Patent Document ID: 9208448
Application ID: 13689467

Base Claim:
1. A method for learning a data format, the method comprising: inputting an initial description of a data format and a batch of data comprising data in a new data format not covered by the initial description, instructions to use the first description to parse the records in the data source; discarding records in the input data that parse successfully, instructions to collect records that fail to parse, instructions to accumulate a quantity, M of records that fail to parse, returning a modified description that extends the initial description to cover the new data, transforming the first description, D into a second description D′ to accommodate differences between the input data format and the first description D by introducing options where a piece of data was missing in the input data and introducing unions where a new type of data was found in the input data; and using a non-incremental format inference system to infer descriptions for the aggregated portions of input data that did not parse using the first description D; and ranking the parses by a metric that measures their quality and return only the top quantity k of the parses metric comprises a triple: m=(e, s, c), where e is a quantity of errors, s is a quantity of characters skipped during Sync token recovery, and c is a quantity of characters correctly parsed.

---

Claim 4:
4. The method of claim 1 , the method further storing data in a learn node data structure embedded in the computer readable medium containing data indicating an accumulation of the data that did not parse using the first description that needs to be learned.