Patent ID: 8051060

Claim:
A method executed by a processor, on a data file having records, where each record is delimited by a record delimiter information unit, R, and each record has a plurality of fields that are delimited by a field delimiter information unit, F, and where said information units R and F are not known, for identifying said information unit R and said information unit F, where R and F are taken from a set of permissible delimiters, comprising the steps of: choosing a pair of candidate information units, R,F; parsing said data file based on said candidate information units, R,F, to result in records, wherein each of the records has a number of fields; computing a field-wise generalized entropy of said data file; operating on said field-wise generalized entropy value; returning to said step of choosing, if a pair of information units, R,F, exists in said set of permissible delimiters that has not been considered as candidate information units, R,F; and choosing the information unit pair R,F, as delimiters R and F, respectively, that yielded smallest of said field-wise generalized entropy value, to transform said data file into a file of cognizable records.