Patent ID: 7590624

Claim:
A method for determining if the data of a data set includes duplicate data elements, the method comprising: a) receiving a data set having a plurality of data elements; b) dividing each data element into N data segments defining thereby N data keys for each data element; c) obtaining intermediate data values for each of the data elements; d) sorting the data elements using at least the intermediate data values wherein said sorting includes: conducting a first sorting of the data elements using the intermediate value for each data element; identifying data elements that have no matching intermediate values as unique data elements; identifying data elements having the same intermediate values as other data elements as potential duplicate data elements; and searching the potential duplicate data elements using the data keys to determine if any of the potential duplicate data elements match all the data keys wherein the searching comprises: 1) segregating the potential duplicate data elements into groups of data elements wherein each group is comprised of potential duplicate data elements having a same intermediate value; 2) beginning with a first group of the groups of data elements; said data key sorting comprises: a) comparing a first data key for each data element in the first group; b) where the first data key of a data element does not match any other first data keys for the first group of data elements the data element is identified as unique and no further processing is performed on the data element; c) where the first data key of a data element matches the first data key of another data element of the first group, the data element remains identified as a potential duplicate data element and further processing is performed on the data element; d) the remaining potential duplicate data elements are subject to further processing as follows: i) comparing a next data key for each potential data element in the first group, where the potential duplicate data element does not match any of the next data keys for the potential duplicate data elements of the first group of data elements, the potential duplicate data element is identified as unique and no further processing is performed on the data element; and ii) if the next data key of the potential duplicate data element does match with another next data key for the potential duplicate data elements of the first group of data elements, the potential duplicate data element remains identified as a potential duplicate data element; iii) operations d)(i)-d(ii) are repeated recursively until no potential duplicate data elements remain or until at least two potential duplicate data elements are identified that match all N data keys and said data elements are identified as duplicates; 3) repeating operations 2)(a), (2)(b), 2)(c), 2)(d)(i)-(iii) for each of the remaining groups until all duplicate data elements, if any, are identified; and e) identifying duplicate data elements.