Patent ID: 9720986
Date: 2017-08-01
CPC Classifications: G06F

Claim:
1. A method for integrating data into a database, the method comprising: storing data comprising a plurality of records which each comprise a plurality of attributes; analyzing a sample of records from the plurality of records by: identifying duplicate pairs of records in the sample records; and analyzing each attribute of each record of the duplicate pairs of records to identify a respective attribute condition which is indicative that the pairs of records are duplicates; comparing each attribute of a record with the respective attribute condition and, if the attribute satisfies the attribute condition, allocating the record to a disjoint group which comprises records with an attribute that satisfies the same respective attribute condition; identifying duplicate pairs of records in the records in each disjoint group; identifying duplicate pairs of records in records that are not allocated to a disjoint group; consolidating each duplicate pair of records into one consolidated record and storing the consolidated record in an integrated database; and identifying distinctive features of records; wherein identifying duplicate pairs of records comprises: analyzing the attributes of the records identifying candidate pairs of records, each candidate pair comprising a two records which have at least one similar attribute generating a similarity value which is indicative of the level of similarity between the two records in each candidate pair identifying each candidate pair with a similarity value above a predetermined level as a duplicate pair of records, wherein the similarity value is a conditional probability value which is indicative of the probability of the two records in a candidate pair being duplicates of one another; identifying a learning rule used to identify duplicate pairs of records; wherein the learning each rule comprises calculating a probability distribution by comparing the attributes of known duplicate records with a random sample of pairs of records, and wherein the calculation is based on a Bayes' rule.