Fast accurate fuzzy matching

A computer-implemented technique for fuzzy matching. This works quickly yet accurately to determine if a given computer-readable record is represented, by exact match or pretty close match, in a large collection of computer-readable records. Further tools may be provided to assess the character of the match.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention concerns digital data processing software and/or hardware to quickly yet accurately determine if a given computer-readable record is represented, by exact match or pretty close match, in an existing collection of computer-readable records.

2. Description of the Related Art

“Fuzzy matching” refers to a well known assortment of techniques to determine whether searched strings approximately match some given pattern string. These techniques are also known by other names such as approximate matching, inexact matching, fuzzy string searching, etc. Each implementation of fuzzy matching uses some similarity function, that is, an algorithm for determining whether the input and searched strings are similar to each other. One common similarity function is Levenshtein distance, and another is n-gram distance.

The commercial market already contains various products that employ fuzzy matching. One example is the Hunter software of Experian, which is intended to detect fraud in the customer acquisition process. Another example is found in the products of Identity Systems, formerly known as Search Software America, which provides various software products aimed at searching, finding, matching, and grouping identity data, regardless of structure, format, location, duplication, omissions or errors. Other examples are found in the products of IBM Entity Analytic Solutions (EAS), which aims to help organizations recognize the entities with which they are doing business. EAS is said to provide real time recognition and resolution, in context with existing business applications.

Although these systems provide certain benefits, Fair Isaac Corporation is interested in improving the performance and efficiency of fuzzy matching programs, since various Fair Isaac products do (or could) beneficially employ fuzzy matching. Fair Isaac has identified some areas of possible focus and some potential shortcomings of existing technology. For one, the computational complexity and cost associated with a brute-force, field by field fuzzy matching against each individual record in a reference database (e.g., a fraud file) is prohibitive in practice. Second, existing approaches can give misleading results when strong matches occur on weak data (such as the strong or identical match of a common first name such as “John”). Third, better control over the manner of fuzzy matching is desired. Fourth, the existing approaches are not as modular and easily extensible as some might like.

In view of these concerns, the existing fuzzy matching products are not completely adequate for all intended applications.

SUMMARY OF THE INVENTION

Broadly, the present disclosure concerns a new technique for fuzzy matching. This works to quickly yet accurately determine if a given computer-readable record is represented, by exact match or pretty close match, in a large collection of computer-readable records. Further tools may be provided to assess the character of the match.

The teachings of this disclosure may be implemented as a method, apparatus, logic circuit, storage medium, or a combination of these. This disclosure provides a number of other advantages and benefits, which should be apparent from the following description.

DETAILED DESCRIPTION

1. Overall Structure

One aspect of the present disclosure concerns a computer system (100) with various components that are configured to perform expedited, accurate fuzzy matching. Broadly, the system100includes a computer106, storage120, user input/output (I/O)108, and other computers112. Various interfaces110interconnect these components. Of course, other components may be added to these, but this architecture provides a starting point to illustrate the primary features of this disclosure.

The computer106and its subcomponents106a-106hare data processing entities, and these may be implemented by one or more hardware devices, software devices, a portion of one or more hardware or software devices, or a combination of the foregoing. Some examples are discussed below inFIGS. 2-4. As one example, the computer106may be implemented by a computer workstation, mainframe computer, distributed computing arrangement, personal computer, server, or other computing machine appropriate to the implementation. In this example, the subcomponents106a-106hare implemented by processes, subroutines, object oriented programs, Java Applets, processing threads, machine code, or other software programming of the computer106hardware.

Broadly, each of the key generators106a-106bacts to receive an input string and compute an output key according to a predetermined computational formula. Under this formula, a given input string will always produce the same output key. However, several input strings (with certain types of similarities, as discussed below) will also produce the same output key. Therefore, the key generators106a-106bserve to “fuzzify” input, and provide a “many to one” mapping between input strings and keys. Under this regime, two input strings that produce the same output key must be similar in some ways. These similarities are prescribed by the details of the key computing formula.

In the illustrated example, the key generator106aapplies one fuzzification formula, whereas the key generator106bapplies a different formula. Alternatively, the key generator106bmay be eliminated, in which case the computer106employs a single fuzzification formula. In still another alternative, the system100may include three, four, five, or any greater number of key generators. In one example, the system100may provide a different key generator for each different field (of current or anticipated records). Operational details of the key generators106a-106bare described in appropriate detail below, under the heading “Operation.”

The comparison engine106cproduces an initial pool of candidate records by applying fuzzy matching to a given input record and records of a reference database122. In this operation, the engine106cemploys the key generator106ato produce new keys for the input record, and as to the existing records (122) the engine106cuses previous output of the key generator106astored in a key database124. Operational details of the comparison engine106care described in appropriate detail below, under the heading “Operation.”

As mentioned above, the comparison engine106cproduces a pool of candidate records. The filter106dacts to reduce or “filter” the candidate pool of records by applying various statistical analyses. Operational details of the filter106dare described in appropriate detail below, under the heading “Operation.”

The analyzer106eanalyzes the candidate pool, providing one basis for other components to reduce the candidate pool even further as discussed below. In one embodiment, the analyzer106eapplies a second stage of fuzzy matching, which employs keys previously prepared by the key generator106aor employs the key generator106bto produce completely new keys for both input record and reference records (122). Operational details of the analyzer106eare described in appropriate detail below, under the heading “Operation.”

The scorer106fapplies a predetermined statistical analysis to the filtered, analyzed candidate pool in order to evaluate, score, rank, or otherwise assess these records relative to each other to relative to a predetermined standard. The scorer106fmay be omitted, if appropriate to the intended application. For instance, the end user may not care about scoring. Or, scoring may be unnecessary if the computer106employs a powerful analyzer106ethat limits the final record pool to manageable levels. Operational details of the scorer106fare described in appropriate detail below, under the heading “Operation.”

The pruner106gacts to reduce the candidate pool even further according to output from the analyzer106eand/or the scorer106f. Operational details of the pruner106gare described in appropriate detail below, under the heading “Operation.”

The controller106hdirects the overall operation of the other components106a-106g, coordinating the various processing stages to produce a final result. The controller106hmay perform other functions related to management of the computer106, such as managing peripheral hardware, performing functions unrelated to fuzzy matching, etc.

The system100includes one or more interfaces110to interface the computer106with peripheral hardware and/or software such as user I/O108, other computers112, and digital data storage120. Accordingly, the interfaces110include any of the following, as appropriate to serve the architecture and functionality described herein: telephone modems, cable modems, T1 interface, routers, Ethernet cards, IDE or EIDE units, satellite modems, wireless transceivers, USB interfaces, Fire wire ports, PS/2 ports, key ring networks, local area networks, wide area networks, infrared ports, etc.

Other Computers112

Optionally, the system100may be interfaced with one or more other computers112to receive input and/or provide output. As one option, the user I/O108may be omitted, with this user input/output occurring at one or more remote computes112. Or, the system may work free of user input/output, with input/output coming from external machines112instead of humans.

The storage120provides digital data storage, various embodiments of which are described below in greater detail under the heading “Storage Media.” The storage120includes a reference database122, key database124, and count database126. Any or all of the components120,122,124,126may be provided by relational databases, linked lists, tables, stacks, queues, or any collection of records that is structured and computer-readable, amenable for a computer program to consult and answer queries.

As mentioned below, one function of the system100is to determine if a given input record is represented in a collection of existing records. In this context, the reference database122provides the existing collection of records. The reference database122provides functional rows and columns representing records and fields, respectively. The database122may include virtually any type of data, such as a collection of current customers, past customers, perpetrators of fraud, recipients of a government benefit, etc. Or, apart from people, the database may represent other data concerning machine parts, vehicles, financial transactions, packets of communication, or any other tangible of intangible thing.

The key database124contains keys corresponding to given fields of each record in the reference database122. The “given” fields, namely those having keys, may be some or even all fields in the reference database122. There is not necessarily a one-to-one relationship between fields and keys, as several fields might be used to generate a single key, or a single field might by itself or in combination with other fields be used to generate several keys. The key database124may be incorporated into the reference database122, or it may be a separate database (as shown) linked to the reference database by appropriate pointer, reference, or other link (124a). In the presently illustrated example, the fields of the reference database122having keys are those of a set illustrated by item702inFIG. 7. This is explained in greater detail below.

The count database126contains statistical data concerning (1) the values in the various fields of the reference database122, or (2) the occurrence of the keys in the key database124, or (3) both of these. The nature of the statistical data is discussed in greater detail below. The count database126may be incorporated into the databases122and/or124, or it may be a separate database (as shown) linked to the respective databases by appropriate pointer, reference, or other links (126a,126b).

2. Exemplary Digital Data Processing Apparatus

As mentioned above, data processing entities (such as the computer106and/or its various subcomponents106a-106h) may be implemented in various forms.

Some examples include a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

As a more specific example,FIG. 2shows a digital data processing apparatus200. The apparatus200includes a processor202, such as a microprocessor, personal computer, workstation, controller, microcontroller, state machine, or other processing machine, coupled to a digital data storage204. In the present example, the storage204includes a fast-access storage206, as well as nonvolatile storage208. The fast-access storage206may be used, for example, to store the programming instructions executed by the processor202. The storage206and208may be implemented by various devices, such as those discussed in greater detail in conjunction withFIGS. 3 and 4. Many alternatives are possible. For instance, one of the components206,208may be eliminated; furthermore, the storage204,206, and/or208may be provided on-board the processor202, or even provided externally to the apparatus200.

The apparatus200also includes an input/output210, such as a connector, line, bus, cable, buffer, electromagnetic link, network, modem, transducer, IR port, antenna, or other means for the processor202to exchange data with other hardware external to the apparatus200.

3. Storage Media

As mentioned above, various instances of digital data storage may be used, for example, to provide storage used by the system100(FIG. 1), to embody the storage204and208(FIG. 2), etc. Depending upon its application, this digital data storage may be used for various functions, such as storing data, or to store machine-readable instructions. These instructions may themselves aid in carrying out various processing functions, or they may serve to install a software program upon a computer, where such software program is then executable to perform other functions related to this disclosure.

In any case, the storage media may be implemented by nearly any mechanism to digitally storage machine-readable signals. One example is optical storage such as CD-ROM, WORM, DVD, digital optical tape, disk storage300(FIG. 3), or other optical storage. Another example is direct access storage, such as a conventional “hard drive”, redundant array of inexpensive disks (“RAID”), or another direct access storage device (“DASD”). Another example is serial-access storage such as magnetic or optical tape. Still other examples of digital data storage include electronic memory such as ROM, EPROM, flash PROM, EEPROM, memory registers, battery backed-up RAM, etc.

An exemplary storage medium is coupled to a processor so the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. In another example, the processor and the storage medium may reside in an ASIC or other integrated circuit.

In contrast to storage media that contain machine-executable instructions (as described above), a different embodiment uses logic circuitry to implement processing features such as the computer106and/or any one or more of components106a-106h.

Depending upon the particular requirements of the application in the areas of speed, expense, tooling costs, and the like, this logic may be implemented by constructing an application-specific integrated circuit (ASIC) having thousands of tiny integrated transistors. Such an ASIC may be implemented with CMOS, TTL, VLSI, or another suitable construction. Other alternatives include a digital signal processing chip (DSP), discrete circuitry (such as resistors., capacitors, diodes, inductors, and transistors), field programmable gate array (FPGA), programmable logic array (PLA), programmable logic device (PLD), and the like.

FIG. 4shows an example of logic circuitry in the form of an integrated circuit400.

Having described the structural features of the present disclosure, the operational aspect of the disclosure will now be described. The steps of any method, process, or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by hardware, or in a combination of the two.

A basic implementation of fuzzy matching would be to query every field, or collection of fields, of every single record in the reference database122, and compute a fuzzy match against every field, or collection of fields, of an input record using any one of various known algorithms. One issue with such an implementation is that it scales poorly, and becomes impractical once the reference database reaches a few ten thousand records. To improve on this, the present disclosure performs a pre-filter of the records that is itself fuzzily compared.

The present implementation of fuzzy matching pre-computes a condensed key (or signature or token or other computational output) of the actual reference record fields, or collection of fields, to be matched, and stores these pre-computed keys (herein termed the “fuzzed-up fields” even in the case of collections of fields being used to generate the keys) in a separate table (124). Instead of then having to query the complete reference database122and compute a fuzzy match on each record, the computer106(in one embodiment) can reduce the list of match candidates from the reference database122to only those records that have at least one pre-fuzzed-up field, or collection of fields, (from the key database124) exactly equal to the corresponding fuzzed-up field, or collection of fields, of the application to be matched. Other alternatives to this technique to reduce the list of match candidates are disclosed. Nevertheless, this gain in computational efficiency contributes to making the problem computationally tractable.

Additionally, the computer106can store (along with the fuzzed-up fields) the frequency with how often they occur in the data, since rare keys that match provide more information than matching of a common key. This data (126) is then available to optionally compute a weighted fuzzy match score by considering all fields used for the match, and summing up the field-level match results.

In this description, there are widespread references to the act of computing keys from record fields. It is emphasized that this disclosure contemplates, but does not require, a one-to-one relationship between fields and keys. For instance, several fields might be used to generate a single key. Or, a single field might by itself or in combination with other fields be used to generate several keys. Nevertheless, for ease of reading (but without any intended limitation), the following description references the computation of keys from record fields in terms of the basic case, where there is a one-to-one relationship.

2. Overall Sequence of Operation

Introduction

FIG. 5shows one example of a fuzzy matching sequence500. For ease of explanation, but without any intended limitation, the example500is described in the specific context of the system100(FIG. 1).

Select Fields

In step502, system architects select which fields of the reference database122will have pre-computed, pre-stored keys. Pre-computing keys for various fields of the reference database122will help speed fuzzy matching operations performed later. The designers in step502may select any fields that will be likely involved in fuzzy matching. To cite a simplified example, one set of selected fields may be: first name, last name, social security number, street name, city, street address, and home telephone. InFIG. 7, these fields (for which keys will be pre-computed and pre-stored) are illustrated by the set702. More restrictive sets706,704are used later in the sequence500, as discussed below.

Compute Keys

For ease of discussion, records represented in the reference database122will be referred to as “reference records.” In step504, the controller106hinstructs the key generator106ato compute one key for each of the fields selected in502(i.e., the fields702), for each record in the reference database122. In the simplified example given above, step504starts with a first record in the reference database122, and computes a key for each value of first name, last name, social security number, street name, city, street address, and home telephone fields. Step504re-performs this operation for every remaining record in the reference database124. The controller106hor key generator106astores the computed keys in the key database124.

As an alternative, instead of using the key generator106afor all fields, the controller106hmay use different key generators for different fields or collections of fields.

As a further alternative, instead of performing step502on an existing reference database, step502may be performed from the beginning with an empty reference database122, whenever new records arrive for storage in the reference database124. Or, step502may be performed on the set of records in the reference database122as of a certain date, and then repeated in real time whenever new records arrive.

In order to compute the keys, the key generator106amay use any computational formula such that each key is produced exclusively by a set of input values having certain similarities to each other. Thus, there is a many-to-one mapping from potential input strings to keys. One example of computational formula is the well known Soundex phonetic algorithm (“Russel Soundex”), which is addressed in various issued U.S. patents, technical journal articles, and at least one book. Other examples include the Celko Improved Soundex algorithm, Metaphone algorithm, Double-Metaphone, Daitch-Mokotoff (D-M) Soundex, etc. A further example is the well known NYSIS algorithm developed by the New York State Identification and Intelligence System as an improvement to Soundex. In addition or instead of these, the key generator106amay employ any other key generating formula based on phonics, numerical, double metaphonics, etc. Optionally, different key generators (such as106a-106b) may be invoked to vary the key formula for different fields. For example, one of key generators106a-106bmay act to simply copy-over complete, raw, or original data from the reference record into the key database124in the case of certain specified fields. It may prove helpful, for example, to have such field values available for later processing (as discussed below). Also, there may be fields containing unique data (such as social security numbers) for which is it unnecessary or undesirable to compute keys.

Optionally, at this time, step505may statistically analyze the reference database122in order to compute and store count data in the database126. In this regard, the controller106hin step505computes statistical data concerning the values in the various fields of the reference database122, and/or the keys in the database124. In the case of the reference database122, this statistical data includes a statistical breakdown of field values, for example by count (number of occurrences), frequency of occurrence, percentile, or other assessment of values in some or all fields. This data may be broken down further by field. For example, the count database126may indicate that a given field value (“Oslo”) occurs one hundred times in the “City Name” field of the database122, or that “Oslo” occurs in the “City Name” field of twenty five percent of all records in the reference database122. In the case of the key database124, this statistical breakdown includes a statistical breakdown of the keys. For example, the count database126may indicate that a given key occurs one thousand times in the database124, or that the given key constitutes forty percent of all keys in the database124, or that the given key occurs four hundred times as to a given reference field.

Although task505may be performed from time to time in batch, alternatives are to calculate statistical data from the beginning with an empty reference database122, calculate statistical data whenever new records arrive for storage in the reference database124, or perform statistical analysis on a set of records in the reference database122as of a certain date, and then repeat in real time for new records that arrive.

Arrival of Input Record

In step506, the computer106(and more particularly, the controller106h) receives a record to evaluate. For ease of explanation, this is interchangeably referred to as the “given” record or “input” record. This record may come from the user I/O108, for instance if the record is entered or submitted or identified by a human user such as a customer, system administrator, software user, or other person. Alternatively, the record may be submitted or entered or identified by a remote computer112. As another alternative, the computer106itself may identify the record. For example, users may enter new records into a cache (not shown), and the computer106takes up cached records for processing in order of entry or another order. Or, the computer106may self-select records from the reference database122for screening or evaluation, unrelated to input of any new record. Step506may observe any of these approaches, or a combination.

Fuzzy Matching

In step507-508, the computer106performs fuzzy matching upon the input record. In one sense, the pre-computation of keys (from step504) may be considered an early party of fuzzy matching, too. In step507, responsive to receiving the input record (506), the controller106hdirects the key generator106ato compute keys for fields of the given record507. To expedite the overall process500, the key generator106aonly computes keys for a limited set of fields of the given record. For example, step507may limit key computation to first name, last name, and social security number fields.FIG. 7illustrates this limited set at704, and shows that this set704is a smaller subset of the entire set of fields (702) for which keys were computed and stored (504) in the key database124. By limiting fuzzy matching to the field set704(instead of the entire set702) this expedites the overall process500.

In step508, the comparison engine106ccompares the keys for the input record (computed in step507) with the corresponding, pre-computed keys (stored in124) of each reference record. In conformance with the limited fuzzy matching strategy, and since step507only computed keys for a limited set (704) of fields, step508only compares keys of the limited set704of fields as between the given record and the reference records. In the previously introduced example, step508will compare the keys for the given record's first name, last name, and social security number fields to each reference record's respective keys for those same fields, looking to see if the keys match identically.

If at least one key of the given record matches a key for the same field of the reference record, that reference record is added to a “candidate pool.” For example, if the given record's key for last name matches a particular reference record's key for last name, the reference record is added to the candidate pool. In one embodiment, where each reference record is given a unique record number in the database122, addition of a record to the candidate pool may be carried out by recording the record's number in a list.

Filtering

In step509, the filter106dhones the candidate pool by removing records from the candidate pool whose fuzzy match with the input record is weak (according to predetermined criteria). Broadly stated, this is carried out by statistically analyzing the nature of the matches found in508. For greater speed, this process utilizes the statistical data stored in the count database126. Filtering (509) is discussed in greater detail below, with reference to the sequence600(FIG. 6).

Analysis

After filtering (step509), the next step in the process500is to analyze the candidate pool (510). Broadly, in step510the analyzer106eperforms more comprehensive fuzzy matching than was conducted in steps507-508. Now that the process500has narrowed the reference records down to a candidate pool (step508) and further filtered that pool (step509), it is possible to perform more comprehensive or comprehensive fuzzy matching without great sacrifice in computational effort. Thus, in step510the analyzer106eperforms fuzzy matching as between the input record and the records of the filtered and pruned candidate pool. Renewed fuzzy matching (510) may be carried out in various ways, two of which are described as follows.

Step513describes one exemplary technique. Here, the last part of the previously conducted fuzzy matching is repeated, but applied to a broader set of fields. In other words, and as compared to the fuzzy matching of steps507-508(based upon fields of the set704shown inFIG. 7), the renewed fuzzy matching of step513involves a greater number of fields (for example, the set706or even the set702). Advantageously, then, the field set706is substantially greater than the field set704. In the present example, the field set706includes first name, last name, social security number, street name, and city. Fuzzy matching of step513employs the same keys computed in steps504,507along with additional keys that must be computed for the input record (for fields not having keys computed in507but are part of the fuzzy matching513). For computing these added keys, the key generator106ais used.

In contrast to step513, steps511-512describe an alternative technique for technique. This technique employs the key generator106binstead of the key generator106a. For each record in the filtered pool, step511retrieves the complete, original, or raw record from the reference database122(or at minimum, the complete, original, or raw values of the fields706). Then, step512computes new keys upon these field and the corresponding fields of the input record. Using these, step512performs fuzzy matching on all fields of the field set706, as between the input record and the records of the filtered candidate pool. In this example, then, the approach of steps511-512is enhanced relative to the earlier fuzzy matching (steps504,507,508) because it considers a greater number of fields (706or even702) than the field set (704) used in steps504,507,508. Additionally, the approach of steps511-512is further enhanced because it employs an enhanced fuzzy matching formula, namely that of the key generator106brather than106a. As to the fuzzy matching formula, this may use a similar key formula as discussed before (e.g., Soundex, NYSIS, etc.) but with different resolution, bit sampling, comparison or combination of multiple formulas, etc. Alternatively, the fuzzy matching formula of step512may conduct analysis unrelated to keys, with one example being the Levenshtein edit distance.

Instead of using the key generator106bfor all fields in step512, the following is one alternative. As an example, this may be used in the embodiment (described above) where step504used different key generators for different fields or collections of fields. Like step502, step512may use different key generators for different fields. However, in this example, the set of key generators used here is different than the set of key generators used in step514.

Ultimately, step510produces a list of records referred to as a final candidate pool.

Scoring

Next, in step514the scorer106fscores the candidate pool according to the analysis of step510. Broadly, the scorer106fanalyzes records of the analyzed candidate pool to evaluate, score, rank, or otherwise assess these records relative to each other or to relative to a predetermined standard. In one example, scoring may consider factors such as inverse term frequency, i.e., terms that occur more often are given a lower score contribution than terms that occur infrequently with more significance. As a different example, step514may act to compute a weighted fuzzy match score by considering all fields used for the match, and summing up the field-level match results. Scoring may be implemented using these, or a combination of these, or a variety of different known techniques described in the numerous patents and patent publications of Fair Isaac Corporation. Optionally, step514may also produce a reason code, indicating an explanation for a given record's score.

Step514is optional, however, and may omitted without departing from this disclosure. As a further alternative, scoring514may performed at a different occasion in the sequence500. One example is between steps508-509, in which case operation509may utilize scores in performing filtering. As another example, scoring may be performed between steps509-510, or during step510. Thus, step510may utilize scoring information in performing its analysis.

After the optional scoring (step514), the sequence500presents two options514a-514b. In option514a, the pruner106gprunes the candidate pool (step515) according to output from the analysis (step510) and scoring (step514, if applicable). To provide some examples, some examples of pruning include setting a score threshold and removing all candidates receiving a score below it, or setting several score thresholds to be used based on which fields did match, or limiting the absolute number of candidates and removing all but the highest scoring ones in case this number was exceeded, or any combination of these, or no pruning at all.

After step515, the controller106hin step516provides an output identifying the records of the pruned pool and/or the computed scores of the pruned records. Also in step516, the controller106hrenders this output to a site such as the user I/O108or another computer112. As an alternative, the controller106hmay cache the output for retrieval on demand by a user, remote machine, or automated process.

In contrast to option514a, in option514bthe sequence500skips step514aand performs step516as discussed above, only with regard to the un-pruned candidate pool.

FIG. 6shows one exemplary process600for conducting filtering as per step509(FIG. 5). The process600is carried out by the filter106d. The process600works by “picking” certain candidates to retain in the candidate pool and excluding the rest.

As mentioned above, steps507-508perform fuzzy matching upon the input record as to the reference records. If at least one key of the input record matches a key for the same field of the reference record, that reference record is added to the candidate pool. For example, if the given record's key for last name matches a particular reference record's key for last name, the reference record is added to the candidate pool. Accordingly, a reference record may be qualified to enter the candidate pool for numerous reasons, i.e., multiple fields that demonstrate a fuzzy match with a corresponding field of the input record.

With this in mind, step602begins with a sub-group of the candidate pool. Namely, step602begins with a set of all reference records that entered the candidate pool (step508) due to a fuzzy match occurring in a first field. The first field may be selected on any appropriate basis, such as arbitrarily, field order, alphabetic order of field names, etc. The field under discussion at any one iteration of the process600is referred to as the “current” field (for brevity the term “current field” is used even in case of a collection of fields).

Next, step604considers frequency data for the current field of the input record. As mentioned above, the sub-group of candidates under discussion qualified for the candidate pool for at least the following reason—the current field of the reference record was a fuzzy match to the current field of the input record. This is why the current field of the input record is examined in step604. More particularly, step604references the database126to determine the count of the data from the current field of the input record.

For example, where the current field is a “City” field, step604may reveal that the current field of the input record is “Oslo” and this occurs in 1,263 records of the reference database122. If the input record also qualified for the candidate pool based on a fuzzy match of another field, or another collection of fields, this is irrelevant for the present analysis.

Next, step606asks whether the number from step604is greater than a prescribed threshold (“NMAX”). If so, step607refrains from “picking” the current sub-group of reference records. In the present example, NMAX is set to one thousand. Thus, step606is satisfied in the present example because Oslo occurs in over one thousand reference records.

In a different example, if the threshold (NMAX) were set to 5,000, then step606would not be satisfied, and the process600would advance to step608. Step608asks whether the number from step604is less than a second threshold (NKEEP). The threshold NKEEP is set sufficiently low (e.g., fifty records) so as to identify highly meaningful matches. Thus, the filter gives precedence to less common matches, which are statistically more relevant. In the present case, step608is not satisfied since Oslo occurs in more than fifty records. Thus, step612(discussed below) is performed. On the other hand, if Oslo occurred in less than fifty records, step608would be satisfied, and step610would occur. Step610“picks” the current sub-group of reference records. The utility of “picking” record groups is discussed in greater detail below.

As mentioned above, if step608is not satisfied, then step612occurs. Step612reduces the candidate pool belonging to the current field given that the size of this pool is larger than NKEEP but smaller or equal to NMAX. This reduction may be based on rules or statistical analysis, may be random, or a combination thereof. Some examples of this include:(a) Random selection of NKEEP many records,(b) Sort the records in increasing order by how many other fields have matched as well.(c) Assuming that all fields are sorted by a preset importance value, sort the records from the pool by whether they match also in the most important other field, or in the second-most important other field, etc.(d) Using pre-assigned weights for each field, for each record in the candidate pool compute a match score by summing the weights of those fields that match. Then sort the candidate records by their achieved sum of weights.(e) Require that the candidate records also match at least a certain number (e.g., one or two) of other fields.(f) Require that the candidate records also match a specific selection of (e.g., one or two) other fields.
For methods (b) through (f), take the top NKEEP many records. Tie breaks may be resolved randomly, or with the use of some of the other rules.

After any of steps607,610,612a, or612bcomplete, step616asks whether there are any remaining candidate sub-groups to consider. Namely, step616asks whether the process604-612bhas progressed through all fields, or all collection of fields, and corresponding keys used in step508. If not, step618advances to the next candidate group (based upon the next field of fuzzy matching), and then re-performs step604on this basis.

On the other hand, when step616finds that all candidate groups have been considered, step620carries out a filtering operation. Namely, step620creates a filtered candidate pool consisting of those reference records of the candidate pool that were “picked” in steps610or612a. The process600ends in step622.

The disclosed system100and processes500,600may be applied in a number of different contexts. Without any intended limitation, the following is a sampling of different applications.

In one example, these techniques are employed in fraud screening. During processing of credit applications, for example, there is usually some kind of fraud screening. Credit applications are often compared against previously identified frauds. Since the fraudsters are aware of this customary check, they tend to vary their application data slightly, maybe change a spelling here or there, or change some components of the address or other information. The challenge is then to still match the application against the fraud file in a non-exact, or fuzzy, manner. Of course, the matching of applications does not have to be limited to matching against the previously identified fraud file, one can also match against previous applications, or the customer master file. In fact, fuzzy matching does not necessarily concern applications, but could be used for any kind of string data.

In another context, these techniques may be employed when people apply for government assistance, to conduct same entity analysis to see if they are already receiving assistance or another disqualifying benefit. In another context, a corporation may employ the system100to periodically conduct redundancy analysis in its databases, by recognizing records that (despite trivial differences) are really represent the same person or entity. For instance, a club or association may reduce magazine mailing costs by periodically screening its subscription list to identify cases where the same people mistakenly appear twice, causing two magazines to be sent to the same person. Furthermore, the system100is useful in numerous computing applications that seek to perform some type of same entity analysis.

C. Other Embodiments

While the foregoing disclosure shows a number of illustrative embodiments, it will be apparent to those skilled in the art that various changes and modifications can be made herein without departing from the scope of the invention as defined by the appended claims. Accordingly, the disclosed embodiment are representative of the subject matter which is broadly contemplated by the present invention, and the scope of the present invention fully encompasses other embodiments which may become obvious to those skilled in the art, and that the scope of the present invention is accordingly to be limited by nothing other than the appended claims.

All structural and functional equivalents to the elements of the above-described embodiments that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the present claims. Moreover, it is not necessary for a device or method to address each and every problem sought to be solved by the present invention, for it to be encompassed by the present claims. Furthermore, no element, component, or method step in the present disclosure is intended to be dedicated to the public regardless of whether the element, component, or method step is explicitly recited in the claims. No claim element herein is to be construed under the provisions of 35 USC 112, sixth paragraph, unless the element is expressly recited using the phrase “means for” or, in the case of a method claim, the phrase “step for”.

Furthermore, although elements of the invention may be described or claimed in the singular, reference to an element in the singular is not intended to mean “one and only one” unless explicitly so stated, but shall mean “one or more”. Additionally, ordinarily skilled artisans will recognize that operational sequences must be set forth in some specific order for the purpose of explanation and claiming, but the present invention contemplates various changes beyond such specific order.

In addition, those of ordinary skill in the relevant art will understand that information and signals may be represented using a variety of different technologies and techniques. For example, any data, instructions, commands, information, signals, bits, symbols, and chips referenced herein may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, other items, or a combination of the foregoing.

Moreover, ordinarily skilled artisans will appreciate that any illustrative logical blocks, modules, circuits, and process steps described herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.