Package: csvmatch
CSV MatchFind fuzzy matches between CSV files. Based onTextmatch.Fuzzy matching is the art and science of connecting up bits of information that are written differently but represent the samething– such as a person or company.InstallingThe best way to install is withpipx, which can be installed with Homebrew:brew install pipx. Then:pipx install csvmatchGetting startedThe best way to approach fuzzy matching with CSV Match is to start with an exact match. From there, you can incrementally improve the results by telling CSV Match about relevant information that should be taken into account and irrelevant information that should be disregarded. Experiment with different approaches. It is helpful to know what the data looks like, and how it has been collected.The input files can be in CSV or Parquet format. The output results will be a CSV file.Exampledata1.csv:nameplacecodenameSam CollinsVietnamnoneRoy BlandLondonSoldierGeorge SmileyLondonBeggermanBill HaydonLondonTailorPerçy AllélíneLondonTinkerKretzschmarHamburgnoneOliver LaconLondonnoneJim PrideauxSlovakianonePeter Guillam EsqBrixtonnoneToby EsterhaseViennaPoormanConnie SachsOxfordnonedata2.csv:Person NameLocationMaria Andreyevna OstrakovaRussiaKonny SaksOxfordTony EsterhaseViennaPeter GuillamBrixtonMr Jim PrideauxCzech RepublicLacon OliverCambridgeClaus KretzschmarHamburgRichard BlandLondonRoy RodgersRomaniaPercy AllelineLondonBill-HaydonLondonGeorge SMILEYLondonRoy BlandUKSam CollinsVietnamTo run an exact match on thenamecolumn from the first file againstPerson Namefrom the second:$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    > matches.csvThe resulting matches include the two names which are written exactly the same:nameplacecodenamePerson NameLocationRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamMatches aremany-to-many, ie. it is possible for one row in the first file to match several rows in the second, and vice-versa.[!TIP]
There is a tradeoff between false negatives and false positives – it is often better to have some incorrect matches in your results that can be manually checked afterwards than to have correct matches missing.UsageCSV Match only requires two arguments, the first file followed by the second. All others are optional, described below:FieldsThefields1andfields2arguments accept one or more column names that should be used for the match. If the column name has a space it should be wrapped in quotes. These should be in the same order for both files – the first column specified for the first file will be compared against the first column specified for the second file, and so on. Defaults to comparing all columns. This flag can be specified multiple times forblocking.ExampleTo match on thenameandplacecolumns from the first file againstPerson NameandLocationfrom the second:$ csvmatch \
  data1.csv \
  data2.csv \
  --fields1 name place \
  --fields2 'Person Name' Location \
  > matches.csvThe resulting matches include the single name-place pair which is the same in both files:nameplacecodenamePerson NameLocationSam CollinsVietnamnoneSam CollinsVietnamIgnoranceTheignoresargument accepts one or more characteristics which should be disregarded for two records to be considered a match. This flag can be specified multiple times forblocking.Combining different forms of ignorance can be quite powerful. The order in which you specify them is not significant.caseignores how text is capitalised.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore case \
    > matches.csvThe resulting matches include George Smiley, whose surname is in all-capitals in the second file:nameplacecodenamePerson NameLocationGeorge SmileyLondonBeggermanGeorge SMILEYLondonRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamnonalphaignores anything that isn't a number or a letter.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore nonalpha \
    > matches.csvThe resulting matches include Bill Haydon, whose name is written with a hypen in the second file:nameplacecodenamePerson NameLocationBill HaydonLondonTailorBill-HaydonLondonRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamnonlatinignores nonlatin characters – so a character such as anéwill be seen as if it were ane.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore nonlatin \
    > matches.csvThe resulting matches include Percy Alleline, whose name is written with several diacritics in the first file:nameplacecodenamePerson NameLocationPerçy AllélíneLondonTinkerPercy AllelineLondonRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamwords-leadingignores all words except the last. This is useful for matching on surnames only.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore words-leading \
    > matches.csvThe resulting matches include Toby and Tony Esterhase, Jim Prideaux and Mr Jim Prideaux, Kretzschmar and Claus Kretzschmar, as well as Roy and Richard Bland:nameplacecodenamePerson NameLocationToby EsterhaseViennaPoormanTony EsterhaseViennaJim PrideauxSlovakianoneMr Jim PrideauxCzech RepublicKretzschmarHamburgnoneClaus KretzschmarHamburgRoy BlandLondonSoldierRichard BlandLondonRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamwords-tailingignore all words except the first.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore words-tailing \
    > matches.csvThe resulting matches include Peter Guillam Esq and Peter Guillam, Roy Bland and Roy Rodgers, as well as the two capitalisations of George Smiley:nameplacecodenamePerson NameLocationPeter Guillam EsqBrixtonnonePeter GuillamBrixtonRoy BlandLondonSoldierRoy RodgersRomaniaGeorge SmileyLondonBeggermanGeorge SMILEYLondonRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamwords-orderignores the order in which the words are given. This is useful for matching names given surname-first with those given forename-first.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore words-order \
    > matches.csvThe resulting matches include Oliver Lacon, whose name is written surname-first in the second file:nameplacecodenamePerson NameLocationOliver LaconLondonnoneLacon OliverCambridgeRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamtitlesignores common English name prefixes such as Mr and Ms. There is a full list of these titleshere.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore titles \
    > matches.csvThe resulting matches include Jim Prideaux, who has the title 'Mr' in the second file:nameplacecodenamePerson NameLocationJim PrideauxSlovakianoneMr Jim PrideauxCzech RepublicRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamregexignores terms specific to your data using a given regular expression. This is specified inline:regex=EXPRESSION.ExampleTo use the regular expressionEsq$to ignore the word 'Esq' where it appear at the end of a value:$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore 'regex= Esq$' \
    > matches.csvThe resulting matches include Peter Guillam, who has the name suffix 'Esq' in the first file:nameplacecodenamePerson NameLocationPeter Guillam EsqBrixtonnonePeter GuillamBrixtonRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamMethods & thresholdsThemethodsargument accepts one or more methods. This lets you specify the algorithm which is used to do the matching. Multiple methods are used forblocking.There are three different categories of method:Comparedmethods work by comparing every row from the first file with every row from the second, producing a number that represents the degree of the match. This means the amount of time required to run a match grows exponentially with the size of the input files. However, they are still useful for larger matches when usingblocking.Appliedmethods transform text into a different representation before they are matched up. These methods are quicker than compared ones, though no meaningful matching degree number is generated – either they match or they don't.Custommethods have their own individual approach. CSV Match only has one,Bilenko. It generates a matching degree number.For those matching methods that generate a matching degree number there is then a threshold filter for any two records to be considered to be a match – you can adjust this with the--thresholdargument, which accepts one or more numbers between 0.0 and 1.0, defaulting to 0.6. Multiple thresholds are also used forblocking.You can also include the matching degree number as a column by specifying it in theoutputs.[!WARNING]
When working with names of people, exact matches, even when other pieces of information such as birthdays are included, are not a guarantee that the two names actually refer to the same human. Furthermore, the chance of a mismatch is unintuitively high – as illustrated bythe birthday paradox.literalis the default – it matches strings exactly, after ignored characteristics have been taken into account.levenshteinUses theDamerau-Levenshteinstring distance metric that counts the number of changes that would have to be made to transform one string into another. Performs compared matching. Where two strings are of different lengths the longer string is used as the denominator for the threshold filter. Good at picking up typos and other small differences in spelling.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --method levenshtein \
    > matches.csvThe resulting matches include various names with small typographical differences, though the most emblematic of this matching method would be Toby and Tony Esterhase:nameplacecodenamePerson NameLocationSam CollinsVietnamnoneSam CollinsVietnamRoy BlandLondonSoldierRoy BlandUKGeorge SmileyLondonBeggermanGeorge SMILEYLondonBill HaydonLondonTailorBill-HaydonLondonPerçy AllélíneLondonTinkerPercy AllelineLondonKretzschmarHamburgnoneClaus KretzschmarHamburgJim PrideauxSlovakianoneMr Jim PrideauxCzech RepublicPeter Guillam EsqBrixtonnonePeter GuillamBrixtonToby EsterhaseViennaPoormanTony EsterhaseViennajarouses theJaro-Winklerstring distance metric that counts characters in common, though it considers differences near the start of the string to be more significant than differences near the end. Performs compared matching. It tends to work better than Levenshtein for shorter strings of text.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --method jaro \
    > matches.csvThe resulting matches includes many more matches thanlevenshtein, though also many more false positives:nameplacecodenamePerson NameLocationSam CollinsVietnamnonePercy AllelineLondonSam CollinsVietnamnoneSam CollinsVietnamRoy BlandLondonSoldierRichard BlandLondonRoy BlandLondonSoldierRoy RodgersRomaniaRoy BlandLondonSoldierBill-HaydonLondonRoy BlandLondonSoldierRoy BlandUKGeorge SmileyLondonBeggermanGeorge SMILEYLondonBill HaydonLondonTailorBill-HaydonLondonBill HaydonLondonTailorRoy BlandUKPerçy AllélíneLondonTinkerPeter GuillamBrixtonPerçy AllélíneLondonTinkerPercy AllelineLondonKretzschmarHamburgnoneClaus KretzschmarHamburgJim PrideauxSlovakianoneMr Jim PrideauxCzech RepublicPeter Guillam EsqBrixtonnonePeter GuillamBrixtonToby EsterhaseViennaPoormanTony EsterhaseViennaToby EsterhaseViennaPoormanRoy RodgersRomaniaConnie SachsOxfordnoneKonny SaksOxfordmetaphoneuses theDouble Metaphonephonetic encoding algorithm to convert words into a representation of how they are pronounced. Performs applied matching. Tends to work better for data which has been transcribed or transliterated.Example$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --method metaphone \
    > matches.csvThe resulting matches includes those with nonalphanumeric differences, as well as Connie Sachs and Konny Saks, names written quite differently that would be pronounced the same:nameplacecodenamePerson NameLocationConnie SachsOxfordnoneKonny SaksOxfordRoy BlandLondonSoldierRoy BlandUKGeorge SmileyLondonBeggermanGeorge SMILEYLondonSam CollinsVietnamnoneSam CollinsVietnamBill HaydonLondonTailorBill-HaydonLondonbilenkousesDedupe, a library built by Forest Gregg and Derek Eder based on the work of Mikhail Bilenko that will ask you to train it by asking whether different pairs of records should match. The information you give it is then extrapolated to match up the rest of the file. The more examples you give it, the better the results will be. At minimum, try to provide 10 positive matches and 10 negative matches. Performs custom matching.BlockingBlocking is the approach of performing multiple matches, with subsequent matches only applying to the subset of matches resulting from the previous match. This can make matches both quicker and more precise. This is an advanced topic, and can be ignored if you are happy with the quality of matches and are dealing with smaller files.In a 'regular' match, you are really just matching using a single block. Each block is defined by: a list of fields for each file, a list of ignores, a method, and a threshold. To perform a blocked match CSV Match needs to know each of these things for each block. You specify these through list arguments, or through outer lists for those arguments where the block requires a list itself. If you specify one of these things less than the total number of blocks – such as if you had two blocks, but specified the threshold once – that value will then be used for all subsequent blocks.ExampleTo specify a first block that does a case-insensitive literal match on surnames, then a second block performing a Levenshtein match on forenames:$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --ignore case words-leading \
    --ignore words-tailing \
    --method literal levenshtein \
    > matches.csvnameplacecodenamePerson NameLocationKretzschmarHamburgnoneClaus KretzschmarHamburgGeorge SmileyLondonBeggermanGeorge SMILEYLondonRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamOutputsTheoutputargument accepts a list of column names which should appear in the output, each prefixed with a number and a dot indicating which file that field is from. They are case-sensitive, and can be in any order you desire. It defaults to all columns in the first file, followed by all columns in the second.There are some special column definitions:1*and2*expand into all columns from the first and second files respectively, anddegreewill add a column with the matching degree number.ExampleTo include every column from the second file, followed by thecodenamecolumn from the first, followed by the matching degree:$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --method levenshtein \
    --output '2*' 1.codename degree \
    > matches.csvPerson NameLocationcodenamedegreeSam CollinsVietnamnone1.0Roy BlandUKSoldier1.0George SMILEYLondonBeggerman0.61538464Bill-HaydonLondonTailor0.90909094Percy AllelineLondonTinker0.78571427Claus KretzschmarHamburgnone0.64705884Mr Jim PrideauxCzech Republicnone0.8Peter GuillamBrixtonnone0.7647059Tony EsterhaseViennaPoorman0.9285714Join typesThejoinargument takes a string that indicates what other nonmatching records should be included in the output. Aleft-outerjoin will return everything from the first file, whether there was a match or not, aright-outerto do the same but for the second file, and afull-outerto return everything from both files. Where two rows didn't match the values will be blank. Defaults to aninnerjoin, where only successful matches are returned.ExampleTo include all rows from the first file, but only those that match from the second:$ csvmatch \
    data1.csv \
    data2.csv \
    --fields1 name \
    --fields2 'Person Name' \
    --join left-outer \
    > matches.csvnameplacecodenamePerson NameLocationRoy BlandLondonSoldierRoy BlandUKSam CollinsVietnamnoneSam CollinsVietnamGeorge SmileyLondonBeggermanBill HaydonLondonTailorPerçy AllélíneLondonTinkerKretzschmarHamburgnoneOliver LaconLondonnoneJim PrideauxSlovakianonePeter Guillam EsqBrixtonnoneToby EsterhaseViennaPoormanConnie SachsOxfordnone