Package: mergic
mergicworkflow support for reproducible deduplication and mergingSay you have a bunch of strings, and some of them are different but refer to the same thing. Maybe it’s just some long list, maybe it’s the contents of two key columns from data sets that you’d like to merge.David Copperfield
Lance Burton
Dave Copperfield
Levar BurtonHere’s what you can do withmergic:Givemergicall your identifiers, one per line. If they are in a file calledoriginals.txt:mergiccalcoriginals.txtYou will see output about the possible groupingsmergiccan produce based on its default distance function. (It’s easy to use a custom distance function—see below.)num groups, max group, num pairs, cutoff
----------------------------------------
         4,         1,         0, -0.909090909091
         3,         2,         1, 0.0909090909091
         2,         2,         2, 0.25
         1,         4,         6, 0.714285714286There are 4 groupings thatmergiccan make with the given distance function, ranging from a group for every name to a single group for all four names.Thenum groupsis the number of groups thatmergiccan make with the givencutoff.Themax groupis the number of items in the largest group for that grouping.Thenum pairscolumn indicates the number of within-group links that correspond to the grouping. In some sense this represents how much work it is to check all the linkages thatmergicis making. Thinking in groups is much better than thinking in individual pairwise comparisons, but the metric is useful for summarizing how much linking is happening.Thecutoffdetermines which pairs of items are put in the same group. If the distance between two items is equal to or less than thecutoff, those items will be grouped together.Select a cutoff to produce the grouping you would like to see. If you would like to use the cutoff 0.3 and put the results in a file calledgrouping.json:mergicmakeoriginals.txt0.3>grouping.jsonNowgrouping.jsoncontains a grouping of your original data, specified inJSON. It’s designed to be human-readable, and human-editable, so you can check it and improve it as needed. The largest groups will be at the top, and similar items will be near one another.{
    "Lance Burton": [
        "Lance Burton",
        "Levar Burton"
    ],
    "David Copperfield": [
        "Dave Copperfield",
        "David Copperfield"
    ]
}There are two proposed groups, with proposed names “Lance Burton” and “David Copperfield”. You probably want to copy the produced file and edit the copy.cpgrouping.jsongrouping_fixed.json# edit `grouping_fixed.json`You can edit the proposed group names—the keys of the object—if you like. The original values from the data are in the array values of the object, and you can move them around to correct the grouping. In this case you could also re-runmergicwith a cutoff of 0.1 to get a correct grouping:{
    "David Copperfield": [
        "Dave Copperfield",
        "David Copperfield"
    ],
    "Lance Burton": [
        "Lance Burton"
    ],
    "Levar Burton": [
        "Levar Burton"
    ]
}Now thatgrouping_fixed.jsonis as perfect as it can be, you can move forward.You can also compare your two JSON grouping files and see what you changed:mergicdiffgrouping.jsongrouping_fixed.json>diff.jsonNow the filediff.jsoncontains just what’s different betweengrouping.jsonandgrouping_fixed.json. Themergic diffcommand is analogous to regulardifffor text files, but it is aware of the JSON partition format so it can capture changes intelligently.You can apply changes to amergic-produced file to recover your edited version.mergicapplygrouping.jsondiff.json>grouping_new.jsonNowgrouping_new.jsonis equivalent togrouping_fixed.json, as you can verify:mergicdiffgrouping_fixed.jsongrouping_new.json# {}  // (no changes)In this way you have a complete and verifiable record of your work, at the level of whole files and also at the level of changes made by hand.The JSON grouping format is very convenient for humans, but for tabular data a merge table is more useful. A merge table has one column with the original values from your data and one column with the new keys. These are namedoriginalandmergicin the output:mergictablegrouping_fixed.json>merge_table.csvThe filemerge_table.csvlooks like this:original,mergic
Lance Burton,Lance Burton
Levar Burton,Levar Burton
Dave Copperfield,David Copperfield
David Copperfield,David CopperfieldThis merge table can now be used with any tabular data system. For merges, first merge it on to both tables and then merge by themergickey. For deduplication, merge it on to the table(s) of interest and then use themergiccolumn as you would have used the original data.InstallationpipinstallmergicUsing a Custom Distance FunctionThemergicpackage provides a command-line script calledmergicthat uses Python’s built-indifflib.SequenceMatcher.ratio()for calculating string distances, but a major strength ofmergicis that it enables easy customization of the distance function via themergic.Blenderclass. Making a custommergicscript is as easy as:#!/usr/bin/env python# custom_mergic.pyimportmergic# Any custom distance you want to try! e.g.,defmy_distance(a,b):returnabs(len(a)-len(b))mergic.Blender(my_distance).script()Nowcustom_mergic.pycan be used just like the standardmergicscript!There is also a function that determines how to make a key for a group. It takes a list and returns a string. By defaultmergic.Blenderwill use the longest of a group’s values (the first longest, in stable sorted order). You can also supply a custom function for generating keys.Distances You Might LikeLevenshtein string edit distance: The classic! It has many implementations; one of them ispython-Levenshtein.# pip install python-LevenshteinimportLevenshteinLevenshtein.distance("fuzzy","wuzzy")# 1SeatGeek’sfuzzywuzzy: As described in theirblog post, some people have found these variants to work well in practice. Responses fromfuzzywuzzyare phrased as integer percent similarities; one way to make a distance is to subtract from 100.# pip install fuzzywuzzyfromfuzzywuzzyimportfuzz100-fuzz.ratio("Levensthein","Leviathan")# 50There are a ton of distances, even just within the two packages mentioned!You can also make your own!