Package: pandas-dedupe
pandas-dedupeThe Dedupe library made easy with Pandas.Installationpip install pandas-dedupeVideo TutorialsBasic DeduplicationBasic UsageA training file and a settings file will be created while running Dedupe.
Keeping these files will eliminate the need to retrain your model in the future.If you would like to retrain your model from scratch, just delete the settings and training files.Deduplication (dedupe_dataframe)dedupe_dataframeis for deduplication when you have data that can contain multiple records that can all refer to the same entityimportpandasaspdimportpandas_dedupe#load dataframedf=pd.read_csv('test_names.csv')#initiate deduplicationdf_final=pandas_dedupe.dedupe_dataframe(df,['first_name','last_name','middle_initial'])#send output to csvdf_final.to_csv('deduplication_output.csv')Gazetteer deduplication (gazetteer_dataframe)gazetteer_dataframeis for matching a messy dataset against a 'canonical dataset' (i.e. the gazette)importpandasaspdimportpandas_dedupe#load dataframedf_clean=pd.read_csv('gazette.csv')df_messy=pd.read_csv('test_names.csv')#initiate deduplicationdf_final=pandas_dedupe.gazetteer_dataframe(df_clean,df_messy,'fullname',canonicalize=True)#send output to csvdf_final.to_csv('gazetteer_deduplication_output.csv')Matching / Record LinkageUse identical field names when linking dataframes.
Record linkage should only be used on dataframes that have been deduplicated.importpandasaspdimportpandas_dedupe#load dataframesdfa=pd.read_csv('file_a.csv')dfb=pd.read_csv('file_b.csv')#initiate matchingdf_final=pandas_dedupe.link_dataframes(dfa,dfb,['field_1','field_2','field_3','field_4'])#send output to csvdf_final.to_csv('linkage_output.csv')Advanced UsageCanonicalize FieldsThe canonicalize parameter will standardize names in a given cluster. Original fields are also kept.pandas_dedupe.dedupe_dataframe(df,['first_name','last_name','payment_type'],canonicalize=True)Update Threshold (dedupe_dataframe and gazetteer_dataframe only)Group records into clusters only if the cophenetic similarity of the cluster is greater than
the threshold.pandas_dedupe.dedupe_dataframe(df,['first_name','last_name'],threshold=.7)Update Existing Model (dedupe_dataframe and gazetteer_dataframe only)IfTrue, it allows a user to update the existing model.pandas_dedupe.dedupe_dataframe(df,['first_name','last_name'],update_model=True)Recall Weight & Sample SizeThededupe_dataframe()function has two optional parameters specifyingrecall_weightandsample_size:recall_weight- Ranges from 0 to 2. When set to 2, we are saying we care twice as much
about recall than we do about precision.sample_size- Specifies the sample size used for training as a float from 0 to 1.
By default it is 30% (0.3) of our data.Specifying TypesIf you'd like to specify dates, spatial data, etc, do so here. The structure must be like so:('field', 'type', 'additional_parameter). theadditional_parametersection can be omitted.
The default type isString.See the full list of typesbelow.# Price Examplepandas_dedupe.dedupe_dataframe(df,['first_name','last_name',('salary','Price')])# has missing Examplepandas_dedupe.link_dataframes(df,['SSN',('bio_pgraph','Text'),('salary','Price','has missing')])# crf Examplepandas_dedupe.dedupe_dataframe(df,[('first_name','String','crf'),'last_name',(m_initial,'Exact')])TypesDedupe supports a variety of datatypes; a full list with documentation can be foundhere.pandas-dedupe officially supports the following datatypes:String- Standard string comparison using string distance metric. This is the default type.Text- Comparison for sentences or paragraphs of text. Uses cosine similarity metric.Price- For comparing positive, non zero numerical values.DateTime- For comparing dates.LatLong- (39.990334, 70.012) will not match to (40.01, 69.98) using a string distance
metric, even though the points are in a geographically similar location. The LatLong type resolves
this by calculating the haversine distance between compared coordinates. LatLong requires
the field to be in the format (Lat, Long). The value can be a string, a tuple containing two
strings, a tuple containing two floats, or a tuple containing two integers. If the format
is not able to be processed, you will get a traceback.Exact- Tests whether fields are an exact match.Exists- Sometimes, the presence or absence of data can be useful in predicting a match.
The Exists type tests for whether both, one, or neither of fields are null.Additional supported parameters are:has missing- Can be used if one of your data fields contains null valuescrf- Use conditional random fields for comparisons rather than distance metric. May be more
accurate in some cases, but runs much slower. Works with String and ShortString types.ContributorsTyler Marrs- Refactored code, added docstrings, addedthresholdparameterTawni Marrs- refactored code, added docstringsieriii- Addedupdate_modelparameter, updated codebase to useDedupe 2.0, added support for multiprocessing, addedgazetteer_dataframe.Daniel Marczin- Extensive updates to documentation to enhance readability.CreditsMany thanks to folks atDataMadefor making the theDedupe librarypublicly available. People interested in a code-free implementation of the dedupe library can find a link here:Dedupe.io.