Package: csvdedupe
csvdedupeCommand line tools for using thededupe python libraryfor deduplicating CSV files.Two easy commands:csvdedupe- takes a messy input file or STDIN pipe and identifies duplicates.csvlink- takes two CSV files and finds matches between them.Read more about csvdedupe on OpenNews SourceInstallation and dependenciespip install csvdedupeGetting Startedcsvdedupecsvdedupetakes a messy input file or STDIN pipe and identifies duplicates. To get started, pick one of three deduping strategies: callcsvdedupewith arguments, pipe your file using UNIX, or define a config file.Provide an input file, field names, and output file:csvdedupeexamples/csv_example_messy_input.csv\--field_names"Site name"AddressZipPhone\--output_fileoutput.csvorPipe it, UNIX style:catexamples/csv_example_messy_input.csv|csvdedupe--skip_training\--field_names"Site name"AddressZipPhone>output.csvorDefine everything in a config file:csvdedupeexamples/csv_example_messy_input.csv\--config_file=config.jsonYour config file may look like this:{"field_names":["Site name","Address","Zip","Phone"],"field_definition":[{"field":"Site name","type":"String"},{"field":"Address","type":"String"},{"field":"Zip","type":"String","Has Missing":true},{"field":"Phone","type":"String","Has Missing":true}],"output_file":"examples/output.csv","skip_training":false,"training_file":"training.json","sample_size":150000,"recall_weight":2}To usecsvdedupeyou absolutely need:inputa CSV file name or piped CSV file to deduplicateEither--config_filePath to configuration file.Or--field_namesList of column names for dedupe to pay attention toYou may also need:--output_file OUTPUT_FILECSV file to store deduplication results (default:
None)--destructiveOutput file will contain unique records only--skip_trainingSkip labeling examples by user and read training from
training_file only (default: False)--training_file TRAINING_FILEPath to a new or existing file consisting of labeled
training examples (default: training.json)--sample_size SAMPLE_SIZENumber of random sample pairs to train off of
(default: 150000)--recall_weight RECALL_WEIGHTThreshold that will maximize a weighted average of our
precision and recall (default: 2)-d,--delimiterDelimiting character of the input CSV file (default: ,)-h,--helpshow help message and exitcsvlinkcsvlinktakes two CSV files and finds matches between them.Provide an input file, field names, and output file:csvlinkexamples/restaurant-1.csvexamples/restaurant-2.csv\--field_namesnameaddresscitycuisine\--output_fileoutput.csvorLine up different field names from each file:csvlinkexamples/restaurant-1.csvexamples/restaurant-2.csv\--field_names_1nameaddresscitycuisine\--field_names_2restaurantstreetcitytype\--output_fileoutput.csvorPipe the output to STDOUT:csvlinkexamples/restaurant-1.csvexamples/restaurant-2.csv\--field_namesnameaddresscitycuisine\>output.csvorDefine everything in a config file:csvlinkexamples/restaurant-1.csvexamples/restaurant-2.csv\--config_file=config.jsonYour config file may look like this:{"field_names_1":["name","address","city","cuisine"],"field_names_2":["restaurant","street","city","type"],"field_definition":[{"field":"name","type":"String"},{"field":"address","type":"String"},{"field":"city","type":"String","Has Missing":true},{"field":"cuisine","type":"String","Has Missing":true}],"output_file":"examples/output.csv","skip_training":false,"training_file":"training.json","sample_size":150000,"recall_weight":2}To usecsvlinkyou absolutely need:inputtwo CSV file names to join togetherEither--config_filePath to configuration file.Or--field_names_1List of column names in first file for dedupe to pay attention to--field_names_2List of column names in second file for dedupe to pay attention toYou may also need:--output_file OUTPUT_FILECSV file to store deduplication results (default:
None)--inner_joinOnly return matches between datasets--skip_trainingSkip labeling examples by user and read training from
training_file only (default: False)--training_file TRAINING_FILEPath to a new or existing file consisting of labeled
training examples (default: training.json)--sample_size SAMPLE_SIZENumber of random sample pairs to train off of
(default: 150000)--recall_weight RECALL_WEIGHTThreshold that will maximize a weighted average of our
precision and recall (default: 2)-d,--delimiterDelimiting character of the input CSV file (default: ,)-h,--helpshow help message and exitTrainingThesecret sauceof csvdedupe is human input. In order to figure out the best rules to deduplicate a set of data, you must give it a set of labeled examples to learn from.The more labeled examples you give it, the better the deduplication results will be. At minimum, you should try to provide10 positive matchesand10 negative matches.The results of your training will be saved in a JSON file (training.json, unless specified otherwise with the--training-fileoption) for future runs of csvdedupe.Here's an example labeling operation:Phone:2850617Address:3801s.wabash
Zip:
Sitename:adas.mckinleyst.thomascdc

Phone:2850617Address:3801swabashave
Zip:
Sitename:adas.mckinleycommunityservices-mckinley-st.thomas

Dotheserecordsrefertothesamething?(y)es/(n)o/(u)nsure/(f)inishedOutputcsvdedupeattempts to identify all the rows in the csv that refer to the same thing. Each group of
such records are called a cluster.csvdedupereturns your input file with an additional column calledCluster ID,
that either is the numeric id (zero-indexed) of a cluster of grouped records or anxif csvdedupe believes
the record doesn't belong to any cluster.csvlinkoperates in much the same way ascsvdedupe, but will flatten both CSVs in to one
output file similar to a SQLOUTER JOINstatement. You can use the--inner_joinflag to exclude rows that don't match across the two input files, much like an INNER JOIN.Preprocessingcsvdedupe attempts to convert all strings to ASCII, ignores case, new lines, and padding whitespace. This is all
probably uncontroversial except the conversion to ASCII. Basically, we had to choose between two ways of handling
extended characters.distance("Tomas", "Tomás')  = distance("Tomas", "Tomas")ordistance("Tomas, "Tomás") = distance("Tomas", "Tomzs")We chose the first option. While it is possible to do something more sophisticated, this option seems to work pretty well
for Latin alphabet languages.TestingUnit tests of core csvdedupe functionspipinstall-rrequirements-test.txt
nosetestsCommunityDedupe Google groupIRC channel, #dedupe on irc.freenode.netRecipesCombining and deduplicating files from different sources.Lets say we have a few sources of early childhood programs in Chicago and we'd like to get a canonical list.
Let's do it withcsvdedupe,csvkit, and some other common command line tools.Alignment and stackingOur first task will be to align the files and have the same data in the same columns for stacking.First, let's look at the headers of the files.File 1> head -1 CPS_Early_Childhood_Portal_Scrape.csv"Site name","Address","Phone","Program Name","Length of Day"File 2> head -1 IDHS_child_care_provider_list.csv"Site name","Address","Zip Code","Phone","Fax","IDHS Provider ID"So, we'll have to add "Zip Code", "Fax", and "IDHS Provider ID"
toCPS_Early_Childhood_Portal_Scrape.csv, and we'll have to add "Program Name",
"Length of Day" toIDHS_child_care_provider_list.csv.> cd examples> sed '1 s/$/,"Zip Code","Fax","IDHS Provider ID"/' CPS_Early_Childhood_Portal_Scrape.csv > input_1a.csv> sed '2,$s/$/,,,/' input_1a.csv > input_1b.csv> sed '1 s/$/,"Program Name","Length of Day"/' IDHS_child_care_provider_list.csv > input_2a.csv> sed '2,$s/$/,,/' input_2a.csv > input_2b.csvNow, we reorder the columns in the second file to align to the first.> csvcut -c "Site name","Address","Phone","Program Name","Length of Day","Zip Code","Fax","IDHS Provider ID" \input_2b.csv > input_2c.csvAnd we are finally ready to stack.> csvstack -g CPS_Early_Childhood_Portal_Scrape.csv,IDHS_child_care_provider_list.csv \-n source \input_1b.csv input_2c.csv > input.csvDedupe it!And now we can dedupe> cat input.csv | csvdedupe --field_names "Site name" Address "Zip Code" Phone > output.csvLet's sort the output by duplicate IDs, and we are ready to open it in your favorite spreadsheet program.> csvsort -c "Cluster ID" output.csv > sorted.csvErrors and BugsIf something is not behaving intuitively, it is a bug, and should be reported.
Report ithere.Patches and Pull RequestsWe welcome your ideas! You can make suggestions in the form ofgithub issues(bug reports, feature requests, general questions), or you can submit a code contribution via a pull request.How to contribute code:Fork the project.Make your feature addition or bug fix.Send us a pull request with a description of your work! Don't worry if it isn't perfect - think of a PR as a start of a conversation, rather than a finished product.Copyright and AttributionCopyright (c) 2016 DataMade. Released under theMIT License.