Package: text-dedup
FeaturesThis repository contains a collection of text deduplication scripts that are ready to use, or modify based on your needs:MinHash + MinHashLSH, including a spark implementation suitable for large (TB) datasets64 or 128 bit SimHashSuffixArray SubstringBloom FilterExact HashI also have big plans for the future:Memory benchmark for streaming processingInter-dataset deduplicationRewrite suffix array in PythonA collections of other deduplication methods: SuperMinHash, ProbMinHash, TreeMinHash, BagMinHash,Optimal Densification for Fast and Accurate Minwise Hashing,Fast Similarity SketchingA collections of other deduplication methods used in other places:CCNet.However, I do not intent to build a general purpose deduplication library, which was the goal of this repo early on. I will gradually retire the pypi package as well. The reason behind it is that each use-case can be wildly different and requires careful design and consideration. I sincerely encourage you to read the script first (they are relatively short) so you can understand what are at stake here when using it. You can use it to bootstrap your own script, or just use it as a reference.AcknowledgementsThis repository is inspired by the following projects, and is heavily influenced by lessons learned from my own participation inBigScience (Apache 2.0)andBigCode (Apache 2.0). There is ablog postabout the journey. Feedbacks are welcome!Datasketch(MIT)simhash-pyandsimhash-cpp(MIT)Deduplicating Training Data Makes Language Models Better(Apache 2.0)Gaoya(MIT)Quick ExamplesPySpark with DataProcNot a lot of people have access to enough compute resources or the need to deduplicate TB-scale datasets, but if you do, this is a good example of how to use it with GCP DataProc.MODIFYtext_dedup/minhash_spark.pyFOR YOUR OWN PROJECT AND DATASET FIRST!exportCLUSTER_NAME=chenghao-tempexportPROJECT_ID=xx

gclouddataprocclusterscreate$CLUSTER_NAME\--enable-component-gateway\--regionus-central1\--zoneus-central1-a\--master-machine-typec2d-standard-16\--master-boot-disk-size500\--num-workers10\--worker-machine-typec2d-standard-16\--worker-boot-disk-size500\--image-version2.0-debian10\--project$PROJECT_IDgclouddataprocjobssubmitpyspark--cluster${CLUSTER_NAME}\--regionus-central1\--jarsgs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar\--driver-log-levelsroot=WARN\--properties="spark.executor.memory"="50g","spark.driver.memory"="8g","spark.executor.cores"="14"\minhash_spark.py--\--table"huggingface-science-codeparrot.the_stack_java.java"\--output"gs://chenghao-data/dataproc_output/deduplicated"For reference, the script finished deduplicating 42 million rows in less than 40 minutes with above settings (160 cores, 640GB memory in total), while the python version would take around 10 hours with a 80-core machine with 1.8TB memory.In the following part, we are going to deduplicate one dataset:glsubset ofoscar-corpus/OSCAR-2201.Suffix Array Substring Exact Deduplication# inputpython-mtext_dedup.suffix_array\--path"oscar-corpus/OSCAR-2201"\--name"gl"\--split"train"\--cache_dir"./cache"\--output"output/suffix_array/oscar_gl_dedup"\--column"text"\--google_repo_path"/Users/chenghao/Downloads/Projects/text-dedup/deduplicate-text-datasets"# outputINFOLoading:2.75seconds
INFOPreprocessing:4.78seconds
INFOSuffixArray:98.29seconds
INFOSelfSimilar:4.24seconds
INFORestore:0.25seconds
INFODeduplicate:6.23seconds
INFOSaving:8.91seconds
INFOTotal:125.45seconds
INFOBefore:180332342bytes(88803)INFOAfter:97646271bytes(40404)MinHash Near Deduplication# inputpython-mtext_dedup.minhash\--path"oscar-corpus/OSCAR-2201"\--name"gl"\--split"train"\--cache_dir"./cache"\--output"output/minhash/oscar_gl_dedup"\--column"text"\--batch_size10000# outputINFOLoading:2.62seconds
INFOMinHashing:0.08seconds
INFOClustering:2.20seconds
INFOFiltering:0.53seconds
INFOSaving:9.86seconds
INFOTotal:15.29seconds
INFODataNumber(before):88803INFODataNumber(after):44124(49.69%)INFODuplicateNumber:44679(50.31%)INFO🤗HappyDeduplicating🤗SimHash Near Deduplication# inputpython-mtext_dedup.simhash\--path"oscar-corpus/OSCAR-2201"\--name"gl"\--split"train"\--cache_dir"./cache"\--output"output/simhash/oscar_gl_dedup"\--column"text"\--batch_size10000# outputINFOLoading:2.60seconds
INFOSimHashing:0.04seconds
INFOIndexing:28.88seconds
INFOFiltering:0.88seconds
INFOSaving:10.41seconds
INFOTotal:42.80seconds
INFODataNumber(before):88803INFODataNumber(after):46163(51.98%)INFODuplicateNumber:42640(48.02%)INFO🤗HappyDeduplicating🤗Exact Hash Exact Deduplication# inputpython-mtext_dedup.exact_hash\--path"oscar-corpus/OSCAR-2201"\--name"gl"\--split"train"\--cache_dir"./cache"\--output"output/exact_hash/oscar_gl_dedup"\--column"text"\--batch_size1000# outputINFOLoading:2.95s
INFOProcessing:3.79s
INFOFiltering:0.10s
INFOSaving:2.89s
INFOTotal:9.72s
INFOBefore:88803INFOAfter:47049Bloom Filter Exact Deduplication# inputpython-mtext_dedup.bloom_filter\--path"oscar-corpus/OSCAR-2201"\--name"gl"\--split"train"\--cache_dir"./cache"\--output"output/bloom_filter/oscar_gl_dedup"\--error_rate1e-5\--column"text"\--batch_size1000# outputINFOLoading:2.72s
INFOProcessing:4.84s
INFOFiltering:0.10s
INFOSaving:2.88s
INFOTotal:10.54s
INFOBefore:88803INFOAfter:47045BenchmarksA benchmark of different methods here can be found inbenchmarks/wiki40.ipynb. A notebook in evaluating MinHash onpinecone/core-2020-05-10-deduplicationcan be found inbenchmarks/pinecone.ipynb.For quick reference, here are the results:MethodPrecisionRecallF1TimeMinHash0.94640.94460.945524sSimHash*0.90110.69590.7853210sSimHash(Gyawali et al., LREC 2020)0.6970.2470.3647-Exact Title (my implementation)0.83020.55210.6632-Exact Title(Gyawali et al., LREC 2020)0.8300.500.624-*Best SimHash result frombenchmarks/hyperparameter.ipynb.LicenseApache 2.0