Package: deduplify
deduplifyA Python tool to search for and remove duplicated files in messy datasets.Table of Contents:OverviewInstallationFrom PyPIManual InstallationUsageHashing filesComparing filesCleaning upGlobal argumentsContributingOverviewdeduplifyis a Python command line tool that will search a directory tree for duplicated files and optionally remove them.
It generates an MD5 hash for each file recursively under a target directory and identifies the filepaths that generate unique and duplicated hashes. When deleting duplicated files, it deletes those deepest in the directory tree first leaving the last present.Installationdeduplifyhas a minimum Python requirement of v3.7 but has been developed in v3.8.From PyPIFirst, make sure yourpipversion is up-to-date.python-mpipinstall--upgradepipThen installdeduplify.pipinstalldeduplifyManual InstallationBegin by cloning this repository and change into it.gitclonehttps://github.com/Living-with-machines/deduplify.gitcddeduplifyNow run the setup script.
This will install any requirements and the CLI tool into your Python$PATH.pythonsetup.pyinstallUsagededuplifyhas 3 commands:hash,compareandclean.Hashing filesThehashcommand takes a path to a target directory as an argument.
It walks the structure of this directory tree and generates MD5 hashes for all files and outputs a database stored as a JSON file, the name of which can be overwritten using the--dbfile [-f]flag.Each document in the generated database can be described as a dictionary with the following properties:{"filepath":"",#String.Thefullpathtoagivenfile."hash":"",#String.TheMD5hashofthegivenfile."duplicate":bool,#Boolean.Whetherthishashisrepeatedinthedatabase(True)ornot(False).}By default,deduplifygenerates hashes for all files under a directory.
But one or more specific file extensions to search for can be specified using the--extflag.Command line usage:usage:deduplifyhash[-h][-cCOUNT][-v][-fDBFILE][--exts[EXTS]][--restart]dir

positionalarguments:dirPathtodirectorytobeginsearchfrom

optionalarguments:-h,--helpshowthishelpmessageandexit-cCOUNT,--countCOUNTNumberofthreadstoparalleliseover.Default:1-v,--verbosePrintloggingmessagestotheconsole-fDBFILE,--dbfileDBFILEDestinationdatabaseforfilehashes.MustbeaJSONfile.Default:file_hashes.json--exts[EXTS]Alistoffileextensionstosearchfor.--restartRestartarunofhashingfilesandskipoverfilesthathavealreadybeenhashed.Outputfilecontainingadatabaseoffilenamesandhashesmustalreadyexist.Comparing filesThecomparecommand reads in the JSON database generated by runninghash, the name of which can be overwritten using the--infile [-f]flag if the data were saved under a different name.
The command runs a check to test if the stem of the filepath are equivalent for all paths that generated a given hash.
This indicates that the file is a true duplication as since both its name and content match.
If they do not match, this implies that the same content is saved under two different filenames.
In this scenario, a warning is raised asking the user to manually investigate these files.If all the filenames for a given hash match, then the shortest filepath is removed from the list and the rest are returned to be deleted.
To delete files, the user needs to runcomparewith the--purgeflag set.A recommended workflow to ensure that all duplicated files have been removed would be as follows:deduplifyhashtarget_dir# First pass at hashing filesdeduplifycompare--purge# Delete duplicated filesdeduplifyhashtarget_dir# Second pass at hashing filesdeduplifycompare# Compare the filenames again. The code should return nothing to compareCommand line usage:usage:deduplifycompare[-h][-cCOUNT][-v][-fINFILE][--list-files][--purge]optionalarguments:-h,--helpshowthishelpmessageandexit-cCOUNT,--countCOUNTNumberofthreadstoparalleliseover.Default:1-v,--verbosePrintloggingmessagestotheconsole-fINFILE,--infileINFILEDatabasetoanalyse.MustbeaJSONfile.Default:file_hashes.json--list-filesListduplicatedfiles.Default:False--purgeDeletesduplicatedfiles.Default:FalseCleaning upAfter purging duplicated files, the target directory may be left with empty sub-directories.
Running thecleancommand will locate and delete these empty subdirs and remove them.Command line usage:usage:deduplifyclean[-h][-cCOUNT][-v]dir

positionalarguments:dirPathtodirectorytobeginsearchfrom

optionalarguments:-h,--helpshowthishelpmessageandexit-cCOUNT,--countCOUNTNumberofthreadstoparalleliseover.Default:1-v,--verbosePrintloggingmessagestotheconsoleGlobal argumentsThe following flags can be passed to any of the commands ofdeduplify.--verbose [-v]: The flag will print verbose output to the console, as opposed to saving it to thededuplify.logfile.--count [-c]: Some processes withindeduplifycan be parallelised over multiple threads when working with larger datasets.
To do this, include the--countflag with the (integer) number of threads you'd like to parallelise over.
This flag will raise an error if requesting more threads than CPUs available on the host machine.ContributingThank you for wanting to contribute todeduplify! :tada: :sparkling_heart:
To get you started, please read ourCode of ConductandContributing Guidelines.