Package: remdups
Author:Roland PuntaierHomepage:https://github.com/rpuntaie/remdupsLicense:See LICENSE fileremdupsremdupsgenerates a script toremove duplicate filescopy files from another directory to this one, ignoring duplicatesmove files from another directory to this one, ignoring duplicatesThe resulting script should be further inspected before executing it in your shell.UsageOptional. You can choose one or more source+hashing methods, via e.g.:cat > .remdups_c.sha512
cat > .remdups_e.md5All of .remdups_{c,b,d,e,n}.{sha512, sha384, sha256, sha224, sha1, md5}
contribute to the final hash. If you don’t make such a file, the default is:.remdups_c.sha256{‘c’: ‘content’, ‘b’: ‘block’, ‘d’: ‘date’, ‘e’: ‘exif’, ‘n’: ‘name’}Create the hash file by either of (can take a long time):remdups
remdups update
remdups update <fromdir>The hashes are added to all .remdups_x.y. To rehash all files:rm .remdups_*Make a script with rm, mv, cp commands.
It can be repeated with different options until the script is good.$remdups rm -s script.sh
$remdups cp -s script.bat #if you used <fromdir>
$remdups mv -s script.py  #if you used <fromdir>If the file ends in .sh, cp is used and the file names are in linux format.
This is usable also on Windows with MSYS, MSYS2 and CYGWIN.If the file ends in .bat, Windows commands are used.If the file ends in .py, python functions are used.Inspect the script and go back to 2., if necessary.
Changes to the script can also be done with the editor.execute script$./script.shAlternatively you can use remdups from your own python script, or interactively from a python prompt.InstallDirectly from PyPi:$pipinstallremdupsFromgithub: Clone, change to the directory and do$pythonsetup.pyinstallIf you plan to extend this toolfork ongithubclone from your fork to you PCdo an editable install$pipinstall-e.test after change and bring coverage to 100%$py.test--covremdups.py--cov-reportterm-missingconsider sharing changes useful for others (githubpull request).HintFor more advanced file selectionfindcan be used.
The following example ignores directoryoldand produces a hash for all JPEG files:$find.-path"old"-prune-or-not-typed-and-iname"\*.jpg"-execsha256sum{}\;>.remdups_c.sha256Command LineThe following is in addition to the help given with:remdups --helpThe sources for the hashes can be:{'c': 'content', 'b': 'block', 'd': 'date', 'e': 'exif', 'n': 'name'}Don’t includen, because same files with different names cannot be found.cis the best.Do e.g:cat > .remdups_b.sha512
cat > .remdups_c.sha256Fill the hash files from the current directory:remdups updateOr fill the hash files from another directory:remdups update <fromdir>In the latter case the paths in the hash files will have a//or\\to mark the start for the new relatives paths in a subsequentmvorcpcommand.Once the hash files are filled create the script. It depend on the extension used:remdups <command> -s script.sh <options>
remdups <command> -s script.bat <options>
remdups <command> -s script.py <options>commandcan berm,cp,mv.
There is alsodupsofanddupsoftail, but they don’t take a--script, but print the output.--keep-in,--keep-outand--comment-outwill remove different files of a duplicate group.--safewill do a byte-wise comparison, before creating the script. That takes longer.cpandmvalso take--sort: In this case the tree is not recreated, but the files are sorted
to the provided tree structure using the file modification date. Seehttps://docs.python.org/3/library/datetime.html#strftime-strptime-behavior.APIWith your own python script you can load the file for hashing and use the
loaded content immediately to create the new file, if not duplicate.fromremdupsimport*hasher=Hasher()allduplicates=[]forfilename,duplicates,contentinhasher.foreachcontent('.'):ifduplicates:allduplicates.append(f)else:assertcontent!=[]#some .remdups_ must be with (c)ontentnfilename='afilehere'withopen(nfilename,'wb')asnf:forbufincontent:nf.write(buf)shutil.copystat(filename,nfilename)foreachcontent()usesscandir(), but does not add duplicate files to the.remdup_files.forfinhasher.scandir(otherdir,filter=['*.jpg'],exclude=['**/old/*']):duplicates=hasher.duplicates(f)yield(f,duplicates,kw['content'])ifduplicates:hasher.clear(f)else:hasher.update_hashfiles()If you don’t want to keep the content, don’t provide a[]forcontentinscandir.scandir()will hash all files not yet in the.remdup_files and will return the file name.This code resorts a tree by hashing and creating a copy, if not duplicate.importosimportremdupsos.chdir('dir/to/resort/to')withopen('.remdups_c.sha256','w'):passremdups.resort('../some/dir/here',"%y%m/%d_%H%M%S")