Package: duplicate_images
Finding Duplicate ImagesFinds equal or similar images in a directory containing (many) image files.Official home page:https://github.com/lene/DuplicateImagesDevelopment page:https://gitlab.com/lilacashes/DuplicateImagesPyPI page:https://pypi.org/project/duplicate-imagesUsageInstalling:$pipinstallduplicate_imagesPrinting the help screen:$find-dups-hQuick test run:$find-dups$IMAGE_ROOTTypical usage:$find-dups$IMAGE_ROOT--parallel--progress--hash-dbhashes.jsonSupported image formatsJPEG and PNG (tested quite thoroughly)HEIC (experimental support, tested cursorily only)All otherformats supportedby
thepillowPython Imaging Library should work, but are not specifically tested.Explicitly allow huge imagesThePILimage library, which is used as backend, limits the size of images to 178956970 pixels by
default, to guard against memory exhaustion. For larger images, specify the maximum image size in
pixels with the--max-image-pixelsoption.Image comparison algorithmsUse the--algorithmoption to select how equal images are found. The default algorithm isphash.ahash,colorhash,dhash,dhash_vertical,phash,phash_simple,whash: seven different
image hashing algorithms. Seehttps://pypi.org/project/ImageHashfor an introduction on image
hashing andhttps://tech.okcupid.com/evaluating-perceptual-image-hashes-okcupidfor some gory
details which image hashing algorithm performs best in which situation. For a start I recommend
usingphash, and only evaluating the other algorithms ifphashdoes not perform satisfactorily
in your use case.Image similarity threshold configurationUse the--hash-sizeparameter to tune the precision of the hashing algorithms. For thecolorhashalgorithm the hash size is interpreted as the number of bin bits and defaults to 3. For all other
algorithms the hash size defaults to 8. Forwhashit must be a power of 2.Use the--max-distanceparameter to tune how close images should be to be considered duplicates.
The argument is a positive integer. Its value is highly dependent on the algorithm used and the
nature of the images compared, so the best value for your use case can oly be found through
experimentation.NOTE:using the--max-distanceparameter slows down the comparison considerably with large
image collections, making the runtime complexity go from O(N) to O(N2). If you want to
scan collections with at least thousands of images, it is highly recommended to tune the desired
similarity threshold with the--hash-sizeparameter alone, if that is at all possible.Pre-storing and using image hashes to speed up computationUse the--hash-db ${FILE}.jsonor--hash-db ${FILE}.pickleoption to store image hashes in the
file$FILEin JSON or Pickle format and read image hashes from that file if they are already
present there. This avoids having to compute the image hashes anew at every run and can
significantly speed up run times.Handling matching images either as pairs or as groupsBy default, matching images are presented as pairs. With the--groupCLI option, they are handled
as a group containing all matching images.Example:1.jpg,2.jpgand3.jpgin the current folder.are equal.$find-dups.1.jpg2.jpg1.jpg3.jpg2.jpg3.jpg
$find-dups.--group1.jpg2.jpg3.jpgActions for matching image groupsUse the--on-equaloption to select what to do to pairs of equal images. The default action isprint.delete-firstord1: deletes the first of the files in the groupdelete-lastordl: deletes the last of the files in the groupdelete-biggestord>: deletes the file with the biggest sizedelete-smallestord<: deletes the file with the smallest sizesymlink-smaller: delete the smaller files and replace them to a symlink to the biggest fileeog: launches theeogimage viewer to compare the files in the group (deprecatedbyexec)xv: launches thexvimage viewer to compare the files in the group (deprecatedbyexec)print: prints the files in the groupprint_inline: likeprintbut without newlinequote: prints the files in the group quoted for POSIX shellsquote_inline: likequotebut without newlineexec: executes a command (see--execargument below)none: does nothing; may be useful for benchmarking and testingThe--execargument allows calling another program when the--on-equal execoption is given.
You can pass a command line string like--exec "program {1} {2}"where{1}and{2}are
replaced by the matching pair files (or first two files in a group), quoted so the shell recognizes
the files properly. The wildcard{*}expands to all files in a matching group, which when called
with the--groupargument may be more than two images considered equal.Examples:--exec "open -a Preview -W {1} {2}": Opens the files in MacOS Preview app and waits for it.--exec "ls -s {*}": Prints the size (in blocks) next to all files.--exec 'for i in {*}; do dirname $i; basename $i; done': Shows the directory and the filename
separately for all files.Parallel executionUse the--paralleloption to utilize all free cores on your system for calculating image hashes.
Optionally, you can specify the number of processes to use with--parallel $N.To execute the--on-equalactions in parallel, use the--parallel-actionsoption, which also can
take an optional number of processes to use as argument.Excluding subfoldersUse the--exclude-diroption to exclude subfolders of$IMAGE_ROOTfrom the search. The argument
is a regular expression matching the subfolder names to be excluded. Multiple arguments can be
passed to--exclude-dirto exclude multiple subfolders.The argument(s) given to--exclude-dirmay be regular expressions. These regular expressions are
matched only against the directory name, not the file name.ExamplesExclude subfolder$IMAGE_ROOT/foo:$find-dups$IMAGE_ROOT--exclude-dir$IMAGE_ROOT/fooExclude all subfolders namedfooorbar:$find-dups$IMAGE_ROOT--exclude-dirfoobarSlow executionfind-dupscan also use an alternative algorithm which exhaustively compares all images to each
other, being O(N2) in the number of images. This algorithm is selected automatically if--max-distanceis not 0.You can use the--slowoption to use this alternative algorithm specifically. The--slowswitch
is mutually exclusive with the--groupswitch.Progress bar and verbosity control--progressprints a progress bar each for the process of reading the images, and the process of
finding duplicates among the scanned image--debugprints debugging output--quietdecreases the log level by 1 for each time it is called;--debugand--quietcancel
each other outDevelopment notesNeeds Python3, Pillow imaging library andpillow-heifHEIF plugin to run, additionally Wand for
the test suite.Uses Poetry for dependency management.InstallationFrom source:$gitclonehttps://gitlab.com/lilacashes/DuplicateImages.git
$cdDuplicateImages
$pip3installpoetry
$poetryinstallRunning$poetryrunfind-dups$PICTURE_DIRor$poetryrunfind-dups-hfor a list of all possible options.Test suiteRunning it all:$poetryrunpytest
$poetryrunmypyduplicate_imagestests
$poetryrunflake8
$poetryrunpylintduplicate_imagestests
$poetryrunbandit-rduplicate_imagesor simply$.git_hooks/pre-pushSetting the test suite to be run before every push:$cd.git/hooks
$ln-s../../.git_hooks/pre-push.PublishingA tag is created and the new version is published automatically by GitLab CI on every successful
merge tomaster.PrerequisitesFor every Merge Request tomasterit is checked that:theversionnumber inpyproject.tomlis not an already existing git tagtheCHANGELOG.mdcontains an entry for the current version numberPyPIThere is a job in GitLab CI for publishing topypi.orgthat runs as soon as a new tag is added,
which happens automatically whenever a MR is merged. The tag is the same as theversionin thepyproject.tomlfile. For every MR it needs to be ensured that theversionis not the same as an
already existing tag.To publish the package on PyPI manually:$poetryconfigrepositories.testpypihttps://test.pypi.org/legacy/
$poetrybuild
$poetrypublish--username$PYPI_USER--password$PYPI_PASSWORD--repositorytestpypi&&\poetrypublish--username$PYPI_USER--password$PYPI_PASSWORD(obviously assuming here that username and password are the same on PyPI and TestPyPI)Updating GitHub mirrorThe GitHub repogit@github.com:lene/DuplicateImages.gitis set up as a push mirror in GitLab CI,
but mirroring is flaky at the time and may or may not succeed. The CI jobPushToGithubshould take
care of mirroring to GitHub after every merge tomaster.To push to the GitHub repository manually (assuming the GitHub repo is set up as remotegithub):$gitcheckoutmaster
$gitfetch
$gitpull--rebase
$gittag# to check that the latest tag is present$gitpush--tagsgithubmasterCreating Releases on GitHubThe CI jobCreateGithubReleasecreates a Release on GitHub, which can then be found underhttps://github.com/lene/DuplicateImages/releases.ProfilingCPU timeTo show the top functions by time spent, including called functions:$poetryrunpython-mcProfile-stottime./duplicate_images/duplicate.py\--algorithm$ALGORITHM--action-equalnone$IMAGE_DIR2>&1|head-n15or, to show the top functions by time spent in the function alone:$poetryrunpython-mcProfile-scumtime./duplicate_images/duplicate.py\--algorithm$ALGORITHM--action-equalnone$IMAGE_DIR2>&1|head-n15Memory usage$poetryrunfil-profilerun./duplicate_images/duplicate.py\--algorithm$ALGORITHM--action-equalnone$IMAGE_DIR2>&1This will open a browser window showing the functions using the most memory (seehttps://pypi.org/project/filprofilerfor more details).ContributorsLene Preuss (https://github.com/lene): primary developerMike Reiche (https://github.com/mreiche): support for arbitrary actions, speedupshttps://github.com/beijingjazzpanda: bug fix