Package: difPy
Duplicate Image Finder (difPy)Tired of going through all images in a folder and comparing them manually to check if they are duplicates?The Duplicate Image Finder (difPy) Python packageautomatesthis task for you!Read more on how the algorithm of difPy works in my Medium articleFinding Duplicate Images with Python.For adetailed usage guide, please view the officialdifPy Usage Documentation.DescriptiondifPy searches for images inone or more different folders, compares the images it found and checks whether these are duplicates. It then outputs theimage files classified as duplicatesas well as theimages having the lowest resolutions, so you know which of the duplicate images are safe to be deleted. You can then either delete them manually, or let difPy delete them for you.difPy does not compare images based on their hashes. It compares them based on their tensors i. e. the image content - this allows difPy tonot only search for duplicate images, but also for similar images.difPy leverages Python'smultiprocessing capabilitiesand is therefore able to perform at high performance even on large datasets.Table of ContentsBasic UsageOutputAdditional ParametersCLI UsageBasic UsageTo make difPy search for duplicateswithin one folder:importdifPydif=difPy.build('C:/Path/to/Folder/')search=difPy.search(dif)To search for duplicateswithin multiple folders:importdifPydif=difPy.build(['C:/Path/to/FolderA/','C:/Path/to/FolderB/','C:/Path/to/FolderC/',...])search=difPy.search(dif)Folder paths can be specified as standalone Python strings, or within a list. WithdifPy.build(), difPy first scans the images in the provided folders and builds a collection of images by generating image tensors.difPy.search()then starts the search for duplicate images.OutputdifPy returns various types of output that you may use depending on your use case:I. Search Result DictionaryAJSON formatted collectionof duplicates/similar images (i. e.match groups) that were found. Each match group has a primary image (the key of the dictionary) which holds the list of its duplicates including their filename and MSE (Mean Squared Error). The lower the MSE, the more similar the primary image and the matched images are. Therefore, an MSE of 0 indicates that two images are exact duplicates.search.result>Output:{'C:/Path/to/Image/image1.jpg':[['C:/Path/to/Image/duplicate_image1a.jpg',0.0],['C:/Path/to/Image/duplicate_image1b.jpg',0.0]],'C:/Path/to/Image/image2.jpg':[['C:/Path/to/Image/duplicate_image2a.jpg',0.0]],...}II. Lower Quality FilesAlistof duplicates/similar images that have thelowest qualityamong match groups:search.lower_quality>Output:['C:/Path/to/Image/duplicate_image1.jpg','C:/Path/to/Image/duplicate_image2.jpg',...]Lower quality images then can bemoved to a different location:search.move_to(destination_path='C:/Path/to/Destination/')Ordeleted:search.delete(silent_del=False)III. Process StatisticsAJSON formatted collectionwith statistics on the completed difPy processes:search.stats>Output:{'directory':['C:/Path/to/Folder_A/','C:/Path/to/Folder_B/',...],'process':{'build':{'duration':{'start':'2024-02-18T19:52:39.479548','end':'2024-02-18T19:52:41.630027','seconds_elapsed':2.1505},'parameters':{'recursive':True,'in_folder':False,'limit_extensions':True,'px_size':50,'processes':5},'total_files':{'count':3232},'invalid_files':{'count':0,'logs':{}}},'search':{'duration':{'start':'2024-02-18T19:52:41.630027','end':'2024-02-18T19:52:46.770077','seconds_elapsed':5.14},'parameters':{'similarity_mse':0,'rotate':True,'lazy':True,'processes':5,'chunksize':None},'files_searched':3232,'matches_found':{'duplicates':3030,'similar':0}}}}Additional ParametersdifPy supports the following parameters:difPy.build(*directory,recursive=True,in_folder=False,limit_extensions=True,px_size=50,show_progress=True,processes=None)difPy.search(difpy_obj,similarity='duplicates',rotate=True,lazy=True,show_progress=True,processes=None,chunksize=None)CLI UsagedifPy can also be invoked through the CLI by using the following commands:pythondif.py#working directorypythondif.py-D'C:/Path/to/Folder/'pythondif.py-D'C:/Path/to/Folder_A/''C:/Path/to/Folder_B/''C:/Path/to/Folder_C/'difPy CLI supports the following arguments:dif.py[-h][-DDIRECTORY[DIRECTORY...]][-ZOUTPUT_DIRECTORY][-r{True,False}][-i{True,False}][-le{True,False}][-pxPX_SIZE][-sSIMILARITY][-ro{True,False}][-la{True,False}][-procPROCESSES][-chCHUNKSIZE][-mvMOVE_TO][-d{True,False}][-sd{True,False}][-p{True,False}]If no directory parameter is given in the CLI, difPy willrun on the current working directory.When running from the CLI, the output of difPy is written to files andsaved in the working directoryby default. To change the default output directory, specify the-Z / -output_directoryparameter. The "xxx" in the output filenames is the current timestamp:difPy_xxx_results.jsondifPy_xxx_lower_quality.jsondifPy_xxx_stats.jsonFor adetailed usage guide, please view the officialdifPy Usage Documentation.