This is a python-module to find duplicate files in different directories.

Usage
python  find_duplicates  dir1 dir2 dir3 etc

Result
The programs writes several files to the working directory. The files with extension tmpfiles are used
in the successive steps. The endresult is a file called duplicates_found_timestamp.txt. This is a text file
with a listing of found duplicates. The duplicates are checked by checksum and therefore real duplicates!

What to do with it
It is a peace of cake to edit this textfile with whatever texeditor you have.  Just remove the # before the files
you like to remove. Remove the rest e.g. cat yourfile | grep -v -e ^# > newfile
You then have a file with the names and paths of the files you want to delete. With another one-liner you can delete your
files. cat outputfile | while read ff; do rm -v "$ff";done

Wy all the trouble
It is all little nerdy and you won't need this for small directories. There great and for better programs, that detect
the duplicates for you. However I ran into problems when I tried to sanitize large directory on a mounted server (+30000 files).
This prompted me  to write this program. This little piece of code did the trick for me. It ran for two hours and
gave me the output I needed.  With the resulting text file I was able to decide what files to remove.
The scanning of large directories
was also the reason I created temporary files. In case of a hiccup, there was some sort of control. It is a little
messy by the files are easily recognisable and therefore easy to remove. In the future I will make an option to
remove them automatically.

This will be a work in progress. As I borrowed heavily from others, especially from the people of stackoverflow
,this code is free to be used and adapted. When you find errors or if you have suggestions for improvement.
Feel free to leave me message on github.

Yours truly,

HGP