mumbly/PyUtils — pull request #1 by endafarrell (2013-02-16 21:28:49+00:00)
Title: A change which may make it easier to use the dupes.out file.

Hi - I'm using your FileDupeFinder.py to find duplicates and I've made a change that helps me to then get rid of the duplicates. Hopefully it might be of use to you ;-)

/e

The first few changes are cosmetic, but as I have 10s of thousands of photos
(which contain many many duplicates due to changing photo management software
too often) I found that I was spending a lot of time in the calculation of the
full md5 and the script /seemed/ to hang.

The last change is one where I print a CSV with the colum order changed and the
lenght of the filename being included. The reason for these are best seen
through a short example.

The next block is the first few lines of my dupes.out:
{code}
1151395,2061592c7402a3d5dbd2caa97c62b321,54,32333374,"/Users/enda/Pictures/ApertureExports/2009/IMG_5039.jpg"
1151395,2061592c7402a3d5dbd2caa97c62b321,62,32731626,"/Users/enda/Pictures/ApertureExports/2009/2009-04/IMG_5039.jpg"
1151395,2061592c7402a3d5dbd2caa97c62b321,62,32731627,"/Users/enda/Pictures/ApertureExports/2009/2009-04/IMG_5039 (1).jpg"
1166554,ce9a988a3aedef241f1e9f5bb38a7ac7,58,32380095,"/Users/enda/Pictures/ApertureExports/2009/IMG_1853 (1).jpg"
1166554,ce9a988a3aedef241f1e9f5bb38a7ac7,54,32380096,"/Users/enda/Pictures/ApertureExports/2009/IMG_1853.jpg"
1167463,bdbf000fcbfcffb55f44eb548de3586f,58,32380454,"/Users/enda/Pictures/ApertureExports/2009/IMG_1893 (1).jpg"
1167463,bdbf000fcbfcffb55f44eb548de3586f,54,32380455,"/Users/enda/Pictures/ApertureExports/2009/IMG_1893.jpg"
1187517,5d55ecef889588be9e62df725a88f690,54,32333096,"/Users/enda/Pictures/ApertureExports/2009/IMG_3032.jpg"
1187517,5d55ecef889588be9e62df725a88f690,62,32731446,"/Users/enda/Pictures/ApertureExports/2009/2009-04/IMG_3032.jpg"
1197276,7d0c03f256299ecd15746776d0a5e484,54,32335523,"/Users/enda/Pictures/ApertureExports/2009/IMG_3303.jpg"
1197276,7d0c03f256299ecd15746776d0a5e484,62,32732405,"/Users/enda/Pictures/ApertureExports/2009/2009-05/IMG_3303.jpg"
{code}

Here we can see that I have duplicates in different subdirs and sometimes with
" (1)" added to the name. I have thousands of dupes like this, so I don't want
to have to manually process such a file. Here's what I do:

I want to keep the shortest-named file. This gives me the "top-level" directory
for a dupe, or the original (before the " (1)" was added. That's why the first
"varying" column is the len(filename).

I then send my dupes.out through sort so that in a set of dupes - two, three or
more - the shortest one is "on top". As we want this third column to be sorted
numerically we have:
sort -t',' -k1,2 -k3n

Next comes an awk one-liner that needs explaining:
awk -v sz=-1 -F ',' '$0 ~ sz "," md5 { print "rm " $5}; {sz=$1; md5=$2;}'

The <<-v sz=-1>> pre-initialises an awk variable called "sz" to a value of "-1".
Later this "sz" becomes the first column (which is the filesize), and so in the
first line the "sz" has a value which cannot be matched.
The <<-F ','>> tells awk that it's a CSV.
The <<$0 ~ sz "," md5>> is a test for awk - find the lines matching the value of
the variables <> and <> seperated by a comma. Our dupes are these such
matches!
The <<{ print "rm " $5};>> gets awk to print the fifth column when the match is
made.
The << {sz=$1; md5=$2;}>> sets the value of the variables <> and <> to
the first and second column of the line it's just read.
all that I then get the next:
{code}
rm "/Users/enda/Pictures/ApertureExports/2009/2009-04/IMG_5039.jpg"
rm "/Users/enda/Pictures/ApertureExports/2009/2009-04/IMG_5039 (1).jpg"
rm "/Users/enda/Pictures/ApertureExports/2009/IMG_1853 (1).jpg"
rm "/Users/enda/Pictures/ApertureExports/2009/IMG_1893 (1).jpg"
rm "/Users/enda/Pictures/ApertureExports/2009/2009-04/IMG_3032.jpg"
rm "/Users/enda/Pictures/ApertureExports/2009/2009-05/IMG_3303.jpg"
{code}