sahib/rmlint — pull request #44 by SeeSpotRun (2014-06-13 12:04:57+00:00)
Title: Added new options, unified some code, fixed minor bug re bind mounts.

Some experimental changes for testing by others:

(1) Added new option "keepallorig" which means no files in 'original'
path(s) prefixed by // will be deleted.
(2) added new option "mustmatchorig" which limits searching to only
find matches wherein at least one file is in 'original' path(s).
(3) this required some structural change so that more than one input
path can be designated as original.

So why add these options?
Consider the case where you are about to reformat a partition and, you
think everything on it is just a backup of other files on your main system.
If you do:
rmlint /path/to/old/filesystem ///all/your/other/files

... then it will take a very long time to run because rmlint will look
for duplicates across all those filesystems.
Instead, with:
rmlint --mustmatchorig --keepallorig /path/to/old/filesystem \
///all/your/other/files

... rmlint will now only look for dupes in the old filesystem which have
originals in your main filesystem.

(4) To make this work, it was going to need extra code in several
routines (at the fingerprint, md5 and paranoid level) to avoid
unneccesary comparisons. Instead it ended up a lot neater to kind of
unify the different comparison routines into one (findmatches), which
sorts into sub-islands of similar files and then calls itself
recursively to submit those sub-islands to the next level of rigour.
I nice spin-off from this is that it makes it very easy if you want to
add some more intermediate check levels of rigour (eg for very large
files, you could do md5 comparisons on 25, 50, 75% of the file).
(5) As part of this, I had to drag a couple of routines across from
mode.c into filter.c (which makes the level of changes look a lot more
severe than it actually was).
(6) The above changes broke the existing method for totalling up the
total GB of files that can be deleted so I put in a simple global
running total instead.
(7) I also changed the -o optional argument to a required one, which
makes for a slightly nicer "-o myfile" style rather than "-omyfile" but
I think it breaks the option for no log file. Anyway maybe this option
still needs a bit more work.
(8) Also there was a potential bug which meant that recursive bind mounts could lead to deletion of a file that wasn't a duplicate. To replicate this:
$ mkdir -p foo/bar/foo2
$ sudo mount --rbind foo foo/bar/foo2
$ (put some files in foo)
$ rmlint foo
This tricked the existing logic in rmlint because only one path was input, which would normally mean no risk of file clashes from different paths.

I think that was about all but may have missed something.

TODO: I don't think the memory 'free' commands are right yet so there
is probably some memory leakage.