steamruler/DuplicateDestroyer — pull request #2 by whisperity (2013-04-06 09:17:33+00:00)
Title: Increase userland communication and improve internal logic

## User communication and feedback
### Dry run switch
While it was a thing with safety in mind to have the user use an `-ok` argument, it pretty much **destroys** how software should work: that by default it should perform its default action, and every other optional change in operation comes afterwards. And the default action is **to** destroy duplicates. So I inverted the code by this logic: instead of specifically setting that we **should do** something, I put a `-d` (dry run) switch to say that **no, we should not**.

### Selection about which file to keep
Also I have added a "menu" where, after the program analyzed everything, the user can select which file to keep. This was done because the previous logic implyed that the first file ever read (in a set of duplicates) is always kept, and everything afterwards is deleted... and that's something I guess is not right. (This change also related to the modifications in the internal logic!)

In my version of the code, the user is now presented to select the one file he/she wishes to keep, and everything else is removed afterwards. (Or alternatively, `-o` to automatically keep the oldest, `-n` to automatically keep the newest one, based on the file's _atime_.)

## Improvements in internal logic
Previously, the software kicked the door down on entering the house and created a glorious overload in memory (ultimately ending up the whole executable crashing down). I have, using the powers of `System.Collections.Generic` and `System.Linq`, improved this so all the data result from filesystem analysis is kept in order and easy for the system.

Also, with this change, the workflow was divided into clear, seperate segments:

1. Calculate the number of files in the target structure.
2. Analyze the structure and search for file sizes.
* As per whisperity/DuplicateDestroyer@152b0a882eeef408e3c69a48dc1ffcdacb5c04ba. I would like to mark that commit **the** most maior improvement to the code so far, as it has this fixed a number of crashes of previous heavy testing (think about running this on your Garry's Mod content folders...).
3. Build a list of files which share the same size. These are called _possible duplicates_ internally.
4. Read contents of all _possible duplicates_ and, just as how it was since the beginning, calculate the hash. This is where I eliminated the need to read all files.
* Using the above example (my Garry's Mod content folder), the number of files to be actually read went down by 10%.
* The file size is ordered `DESC`, so the biggest file is read first.
5. Perform analysis on the read hashes and build a list of files to remove.
6. The user selects the file to keep from each duplicate set.
7. Expunge the remaining files based on the user's choice.
8. `Environment.Exit(0);`

The application became a bit more CPU and somewhat more memory heavy because of the increased data to keep "in mind", but this change eliminated a great deal of **storage heavyness**. And because HDDs are still traditional, this makes the execution lightweight in general. In the previous version, the whole execution was memory and HDD-heavy (constant file read, calculation and removal). Now, only stage 4 (reading contents) is **the** HDD heavy phase, and the CPU is given the biggest load in stage 5.

Also, because of putting the file removal step to the end, I would consider this to be actually **safer**. If and when the program is to crash before it can analyze the structure, no live data is lost.

The test example:
* 131.259 files in the structure.
* 14.938 unique file size found for 120.552 files.
* There are no detailed view like `1.2 GiB: 3 files. 670 MiB: 9 files.` This only means what it says: we have 14k of exact, distinct file sizes and we have 120k _possible duplicates_.
* Also keep in mind that the example has a vast structure of `.svn` folders containing versioning duplicates of content files.
* Bottom line: it is not the best example, but it's the most fitting I can get by using a structure on my computer.
* 32.952 unique content dupicated across 78.853 files.
* Total time taken: 30 minutes, but in dry run, so there was no file removal.

Everything else is to be seen in the source code. :beers: :wink: