deduplicator:
changed variable names to be easier to read
added progress log
commented out debug print statements
fixed bug from last commit
comment