Reconfigured Crawler; Added Program to Remove Queue Duplicates

Note: The Duplicate Removal program will likely miss some duplicates due
to the time it would take to check every ID with every ID ever seen up
to that point. Use at your own discretion.