added duplicate finder and deleter to the multiprocessing piece