Add prevent_duplicates option. If set to true (default is false) will not let into the queue duplicate files. As suggested in #324 and #685.