Crawler now rejects duplicates.
Moving spec to the right location.