Prevent urls from being crawled twice