update crawl to check if there is duplicate url/title