Expanded the allowed urls that the crawler can crawl.