Add option 'cross_host_allowed' to limit to crawl specified URLs.