The crawl cycle is functional. Will auto-extend to new domains.