UX, add extra link
Only show the crawl menu item if the domain has a manual crawl.
Don't run the crawler on create if the crawl is manual.
Change the default for a crawl to 10.000 pages.
Only allow the domain to be force crawled if the domain is updating manually.
Fix the URL crawler because there is no status on create.
Fix the issues with the finished version of the web crawler finished job by looping through the urls to update the redirects
Don't save the content of the robots.txt file if the status code is != 200.
Fix the issues on the urls index.