Finished the robots.txt crawl delay support