Started implementing robot.txt crawl delay support