add robots.txt check for initial page and support for crawl-delay