Added retrieval of "crawl-delay" and "sitemap" entries along with tests.