Adding the crawler politeness criterion and deal to crawl