Make max crawl retries configurable