Updated master_crawl and other files for better crawls