update settings to optimize crawl behavior