Set correct crawl delays for API and pages.