changes to crawlers