Adding support to simple crawl for robots.