checkRobots() in working order with caching