removed stdout prints for robots.txt errors