Added simple check for robots.txt