Adding a test to check for robots.txt