Updating the tests for robots.txt