Added a basic test for robots.txt