added test to follow robot.txt