Adds sample robots.txt file.
Uncomments robots.txt spider tests and makes them pass.