Added a robots.txt crawler