added obtaining robots.txt contents