Checked that a robots.txt exists