Fix robots.txt handling, and stop creating temporary files