Better robots.txt handling