Proper support for robots.txt