Integration of robots.txt support