Integrated robots.txt