Add robots.txt handling.