Check user agents against robots.txt files