catch remove exceptions
do a recheck on robots.txt.