A fix to honor robots.txt. Trying a better fix.