bugfix for robots.txt exclusion logging