Ignoring errors from AhrefsBot TurnitinBot crawlers
More bots (including slurp from Yahoo)
Ignoring DotBot errors