alphagov/static — pull request #408 by samjsharpe (2014-05-26 08:28:10+00:00)
Title: Block ahrefs bot in robots.txt

This bot crawls us very very regularly (sometimes more than once a day) and
causes spikes in our load and exceptions. It seems to serve no useful purpose
to us, in that it's not a search engine that users would use.

> 1. What is ahrefs.com
> Ahrefs.com is an independent tool for SEO analysis with a wide range
> of features. It is designed, first of all, for SEO specialists and
> site owners but may be of interest to other concerned Internet
> researchers.
> https://ahrefs.com/faqs.php

This change to the robots.txt is the recommended way to stop the bot crawling.
See https://ahrefs.com/robot