Automattic/Ad-Code-Manager — pull request #76 by nickdaugherty (2013-05-01 23:37:04+00:00)
Title: Add robots.txt entries for ad provider's crawler automatically

Google recommends explicitly setting a User-agent for their ads crawler - see http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1061943. If the User-agent isn't found in robots.txt, Google creates an alert in the AdSense dashboard (and emails the owner, I believe), asking the owner to add an entry to their robots.txt.

We can do this step automatically using the do_robotstxt action, making it even easier for users to add ads to their site. This Pull Request adds this support to `ACM_Provider` by adding a new `$crawler_user_agent` member to the class, which is meant to be set by child classes. If the ad provider doesn't have a crawler, leaving this member null will bypass the robots.txt modification.

Whether or not robots.txt is modified can be controlled by the `acm_should_do_robotstxt` filter, which accepts a boolean and the `ACM_Provider` instance.

The disallowed paths default to blank (we might want to make this mimic the default WP behavior, which is to disallow /wp-admin - feedback welcome) and are filterable with `acm_robotstxt_disallow`, which accepts / returns an array of disallowed paths and the `ACM_Provider` instance.

If the blog is marked as private, the site root is disallowed (which is the default behavior in WP).

Wanted to run this by anyone who is interested for feedback before committing.

Fixes #75