paulirish/html5-boilerplate — Issue #804: Implement X-Robots-tag headers (avoid dev/stage websites "leaking" into SERPs)
Author: maniqui
Created: 2011-10-10 20:14:31+00:00
State: open

Short version: Disallowing crawlers on robots.txt isn't enough.
See:
http://yoast.com/prevent-site-being-indexed/
http://code.google.com/web/controlcrawlindex/docs/robots_meta_tag.html
Matt Cutt (from Google Webmaster Central) on this topic:
http://www.youtube.com/watch?v=KBdEwpRQRD0

Long version:

Yes, compliant robots won't crawl it, but that doesn't exactly means that the website (well, its URL) won't get indexed.
Matt Cutt explains it fairly well in the linked video, but here's the short version: if your website gets linked from somewhere else (other websites?) out there, Google may then list your website's URL in SERPs, even if Google respects your robots.txt.

At least, I've experienced (probably, I'm the guilty one for being careless, for not doing the robots.txt disallow dance) one or two times that development websites are "leaked" into Google SERPs.
Why? I can't tell exactly.
Maybe I or other developer or the client "talked" about the dev website (dev.example.com) on some email exchange via GMail, and thus, the Big Google Brother "got to know" about the existence of dev.example.com...

So, once your development website gets leaked into SERPs, you probably want to remove it from there.
Doing the robots.txt disallow dance may not help on that, as your website already leaked.
Thus, X-Robots-tag headers may be a good approach to solve the situation, imo.
Even more, this approach **is not** to be used in tandem with disallowing crawlers via robots.txt.

From [Google's docs on Control Crawling and Indexing](http://code.google.com/web/controlcrawlindex/docs/robots_meta_tag.html):

> Combining crawling with indexing / serving directives
Robots meta tags and X-Robots-Tag HTTP headers are discovered when a URL is crawled. If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving directives will not be found and will therefore be ignored. If indexing or serving directives must be followed, the URLs containing those directives cannot be disallowed from crawling.

So, here is the proposed snippet to add in .htaccess.
Please, feel free to improve the wording I've used to try to explain this.

# -----------------------------------------------------------------------------------------
# Disable URL indexing by crawlers (FOR DEVELOPMENT/STAGE)
# -----------------------------------------------------------------------------------------

# Avoid search engines (Google, Yahoo, etc) indexing website's content
# http://yoast.com/prevent-site-being-indexed/
# http://code.google.com/web/controlcrawlindex/docs/robots_meta_tag.html
# Matt Cutt (from Google Webmaster Central) on this topic:
# http://www.youtube.com/watch?v=KBdEwpRQRD0

# IMPORTANT: serving this headers is recommended only for
# development/stage websites (or for live websites that don't
# want to be indexed). This will avoid the website
# being indexed in SERPs (search engines result pages).
# This is a better approach than using robots.txt
# to disallow the SE robots crawling your website,
# because disallowing the robots doesn't exactly
# mean that your website won't get indexed (read links above).

#
# Header set X-Robots-Tag "noindex, nofollow, noarchive"
#
# Header set X-Robots-Tag "noindex, noarchive, nosnippet"
#
#