koenbok/Cactus — pull request #50 by benjaminestes (2013-04-22 16:40:08+00:00)
Title: Reconciled robots.txt with protocol

Reconciled robots.txt with Robots Exclusion Protocol. Defaults now specify that all pages are available and sitemap resides at /sitemap.xml.

See http://www.robotstxt.org/robotstxt.html and http://support.google.com/webmasters/bin/answer.py?hl=en&answer=183669 for more details.