Reconciled robots.txt with protocol

Reconciled robots.txt with Robots Exclusion Protocol. Defaults now specify that all pages are available and sitemap resides at /sitemap.xml.

See http://www.robotstxt.org/robotstxt.html and http://support.google.com/webmasters/bin/answer.py?hl=en&answer=183669 for more details.
Merge pull request #1 from benjaminestes/patch-1

Reconciled robots.txt with protocol