Update robots.txt to disallow bots.

* Add a functional test to test the contents.

fixes #84
Merge pull request #246 from mozilla/issue-84-robots-txt

Update robots.txt to disallow bots.