Update robots.txt to disallow bots.

* Add a functional test to test the contents.

fixes #84