[gh-689] Remove the robotstxt module and restore robots.txt file.
Merge pull request #693 from tsteiner2/issue-689

Closes #689