Fix robots.txt and noarchive uppercase; add tests.
Merge pull request #601 from jcushman/dev-ohnorobot

Fix robots.txt and noarchive uppercase; add tests.