Fix robots.txt and noarchive uppercase; add tests.

It looks like there were bugs in our auto-darchive logic -- "noarchive" was missed if it wasn't lowercase, and robots.txt was missed if the mime type was wrong. This fixes those cases and adds some test cases to ensure proper darchiving.