Loosen robots.txt content-type validation

We should check for any form of `text/plain` - we're not supposed to care what the charset is, for example.