Be slightly more forgiving on robots.txt content type

I.e. allow `text/plain; charset=utf8`