Use our user-agent when parsing robots.txt
Making it multibyte safe