Improve performance when parsing very large robots.txt files