Streaming parser for robots.txt files