Pretty much, yes, that is exactly how it is being used. It used to be in the parser but was done subsequent to `get_robots_text` (http request). That request did not belong in the parser so when I moved that call out, I moved this out as well. I also did this since we had to do a little song-and-dance with `StringIO` given we use `URI::HTTP` (which uses `OpenURI`) for the http call.

Shoudl I add documentation?