I totally agree. In fact, it took a great deal of restraint to not refactor the entire parser. I left it alone for a couple reasons:

1. it is battle tested: we have been using this parser ([in a different form](https://github.com/fizx/robots/blob/master/lib/robots.rb)) for years
2. I was trying to keep this port as simple as possible.

I do plan on cleaning up the parser but when I do that, I will need a day or two as I also plan to refactor it so that the parser is compliant with rfc specs.

If you still believe this change should happen now, I am happy to make the change.