Add some unit tests, and simplifications in StringReader::getTokens()

These patches don't change any behavior (except for adding the `--test` option); I was mainly exploring the StringReader code to figure out why my grammar file was putting it in an infinite loop (my syntax was bad), and saw a few places where things could be simplified.

Anyway, feel free to discard all this. Just messing around really. :)