elm-city-craftworks/turing_tarpit — pull request #5 by steveklabnik (2012-01-18 17:37:38+00:00)
Title: Broke out a Tokenizer class from Scanner

Private is a smell. Especially when you have a bunch of private methods.
It usually means there's a class hiding inside.

I drew this diagram of dependencies between methods:
http://i.imgur.com/PPhXb.jpg

As you can see, the Scanner class at the bottom is strange: only
one method gets used from outside, and everything else is private.
This suggests that the cluster of methods that _aren't_ \#next should
be some sort of class. The best strategy is to look at the names and
see if they feel grouped in some way.

It wasn't all sunshine in paradise, however. Reading the method names,
I kept saying 'man, this feels like a Scanner to me.' So I decided
to turn the tables: what does \#next imply?

To me, it implies a Tokenizer. Ahhhh, Java! But that's really what
the Interpreter wants: a stream of tokens. Damn the logic for actually
getting them. So I renamed Scanner to Tokenizer, and broke out
the private methods into a class named Scanner.

This feels better from an SRP standpoint as well. The next method is
all about the logic concerning which token we want next, but all the
others are about the manipulation of the actual data structure itself.
\#next is a higher level interface built on top of the lower-level
Scanner. Rock.

I also had to update the tests.