finished tokenizer. can be modified to use something other than strings to represent tokens.
its ugly.