added test for encoding weirdness
moved test to TokenizerSpecTest
changed to handle tokens with whitespace/control chars
Merge pull request #33 from afader/master

fixed bug in computeOffsets