knowitall/nlptools — pull request #33 by afader (2013-12-03 21:43:44+00:00)
Title: fixed bug in computeOffsets

The bug was actually in `computeOffsets`. Some tokenizers can return tokens containing whitespace or control characters. The previous implementation would strip these out when computing offsets.

This new implementation should handle it. Also added unit test for it.