knowitall/nlptools — pull request #32 by afader (2013-12-03 07:10:39+00:00)
Title: added test for encoding weirdness

I found a bug in `ClearTokenizer` shown by this test case. It throws an exception.

The root of the problem is in `Tokenizer.computeOffsets` https://github.com/afader/nlptools/blob/master/core/src/main/scala/edu/knowitall/tool/tokenize/Tokenizer.scala#L19-L42

Here's what happens:
* send a weird string to `ClearTokenizer.tokenize`
* the underlying tokenizer returns a sequence of string tokens, where the last token is not actually a substring of the input string (for whatever reason)
* `computeOffsets` assumes that the tokens are all substrings of the original string, and then throws when this is violated

Not sure what the correct behavior should be. It would be nice if this wouldn't throw an exception i.e. you should be able to call `tokenize` on any string, even if it has some funny bytes.

Maybe the offsets should not always be available?