scrapinghub/webstruct — pull request #20 by chekunkov (2014-06-07 12:30:06+00:00)
Title: Tokenizer fixes and span_tokenize method

Tokenizer from #15 had issues like don't splitting dot at the end of the sentence as separate token

```
40006,40007c40017
< community
< .
---
> community.
41148,41149c41158
< Reserved
< .
---
> Reserved.
```

Now this issue should be fixed.

Also I've refactored code and added span_tokenize method (@kmike I remember you said it would be nice to have this method)

Performance wasn't heart much

```
X, y = webstruct.HtmlTokenizer().tokenize(trees)

CPU times: user 3.42 s, sys: 32 ms, total: 3.46 s
Wall time: 3.45 s
```