zencephalon/Tactful_Tokenizer — pull request #7 by SergeyKishenin (2014-04-30 05:16:49+00:00)
Title: Recognition of russian abbr names

It will be cool if tokenizer will recognize russian abbr names like "В.Ф. Иванов" and make it a single token.

I've checked in irb a regexp that should do that but when i put it to word_tokenizer.rb nothing happens.
I put the following there:

```[/([\p{Word}]{1}\.[\p{Word}]{1}\.)\s([\p{Word}]{1,}[^\.\s])/u, '\1\2']```

Any ideas?