camspiers/statistical-classifier — pull request #14 by corentin-larose (2014-01-02 19:29:57+00:00)
Title: Added tokenization capability for accented languages

This little change allows the current tokenizer to tokenize accented languages as french. Plus preg_split should be way faster than preg_match_all for the tokenization purpose.