lingpy/lingpy — pull request #152 by LinguList (2014-12-15 19:35:58+00:00)
Title: Increasing the accuracy of default tokenization/segmentation in LingPy

This PR does not change much, but it adds

* a new and long list based on the [BDPA](http://alignments.lingpy.org) in order to illustrate the capability of default ipa-string segmentation in LingPy
* more background-data on common diacritics which are used as a default for handling tokenization in LingPy

This PR further illustrates the benefits of ipa2tokens:

* consistency with more than 10 000 unique sequences written in IPA transcription
* capability to handle tones
* capability to handle accents
* capability to handle combining characters that make one token out things that would otherwise be treated as multiple tokens
* speed in terms of computation (ipa2tokens is twice as fast as the orthoprofile-tokenizer, and consistent with all test items, please see [this example](https://github.com/LinguList/cookbook/blob/master/timing/timer.py) in the future "lingpy.cookbook" where this is nicely illustrated with ipa2tokens taking 1.38 seconds on my computer vs. 3.02 seconds for the alternative solution

When testing this PR on your machine, make sure that you clean your lingpy.cache, since otherwise the changes won't work.