lingpy/lingpy — pull request #110 by bambooforest (2014-11-06 15:44:27+00:00)
Title: Clean scripts directory

Removed non-lingpy specific tokenization files (tests being moved to proper unit tests for the tokenizer; the rest doesn't belong here).

Note the lingpy/scripts tokenize directory can be deleted, I think. @lingpy have a look at testtokenizer.py

PS: Next time I'll make sure I'm PR'ing on a branch that's not master on my fork.