Package: eleve
What is ELeVE ?ELeVE is a library intended for computing an “autonomy estimate” score for substrings (all n-grams) in a corpus of text.The autonomy score is based on normalised variation of branching entropies (nVBE) of strings, See[MagistrySagot2012]for a definiton of these termsIt was developed mainly for unsupervised segmentation of mandarin Chinese, but is language independant and was successfully used in research on other tasks like keyphrase extraction.Full documentation is available onhttp://pythonhosted.org/eleve/.In a nutshellHere is a simple “getting started”. First you have to train a model:>>> from eleve import MemoryStorage
>>>
>>> storage = MemoryStorage()
>>>
>>> # Then the training itself:
>>> storage.add_sentence(["I", "like", "New", "York", "city"])
>>> storage.add_sentence(["I", "like", "potatoes"])
>>> storage.add_sentence(["potatoes", "are", "fine"])
>>> storage.add_sentence(["New", "York", "is", "a", "fine", "city"])And then you cat query it:>>> storage.query_autonomy(["New", "York"])
2.0369977951049805
>>> storage.query_autonomy(["like", "potatoes"])
-0.3227022886276245Eleve also store n-gram’s occurence count:>>> storage.query_count(["New", "York"])
2
>>> storage.query_count(["New", "potatoes"])
0
>>> storage.query_count(["I", "like", "potatoes"])
1
>>> storage.query_count(["potatoes"])
2Then, you can use it for segmentation, using an algorithm that look for the solution which maximize nVBE of resulting words:>>> from eleve import Segmenter
>>> s = Segmenter(storage)
>>> # segment up to 4-grams, if we used the same storage as before.
>>>
>>> s.segment(["What", "do", "you", "know", "about", "New", "York"])
[['What'], ['do'], ['you'], ['know'], ['about'], ['New', 'York']]InstallationYou will need some dependencies. On Ubuntu:$ sudo apt-get install python3-dev libboost-python-dev libboost-filesystem-dev libleveldb-devThen to install eleve:$ pip install eleveor if you have a local clone of source folder:$ python setup.py installGet the sourceSource are stored ongithub:$ git clone https://github.com/kodexlab/eleveContributeInstall the development environment:$ git clone https://github.com/kodexlab/eleve
$ cd eleve
$ virtualenv ENV -p /usr/bin/python3
$ source ENV/bin/activate
$ pip install -r requirements.txt
$ pip install -r requirements.dev.txtPull requests are welcome!To run tests:$ make testallTo build the doc:$ make docthen open:docs/_build/html/index.htmlWarning: You need to haveeleveaccesible in the python path to run tests (and to build doc).
For that you can installeleveas a link in local virtualenv:$ pip install -e .(Note: this is indicated in pytestgood practice)ReferencesIf you useelevefor an academic publication, please cite this paper:[MagistrySagot2012]Magistry, P., & Sagot, B. (2012, July). Unsupervized word segmentation: the case for mandarin chinese. In Proceedings of the 50th Annual Meeting of the ACL: Short Papers-Volume 2 (pp. 383-387).http://www.aclweb.org/anthology/P12-2075Copyright, license and authorsCopyright (C) 2014-2015 Kodex⋅Lab., 2020 Pierre Magistryeleveis available under theLGPL Version 3license.elevewas originaly designed and prototyped byPierre Magistryduring its PhD. It then has been completly rewriten byKorantin AugusteandEmmanuel Navarro(with the help of Pierre).For the sake of easier installation and evolution, it was later refactored in Cython by Pierre Magistry.