Package: text-sentence
Text tokenizer and sentence splitterLibrary “text-sentence” is text tokenizer and sentence splitter.Input is for main function is text, list of known names and abbreviations.
Result is list of tokens. Each token has type and other attributes i.e.:is word,is number,is roman number,is sentence end,is abbreviation,is name,is contraction,is end of chapteretc.Determining end of sentenceneeds special logic and care what is the main
reason for naming package with “text-sentence”.TAGStokenization, sentence splitter, sentencer, chapter, names, abbreviationAUTHORRobert Lujo, Zagreb, Croatia, find mail address in LICENCEFEATURESTo name the most important:TODO: …System is based on unicode strings.CheckGetting started.INSTALLATIONInstallation instructions - if you have installed pip packagehttp://pypi.python.org/pypi/pip:pip install text-sentenceIf not, then do it old-fashioned way:download zip fromhttp://pypi.python.org/pypi/text-sentence/unzipopen shellgo to distribution directorypython setup.py installDevelopment version you can see athttp://bitbucket.org/trebor74hr/text-sentence.or Mercurial clone with:hg clone https://bitbucket.org/trebor74hr/text-sentenceGETTING STARTEDUsage example - start python shell:>>> from text_sentence import Tokenizer
>>> t = Tokenizer()
>>> list(t.tokenize("This is first sentence. This is second one!And this is third, is it?"))
[T('this'/sent_start), T('is'), T('first'), T('sentence'), T('.'/sent_end),
 T('this'/sent_start), T('is'), T('second'), T('one'), T('!'/sent_end),
 T('and'/sent_start), T('this'), T('is'), T('third'), T(','/inner_sep),
 T('is'), T('it'), T('?'/sent_end)]More samples can be found in tests:http://bitbucket.org/trebor74hr/text-sentence/src/tip/text_sentence/test_sentence.txtFurtherSince there is currently no good documentation, the best source of
further information is by reading tests inside of module and
tests test_sentence. More information inRunning tests.
You can allways read a source.DOCUMENTATIONCurrently there is no documentation. In progress …SUPPORTSince this project is limited by my free time, support is limited.REPORT BUG OR REQUEST FEATUREIf you encounter bug, the best is to report it to the bitbucket web pagehttp://bitbucket.org/trebor74hr/text-sentence.The best way to contact me is by mail (find in LICENCE).TODO list is in readme.txt (dev version).CONTRIBUTIONSince this project is not currently in the stable API phase, contribution
should wait for a while.RUNNING TESTSAll tests are doctests (not unittests). There are two type of tests in the
package:doctests in module i.e. in __init__.pydoctests in test_sentence.txtRunning module directly will run 1. and 2.To run tests:goto text_sentence directoryrun tests by running module, e.g.:> python __init__.py
__main__: running doctests
test_sentence.txt: running doctestsother with:> python -m"text_sentence"TODOvarious things, see readme.txt in dev version for details.CHANGES0.14ulr1 100621:is_contraction token attribute - e.g. isn’t or oš’0.13ulr1 100619:sample in getting started0.12ulr1 100619:test_sentence.txt installationreadme fix main title0.11ulr1 100618:adapted tests__init__.py and sentence.py0.10ulr1 100617:first installable release