Package: text-hr
Morphological/Inflection/Lemmatization Engine for Croatian language“text-hr” is Morphological/Inflectional/Lemmatization Engine for Croatian
language written in Python programming language. Includes stopwords and
Part-Of-Speech tagging engine (POS tagging) based on inverse inflection
algorithm for detection.Since API is not freezed, this project is still in alpha.TAGSCroatian language, lemmatization, stemming, inflection, python, natural
language processing (NLP), Part-of-speech (POS) tagging, stopwords, inverse
inflection, morphological lexiconOZNAKEHrvatski jezik, lematizacija, Python biblioteka, morfologija, infleksija,
obrnuta infleksija, prepoznavanje vrsta riječi, računalna obrada govornog
jezika, zaustavne riječi, morfološki leksikonAUTHORRobert Lujo, Zagreb, Croatia, find mail address in LICENCEFEATURESTo name the most important:inflection system - for producing all forms of one worddetection of word types (POS tagging) - from existing list of word formslist of stopwordsSystem is based on unicode strings, default codepage to convert from and to
string is cp-1250.CheckGetting started.INSTALLATIONInstallation instructions - if you have installed pip packagehttp://pypi.python.org/pypi/pip:pip install text-hrIf not, then do it old-fashioned way:download zip fromhttp://pypi.python.org/pypi/text-hr/unzipopen shellgo to distribution directorypython setup.py installGETTING STARTEDThere are three important parts that this project provides:Inflection system- for producing all forms of one wordDetection of word types (POS tagging)- from existing list of word formsList of stopwordsInflection systemUsage example - start python shell:>>> from text_hr import Verb
>>> v = Verb("platiti")
>>> for k in sorted(v.forms.keys()):
...     print(k, v.forms[k])
...
AOR/P/1 [u'platismo']
AOR/P/2 [u'platiste']
AOR/P/3 [u'plati\u0161e']
AOR/S/1 [u'platih']
AOR/S/2 [u'plati']
AOR/S/3 [u'plati']
IMP/P/1 [u'platasmo', u'pla\u0107asmo', u'platijasmo']
IMP/P/2 [u'plataste', u'pla\u0107aste', u'platijaste']
IMP/P/3 [u'platahu', u'pla\u0107ahu', u'platijahu']
...
VA_PA//P_O+S+V+N [u'pla\u0107eno']
X_INF// [u'platiti']
X_VAD_PAS// [u'plativ\u0161i']
X_VAD_PRE// [u'plate\u0107i']
X_VAD_PRE// [u'plate\u0107i']Detection of word types (POS tagging)TODO: to be done - check test_detect.txt for samples, and detect.py for the logic:First example in test_detect.txt:>>> from text_hr.detect import WordTypeRecognizerExample
>>> def test_it(word_list, wt_filter=None, level=2):
...     wdh = WordTypeRecognizerExample(word_list, silent=True)
...     if not wt_filter is None:
...         wdh.detect(wt_filter=wt_filter, level=level)  # e.g. wt_filter=["N"]
...     else:
...         wdh.detect(level=level)  # all word types
...     lines_file = LinesFile()
...     wdh.dump_result(lines_file) # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS
...     print("\n".join(lines_file.lines))
...     return wdh

>>> class LinesFile(object):
...     def __init__(self):
...         self.lines = []
...     def write(self, s):
...         self.lines.append(repr(s.rstrip()))

>>> word_list = [
...   "Broj    84"
... , "broji   34"
... , "Brojila  28"
... , "broje   23"
... , "brojeći 22"
... , "brojim   7"
... , "brojimo  5"
... , "brojiš   4"
... , "brojahu  2"
... , "brojaše  1"
... , "brojite  1"
... , "-brijestovu 1"
... , "brijestovi 1"   #the only one checked with endswith, but all other will be checked with get_freq
... , "-brijestove 1"
... , "-brijestova 1"
... ]

Lowest quality, but fastest
>>> wdh = test_it(word_list, level=4) # doctest: +ELLIPSIS
" 10/  183 -> brojati              (u'V-XX_-_JATI-je\\u0107i-0') 84/broj,34/broji,23/broje,22/broje\xe6i,7/brojim,5/brojimo,4/broji\x9a,2/brojahu,1/brojite,1/broja\x9ae"List of stopwordsIs located in std_words.txt, and you can read it directly from herehttp://bitbucket.org/trebor74hr/text-hr/src/tip/text_hr/std_words.txtThe list can be updated like this:>>> import text_hr
>>> text_hr.dump_all_std_words()
Totaly 2904 word forms dumped to r:\hg-clones\python\text-hr\text_hr\std_words.txt in codepage utf8Iteration over all words goes like this:from text_hr import get_all_std_words

for word_base, l_key, cnt, _suff_id, wform_key, wform in get_all_std_words():
    print(word_base, l_key, cnt, _suff_id, wform_key, wform)FurtherSince there is currently no good documentation, the best source of
further information is by reading tests inside of modules and
tests in tests directory (dev version). More information inRunning tests.
You can allways read a source.DOCUMENTATIONCurrently there is no documentation. In progress …SUPPORTSince this project is limited by my free time, support is limited.REPORT BUG OR REQUEST FEATUREIf you encounter bug, the best is to report it to the bitbucket web pagehttp://bitbucket.org/trebor74hr/text-hr.If there will be an interest for development for other inflection rich
languages, I’d be glad to decouple language specific code and create new
project that will be capable to deal with multiple languages.The best way to contact me is by mail (find in LICENCE).TODO list is in readme.txt (dev version).CONTRIBUTIONSince this project is not currently in the stable API phase, contribution
should wait for a while.RUNNING TESTSAll tests are doctests (not unittests). There are three type of tests in the
package:doctests in each module - e.g. in verbs.pydoctests in tests/test_*.txt - only development versiontests which are not automatically compared - i.e. in special call mode
detect.py can produce output file which needs to be compared
manually with some existing file. Such test(s) are very slow. This needs
to be changed to be automatic.Running each module directly will run 1. and 2. if running from development
version. To get development version
To use development version (http://bitbucket.org/trebor74hr/text-hr):hg clone https://bitbucket.org/trebor74hr/text-hrcreate text_hr.pth in python site-packages directory with path to text-hr e.g.:r:\hg-clones\python\text-hrTo run all tests:go to tests directoryrun tests.py like (with sample output):> python tests.py
testing module   __init__
testing module   adjectives
...
testing textfile R:\hg-clones\python\text-hr\tests\test_adj.txt
...
testing textfile R:\hg-clones\python\text-hr\tests\test_verbs_type.txtTo run tests for just one module:goto text_hr directoryrun tests by running module, e.g.:> py pronouns.py
__main__: running doctests
..\tests\test_pronouns.txt: running doctestsin the case you’re not running from dev version, you’ll get output like
this:> py pronouns.py
__main__: running doctests
..\tests\test_pronouns.txt: Not found, skippingADDITIONALMaster thesis pdf in Croatian (134 pages) with title:Lociranje sličnih logičkih cjelina u tekstualnim
dokumentima na hrvatskome jezikucan be found at:http://bitbucket.org/trebor74hr/text-hr/downloads/magistarski-konacni.pdfTODOvarious things, see readme.txt for details.CHANGES0.20RL 200507migration to python 3+, tested on python 3.7, all tests pass0.18RL 121210fixed wrong readme on bitbucket homepage0.17RL 100617utf-8 setup0.16RL 100617master thesis pdf added to repository (in Croatian, 134 pages)0.15RL 100617minor changes0.14RL 100617beta releasetags: lemmatization, stemming0.13RL 100610:text_hr package reorganized (__init__.py with __all__ and imports …)word_types.py removedstd_words.txt0.12RL 100608 :READMEenabled tests from tests.py for allenabled tests from directly from each modules0.11RL 100607:recreated repo at bitbucketno .suff_registry.pickle and testing_*.out put in zip0.10RL 100605:first installable release