Package: udar
UDAR(enie)UDARDoesAccentedRussian: A finite-state morphological
analyzer of Russian that handles stressed wordforms.A python wrapper for theRussian finite-state
transducerdescribed
originally in chapter 2 ofmy dissertation.If you use this work in your research please cite the following:Reynolds, Robert J. "Russian natural language processing for computer-assisted language learning: capturing the benefits of deep morphological analysis in real-life applications" PhD Diss., UiT–The Arctic University of Norway, 2016.https://hdl.handle.net/10037/9685Feature requests, issues, and pull requests are welcome!Non-python dependenciesThis repository usesgit-lfsfor large files. Be sure todownload and install itbefore you clone/commit.For all features to be available, you should havehfstandvislcg3installed as command-line utilities. Specifically,hfstis needed for
FST-based tokenization, andvislcg3is needed for grammatical disambiguation.
The version used to successfully test the code is included in each commit inthis file. The recommended method
for installing these dependencies is as follows:MacOS$curlhttps://apertium.projectjj.com/osx/install-nightly.sh|sudobashDebian / Ubuntu$wgethttps://apertium.projectjj.com/apt/install-nightly.sh-O-|sudobash
$sudoapt-getinstallcg3hfsthfst-devInstallationUntil the first stable version is released onPyPI,udarcan be installed directly from this repository usingpip:$python3-mpipinstall--usergit+https://github.com/reynoldsnlp/udarIntroductionNB! Documentation is currently limited to docstrings. I recommend that you
usehelp()frequently to see how to use classes and methods. For example,
to see what options are available for building aDocument, tryhelp(Document).The most common use-case is to use theDocumentconstructor to automatically
tokenize and analyze a text. If youprint()aDocumentobject, the result
is anXFST/HFSTstream:importudardoc1=udar.Document('Мы удивились простоте системы.')print(doc1)# Мы	мы+Pron+Pers+Pl1+Nom	0.000000## удивились	удивиться+V+Perf+IV+Pst+MFN+Pl	5.078125## простоте	простота+N+Fem+Inan+Sg+Dat	4.210938# простоте	простота+N+Fem+Inan+Sg+Loc	4.210938## системы	система+N+Fem+Inan+Pl+Acc	5.429688# системы	система+N+Fem+Inan+Pl+Nom	5.429688# системы	система+N+Fem+Inan+Sg+Gen	5.429688## .	.+CLB	0.000000Passing the argumentdisambiguate=True, or runningdoc1.disambiguate()after the fact will run a Constraint Grammar to remove as many ambiguous
readings as possible.This grammar is far from complete, so some ambiguous
readings will remain.Data objectsDocumentobjectPropertyTypeDescriptiontextstrOriginal text of this documentsentencesList[Sentence]List of sentences in this documentnum_tokensintNumber of tokens in this documentfeaturestupleudar.features.FeatureExtractorstores extracted features hereDocumentobjects have convenient methods for adding stress or converting to
phonetic transcription.MethodReturn typeDescriptionstressedstrThe original text of the document with stress marksphoneticstrThe original text converted to phonetic transcriptiontransliteratestrThe original text converted to Romanized Cyrillic (default=Scholarly)disambiguateNoneDisambiguate readings using the Constraint Grammarcg3_strstrAnalysis stream in theVISL-CG3 formatfrom_cg3DocumentCreateDocumentfromVISL-CG3 format streamhfst_strstrAnalysis stream in the XFST/HFST formatfrom_hfstDocumentCreateDocumentfrom XFST/HFST format streamto_dictlistConvert to a complex list objectto_jsonstrConvert to a JSON stringExamplesstressed_doc1=doc1.stressed()print(stressed_doc1)# Мы́ удиви́лись простоте́ систе́мы.ambig_doc=udar.Document('Твои слова ничего не значат.',disambiguate=True)print(sorted(ambig_doc[1].stresses()))# Note that слова is still ambiguous# ['сло́ва', 'слова́']print(ambig_doc.stressed(selection='safe'))# 'safe' skips сло́ва and слова́# Твои́ слова ничего́ не зна́чат.print(ambig_doc.stressed(selection='all'))# 'all' combines сло́ва and слова́# Твои́ сло́ва́ ничего́ не зна́чат.print(ambig_doc.stressed(selection='rand')in{'Твои́ сло́ва ничего́ не зна́чат.','Твои́ слова́ ничего́ не зна́чат.'})# 'rand' randomly chooses between сло́ва and слова́# Truephonetic_doc1=doc1.phonetic()print(phonetic_doc1)# мы́ уд'ив'и́л'ис' пръстʌт'э́ с'ис'т'э́мы.SentenceobjectPropertyTypeDescriptiondocDocument"Back pointer" to the parent document of this sentencetextstrOriginal text of this sentencetokensList[Token]The list of tokens in this sentenceidstr(optional) Sentence id, if assigned at creationMethodReturn typeDescriptionstressedstrThe original text of the sentence with stress marksphoneticstrThe original text converted to phonetic transcriptiontransliteratestrThe original text converted to Romanized Cyrillic (default=Scholarly)disambiguateNoneDisambiguate readings using the Constraint Grammarcg3_strstrAnalysis stream in theVISL-CG3 formatfrom_cg3SentenceCreateSentencefromVISL-CG3 format streamhfst_strstrAnalysis stream in the XFST/HFST formatfrom_hfstSentenceCreateSentencefrom XFST/HFST format streamto_dictlistConvert to a complex list objectto_jsonstrConvert to a JSON stringTokenobjectPropertyTypeDescriptionidstrThe index of this token in the sentence, 1-basedtextstrThe original text of this tokenmiscstrMiscellaneous annotations with regard to this tokenlemmasSet[str]All possible lemmas, based on remaining readingsreadingsList[Reading]List of readings not removed by the Constraint Grammarremoved_readingsList[Reading]List of readings removed by the Constraint GrammardeprelstrThe dependency relation between this word and its syntactic head. Example: ‘nmod’.MethodReturn typeDescriptionstressesSet[str]All possible stressed wordforms, based on remaining readingsstressedstrThe original text of the sentence with stress marksphoneticstrThe original text converted to phonetic transcriptionmost_likely_readingReading"Most likely" reading (may be partially random selection)most_likely_lemmasList[str]List of lemma(s) from the "most likely" readingtransliteratestrThe original text converted to Romanized Cyrillic (default=Scholarly)force_disambiguateNoneFully disambiguate readings using methodsother thanthe Constraint Grammarcg3_strstrAnalysis stream in theVISL-CG3 formathfst_strstrAnalysis stream in the XFST/HFST formatto_dictdictConvert to adictobjectto_jsonstrConvert to a JSON stringReadingobjectPropertyTypeDescriptionsubreadingsList[Subreading]Usually only one subreading, but multiple subreadings are possible for complexTokens.lemmasList[str]Lemmas from all subreadingsgrouped_tagsList[Tag]The part-of-speech, morphosyntactic, semantic and other tags from all subreadingsweightstrWeight indicating the likelihood of the reading, without respect to contextcg_rulestrReference to the rule in the constraint grammar that removed/selected/etc. this reading. If no action has been taken on this reading, then''.is_most_likelyboolIndicates whether this reading has been selected as the most likely reading of itsToken. Note that some selection methods may be at least partiallyrandom.MethodReturn typeDescriptioncg3_strstrAnalysis stream in theVISL-CG3 formathfst_strstrAnalysis stream in the XFST/HFST formatgeneratestrGenerate the wordform from this readingreplace_tagNoneReplace a tag in this readingdoes_not_conflictboolDetermine whether reading from external tagset (e.g. Universal Dependencies) conflicts with this readingto_dictlistConvert to alistobjectto_jsonstrConvert to a JSON stringSubreadingobjectPropertyTypeDescriptionlemmastrThe lemma of the subreadingtagsList[Tag]The part-of-speech, morphosyntactic, semantic and other tagstagsetSet[Tag]Same astags, but for faster membership testing (inReading)MethodReturn typeDescriptioncg3_strstrAnalysis stream in theVISL-CG3 formathfst_strstrAnalysis stream in the XFST/HFST formatreplace_tagNoneReplace a tag in this readingto_dictdictConvert to adictobjectto_jsonstrConvert to a JSON stringTagobjectPropertyTypeDescriptionnamestrThe name of this tagms_featstrMorphosyntactic feature that this tag is associated with (e.g.Dathas ms_featCASE)detailstrDescription of the tag's purpose or meaningis_L2_errorboolWhether this tag indicates a second-language learner errorMethodReturn typeDescriptioninfostrAlias forTag.detailConvenience functionsA number of functions are included, both for convenience, and to give concrete
examples for using the API.noun_distractors()This function generates all six cases of a given noun. If the given noun is
singular, then the function generates singular forms. If the given noun is
plural, then the function generates plural forms. Such a list can be used in a
multiple-choice exercise, hence the namedistractors.sg_paradigm=udar.noun_distractors('словом')print(sg_paradigm=={'сло́ву','сло́ве','сло́вом','сло́ва','сло́во'})# Truepl_paradigm=udar.noun_distractors('словах')print(pl_paradigm=={'слова́м','слова́','слова́х','слова́ми','сло́в'})# TrueIf unstressed forms are desired, simply pass the argumentstressed=False.diagnose_L2()This function will take a text string as the argument, and will return a
dictionary of all the types of L2 errors in the text, along with examples of
the error.diag=udar.diagnose_L2('Етот малчик говорит по-русски.')print(diag=={'Err/L2_e2je':{'Етот'},'Err/L2_NoSS':{'малчик'}})# Truetag_info()This function will look up the meaning of any tag used by the analyzer.print(udar.tag_info('Err/L2_ii'))# L2 error: Failure to change ending ие to ии in +Sg+Loc or +Sg+Dat, e.g. к Марие, о кафетерие, о знаниеUsing the transducers manuallyThe transducers come in two varieties: theAnalyzerclass and theGeneratorclass. For memory efficiency, I recommend using theget_analyzerandget_generatorfunctions, which ensure that each flavor of the transducers
remains a singleton in memory.AnalyzerTheAnalyzercan be initialized with or without analyses for second-language
learner errors using the keywordL2_errors.analyzer=udar.get_analyzer()# by default, L2_errors is FalseL2_analyzer=udar.get_analyzer(L2_errors=True)Analyzers are callable. They take a tokenstrand return a sequence of
reading/weighttuples.raw_readings1=analyzer('сло́ва')print(raw_readings1)# (('слово+N+Neu+Inan+Sg+Gen', 5.9755859375),)raw_readings2=analyzer('слова')print(raw_readings2)# (('слово+N+Neu+Inan+Pl+Acc', 5.9755859375), ('слово+N+Neu+Inan+Pl+Nom', 5.9755859375), ('слово+N+Neu+Inan+Sg+Gen', 5.9755859375))GeneratorTheGeneratorcan be initialized in three varieties: unstressed, stressed,
and phonetic.generator=udar.get_generator()# unstressed by defaultstressed_generator=udar.get_generator(stressed=True)phonetic_generator=udar.get_generator(phonetic=True)Generators are callable. They take aReadingor raw readingstrand
return a surface form.print(stressed_generator('слово+N+Neu+Inan+Pl+Nom'))# слова́Working withTokens andReadingssYou can easily check if a morphosyntactic tag is in aToken,Reading,
orSubreadingusingin:token2=udar.Token('слова',analyze=True)print(token2)# слова [слово_N_Neu_Inan_Pl_Acc  слово_N_Neu_Inan_Pl_Nom  слово_N_Neu_Inan_Sg_Gen]print('Gen'intoken2)# do any of the readings include Genitive case?# Trueprint('слово'intoken2)# does not work for lemmas; use `in Token.lemmas`# Falseprint('слово'intoken2.lemmas)# TrueYou can make a filtered list of aToken's readings using the following idiom:pl_readings=[readingforreadingintoken2if'Pl'inreading]print(pl_readings)# [Reading(слово+N+Neu+Inan+Pl+Acc, 5.975586, ), Reading(слово+N+Neu+Inan+Pl+Nom, 5.975586, )]Related projectshttps://github.com/mikahama/uralicNLP