Package: simplemma
PurposeLemmatizationis the process of grouping together the inflected forms of a word so they can be analysed as a single item, identified by the word’s lemma, or dictionary form. Unlike stemming, lemmatization outputs word units that are still valid linguistic forms.In modern natural language processing (NLP), this task is often indirectly tackled by more complex systems encompassing a whole processing pipeline. However, it appears that there is no straightforward way to address lemmatization in Python although this task can be crucial in fields such as information retrieval and NLP.Simplemmaprovides a simple and multilingual approach to look for base forms or lemmata. It may not be as powerful as full-fledged solutions but it is generic, easy to install and straightforward to use. In particular, it does not need morphosyntactic information and can process a raw series of tokens or even a text with its built-in tokenizer. By design it should be reasonably fast and work in a large majority of cases, without being perfect.With its comparatively small footprint it is especially useful when speed and simplicity matter, in low-resource contexts, for educational purposes, or as a baseline system for lemmatization and morphological analysis.Currently, 49 languages are partly or fully supported (see table below).InstallationThe current library is written in pure Python with no dependencies:pip install simplemmapip3where applicablepip install-Usimplemmafor updatesUsageWord-by-wordSimplemma is used by selecting a language of interest and then applying the data on a list of words.>>>importsimplemma# get a wordmyword='masks'# decide which language to use and apply it on a word form>>>simplemma.lemmatize(myword,lang='en')'mask'# grab a list of tokens>>>mytokens=['Hier','sind','Vaccines']>>>fortokeninmytokens:>>>simplemma.lemmatize(token,lang='de')'hier''sein''Vaccines'# list comprehensions can be faster>>>[simplemma.lemmatize(t,lang='de')fortinmytokens]['hier','sein','Vaccines']Chaining several languages can improve coverage, they are used in sequence:>>>fromsimplemmaimportlemmatize>>>lemmatize('Vaccines',lang=('de','en'))'vaccine'>>>lemmatize('spaghettis',lang='it')'spaghettis'>>>lemmatize('spaghettis',lang=('it','fr'))'spaghetti'>>>lemmatize('spaghetti',lang=('it','fr'))'spaghetto'For certain languages a greedier decomposition is activated by default as it can be beneficial, mostly due to a certain capacity to address affixes in an unsupervised way. This can be triggered manually by setting thegreedyparameter toTrue.This option also triggers a stronger reduction through a further iteration of the search algorithm, e.g. “angekündigten” → “angekündigt” (standard) → “ankündigen” (greedy). In some cases it may be closer to stemming than to lemmatization.# same example as before, comes to this result in one step>>>simplemma.lemmatize('spaghettis',lang=('it','fr'),greedy=True)'spaghetto'# German case described above>>>simplemma.lemmatize('angekündigten',lang='de',greedy=True)'ankündigen'# 2 steps: reduction to infinitive verb>>>simplemma.lemmatize('angekündigten',lang='de',greedy=False)'angekündigt'# 1 step: reduction to past participleThe additional functionis_known()checks if a given word is present in the language data:>>>fromsimplemmaimportis_known>>>is_known('spaghetti',lang='it')TrueTokenizationA simple tokenization function is included for convenience:>>>fromsimplemmaimportsimple_tokenizer>>>simple_tokenizer('Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.')['Lorem','ipsum','dolor','sit','amet',',','consectetur','adipiscing','elit',',','sed','do','eiusmod','tempor','incididunt','ut','labore','et','dolore','magna','aliqua','.']# use iterator instead>>>simple_tokenizer('Lorem ipsum dolor sit amet',iterate=True)The functionstext_lemmatizer()andlemma_iterator()chain tokenization and lemmatization. They can takegreedy(affecting lemmatization) andsilent(affecting errors and logging) as arguments:>>>fromsimplemmaimporttext_lemmatizer>>>sentence='Sou o intervalo entre o que desejo ser e os outros me fizeram.'>>>text_lemmatizer(sentence,lang='pt')# caveat: desejo is also a noun, should be desejar here['ser','o','intervalo','entre','o','que','desejo','ser','e','o','outro','me','fazer','.']# same principle, returns a generator and not a list>>>fromsimplemmaimportlemma_iterator>>>lemma_iterator(sentence,lang='pt')Caveats# don't expect too much though# this diminutive form isn't in the model data>>>simplemma.lemmatize('spaghettini',lang='it')'spaghettini'# should read 'spaghettino'# the algorithm cannot choose between valid alternatives yet>>>simplemma.lemmatize('son',lang='es')'son'# valid common name, but what about the verb form?As the focus lies on overall coverage, some short frequent words (typically: pronouns and conjunctions) may need post-processing, this generally concerns a few dozens of tokens per language.The current absence of morphosyntactic information is both an advantage in terms of simplicity and an impassable frontier regarding lemmatization accuracy, e.g. disambiguation between past participles and adjectives derived from verbs in Germanic and Romance languages. In most cases,simplemmaoften does not change such input words.The greedy algorithm seldom produces invalid forms. It is designed to work best in the low-frequency range, notably for compound words and neologisms. Aggressive decomposition is only useful as a general approach in the case of morphologically-rich languages, where it can also act as a linguistically motivated stemmer.Bug reports over theissues pageare welcome.Language detectionLanguage detection works by providing a text and tuplelangconsisting of a series of languages of interest. Scores between 0 and 1 are returned.Thelang_detector()function returns a list of language codes along with scores and adds “unk” for unknown or out-of-vocabulary words. The latter can also be calculated by using the functionin_target_language()which returns a ratio.# import necessary functions>>>fromsimplemma.langdetectimportin_target_language,lang_detector# language detection>>>lang_detector('"Moderní studie narazily na několik tajemství." Extracted from Wikipedia.',lang=("cs","sk"))[('cs',0.625),('unk',0.375),('sk',0.125)]# proportion of known words>>>in_target_language("opera post physica posita (τὰ μετὰ τὰ φυσικά)",lang="la")0.5Supported languagesThe following languages are available using theirBCP 47 language tag, which is usually theISO 639-1 codebut if no such code exists, aISO 639-3 codeis used instead:Available languages (2022-01-20)CodeLanguageForms (10³)Acc.CommentsastAsturian124bgBulgarian204caCatalan579csCzech1870.89on UD CS-PDTcyWelsh360daDanish5540.92on UD DA-DDT, alternative:lemmydeGerman6750.95on UD DE-GSD, see alsoGerman-NLP listelGreek1810.88on UD EL-GDTenEnglish1310.94on UD EN-GUM, alternative:LemmInflectenmMiddle English38esSpanish6650.95on UD ES-GSDetEstonian119low coveragefaPersian12experimentalfiFinnish3,199seethis benchmarkfrFrench2170.94on UD FR-GSDgaIrish372gdGaelic48glGalician384gvManx62hbsSerbo-Croatian656Croatian and Serbian lists to be added laterhiHindi58experimentalhuHungarian458hyArmenian246idIndonesian170.91on UD ID-CSUIisIcelandic174itItalian3330.93on UD IT-ISDTkaGeorgian65laLatin843lbLuxembourgish305ltLithuanian247lvLatvian164mkMacedonian56msMalay14nbNorwegian (Bokmål)617nlDutch2500.92on UD-NL-AlpinonnNorwegian (Nynorsk)56plPolish3,2110.91on UD-PL-PDBptPortuguese9240.92on UD-PT-GSDroRomanian311ruRussian595alternative:pymorphy2seNorthern Sámi113skSlovak8180.92on UD SK-SNKslSlovene136sqAlbanian35svSwedish658alternative:lemmyswSwahili10experimentaltlTagalog32experimentaltrTurkish1,2320.89on UD-TR-BounukUkrainian370alternative:pymorphy2Low coveragementions means one would probably be better off with a language-specific library, butsimplemmawill work to a limited extent. Open-source alternatives for Python are referenced if possible.Experimentalmentions indicate that the language remains untested or that there could be issues with the underlying data or lemmatization process.The scores are calculated onUniversal Dependenciestreebanks on single word tokens (including some contractions but not merged prepositions), they describe to what extent simplemma can accurately map tokens to their lemma form. They can be reproduced by concatenating all available UD files and by using the scriptudscore.pyin thetests/folder.This library is particularly relevant as regards the lemmatization of less frequent words. Its performance in this case is only incidentally captured by the benchmark above. In some languages, a fixed number of words such as pronouns can be further mapped by hand to enhance performance.SpeedOrders of magnitude given for reference only, measured on an old laptop to give a lower bound:Tokenization: > 1 million tokens/secLemmatization: > 250,000 words/secInstalling the most recent Python version can improve speed.Optional pre-compilation withmypycpip3 install mypyclone or download the source code from the repositorypython3 setup.py--use-mypycbdist_wheelpip3 installdist/*.whl(where*is the compiled wheel)Roadmap[-] Add further lemmatization lists[ ] Grammatical categories as option[ ] Function as a meta-package?[ ] Integrate optional, more complex models?Credits and licensesSoftware under MIT license, for the linguistic information databases seelicensesfolder.The surface lookups (non-greedy mode) use lemmatization lists derived from various sources, ordered by relative importance:Lemmatization listsby Michal Měchura (Open Database License)Wiktionary entries packaged by theKaikki projectFreeLing projectspaCy lookups dataUnimorph ProjectWikinflection corpusby Eleni Metheniti (CC BY 4.0 License)ContributionsSee thislist of contributorsto the repository.Feel free to contribute, notably byfiling issuesfor feedback, bug reports, or links to further lemmatization lists, rules and tests.Contributions by pull requests ought to follow the following conventions: code style withblack, type hinting withmypy, included tests withpytest.Other solutionsSee lists:German-NLPandother awesome-NLP lists.For a more complex and universal approach in Python seeuniversal-lemmatizer.ReferencesTo cite this software:Barbaresi A. (year). Simplemma: a simple multilingual lemmatizer for Python [Computer software] (Versionversion number). Berlin, Germany: Berlin-Brandenburg Academy of Sciences. Available fromhttps://github.com/adbar/simplemmaDOI: 10.5281/zenodo.4673264This work draws from lexical analysis algorithms used in:Barbaresi, A., & Hein, K. (2017).Data-driven identification of German phrasal compounds. In International Conference on Text, Speech, and Dialogue Springer, pp. 192-200.Barbaresi, A. (2016).An unsupervised morphological criterion for discriminating similar languages. In 3rd Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2016), Association for Computational Linguistics, pp. 212-220.Barbaresi, A. (2016).Bootstrapped OCR error detection for a less-resourced language variant. In 13th Conference on Natural Language Processing (KONVENS 2016), pp. 21-26.