Package: pystempel
Python port of Stempel, an algorithmic stemmer for Polish language, originally written in Java.The original stemmer has been implemented as part ofEgothor Project, taken virtually unchanged toStempel Stemmer Java libraryby Andrzej Białecki and next included as part ofApache Lucene,
a free and open-source search engine library. It is also used byElastic Searchsearch engine.This package includes also high-quality stemming tables for Polish: original one pretrained by
Andrzej Białecki on 20,000 training sets, and new one, pretrained on 259,080 training sets
from Polimorf dictionary by me.The port does not include code for compiling stemming tables.How to useInstall in your local environment:pip install pystempelUse in your code:>>>fromstempelimportStempelStemmerChoose original (called default) version of a stemmer:>>>stemmer=StempelStemmer.default()or a version with new stemming table pretrained on training sets from Polimorf dictionary:>>>stemmer=StempelStemmer.polimorf()Stem:>>>forwordin['książka','książki','książkami','książkowa','książkowymi']:...print(stemmer.stem(word))...książekksiążekksiążekksiążkowyksiążkowyChoosing stemming tablePerformance between original (default) and new stemming table (Polimorf-based) varies significantly.
The stemmer for the default stemming table isunderstemming, i.e., for multiple forms of the
same lemma provides different stems more often (63%) than when using Polimorf-based stemming table
(13%). However, the file footprint of the latter is bigger (2.2MB vs 0.3MB). Also loading takes
longer (7.5 seconds vs. 1.3 seconds), though this happens only once, when a stemmer is created. Also,
for original stemming table, the stemmer stems slightly faster: ~60000 vs ~51000 words per second.
SeeEvaluation Jupyter Notebookfor the detailed evaluation results.Note also, that the licensing schema of both stemming tables differs, and hence licensing of
data generated with each one. See “Licensing” section for the details.Choosing between port and wrapperIf you work on an NLP project in Python you can choose between Python port and Python wrapper.
Python port is what pystempel tries to achieve: translation from Java implementation to Python.
Python wrapper is what I used intests: Python functions to call the original Java implementation of
stemmer. You can find more about wrappers and ports inStackoverflow comparision post. Here, I
compare both approaches to help you decide:Same accuracy. I have verified Python port by comparing its output
with output of original Java implementation for 331224 words from Free Polish dictionary
(sjp.pl) and for 100% of words it returns same output.Similar performance. For mentioned dataset both stemmer versions achieved comparable performance.
Python port completed stemming in 4.4 seconds, while Python wrapper – in 5 seconds (Intel Core
i5-6000 3.30 GHz, 16GB RAM, Windows 10, OpenJDK)Different setup. Python wrapper requires additionally installation of Cython and pyjnius.
Python wrapper will make alsodebugging harder(switching between two programming languages).OptionsTo disable a progress bar when loading stemming tables, set environment variableDISABLE_TQDM=True.Development setupTo setup environment for development you will needAnacondainstalled.conda env create --file environment.yml
conda activate pystempel-env
pre-commit installTo run tests:curl https://repo1.maven.org/maven2/org/apache/lucene/lucene-analyzers-stempel/8.1.1/lucene-analyzers-stempel-8.1.1.jar > stempel-8.1.1.jar
python -m pytest ./To run benchmark:set PYTHONPATH=%PYTHONPATH%;%cd%
python tests\test_benchmark.pyLicensingCode: Most of the code is covered byEgothorOpen Source License, an Apache-style license.
The rest of the code is covered by theApache License 2.0. This should be clear from a preamble
of each file.Data:The original pretrained stemming table is covered byApache License 2.0.The new pretrained stemming table is covered by2-Clause BSD License, similarly to thePolimorf dictionaryit has been derived from. The copyright owner of both the stemming table
and the dictionary isInstitute of Computer Science at Polish Academy of Science(IPI PAN).Polish dictionary used by the unit tests comes fromsjp.pland is covered byApache License 2.0as well.AlternativesEstemis Erlang wrapper (not port) for Stempel stemmer.pl_stemmeris a Python stemmer based on Porter’s Algorithm.polish-stemis a Python stemmer using Finite State Transducers.