# PYHFST: A Pure Python Implementation of HFST

Khalid Alnajjar, Mika Hämäläinen

► To cite this version:

Khalid Alnajjar, Mika Hämäläinen. PYHFST: A Pure Python Implementation of HFST. Lightning Proceedings of NLP4DH and IWCLUL 2023, Dec 2023, Tokyo, Japan. pp.32-35. hal-04429719

HAL Id: hal-04429719

https://hal.science/hal-04429719v1

Submitted on 31 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# PYHFST: A Pure Python Implementation of HFST

Khalid Alnajjar¹ and Mika Hämäläinen²

¹Rootroo Ltd, Finland

²Metropolia University of Applied Sciences, Finland

In this paper, we describe our work on reimplementing HFST optimized lookup on Python. Our tool is called PYHFST and it is available on GitHub (https://github.com/Rootroo-ltd/pyhfst), PyPi (https://pypi.org/project/pyhfst/) and Zenodo (https://zenodo.org/records/7791470).

HFST (Helsinki Finite State Transducer) (Lindén et al., 2011) is, without a doubt, one of the core technologies used in NLP for Uralic languages. It powers numerous morphological transducers for different Uralic languages (Pirinen, 2015; Trosterud et al., 2017; Kaalep et al., 2018; Rueter et al., 2021) and as such, it is an essential building block of GiellaLT (Moshagen et al., 2013), Ve'rdd (Alnajjar et al., 2020) and UralicNLP (Hämäläinen, 2019) among others.

<figure><img src="image_2.png" /></figure>
The problem, however, is that as years go by, the Python implementation of HFST has fallen behind the development. The Python package of HFST has become divided into two packages: hfst and hfst-dev. At the time of writing, the latest version of hfst was released in 2018 and it completely lacks support for Windows 64 and ARM-based Macs. Furthermore, it has precompiled binaries for several Python versions, out of which, 3.7 is the latest supported Python. There is a possibility of compiling the Python package from source code, but in practice this is not possible on latest Macs anymore without updating the source code.

There is a more recent version of hfst-dev but it mainly provides support for Linux. The most recent Python version that is supported is 3.10, but only on Linux. There is no support for any Windows version or any ARM-based systems, including latest Macs. There is no source code distribution available either, which means that the installation just fails on any unsupported platform.

Throughout history, HFST has always been a pain in the neck to install. For this reason, we undertook the challenge of porting HFST optimised lookup to pure Python. The main functionality we need is the ability to load and run HFST transducers. We took the Java implementation (Mäkelä et al., 2017) and made a quick and dirty conversion of the source code to Python using ChatGPT. The converted code did not work fully, so we spent a good amount of time debugging and optimising the code.

The biggest issue was that the pure Python version did not work fast enough. For this reason, we implemented our code in Cython as well. This makes it possible to pip install our pyhfst library on any system because it is pure Python. By passing an extra argument during the installation, the Cython version gets installed, which brings a massive speed boost to the library.
Pyhfst is still slower than hfst or hfst-dev. However, it is the easiest option for running HFST on Python on a modern operating system, a modern version of Python and an ARM chip. Phfst is also better than the other alternatives developed by different people in the sense that it is the only non-official Python implementation that supports both weighted and unweighted transducers.

## References

Alnajjar, K., Hämäläinen, M., Rueter, J., & Partanen, N. (2020). Ve’rdd. Narrowing the Gap between Paper Dictionaries, Low-Resource NLP and Community Involvement. In Proceedings of the 28th International Conference on Computational Linguistics: System Demonstrations (pp. 1-6).

Hämäläinen, M. (2019). UralicNLP: An NLP library for Uralic languages. Journal of open source software.

Kaalep, H. J., Moshagen, S. N., & Trosterud, T. (2018). Estonian Morphology in the Giella Infrastructure. In Baltic HLT (pp. 47-54).

Lindén, K., Axelson, E., Hardwick, S., Pirinen, T. A., & Silfverberg, M. (2011). Hfst—framework for compiling and applying morphologies. In Systems and Frameworks for Computational Morphology: Second International Workshop, SFCM 2011, Zurich, Switzerland, August 26, 2011. Proceedings 2 (pp. 67-85). Springer Berlin Heidelberg.
Moshagen, S., Pirinen, T. A., & Trosterud, T. (2013).
Building an open-source development infrastructure for
language technology projects. In Proceedings of the 19th
Nordic Conference of Computational Linguistics
(NODALIDA 2013) (pp. 343-352).

Mäkelä, E., Asplund, M., & Lahti, P.S. (2016). seco-hfst: Thread safe Java version of Helsinki Finite State Transducer Lookup (v1.1.5). Zenodo. https://doi.org/10.5281/zenodo.56575

Pirinen, T. A. (2015). Omorfi–free and open source
morphological lexical database for Finnish. In Proceedings
of the 20th Nordic conference of computational linguistics
(NODALIDA 2015) (pp. 313-315).

Rueter, J., Partanen, N., Hämäläinen, M., & Trosterud, T. (2021). Overview of open-source morphology development for the Komi-Zyrian language: Past and future. In Proceedings of the Seventh International Workshop on Computational Linguistics of Uralic Languages. The Association for Computational Linguistics.

Trosterud, S. R., Trosterud, T., Räisänen, A. K., Niiranen, L., Haavisto, M., & Maliniemi, K. (2017). A morphological analyser for Kven. In Proceedings of the Third Workshop on Computational Linguistics for Uralic Languages (pp. 76-88).