Package: huspacy
HuSpaCy is aspaCylibrary providing industrial-strength Hungarian language processing facilities through spaCy models.
The released pipelines consist of a tokenizer, sentence splitter, lemmatizer, tagger (predicting morphological features as well), dependency parser and a named entity recognition module.
Word and phrase embeddings are also available through spaCy's API.
All models have high throughput, decent memory usage and close to state-of-the-art accuracy.
A live demo is availablehere, model releases are published toHugging Face Hub.This repository contains material to build HuSpaCy and all of its models in a reproducible way.InstallationTo get started using the tool, first, we need to download one of the models. The easiest way to achieve this is to installhuspacy(fromPyPI) and then fetch a model through its API.pipinstallhuspacyimporthuspacy# Download the latest CPU optimized modelhuspacy.download()Install the models directlyYou can install the latest models directly from 🤗 Hugging Face Hub:CPU optimizedlarge model:pip install https://huggingface.co/huspacy/hu_core_news_lg/resolve/main/hu_core_news_lg-any-py3-none-any.whlGPU optimizedtransformers model:pip install https://huggingface.co/huspacy/hu_core_news_trf/resolve/main/hu_core_news_trf-any-py3-none-any.whlTo speed up inference on GPUs, CUDA should be installed as described inhttps://spacy.io/usage.Read more on the modelshereQuickstartHuSpaCy is fully compatible withspaCy's API, newcomers can easily get started withspaCy 101guide.Although HuSpacy models can be loaded withspacy.load(...), the tool provides convenience methods to easily access downloaded models.# Load the model using spacy.load(...)importspacynlp=spacy.load("hu_core_news_lg")# Load the default large model (if downloaded)importhuspacynlp=huspacy.load()# Load the model directly as a moduleimporthu_core_news_lgnlp=hu_core_news_lg.load()To process texts, you can simply call the loaded model (i.e. thenlpcallable object)doc=nlp("Csiribiri csiribiri zabszalma - négy csillag közt alszom ma.")As HuSpaCy is built on spaCy, the returneddocdocumentcontains all the annotations given by the pipeline components.API Documentation is available inour website.Models overviewWe provide several pretrained models:hu_core_news_lgis a CNN-based large model which achieves a good
balance between accuracy and processing speed. This default model provides tokenization, sentence splitting,
part-of-speech tagging (UD labels w/ detailed morphosyntactic features), lemmatization, dependency parsing and named
entity recognition and ships with pretrained word vectors.hu_core_news_trfis built
onhuBERTand provides the same functionality as the large model
except the word vectors. It comes with much higher accuracy in the price of increased computational resource usage.
We suggest using it with GPU support.hu_core_news_mdgreatly improves onhu_core_news_lg's
throughput by loosing some accuracy. This model could be a good choice when processing speed is crucial.hu_core_news_trf_xlis an experimental model built
onXLM-RoBERTa-large. It provides the same functionality as
thehu_core_news_trfmodel, however it comes with slightly higher accuracy in the price of significantly increased
computational resource usage.
We suggest using it with GPU support.HuSpaCy's model versions followsspaCy's versioning scheme.A demo of the models is available atHugging Face Spaces.To read more about the model's architecture we suggest
readingthe relevant sections from spaCy's documentation.ComparisonModelsmdlgtrftrf_xlEmbeddings100d floret300d florettransformer:huBERTtransformer:XLM-RoBERTa-largeTarget hardwareCPUCPUGPUGPUAccuracy⭑⭑⭑⭒⭑⭑⭑⭑⭑⭑⭑⭑⭒⭑⭑⭑⭑⭑Resource usage⭑⭑⭑⭑⭑⭑⭑⭑⭑⭑⭑⭒CitationIf you use HuSpaCy or any of its models, please cite it as:@InProceedings{HuSpaCy:2023,author={"Orosz, Gy{\"o}rgy and Szab{\'o}, Gerg{\H{o}} and Berkecz, P{\'e}ter and Sz{\'a}nt{\'o}, Zsolt and Farkas, Rich{\'a}rd"},editor={"Ek{\v{s}}tein, Kamil and P{\'a}rtl, Franti{\v{s}}ek and Konop{\'i}k, Miloslav"},title={{"Advancing Hungarian Text Processing with HuSpaCy: Efficient and Accurate NLP Pipelines"}},booktitle={{"Text, Speech, and Dialogue"}},year="2023",publisher={{"Springer Nature Switzerland"}},address={{"Cham"}},pages="58--69",isbn="978-3-031-40498-6"}@InProceedings{HuSpaCy:2021,title={{HuSpaCy: an industrial-strength Hungarian natural language processing toolkit}},booktitle={{XVIII. Magyar Sz{\'a}m{\'\i}t{\'o}g{\'e}pes Nyelv{\'e}szeti Konferencia}},author={Orosz, Gy{\"o}rgy and Sz{\' a}nt{\' o}, Zsolt and Berkecz, P{\' e}ter and Szab{\' o}, Gerg{\H o} and Farkas, Rich{\' a}rd},location={{Szeged}},pages="59--73",year={2022},}ContactFor feature requests, issues and bugs please use theGitHub Issue Tracker. Otherwise, reach out to us in theDiscussion Forum.AuthorsHuSpaCy is implemented in theSzegedAIteam, coordinated byOrosz Györgyin theHungarian AI National Laboratory, MILABprogram.LicenseThis library is released under theApache 2.0 LicenseTrained models have their own license (CC BY-SA 4.0) as described on themodels page.