Package: calamanCy
calamanCy: NLP pipelines for TagalogcalamanCyis a Tagalog natural language preprocessing framework made withspaCy. Its goal is to provide pipelines and datasets for
downstream NLP tasks. This repository contains material for using calamanCy,
reproduction of results, and guides on usage.calamanCy takes inspiration from other language-specificspaCy Universe frameworkssuch asDaCy,huSpaCy,
andgraCy. The name is based fromcalamansi,
a citrus fruit native to the Philippines and used in traditional Filipino cuisine.🔧 InstallationTo get started with calamanCy, simply install it usingpipby running the
following line in your terminal:pipinstallcalamanCyDevelopmentIf you are developing calamanCy, first clone the repository:gitclonegit@github.com:ljvmiranda921/calamanCy.gitThen, create a virtual environment and install the dependencies:python-mvenvvenv
venv/bin/pipinstall-e.# requires pip>=23.0venv/bin/pipinstall.[dev]# Activate the virtual environmentsourcevenv/bin/activateor alternatively, usemake dev.Running the testsWe usepytestas our test runner:python-mpytest--pyargscalamancy👩‍💻 UsageTo use calamanCy you first have to download either the medium, large, or
transformer model. To see a list of all available models, run:importcalamancyfrommodelincalamancy.models():print(model)# ..# tl_calamancy_md-0.1.0# tl_calamancy_lg-0.1.0# tl_calamancy_trf-0.1.0To download and load a model, run:nlp=calamancy.load("tl_calamancy_md-0.1.0")doc=nlp("Ako si Juan de la Cruz")Thenlpobject is an instance of spaCy'sLanguageclassand you can use it as any other spaCy
pipeline. You can alsoaccess these models on Hugging Face🤗.📦 Models and DatasetscalamanCy provides Tagalog models and datasets that you can use in your spaCy
pipelines. You can download them directly or use thecalamancyPython library
to access them. The training procedure for each pipeline can be found in themodels/directory. They are further subdivided into versions. Each folder is
an instance of aspaCy project.Here are the models for the latest release:ModelPipelinesDescriptiontl_calamancy_md (73.7 MB)tok2vec, tagger, morphologizer, parser, nerCPU-optimized Tagalog NLP model. Pretrained using the TLUnified dataset. Using floret vectors (50k keys)tl_calamancy_lg (431.9 MB)tok2vec, tagger, morphologizer, parser, nerCPU-optimized large Tagalog NLP model. Pretrained using the TLUnified dataset. Using fastText vectors (714k)tl_calamancy_trf (775.6 MB)transformer, tagger, parser, nerGPU-optimized transformer Tagalog NLP model. Uses roberta-tagalog-base as context vectors.📓 APIThe calamanCy library contains utility functions that help you load its models
and infer on your text.  You can think of these functions as "syntactic sugar"
to the spaCy API. We highly recommend checking out thespaCy Doc
object, as it provides the most flexibility.LoadersThe loader functions provide an easier interface to download calamanCy models.
These models are hosted onHuggingFaceso you can try them out first before downloading.functionget_latest_versionReturn the latest version of a calamanCy model.ArgumentTypeDescriptionmodelstrThe string indicating the model.RETURNSstrThe latest version of the model.functionmodelsGet a list of valid calamanCy models.ArgumentTypeDescriptionRETURNSList[str]List of valid calamanCy modelsfunctionloadLoad a calamanCy model as aspaCy language pipeline.ArgumentTypeDescriptionmodelstrThe model to download. See the available models atcalamancy.models().forceboolForce download the model. Defaults toFalse.**kwargsdictAdditional arguments tospacy.load().RETURNSLanguageA spaCy language pipeline.InferenceBelow are lightweight utility classes for users who are not familiar with spaCy's
primitives. They are only useful for inference and not for training. If you wish
to train on top of these calamanCy models (e.g., text categorization,
task-specific NER, etc.), we advise you to follow the standardspaCy training
workflow.General usage: first, you need to instantiate a class with the name of a model.
Then, you can use the__call__method to perform the prediction. The output
is of the typeIterable[Tuple[str, Any]]where the first part of the tuple
is the token and the second part is its label.methodEntityRecognizer.__call__Perform named entity recognition (NER).  By default, it uses the v0.1.0 ofTLUnified-NERwith the following entity labels:PER (Person), ORG (Organization), LOC
(Location).ArgumentTypeDescriptiontextstrThe text to get the entities from.YIELDSIterable[Tuple[str, str]]the token and its entity in IOB format.methodTagger.__call__Perform parts-of-speech tagging. It uses the annotations from theTRGandUgnayantreebanks with the following tags:ADJ, ADP, ADV, AUX, DET, INTJ, NOUN, PART,
PRON, PROPN, PUNCT, SCONJ, VERB.ArgumentTypeDescriptiontextstrThe text to get the POS tags from.YIELDSIterable[Tuple[str, Tuple[str, str]]]the token and its coarse- and fine-grained POS tag.methodParser.__call__Perform syntactic dependency parsing. It uses the annotations from theTRGandUgnayantreebanks.ArgumentTypeDescriptiontextstrThe text to get the dependency relations from.YIELDSIterable[Tuple[str, str]]the token and its dependency relation.📝 Reporting IssuesIf you have questions regarding the usage ofcalamanCy, bug reports, or just
want to give us feedback after giving it a spin, please use theIssue
tracker. Thank you!📜 CitationIf you are citing the open-source software, please use:@misc{miranda2023calamancy,
    title={{calamanCy: A Tagalog Natural Language Processing Toolkit}}, 
    author={Lester James V. Miranda},
    year={2023},
    eprint={2311.07171},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}If you are citing theNER dataset, please use:@misc{miranda2023developing,
    title={{Developing a Named Entity Recognition Dataset for Tagalog}}, 
    author={Lester James V. Miranda},
    year={2023},
    eprint={2311.07161},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}