Package: pie-extended
Pie ExtendedWarning: This software is only compatible with up to Python 3.7 for the moment.Extension forpieto include taggers with their models and pre/postprocessors.Pie is a wonderful tool to train models. And most of the time, it will be enough. Whatpie_extendedis proposing here
is to provide you with the necessary tools to share your models with customized pre- and post-processing.The current system provide an easier access to addingcustomized:normalization of your text,sentence tokenization,word tokenization,disambiguation,output formattingCite as@software{thibault_clerice_2020_3883590,
  author       = {Clérice, Thibault},
  title        = {Pie Extended, an extension for Pie with pre-processing and post-processing},
  month        = jun,
  year         = 2020,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.3883589},
  url          = {https://doi.org/10.5281/zenodo.3883589}
}Current supported languagesClassical Latin (Model:lasla)Ancient Greek (Model:grc)Old French (Model:fro)Early Modern French (Model:freem)Classical French (Model:fr)Old Dutch (Model:dum)If you trained models and want some help sharing them with Pie Extended, open an issue :)InstallTo install, simply dopip install pie-extended. Then, look at all available models.WARNING: if you don't have a GPU or CUDAPlease, in case of doubt, runpip install pie-extended  --extra-index-url https://download.pytorch.org/whl/cpuRun on terminalBut on top of that, it provides a quick and easy way to use others models ! For example, in a shell :pie-extendeddownloadlasla
pie-extendedinstall-addonslasla
pie-extendedtaglaslayour_file.txtwill give you access to all you need !Python APIYou can run the lemmatizer in your own scripts and retrieve token annotations as dictionaries:fromtypingimportListfrompie_extended.cli.utilsimportget_tagger,get_model,download# In case you need to downloaddo_download=Falseifdo_download:fordlindownload("lasla"):x=1# model_path allows you to override the model loaded by another .tarmodel_name="lasla"tagger=get_tagger(model_name,batch_size=256,device="cpu",model_path=None)sentences:List[str]=["Lorem ipsum dolor sit amet, consectetur adipiscing elit. "]# Get the main object from the model (: data iterator + postprocesorfrompie_extended.models.lasla.importsimportget_iterator_and_processorforsentence_groupinsentences:iterator,processor=get_iterator_and_processor()print(tagger.tag_str(sentence_group,iterator=iterator,processor=processor))will result in[{'form':'lorem','lemma':'lor','POS':'NOMcom','morph':'Case=Acc|Numb=Sing','treated':'lorem'},{'form':'ipsum','lemma':'ipse','POS':'PROdem','morph':'Case=Acc|Numb=Sing','treated':'ipsum'},{'form':'dolor','lemma':'dolor','POS':'NOMcom','morph':'Case=Nom|Numb=Sing','treated':'dolor'},{'form':'sit','lemma':'sum1','POS':'VER','morph':'Numb=Sing|Mood=Sub|Tense=Pres|Voice=Act|Person=3','treated':'sit'},{'form':'amet','lemma':'amo','POS':'VER','morph':'Numb=Sing|Mood=Sub|Tense=Pres|Voice=Act|Person=3','treated':'amet'},{'form':',','lemma':',','pos':'PUNC','morph':'MORPH=empty','treated':','},{'form':'consectetur','lemma':'consector2','POS':'VER','morph':'Numb=Sing|Mood=Sub|Tense=Pres|Voice=Dep|Person=3','treated':'consectetur'},{'form':'adipiscing','lemma':'adipiscor','POS':'VER','morph':'Tense=Pres|Voice=Dep','treated':'adipiscing'},{'form':'elit','lemma':'elio','POS':'VER','morph':'Numb=Sing|Mood=Ind|Tense=Pres|Voice=Act|Person=3','treated':'elit'},{'form':'.','lemma':'.','pos':'PUNC','morph':'MORPH=empty','treated':'.'}]Add a modelCreate a package in./pie_extended/models/. Exemple:foo.Add the name of the package in./pie_extended/models/__init__.pyin the variablemodules.In the modulepie_extended.models.foo, we should find the following variable:Models: a string with filenames and tasks for Pie.DESC: a METADATA object that bears information about the modelDOWNLOADS: A list of file to download.frompie_extended.utilsimportMetadata,File,get_pathDESC=Metadata("Foo""language",["Author 1","Author 2"],"A readable description","A link to more information")DOWNLOADS=[File("/a/link/to/a/file","local_name_of_the_file.tar")]Models="<{},task1,task2><{},lemma,pos>".format(get_path("foo","local_name_of_the_file.tar"))In the modulepie_extended.models.foo.imports, we should find the following content:get_iterator_and_processor: a function that returns aDataIteratorand aProcessor(optionally)addons: a function that installs add-ons(optionally)Disambiguator: a disambiguator instance (or an object creator that returns one)Check for a simple example inpie_extended.models.fro.importsand a more complex one
inpie_extended.models.lasla.importsInstall development version (⚠ for development only)Clone the repository, create an environment, and thenpythonsetup.pydevelopWarningThis is an extremely early build, subject to change here and there. But it is functional !