Package: tensorflow-onmttok-ops
OpenNMT Tokenizer TensorFlow OpsDISCLAIMER: This package is not published by the OpenNMT authors.Full credits forOpenNMT TokenizerandOpenNMT-tfgoes to their respectively
authors.This project aims to wrapOpenNMT Tokenizerinto TensorFlow Ops.It's primarily intended to be used as an addition to theOpenNMT-tfframework,
in order to remove the need of applying tokenization and/or
detokenization outside of a serving environment (e.g. TensorFlow Serving).CompatibilityTensorFlow2.1,2.2OpenNMT-tf >=2.6.0for usage in conjunction with OpenNMT-tfInstallationPrerequisites :A Linux environment (manylinux2014eligible)Python3.5,3.6,3.7or3.8Install the package with pip :pipinstalltensorflow-onmttok-opsUsageAvailable Tokenizer optionsThe majority of the OpenNMT Tokenizeroptionsare available.However, providingBPEorSentencePiecemodels is not supported,
and by extension, setting the tokenizermodetononeis not supported.You thereforecannotuse the following options :bpe_model_pathsp_model_pathsp_nbest_sizesp_alphavocabulary_pathvocabulary_thresholdNote:Tokenizer options are defined at graph construction time
and are constants.Tokenizationimporttensorflow_onmttokastf_onmttoktokens=tf_onmttok.tokenize(["Hello, how are you?"],mode="conservative")Detokenizationimporttensorflow_onmttokastf_onmttoktext=tf_onmttok.detokenize(["How","are","you","?"],mode="space")With OpenNMT-tfUsage with OpenNMT-tf is pretty straightforward.This package comes with a built-in tokenizer
in order to make usage of the ops.Before training your model, register the tokenizer as follows :fromtensorflow_onmttokimportregister_opennmt_in_graph_tokenizerregister_opennmt_in_graph_tokenizer()See thecomplete exampleNow that the tokenizer is registered, you can use theOpenNMTInGraphTokenizerclass instead ofOpenNMTTokenizerin your
tokenization configuration files, e.g. :type:OpenNMTInGraphTokenizerparams:mode:conservativecase_feature:trueThat's it ! You can now train your model as usual.
YourExportedModelwill now expect atextinput instead oftokensandlength.Note: Tokenization resources will not be exported
to theassets.extradirectory.Build TF Serving with this OpsThis guide will show you how to build TensorFlow Serving
with this ops.PrerequisitesYou have already cloned the
TF Serving>= 2.1.0repository,
and have all tools installed for building itYou have installed CMake3.1.0or newerBuildingAdd the Ops sourcesFirst, download thereleaseof your choice.Inside the TF Serving sources folder, create a directory
namedcustom_opsand copy the content of thetensorflow_onmttokdirectory into it.$cd<tf_serving_sources>
$mkdirtensorflow_serving/custom_ops
$cp-r<op_sources>/tensorflow_onmttoktensorflow_serving/custom_opsReference the OpsEdittensorflow_serving/model_servers/BUILDto reference
the Ops build target :SUPPORTED_TENSORFLOW_OPS=[..."//tensorflow_serving/custom_ops/tensorflow_onmttok:onmttok_ops"]Build OpenNMT Tokenizer from sourcesThe last step is to build a static version of the
OpenNMT Tokenizer library.This repository provides a shell script
that will build it with CMake.$cd<op_sources>
$chmod+xbuild_tokenizer.sh&&./build_tokenizer.shNote: Passsudoargument to thebuild_tokenizer.shscript
to execute themake installcommand with sudo.Build TensorFlow ServingYou can now build TensorFlow Serving as usual.