Package: tftokenizers
TFtftransformersConverting Hugginface tokenizers to Tensorflow tokenizers. The main reason is to be able to bundle the tokenizer and model into one Reusable SavedModel, inspired by theTensorflow Official Guide on tokenizersSource Code:https://github.com/Hugging-Face-Supporter/tftokenizersModels we know works:"bert-base-cased""bert-base-uncased""bert-base-multilingual-cased""bert-base-multilingual-uncased"# Distilled"distilbert-base-cased""distilbert-base-multilingual-cased""microsoft/MiniLM-L12-H384-uncased"# Non-english"KB/bert-base-swedish-cased""bert-base-chinese"ExamplesThis is an example of how one can use Huggingface model and tokenizers bundled together as aReusable SavedModeland yields the same result as using the model and tokenizer from Huggingface 🤗importtensorflowastffromtransformersimportTFAutoModelfromtftokenizersimportTFModel,TFAutoTokenizer# Load base models from Huggingfacemodel_name="bert-base-cased"model=TFAutoModel.from_pretrained(model_name)# Load converted TF tokenizertokenizer=TFAutoTokenizer.from_pretrained(model_name)# Create a TF Reusable SavedModelcustom_model=TFModel(model=model,tokenizer=tokenizer)# Tokenizer and model can handle `tf.Tensors` or regular stringstf_string=tf.constant(["Hello from Tensorflow"])s1="SponGE bob SQuarePants is an avenger"s2="Huggingface to Tensorflow tokenizers"s3="Hello, world!"output=custom_model(tf_string)output=custom_model([s1,s2,s3])# We can now pass input as tensorsoutput=custom_model(inputs=tf.constant([s1,s2,s3],dtype=tf.string,name="inputs"),)# Save tokenizersaved_name="reusable_bert_tf"tf.saved_model.save(custom_model,saved_name)# Load tokenizerreloaded_model=tf.saved_model.load(saved_name)output=reloaded_model([s1,s2,s3])print(output)Setupgitclonehttps://github.com/Hugging-Face-Supporter/tftokenizers.gitcdtftokenizers
poetryinstall
poetryshellRunTo convert a Huggingface tokenizer to Tensorflow, first choose one from the models or tokenizers from the Huggingface hub to download.NOTECurrently only BERT models work with the converter.DownloadFirst download tokenizers from the hub by name. Either run the bash script do download multiple tokenizers or download a single tokenizer with the python script.The idea is to eventually only to automatically download and convertpythontftokenizers/download.py-nbert-base-uncased
bashscripts/download_tokenizers.shConvertConvert downloaded tokenizer from Huggingface format to Tensorflowpythontftokenizers/convert.pyBefore CommitmakebuildFAQHow to know what tokenizer is used?TL;DRfromtransformersimportAutoTokenizername="bert-base-cased"tokenizer=AutoTokenizer.from_pretrained(name)# IF the tokenizer is fast:print(tokenizer.is_fast)# Base tokenizer modelprint(type(tokenizer.backend_tokenizer.model))# Check if it is a SentencePiece tokenizer# Should be `vocab.txt` or `vocab.json` if not SentencePiece tokenizer# SencePiece if "vocab_file":#   "sentencepiece.bpe.model"print(tokenizer.vocab_files_names)# Else# Find if the model is a SentencePiece model withprint(vars(tokenizer).get("spm_file",None))# print(vars(tokenizer).get("sp_model", None)):memo: Read More:And the components of the tokenizers described [here](https://huggingface.co/docs/tokenizers/python/latest/components.html) as:
- Normalizers
- Pre tokenizers
- [Models](https://huggingface.co/docs/tokenizers/python/latest/components.html#models)
- PostProcessor
- DecodersWhen loading a tokenizer with Huggingface transformers, it maps the name of the model from the Huggingface Hub to the correct model and tokenizer available there, if not it will try to to find a folder on your local computer with that name.Additionally, tokenizers from Huggingface are defined in multiple different steps using the Huggingface tokenizer library. For those interested, you can look into the different components of that library of how the composition of a tokenizer workshere. There is also a great guide documenting how composition of tokenizers are done in thisMedium articleWhat tokenizers are used by what models?:memo: Read More:As stated in the section above, you will need to look at each model to inspect the type of tokenizer it is using, but in general there are just a few "base tokenizers / models". See [Huggingface documentation](https://huggingface.co/docs/transformers/tokenizer_summary) for explanation on how these "base tokenizers" are definedBase Tokenizer NamesModel ImplementationsSentencePiece tokenizers can either be BPE (rare if the tokenizers is fast) or Unigram (all Unigram == SentencePiece)BPE = tokenizers.models.BPEImplemented bybyte-pair BPE,char-level BPE, (SentencePiece BPE)Used byGPT,XLNet,FlauBERT,RoBERTa,GPT-2,GPT-j,GPT-neo,BART,XLM-RoBERTaUnigram = tokenizers.models.UnigramImplemented bySentencePiece UnicodeUsed byAllT5modelsWordPiece = tokenizers.models.WordPieceImplemented byBert WordPieceUsed byBERT,mBERT,miniLM, distilled versions of BERTSentencePieceSentencePiece is a method for creating sub-word tokenizations.
It supports BPE and Unigram.SentencePiece is a separate C++ implemented library with python and Tensorflow bindings.
The vocabulary is bundled into:For fast models:"vocab_file_names":`sentencepiece.bpe.model` for "BPE" and
`spiece.model` for UnigramFor slow models:"vocab_file_names":'source_spm': 'source.spm',
'target_spm': 'target.spm',
'vocab': 'vocab.json'"spm_files":will be a single file or a list of files
...Used by:Fast:T5modelsSlow:facebook/m2m100_418M,facebook/wmt19-en-deHow to implement the tokenizers from Huggingface to Tensorflow?You will need to download the Huggingface tokenizer of your choice, determine the type of the tokenizer (is_fast, tokenizer type andvocab_file_names). Then map the tokenizer used to the Tensorflow supported equivalent:https://github.com/tensorflow/text/issues/422BPEandUnigram:All BPE implementations for Tensorflow is backed by SentencePieceSentencePiece in TensorFlowOfficial Answer 1Official Answer 2How to load a SentencePiece modelInput will need to be TensorsHow to load model from vocabWordPiece:BertTokenizerorWordPieceorFastWordPiecehttps://github.com/tensorflow/text/issues/116https://github.com/tensorflow/text/issues/414What other ways are there to convert a tokenizer?:memo: Read More:With `tfokenizers` there are three ways to use the package:importtensorflowastfimporttensorflow_textastextfromtransformersimportAutoTokenizer,TFAutoModelfromtransformers.utils.loggingimportset_verbosity_errorfromtftokenizers.fileimport(get_filename_from_path,get_vocab_from_path,load_json)fromtftokenizers.modelimportTFModelfromtftokenizers.tokenizerimportTFAutoTokenizer,TFTokenizerBaseset_verbosity_error()tf.get_logger().setLevel("ERROR")pretrained_model_name="bert-base-cased"# a) by model_nametf_tokenizer=TFAutoTokenizer.from_pretrained(pretrained_model_name)# b) bundled with the model, similar to TFHubmodel=TFAutoModel.from_pretrained(pretrained_model_name)custom_model=TFModel(model=model,tokenizer=tf_tokenizer)# c) from source, using the saved files of a transformers tokenizer# Make sure you run download.py or the download script firstPATH="saved_tokenizers/bert-base-uncased"vocab=get_vocab_from_path(PATH)vocab_path=get_filename_from_path(PATH,"vocab")config=load_json(f"{PATH}/tokenizer_config.json")tokenizer_spec=load_json(f"{PATH}/tokenizer.json")special_tokens_map=load_json(f"{PATH}/special_tokens_map.json")tokenizer_base_params=dict(lower_case=True,token_out_type=tf.int64)tokenizer_base=text.BertTokenizer(vocab_path,**tokenizer_base_params)custom_tokenizer=TFTokenizerBase(vocab_path=vocab_path,tokenizer_base=tokenizer_base,hf_spec=tokenizer_spec,config=config,)How to save Huggingface Tokenizer files locally?:memo: Read More:To download the files used by Huggingface tokenizers, you can either download one by namepython tftokenizers/download.py -n KB/bert-base-swedish-casedor download multiplebash scrips/download_tokenizers.shWIPConvert a BERT tokenizer from Huggingface to TensorflowMake a TF Reusabel SavedModel with Tokenizer and Model in the same class. Emulate how the TF Hub example for BERT works.Find methods for identifying the base tokenizer model and map those settings and special tokens to new tokenizersExtend the tokenizers to more tokenizer types and identify them from a huggingface model nameDocument how others can use the library and document the different stages in the processImprove the conversion pipeline (s.a. Download and export files if not passed in or available locally)model_max_lengthshould be regulated. However, some newer models have the max_lenght for tokenizers at 1000_000_000Support more tokenizers, starting with SentencePieceIdentify tokenizer conversion limitationsSupport encoding of two sentences at a timeRefAllow the tokenizers to be used for Masking (MLM)Ref