Package: openvino-tokenizers
OpenVINO TokenizersOpenVINO Tokenizers adds text processing operations to OpenVINO.FeaturesPerform tokenization and detokenization without third-party dependenciesConvert a HuggingFace tokenizer into OpenVINO model tokenizer and detokenizerCombine OpenVINO models into a single modelAdd greedy decoding pipeline to text generation modelInstallation(Recommended) Create and activate virtual env:python3-mvenvvenvsourcevenv/bin/activate# orcondacreate--nameopenvino_tokenizercondaactivateopenvino_tokenizerMinimal InstallationUse minimal installation when you have a converted OpenVINO tokenizer:pipinstallopenvino-tokenizers# orcondainstall-cconda-forgeopenvinoopenvino-tokenizersConvert Tokenizers InstallationIf you want to convert HuggingFace tokenizers into OpenVINO tokenizers:pipinstallopenvino-tokenizers[transformers]# orcondainstall-cconda-forgeopenvinoopenvino-tokenizers&&pipinstalltransformers[sentencepiece]tiktokenBuild and install from source afterOpenVINO installationsourcepath/to/installed/openvino/setupvars.sh
gitclonehttps://github.com/openvinotoolkit/openvino_contrib.gitcdopenvino_contrib/modules/custom_operations/
pipinstall.[transformers]Build and install for developmentsourcepath/to/installed/openvino/setupvars.sh
gitclonehttps://github.com/openvinotoolkit/openvino_contrib.gitcdopenvino_contrib/modules/custom_operations/
pipinstall-e.[all]# verify installation by running testscduser_ie_extensions/tokenizer/python/tests/
pytest.C++ InstallationYou can use converted tokenizers in C++ pipelines with prebuild binaries.Download OpenVINO archive distribution for your OS fromhereand extract the archive.Download OpenVINO Tokenizers prebuild libraries fromhere. To ensure compatibility first three numbers of OpenVINO Tokenizers version should match OpenVINO version and OS.Extract OpenVINO Tokenizers archive into OpenVINO installation directory:Windows:<openvino_dir>\runtime\bin\intel64\Release\MacOS_x86:<openvino_dir>/runtime/lib/intel64/Release/MacOS_arm64:<openvino_dir>/runtime/lib/arm64/Release/Linux_x86:<openvino_dir>/runtime/lib/intel64/Linux_arm64:<openvino_dir>/runtime/lib/aarch64/After that you can add binary extension in the code with:core.add_extension("user_ov_extensions.dll")for Windowscore.add_extension("libuser_ov_extensions.dylib")for MacOScore.add_extension("libuser_ov_extensions.so")for Linuxandread/compileconverted (de)tokenizers models.Usage:warning: OpenVINO Tokenizers can be inferred on aCPUdevice only.Convert HuggingFace tokenizerOpenVINO Tokenizers ships with CLI tool that can convert tokenizers from Huggingface Hub
or Huggingface tokenizers saved on disk:convert_tokenizercodellama/CodeLlama-7b-hf--with-detokenizer-ooutput_dirThere is alsoconvert_tokenizerfunction that can convert tokenizer python object.importnumpyasnpfromtransformersimportAutoTokenizerfromopenvinoimportcompile_model,save_modelfromopenvino_tokenizersimportconvert_tokenizerhf_tokenizer=AutoTokenizer.from_pretrained("bert-base-uncased")ov_tokenizer=convert_tokenizer(hf_tokenizer)compiled_tokenzier=compile_model(ov_tokenizer)text_input=["Test string"]hf_output=hf_tokenizer(text_input,return_tensors="np")ov_output=compiled_tokenzier(text_input)foroutput_nameinhf_output:print(f"OpenVINO{output_name}={ov_output[output_name]}")print(f"HuggingFace{output_name}={hf_output[output_name]}")# OpenVINO input_ids = [[ 101 3231 5164  102]]# HuggingFace input_ids = [[ 101 3231 5164  102]]# OpenVINO token_type_ids = [[0 0 0 0]]# HuggingFace token_type_ids = [[0 0 0 0]]# OpenVINO attention_mask = [[1 1 1 1]]# HuggingFace attention_mask = [[1 1 1 1]]# save tokenizer for later usesave_model(ov_tokenizer,"openvino_tokenizer.xml")loaded_tokenizer=compile_model("openvino_tokenizer.xml")loaded_ov_output=loaded_tokenizer(text_input)foroutput_nameinhf_output:assertnp.all(loaded_ov_output[output_name]==ov_output[output_name])Connect Tokenizer to a ModelTo infer and convert the original model, install torch or torch-cpu to the virtual environment.fromtransformersimportAutoTokenizer,AutoModelForSequenceClassificationfromopenvinoimportcompile_model,convert_modelfromopenvino_tokenizersimportconvert_tokenizer,connect_modelscheckpoint="mrm8488/bert-tiny-finetuned-sms-spam-detection"hf_tokenizer=AutoTokenizer.from_pretrained(checkpoint)hf_model=AutoModelForSequenceClassification.from_pretrained(checkpoint)text_input=["Free money!!!"]hf_input=hf_tokenizer(text_input,return_tensors="pt")hf_output=hf_model(**hf_input)ov_tokenizer=convert_tokenizer(hf_tokenizer)ov_model=convert_model(hf_model,example_input=hf_input.data)combined_model=connect_models(ov_tokenizer,ov_model)compiled_combined_model=compile_model(combined_model)openvino_output=compiled_combined_model(text_input)print(f"OpenVINO logits:{openvino_output['logits']}")# OpenVINO logits: [[ 1.2007061 -1.4698029]]print(f"HuggingFace logits{hf_output.logits}")# HuggingFace logits tensor([[ 1.2007, -1.4698]], grad_fn=<AddmmBackward0>)Use Extension With Converted (De)Tokenizer or Model With (De)TokenizerImportopenvino_tokenizerswill add all tokenizer-related operations to OpenVINO,
after which you can work with saved tokenizers and detokenizers.importnumpyasnpimportopenvino_tokenizersfromopenvinoimportCorecore=Core()# detokenizer from codellama sentencepiece modelcompiled_detokenizer=core.compile_model("detokenizer.xml")token_ids=np.random.randint(100,1000,size=(3,5))openvino_output=compiled_detokenizer(token_ids)print(openvino_output["string_output"])# ['sc�ouition�', 'intvenord hasient', 'g shouldwer M more']Text generation pipelineimportnumpyasnpfromopenvinoimportcompile_model,convert_modelfromtransformersimportAutoModelForCausalLM,AutoTokenizerfromopenvino_tokenizersimportadd_greedy_decoding,convert_tokenizer# Use different repo for the tokenizer because the original repo doesn't have .model file# Sentencepiece(Unigram) tokenizer supported only with .model filetokenizer_checkpoint="microsoft/Llama2-7b-WhoIsHarryPotter"model_checkpoint="nickypro/tinyllama-15M"hf_tokenizer=AutoTokenizer.from_pretrained(tokenizer_checkpoint)hf_model=AutoModelForCausalLM.from_pretrained(model_checkpoint,use_cache=False)# convert hf tokenizertext_input=["Quick brown fox was"]ov_tokenizer,ov_detokenizer=convert_tokenizer(hf_tokenizer,with_detokenizer=True)compiled_tokenizer=compile_model(ov_tokenizer)# transform input text into tokensov_input=compiled_tokenizer(text_input)hf_input=hf_tokenizer(text_input,return_tensors="pt")# convert Pytorch model to OpenVINO IR and add greedy decoding pipeline to itov_model=convert_model(hf_model,example_input=hf_input.data)ov_model_with_greedy_decoding=add_greedy_decoding(ov_model)compiled_model=compile_model(ov_model_with_greedy_decoding)# generate new tokensnew_tokens_size=10prompt_size=ov_input["input_ids"].shape[-1]input_dict={output.any_name:np.hstack([tensor,np.zeros(shape=(1,new_tokens_size),dtype=np.int_)])foroutput,tensorinov_input.items()}foridxinrange(prompt_size,prompt_size+new_tokens_size):output=compiled_model(input_dict)["token_ids"]input_dict["input_ids"][:,idx]=output[:,idx-1]input_dict["attention_mask"][:,idx]=1ov_token_ids=input_dict["input_ids"]hf_token_ids=hf_model.generate(**hf_input,min_new_tokens=new_tokens_size,max_new_tokens=new_tokens_size,temperature=0,# greedy decoding)# decode model outputcompiled_detokenizer=compile_model(ov_detokenizer)ov_output=compiled_detokenizer(ov_token_ids)["string_output"]hf_output=hf_tokenizer.batch_decode(hf_token_ids,skip_special_tokens=True)print(f"OpenVINO output string: `{ov_output}`")# OpenVINO output string: `['Quick brown fox was walking through the forest. He was looking for something']`print(f"HuggingFace output string: `{hf_output}`")# HuggingFace output string: `['Quick brown fox was walking through the forest. He was looking for something']`Supported Tokenizer TypesHuggingfaceTokenizer TypeTokenizer Model TypeTokenizerDetokenizerFastWordPiece✅❌BPE✅✅Unigram❌❌LegacySentencePiece .model✅✅Customtiktoken✅✅Test ResultsThis report is autogenerated and includes tokenizers and detokenizers tests. TheOutput Matched, %column shows the percent of test strings for which the results of OpenVINO and Hugingface Tokenizers are the same. To update the report runpytest tokenizers_test.py --update_readmeinmodules/custom_operations/user_ie_extensions/tokenizer/python/testsdirectory.Output Match by Tokenizer TypeTokenizer TypeOutput Matched, %Number of TestsBPE95.823420SentencePiece86.282880Tiktoken97.69216WordPiece82.12520Output Match by ModelTokenizer TypeModelOutput Matched, %Number of TestsBPEEleutherAI/gpt-j-6b98.33180BPEEleutherAI/gpt-neo-125m98.33180BPEEleutherAI/gpt-neox-20b97.78180BPEEleutherAI/pythia-12b-deduped97.78180BPEKoboldAI/fairseq-dense-13B98.89180BPESalesforce/codegen-16B-multi97.22180BPEai-forever/rugpt3large_based_on_gpt297.78180BPEbigscience/bloom99.44180BPEdatabricks/dolly-v2-3b97.78180BPEfacebook/bart-large-mnli97.22180BPEfacebook/galactica-120b98.33180BPEfacebook/opt-66b98.89180BPEgpt297.22180BPElaion/CLIP-ViT-bigG-14-laion2B-39B-b160k61.11180BPEmicrosoft/deberta-base96.11180BPEroberta-base96.11180BPEsentence-transformers/all-roberta-large-v196.11180BPEstabilityai/stablecode-completion-alpha-3b-4k98.33180BPEstabilityai/stablelm-tuned-alpha-7b97.78180SentencePieceNousResearch/Llama-2-13b-hf100.00180SentencePieceNousResearch/Llama-2-13b-hf_slow100.00180SentencePieceTHUDM/chatglm2-6b100.00180SentencePieceTHUDM/chatglm2-6b_slow100.00180SentencePieceTHUDM/chatglm3-6b100.00180SentencePieceTHUDM/chatglm3-6b_slow100.00180SentencePiececamembert-base0.00180SentencePiececamembert-base_slow75.00180SentencePiececodellama/CodeLlama-7b-hf100.00180SentencePiececodellama/CodeLlama-7b-hf_slow100.00180SentencePiecemicrosoft/deberta-v3-base93.33180SentencePiecemicrosoft/deberta-v3-base_slow100.00180SentencePiecexlm-roberta-base98.89180SentencePiecexlm-roberta-base_slow98.89180SentencePiecexlnet-base-cased61.11180SentencePiecexlnet-base-cased_slow53.33180TiktokenQwen/Qwen-14B-Chat98.15108TiktokenSalesforce/xgen-7b-8k-base97.22108WordPieceProsusAI/finbert80.0040WordPiecebert-base-multilingual-cased80.0040WordPiecebert-large-cased80.0040WordPiececointegrated/rubert-tiny280.0040WordPiecedistilbert-base-uncased-finetuned-sst-2-english80.0040WordPiecegoogle/electra-base-discriminator80.0040WordPiecegoogle/mobilebert-uncased95.0040WordPiecejhgan/ko-sbert-sts75.0040WordPieceprajjwal1/bert-mini95.0040WordPiecerajiv003/ernie-finetuned-qqp95.0040WordPiecerasa/LaBSE72.5040WordPiecesentence-transformers/all-MiniLM-L6-v275.0040WordPiecesqueezebert/squeezebert-uncased80.0040Recreating Tokenizers From TestsIn some tokenizers, you need to select certain settings so that their output is closer to the Huggingface tokenizers:THUDM/chatglm2-6bdetokenizer always skips special tokens. Useskip_special_tokens=Trueduring conversionTHUDM/chatglm3-6bdetokenizer don't skips special tokens. Useskip_special_tokens=Falseduring conversionAll tested tiktoken based detokenizers leave extra spaces. Useclean_up_tokenization_spaces=Falseduring conversion