Asking about Sentencepiece model
#12
by
bibekyess
- opened
Hello! I am trying to serve this model using serving toolkit like LMDeploy [https://github.com/InternLM/lmdeploy] and they use sentencepiece model. Since Llama-2-Ko uses FastTokenizer provided by HF tokenizers NOT sentencepiece package, I am wondering is it possible to get the sentencepiece model(specifically tokenizer.model file)? I tried to convert tokenizer.json to tokenizer.model but cannot do it succesfully. Or can you suggest any other serving toolkits ?
Thank you! :)
Umm it foes not supported at this moment. Since HF tokenizer is much feasible option to use,
there is no plan to support SPM directly 🥲
bibekyess
changed discussion status to
closed