Phi 3 tokenizer_config has been updated upstream

by smcleod - opened Apr 29

Apr 29

FYI https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/commit/fc313b0514906a348a624cc9421e36f52c60e137

The change updates the tokenizer config for the chat template:

   "chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') %}{{'<|user|>' + '\n' + message['content'] + '<|end|>' + '\n' + '<|assistant|>' + '\n'}}{% elif (message['role'] == 'assistant') %}{{message['content'] + '<|end|>' + '\n'}}{% endif %}{% endfor %}",

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment