Full 1-1 compat with dbrx-instruct tokenizer?

by Qubitium - opened Apr 3

Apr 3

@Xenova Is this full 1-1, forward/reverse, encode/decode exactly behaves like the tiktoken tokenizer in dbrx-instruct? Thank you.

Xenova

Owner Apr 3

It should be! We've tested it on the entire xnli dataset (all languages), and it produces the same result.

Qubitium

Apr 4

@Xenova Thank you. We made a base-tokenizer based on your code with slight modifications to included pad token and to tokens that the original tokenizer added dynamically but never added to encoder. https://huggingface.co/LnL-AI/dbrx-base-tokenizer

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment