Tokenizer

#12
by deleted - opened
deleted

@FancyZhao Is the tokenizer used for Yi similar to Llama's BPE Tokenizer, forget about multilingual tokens only for English?

01-ai org

@FancyZhao Is the tokenizer used for Yi similar to Llama's BPE Tokenizer, forget about multilingual tokens only for English?

Yes, they are both trained with sentencepiece BPE mode.

FancyZhao changed discussion status to closed

Sign up or log in to comment