Xenova

/

claude-tokenizer

like 10

It's the one currently used in their python SDK (which has code snippets updated to use claude 3), so it should be! Although, it would be nice to get confirmation on this from Anthropic. They don't even use the word "tokenizer" in the report they published.

Hey @Xenova is this tokenizer valid for claude 3?

Probably not applicable. See: https://github.com/anthropics/anthropic-sdk-python/blob/9fad441043ff7bfdf8786b64b1e4bbb27105b112/src/anthropic/_client.py#L276-L278

Thanks @secsilm , you're right... they unfortunately haven't released the v3 tokenizer.