Hugging Face's logo

HuggingFaceBio
/
Carbon-3B

Text Generation
Transformers
Safetensors
Upper Grand Valley Dani
llama
genomic
text-generation-inference
Model card Files
xet
 Community
4

tokenizer: expose .vocab property for fast-tokenizer-style callers

#4
by kashif HF Staff - opened
base: refs/heads/main
←
from: refs/pr/4
Discussion Files changed
+8
-0
kashif
 Hugging Face Biology Research org

Adds a vocab property mirroring get_vocab() so downstream tools that expect the fast-tokenizer interface (e.g. llama.cpp's convert_hf_to_gguf.py which does tokenizer.vocab) work without a fallback. No behavior change.
tokenizer: expose .vocab property for fast-tokenizer-style callers4c899c4a
kashif changed pull request status to merged

Sign up or log in to comment