tokenizer / README.md
julien-c's picture
julien-c HF staff
Add BLOOM license to tokenizer (#4)
d43158e
metadata
license: bigscience-bloom-rail-1.0

Tokenizer used for all BLOOM models

Tokenizer information are provided at https://huggingface.co/bigscience/bloom#preprocessing

TODO: point to paper once it comes out with extra details on the tokenizer