Tokenizer
#3
by
Saugatkafley
- opened
How to initialize tokenizer.
The repo also lacks tokenizer_config.json
We have migrated the custom IndicTrans2 tokenizer (for HF compatible IndicTrans2 models) to IndicTransTokenizer repository and it will be maintained there separately from now onwards.
This is automatically installed when you call install.sh script in huggingface_interface.
pranjalchitale
changed discussion status to
closed