Why loading tokenizer is very slow?
#8
by
shreyans92dhankhar
- opened
Loading the tokenizer is very slow, any idea why?
Don't know
Instead of using AutoTokenizer use LlamaTokenizer, it will load it fast
Loading the tokenizer is very slow, any idea why?
Don't know
Instead of using AutoTokenizer use LlamaTokenizer, it will load it fast