Error when loading LlamaTokenizer
Author: creamiracle
Pull request: no

### System Info

centos 7
python version is 3.7.12
transformers 4.28.0.dev0
sentencepiece 0.1.97

### Who can help?

@ArthurZucker  @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

My situation is using a LAN, so I can't using function like tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf", add_eos_token=True) which will download the files.
Becasue of this, I have uploaded the llama-7b-hf files into the LAN, and use the directroy to load, such as tokenizer = LlamaTokenizer.from_pretrained("/home/qilin7/chat/llama-7b-hf", add_eos_token=True)
When I do this, I get the error like below:

The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
The tokenizer class you load from this checkpoint is 'LLaMATokenizer'.
The class this function is called from is 'LlamaTokenizer'.

RuntimeError Traceback (most recent call last)
/tmp/ipykernel_1357/3612021170.py in
5 tokenizer = LlamaTokenizer.from_pretrained("/home/qilin7/chat/llama-7b-hf", add_eos_token=True)
6 tokenizer.pad_token = tokenizer.eos_token
7 tokenizer.pad_token_id = tokenizer.eos_token_id

/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
1818 local_files_only=local_files_only,
1819 _commit_hash=commit_hash,
-> 1820 **kwargs,
1821 )
1822

/opt/conda/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, resolved_vocab_files, pretrained_model_name_or_path, init_configuration, use_auth_token, cache_dir, local_files_only, _commit_hash, *init_inputs, **kwargs)
1963 # Instantiate tokenizer.
1964 try:
-> 1965 tokenizer = cls(*init_inputs, **init_kwargs)
1966 except OSError:
1967 raise OSError(

/opt/conda/lib/python3.7/site-packages/transformers/models/llama/tokenization_llama.py in init(self, vocab_file, unk_token, bos_token, eos_token, pad_token, sp_model_kwargs, add_bos_token, add_eos_token, clean_up_tokenization_spaces, **kwargs)
94 self.add_eos_token = add_eos_token
95 self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs)
---> 96 self.sp_model.Load(vocab_file)
97
98 def getstate(self):

/media/cfs/.pylib/lib/python3.7/site-packages/sentencepiece/init.py in Load(self, model_file, model_proto)
903 if model_proto:
904 return self.LoadFromSerializedProto(model_proto)
--> 905 return self.LoadFromFile(model_file)
906
907

/media/cfs/.pylib/lib/python3.7/site-packages/sentencepiece/init.py in LoadFromFile(self, arg)
308
309 def LoadFromFile(self, arg):
--> 310 return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
311
312 def _EncodeAsIds(self, text, enable_sampling, nbest_size, alpha, add_bos, add_eos, reverse, emit_unk_piece):
RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

### Expected behavior

I had ask the problem on sentencepiece, the had replyed like this
https://github.com/google/sentencepiece/issues/850#issuecomment-1504857453

Thanks.

Comments:
Given the error, the tokenizer checkpoints that you are using are clearly outdated. Convert the model again using `main` then upload them to your LAN. Also not that you using the fast tokenizer will require to have the latest `tokenizers` library