ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.
Author: candowu
Pull request: no

### System Info

4.27.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

i test llama in colab here is my code and output:

!pip install git+https://github.com/huggingface/transformers
!pip install sentencepiece

import torch
from transformers import pipeline,LlamaTokenizer,LlamaForCausalLM

device = "cuda:0" if torch.cuda.is_available() else "cpu"
print(device)

# tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")
# model = LlamaForCausalLM.from_pretrained("decapoda-research/llama-7b-hf")
generator = pipeline(model="decapoda-research/llama-7b-hf", device=device)
generator("I can't believe you did such a ")

ValueError                                Traceback (most recent call last)
[<ipython-input-3-c1d71e177e5a>](https://localhost:8080/#) in <module>
      7 # tokenizer = LlamaTokenizer.from_pretrained("decapoda-research/llama-7b-hf")
      8 # model = LlamaForCausalLM.from_pretrained("decapoda-research/llama-7b-hf")
----> 9 generator = pipeline(model="decapoda-research/llama-7b-hf", device=device)
     10 generator("I can't believe you did such a ")

1 frames
[/usr/local/lib/python3.9/dist-packages/transformers/models/auto/tokenization_auto.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    675 
    676             if tokenizer_class is None:
--> 677                 raise ValueError(
    678                     f"Tokenizer class {tokenizer_class_candidate} does not exist or is not currently imported."
    679                 )

ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.


### Expected behavior

expect output generated info

Comments:
I face the same issue

Hi @candowu, thanks for raising this issue. This is arising, because the `tokenizer` in the [config on the hub](https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer_config.json) points to `LLaMATokenizer`. However, the tokenizer in the library is `LlamaTokenizer`. 

This is likely due to the configuration files being created before the final PR was merged in. 

I cloned the repo and changed the tokenizer in the config file to LlamaTokenizer 
but I got
ValueError: Tokenizer class LlamaTokenizer does not exist or is not currently imported.


For anybody interested I was able to load an earlier saved model with the same issue using my [fork](https://github.com/mbehm/transformers) with the capitalization restored. That being said for future it's probably better to try find or save a new model with the new naming.

@yhifny Are you able to import the tokenizer directly using `from transformers import LlamaTokenizer `? 

If not, can you make sure that you are working from the development branch in your environment using:
`pip install git+https://github.com/huggingface/transformers`

more details [here](https://huggingface.co/docs/transformers/installation#install-from-source).

I can import the `LlamaTokenizer` class, but getting error that `from_pretrained` method is None.  Anyone else having this issue?

As the error message probably mentions, you need to install sentencepiece: `pip install sentencepiece`.

Working now. I swear I had sentencepiece, but probably forgot to reset the runtime 🤦  My bad!

> For anybody interested I was able to load an earlier saved model with the same issue using my [fork](https://github.com/mbehm/transformers) with the capitalization restored. That being said for future it's probably better to try find or save a new model with the new naming.

Thanks, man, your link solved all the problem

> Hi @candowu, thanks for raising this issue. This is arising, because the `tokenizer` in the [config on the hub](https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer_config.json) points to `LLaMATokenizer`. However, the tokenizer in the library is `LlamaTokenizer`.
> 
> This is likely due to the configuration files being created before the final PR was merged in.

Change the **LLaMATokenizer** in tokenizer_config.json into lowercase **LlamaTokenizer** and it works like a charm.

> For anybody interested I was able to load an earlier saved model with the same issue using my [fork](https://github.com/mbehm/transformers) with the capitalization restored. That being said for future it's probably better to try find or save a new model with the new naming.

Thank you so much for this! Works!

> > Hi @candowu, thanks for raising this issue. This is arising, because the `tokenizer` in the [config on the hub](https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer_config.json) points to `LLaMATokenizer`. However, the tokenizer in the library is `LlamaTokenizer`.
> > This is likely due to the configuration files being created before the final PR was merged in.
> 
> Change the **LLaMATokenizer** in tokenizer_config.json into lowercase **LlamaTokenizer** and it works like a charm.

I assume this is applied to the llama-7b cloned repo from HuggingFace right? How can I instantiate the model and the tokenizer after doing that please?

> 

you are a life saver. There docs on the site should be updated for this reference. 

Thank you so much for this! Works! That's amazing!

You can try this for a ather crazy way to find out what is the right casing for the module:

```python
import transformers

from itertools import product
import importlib

def find_variable_case(s, max_tries=1000):
  var_permutations = list(map("".join, product(*zip(s.upper(), s.lower()))))
  # Intuitively, any camel casing should minimize the no. of upper chars.
  # From https://stackoverflow.com/a/58789587/610569
  var_permutations.sort(key=lambda ss: (sum(map(str.isupper, ss)), len(ss)))
  for i, v in enumerate(var_permutations):
    if i > max_tries:
      return
    try:
      dir(transformers).index(v)
      return v
    except:
      continue


v = find_variable_case('LLaMatokenizer')
exec(f"from transformers import {v}")
vars()[v]
```

[out]:


```
transformers.utils.dummy_sentencepiece_objects.LlamaTokenizer
```

I encountered the same issue identified at the thread today 4/2/2023. The post https://github.com/huggingface/transformers/issues/22222#issuecomment-1477171703 fixed the problem for me. 

Thank you.

Hi! I am facing the same problem. I try to import LlamaTokenizer,
But:---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
Cell In[27], line 1
----> 1 from transformers import LlamaTokenizer 

ImportError: cannot import name 'LlamaTokenizer' from 'transformers' (/usr/local/anaconda3/envs/abc/lib/python3.10/site-packages/transformers/__init__.py)

and the version of transformers is "transformers              4.28.0.dev0              pypi_0    pypi"

plz tell me how to fix it.

You need to install the library from source to be able to use the LLaMA model.

> You need to install the library from source to be able to use the LLaMA model.

Thanks! Where can I get it? And how to install it? 
Actually I have already installed transformers 4.28.0.dev0, I'm not sure about what you mean.

You can open the documentation at the [install page](https://huggingface.co/docs/transformers/installation#install-from-source).

Great! I restart my server and it works! thank you !!!

Hi

I installed from source 

git clone https://github.com/huggingface/transformers.git
cd transformers
pip install -e .


 pip list show:

transformers       4.29.0.dev0 D:\myfolder\transformers

but I still have

ValueError: Tokenizer class LlamaTokenizer  does not exist or is not currently imported.



+1 on @thibaudart comment, I have the same issue.

Will this problem be fixed by updating to newest version of transformers, or must we modify the config file manually each time?

You should just using that checkpoint. The maintainers of that repo have made it clear that they are not interested in being compatible with Transformers by ignoring the 62 PRs trying to fix their checkpoints. The huggyllama checkpoints are confirmed to work if you are looking for an alternative (but you should still request the weights to Meta following their official form).

There are now 903 checkpoints for llama on the Hub and only the 4 from decapoda-research do not work since they created them before the PR for Llama was merged into Transformers. We won't break the code for the other 899 checkpoints.

                if( "LLaMATokenizer" == tokenizer_class_candidate ):  ## add these 2 line to solve it.
                    tokenizer_class_candidate = 'LlamaTokenizer'  
                tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)

@MasterLivens  hi, i am currently using colab, which file should i add this code? 

@zhiyixu The code being referred to should go into .../site-packages/transformers/models/auto/tokenization_auto.py

However, what worked for me was updating my transformers and tokenizers package. 
tokenization_auto.py has a mapping of tokenizers at the beginning and I realized that llama wasn't included in the version I had.

> > Hi @candowu, thanks for raising this issue. This is arising, because the `tokenizer` in the [config on the hub](https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer_config.json) points to `LLaMATokenizer`. However, the tokenizer in the library is `LlamaTokenizer`.
> > This is likely due to the configuration files being created before the final PR was merged in.
> 
> Change the **LLaMATokenizer** in tokenizer_config.json into lowercase **LlamaTokenizer** and it works like a charm.

Can you please enlighten me on how this could be achieved please? I'm new to this 

Hi, @nameless0704. First, I would like to thank you for the insightful comment of `changing the LLaMATokenizer in tokenizer_config.json into lowercase LlamaTokenizer `. I am fairly new in this area. May I ask how to Change the LLaMATokenizer in tokenizer_config.json into lowercase LlamaTokenizer? 

I could not figure it out and would like to seek your help. Any information is appreciated. Thank you very much in advance!

I share a experiment, Just replace your llama model to https://huggingface.co/elinas/llama-7b-hf-transformers-4.29 will solve the error like ImportError: cannot import name 'LLaMATokenizer' from 'transformers'

Example of how to use LLaMA AutoTokenizer

```
!pip install tokenizers==0.13.3
!pip install sentencepiece

from transformers import AutoTokenizer, AutoModelForCausalLM

# model_name = "openlm-research/open_llama_3b"
# model_name = "openlm-research/open_llama_7b"
model_name = "openlm-research/open_llama_13b"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
```

> @MasterLivens hi, i am currently using colab, which file should i add this code?

the specified error file in the error message.

> > Hi @candowu, thanks for raising this issue. This is arising, because the `tokenizer` in the [config on the hub](https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer_config.json) points to `LLaMATokenizer`. However, the tokenizer in the library is `LlamaTokenizer`.
> > This is likely due to the configuration files being created before the final PR was merged in.
> 
> Change the **LLaMATokenizer** in tokenizer_config.json into lowercase **LlamaTokenizer** and it works like a charm.

Where is the tokenizer_config.json?

> > > Hi @candowu, thanks for raising this issue. This is arising, because the `tokenizer` in the [config on the hub](https://huggingface.co/decapoda-research/llama-7b-hf/blob/main/tokenizer_config.json) points to `LLaMATokenizer`. However, the tokenizer in the library is `LlamaTokenizer`.
> > > This is likely due to the configuration files being created before the final PR was merged in.
> > 
> > 
> > Change the **LLaMATokenizer** in tokenizer_config.json into lowercase **LlamaTokenizer** and it works like a charm.
> 
> Where is the tokenizer_config.json?

I think this is the location:
.cache/huggingface/hub/models--decapoda-research--llama-65b-hf/snapshots/47d2b93e8c0a3d5d6582bdec13f233ca0527499a/tokenizer_config.json

Please. Im facing the same issue. Can anyone help ? I tried all the above methods.


> Please. Im facing the same issue. Can anyone help ? I tried all the above methods.

I had the same issue and it was solved by:
pip uninstall transformers
pip install transformers

in my code, transformer==4.30.0 can fix it

looked at `tokenization_auto.py` in the transformers package that was installed via `pip install git+https://github.com/huggingface/transformers`

```
 (
                "llama",
                (
                    "LlamaTokenizer" if is_sentencepiece_available() else None,
                    "LlamaTokenizerFast" if is_tokenizers_available() else None,
                ),
            ),
```
I had to install `sentencepiece` to bypass the not found error, running into other errors though :)


I had similar issues. root cause was that I was using python3.7 where pip install transformers was older version 4.18. Once I upgraded to python3.9 pip3 install transformers install 4.30.0

Also make sure you have recent cuda drivers, running only on CPU was very slow for me. 

Hi, I still have the error. I tried all the solutions above....
![Screenshot 2023-08-23 at 22 34 24](https://github.com/huggingface/transformers/assets/35782965/2a9081a4-52ee-46a4-998c-f6c6865085f6)


Do we need to load the tokenizer using LlamaTokenizer or can we use it with AutoTokenizer? If the latter is possible, what is the fully qualified tokenizer model on the hub? I get:
OSError: LLamaTokenizer is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
Where is the tokenizer_config.json stored?


@ndvbd you should be able to use `AutoTokenizer` with any tokenizers on the hub. 
If you have an issue and want us to help you, we really need a small reproducer, and the full traceback. 

For anyone still getting the same error: 
- make sure you are using the correct version of transformers. `print(transformers.__version__)`
- if you are working on a notebook cc @youshikyou make sure to restart the kernel after you have installed the packages, to make sure the changes are taken into account. 
- make sure the repository you are loading from (for example `meta-llama/Llama-2-7b-hg`)  has the correct `LlamaTokenizer` class if you are using `AutoModel`. 

If you have a different issue, make sure to open a new issue and ping me 🤗 

same error on model codellama/CodeLlama-13b-hf
can onyone post a valid json config here ?

No, the json is valid, you are just not working on `main` or at least the release that included `CodeLlamaTokenizer`... (https://github.com/huggingface/transformers/releases/tag/v4.33.1) 

I got the same error. I read that we have to edit 'tokenizer_config.json'. I did that on huggingface by changing the json file available in 'Files and versions' of the 'decapoda-research/llama-7b-hf', but I am not sure how to use the edited changes. This is my code:

`import torch
import transformers
from transformers import AutoTokenizer
from  langchain import LLMChain, HuggingFacePipeline, PromptTemplate

print("Import Complete")

model = "meta-llama/Llama-2-7b-hf
access_token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
tokenizer = AutoTokenizer.from_pretrained(model, use_auth_token=access_token)`

I get the same error- ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.

You should use `model = "path/to/your/decapoda/modifided/model`

Any solution to this error **ValueError: Tokenizer class LlamaTokenizer does not exist or is not currently imported.** ?

Install this library
pip install -U transformers 

I use this hack when loading tokenizer for [BigTranslate](https://hf-mirror.com/James-WYang/BigTranslate). Though loaded, it does not guarantee to work with model inference and training.

The [discussion](https://hf-mirror.com/James-WYang/BigTranslate/discussions/1) suggests I should use [the original codebase](https://github.com/ZNLP/BigTrans/tree/main/model/) for loading the tokenizer "LLaMATokenizer".

```python
import transformers
setattr(transformers, "LLaMATokenizer", transformers.LlamaTokenizer)
tokenizer = transformers.AutoTokenizer("/James-WYang/BigTranslate")
```

You can compare the tokenizers between [BigTranslate](https://github.com/ZNLP/BigTranslate/blob/main/model/llama/tokenization_llama.py) and [huggingface/transformers](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/tokenization_llama.py). I think they are basically the same.