OSError: Can't load tokenizer
Author: amarahiqbal
Pull request: no

### System Info

Python Version: 3.10.11 (main, May 16 2023, 00:28:57) [GCC 11.2.0]
Transformers Version: 4.29.2
Platform - Azure

### Who can help?

 @ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

model_name = "facebook/xmod-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
OSError: Can't load tokenizer for 'facebook/xmod-base'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'facebook/xmod-base' is the correct path to a directory containing all relevant files for a XLMRobertaTokenizerFast tokenizer.


### Expected behavior

The tokenizer is not getting loaded.
OSError: Can't load tokenizer for 'facebook/xmod-base'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'facebook/xmod-base' is the correct path to a directory containing all relevant files for a XLMRobertaTokenizerFast tokenizer.


Comments:
Hey @amarahiqbal, see the model card of this model here: https://huggingface.co/facebook/xmod-base#tokenizer

Namely, this part:
![image](https://github.com/huggingface/transformers/assets/30755778/822925e7-9e03-4b60-9b79-2ff00c6612e1)

Feel free to open a discussion in their repository; ideally, they should copy/paste the tokenizer files from `xlm-roberta-base` into their own repository so that the command you have above works seamlessly. 

Let me know if you'd like me to open a thread there instead!

Hi I can help you to solve this issue ? 
If you want to use the tokenizer of facebook/xmod-base it uses  the below tokenizer.
![image](https://github.com/huggingface/transformers/assets/77787482/8c04d3e7-03c7-4081-a418-1b246d60b1f7)
The code to load the tokenizer is as follows : 
First do : pip install transformers

Now to just load the tokenizer : 

```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('xlm-roberta-base') 
#This is the tokenizer used for the above model you have mentioned facebook/xmod-base


```
But use the above code if you want to use just use the tokenizer that is used by  facebook/xmod-base.

Now if you want to use the model facebook/xmod-base as a whole and not just the tokenizer then use the below code : 

```

from transformers import XmodModel

model = XmodModel.from_pretrained("facebook/xmod-base")
model.set_default_language("en_XX")


```

The HF link for the tokenizer : https://huggingface.co/xlm-roberta-base

from lavis.models import load_model_and_preprocess
from lavis.processors import load_processor
# setup device to use
# from transformers import BertTokenizer

# tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
device = torch.device("cuda") if torch.cuda.is_available() else "cpu"
model, vis_processors, txt_processors = load_model_and_preprocess(name="blip2_feature_extractor", model_type="pretrain", is_eval=True, device=device)
#image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)



OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'bert-base-uncased' is the correct path to a directory containing all relevant files for a BertTokenizer tokenizer.


Maybe I have the same error, but I can't solve it too.  Do you solve the problem?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am getting the same error in Kaggle Notebook. How to solve it?


We can't help you if you don't provide a reproducer :( 

> from lavis.models import load_model_and_preprocess from lavis.processors import load_processor
> 
> # setup device to use
> # from transformers import BertTokenizer
> # tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
> device = torch.device("cuda") if torch.cuda.is_available() else "cpu" model, vis_processors, txt_processors = load_model_and_preprocess(name="blip2_feature_extractor", model_type="pretrain", is_eval=True, device=device) #image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
> 
> OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'bert-base-uncased' is the correct path to a directory containing all relevant files for a BertTokenizer tokenizer.
> 
> Maybe I have the same error, but I can't solve it too. Do you solve the problem?

same issue ！！Will you let me know if you've solved the problem?

> from lavis.models import load_model_and_preprocess from lavis.processors import load_processor
> 
> # setup device to use
> # from transformers import BertTokenizer
> # tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
> device = torch.device("cuda") if torch.cuda.is_available() else "cpu" model, vis_processors, txt_processors = load_model_and_preprocess(name="blip2_feature_extractor", model_type="pretrain", is_eval=True, device=device) #image = vis_processors["eval"](raw_image).unsqueeze(0).to(device)
> 
> OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'bert-base-uncased' is the correct path to a directory containing all relevant files for a BertTokenizer tokenizer.
> 
> Maybe I have the same error, but I can't solve it too. Do you solve the problem?

I'm facing the same problem! How did you solve it?

你需要去看看官方的transformers是什么版本

DariaShel ***@***.***> 于2025年1月20日周一 21:55写道：

> from lavis.models import load_model_and_preprocess from lavis.processors
> import load_processor
> setup device to use from transformers import BertTokenizer tokenizer =
> BertTokenizer.from_pretrained('bert-base-uncased')
>
> device = torch.device("cuda") if torch.cuda.is_available() else "cpu"
> model, vis_processors, txt_processors =
> load_model_and_preprocess(name="blip2_feature_extractor",
> model_type="pretrain", is_eval=True, device=device) #image = vis_processors
> "eval" <http://raw_image>.unsqueeze(0).to(device)
>
> OSError: Can't load tokenizer for 'bert-base-uncased'. If you were trying
> to load it from 'https://huggingface.co/models', make sure you don't have
> a local directory with the same name. Otherwise, make sure
> 'bert-base-uncased' is the correct path to a directory containing all
> relevant files for a BertTokenizer tokenizer.
>
> Maybe I have the same error, but I can't solve it too. Do you solve the
> problem?
>
> I'm facing the same problem! How did you solve it?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/26612#issuecomment-2602489676>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/BFEXZ2VPWXB46GRHPBZGKIL2LT54NAVCNFSM6AAAAABPAM35H6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMBSGQ4DSNRXGY>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>