Inference API: Can't load tokenizer using from_pretrained, please update its configuration: No such file or directory (os error 2)
Author: nbravulapalli
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@LysandreJik  @patil-suraj 
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information
I am trying to use the Inference API in the HuggingFace Hub with a version of GPT-2 I finetuned on a custom task.

## To reproduce
When I try to use the api, the following error comes
![image](https://user-images.githubusercontent.com/87538360/140842732-f247ff10-5bc3-4d85-be25-b0fbc7c25377.png)

Steps to reproduce the behavior:
Here is the files I have in my private repo:
![image](https://user-images.githubusercontent.com/87538360/140842797-051c1980-03d1-46c5-880e-88e69dd569b2.png)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I uploaded the tokenizer files to colab, and I was able to instantiate a tokenizer with the from_pretrained method, so I don't know why the inference api throws an error.

Comments:
cc @Narsil 

Did you fix anything ? It seemed to be working now, no ?

@Narsil I downloaded the tokenizer.json file from the original gpt2-medium checkpoint from the hub and I added it to my model's repo and it works now. However, this file is not produced automatically by the 'save_pretrained()' method of the hugginface GPT2LMHeadModel class, or the AutoTokenizer class . When loading a tokenizer manually using the AutoTokenizer class in Google Colab, this 'tokenizer.json' file isn't necessary (it loads correctly given just the files from AutoTokenizer.save_pretrained() method). Was my solution of adding the tokenizer.json correct, or will it cause any hidden errors?

@nbravulapalli .

The `tokenizer.json` shouldn't be necessary (It does speed up loading time though). I can't tell you why it wasn't able to load before.

Glad you could work it out. Closing this for now, feel free to reopen.

Thanks @Narsil and @patil-suraj , 

Also, I purchased a Inference API subscription, but I had question about the pricing structure, so I emailed 'api-enterprise@huggingface.co', but I haven't gotten a response. Can you please take a look at that?

Thank you!

I'm facing this exact same issue on jamiealexandre/curriculum-breadcrumbs-gpt2 (private, but feel free to look, assuming you have access). I tried uploading tokenizer.json from the base gpt2 model (which I used as a base for finetuning), but it doesn't seem to have made a difference.

I get that same error message whether trying to generate text through the web UI or the hosted API.

When I run `GPT2Tokenizer.from_pretrained` using the path to my local clone of the repo, it loads successfully.

I noticed that the gpt2 repo didn't have the tokenizer_config.json in it, whereas mine did, so I deleted that file and now it seems to be working!

That file was automatically created and pushed when I did `tokenizer.push_to_hub("curriculum-breadcrumbs-gpt2", private=True, use_auth_token=True)`. I'm guessing it should be excluded? From looking inside it, my guess is that it contains hard-coded local paths that don't work once in the cloud.

`tokenizer_config.json` is necessary  for some additional information in the tokenizer. Original `gpt2` repo might be different, but there's some code for legacy models to make sure everything works smoothly for those.

The path within that file is indeed something to look into but it should work nonetheless.

Same problem here, any idea of how to fix it? I have other models that work fine but they contain the `tokenizer.json` file, which is not needed.

Hi @elozano98,

Do you mind sharing the name of the model ? (or send an email to `api-enterprise@huggingface.co`).

The API really only does `AutoTokenizer.from_pretrained("your_model_name").

The usual issue is with improper files (which means it shouldn't load locally either with that code).
Occasionally there are issues with `spm` + `bpe` (which is a rare combination) which just takes extremely long to load (because file formats are different, `tokenizers` has to go through O(n²) tokens to reconstruct its own map. This can be completely avoided  by simply saving `tokenizer.json`. In python: 
```python
tokenizer._tokenizer.save("tokenizer.json")
```

Another thing could be a recent dependency not yet added on the API (but I don't think there was one for tokenizers recently).


Hi @Narsil, I am facing the same issue, I believe this is happening because in my tokenizer_config.json, the file location for the "tokenizer_file" is given as "/root/.cache/huggingface/transformers/75abb59d7a06f4f640158a9bfcde005264e59e8d566781ab1415b139d2e4c603.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4". I believe if I change this, my model would not throw this error. But I am not sure, what should I replace that with, can you tell me?

Hi @debjyoti003 ,

This `tokenizer_file` location is not read I think so I don't think there's an issue with this being in your file.
Do you mind providing the model name so we can look it up ?

If you want to check yourself you can follow those instructions.

When you have a local directory, let's say : `my_awesome_model`.

Does:
```python
AutoTokenizer.from_pretrained("./my_awesome_model")
``` 
work ?

Then, once you uploaded it to you account on hf.co, does

```python
AutoTokenizer.from_pretrained("debjyoti003/my_awesome_model", use_auth_token=True)
```
work ?

If both those work, then the error most likely comes from the `api-inference`. It *might* be a tag issue where the api expects a different kind of model that the one provided (possible but unlikely).
If only the first one works, then you probably need to upload more files to the hub. Something else could be at play, don't hesitate to ping on this with the actual model so we can take a look
I don't expect that None of these work. :)

Hi @Narsil yeah, both are working, but after exploring some of the different model files, I found that most have been trained with pytorch backend, and they have pytorch_model.bin file. But in my case that file (pytorch_model.bin) is not getting generated, what do you think @Narsil is this the reason this issue is happening? The model name is distilbert-base-uncased

Just chiming in that I ran into this issue fine-tuning [distilgpt2](https://huggingface.co/distilgpt2) following [this example notebook]( https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling.ipynb).

I was able to get it to work by manually copying [tokenizer.json](https://huggingface.co/distilgpt2/blob/main/tokenizer.json) into my repo after the notebook posted it to huggingface.

Is there a step I can add to the notebook to include this file or am I missing something else?

@shiffman ,

I am not sure what are the steps for `push_to_hub` to upload the tokenizer.
```python
tokenizer.push_to_hub("sgugger/my-awesome-model")
```

Might be necessary

@sgugger  can you confirm ?

Yes, as highlighted by the [documentation](https://huggingface.co/docs/transformers/model_sharing#use-the-pushtohub-function).

Ah, thank you! Would it be helpful for me to pull request a fix to this notebook?

https://github.com/huggingface/notebooks/blob/master/examples/language_modeling.ipynb

I could add it to the cell with `trainer.push_to_hub()`?

```python
trainer.push_to_hub()
tokenizer.push_to_hub()
```

<img width="678" alt="Screen Shot 2022-03-22 at 9 16 44 AM" src="https://user-images.githubusercontent.com/191758/159490487-b8f12156-496a-4764-a266-e219c4985764.png">

(Incidentally I also found another issue with the notebook where the string data is concatenated without any whitespace between training samples. I am working on updating that along with investigating whether I can add padding for data samples that are unrelated and should not be concatenated. Any advice or thoughts welcome!)


Ah yes, the `tokenizer` has not been passed to the `Trainer` in this notebook, so needs to be pushed separately. Feel free to open a PR on the notebooks repo to fix this!

Hi,

Having the same issue,
Steps followed :
- Trained the model (t5-base) using custom PyTorch (no Trainer).
- Loading model and tokenizer locally works fine using T5Tokenizer (not using AutoTokenizer ).
- Pushed the model to HuggingFace hub using model.push_to_hub() and tokenizer.push_to_hub()

Behavior : 
- Loaded tokenizer from hub using AutoTokenizer doesn't work.
- Loading using T5Tokenizer also from hub works.
- Looking at the files directory in the hub, only seeing tokenizer_config.json !
- Interface API gives the error : Can't load tokenizer using from_pretrained, please update its configuration: No such file or directory (os error 2)


I am having the same issue with a private repository of mine. I did everything listed in this thread yet nothing. Can anyone help please help @Narsil @sgugger ?

As context, this is happening with the api inference ```import requests
API_URL = "https://api-inference.huggingface.co/models/xxxxxx/xxxxxxxx"
API_TOKEN = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
headers = {"Authorization": f"Bearer {API_TOKEN}"}
def query(filename):
    with open(filename, "rb") as f:
        data = f.read()
    response = requests.request("POST", API_URL, headers=headers, data=data)
    return json.loads(response.content.decode("utf-8"))
data = query("Recording.wav")
print(data)```

When I use the standard inference script ```from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC, Wav2Vec2CTCTokenizer
from datasets import load_dataset
import torch

token='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
processor = Wav2Vec2Processor.from_pretrained("xxxx/xxxxxx", use_auth_token=token)
tokenizer = Wav2Vec2CTCTokenizer.from_pretrained("xxxx/xxxxxx", use_auth_token=token)
model = Wav2Vec2ForCTC.from_pretrained("xxxx/xxxxxx", use_auth_token=token)

tokenizer.push_to_hub("xxxx/xxxxxx", use_auth_token=token)
processor.push_to_hub("xxxx/xxxxxx", use_auth_token=token)

ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
input_values = processor(ds[0]["audio"]["array"], return_tensors="pt", padding="longest").input_values  # Batch size 1
 
logits = model(input_values).logits
predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)
print('Testing ASR Decoded: {}'.format(transcription))``` This perfectly works

@Narsil @sgugger 

Do you mind creating a new issue with all the necessary information?

Ideally including the model id (if you can ofc).

The API does in substance:

```python
pipe = pipeline(model="xxx/xxx")
out = pipe(audio_filename)
print(out)
```
That might yield errors not seen in the code you include.


@Narsil which information do you need? 

When I use the code you provided above, it does work as intended
```
from transformers import pipeline
token='xxxxxx'
pipe = pipeline(model="xxx/xxxx", use_auth_token=token)
out = pipe("Recording.wav")
print(out)
```

Then I think I would need the model id to see since something must be wrong in what the API sees it seems.
Is is something you could do  ?

Otherwise maybe try to setup a dummy model in the same way ?

The code you shared works on public models like https://huggingface.co/facebook/wav2vec2-base-960h

Is it maybe a ngram enabled wav2vec2 ? (In which case you should have had a warning in your local pipeline code)

Is it okay if I share the model ID but have it private still?

Yes, the code and even the hosted inference API works for that model, even for many more finetuned versions. I tried to browse through lots of them - yet nothing seems to be working. Sometimes it runs, sometimes it does not - and the randomness is not good

@Narsil 

> Is it okay if I share the model ID but have it private still?

Yes, just share the org or username if you want, I have production access to see the faulty deployments. (just there are quite a few atm so hard to say what's wrong)

> and the randomness is not good

This is a very odd queue indeed, probably very interesting lead !

@Narsil the org username is lelapa. There’s a single model there

Thanks for the info, it seems there's indeed a bug in `transformers` with tokens.

I opened a PR with a fix, which will land on the API asap. Thanks for the report (next time, don't hesitate to create a new issue though and link the old one if you're not sure)

Thanks @Narsil I will stay tuned for the feedback on the resolution of this

@Narsil Any updates?

Thanks for the ping, should be good now ! (Sorry the llamav2 release took a bit too much attention from me :) )

Can you confirm ?

@Narsil Haha I understand. Yes I can confirm it is working well