Unable to load Wav2Vec2 fine-tuned models from local files
Author: peterwolf-smarsh
Pull request: no

## Environment info
- `transformers` version: 4.16.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.7
- PyTorch version (GPU?): 1.10.2 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten

## Information

I am fine tuning a XLS-R model

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I fine tuned a model (facebook/wav2vec2-large-xlsr-53) using a variant of the example script https://huggingface.co/blog/fine-tune-wav2vec2-english

It produced a file speech_recognition_model.pt and a directory of checkpoints, as expected.

However, I am unable to load either PT or checkpoint.  For example, when I do this
```
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
MODEL_ID = "speech_recognition_model.pt"
processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
model = Wav2Vec2ForCTC.from_pretrained(MODEL_ID)
```
I get this error
```
Traceback (most recent call last):
  File "/Users/peter.wolf/opt/miniconda3/envs/wav2vec/lib/python3.9/site-packages/transformers/feature_extraction_utils.py", line 423, in get_feature_extractor_dict
    text = reader.read()
  File "/Users/peter.wolf/opt/miniconda3/envs/wav2vec/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte
```
And when I try a checkpoint
```
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
MODEL_ID = "./training/checkpoint-90500/"
processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
model = Wav2Vec2ForCTC.from_pretrained(MODEL_ID)
```
I get this
```
Traceback (most recent call last):
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/pydevd.py", line 1483, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm CE.app/Contents/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/peter.wolf/dev/wav2vec_experiments/src/pt.py", line 3, in <module>
    processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
  File "/Users/peter.wolf/opt/miniconda3/envs/wav2vec/lib/python3.9/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py", line 127, in from_pretrained
    tokenizer = Wav2Vec2CTCTokenizer.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/Users/peter.wolf/opt/miniconda3/envs/wav2vec/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1757, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for './training/checkpoint-90500/'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure './training/checkpoint-90500/' is the correct path to a directory containing all relevant tokenizer files.
```

## Expected behavior

As far as I can tell, my code only differs from the examples by not pushing the models to Huggingface.  I would expect that I can use both pretrained models (PT file) and checkpoints to do inference.  However, I am new to this code, so it is likely that I am making some kind of error... but I can not see it.

Comments:
Hey @peterwolf-smarsh,

Pretrained checkpoints do not contain a tokenizer, therefore you **cannot** load a processor from a pretrained checkpoint. E.g.:

```python
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
MODEL_ID = "./training/checkpoint-90500/"
processor = Wav2Vec2Processor.from_pretrained(MODEL_ID)
```

fails if `MODEL_ID` is just a pretrained tokenizer.

You should however be able to load the CTC model correctly. To me it looks like your checkpoint is broken in some way. 
Could you maybe try to upload the model checkpoint to the Hub? Otherwise there is no way we can help you with the issue sadly 


Thank you Patrick for the quick help.  I have uploaded my PT file and checkpoints directory

https://huggingface.co/opus111/issue-16060-pt-file/tree/main

Note that I am able to load the model.  This works
```
MODEL_ID = "speech_recognition_model.pt"
model = Wav2Vec2ForCTC.from_pretrained(MODEL_ID)
```
Perhaps I just don't understand how to load a processor from the results of training

In order to load a processor, you need both tokenizer files and feature extractor files. They seem to be missing in your repo at the moment. If you compare this to [this repo](https://huggingface.co/facebook/wav2vec2-base-960h/tree/main) you see that there are all the files necessary for the tokenizer and feature extractor

Hmmm interesting...  I assumed that those files should be saved automatically as part of checkpointing.  Is there something I need to do, or is this a bug?  Here is my training code below:
```
    model = Wav2Vec2ForCTC.from_pretrained(
        "facebook/wav2vec2-base",
        ctc_loss_reduction="mean",
        pad_token_id=processor.tokenizer.pad_token_id,
    )
    model.freeze_feature_extractor()

    training_args = TrainingArguments(
      output_dir="training",
      group_by_length=True,
      per_device_train_batch_size=32,
      evaluation_strategy="steps",
      num_train_epochs=30,
      fp16=True,
      gradient_checkpointing=True,
      save_steps=500,
      eval_steps=500,
      logging_steps=500,
      learning_rate=1e-4,
      weight_decay=0.005,
      warmup_steps=1000,
      save_total_limit=2,
    )

    trainer = Trainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=train_prepared,
        eval_dataset=test_prepared,
        tokenizer=processor.feature_extractor,
    )

    trainer.train()
```

I see! Yeah good point actually! The way it's written right now would only save the `feature_extractor` and the model. You will need to save the tokenizer manually since it's not passed into the Trainer. 

So in short you'll need a line like this one: https://github.com/huggingface/transformers/blob/0c55d47cded0c711a4359afb04d73ffb86b7d913/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L647 somewhere in the script

Excellent!  Thank you so much for the help, and thank you for this wonderful package and all the tutorials :-D

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

FYI: I ran into the exact same issue, thanks for the hint to save the tokenizer manually.