Please add an example for run_speech_recognition_ctc for eval only
Author: XekriRedmane
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.11.2
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO


### Who can help?

@sgugger

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Remove any training output directory.
2. Delete the cache.
3. Modify any of the examples, by removing --do_train.
4. Run. You should run into an exception:

```
  File "/home/xekri/eo-reco/run_speech_recognition_ctc.py", line 824, in <module>
    main()
  File "/home/xekri/eo-reco/run_speech_recognition_ctc.py", line 575, in main
    vocab_dict = create_vocabulary_from_data(
  File "/home/xekri/eo-reco/run_speech_recognition_ctc.py", line 351, in create_vocabulary_from_data
    remove_columns=datasets["train"].column_names,
  File "/home/xekri/.env/lib/python3.10/site-packages/datasets/dataset_dict.py", line 57, in __getitem__
    return super().__getitem__(k)
KeyError: 'train'
```

Note: line numbers may be different, since I modified the file with debug.

### Expected behavior

1. An example that clearly shows a working evaluation-only set of arguments.
2. No exception during evaluation-only.

Comments:
cc @sanchit-gandhi 

Hey @RobertBaruch, thanks for the issue! I would advocate against using a training script for evaluation only. The reason is that when we use a training script with HF Trainer, we still initialise optimiser states for our model, even if we're not doing any training. For common optimisers, such as Adam or AdamW, there are two optimiser states per trainable model parameter. This means for a 1GB model, we have 2GB of optimiser states, both of which are on the GPU. So immediately, 2GB of GPU memory is consumed by optimiser states we don't actually use.

What I would encourage you to do instead is run inference using `pipeline`. In doing so, you'll only put the model weights on the GPU, so you'll have more GPU memory and thus be able to run larger batch sizes. Here's a code snippet that achieves this:
```python
from datasets import load_dataset
from transformers import pipeline
from transformers.pipelines.pt_utils import KeyDataset
from tqdm.auto import tqdm

pipe = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-base-960h", device=0)
dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")

all_preds = []
for out in tqdm(pipe(KeyDataset(dataset, "file"), batch_size=8)):
    all_preds.append(out)

print(all_preds)
```
It's adapted from the docs, where you can read more about pipeline: https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.pipeline

What I really want is the ability to compute the loss, CER, and WER for the
predictions, which I don't think pipeline gives?

On Wed, May 31, 2023, 07:14 Sanchit Gandhi ***@***.***> wrote:

> Hey @RobertBaruch <https://github.com/RobertBaruch>, thanks for the
> issue! I would advocate against using a training script for evaluation
> only. The reason is that we still initialise optimiser states for our
> model, even if we're not doing any training. For common optimisers, such as
> Adam or AdamW, there are two optimiser states per trainable model
> parameter. This means for a 1GB model, we have 2GB of optimiser states,
> both of which are on the GPU. So immediately, 2GB of GPU memory is consumed
> by optimiser states we don't actually use.
>
> What I would encourage you to do instead is run inference using pipeline.
> In doing so, you'll only put the model weights on the GPU, so you'll have
> more GPU memory and thus be able to run larger batch sizes. Here's a code
> snippet that achieves this:
>
> from datasets import load_datasetfrom transformers import pipelinefrom transformers.pipelines.pt_utils import KeyDatasetfrom tqdm.auto import tqdm
> pipe = pipeline("automatic-speech-recognition", model="facebook/wav2vec2-base-960h", device=0)dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
> all_preds = []for out in tqdm(pipe(KeyDataset(dataset, "file"), batch_size=8)):
>     all_preds.append(out)
> print(all_preds)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/23882#issuecomment-1570321793>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AANTRDTOY32TU52OXW6AQVLXI5G3HANCNFSM6AAAAAAYU243JQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Well, I could get CER and WER just by comparing the result with the target via `jiwer`. But a loss would be nice to see.

Hey @RobertBaruch - in this case, you can use the `model` + `processor` API. Does this docstring answer your question? https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2ForCTC.forward.example

Definitely, thank you!