KeyError: 'input_ids' on Whisper training with include_inputs_for_metrics
Author: dmurillo976s
Pull request: no

### System Info

- `transformers` version: 4.32.0.dev0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This link https://colab.research.google.com/drive/1mMolQVClnnC_hi1J6DeCUOElJtatSXoz?usp=sharing points to a Colab notebook that reproduces the issue. It is a slightly modified version from the official post in https://huggingface.co/blog/fine-tune-whisper

Basically, whenever setting `include_inputs_for_metrics=True` for a training with whisper, the error message and stack trace below appear. 

As a clarification, I was just experimenting with `include_inputs_for_metrics`. I realize it doesn't make much sense in this context to receive the bare inputs in `compute_metrics`. It would be better to receive any other type of metadata from other dataset fields.

Nevertheless, it seems like this is a bug, given that these models receive their inputs as `input_features` and not `input_ids`.

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[<ipython-input-20-3435b262f1ae>](https://localhost:8080/#) in <cell line: 1>()
----> 1 trainer.train()

6 frames
[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1524             self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size
   1525         )
-> 1526         return inner_training_loop(
   1527             args=args,
   1528             resume_from_checkpoint=resume_from_checkpoint,

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   1886                     self.control = self.callback_handler.on_step_end(args, self.state, self.control)
   1887 
-> 1888                     self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
   1889                 else:
   1890                     self.control = self.callback_handler.on_substep_end(args, self.state, self.control)

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _maybe_log_save_evaluate(self, tr_loss, model, trial, epoch, ignore_keys_for_eval)
   2211                     metrics.update(dataset_metrics)
   2212             else:
-> 2213                 metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
   2214             self._report_to_hp_search(trial, self.state.global_step, metrics)
   2215 

[/usr/local/lib/python3.10/dist-packages/transformers/trainer_seq2seq.py](https://localhost:8080/#) in evaluate(self, eval_dataset, ignore_keys, metric_key_prefix, **gen_kwargs)
    157         self._gen_kwargs = gen_kwargs
    158 
--> 159         return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
    160 
    161     def predict(

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in evaluate(self, eval_dataset, ignore_keys, metric_key_prefix)
   2919 
   2920         eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else self.evaluation_loop
-> 2921         output = eval_loop(
   2922             eval_dataloader,
   2923             description="Evaluation",

[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in evaluation_loop(self, dataloader, description, prediction_loss_only, ignore_keys, metric_key_prefix)
   3109             # Prediction step
   3110             loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
-> 3111             inputs_decode = self._prepare_input(inputs["input_ids"]) if args.include_inputs_for_metrics else None
   3112 
   3113             if is_torch_tpu_available():

[/usr/local/lib/python3.10/dist-packages/transformers/feature_extraction_utils.py](https://localhost:8080/#) in __getitem__(self, item)
     84         """
     85         if isinstance(item, str):
---> 86             return self.data[item]
     87         else:
     88             raise KeyError("Indexing with integers is not available when using Python based feature extractors")

KeyError: 'input_ids'
```

### Expected behavior

The dummy training should be able to complete its validation loop.

Comments:
The `include_inputs_for_metrics=True` feature only supports text models, not other modalities for now.

Noted. Thanks for the quick response! I couldn't find anything that suggested so in the docs, so I hope this github issue comes in handy if anyone else tries this. 

I'll go ahead and close the issue.

You don't need to close this, we should probably support other modalities better in the Trainer ;-) I was mainly stating that `input_ids` is hard-coded in this code and we would need to update it.

Oh ok. I opened it again then. Thanks!

Should be fixed by the PR linked above if you want to try.

Hi. That's awesome! I just tried it in the colab notebook I shared and it worked nicely. Thanks!

please can you tell me the solution!!!

Hey @rokayabencheikh - could you confirm that you're using the latest version of the Transformers library? i.e. with:
```
pip install --upgrade transformers
```

thenk you