ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds Transformers Translation Tutorial Repro
Author: SoyGema
Pull request: no

### System Info

### Context 
Hello There!
First and foremost, congrats for Transformers Translation[ tutorial](https://huggingface.co/docs/transformers/tasks/translation). 👍 
 It serves as a Spark for building english-to-many  translation languages models!
I´m following it along with TF mostly reproducing it in a jupyter Notebook with TF for mac with GPU enabled
Using the following dependency versions. 

```
tensorflow-macos==2.9.0
tensorflow-metal==0.5.0
transformers ==4.29.2
```
_* NOTE : tensorflow-macos dependencies are [fixed ](https://developer.apple.com/forums/thread/721619) for ensuring GPU training_



### Who can help?

 @ArthurZucker @younesbelkada  
@gante maybe?

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

### Issue Description
Im finding the following error when **fitting the model** for finetunning a model coming from [TFAutoModelForSeq2SeqLM](https://huggingface.co/docs/transformers/v4.30.0/en/model_doc/auto#transformers.TFAutoModelForSeq2SeqLM) autoclass
```
with tf.device('/device:GPU:0'):
    model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=1, callbacks= callbacks ) 
```

It is returning 
```
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
        
        
        Call arguments received by layer "decoder" (type TFT5MainLayer):
          • self=None
          • input_ids=None
          • attention_mask=None
          • encoder_hidden_states=tf.Tensor(shape=(32, 96, 512), dtype=float32)
          • encoder_attention_mask=tf.Tensor(shape=(32, 96), dtype=int32)
          • inputs_embeds=None
          • head_mask=None
          • encoder_head_mask=None
          • past_key_values=None
          • use_cache=True
          • output_attentions=False
          • output_hidden_states=False
          • return_dict=True
          • training=False
    
    
    Call arguments received by layer "tft5_for_conditional_generation" (type TFT5ForConditionalGeneration):
      • self={'input_ids': 'tf.Tensor(shape=(32, 96), dtype=int64)', 'attention_mask': 'tf.Tensor(shape=(32, 96), dtype=int64)'}
      • input_ids=None
      • attention_mask=None
      • decoder_input_ids=None
      • decoder_attention_mask=None
      • head_mask=None
      • decoder_head_mask=None
      • encoder_outputs=None
      • past_key_values=None
      • inputs_embeds=None
      • decoder_inputs_embeds=None
      • labels=None
      • use_cache=None
      • output_attentions=None
      • output_hidden_states=None
      • return_dict=None
      • training=False
```


### Backtrace
Tried:
* Remove callbacks : The model is trained, but of course not loaded into the Hub, nor the metrics computed


* Followed #16234 , this[ comment ](https://github.com/huggingface/transformers/issues/16234#issuecomment-1071114294) and **ensured that Im using AutoTokenizer.** This glimpsed that this could be related to TFAutoModelForSeq2SeqLM . 
```
model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)
```
Seems to be working correctly. Therefore I assume that the  **pre-trained model is loaded** 

* Also followed #21116 and added `save_strategy=no` argument in [PushToCallBack ](https://github.com/huggingface/transformers/issues/21116#issuecomment-1382869967) , but the error persisted


### Expected behavior

Model trained should be uploaded to the Hub. 
The folder appears empty , there is an error


### Hypothesis

At this point, what Im guessing is that once I load the model I shall redefine the verbose error trace?
Any help please of how to do this ? :) or how can I fix it ? Do I have to define a specific Trainer ? Any idea of where I can find this in docs?



Comments:
Hey @SoyGema 👋 

From your exception, I believe the issue is at the data preparation stage -- it is pretty much complaining that your dataset has no labels. Have you followed the data preprocessing steps described [here](https://huggingface.co/docs/transformers/tasks/translation#preprocess)?

Hello there @gante ! Thanks for your quick response and help ! 
I really appreciate it . 🥇 
I´ve uploaded the notebook [here](https://github.com/SoyGema/The-Lord-of-The-Words-The-two-frameworks/blob/main/notebooks/TransformerTranslationPOCV2.ipynb) . As far as I can understand (let me know if Im missing something here ), Im using the preprocessing function. 

In fact, the _tokenized_books (cell 16) returns something in the form of

```
DatasetDict({
    train: Dataset({
        features: ['id', 'translation', 'input_ids', 'attention_mask', 'labels'],
        num_rows: 1123
    })
    test: Dataset({
        features: ['id', 'translation', 'input_ids', 'attention_mask', 'labels'],
        num_rows: 281
    })
})
```

And _data_collator_ (cell 19) returns something like

```
DataCollatorForSeq2Seq(tokenizer=T5Tokenizer(name_or_path='t5-small', vocab_size=32100, model_max_length=512, is_fast=False, padding_side='right', truncation_side='right', special_tokens={'eos_token': '
', 'unk_token': '<unk>', 'pad_token': '<pad>', 'additional_special_tokens': ['<extra_id_0>', .....
```

Am I missing something from the video that should be in code ?
for quick testing purposes, Im with **pt_to_en** dataset, that seems to have same characteristics. I've checked that tokenized_books function returns the same data structure type in **pt_to_en** that in **fr_to_en** dataset

My apologies in advance for the extremely notebook verbose code regarding GPU low level operation use. I am trying to optimize for that therefore all trace. 

Thanks so so much for your time on this
Happy if you can point me on the right direction! 👍 



Hey @SoyGema 👋 

Your `KerasMetricCallback` was missing `predict_with_generate=True` -- metrics that rely on text generation must pass this flag, as generating text is different from a model forward pass. It should become `metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_test_set, predict_with_generate=True)`

For future reference in case you encounter further bugs, have a look at our complete translation example: https://github.com/huggingface/transformers/blob/main/examples/tensorflow/translation/run_translation.py

Hello there @gante 👋

Thanks  for the reference. I'm definetly having this as a north script and also using it !
Been thinking about how to structure this exploration and also _indexing the roadblocks/bugs/solutions so other users can benefit from it_ . 

I'm closing this issue (as it is solved but other arised )and probably open another ones in [my own repo](https://github.com/SoyGema/The-Lord-of-The-Words-The-two-frameworks) as it goes so issues are unitary-structured . Hope this makes sense. Hope I can take it from there and not disturb you!

Thanks again!

Just for Reproducibility. If someone wants to go through the script example. Documentation about flag configuration and more can be found [here](https://huggingface.co/docs/transformers/run_scripts)