Issue: List index out of range when using Seq2SeqTrainer
Author: DidiDerDenker
Pull request: no

## Environment info
- `transformers` version: v4.5.0
- Platform: Google Colab
- Python version: Python 3.7
- Using GPU in script? Yes

## Who can help
- tokenizers: @LysandreJik
- trainer: @sgugger

## Information
I am using a pre-trained Bert in order to train an abstractive summarization model. The problem arises when using my own colab notebook. The error arises during validation, sometimes sooner or later. The code is very similar to: https://colab.research.google.com/drive/1WIk2bxglElfZewOHboPFNj8H44_VAyKE?usp=sharing#scrollTo=Gw3IZYrfKl4Z

## To reproduce
Here are a few code snippiets to reproduce this behavior:

```ruby
import transformers as ft

training_args = ft.Seq2SeqTrainingArguments(
    predict_with_generate=True,
    evaluation_strategy="steps",
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    output_dir=path_output,
    warmup_steps=1000,
    save_steps=2000,
    logging_steps=100,
    eval_steps=2000,
    save_total_limit=1,
    fp16=True
)

trainer = ft.Seq2SeqTrainer(
    model=tf2tf,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_data,
    eval_dataset=val_data,
    tokenizer=tokenizer
)

trainer.train()
```

Error message:
```ruby
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-23-38bced663988> in <module>()
      9 )
     10 
---> 11 trainer.train()

12 frames

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in <genexpr>(.0)
    878     @staticmethod
    879     def _unnest(py_dict):
--> 880         return dict((key, array[0]) for key, array in py_dict.items())
    881 
    882     @staticmethod

IndexError: list index out of range
```

## Expected behavior

The training should go through without errors, as in previous versions. I would be happy if someone knows what I need to adjust in the code to make it run. Thanks :)

Comments:
The error seems to come from your dataset, and you did not share the code you used to create and process it, so there is little we can do to help.

Thanks for the quick reply. Here is the code I used to prepare the data:

```
train_data = datasets.load_dataset("cnn_dailymail", "3.0.0", split="train")
val_data = datasets.load_dataset("cnn_dailymail", "3.0.0", split="validation[:10%]")
test_data = datasets.load_dataset("cnn_dailymail", "3.0.0", split="test[:5%]")

encoder_max_length = 512
decoder_max_length = 128
batch_size = 4 # 16

def process_data_to_model_inputs(batch):
    inputs = tokenizer(batch["article"], padding="max_length", truncation=True, max_length=encoder_max_length)
    outputs = tokenizer(batch["highlights"], padding="max_length", truncation=True, max_length=decoder_max_length)

    batch["input_ids"] = inputs.input_ids
    batch["attention_mask"] = inputs.attention_mask
    batch["decoder_input_ids"] = outputs.input_ids
    batch["decoder_attention_mask"] = outputs.attention_mask
    batch["labels"] = outputs.input_ids.copy()
    batch["labels"] = [[-100 if token == tokenizer.pad_token_id else token for token in labels] for labels in batch["labels"]]

    return batch

train_data = train_data.shuffle()

train_data = train_data.map(
    process_data_to_model_inputs, 
    batched=True, 
    batch_size=batch_size, 
    remove_columns=["article", "highlights"] # "id"
)

train_data.set_format(
    type="torch",
    columns=["input_ids",
             "attention_mask",
             "decoder_input_ids",
             "decoder_attention_mask",
             "labels"]
)

val_data = val_data.shuffle()

val_data = val_data.map(
    process_data_to_model_inputs, 
    batched=True, 
    remove_columns=["article", "highlights"] # id
)

val_data.set_format(
    type="torch",
    columns=["input_ids",
             "attention_mask",
             "decoder_input_ids",
             "decoder_attention_mask",
             "labels"]
)
```

Then I loaded the pre-trained models and set parameters as in the original notebook. If more information is needed, please let me know.

The evaluation runs without any problem on my side, with the code you provided and the rest from the notebook you mentioned. Are you sure you have the latest version of the `Datasets` library installed?
Otherwise, could you share a colab or a full script reproducing the error?

Thank you! An update to the latest version of `Datasets` solved my problem.