output_scores causes trainer.predict to error out
Author: tanyaroosta
Pull request: no

## Environment info

- `transformers` version: 4.17.0
- Platform: Linux-4.14.252-131.483.amzn1.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.13
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@[patrickvonplaten](https://github.com/patrickvonplaten)

Models: BARTForConditionalGeneration

Library:
- Text generation: @patrickvonplaten @narsil
- Trainer: @sgugger

## Information

Model I am using : BARTForConditionalGeneration

The problem arises when using:

* [ x] my own modified scripts: (give details below):  
I am using the official example from https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization.py
 with one modification to add output_scores and return_dict_in_generate in the PretrainedConfig

The tasks I am working on is:
* [ x] my own task or dataset: (give details below)
Finetune a BART model with my own dataset and predict on a testset

## To reproduce

Steps to reproduce the behavior:

1. Get the script from https://github.com/huggingface/transformers/blob/main/examples/pytorch/summarization/run_summarization.py
2. add the following lines (in asterisks) to the pretrained config loading, based on the documentation at: https://huggingface.co/transformers/v4.7.0/_modules/transformers/configuration_utils.html#PretrainedConfig

```
 config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        **return_dict_in_generate=True,
        output_scores=True,**
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        use_auth_token=True if model_args.use_auth_token else None,
    )
**assert config.return_dict_in_generate==True
assert config.output_scores==True**
```

3. Run the script as follows:

`python -m torch.distributed.launch --nproc_per_node=1 run_summarization.py     --model_name_or_path sshleifer/distilbart-xsum-12-6      --do_train False    --do_predict True     --do_eval False     --train_file  trainAll_allDesc_noCQA.csv     --validation_file val.csv     --test_file testAll.csv     --output_dir  /home/ec2-user/SageMaker/output/     --per_device_train_batch_size=1     --per_device_eval_batch_size=1      --overwrite_output_dir     --predict_with_generate True    --num_train_epochs 10     --text_column text     --summary_column summary     --learning_rate 3e-5     --weight_decay 0.01     --adam_beta2 0.98     --warmup_steps 5000 --generation_max_length 35`


The error snippet:

```
03/23/2022 13:31:38 - INFO - __main__ - *** Predict ***
[INFO|trainer.py:2389] 2022-03-23 13:31:38,756 >> ***** Running Prediction *****
[INFO|trainer.py:2391] 2022-03-23 13:31:38,756 >>   Num examples = 5126
[INFO|trainer.py:2394] 2022-03-23 13:31:38,756 >>   Batch size = 1
Traceback (most recent call last):
  File "run_summarization.py", line 737, in <module>
    main()
  File "run_summarization.py", line 690, in main
    predict_dataset, metric_key_prefix="predict", max_length=max_length, num_beams=num_beams
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer_seq2seq.py", line 119, in predict
    return super().predict(test_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer.py", line 2332, in predict
    test_dataloader, description="Prediction", ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer.py", line 2431, in evaluation_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/transformers/trainer_seq2seq.py", line 180, in prediction_step
    if generated_tokens.shape[-1] < gen_kwargs["max_length"]:
AttributeError: 'BeamSearchEncoderDecoderOutput' object has no attribute 'shape'
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p36/lib/python3.6/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
```

## full script:
https://github.com/tanyaroosta/summarization/blob/main/run_summarization.py

## Expected behavior

If I use the same exact command to run the script, when I don't have the two added config arguments, everything runs fine, and it generates the predictions.  However, as soon as I add the output_scores=True to the pretrained config, it errors out with the error shown above. I also noticed that if I print the model.config after loading the model, the two added parameters don't show up, but I am not sure if that is necessarily an issue.  


Comments:
Yes, the `Seq2SeqTrainer` does not support `return_dict_in_generate` for now, so that is expected. In the meantime, using the `no_trainer` example would allow you to customize everything to your needs.

thanks @sgugger for the quick reply.  Does it support `output_scores`? I took out the `return_dict_in_generate` and left `output_scores=True`.  However, it seems no scores are printed in the generated_predictions.txt.  It didn't error out though...

Not sure it will accumulate them either, the loop is focused on the generation, not additional outputs.

thanks, so for the time being, the safest thing is to use the `no_trainer` script? @sgugger 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@sgugger 
In my case, When inference.
I have to get output score, and error accure even now 😥
I think code use like this,
``` python3
trainer = Seq2SeqTrainer(
    model=model,
    data_collator=data_collator,
    args=training_args,
)
gen_kwargs = {"output_scores": True, "return_dict_in_generate": True}
output = trainer.predict(test_dataset, **gen_kwargs)
```
and, override `Seq2SeqTrainer.prediction_step`
pseudo code...
```
output= self.model.generate(
    generation_inputs,
    **gen_kwargs,
)
IF gen_kwargs.get("return_dict_in_generate") or (generated_tokens is  BeamSearchDecoderOnlyOutput or BeamSearchEncoderDecoderOutput) ?
    generated_tokens = output["sequences"]
....
return (loss, generated_toeksn, labels, output)
```

how about this? if it is fine, can i contribute to trainer_seq2seq.py? (not override, update `prediction_step`)