False truncation of generated sequences when calling Seq2SeqTrainer.predict with num_return_sequences>1
Author: namespace-Pt
Pull request: no

### System Info

- `transformers` version: 4.30.0
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@sgugger @ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Define `compute_metrics` function to inspect predictions' shape
    ```
    def compute_metrics(eval_preds):
        preds, labels = eval_preds
        print(preds.shape)
    ```
2. Create training arguments `args = TrainingArguments(do_predict=True)`
3. Instantiate `trainer = Seq2SeqTrainer(model=model, args=args, tokenizer=tokenizer, compute_metrics=compute_metrics)`
4. Call predict method `trainer.predict(test_dataset, num_return_sequences=2, max_new_tokens=32, do_sample=True)`

I expect the predictions to be of shape `8 * 2 * k` or `16 * k` where k is the generated sequence length. However, it is always `8 * k`. I find out the `generated_tokens` [here](https://github.com/huggingface/transformers/blob/fe861e578f50dc9c06de33cd361d2f625017e624/src/transformers/trainer_seq2seq.py#L276) is `16 * k` while it is truncated to `8 * k` [here](https://github.com/huggingface/transformers/blob/fe861e578f50dc9c06de33cd361d2f625017e624/src/transformers/trainer.py#L3330), which is incorrect to me. 

### Expected behavior

I think when `num_return_sequences > 1`, the output tensor (`generated_tokens`) should be of shape `2 * 8 * k` instead of `16 * k`. Maybe you can add a simple parameter (e.g. `batch_size_alone=True/False`) to determine how the output is aligned.

By the way, I think currently there are too many ways to set generation configurations when prediction (kwargs, model configs, and default configs). Maybe they should be simplified.

Comments:
It's true that `num_return_sequences>1` is not supported by the `Seq2SeqTrainer`. If you have an idea of a fix, I'm happy to look at a PR!

As for setting generation parameters, the recommended way is to use the generation config (using the model config is deprecated).

@sgugger Wow I find opening a pull request is quite complicated as there are so many tests. I pushed a simple fix in my forked repo [here](https://github.com/namespace-Pt/transformers/commit/fbcbda33522186a84a073a43fef864eecc0a29f2). Hope that helps :)

@sgugger What is the recommended way of using `Seq2SeqTrainer` with `predict_with_generate=True` and `num_return_sequences>1` in distributed inference setup, for example with `trainer.predict()`?

Currently, I have the following solution.

I am passing a custom function to `preprocess_logits_for_metrics`. Since, I predict with generate, I actually do not get logits but the generated tokens in this function as input. The input shape is `(number_of_samples*num_return_sequences, sequence_length).` which would get truncated. Therefore, I reshape the tensor to shape `(number_of_samples, seq_len * num_return_sequences)`. 

It works but is there is a better way you would recommend?