nested_concat assumes lists are equal in size
Author: YoraiLevi
Pull request: no

### System Info

I have stumbled upon a quirk while trying to figure out how to calculate custom metrics.

using a detr model for object detection and a provided trainer with a dataset with that has a smaller last batch I am missing labels in the custom metric input.


the length of `batched_labels` in the metric is per the length of the last batch (smaller) and isn't merged like the other fields via the https://github.com/huggingface/transformers/blob/0afa5071bd84e44301750fdc594e33db102cf374/src/transformers/trainer_pt_utils.py#L105 on line  
https://github.com/huggingface/transformers/blob/0afa5071bd84e44301750fdc594e33db102cf374/src/transformers/trainer.py#L3237
for equal sized batches it works fine. (total eval_dataset size is 16, batched automatically into 4's)
![image](https://github.com/huggingface/transformers/assets/50873841/72c1b3ef-8549-4f3f-885a-70e96bc5948a)

![image](https://github.com/huggingface/transformers/assets/50873841/bb077e7a-f52e-4001-9d18-8e00116a367b)


### Who can help?

@muellerz @pacman100

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

setup similar to: https://huggingface.co/docs/transformers/main/en/tasks/object_detection#object-detection

```
def compute_metrics(eval_pred: EvalPrediction): 
    (loss_dict, logits, pred_boxes, last_hidden_state, encoder_last_hidden_state), batched_labels = eval_pred
    # (loss_dict, logits, pred_boxes, last_hidden_state, encoder_last_hidden_state), labels = eval_pred
    outputs = DetrObjectDetectionOutput(#loss_dict=loss_dict,
                                        logits=torch.from_numpy(logits),
                                        pred_boxes=torch.from_numpy(pred_boxes),
                                        last_hidden_state=None,
                                        decoder_hidden_states=None,
                                        )
    number_of_image_ids_in_each_indecies = [batched_label['image_id'].shape for batched_label in batched_labels]
    print(number_of_image_ids_in_each_indecies)
 ```

trainer,
```
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=collate_fn,
    train_dataset=ds_train_augmented,
    eval_dataset=ds_val_augmented,
    tokenizer=image_processor,
    compute_metrics=compute_metrics,
)
```

### Expected behavior

to pad it to the last? to append it to a growing list?
anything but this that retains the data.

Comments:
Similarly,
`len(batched_labels[0]['orig_size'])` is 7 and not 8 or any multiple of 2.
![image](https://github.com/huggingface/transformers/assets/50873841/c25c8e1c-b8ea-40e5-960b-99fab235852f)



It's impossible to know where a bounding box originated from
![image](https://github.com/huggingface/transformers/assets/50873841/8768ad48-f967-4035-81a7-cd21eab67ca6)
![image](https://github.com/huggingface/transformers/assets/50873841/5162c68b-d311-4188-a049-49ab78527a4d)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

hello, @YoraiLevi  did you resolve this issue?

@rhajou i wrote a custom trainer and fixed the above issue locally for myself with my suggestion in my pull request, doubt it works now because there were a lot of changes in the repository since, You can see the custom trainer in my "Intro to deep learning final project" repo however i am pretty sure it's missing something for it to work.