trainer.evaluate gives different loss values for object detection depending on batch size
Author: erickrf
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-5.15.133+-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, Tesla T4
- Using distributed or parallel set-up in script?: No

### Who can help?

@pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Train or fine-tune an object detection transformer like DeTR.
2. Create new `Trainer` objects with different `per_device_eval_batch_size`
3. Run evaluate with each of them

In more detail:

```
from transformers import DetrImageProcessor, DetrForObjectDetection, TrainingArguments, Trainer
from datasets import load_dataset
import numpy as np

cppe5 = load_dataset("cppe-5")
categories = cppe5['train'].features['objects'].feature['category'].names

id2label = {index: x for index, x in enumerate(categories, start=0)}
label2id = {v: k for k, v in id2label.items()}

model_name = "facebook/detr-resnet-50"
image_processor = DetrImageProcessor.from_pretrained(model_name)
detr = DetrForObjectDetection.from_pretrained(
    model_name,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)

def formatted_anns(image_id, category, area, bbox):
    annotations = []
    
    for i in range(0, len(category)):
        new_ann = {
            "image_id": image_id,
            "category_id": category[i],
            "isCrowd": 0,
            "area": area[i],
            "bbox": list(bbox[i]),
        }
        annotations.append(new_ann)

    return annotations


def transform_aug_ann(examples):
    image_ids = examples["image_id"]
    images, bboxes, area, categories = [], [], [], []
    
    for image, objects in zip(examples["image"], examples["objects"]):
        image = np.array(image.convert("RGB"))[:, :, ::-1]

        area.append(objects["area"])
        images.append(image)
        bboxes.append(objects["bbox"])
        categories.append(objects["category"])

    targets = [
        {"image_id": id_, "annotations": formatted_anns(id_, cat_, ar_, box_)}
        for id_, cat_, ar_, box_ in zip(image_ids, categories, area, bboxes)
    ]

    return image_processor(images=images, annotations=targets, return_tensors="pt")


def collate_fn(batch):
    pixel_values = [item["pixel_values"] for item in batch]
    encoding = image_processor.pad(pixel_values, return_tensors="pt")
    labels = [item["labels"] for item in batch]
    batch = {}
    batch["pixel_values"] = encoding["pixel_values"]
    batch["pixel_mask"] = encoding["pixel_mask"]
    batch["labels"] = labels
    return batch

cppe5["train"] = cppe5["train"].with_transform(transform_aug_ann)

training_args = TrainingArguments(
    output_dir="model/tests",
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    num_train_epochs=1,
    fp16=False,
    save_steps=200,
    logging_steps=200,
    learning_rate=1e-5,
    weight_decay=1e-4,
    save_total_limit=1,
    remove_unused_columns=False,
)
trainer = Trainer(
    model=detr,
    args=training_args,
    data_collator=collate_fn,
    train_dataset=cppe5["train"],
    tokenizer=image_processor,
)
trainer.train()
```

After training, I run 

```
trainer.evaluate(cppe5['test'])

training_args = TrainingArguments(
    output_dir="model/tests",
    per_device_eval_batch_size=3,
    remove_unused_columns=False,
)
trainer = Trainer(
    model=detr,
    args=training_args,
    data_collator=collate_fn,
)

trainer.evaluate(cppe5['test'])
```

And the loss values will differ.

### Expected behavior

The same loss value was expected regardless of the evaluation batch size.

Comments:
cc @NielsRogge maybe padding? 

Seems to be the same problem as #28153 

@erickrf #28153 has now been merged into main. Could you install from source and retry? 

 I managed to install from source, but to do this, I had to:
- upgrade mlflow to 2.10.2
- upgrade torch to 2.20 to avoid having this error `ValueError: prefetch_factor option could only be specified in multiprocessing.let num_workers > 0 to enable multiprocessing.` I was based on the solution of @amyeroberts  [here](https://github.com/huggingface/transformers/issues/29040#issuecomment-1953047354) .
I re did the tests and: 
- with `do_pad=True` ->  change in batch_size will result in **different** loss
- with `do_pad=False` -> change in batch_size will result in **different** loss

@rhajou, Are all the raw images in your pipeline the same size? It shouldn't be possible to pass in batches if not, as tensors can't be created from different sized images. 

How different are the loss values? 

One thing to note, is that for DETR, [it's known for there to be a difference in values for different batch sizes](https://github.com/facebookresearch/detr?tab=readme-ov-file#evaluation), they use batch size 2 for their own benchmark evaluations.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.