Size of tensors mismatch when training object detection model
Author: Rayndell
Pull request: no
Labels: name: bug

### System Info

Unfortunately transformers env fails with the following error: NameError: name 'CompletionCreateParamsStreaming' is not defined. Please find manuel configuration instead:
System: Linux 6.1.0-40-amd64 (Docker)
Python version: 3.12.13
transformers==5.12.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to fine tune a RT DETR v2 detection model on my custom dataset. I am following the notebook provided here: https://github.com/qubvel/transformers-notebooks/blob/main/notebooks/RT_DETR_v2_finetune_on_a_custom_dataset.ipynb, but I am facing some issues regarding a mismatch between tensor sizes during training. Here is the full stack trace:

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Training size: 4
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5.28k/5.28k [00:00<00:00, 5.84MB/s]
model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 172M/172M [00:03<00:00, 51.5MB/s]
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 770/770 [00:00<00:00, 4964.97it/s]
[transformers] RTDetrV2ForObjectDetection LOAD REPORT from: PekingU/rtdetr_v2_r50vd
Key                                                 | Status   |                                                                                         
----------------------------------------------------+----------+-----------------------------------------------------------------------------------------
model.denoising_class_embed.weight                  | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([81, 256]) vs model:torch.Size([71, 256])
model.decoder.class_embed.{0, 1, 2, 3, 4, 5}.weight | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([80, 256]) vs model:torch.Size([70, 256])
model.decoder.class_embed.{0, 1, 2, 3, 4, 5}.bias   | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([80]) vs model:torch.Size([70])          
model.enc_score_head.bias                           | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([80]) vs model:torch.Size([70])          
model.enc_score_head.weight                         | MISMATCH | Reinit due to size mismatch - ckpt: torch.Size([80, 256]) vs model:torch.Size([70, 256])

Notes:
- MISMATCH:     ckpt weights were loaded, but they did not match the original empty weight shapes.
  0%|                                                                                                                                         | 0/100 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/workspace/detection/test_rt_detr.py", line 215, in <module>
    trainer.train()
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1433, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1515, in _inner_training_loop
    self._run_epoch(
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1743, in _run_epoch
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1915, in training_step
    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1987, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 907, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1867, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 907, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1522, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 1036, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/output_capturing.py", line 252, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1196, in forward
    fused_feature_map = torch.concat([top_fpn_feature_map, backbone_feature_map], dim=1)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 122 but got size 121 for tensor number 1 in the list.
  0%|          | 0/100 [00:01<?, ?it/s] 

A strange thing for starters is that it seems that an epoch will be 100 iterations, while my dataset sample is only 4 images...

Below you will find a code snippet to reproduce. Additional files needed to run the code (csv, png, ...) are provided as attachments. Note that my annotations are in PASCAL VOC format ([x1 y1 x2 y2]).

```python
from dataset import get_dataloader
import albumentations as A
from PIL import Image, ImageDraw
import numpy as np
import torch
import pandas as pd
from transformers import AutoImageProcessor, AutoModelForObjectDetection, TrainingArguments, Trainer
from dataclasses import dataclass
from transformers.image_transforms import center_to_corners_format
from torchmetrics.detection.mean_ap import MeanAveragePrecision

def get_classes(file_path):
    classes = ['Background']    # First class is always the background
    with open(file_path, 'r') as f:
        classes += f.read().splitlines()
    return classes

def parse_bool(value):
    return value.strip().lower() in ["true", "1", "yes"]

class ObjectDetectionDataset(Dataset):
    def __init__(self, dataframe, database_path, transform=None):
        # Retrieve unique image paths
        image_paths = list(dataframe['image_path'].unique())
        
        self.image_paths = []
        self.annotations = {}
        for image_path in image_paths:
            row = dataframe[dataframe['image_path'] == image_path].iloc[0]
            # Transform local path into absolute path
            if database_path:
                image_path = os.path.join(database_path, image_path)
                if not os.path.exists(image_path):
                    print(f"Warning: image file {image_path} was not found.")
                    continue
                    
            self.image_paths.append(image_path)
            # Retrieve box annotations
            annotations = ast.literal_eval(row['annotations'])
            self.annotations[image_path] = annotations
            
        self.transform = transform

    def __len__(self):
        return len(self.image_paths)

    def __getitem__(self, idx):
        
        # Retrieve the image path and its annotations
        image_path = self.image_paths[idx]
        annotations = self.annotations[image_path]
        
        image = Image.open(image_path).convert("RGB")

        # We put the image into a TVImage in order to apply the geometrical data augmentation
        image = TVImage(image, dtype=torch.uint8)
        
        height, width = image.shape[-2:]

        # If no annotations, we fill the image's target with zeros for the bbox and labels values
        if len(annotations) == 0:
            empty_boxes = BoundingBoxes(torch.empty((0, 4), dtype=torch.float32), format="XYXY", canvas_size=(height, width))
            empty_labels = torch.empty((0,), dtype=torch.int64)
            target = {"boxes": empty_boxes, "labels": empty_labels}
            if self.transform:
                image, target = self.transform(image, target)
            return image, target

        # We retrieve boxes and labels
        boxes = [annotation['bndbox'] for annotation in annotations]
        labels = [annotation['label'] for annotation in annotations]
        
        for i in range(len(boxes)):
            if boxes[i][0] < 0:
                boxes[i][0] = 0
            if boxes[i][2] > width - 1:
                boxes[i][2] = width - 1
            if boxes[i][1] < 0:
                boxes[i][1] = 0
            if boxes[i][3] > height - 1:
                boxes[i][3] = height - 1
        
        labels_tensor = torch.tensor(labels, dtype=torch.int64)
        boxes_tensor = BoundingBoxes(torch.tensor(boxes, dtype=torch.float32), format="XYXY", canvas_size=(height, width))

        # Create the target dictionary
        target = {"boxes": boxes_tensor, "labels": labels_tensor}
        
        # Apply the image transformations (preprocessing and data augmentation)
        if self.transforms:
            image, target = self.transforms(image, target)
        
        return image, target

class COCODataset(ObjectDetectionDataset):
    """
    Class creating a dataset from a pandas dataframe
    """
    def __init__(self, dataframe, database_path, image_processor, transform=None):
        super().__init__(dataframe, database_path, transform)
        self.image_processor = image_processor

    @staticmethod
    def format_image_annotations_as_coco(idx, labels, boxes):
        """Format one set of image annotations to the COCO format

        Args:
            idx (int): image index, will be transformed into str image_id
            labels (List[int]): list of categories/class labels corresponding to provided bounding boxes
            boxes (List[Tuple[float]]): list of bounding boxes in COCO format
            ([center_x, center_y, width, height] in absolute coordinates)

        Returns:
            dict: {
                "image_id": image id,
                "annotations": list of formatted annotations
            }
        """
        annotations = []
        for label, bbox in zip(labels, boxes):
            formatted_annotation = {
                "image_id": idx,
                "category_id": label,
                "bbox": list(bbox),
                "iscrowd": 0,
                "area": bbox[2] * bbox[3],
            }
            annotations.append(formatted_annotation)

        return {
            "image_id": idx,
            "annotations": annotations,
        }

    def __len__(self):
        return len(self.image_paths)

    def __getitem__(self, idx):
        # Retrieve the image path and its annotations
        image_path = self.image_paths[idx]
        annotations = self.annotations[image_path]

        # Convert image to RGB numpy array
        image = Image.open(image_path)
        image = np.array(image.convert("RGB"))

        # We retrieve boxes and labels
        boxes = []
        labels = []
        for annotation in annotations:
            bbox = annotation['bndbox']
            # Convert bbox values in COCO format
            width = bbox[2] - bbox[0]
            height = bbox[3] - bbox[1]
            center_x = bbox[0]
            center_y = bbox[1]
            boxes.append([center_x, center_y, width, height])
            labels.append(annotation['label'])

        # Apply augmentations
        if self.transform:
            transformed = self.transform(image=image, bboxes=boxes, labels=labels)
            image = transformed["image"]
            boxes = transformed["bboxes"]
            labels = transformed["labels"]

        # Format annotations in COCO format for image_processor
        formatted_annotations = self.format_image_annotations_as_coco(idx, labels, boxes)

        # Apply the image processor transformations: resizing, rescaling, normalization
        result = self.image_processor(
            images=image, annotations=formatted_annotations, return_tensors="pt"
        )

        # Image processor expands batch dimension, lets squeeze it
        result = {k: v[0] for k, v in result.items()}

        return result

def get_dataloader(csv_path, database_path, batch_size, data_transforms=None, shuffle=True, coco_preprocessor=None) -> DataLoader:
    # Get a pandas DataFrame from the CSV file
    dataframe = pd.read_csv(csv_path)
    
    # Create the dataset from it
    if coco_preprocessor:
        if len(dataframe) % batch_size != 0:
            dataframe = dataframe[:batch_size * (len(dataframe) // batch_size)]
        dataset = COCODataset(dataframe, database_path, coco_preprocessor, data_transforms)
    else:
        dataset = ObjectDetectionDataset(dataframe, database_path, data_transforms)
    #dataset = Subset(dataset, range(batch_size*2))
    dataloader = DataLoader(dataset, batch_size, shuffle=shuffle, collate_fn=custom_collate_fn)
    
    return dataloader, dataset

checkpoint = "PekingU/rtdetr_v2_r50vd"
resize_height = 1216
resize_width = 1936

train_csv = "train_sample.csv"
val_csv = "val_sample.csv"
database_path = "."
classes_file = "classes.txt"
output_dir = "output"

train_augmentation_and_transform = A.Compose(
    [
        A.ColorJitter(brightness=(0.7, 1.3), contrast=(0.6, 1.4), p=0.5)
    ],
    bbox_params=A.BboxParams(format="coco", label_fields=["labels"], clip=True, min_area=5, min_width=1, min_height=1),
)

# to make sure boxes are clipped to image size and there is no boxes with area < 1 pixel
validation_transform = A.Compose(
    [A.NoOp()],
    bbox_params=A.BboxParams(format="coco", label_fields=["labels"], clip=True, min_area=1, min_width=1, min_height=1),
)

# Read the class names
classes = get_classes(classes_file)    
num_classes = len(classes)
id2label = {index: x for index, x in enumerate(classes, start=0)}
label2id = {v: k for k, v in id2label.items()}

image_processor = AutoImageProcessor.from_pretrained(
    checkpoint,
    do_resize=True,
    size={"width": resize_width, "height": resize_height},
    backend="torchvision",
)

batch_size = 2
train_dataloader, train_dataset = get_dataloader(train_csv, database_path, batch_size, train_augmentation_and_transform, shuffle=True, coco_preprocessor=image_processor)
val_dataloader, val_dataset = get_dataloader(val_csv, database_path, batch_size, validation_transform, shuffle=False, coco_preprocessor=image_processor)

def collate_fn(batch):
    data = {}
    data["pixel_values"] = torch.stack([x["pixel_values"] for x in batch])
    data["labels"] = [x["labels"] for x in batch]
    return data

@dataclass
class ModelOutput:
    logits: torch.Tensor
    pred_boxes: torch.Tensor

class MAPEvaluator:

    def __init__(self, image_processor, threshold=0.00, id2label=None):
        self.image_processor = image_processor
        self.threshold = threshold
        self.id2label = id2label

    def collect_image_sizes(self, targets):
        """Collect image sizes across the dataset as list of tensors with shape [batch_size, 2]."""
        image_sizes = []
        for batch in targets:
            batch_image_sizes = torch.tensor(np.array([x["size"] for x in batch]))
            image_sizes.append(batch_image_sizes)
        return image_sizes

    def collect_targets(self, targets, image_sizes):
        post_processed_targets = []
        for target_batch, image_size_batch in zip(targets, image_sizes):
            for target, size in zip(target_batch, image_size_batch):

                # here we have "yolo" format (x_center, y_center, width, height) in relative coordinates 0..1
                # and we need to convert it to "pascal" format (x_min, y_min, x_max, y_max) in absolute coordinates
                height, width = size
                boxes = torch.tensor(target["boxes"])
                boxes = center_to_corners_format(boxes)
                boxes = boxes * torch.tensor([[width, height, width, height]])

                labels = torch.tensor(target["class_labels"])
                post_processed_targets.append({"boxes": boxes, "labels": labels})
        return post_processed_targets

    def collect_predictions(self, predictions, image_sizes):
        post_processed_predictions = []
        for batch, target_sizes in zip(predictions, image_sizes):
            batch_logits, batch_boxes = batch[1], batch[2]
            output = ModelOutput(logits=torch.tensor(batch_logits), pred_boxes=torch.tensor(batch_boxes))
            post_processed_output = self.image_processor.post_process_object_detection(
                output, threshold=self.threshold, target_sizes=target_sizes
            )
            post_processed_predictions.extend(post_processed_output)
        return post_processed_predictions

    @torch.no_grad()
    def __call__(self, evaluation_results):

        predictions, targets = evaluation_results.predictions, evaluation_results.label_ids

        image_sizes = self.collect_image_sizes(targets)
        post_processed_targets = self.collect_targets(targets, image_sizes)
        post_processed_predictions = self.collect_predictions(predictions, image_sizes)

        evaluator = MeanAveragePrecision(box_format="xyxy", class_metrics=True)
        evaluator.warn_on_many_detections = False
        evaluator.update(post_processed_predictions, post_processed_targets)

        metrics = evaluator.compute()

        # Replace list of per class metrics with separate metric for each class
        classes = metrics.pop("classes")
        map_per_class = metrics.pop("map_per_class")
        mar_100_per_class = metrics.pop("mar_100_per_class")
        for class_id, class_map, class_mar in zip(classes, map_per_class, mar_100_per_class):
            class_name = id2label[class_id.item()] if id2label is not None else class_id.item()
            metrics[f"map_{class_name}"] = class_map
            metrics[f"mar_100_{class_name}"] = class_mar

        metrics = {k: round(v.item(), 4) for k, v in metrics.items()}

        return metrics

eval_compute_metrics_fn = MAPEvaluator(image_processor=image_processor, threshold=0.01, id2label=id2label)

model = AutoModelForObjectDetection.from_pretrained(
    checkpoint,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)

training_args = TrainingArguments(
    output_dir=output_dir,
    num_train_epochs=int(config["training_params"]["epochs"]),
    max_grad_norm=0.1,
    learning_rate=float(config["training_params"]["learning_rate"]),
    warmup_steps=300,
    per_device_train_batch_size=batch_size,
    dataloader_num_workers=1,
    metric_for_best_model="eval_map",
    greater_is_better=True,
    load_best_model_at_end=True,
    eval_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=2,
    remove_unused_columns=False,
    eval_do_concat_batches=False,
    report_to="tensorboard",  # or "wandb"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    processing_class=image_processor,
    data_collator=collate_fn,
    compute_metrics=eval_compute_metrics_fn,
)

trainer.train()
```

### Expected behavior

I would expect the training to run correctly on the custom dataset

[sample.tar.gz](https://github.com/user-attachments/files/28950403/sample.tar.gz)

Comments:
HI @Rayndell the hybrid encoder('fused_feature_map = torch.concat([top_fpn_feature_map, backbone_feature_map], dim = 1)'), and it comes from your resize width. RT-DETR's backbone has a total stride of 32, and the encoder's FPN upsamples the stride-32 feature map * 2 and concatenates it with the stride-16 lateral(1936 / 16 = 121), your 'width = 1936'  can't be divided by 32 (1936/32 =60.5) and it's become 61 * 2 = 122 mismatch with 121

Thanks, it seems to work now. However I wonder if this information is available somewhere, I wasn't able to find it in any documentation. I am also concerned about these size mismatches warnings (I know some of them are due to a different number of classes), I hope that the model is not training from scratch?

<img width="1076" height="167" alt="Image" src="https://github.com/user-attachments/assets/24a325a9-82fd-4b83-8c74-84ddaee32d32" />

I don't know whether it's written somewhere actually sry, but for the second problem with 'ignore_mismatched_sizes=True', we just load the matched things from pretrained model, and the mismatch one(like your datasets) we pop out a mismatch warning and reintalized it

I was able to make things run, but I ran into another problem. After some epochs it crashes due to a lack of memory:

ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1285, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/queue.py", line 180, in get
    self.not_empty.wait(remaining)
  File "/usr/lib/python3.12/threading.py", line 359, in wait
    gotit = waiter.acquire(True, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 365) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/detection/test_rt_detr.py", line 213, in <module>
    trainer.train()
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1433, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1515, in _inner_training_loop
    self._run_epoch(
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1816, in _run_epoch
    self._maybe_log_save_evaluate(
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2095, in _maybe_log_save_evaluate
    metrics = self._evaluate(trial, ignore_keys_for_eval)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 3016, in _evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2603, in evaluate
    output = self.evaluation_loop(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2718, in evaluation_loop
    for step, inputs in enumerate(dataloader):
                        ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/data_loader.py", line 585, in __iter__
    current_batch = next(dataloader_iter)
                    ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 734, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1492, in _next_data
    idx, data = self._get_data()
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1444, in _get_data
    success, data = self._try_get_data()
                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1298, in _try_get_data
    raise RuntimeError(
RuntimeError: DataLoader worker (pid(s) 365) exited unexpectedly
  2%|▏         | 7607/380350 [25:15<20:37:25,  5.02it/s]

It happened after something like 10 epochs. Can this be a memory leak somewhere? My batch size is already at 1 (it crashes after 2 epochs with a batch size of 2) and I cannot decrease it further.

hi @Rayndell Two things to try:

1. **Confirm the cause:** set `dataloader_num_workers=0` in `TrainingArguments`. If the crash goes away, it's definitely the shared-memory issue (workers no longer use shm). This is slower but a good diagnostic.

2. **Real fix (if Docker):** restart the container with a bigger shared-memory segment, e.g. `docker run --shm-size=1g ...` (or `--ipc=host`). Then you can keep `num_workers > 0` for speed. this seems like not a OOM but maybe you running docker or something that using shared memory

Thank you, it was indeed a shared memory related issue. However I have other troubles now during training, it seems that there is a lack of CUDA memory at some time during the training:

Traceback (most recent call last):                                                                                                    | 44/268 [00:24<02:03,  1.81it/s]
  File "/workspace/detection/test_rt_detr.py", line 213, in <module>
    trainer.train()
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1433, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1515, in _inner_training_loop
    self._run_epoch(
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1816, in _run_epoch
    self._maybe_log_save_evaluate(
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2095, in _maybe_log_save_evaluate
    metrics = self._evaluate(trial, ignore_keys_for_eval)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 3016, in _evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2603, in evaluate
    output = self.evaluation_loop(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2728, in evaluation_loop
    losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2982, in prediction_step
    loss, outputs = self.compute_loss(
                    ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1987, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 907, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1867, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 907, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1514, in forward
    features = self.backbone(pixel_values, pixel_mask)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 810, in forward
    features = self.model(pixel_values).feature_maps
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 907, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/backbone_utils.py", line 172, in wrapper
    output = forward_function(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr/modeling_rt_detr_resnet.py", line 400, in forward
    embedding_output = self.embedder(pixel_values)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr/modeling_rt_detr_resnet.py", line 112, in forward
    embedding = self.embedder(pixel_values)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/container.py", line 244, in forward
    input = module(input)
            ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr/modeling_rt_detr_resnet.py", line 66, in forward
    hidden_states = self.normalization(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 761, in forward
    return x * scale + bias
           ~~~~~~~~~~^~~~~~
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.11 GiB. GPU 0 has a total capacity of 31.47 GiB of which 481.06 MiB is free. Including non-PyTorch memory, this process has 30.98 GiB memory in use. Of the allocated memory 25.30 GiB is allocated by PyTorch, and 5.31 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
  2%|▏         | 2535/126750 [22:59<18:46:50,  1.84it/s] 

This happened after 5 epochs (not right away), I tried to decrease the batch size but it still happens. Is there some memory that is not properly deallocated?

I am closing this issue since I have opened a new one for it. Thanks for your answers.