Out of memory after some epochs while training RT-DETR v2 model
Author: Rayndell
Pull request: no
Labels: name: bug

### System Info

Unfortunately transformers env fails with the following error: NameError: name 'CompletionCreateParamsStreaming' is not defined. Please find manuel configuration instead:
System: Linux 6.1.0-40-amd64 (Docker)
Python version: 3.12.13
transformers==5.12.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I am trying to fine tune a RT DETR v2 detection model on my custom dataset. I am following the notebook provided here: https://github.com/qubvel/transformers-notebooks/blob/main/notebooks/RT_DETR_v2_finetune_on_a_custom_dataset.ipynb, but I am facing some issues regarding a lack of memory after several epochs:

{'loss': '39.8', 'grad_norm': '150.6', 'learning_rate': '0.0009995', 'epoch': '0.06573'}                                                                              
{'loss': '46.91', 'grad_norm': '1.044e+04', 'learning_rate': '0.0009982', 'epoch': '0.1315'}                                                                          
{'loss': '49.9', 'grad_norm': '9.306e+04', 'learning_rate': '0.0009968', 'epoch': '0.1972'}                                                                           
{'loss': '48.61', 'grad_norm': '1022', 'learning_rate': '0.0009955', 'epoch': '0.2629'}                                                                               
{'loss': '47.9', 'grad_norm': '1.658e+05', 'learning_rate': '0.0009942', 'epoch': '0.3286'}                                                                           
{'loss': '47.41', 'grad_norm': '1.303e+05', 'learning_rate': '0.0009929', 'epoch': '0.3944'}                                                                          
{'loss': '47.88', 'grad_norm': '2.339e+04', 'learning_rate': '0.0009916', 'epoch': '0.4601'}                                                                          
{'loss': '47.76', 'grad_norm': '1.749e+05', 'learning_rate': '0.0009903', 'epoch': '0.5258'}                                                                          
{'loss': '49.36', 'grad_norm': '5.75e+05', 'learning_rate': '0.000989', 'epoch': '0.5916'}                                                                            
{'loss': '52.91', 'grad_norm': '4.086e+05', 'learning_rate': '0.0009876', 'epoch': '0.6573'}                                                                          
{'loss': '53.59', 'grad_norm': '3.539e+04', 'learning_rate': '0.0009863', 'epoch': '0.723'}                                                                           
{'loss': '49.96', 'grad_norm': '1.536e+04', 'learning_rate': '0.000985', 'epoch': '0.7887'}                                                                           
{'loss': '55.76', 'grad_norm': '221.7', 'learning_rate': '0.0009837', 'epoch': '0.8545'}                                                                              
{'loss': '52.76', 'grad_norm': '4660', 'learning_rate': '0.0009824', 'epoch': '0.9202'}                                                                               
{'loss': '39.24', 'grad_norm': '1.069e+04', 'learning_rate': '0.0009811', 'epoch': '0.9859'}                                                                          
  2%|██▍                                                                                                                     | 7607/380350 [25:11<23:35:13,  4.39it/s]ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm).
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1285, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/queue.py", line 180, in get
    self.not_empty.wait(remaining)
  File "/usr/lib/python3.12/threading.py", line 359, in wait
    gotit = waiter.acquire(True, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 365) is killed by signal: Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/detection/test_rt_detr.py", line 213, in <module>
    trainer.train()
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1433, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1515, in _inner_training_loop
    self._run_epoch(
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1816, in _run_epoch
    self._maybe_log_save_evaluate(
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2095, in _maybe_log_save_evaluate
    metrics = self._evaluate(trial, ignore_keys_for_eval)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 3016, in _evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2603, in evaluate
    output = self.evaluation_loop(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2718, in evaluation_loop
    for step, inputs in enumerate(dataloader):
                        ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/accelerate/data_loader.py", line 585, in __iter__
    current_batch = next(dataloader_iter)
                    ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 734, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1492, in _next_data
    idx, data = self._get_data()
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1444, in _get_data
    success, data = self._try_get_data()
                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/data/dataloader.py", line 1298, in _try_get_data
    raise RuntimeError(
RuntimeError: DataLoader worker (pid(s) 365) exited unexpectedly
  2%|▏         | 7607/380350 [25:15<20:37:25,  5.02it/s]

Here is a code snippet to reproduce:

```python
from dataset import get_dataloader
import albumentations as A
from PIL import Image, ImageDraw
import numpy as np
import torch
import pandas as pd
from transformers import AutoImageProcessor, AutoModelForObjectDetection, TrainingArguments, Trainer
from dataclasses import dataclass
from transformers.image_transforms import center_to_corners_format
from torchmetrics.detection.mean_ap import MeanAveragePrecision

def get_classes(file_path):
    classes = ['Background']    # First class is always the background
    with open(file_path, 'r') as f:
        classes += f.read().splitlines()
    return classes

def parse_bool(value):
    return value.strip().lower() in ["true", "1", "yes"]

class ObjectDetectionDataset(Dataset):
    def __init__(self, dataframe, database_path, transform=None):
        # Retrieve unique image paths
        image_paths = list(dataframe['image_path'].unique())
        
        self.image_paths = []
        self.annotations = {}
        for image_path in image_paths:
            row = dataframe[dataframe['image_path'] == image_path].iloc[0]
            # Transform local path into absolute path
            if database_path:
                image_path = os.path.join(database_path, image_path)
                if not os.path.exists(image_path):
                    print(f"Warning: image file {image_path} was not found.")
                    continue
                    
            self.image_paths.append(image_path)
            # Retrieve box annotations
            annotations = ast.literal_eval(row['annotations'])
            self.annotations[image_path] = annotations
            
        self.transform = transform

    def __len__(self):
        return len(self.image_paths)

    def __getitem__(self, idx):
        
        # Retrieve the image path and its annotations
        image_path = self.image_paths[idx]
        annotations = self.annotations[image_path]
        
        image = Image.open(image_path).convert("RGB")

        # We put the image into a TVImage in order to apply the geometrical data augmentation
        image = TVImage(image, dtype=torch.uint8)
        
        height, width = image.shape[-2:]

        # If no annotations, we fill the image's target with zeros for the bbox and labels values
        if len(annotations) == 0:
            empty_boxes = BoundingBoxes(torch.empty((0, 4), dtype=torch.float32), format="XYXY", canvas_size=(height, width))
            empty_labels = torch.empty((0,), dtype=torch.int64)
            target = {"boxes": empty_boxes, "labels": empty_labels}
            if self.transform:
                image, target = self.transform(image, target)
            return image, target

        # We retrieve boxes and labels
        boxes = [annotation['bndbox'] for annotation in annotations]
        labels = [annotation['label'] for annotation in annotations]
        
        for i in range(len(boxes)):
            if boxes[i][0] < 0:
                boxes[i][0] = 0
            if boxes[i][2] > width - 1:
                boxes[i][2] = width - 1
            if boxes[i][1] < 0:
                boxes[i][1] = 0
            if boxes[i][3] > height - 1:
                boxes[i][3] = height - 1
        
        labels_tensor = torch.tensor(labels, dtype=torch.int64)
        boxes_tensor = BoundingBoxes(torch.tensor(boxes, dtype=torch.float32), format="XYXY", canvas_size=(height, width))

        # Create the target dictionary
        target = {"boxes": boxes_tensor, "labels": labels_tensor}
        
        # Apply the image transformations (preprocessing and data augmentation)
        if self.transforms:
            image, target = self.transforms(image, target)
        
        return image, target

class COCODataset(ObjectDetectionDataset):
    """
    Class creating a dataset from a pandas dataframe
    """
    def __init__(self, dataframe, database_path, image_processor, transform=None):
        super().__init__(dataframe, database_path, transform)
        self.image_processor = image_processor

    @staticmethod
    def format_image_annotations_as_coco(idx, labels, boxes):
        """Format one set of image annotations to the COCO format

        Args:
            idx (int): image index, will be transformed into str image_id
            labels (List[int]): list of categories/class labels corresponding to provided bounding boxes
            boxes (List[Tuple[float]]): list of bounding boxes in COCO format
            ([center_x, center_y, width, height] in absolute coordinates)

        Returns:
            dict: {
                "image_id": image id,
                "annotations": list of formatted annotations
            }
        """
        annotations = []
        for label, bbox in zip(labels, boxes):
            formatted_annotation = {
                "image_id": idx,
                "category_id": label,
                "bbox": list(bbox),
                "iscrowd": 0,
                "area": bbox[2] * bbox[3],
            }
            annotations.append(formatted_annotation)

        return {
            "image_id": idx,
            "annotations": annotations,
        }

    def __len__(self):
        return len(self.image_paths)

    def __getitem__(self, idx):
        # Retrieve the image path and its annotations
        image_path = self.image_paths[idx]
        annotations = self.annotations[image_path]

        # Convert image to RGB numpy array
        image = Image.open(image_path)
        image = np.array(image.convert("RGB"))

        # We retrieve boxes and labels
        boxes = []
        labels = []
        for annotation in annotations:
            bbox = annotation['bndbox']
            # Convert bbox values in COCO format
            width = bbox[2] - bbox[0]
            height = bbox[3] - bbox[1]
            center_x = bbox[0]
            center_y = bbox[1]
            boxes.append([center_x, center_y, width, height])
            labels.append(annotation['label'])

        # Apply augmentations
        if self.transform:
            transformed = self.transform(image=image, bboxes=boxes, labels=labels)
            image = transformed["image"]
            boxes = transformed["bboxes"]
            labels = transformed["labels"]

        # Format annotations in COCO format for image_processor
        formatted_annotations = self.format_image_annotations_as_coco(idx, labels, boxes)

        # Apply the image processor transformations: resizing, rescaling, normalization
        result = self.image_processor(
            images=image, annotations=formatted_annotations, return_tensors="pt"
        )

        # Image processor expands batch dimension, lets squeeze it
        result = {k: v[0] for k, v in result.items()}

        return result

def get_dataloader(csv_path, database_path, batch_size, data_transforms=None, shuffle=True, coco_preprocessor=None) -> DataLoader:
    # Get a pandas DataFrame from the CSV file
    dataframe = pd.read_csv(csv_path)
    
    # Create the dataset from it
    if coco_preprocessor:
        if len(dataframe) % batch_size != 0:
            dataframe = dataframe[:batch_size * (len(dataframe) // batch_size)]
        dataset = COCODataset(dataframe, database_path, coco_preprocessor, data_transforms)
    else:
        dataset = ObjectDetectionDataset(dataframe, database_path, data_transforms)
    #dataset = Subset(dataset, range(batch_size*2))
    dataloader = DataLoader(dataset, batch_size, shuffle=shuffle, collate_fn=custom_collate_fn)
    
    return dataloader, dataset

checkpoint = "PekingU/rtdetr_v2_r50vd"
resize_height = 1216
resize_width = 1920

train_csv = "train.csv"
val_csv = "val.csv"
database_path = "."
classes_file = "classes.txt"
output_dir = "output"

train_augmentation_and_transform = A.Compose(
    [
        A.ColorJitter(brightness=(0.7, 1.3), contrast=(0.6, 1.4), p=0.5)
    ],
    bbox_params=A.BboxParams(format="coco", label_fields=["labels"], clip=True, min_area=5, min_width=1, min_height=1),
)

# to make sure boxes are clipped to image size and there is no boxes with area < 1 pixel
validation_transform = A.Compose(
    [A.NoOp()],
    bbox_params=A.BboxParams(format="coco", label_fields=["labels"], clip=True, min_area=1, min_width=1, min_height=1),
)

# Read the class names
classes = get_classes(classes_file)    
num_classes = len(classes)
id2label = {index: x for index, x in enumerate(classes, start=0)}
label2id = {v: k for k, v in id2label.items()}

image_processor = AutoImageProcessor.from_pretrained(
    checkpoint,
    do_resize=True,
    size={"width": resize_width, "height": resize_height},
    backend="torchvision",
)

batch_size = 1
train_dataloader, train_dataset = get_dataloader(train_csv, database_path, batch_size, train_augmentation_and_transform, shuffle=True, coco_preprocessor=image_processor)
val_dataloader, val_dataset = get_dataloader(val_csv, database_path, batch_size, validation_transform, shuffle=False, coco_preprocessor=image_processor)

def collate_fn(batch):
    data = {}
    data["pixel_values"] = torch.stack([x["pixel_values"] for x in batch])
    data["labels"] = [x["labels"] for x in batch]
    return data

@dataclass
class ModelOutput:
    logits: torch.Tensor
    pred_boxes: torch.Tensor

class MAPEvaluator:

    def __init__(self, image_processor, threshold=0.00, id2label=None):
        self.image_processor = image_processor
        self.threshold = threshold
        self.id2label = id2label

    def collect_image_sizes(self, targets):
        """Collect image sizes across the dataset as list of tensors with shape [batch_size, 2]."""
        image_sizes = []
        for batch in targets:
            batch_image_sizes = torch.tensor(np.array([x["size"] for x in batch]))
            image_sizes.append(batch_image_sizes)
        return image_sizes

    def collect_targets(self, targets, image_sizes):
        post_processed_targets = []
        for target_batch, image_size_batch in zip(targets, image_sizes):
            for target, size in zip(target_batch, image_size_batch):

                # here we have "yolo" format (x_center, y_center, width, height) in relative coordinates 0..1
                # and we need to convert it to "pascal" format (x_min, y_min, x_max, y_max) in absolute coordinates
                height, width = size
                boxes = torch.tensor(target["boxes"])
                boxes = center_to_corners_format(boxes)
                boxes = boxes * torch.tensor([[width, height, width, height]])

                labels = torch.tensor(target["class_labels"])
                post_processed_targets.append({"boxes": boxes, "labels": labels})
        return post_processed_targets

    def collect_predictions(self, predictions, image_sizes):
        post_processed_predictions = []
        for batch, target_sizes in zip(predictions, image_sizes):
            batch_logits, batch_boxes = batch[1], batch[2]
            output = ModelOutput(logits=torch.tensor(batch_logits), pred_boxes=torch.tensor(batch_boxes))
            post_processed_output = self.image_processor.post_process_object_detection(
                output, threshold=self.threshold, target_sizes=target_sizes
            )
            post_processed_predictions.extend(post_processed_output)
        return post_processed_predictions

    @torch.no_grad()
    def __call__(self, evaluation_results):

        predictions, targets = evaluation_results.predictions, evaluation_results.label_ids

        image_sizes = self.collect_image_sizes(targets)
        post_processed_targets = self.collect_targets(targets, image_sizes)
        post_processed_predictions = self.collect_predictions(predictions, image_sizes)

        evaluator = MeanAveragePrecision(box_format="xyxy", class_metrics=True)
        evaluator.warn_on_many_detections = False
        evaluator.update(post_processed_predictions, post_processed_targets)

        metrics = evaluator.compute()

        # Replace list of per class metrics with separate metric for each class
        classes = metrics.pop("classes")
        map_per_class = metrics.pop("map_per_class")
        mar_100_per_class = metrics.pop("mar_100_per_class")
        for class_id, class_map, class_mar in zip(classes, map_per_class, mar_100_per_class):
            class_name = id2label[class_id.item()] if id2label is not None else class_id.item()
            metrics[f"map_{class_name}"] = class_map
            metrics[f"mar_100_{class_name}"] = class_mar

        metrics = {k: round(v.item(), 4) for k, v in metrics.items()}

        return metrics

eval_compute_metrics_fn = MAPEvaluator(image_processor=image_processor, threshold=0.01, id2label=id2label)

model = AutoModelForObjectDetection.from_pretrained(
    checkpoint,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)

training_args = TrainingArguments(
    output_dir=output_dir,
    num_train_epochs=50,
    max_grad_norm=0.1,
    learning_rate=0.001,
    warmup_steps=300,
    per_device_train_batch_size=batch_size,
    dataloader_num_workers=1,
    metric_for_best_model="eval_map",
    greater_is_better=True,
    load_best_model_at_end=True,
    eval_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=2,
    remove_unused_columns=False,
    eval_do_concat_batches=False,
    report_to="tensorboard",  # or "wandb"
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    processing_class=image_processor,
    data_collator=collate_fn,
    compute_metrics=eval_compute_metrics_fn,
)

trainer.train()
```

I have also tried with batch_size=2 and the crash occurs after 2 or 3 epochs. I am suspecting a memory leak since the training does well on a smaller dataset.

### Expected behavior

The training is expected to run without crashing due to lack of memory.

Comments:
Thank you for the in-depth explaantion, it was indeed a shared memory related issue. However I have other troubles now during training, it seems that there is a lack of CUDA memory at some time during the training:

Traceback (most recent call last): | 44/268 [00:24<02:03, 1.81it/s]
File "/workspace/detection/test_rt_detr.py", line 213, in
trainer.train()
File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1433, in train
return inner_training_loop(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1515, in _inner_training_loop
self._run_epoch(
File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1816, in _run_epoch
self._maybe_log_save_evaluate(
File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2095, in _maybe_log_save_evaluate
metrics = self._evaluate(trial, ignore_keys_for_eval)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 3016, in _evaluate
metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2603, in evaluate
output = self.evaluation_loop(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2728, in evaluation_loop
losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 2982, in prediction_step
loss, outputs = self.compute_loss(
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/trainer.py", line 1987, in compute_loss
outputs = model(**inputs)
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 907, in wrapper
output = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1867, in forward
outputs = self.model(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 907, in wrapper
output = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 1514, in forward
features = self.backbone(pixel_values, pixel_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 810, in forward
features = self.model(pixel_values).feature_maps
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py", line 907, in wrapper
output = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/backbone_utils.py", line 172, in wrapper
output = forward_function(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr/modeling_rt_detr_resnet.py", line 400, in forward
embedding_output = self.embedder(pixel_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr/modeling_rt_detr_resnet.py", line 112, in forward
embedding = self.embedder(pixel_values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/container.py", line 244, in forward
input = module(input)
^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr/modeling_rt_detr_resnet.py", line 66, in forward
hidden_states = self.normalization(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1784, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/transformers/models/rt_detr_v2/modeling_rt_detr_v2.py", line 761, in forward
return x * scale + bias
~~~~~~~~~~^~~~~~
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.11 GiB. GPU 0 has a total capacity of 31.47 GiB of which 481.06 MiB is free. Including non-PyTorch memory, this process has 30.98 GiB memory in use. Of the allocated memory 25.30 GiB is allocated by PyTorch, and 5.31 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2%|▏ | 2535/126750 [22:59<18:46:50, 1.84it/s]

This happened after 5 epochs (not right away), I tried to decrease the batch size but it still happens. Is there some memory that is not properly deallocated?

This looks like a GPU memory fragmentation issue rather than a bug in transformers. The OOM is from memory fragmentation caused by the large input resolution (1216x1920). Try PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True and adding torch.cuda.empty_cache() in your compute_metrics function.

Hello, thank you for your answer. However the solution you propose changed nothing, I still have the same memory error.

I also tried the fix proposed by @gmrnlg1971 (detaching tensors) but it did not work either.

Hi @Rayndell, sorry to hear the detach fix did not resolve the OOM for you — high-resolution inputs like 1216×1920 put serious pressure on fragmented GPU memory, so a single change often is not enough. Here are a few targeted strategies that should help:

**1. Reduce `max_num_pages` in the dataset config**
If your dataset config exposes a `max_num_pages` parameter, lowering it directly reduces the maximum sequence length fed to the model at once, which is usually the quickest win for large-resolution inputs.

**2. Enable gradient checkpointing**
This trades compute for memory by recomputing activations during the backward pass rather than storing them:
```python
model.gradient_checkpointing_enable()
```
This can cut activation memory by 30–60% with only a modest throughput penalty.

**3. Free the fragment cache inside the training loop**
CUDA memory fragmentation accumulates quickly at high resolutions. Adding a periodic `empty_cache()` call helps:
```python
for step, batch in enumerate(dataloader):
    loss = model(**batch).loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if step % 50 == 0:
        torch.cuda.empty_cache()
```

**4. Set `PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128`**
This tells PyTorch's caching allocator to avoid holding onto very large contiguous blocks, which is a primary driver of fragmentation at high resolutions. Set it before launching your training script:
```bash
export PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:128
```
or in Python:
```python
import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128"
```

I would recommend trying these in combination — particularly gradient checkpointing + the allocator config — as they address complementary causes of OOM. Feel free to share the full traceback if the issue persists and I'll take a closer look! 🙂

If this could help, it seems that the crash occur at the eval, since setting eval_strategy="no" prevents from crashes. Could this be a bug in the MAPEvaluator given in the notebook? I tried to move all I could to the cpu (by detaching the evaluation and ground truth tensors), but none of it works. I will try what you suggest but it does not seem to be a fragmented memory related problem.

do compute_metrics = None
if crash disappear that means culprit is most likely MAPEvaluator  or some memory accumulation happening during metric computation
if crash still happen culprit is most likely  evaluation pipeline such as : -  trainer evaluation loop, prediction accumulation, Rt-detr output

Yes, the crash disappears when I disable the evaluation, but I have currently no clue on what to change to prevent this.

I already tried setting eval_accumulation_steps=1 and per_device_eval_batch_size=1, without success.

Actually the program seems to crash during the first evaluation process:

<img width="1177" height="56" alt="Image" src="https://github.com/user-attachments/assets/d25c77b3-58c2-4805-a547-b32bafec80d4" />

I'm not sure but this might be another possibility

If it crashes on first evaluation then 
evaluation dataset is too large so this can explode memory immediately during the first evaluation.

you can use small validation subset with this if crash disappearss then that would suggest evaluation memory usage scales with dataset size

if not then then I would start suspecting a bug in the  evaluation pipeline itself or metric code.

Hello,

I have finally solved this problem. It was in fact during the evaluation, where the processed stored all the outputs on the GPU which resulted in a lack of GPU memory. As a workaround, I retained only the outputs with a score > threshold:

```python
def preprocess_logits_for_metrics(outputs, labels):
    probs = logits.softmax(-1)
    scores, labels_idx = probs.max(-1)  # (B, N)

    keep_logits = []
    keep_boxes = []
    threshold = 0.2
    for b in range(scores.shape[0]):
        mask = scores[b] > threshold

        if mask.sum() == 0:
            top1 = scores[b].argmax().unsqueeze(0)
            mask = torch.zeros_like(mask)
            mask[top1] = True

        keep_logits.append(logits[b][mask])
        keep_boxes.append(boxes[b][mask])

    return keep_logits, keep_boxes

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=val_dataset,
    processing_class=image_processor,
    preprocess_logits_for_metrics=preprocess_logits_for_metrics,
    data_collator=collate_fn,
    compute_metrics=eval_compute_metrics_fn,
)
```

I also added these parameters in the TrainingArguments:

```python
remove_unused_columns=False,
eval_do_concat_batches=False,
eval_accumulation_steps=1,
```

However I am still unable to obtain proper results on my dataset: the loss stays big, with big gradients, and it seems that the model isn't learning anything. I tried different strategies: freezing the backbone, reducing the learning rate, trying different optimizers and schedulers, ... Can you guys help me on this? My dataset has large images, which I reduce to 1920x1216 pixels (I cannot reduce it further since some objects are very small). One challenging issue is that on top of that some objects are well represented and are very big and elongated (see example below). I also tried a version of the dataset where these objects are not present but the problem still remain.

<img width="1936" height="1216" alt="Image" src="https://github.com/user-attachments/assets/52d422ed-7353-452e-923c-728bd438eb8c" />

Here is my current configuration:

```python
checkpoint = "PekingU/rtdetr_v2_r50vd"

train_augmentation_and_transform = A.Compose(
    [
        A.ColorJitter(brightness=(0.7, 1.3), contrast=(0.6, 1.4), p=0.5)
    ],
    bbox_params=A.BboxParams(format="coco", label_fields=["labels"], clip=True, min_area=5, min_width=1, min_height=1),
)

training_args = TrainingArguments(
    output_dir=output_dir,
    num_train_epochs=30,
    max_grad_norm=0.1,
    optim="sgd",
    learning_rate=0.0001,
    lr_scheduler_type="cosine",
    warmup_steps=300,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=4,
    dataloader_num_workers=2,
    metric_for_best_model="eval_map",
    greater_is_better=True,
    load_best_model_at_end=False,
    eval_strategy="epoch",
    save_strategy="epoch",
    logging_strategy="steps",
    save_total_limit=3,
    remove_unused_columns=False,
    eval_do_concat_batches=False,
    eval_accumulation_steps=1,
    logging_steps=100,
    fp16=False,
    prediction_loss_only=False,
    report_to="tensorboard",  # or "wandb"
)
```

Am I doing something wrong here? I also carefully double-checked my data and what is being fed to the network during training, everything is fine.