OWL-VIT Finetuning code for custom dataset in Hugging Face
Author: solomonmanuelraj
Pull request: no

### Feature request



Hi team,

when i am trying to finetune owl-vit base 32 model with custom data cppe-5 i am receiving the following error in the time of trainer.train() function.

######################################################################################################################

ValueError Traceback (most recent call last)
Cell In[40], line 11
1 from transformers import Trainer
3 trainer = Trainer(
4 model=lora_model,
5 args=training_args,
(...)
8 tokenizer=processor,
9 )
---> 11 trainer.train()

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
1535 hf_hub_utils.enable_progress_bars()
1536 else:
-> 1537 return inner_training_loop(
1538 args=args,
1539 resume_from_checkpoint=resume_from_checkpoint,
1540 trial=trial,
1541 ignore_keys_for_eval=ignore_keys_for_eval,
1542 )

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
1851 self.control = self.callback_handler.on_step_begin(args, self.state, self.control)
1853 with self.accelerator.accumulate(model):
-> 1854 tr_loss_step = self.training_step(model, inputs)
1856 if (
1857 args.logging_nan_inf_filter
1858 and not is_torch_tpu_available()
1859 and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
1860 ):
1861 # if loss is nan or inf simply add the average of previous logged losses
1862 tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2735, in Trainer.training_step(self, model, inputs)
2732 return loss_mb.reduce_mean().detach().to(self.args.device)
2734 with self.compute_loss_context_manager():
-> 2735 loss = self.compute_loss(model, inputs)
2737 if self.args.n_gpu > 1:
2738 loss = loss.mean() # mean() to average on multi-gpu parallel training

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2776, in Trainer.compute_loss(self, model, inputs, return_outputs)
2774 else:
2775 if isinstance(outputs, dict) and "loss" not in outputs:
-> 2776 raise ValueError(
2777 "The model did not return a loss from the inputs, only the following keys: "
2778 f"{','.join(outputs.keys())}. For reference, the inputs it received are {','.join(inputs.keys())}."
2779 )
2780 # We don't use .loss here since the model may return tuples instead of ModelOutput.
2781 loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]

ValueError: The model did not return a loss from the inputs, only the following keys: logits,pred_boxes,text_embeds,image_embeds,class_embeds,text_model_output,vision_model_output. For reference, the inputs it received are input_ids,attention_mask,pixel_values

#####################################################################################################################

Collate_fn() definition
#######################################################################################################################
def collate_fn(batch):
input_ids = torch.Tensor([item["input_ids"].tolist() for item in batch]).int()
input_ids = input_ids.to(device)
attention_mask = torch.Tensor([item["attention_mask"].tolist() for item in batch]).int()
attention_mask = attention_mask.to(device)
pixel_values = torch.Tensor([item["pixel_values"].tolist() for item in batch])
pixel_values = pixel_values.to(device)

batch = {}
batch["input_ids"] = input_ids
batch["attention_mask"] = attention_mask
batch["pixel_values"] = pixel_values
print(batch)
return batch

####################################################################################################################

i am using cppe-5 dataset from HF for custom training and testing.

let me know your feedback comments.


### Motivation

Fine tuning the owl-vit model in custom dataset using HF Trainer. It will help to PEFT fine tune the model with Lora 

### Your contribution

Will test this feature with the custom dataset

Comments:
cc @younesbelkada 

Hi @solomonmanuelraj 
Thanks for the issue, OwlViT cannot be used out of the box with HF trainer, you need either to subclass it with a new class and overwrite the method that computes the loss or design your own custom training loop.
You can have a look at this notebook from @NielsRogge : https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb and start from it

@younesbelkada  thanks for your quick update. 

Hi @younesbelkada 

Thanks for your help. As per your input, i created a sub class for HF trainer and implemented the custom loss function ( referred https://www.kaggle.com/code/bibhasmondal96/detr-from-scratch & https://github.com/facebookresearch/detr/blob/main/models/matcher.py for the custom loss implementation) when i am running with CPPE dataset i am receiving the following error. I am running this code in a single GPU machine. 

program code and error list attached.

```python
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

from datasets import load_dataset
cppe5 = load_dataset("cppe-5")

categories = cppe5["train"].features["objects"].feature["category"].names
id2label = {index: x for index, x in enumerate(categories, start=0)}
label2id = {v: k for k, v in id2label.items()}

id2label
text_inputs =list(id2label.values())
print(text_inputs)

remove_idx = [590, 821, 822, 875, 876, 878, 879]
keep = [i for i in range(len(cppe5["train"])) if i not in remove_idx]

## considering only 100 training images
keep = keep[0:100]
cppe5["train"] = cppe5["train"].select(keep)

from transformers import AutoImageProcessor
from transformers import AutoProcessor

image_processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50")

checkpoint = "google/owlvit-base-patch32"
processor = AutoProcessor.from_pretrained(checkpoint)

from transformers import AutoModelForZeroShotObjectDetection

model = AutoModelForZeroShotObjectDetection.from_pretrained(
    checkpoint,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)
model.to(device)



import albumentations
import numpy as np

transform = albumentations.Compose(
    [
        albumentations.Resize(480, 480),
        albumentations.HorizontalFlip(p=1.0),
        albumentations.RandomBrightnessContrast(p=1.0),
    ],
    bbox_params=albumentations.BboxParams(format="coco", label_fields=["category"]),
)

def formatted_anns(image_id, category, area, bbox):
    annotations = []
    for i in range(0, len(category)):
        new_ann = {
            "image_id": image_id,
            "category_id": category[i],
            "isCrowd": 0,
            "area": area[i],
            "bbox": list(bbox[i]),
        }
        annotations.append(new_ann)

    return annotations

# transforming a batch
def transform_aug_ann(examples):
    image_ids = examples["image_id"]
    images, bboxes, area, categories = [], [], [], []
    transformed_data = []
    for image, objects in zip(examples["image"], examples["objects"]):
        image = np.array(image.convert("RGB"))[:, :, ::-1]
        out = transform(image=image, bboxes=objects["bbox"], category=objects["category"])

        area.append(objects["area"])
        images.append(out["image"])
        bboxes.append(out["bboxes"])
        categories.append(out["category"])
        transformed_data.append(processor(text=text_inputs, images=image, return_tensors="pt"))

    
    return {"transformed_data":transformed_data}

# transforming a batch
def transform_aug_ann_labels(examples):
    image_ids = examples["image_id"]
    images, bboxes, area, categories = [], [], [], []
    for image, objects in zip(examples["image"], examples["objects"]):
        image = np.array(image.convert("RGB"))[:, :, ::-1]
        out = transform(image=image, bboxes=objects["bbox"], category=objects["category"])

        area.append(objects["area"])
        images.append(out["image"])
        bboxes.append(out["bboxes"])
        categories.append(out["category"])

    targets = [
        {"image_id": id_, "annotations": formatted_anns(id_, cat_, ar_, box_)}
        for id_, cat_, ar_, box_ in zip(image_ids, categories, area, bboxes)
    ]

    return image_processor(images=images, annotations=targets, return_tensors="pt")

transform_1 = cppe5["train"].with_transform(transform_aug_ann)
transform_2 = cppe5["train"].with_transform(transform_aug_ann_labels)

import pandas as pd
from datasets import Dataset

data = []
for i in range(len(transform_1)):
    dict_ = {}
    dict_["input_ids"] = transform_1[i]["transformed_data"]["input_ids"]
    dict_["attention_mask"] = transform_1[i]["transformed_data"]["attention_mask"]
    dict_["pixel_values"] = transform_1[i]["transformed_data"]["pixel_values"][0]
    dict_["labels"] = transform_2[i]["labels"]
    data.append(dict_)

# Preprocessed Training Data
train_dataset = Dataset.from_list(data)
train_dataset.features

# Using Detr-Loss calculation https://github.com/facebookresearch/detr/blob/main/models/matcher.py
# https://www.kaggle.com/code/bibhasmondal96/detr-from-scratch

class BoxUtils(object):
    @staticmethod
    def box_cxcywh_to_xyxy(x):
        x_c, y_c, w, h = x.unbind(-1)
        b = [(x_c - 0.5 * w), (y_c - 0.5 * h),
             (x_c + 0.5 * w), (y_c + 0.5 * h)]
        return torch.stack(b, dim=-1)

    @staticmethod
    def box_xyxy_to_cxcywh(x):
        x0, y0, x1, y1 = x.unbind(-1)
        b = [(x0 + x1) / 2, (y0 + y1) / 2,
             (x1 - x0), (y1 - y0)]
        return torch.stack(b, dim=-1)

    @staticmethod
    def rescale_bboxes(out_bbox, size):
        img_h, img_w = size
        b = BoxUtils.box_cxcywh_to_xyxy(out_bbox)
        b = b * torch.tensor([img_w, img_h, img_w, img_h], dtype=torch.float32)
        return b

    @staticmethod
    def box_area(boxes):
        """
        Computes the area of a set of bounding boxes, which are specified by its
        (x1, y1, x2, y2) coordinates.
        Arguments:
            boxes (Tensor[N, 4]): boxes for which the area will be computed. They
                are expected to be in (x1, y1, x2, y2) format
        Returns:
            area (Tensor[N]): area for each box
        """
        return (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1])
        
    @staticmethod
    # modified from torchvision to also return the union
    def box_iou(boxes1, boxes2):
        area1 = BoxUtils.box_area(boxes1)
        area2 = BoxUtils.box_area(boxes2)

        lt = torch.max(boxes1[:, None, :2], boxes2[:, :2])  # [N,M,2]
        rb = torch.min(boxes1[:, None, 2:], boxes2[:, 2:])  # [N,M,2]

        wh = (rb - lt).clamp(min=0)  # [N,M,2]
        inter = wh[:, :, 0] * wh[:, :, 1]  # [N,M]

        union = area1[:, None] + area2 - inter

        iou = inter / union
        return iou, union

    @staticmethod
    def generalized_box_iou(boxes1, boxes2):
        """
        Generalized IoU from https://giou.stanford.edu/
        The boxes should be in [x0, y0, x1, y1] format
        Returns a [N, M] pairwise matrix, where N = len(boxes1)
        and M = len(boxes2)
        """
        # degenerate boxes gives inf / nan results
        # so do an early check
        assert (boxes1[:, 2:] >= boxes1[:, :2]).all()
        assert (boxes2[:, 2:] >= boxes2[:, :2]).all()
        iou, union = BoxUtils.box_iou(boxes1, boxes2)

        lt = torch.min(boxes1[:, None, :2], boxes2[:, :2])
        rb = torch.max(boxes1[:, None, 2:], boxes2[:, 2:])

        wh = (rb - lt).clamp(min=0)  # [N,M,2]
        area = wh[:, :, 0] * wh[:, :, 1]

        return iou - (area - union) / area
```

```
import torch
import torch.nn as nn
import torch.nn.functional as F

from scipy.optimize import linear_sum_assignment

class HungarianMatcher(nn.Module):
    """This class computes an assignment between the targets and the predictions of the network
    For efficiency reasons, the targets don't include the no_object. Because of this, in general,
    there are more predictions than targets. In this case, we do a 1-to-1 matching of the best predictions,
    while the others are un-matched (and thus treated as non-objects).
    """

    def __init__(self, cost_class: float = 1, cost_bbox: float = 1, cost_giou: float = 1):
        """Creates the matcher
        Params:
            cost_class: This is the relative weight of the classification error in the matching cost
            cost_bbox: This is the relative weight of the L1 error of the bounding box coordinates in the matching cost
            cost_giou: This is the relative weight of the giou loss of the bounding box in the matching cost
        """
        super().__init__()
        self.cost_class = cost_class
        self.cost_bbox = cost_bbox
        self.cost_giou = cost_giou
        assert cost_class != 0 or cost_bbox != 0 or cost_giou != 0, "all costs cant be 0"

    @torch.no_grad()
    def forward(self, outputs, targets):
        """ Performs the matching
        Params:
            outputs: This is a dict that contains at least these entries:
                 "pred_logits": Tensor of dim [batch_size, num_queries, num_classes] with the classification logits
                 "pred_boxes": Tensor of dim [batch_size, num_queries, 4] with the predicted box coordinates
            targets: This is a list of targets (len(targets) = batch_size), where each target is a dict containing:
                 "labels": Tensor of dim [num_target_boxes] (where num_target_boxes is the number of ground-truth
                           objects in the target) containing the class labels
                 "boxes": Tensor of dim [num_target_boxes, 4] containing the target box coordinates
        Returns:
            A list of size batch_size, containing tuples of (index_i, index_j) where:
                - index_i is the indices of the selected predictions (in order)
                - index_j is the indices of the corresponding selected targets (in order)
            For each batch element, it holds:
                len(index_i) = len(index_j) = min(num_queries, num_target_boxes)
        """
        print(outputs.keys())
        bs, num_queries = outputs["logits"].shape[:2]

        # We flatten to compute the cost matrices in a batch
        out_prob = outputs["logits"].flatten(0, 1).softmax(-1)  # [batch_size * num_queries, num_classes]
        out_bbox = outputs["pred_boxes"].flatten(0, 1)  # [batch_size * num_queries, 4]

        # Also concat the target labels and boxes
        tgt_ids = torch.cat([v["class_labels"] for v in targets])
        print("Index ",type(tgt_ids))
        print(tgt_ids)
        tgt_ids = tgt_ids.int()
        print("Index ",type(tgt_ids))
        print(tgt_ids)

        tgt_bbox = torch.cat([v["boxes"] for v in targets])

        # Compute the classification cost. Contrary to the loss, we don't use the NLL,
        # but approximate it in 1 - proba[target class].
        # The 1 is a constant that doesn't change the matching, it can be ommitted.
        cost_class = -out_prob[:, tgt_ids]

        # Compute the L1 cost between boxes
        cost_bbox = torch.cdist(out_bbox, tgt_bbox, p=1)

        # Compute the giou cost betwen boxes
        cost_giou = -BoxUtils.generalized_box_iou(
            BoxUtils.box_cxcywh_to_xyxy(out_bbox),
            BoxUtils.box_cxcywh_to_xyxy(tgt_bbox)
        )

        # Final cost matrix
        C = self.cost_bbox * cost_bbox + self.cost_class * cost_class + self.cost_giou * cost_giou
        C = C.view(bs, num_queries, -1).cpu()

        sizes = [len(v["boxes"]) for v in targets]
        indices = [linear_sum_assignment(c[i]) for i, c in enumerate(C.split(sizes, -1))]
        return [(torch.as_tensor(i, dtype=torch.int64), torch.as_tensor(j, dtype=torch.int64)) for i, j in indices]

class SetCriterion(nn.Module):
    """ This class computes the loss for DETR.
    The process happens in two steps:
        1) we compute hungarian assignment between ground truth boxes and the outputs of the model
        2) we supervise each pair of matched ground-truth / prediction (supervise class and box)
    """
    def __init__(self, num_classes, matcher, weight_dict, eos_coef, losses):
        """ Create the criterion.
        Parameters:
            num_classes: number of object categories, omitting the special no-object category
            matcher: module able to compute a matching between targets and proposals
            weight_dict: dict containing as key the names of the losses and as values their relative weight.
            eos_coef: relative classification weight applied to the no-object category
            losses: list of all the losses to be applied. See get_loss for list of available losses.
        """
        super().__init__()
        self.num_classes = num_classes
        self.matcher = matcher
        self.weight_dict = weight_dict
        self.eos_coef = eos_coef
        self.losses = losses
        empty_weight = torch.ones(self.num_classes + 1)
        empty_weight[-1] = self.eos_coef
        self.register_buffer('empty_weight', empty_weight)

    def loss_labels(self, outputs, targets, indices, num_boxes):
        """Classification loss (NLL)
        targets dicts must contain the key "labels" containing a tensor of dim [nb_target_boxes]
        """
        print("loss_labels",outputs.keys())
        assert 'logits' in outputs
        src_logits = outputs['logits']

        idx = self._get_src_permutation_idx(indices)
        target_classes_o = torch.cat([t["class_labels"][J] for t, (_, J) in zip(targets, indices)]).to(torch.int64)
        target_classes = torch.full(src_logits.shape[:2], self.num_classes,
                                    dtype=torch.int64, device=src_logits.device).to(torch.int64)
        target_classes[idx] = target_classes_o

        loss_ce = F.cross_entropy(src_logits.transpose(1, 2), target_classes, self.empty_weight)
        losses = {'loss_ce': loss_ce}
        return losses

    @torch.no_grad()
    def loss_cardinality(self, outputs, targets, indices, num_boxes):
        """ Compute the cardinality error, ie the absolute error in the number of predicted non-empty boxes
        This is not really a loss, it is intended for logging purposes only. It doesn't propagate gradients
        """
        pred_logits = outputs['logits']
        device = pred_logits.device
        tgt_lengths = torch.as_tensor([len(v["class_labels"]) for v in targets], device=device)
        # Count the number of predictions that are NOT "no-object" (which is the last class)
        card_pred = (pred_logits.argmax(-1) != pred_logits.shape[-1] - 1).sum(1)
        card_err = F.l1_loss(card_pred.float(), tgt_lengths.float())
        losses = {'cardinality_error': card_err}
        return losses

    def loss_boxes(self, outputs, targets, indices, num_boxes):
        """Compute the losses related to the bounding boxes, the L1 regression loss and the GIoU loss
           targets dicts must contain the key "boxes" containing a tensor of dim [nb_target_boxes, 4]
           The target boxes are expected in format (center_x, center_y, w, h), normalized by the image size.
        """
        assert 'pred_boxes' in outputs
        idx = self._get_src_permutation_idx(indices)
        src_boxes = outputs['pred_boxes'][idx]
        target_boxes = torch.cat([t['boxes'][i] for t, (_, i) in zip(targets, indices)], dim=0)

        loss_bbox = F.l1_loss(src_boxes, target_boxes, reduction='none')

        losses = {}
        losses['loss_bbox'] = loss_bbox.sum() / num_boxes

        loss_giou = 1 - torch.diag(BoxUtils.generalized_box_iou(
            BoxUtils.box_cxcywh_to_xyxy(src_boxes),
            BoxUtils.box_cxcywh_to_xyxy(target_boxes))
        )
        losses['loss_giou'] = loss_giou.sum() / num_boxes
        return losses

    def _get_src_permutation_idx(self, indices):
        # permute predictions following indices
        batch_idx = torch.cat([torch.full_like(src, i) for i, (src, _) in enumerate(indices)])
        src_idx = torch.cat([src for (src, _) in indices])
        return batch_idx, src_idx

    def _get_tgt_permutation_idx(self, indices):
        # permute targets following indices
        batch_idx = torch.cat([torch.full_like(tgt, i) for i, (_, tgt) in enumerate(indices)])
        tgt_idx = torch.cat([tgt for (_, tgt) in indices])
        return batch_idx, tgt_idx

    def get_loss(self, loss, outputs, targets, indices, num_boxes, **kwargs):
        loss_map = {
            'labels': self.loss_labels,
            'cardinality': self.loss_cardinality,
            'boxes': self.loss_boxes,
        }
        assert loss in loss_map, f'do you really want to compute {loss} loss?'
        return loss_map[loss](outputs, targets, indices, num_boxes, **kwargs)

    def forward(self, outputs, targets):
        """ This performs the loss computation.
        Parameters:
             outputs: dict of tensors, see the output specification of the model for the format
             targets: list of dicts, such that len(targets) == batch_size.
                      The expected keys in each dict depends on the losses applied, see each loss' doc
        """
        print("output type")
        print(type(outputs))
        print("target type")
        print(targets)
        outputs_without_aux = {k: v for k, v in outputs.items() if k != 'aux_outputs'}

        # Retrieve the matching between the outputs of the last layer and the targets
        indices = self.matcher(outputs_without_aux, targets)

        # Compute the average number of target boxes accross all nodes, for normalization purposes
        num_boxes = sum(len(t["class_labels"]) for t in targets)
        num_boxes = torch.as_tensor([num_boxes], dtype=torch.float, device=next(iter(outputs.values())).device)

        # Compute all the requested losses
        losses = {}
        for loss in self.losses:
            losses.update(self.get_loss(loss, outputs, targets, indices, num_boxes))

        return losses
    
def collate_fn(batch):
    #input_ids = torch.Tensor([item["input_ids"].tolist() for item in batch]).int()
    input_ids = torch.Tensor([item["input_ids"] for item in batch]).int()
    input_ids = input_ids.to(device)
    # attention_mask = torch.Tensor([item["attention_mask"].tolist() for item in batch]).int()
    attention_mask = torch.Tensor([item["attention_mask"] for item in batch]).int()
    attention_mask = attention_mask.to(device)
    # pixel_values = torch.Tensor([item["pixel_values"].tolist() for item in batch])
    pixel_values = torch.Tensor([item["pixel_values"] for item in batch])
    pixel_values = pixel_values.to(device)
    labels = []
    for item in batch:
        for (key, value) in item["labels"].items():
            item["labels"][key] = torch.Tensor(value).to(device)
        labels.append(item["labels"])
     
    batch = {}
    batch["input_ids"] = input_ids
    batch["attention_mask"] = attention_mask
    batch["pixel_values"] = pixel_values
    batch["labels"] = labels
    #print(batch)
    return batch



from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="owlvit-base-patch32_FT_cppe5",
    per_device_train_batch_size=1,
    num_train_epochs=2,
    fp16=True,
    save_steps=200,
    logging_steps=50,
    learning_rate=1e-5,
    weight_decay=1e-4,
    save_total_limit=2,
    remove_unused_columns=False,
    push_to_hub=False,
    dataloader_pin_memory=False,
    gradient_accumulation_steps=1
)

# custom loss
def custom_loss(logits, labels):
    num_classes = 4
    matcher = HungarianMatcher(cost_class = 1, cost_bbox = 5, cost_giou = 2)
    weight_dict = {'loss_ce': 1, 'loss_bbox': 5, 'loss_giou': 2}
    losses = ['labels', 'boxes', 'cardinality']
    criterion = SetCriterion(num_classes, matcher=matcher, weight_dict=weight_dict, eos_coef=0.1, losses=losses)
    criterion.to(device)
    print("logits",type(logits))
    print("labels",type(labels))
    loss = criterion(logits, labels)
    return loss

# subclass trainer
from transformers import Trainer

class CustomTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")

        inputs["input_ids"] = inputs["input_ids"][0]
        inputs["attention_mask"] = inputs["attention_mask"][0]
        print(inputs["attention_mask"].shape)
        outputs = model(**inputs, return_dict=True)

        print(outputs.keys())
        print(outputs.logits)
        print(labels)
        print("before custom loss calling")
        loss = custom_loss(outputs, labels)
        print("after custom loss calling")
        return (loss, outputs) if return_outputs else loss

# use new trainer
trainer = CustomTrainer(
    model=model,
    args=training_args,
    data_collator=collate_fn,
    train_dataset=train_dataset,
    tokenizer=processor
)

trainer.train()
```


**Error List**

```
TypeError                                 Traceback (most recent call last)
Cell In[25], [line 1](vscode-notebook-cell:?execution_count=25&line=1)
----> [1](vscode-notebook-cell:?execution_count=25&line=1) trainer.train()

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537), in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   [1535](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1535)         hf_hub_utils.enable_progress_bars()
   [1536](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1536) else:
-> [1537](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537)     return inner_training_loop(
   [1538](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1538)         args=args,
   [1539](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1539)         resume_from_checkpoint=resume_from_checkpoint,
   [1540](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1540)         trial=trial,
   [1541](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1541)         ignore_keys_for_eval=ignore_keys_for_eval,
   [1542](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1542)     )

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854), in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   [1851](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1851)     self.control = self.callback_handler.on_step_begin(args, self.state, self.control)
   [1853](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1853) with self.accelerator.accumulate(model):
-> [1854](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854)     tr_loss_step = self.training_step(model, inputs)
   [1856](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1856) if (
   [1857](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1857)     args.logging_nan_inf_filter
   [1858](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1858)     and not is_torch_tpu_available()
   [1859](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1859)     and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
   [1860](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1860) ):
   [1861](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1861)     # if loss is nan or inf simply add the average of previous logged losses
   [1862](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1862)     tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2744](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2744), in Trainer.training_step(self, model, inputs)
   [2742](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2742)         scaled_loss.backward()
   [2743](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2743) else:
-> [2744](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2744)     self.accelerator.backward(loss)
   [2746](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2746) return loss.detach() / self.args.gradient_accumulation_steps

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1901](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1901), in Accelerator.backward(self, loss, **kwargs)
   [1899](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1899)     print("self.gradient_accumulation_steps :- ",self.gradient_accumulation_steps)
   [1900](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1900)     print("self.gradient_accumulation_steps type:- ",type(self.gradient_accumulation_steps))
-> [1901](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1901)     loss = loss / self.gradient_accumulation_steps
   [1902](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1902) if self.distributed_type == DistributedType.DEEPSPEED:
   [1903](https://vscode-remote+ssh-002dremote-002bsolomon-002ddesktop.vscode-resource.vscode-cdn.net/home/lfo2kor/foundation_models/lora/vision_models/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1903)     self.deepspeed_engine_wrapped.backward(loss, **kwargs)

TypeError: unsupported operand type(s) for /: 'dict' and 'int'


#######################################################################################

$ python /home/lfo2kor/foundation_models/lora/vision_models/owl_vit_cus_trainer.py
cuda
['Coverall', 'Face_Shield', 'Gloves', 'Goggles', 'Mask']
Could not find image processor class in the image processor config or the model config. Loading based on pattern matching with the model's feature extractor configuration.
The `max_size` parameter is deprecated and will be removed in v4.26. Please specify in `size['longest_edge'] instead`.
  0%|                                                                   | 0/200 [00:00<?, ?it/s]torch.Size([5, 16])
odict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
tensor([[[ -8.9141,  -8.8359,  -8.8750,  -8.7734,  -9.2188],
         [ -9.8984,  -9.9844,  -9.4922,  -9.6719, -10.5625],
         [-10.1406, -10.2344,  -9.9375,  -9.8516, -10.8438],
         ...,
         [-11.0312, -10.8125,  -9.4922, -10.1797, -11.2500],
         [-10.9844, -10.6875, -10.7109,  -9.8047, -10.7266],
         [-16.1094, -15.5938, -15.7812, -15.3125, -15.8438]]], device='cuda:0',
       grad_fn=<ToCopyBackward0>)
[{'area': tensor([32777.7773, 21233.3320, 21000.0000], device='cuda:0'), 'boxes': tensor([[0.4872, 0.2912, 0.0625, 0.1105],
        [0.4328, 0.7111, 0.0525, 0.0852],
        [0.5979, 0.3900, 0.0450, 0.0983]], device='cuda:0'), 'class_labels': tensor([4., 2., 2.], device='cuda:0'), 'image_id': tensor([429.], device='cuda:0'), 'iscrowd': tensor([0., 0., 0.], device='cuda:0'), 'orig_size': tensor([480., 480.], device='cuda:0'), 'size': tensor([800., 800.], device='cuda:0')}]
before custom loss calling
logits <class 'transformers.models.owlvit.modeling_owlvit.OwlViTObjectDetectionOutput'>
labels <class 'list'>
output type
<class 'transformers.models.owlvit.modeling_owlvit.OwlViTObjectDetectionOutput'>
target type
[{'area': tensor([32777.7773, 21233.3320, 21000.0000], device='cuda:0'), 'boxes': tensor([[0.4872, 0.2912, 0.0625, 0.1105],
        [0.4328, 0.7111, 0.0525, 0.0852],
        [0.5979, 0.3900, 0.0450, 0.0983]], device='cuda:0'), 'class_labels': tensor([4., 2., 2.], device='cuda:0'), 'image_id': tensor([429.], device='cuda:0'), 'iscrowd': tensor([0., 0., 0.], device='cuda:0'), 'orig_size': tensor([480., 480.], device='cuda:0'), 'size': tensor([800., 800.], device='cuda:0')}]
dict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
Index  <class 'torch.Tensor'>
tensor([4., 2., 2.], device='cuda:0')
Index  <class 'torch.Tensor'>
tensor([4, 2, 2], device='cuda:0', dtype=torch.int32)
loss_labels odict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
cuda:0
torch.Size([1, 5, 576])
cuda:0
torch.Size([1, 576])
weight status ----------
cuda:0
tensor([1.0000, 1.0000, 1.0000, 1.0000, 0.1000], device='cuda:0')
cuda:0
<class 'torch.Tensor'>
torch.Size([5])
reduction status ----
mean
ignore_index status -----
-100
label_smoothing status -------
0.0
after custom loss calling
loss value :- {'loss_ce': tensor(2.0748, device='cuda:0', grad_fn=<NllLoss2DBackward0>), 'loss_bbox': tensor([0.0386], device='cuda:0', grad_fn=<DivBackward0>), 'loss_giou': tensor([0.4936], device='cuda:0', grad_fn=<DivBackward0>), 'cardinality_error': tensor(563., device='cuda:0')}
loss type :- <class 'dict'>
self.gradient_accumulation_steps :-  1
self.gradient_accumulation_steps type:-  <class 'int'>
Traceback (most recent call last):
  File "/home/lfo2kor/foundation_models/lora/vision_models/owl_vit_cus_trainer.py", line 499, in <module>
    trainer.train()
  File "/home/lfo2kor/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py", line 1537, in train
    return inner_training_loop(
  File "/home/lfo2kor/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py", line 1854, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/lfo2kor/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py", line 2744, in training_step
    self.accelerator.backward(loss)
  File "/home/lfo2kor/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py", line 1901, in backward
    loss = loss / self.gradient_accumulation_steps
TypeError: unsupported operand type(s) for /: 'dict' and 'int'
  0%|          | 0/200 [00:02<?, ?it/s] 
```

Thanks a lot for sharing the snippet @solomonmanuelraj ! 🙏 
I think to fix that you need to return a tensor instead of a dictionary for the loss, can you try to return a dict by somehow aggregating all inidivual losses from your loss dict and return that instead ? 

Hi @younesbelkada 
    yes. changed the code to return the loss tensor (loss = sum(loss.values()) instead of a loss dictionary. Now it is working fine. thanks for your quick feedback.

Thanks very much @solomonmanuelraj !

Hi @younesbelkada ,

Lora is working fine for Owl-Vit. now i am applying for Qlora for OWL-ViT finetuning. Mainly following this notebook ( https://github.com/peremartra/Large-Language-Model-Notebooks-Course/blob/main/5-Fine%20Tuning/QLoRA_Tuning_PEFT.ipynb) I have given the 8 bit config 

bnb_config = BitsAndBytesConfig(
    load_in_8bit=True,
    bnb_8bit_use_double_quant=True,
    bnb_8bit_quant_type="nf4",
    bnb_8bit_compute_dtype=torch.bfloat16,
)

device_map = {"": 0}
foundation_model = AutoModelForZeroShotObjectDetection.from_pretrained(model_name,                                                                  
                    quantization_config=bnb_config,torch_dtype=torch.bfloat16,
                    device_map=device_map)

i receive the following error. Need your help. Attached the error trace.

###########################################################################################
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[25], [line 1](vscode-notebook-cell:?execution_count=25&line=1)
----> [1](vscode-notebook-cell:?execution_count=25&line=1) trainer.train()

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537), in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   [1535](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1535)         hf_hub_utils.enable_progress_bars()
   [1536](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1536) else:
-> [1537](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537)     return inner_training_loop(
   [1538](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1538)         args=args,
   [1539](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1539)         resume_from_checkpoint=resume_from_checkpoint,
   [1540](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1540)         trial=trial,
   [1541](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1541)         ignore_keys_for_eval=ignore_keys_for_eval,
   [1542](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1542)     )

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854), in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   [1851](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1851)     self.control = self.callback_handler.on_step_begin(args, self.state, self.control)
   [1853](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1853) with self.accelerator.accumulate(model):
-> [1854](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854)     tr_loss_step = self.training_step(model, inputs)
   [1856](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1856) if (
   [1857](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1857)     args.logging_nan_inf_filter
   [1858](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1858)     and not is_torch_tpu_available()
   [1859](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1859)     and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
   [1860](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1860) ):
   [1861](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1861)     # if loss is nan or inf simply add the average of previous logged losses
   [1862](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1862)     tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2744](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2744), in Trainer.training_step(self, model, inputs)
   [2742](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2742)         scaled_loss.backward()
   [2743](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2743) else:
-> [2744](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2744)     self.accelerator.backward(loss)
   [2746](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2746) return loss.detach() / self.args.gradient_accumulation_steps

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1907](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1907), in Accelerator.backward(self, loss, **kwargs)
   [1905](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1905)     return
   [1906](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1906) elif self.scaler is not None:
-> [1907](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1907)     self.scaler.scale(loss).backward(**kwargs)
   [1908](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1908) else:
   [1909](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1909)     loss.backward(**kwargs)

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:492](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:492), in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    [482](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:482) if has_torch_function_unary(self):
    [483](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:483)     return handle_torch_function(
    [484](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:484)         Tensor.backward,
    [485](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:485)         (self,),
   (...)
    [490](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:490)         inputs=inputs,
    [491](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:491)     )
--> [492](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:492) torch.autograd.backward(
    [493](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:493)     self, gradient, retain_graph, create_graph, inputs=inputs
    [494](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:494) )

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:251](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:251), in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    [246](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:246)     retain_graph = create_graph
    [248](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:248) # The reason we repeat the same comment below is that
    [249](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:249) # some Python versions print out the first line of a multi-line function
    [250](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:250) # calls in the traceback and some print out the last line
--> [251](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:251) Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    [252](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:252)     tensors,
    [253](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:253)     grad_tensors_,
    [254](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:254)     retain_graph,
    [255](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:255)     create_graph,
    [256](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:256)     inputs,
    [257](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:257)     allow_unreachable=True,
    [258](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:258)     accumulate_grad=True,
    [259](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:259) )

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:288](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:288), in BackwardCFunction.apply(self, *args)
    [282](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:282)     raise RuntimeError(
    [283](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:283)         "Implementing both 'backward' and 'vjp' for a custom "
    [284](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:284)         "Function is not allowed. You should only implement one "
    [285](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:285)         "of them."
    [286](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:286)     )
    [287](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:287) user_fn = vjp_fn if vjp_fn is not Function.vjp else backward_fn
--> [288](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:288) return user_fn(self, *args)

File [~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:491](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:491), in MatMul8bitLt.backward(ctx, grad_output)
    [485](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:485)     print("state.CxB",state.CxB)
    [486](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:486)     print("State ",state)
    [488](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:488)     CB = (
    [489](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:489)         undo_layout(state.CxB, state.tile_indices)
    [490](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:490)         .to(ctx.dtype_A)
--> [491](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:491)         .mul_(state.SCB.unsqueeze(1).mul(1.0 / 127.0))
    [492](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:492)     )
    [493](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:493)     grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A)
    [494](...//foundation_models/lora/vision_models/owlvit-base-patch32_PeftFT_cppe5/~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:494) else:

RuntimeError: The size of tensor a (32) must match the size of tensor b (4) at non-singleton dimension 0

##############################################################################################
Hi @younesbelkada ,

when i am running the above qlora code with below 4 bits bnbconfig , the fine tuning is taking place without any error. 

bnb_config = BitsAndBytesConfig(
     load_in_4bit=True,
     bnb_4bit_use_double_quant=True,
     bnb_4bit_quant_type="nf4",
     bnb_4bit_compute_dtype=torch.bfloat16
)

once i change the config with below information 

bnb_config = BitsAndBytesConfig(
    load_in_8bit=True
    # bnb_8bit_use_double_quant=True,
    # bnb_8bit_quant_type="int8",
    # bnb_8bit_compute_dtype=torch.bfloat16,
)

i receive the following error trace. 

#########################################################################################
RuntimeError                              Traceback (most recent call last)
Cell In[25], line 1
----> 1 trainer.train()

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1535         hf_hub_utils.enable_progress_bars()
   1536 else:
-> 1537     return inner_training_loop(
   1538         args=args,
   1539         resume_from_checkpoint=resume_from_checkpoint,
   1540         trial=trial,
   1541         ignore_keys_for_eval=ignore_keys_for_eval,
   1542     )

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   1851     self.control = self.callback_handler.on_step_begin(args, self.state, self.control)
   1853 with self.accelerator.accumulate(model):
-> 1854     tr_loss_step = self.training_step(model, inputs)
   1856 if (
   1857     args.logging_nan_inf_filter
   1858     and not is_torch_tpu_available()
   1859     and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
   1860 ):
   1861     # if loss is nan or inf simply add the average of previous logged losses
   1862     tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2744, in Trainer.training_step(self, model, inputs)
   2742         scaled_loss.backward()
   2743 else:
-> 2744     self.accelerator.backward(loss)
   2746 return loss.detach() / self.args.gradient_accumulation_steps

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1907, in Accelerator.backward(self, loss, **kwargs)
   1905     return
   1906 elif self.scaler is not None:
-> 1907     self.scaler.scale(loss).backward(**kwargs)
   1908 else:
   1909     loss.backward(**kwargs)

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/_tensor.py:492, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    482 if has_torch_function_unary(self):
    483     return handle_torch_function(
    484         Tensor.backward,
    485         (self,),
   (...)
    490         inputs=inputs,
    491     )
--> 492 torch.autograd.backward(
    493     self, gradient, retain_graph, create_graph, inputs=inputs
    494 )

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/__init__.py:251, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    246     retain_graph = create_graph
    248 # The reason we repeat the same comment below is that
    249 # some Python versions print out the first line of a multi-line function
    250 # calls in the traceback and some print out the last line
--> 251 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    252     tensors,
    253     grad_tensors_,
    254     retain_graph,
    255     create_graph,
    256     inputs,
    257     allow_unreachable=True,
    258     accumulate_grad=True,
    259 )

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:288, in BackwardCFunction.apply(self, *args)
    282     raise RuntimeError(
    283         "Implementing both 'backward' and 'vjp' for a custom "
    284         "Function is not allowed. You should only implement one "
    285         "of them."
    286     )
    287 user_fn = vjp_fn if vjp_fn is not Function.vjp else backward_fn
--> 288 return user_fn(self, *args)

File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:491, in MatMul8bitLt.backward(ctx, grad_output)
    485     print("state.CxB",state.CxB)
    486     print("State ",state)
    488     CB = (
    489         undo_layout(state.CxB, state.tile_indices)
    490         .to(ctx.dtype_A)
--> 491         .mul_(state.SCB.unsqueeze(1).mul(1.0 / 127.0))
    492     )
    493     grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A)
    494 else:

RuntimeError: The size of tensor a (32) must match the size of tensor b (4) at non-singleton dimension 0

#######################################################################################

your comments will be useful. 

thanks.





thanks.

Hi @younesbelkada,

one more doubt. Once we complete 8bit quantization and fintuning using qlora after that is it possible to export the quantized model in the form of 8 bits onnx file? 

like to do 8 bit quantization aware lora and export the model in onnx format to run in another 8 bit device. 

thanks  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@solomonmanuelraj: Could you please share the Colab notebook to fine-tune Owl-ViT2 using LoRA?

Hi @younesbelkada, I just created a [Colab notebook](https://github.com/NSTiwari/Owl-ViT/blob/main/fine_tune_owl_vit2.ipynb) following the snippet shared by @solomonmanuelraj.

Do you know what's going wrong?

`torch.Size([5, 16])
odict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
tensor([[[ -8.9531,  -8.8750,  -8.9062,  -8.7969,  -9.2500],
         [ -9.9141, -10.0156,  -9.5234,  -9.7031, -10.5938],
         [-10.1797, -10.2656,  -9.9766,  -9.8906, -10.8750],
         ...,
         [-11.0703, -10.8516,  -9.5781, -10.2109, -11.2969],
         [-10.9609, -10.6953, -10.7109,  -9.8281, -10.7266],
         [-16.1406, -15.6406, -15.8203, -15.3438, -15.8906]]], device='cuda:0',
       grad_fn=<ToCopyBackward0>)
[{'area': tensor([32777.7773, 21233.3320, 21000.0000], device='cuda:0'), 'boxes': tensor([[0.4872, 0.2912, 0.0625, 0.1105],
        [0.4328, 0.7111, 0.0525, 0.0852],
        [0.5979, 0.3900, 0.0450, 0.0983]], device='cuda:0'), 'class_labels': tensor([4., 2., 2.], device='cuda:0'), 'image_id': tensor([429.], device='cuda:0'), 'iscrowd': tensor([0., 0., 0.], device='cuda:0'), 'orig_size': tensor([480., 480.], device='cuda:0'), 'size': tensor([800., 800.], device='cuda:0')}]
before custom loss calling
logits <class 'transformers.models.owlvit.modeling_owlvit.OwlViTObjectDetectionOutput'>
labels <class 'list'>
output type
<class 'transformers.models.owlvit.modeling_owlvit.OwlViTObjectDetectionOutput'>
target type
[{'area': tensor([32777.7773, 21233.3320, 21000.0000], device='cuda:0'), 'boxes': tensor([[0.4872, 0.2912, 0.0625, 0.1105],
        [0.4328, 0.7111, 0.0525, 0.0852],
        [0.5979, 0.3900, 0.0450, 0.0983]], device='cuda:0'), 'class_labels': tensor([4., 2., 2.], device='cuda:0'), 'image_id': tensor([429.], device='cuda:0'), 'iscrowd': tensor([0., 0., 0.], device='cuda:0'), 'orig_size': tensor([480., 480.], device='cuda:0'), 'size': tensor([800., 800.], device='cuda:0')}]
dict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
Index  <class 'torch.Tensor'>
tensor([4., 2., 2.], device='cuda:0')
Index  <class 'torch.Tensor'>
tensor([4, 2, 2], device='cuda:0', dtype=torch.int32)
loss_labels odict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
after custom loss calling
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-32-3435b262f1ae>](https://localhost:8080/#) in <cell line: 1>()
----> 1 trainer.train()

1 frames
[/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   1969                     tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)
   1970                 else:
-> 1971                     tr_loss += tr_loss_step
   1972 
   1973                 self.current_flos += float(self.floating_point_ops(inputs))

RuntimeError: output with shape [] doesn't match the broadcast shape [1]`

> @younesbelkada thanks for your quick update.



> ### Feature request
> Hi team,
> 
> when i am trying to finetune owl-vit base 32 model with custom data cppe-5 i am receiving the following error in the time of trainer.train() function.
> 
> ######################################################################################################################
> 
> ValueError Traceback (most recent call last) Cell In[40], line 11 1 from transformers import Trainer 3 trainer = Trainer( 4 model=lora_model, 5 args=training_args, (...) 8 tokenizer=processor, 9 ) ---> 11 trainer.train()
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs) 1535 hf_hub_utils.enable_progress_bars() 1536 else: -> 1537 return inner_training_loop( 1538 args=args, 1539 resume_from_checkpoint=resume_from_checkpoint, 1540 trial=trial, 1541 ignore_keys_for_eval=ignore_keys_for_eval, 1542 )
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval) 1851 self.control = self.callback_handler.on_step_begin(args, self.state, self.control) 1853 with self.accelerator.accumulate(model): -> 1854 tr_loss_step = self.training_step(model, inputs) 1856 if ( 1857 args.logging_nan_inf_filter 1858 and not is_torch_tpu_available() 1859 and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step)) 1860 ): 1861 # if loss is nan or inf simply add the average of previous logged losses 1862 tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2735, in Trainer.training_step(self, model, inputs) 2732 return loss_mb.reduce_mean().detach().to(self.args.device) 2734 with self.compute_loss_context_manager(): -> 2735 loss = self.compute_loss(model, inputs) 2737 if self.args.n_gpu > 1: 2738 loss = loss.mean() # mean() to average on multi-gpu parallel training
> 
> File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2776, in Trainer.compute_loss(self, model, inputs, return_outputs) 2774 else: 2775 if isinstance(outputs, dict) and "loss" not in outputs: -> 2776 raise ValueError( 2777 "The model did not return a loss from the inputs, only the following keys: " 2778 f"{','.join(outputs.keys())}. For reference, the inputs it received are {','.join(inputs.keys())}." 2779 ) 2780 # We don't use .loss here since the model may return tuples instead of ModelOutput. 2781 loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]
> 
> ValueError: The model did not return a loss from the inputs, only the following keys: logits,pred_boxes,text_embeds,image_embeds,class_embeds,text_model_output,vision_model_output. For reference, the inputs it received are input_ids,attention_mask,pixel_values
> 
> #####################################################################################################################
> 
> Collate_fn() definition ####################################################################################################################### def collate_fn(batch): input_ids = torch.Tensor([item["input_ids"].tolist() for item in batch]).int() input_ids = input_ids.to(device) attention_mask = torch.Tensor([item["attention_mask"].tolist() for item in batch]).int() attention_mask = attention_mask.to(device) pixel_values = torch.Tensor([item["pixel_values"].tolist() for item in batch]) pixel_values = pixel_values.to(device)
> 
> batch = {} batch["input_ids"] = input_ids batch["attention_mask"] = attention_mask batch["pixel_values"] = pixel_values print(batch) return batch
> 
> ####################################################################################################################
> 
> i am using cppe-5 dataset from HF for custom training and testing.
> 
> let me know your feedback comments.
> 
> ### Motivation
> Fine tuning the owl-vit model in custom dataset using HF Trainer. It will help to PEFT fine tune the model with Lora
> 
> ### Your contribution
> Will test this feature with the custom dataset

@solomonmanuelraj  Followed your project and created custom loss for multilabels while finetuning owlvit . The loss is too much. Could you please let me know if it was correct. Here is my code https://github.com/rathianandk/binsense

> cc @younesbelkada

Followed your owlvit project and created custom loss for multilabels while finetuning owlvit . The loss is too much. Could you please let me know if it was correct. Here is my code https://github.com/rathianandk/binsense

@rathianandk: Was just running the Colab you shared, the code crashes at line `transformed_data.append(processor(text=examples["text_input"], images=image,return_tensors="pt"))` in `transform_aug_ann` function.

![image](https://github.com/huggingface/transformers/assets/37960279/f9b8790f-cf2e-4bb7-aaa7-55521a7328de)

Please  take latest now added padding=True, truncation=True,

On Mon, May 6, 2024 at 8:04 AM Nitin Tiwari ***@***.***>
wrote:

> @rathianandk <https://github.com/rathianandk>: Was just running the Colab
> you shared, the code crashes at line transformed_data.append(processor(text=examples["text_input"],
> images=image,return_tensors="pt")) in transform_aug_ann function.
>
> image.png (view on web)
> <https://github.com/huggingface/transformers/assets/37960279/f9b8790f-cf2e-4bb7-aaa7-55521a7328de>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/28778#issuecomment-2096251563>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFVFB4JU57WPJEGXHJV4E7DZA6LXZAVCNFSM6AAAAABCRQDP6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJWGI2TCNJWGM>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Please  take latest now added padding=True, truncation=True, and changed
dataset

On Mon, May 6, 2024 at 10:53 AM R A ***@***.***> wrote:

> Please  take latest now added padding=True, truncation=True,
>
> On Mon, May 6, 2024 at 8:04 AM Nitin Tiwari ***@***.***>
> wrote:
>
>> @rathianandk <https://github.com/rathianandk>: Was just running the
>> Colab you shared, the code crashes at line transformed_data.append(processor(text=examples["text_input"],
>> images=image,return_tensors="pt")) in transform_aug_ann function.
>>
>> image.png (view on web)
>> <https://github.com/huggingface/transformers/assets/37960279/f9b8790f-cf2e-4bb7-aaa7-55521a7328de>
>>
>> —
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/transformers/issues/28778#issuecomment-2096251563>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/AFVFB4JU57WPJEGXHJV4E7DZA6LXZAVCNFSM6AAAAABCRQDP6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJWGI2TCNJWGM>
>> .
>> You are receiving this because you were mentioned.Message ID:
>> ***@***.***>
>>
>


Following

Thanks for the update.

On Wed, May 15, 2024 at 12:13 PM Guyda ***@***.***> wrote:

> Following
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/28778#issuecomment-2113283841>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFVFB4PJDPQ4XSDWJJ5GOW3ZCOXUXAVCNFSM6AAAAABCRQDP6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJTGI4DGOBUGE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Hi I wanted to generate custom dataset for googleowl model.  Can anyone help me?

@solomonmanuelraj, after converting the dict to sum, I am getting error
 

> "ValueError: Calculated loss must be on the original device: cuda:0 but device in use is cpu"

Where is it calculating the loss on cpu exactly? I checked all the loss tensor and send them to device:0. 

Whole error:
```
torch.Size([5, 16])
odict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
tensor([[[ -8.9453,  -8.8750,  -8.9062,  -8.7969,  -9.2422],
         [ -9.9062, -10.0156,  -9.5156,  -9.6953, -10.5859],
         [-10.1953, -10.2734,  -9.9766,  -9.8984, -10.8828],
         ...,
         [-11.0625, -10.8438,  -9.5781, -10.2109, -11.2969],
         [-10.9609, -10.6953, -10.7109,  -9.8281, -10.7266],
         [-16.1406, -15.6406, -15.8203, -15.3438, -15.8906]]], device='cuda:0',
       grad_fn=<ToCopyBackward0>)
[{'area': tensor([32777.7773, 21233.3320, 21000.0000], device='cuda:0'), 'boxes': tensor([[0.4872, 0.2912, 0.0625, 0.1105],
        [0.4328, 0.7111, 0.0525, 0.0852],
        [0.5979, 0.3900, 0.0450, 0.0983]], device='cuda:0'), 'class_labels': tensor([4., 2., 2.], device='cuda:0'), 'image_id': tensor([429.], device='cuda:0'), 'iscrowd': tensor([0., 0., 0.], device='cuda:0'), 'orig_size': tensor([480., 480.], device='cuda:0'), 'size': tensor([800., 800.], device='cuda:0')}]
before custom loss calling
logits <class 'transformers.models.owlvit.modeling_owlvit.OwlViTObjectDetectionOutput'>
labels <class 'list'>
output type
<class 'transformers.models.owlvit.modeling_owlvit.OwlViTObjectDetectionOutput'>
target type
[{'area': tensor([32777.7773, 21233.3320, 21000.0000], device='cuda:0'), 'boxes': tensor([[0.4872, 0.2912, 0.0625, 0.1105],
        [0.4328, 0.7111, 0.0525, 0.0852],
        [0.5979, 0.3900, 0.0450, 0.0983]], device='cuda:0'), 'class_labels': tensor([4., 2., 2.], device='cuda:0'), 'image_id': tensor([429.], device='cuda:0'), 'iscrowd': tensor([0., 0., 0.], device='cuda:0'), 'orig_size': tensor([480., 480.], device='cuda:0'), 'size': tensor([800., 800.], device='cuda:0')}]
dict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
Index  <class 'torch.Tensor'>
tensor([4., 2., 2.], device='cuda:0')
Index  <class 'torch.Tensor'>
tensor([4, 2, 2], device='cuda:0', dtype=torch.int32)
loss_labels odict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
In custom loss after loss
loss_tensor returned
after custom loss calling

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[15], line 317
    308 # use new trainer
    309 trainer = CustomTrainer(
    310     model=model,
    311     args=training_args,
   (...)
    314     tokenizer=processor
    315 )
--> 317 trainer.train()

File ~/anaconda3/envs/lvis/lib/python3.11/site-packages/transformers/trainer.py:1885, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1883         hf_hub_utils.enable_progress_bars()
   1884 else:
-> 1885     return inner_training_loop(
   1886         args=args,
   1887         resume_from_checkpoint=resume_from_checkpoint,
   1888         trial=trial,
   1889         ignore_keys_for_eval=ignore_keys_for_eval,
   1890     )

File ~/anaconda3/envs/lvis/lib/python3.11/site-packages/transformers/trainer.py:2227, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   2225 else:
   2226     if tr_loss.device != tr_loss_step.device:
-> 2227         raise ValueError(
   2228             f"Calculated loss must be on the original device: {tr_loss.device} but device in use is {tr_loss_step.device}"
   2229         )
   2230     tr_loss += tr_loss_step
   2232 self.current_flos += float(self.floating_point_ops(inputs))

ValueError: Calculated loss must be on the original device: cuda:0 but device in use is cpu
```

> Hi @younesbelkada, I just created a [Colab notebook](https://github.com/NSTiwari/Owl-ViT/blob/main/fine_tune_owl_vit2.ipynb) following the snippet shared by @solomonmanuelraj.
> 
> Do you know what's going wrong?
> ## `torch.Size([5, 16])
> odict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
> tensor([[[ -8.9531, -8.8750, -8.9062, -8.7969, -9.2500],
> [ -9.9141, -10.0156, -9.5234, -9.7031, -10.5938],
> [-10.1797, -10.2656, -9.9766, -9.8906, -10.8750],
> ...,
> [-11.0703, -10.8516, -9.5781, -10.2109, -11.2969],
> [-10.9609, -10.6953, -10.7109, -9.8281, -10.7266],
> [-16.1406, -15.6406, -15.8203, -15.3438, -15.8906]]], device='cuda:0',
> grad_fn=)
> [{'area': tensor([32777.7773, 21233.3320, 21000.0000], device='cuda:0'), 'boxes': tensor([[0.4872, 0.2912, 0.0625, 0.1105],
> [0.4328, 0.7111, 0.0525, 0.0852],
> [0.5979, 0.3900, 0.0450, 0.0983]], device='cuda:0'), 'class_labels': tensor([4., 2., 2.], device='cuda:0'), 'image_id': tensor([429.], device='cuda:0'), 'iscrowd': tensor([0., 0., 0.], device='cuda:0'), 'orig_size': tensor([480., 480.], device='cuda:0'), 'size': tensor([800., 800.], device='cuda:0')}]
> before custom loss calling
> logits <class 'transformers.models.owlvit.modeling_owlvit.OwlViTObjectDetectionOutput'>
> labels <class 'list'>
> output type
> <class 'transformers.models.owlvit.modeling_owlvit.OwlViTObjectDetectionOutput'>
> target type
> [{'area': tensor([32777.7773, 21233.3320, 21000.0000], device='cuda:0'), 'boxes': tensor([[0.4872, 0.2912, 0.0625, 0.1105],
> [0.4328, 0.7111, 0.0525, 0.0852],
> [0.5979, 0.3900, 0.0450, 0.0983]], device='cuda:0'), 'class_labels': tensor([4., 2., 2.], device='cuda:0'), 'image_id': tensor([429.], device='cuda:0'), 'iscrowd': tensor([0., 0., 0.], device='cuda:0'), 'orig_size': tensor([480., 480.], device='cuda:0'), 'size': tensor([800., 800.], device='cuda:0')}]
> dict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
> Index <class 'torch.Tensor'>
> tensor([4., 2., 2.], device='cuda:0')
> Index <class 'torch.Tensor'>
> tensor([4, 2, 2], device='cuda:0', dtype=torch.int32)
> loss_labels odict_keys(['logits', 'pred_boxes', 'text_embeds', 'image_embeds', 'class_embeds', 'text_model_output', 'vision_model_output'])
> after custom loss calling
> 
> RuntimeError Traceback (most recent call last) in <cell line: 1>() ----> 1 trainer.train()
> 
> 1 frames [/usr/local/lib/python3.10/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval) 1969 tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged) 1970 else: -> 1971 tr_loss += tr_loss_step 1972 1973 self.current_flos += float(self.floating_point_ops(inputs))
> 
> RuntimeError: output with shape [] doesn't match the broadcast shape [1]`

Hi @rathianandk  were you able to solve the Issue? If yes, can you explain how