Detr Loss: "IndexError: tensors used as indices must be long, int, byte or bool tensors"
Author: kimborenn
Pull request: no

### System Info

- `transformers` version: 4.35.0
- Platform: Linux-5.15.0-73-generic-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.0+cu121 (False)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.21
- JaxLib version: 0.4.21
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the provided code:

```
import torchvision
import os
import json

class CocoDetection(torchvision.datasets.CocoDetection):
    def __init__(self, img_folder, feature_extractor, train=True):
        ann_file = os.path.join(img_folder, "../../annotations/instances_train2017.json" if train else "../../annotations/instances_val2017.json")
        super(CocoDetection, self).__init__(img_folder, ann_file)
        self.feature_extractor = feature_extractor
        self.id_mapping = {}
        self.categories = json.loads(Path(ann_file).read_text())['categories']

        # Update 'id' values and populate the id_mapping dictionary
        for i, label in enumerate(self.categories):
            original_id = label['id']
            new_id = i + 1
            label['id'] = new_id
            self.id_mapping[original_id] = new_id
    
    def __getitem__(self, idx):
        img, target = super(CocoDetection, self).__getitem__(idx)
        image_id = self.ids[idx]
        target = {'image_id': image_id, 'annotations': target}
        encoding = self.feature_extractor(images=img, annotations=target, return_tensors="pt")

        item = {}
        item["pixel_values"] = encoding["pixel_values"].squeeze() # remove batch dimension
        item["pixel_mask"] = encoding["pixel_mask"]
        
        class_labels_tensor = encoding["labels"][0]["class_labels"]
        encoding["labels"][0]["class_labels"] = torch.tensor([self.id_mapping[original_id.item()] for original_id in class_labels_tensor])
        
        item["labels"] = encoding["labels"][0] # remove batch dimension

        return item['pixel_values'], item['labels']

from transformers import DetrFeatureExtractor

img_folder = "/home/datasets/coco/images"

feature_extractor = DetrFeatureExtractor.from_pretrained("facebook/detr-resnet-50")

train_dataset = CocoDetection(img_folder=f'{img_folder}/train2017', feature_extractor=feature_extractor)
val_dataset = CocoDetection(img_folder=f'{img_folder}/val2017', feature_extractor=feature_extractor, train=False)


def collate_fn(batch):
    pixel_values = [item[0] for item in batch]
    encoding = feature_extractor.pad(pixel_values, return_tensors="pt")
    labels = [item[1] for item in batch]
    batch = {}
    batch['pixel_values'] = encoding['pixel_values']
    batch['pixel_mask'] = encoding['pixel_mask']
    batch['labels'] = labels
    return batch

from transformers import DetrImageProcessor, TrainingArguments, Trainer, AutoModelForObjectDetection
from pathlib import Path
import torch

categories = [i['name'] for i in train_dataset.categories]

id2label = {index: x for index, x in enumerate(categories, start=0)}
label2id = {v: k for k, v in id2label.items()}

checkpoint = "facebook/detr-resnet-50"
image_processor = DetrImageProcessor.from_pretrained(checkpoint)

model = AutoModelForObjectDetection.from_pretrained(
    checkpoint,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)

training_args = TrainingArguments(
    output_dir='.',
    remove_unused_columns=False,
    save_strategy="epoch",
    learning_rate=5e-5,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    num_train_epochs=3,
    logging_steps=200,
    save_total_limit=3,
)


trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=collate_fn,
    train_dataset=train_dataset,
    eval_dataset=val_dataset
)

trainer.train()



```



### Expected behavior

The training process should proceed without errors, and the model should be trained successfully.

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests, since you are using custom code, unless you isolate that this is comming from transformers / the trainer and not he way you process your data, I would recommend you to ask your question on the [forum](https://discuss.huggingface.co/) instead. I'm sure the community will be of help!

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.