[Bug + workaround] Keypoints 0.0 are confusing ../transformers/models/detr/image_processing_detr.py
Author: ghost
Pull request: no
Labels: name: Good First Issue

### System Info

- transformers: 4.33.1
- Python: 3.9.18
- Macbook pro M2 chip










### Who can help?

@Narsil




### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Trying to fine-tune Detr on a dataset in coco-format following [this](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb) hugging-face suggested notebook.

At cell [16]:
`from torch.utils.data import DataLoader

def collate_fn(batch):
  pixel_values = [item[0] for item in batch]
  encoding = processor.pad(pixel_values, return_tensors="pt")
  labels = [item[1] for item in batch]
  batch = {}
  batch['pixel_values'] = encoding['pixel_values']
  batch['pixel_mask'] = encoding['pixel_mask']
  batch['labels'] = labels
  return batch

train_dataloader = DataLoader(train_dataset, collate_fn=collate_fn, batch_size=4, shuffle=True)
val_dataloader = DataLoader(val_dataset, collate_fn=collate_fn, batch_size=2)
batch = next(iter(train_dataloader))`

I get this error:
```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
/Users/me/Documents/dev/ai_sb/detection/obj_det_train.ipynb Cell 10 line 1
     13 train_dataloader = DataLoader(train_dataset, collate_fn=collate_fn, batch_size=4, shuffle=True)
     14 # val_dataloader = DataLoader(val_dataset, collate_fn=collate_fn, batch_size=2)
---> 16 batch = next(iter(train_dataloader))

File .../.conda/lib/python3.9/site-packages/torch/utils/data/dataloader.py:630, in _BaseDataLoaderIter.__next__(self)
    627 if self._sampler_iter is None:
    628     # TODO(https://github.com/pytorch/pytorch/issues/76750)
    629     self._reset()  # type: ignore[call-arg]
--> 630 data = self._next_data()
    632 self._num_yielded += 1
    633 if self._dataset_kind == _DatasetKind.Iterable and \
    634         self._IterableDataset_len_called is not None and \
    635         self._num_yielded > self._IterableDataset_len_called:

File .../.conda/lib/python3.9/site-packages/torch/utils/data/dataloader.py:675, in _SingleProcessDataLoaderIter._next_data(self)
    673 def _next_data(self):
    674     index = self._next_index()  # may raise StopIteration
--> 675     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    676     if self._pin_memory:
    677         data = _utils.pin_memory.pin_memory(data, self._pin_memory_device)

File .../.conda/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py:51, in _MapDatasetFetcher.fetch(self, possibly_batched_index)
     49         data = self.dataset.__getitems__(possibly_batched_index)
     50     else:
---> 51         data = [self.dataset[idx] for idx in possibly_batched_index]
     52 else:
     53     data = self.dataset[possibly_batched_index]

File .../.conda/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py:51, in <listcomp>(.0)
     49         data = self.dataset.__getitems__(possibly_batched_index)
     50     else:
---> 51         data = [self.dataset[idx] for idx in possibly_batched_index]
     52 else:
     53     data = self.dataset[possibly_batched_index]

/Users/me/Documents/dev/ai_sb/detection/obj_det_train.ipynb Cell 10 line 2
     20 image_id = self.ids[idx]
     21 annotations = {'image_id': image_id, 'annotations': annotations}
---> 22 encoding = self.image_processor(images=images, annotations=annotations, return_tensors="pt")
     23 pixel_values = encoding["pixel_values"].squeeze()
     24 target = encoding["labels"][0]

File .../.conda/lib/python3.9/site-packages/transformers/image_processing_utils.py:546, in BaseImageProcessor.__call__(self, images, **kwargs)
    544 def __call__(self, images, **kwargs) -> BatchFeature:
    545     """Preprocess an image or a batch of images."""
--> 546     return self.preprocess(images, **kwargs)

File .../.conda/lib/python3.9/site-packages/transformers/models/detr/image_processing_detr.py:1253, in DetrImageProcessor.preprocess(self, images, annotations, return_segmentation_masks, masks_path, do_resize, size, resample, do_rescale, rescale_factor, do_normalize, image_mean, image_std, do_pad, format, return_tensors, data_format, input_data_format, **kwargs)
   1251 prepared_annotations = []
   1252 for image, target in zip(images, annotations):
-> 1253     target = self.prepare_annotation(
   1254         image,
   1255         target,
   1256         format,
   1257         return_segmentation_masks=return_segmentation_masks,
   1258         masks_path=masks_path,
   1259         input_data_format=input_data_format,
   1260     )
   1261     prepared_images.append(image)
   1262     prepared_annotations.append(target)

File .../.conda/lib/python3.9/site-packages/transformers/models/detr/image_processing_detr.py:858, in DetrImageProcessor.prepare_annotation(self, image, target, format, return_segmentation_masks, masks_path, input_data_format)
    856 if format == AnnotionFormat.COCO_DETECTION:
    857     return_segmentation_masks = False if return_segmentation_masks is None else return_segmentation_masks
--> 858     target = prepare_coco_detection_annotation(
    859         image, target, return_segmentation_masks, input_data_format=input_data_format
    860     )
    861 elif format == AnnotionFormat.COCO_PANOPTIC:
    862     return_segmentation_masks = True if return_segmentation_masks is None else return_segmentation_masks

File .../.conda/lib/python3.9/site-packages/transformers/models/detr/image_processing_detr.py:333, in prepare_coco_detection_annotation(image, target, return_segmentation_masks, input_data_format)
    331     num_keypoints = keypoints.shape[0]
    332     keypoints = keypoints.reshape((-1, 3)) if num_keypoints else keypoints
--> 333     new_target["keypoints"] = keypoints[keep]
    338 if return_segmentation_masks:
    339     segmentation_masks = [obj["segmentation"] for obj in annotations]

IndexError: boolean index did not match indexed array along dimension 0; dimension is 294 but corresponding boolean dimension is 1
```

### Expected behavior

Nothing.

**My workaround**

To fix this, I had to edit this file in transformers library:
**.conda/lib/python3.9/site-packages/transformers/models/detr/image_processing_detr.py**

I changed this:
```
# if annotations and "keypoints" in annotations[0]:
    #     keypoints = [obj["keypoints"] for obj in annotations]
    #     print("keypoints", keypoints) #TODO: remove
    #     keypoints = np.asarray(keypoints, dtype=np.float32)
    #     num_keypoints = keypoints.shape[0]
    #     keypoints = keypoints.reshape((-1, 3)) if num_keypoints else keypoints
    #     new_target["keypoints"] = keypoints[keep]
```

To this:

```
if annotations and "keypoints" in annotations[0]:
    keypoints = [obj["keypoints"] for obj in annotations]
    # Apply the keep mask here to filter the relevant annotations
    keypoints = [keypoints[i] for i in range(len(keypoints)) if keep[i]]
    
    # converting the filtered keypoints list to a numpy array and reshape it
    keypoints = np.asarray(keypoints, dtype=np.float32)
    num_keypoints = keypoints.shape[0]
    keypoints = keypoints.reshape((-1, 3)) if num_keypoints else keypoints
    
    new_target["keypoints"] = keypoints # We no longer apply keep mask here
```

**Why?**
To ensure that the filtering applied to the keypoints respects its original structure (number of keypoints per annotation). When you reshape keypoints with keypoints.reshape((-1, 3)), it loses the information about which keypoints belong to which annotation.

Here is what needed to be done (at least in my little hack-ish workaround):

1. Before reshaping the keypoints array, I had to apply the keep mask to retain only the annotations I'm are interested in. Only after this cound I reshape the keypoints array to apply further operations.
2. Then, I applied the keep mask on the keypoints list before converting it into a numpy array and reshaping it. This ensured that I only keep the keypoints corresponding to the bounding boxes that satisfy the condition in the keep mask.

Comments:
Hi @duckheada, thanks for raising this issue! 

Would you like to open a PR with this fix?

cc @rafaelpadilla 

@amyeroberts not really. I don't know enough about the library

OK, I'll open it up to the community if someone wants to add this 👍 

@amyeroberts can I work on this?

@amyeroberts  @ArthurZucker  Can you please review this PR?

I will! Sorry for the delay 

Hi @duckheada and @hackpk ,

Thank you for opening this issue. :) 

I was following the referenced notebook ([here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/DETR/Fine_tuning_DetrForObjectDetection_on_custom_dataset_(balloon).ipynb)), but I could not replicate the issue.

Are you working with a dataset other than the "balloon" dataset? If possible, could you share that specific dataset with me? This would help me better understand and duplicate the error you're experiencing.

Hi @rafaelpadilla, yes I was working a different dataset. No I cannot share it.

Hi @duckheada, 
No problem, I totally understand if you can't share it. :)

This error appears if we use they annotations of the keypoints (e.g. `person_keypoints_val2017.json`), which is not in the example of the referenced notebook. That's why I couldn't replicate it initially. 

Thank you for reporting this issue! Great work! 🎉  The solution in PR #26250