Repository: Lightning-AI/torchmetrics

Issue:
MeanAveragePrecision devices mismatch
Version: TorchMetrics 0.7.3

I have noticed a bug in MeanAveragePrecision() class.
How to reproduce:
The code in pytorch-lightning module is: 
```py
__init__(...):
  self.mAP = MeanAveragePrecision()

validation_step(...):
  self.mAP(preds, targets)  # preds and targets have the same device as pl-module.

validation_epoch_end(...):
  mAP_dict = self.mAP.compute()
  self.log_dict(mAP_dict)
  
```
I get a device mismatch error at line 715 in map.py, when I run my code on a GPU.

I dug deeper and found the problem:
On line 781 is the following code:
```
       # move everything to CPU, as we are faster here
        self.detection_boxes = [box.cpu() for box in self.detection_boxes]
        self.detection_labels = [label.cpu() for label in self.detection_labels]
        self.detection_scores = [score.cpu() for score in self.detection_scores]
        self.groundtruth_boxes = [box.cpu() for box in self.groundtruth_boxes]
        self.groundtruth_labels = [label.cpu() for label in self.groundtruth_labels]
```
This means, that all gt boxes and detections are stored in the CPU, but in the function _evaluate_image(...), when we have no predictions but have GT we call function __evaluate_image_preds_no_gt(...) (line 470 in map.py).
In this function we have:
``` 
       return {
            "dtMatches": torch.zeros((nb_iou_thrs, nb_det), dtype=torch.bool, device=self.device),
            "gtMatches": torch.zeros((nb_iou_thrs, nb_gt), dtype=torch.bool, device=self.device),
            "dtScores": scores_sorted,
            "gtIgnore": gt_ignore,
            "dtIgnore": det_ignore,
        }
```
The problem is the part where we return device=self.device (we moved everything to CPU before).

My questions are:
Have I discovered a bug or do I have an error in my code?
Why has nobody else before encountered this problem, since this bug makes this metric unusable?

Thank you for your answers. 

Edit: I forked the report and edited functions __evaluate_image_gt_no_preds and __evaluate_image_preds_no_gt by replacing device=self.device by device=torch.device('cpu') and so far my code is working.
I think this is a critical bug, which leaves the Class unusable. 
Bugfix/map device placement
## What does this PR do?

Fixes #943

## Before submitting

- [x] Was this **discussed/approved** via a Github issue? (no need for typos and docs improvements)
- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/metrics/blob/master/.github/CONTRIBUTING.md), Pull Request section?
- [ ] Did you make sure to **update the docs**?
- [ ] Did you write any new **necessary tests**?

## PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

## Did you have fun?

Make sure you had fun coding 🙃



Discussion:
Hi! thanks for your contribution!, great first issue!
Hi @kuntiik,
I have trouble reproducing the error you get. If I run the simple MAP example:
https://github.com/PyTorchLightning/metrics/blob/master/tm_examples/detection_map.py
where I move everything to `cuda` I do not get any errors.

Could you try updating to master, just to make sure we have not solved the issue between 0.7.3 and now?

Hi,
you tested it on a simple example, which does not catch the error. Try to do the following. Move MeanAveragePrecision to Cuda. Then do update twice: In one of the updates, there has to be a prediction and a ground truth. In the update delete ground truth (pass empty) then try to compute the metrics. I will try to post minimal example asap.
This test fails with the following error:

>       det_matches = torch.cat([e["dtMatches"][:, :max_det] for e in img_eval_cls_bbox], axis=1)[:, inds]
E       RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument tensors in method wrapper___cat)

```{python}
def test_minimal_metrics():
    from torchmetrics.detection import MeanAveragePrecision
    map = MeanAveragePrecision().to(torch.device('cuda:0'))
    preds = [
        dict(
            boxes=torch.Tensor([[258.0, 41.0, 606.0, 285.0]]),
            scores=torch.Tensor([0.536]),
            labels=torch.IntTensor([0]),
        )
    ]
    target = [
        dict(
            boxes=torch.Tensor([[214.0, 41.0, 562.0, 285.0]]),
            labels=torch.IntTensor([0]),
        )
    ]
    map.update(preds, target)
    target = [
        dict(
            boxes=torch.Tensor([]),
            labels=torch.IntTensor([]),
        )
    ]
    map.update(preds, target)
    result = map.compute()
    print(result)

```
Hi @kuntiik,
Thanks for the example that you provided, it does indeed lead to device errors. Going to fix immediately. 


Resolving patch:
diff --git a/torchmetrics/detection/mean_ap.py b/torchmetrics/detection/mean_ap.py
--- a/torchmetrics/detection/mean_ap.py
+++ b/torchmetrics/detection/mean_ap.py
@@ -182,15 +182,15 @@ class MeanAveragePrecision(Metric):
         >>> from torchmetrics.detection.mean_ap import MeanAveragePrecision
         >>> preds = [
         ...   dict(
-        ...     boxes=torch.Tensor([[258.0, 41.0, 606.0, 285.0]]),
-        ...     scores=torch.Tensor([0.536]),
-        ...     labels=torch.IntTensor([0]),
+        ...     boxes=torch.tensor([[258.0, 41.0, 606.0, 285.0]]),
+        ...     scores=torch.tensor([0.536]),
+        ...     labels=torch.tensor([0]),
         ...   )
         ... ]
         >>> target = [
         ...   dict(
-        ...     boxes=torch.Tensor([[214.0, 41.0, 562.0, 285.0]]),
-        ...     labels=torch.IntTensor([0]),
+        ...     boxes=torch.tensor([[214.0, 41.0, 562.0, 285.0]]),
+        ...     labels=torch.tensor([0]),
         ...   )
         ... ]
         >>> metric = MeanAveragePrecision()
@@ -505,11 +505,11 @@ def _evaluate_image(
             det_ignore, torch.logical_and(det_matches == 0, torch.repeat_interleave(ar, nb_iou_thrs, 0))
         )
         return {
-            "dtMatches": det_matches,
-            "gtMatches": gt_matches,
-            "dtScores": scores_sorted,
-            "gtIgnore": gt_ignore,
-            "dtIgnore": det_ignore,
+            "dtMatches": det_matches.to(self.device),
+            "gtMatches": gt_matches.to(self.device),
+            "dtScores": scores_sorted.to(self.device),
+            "gtIgnore": gt_ignore.to(self.device),
+            "dtIgnore": det_ignore.to(self.device),
         }
 
     @staticmethod
@@ -585,7 +585,7 @@ def _summarize(
             else:
                 prec = prec[:, :, area_inds, mdet_inds]
 
-        mean_prec = Tensor([-1]) if len(prec[prec > -1]) == 0 else torch.mean(prec[prec > -1])
+        mean_prec = torch.tensor([-1.0]) if len(prec[prec > -1]) == 0 else torch.mean(prec[prec > -1])
         return mean_prec
 
     def _calculate(self, class_ids: List) -> Tuple[MAPMetricResults, MARMetricResults]:
@@ -623,7 +623,7 @@ def _calculate(self, class_ids: List) -> Tuple[MAPMetricResults, MARMetricResult
         scores = -torch.ones((nb_iou_thrs, nb_rec_thrs, nb_classes, nb_bbox_areas, nb_max_det_thrs))
 
         # move tensors if necessary
-        rec_thresholds_tensor = Tensor(self.rec_thresholds)
+        rec_thresholds_tensor = torch.tensor(self.rec_thresholds)
 
         # retrieve E at each category, area range, and max number of detections
         for idx_cls, _ in enumerate(class_ids):
@@ -771,8 +771,8 @@ def compute(self) -> dict:
         map_val, mar_val = self._summarize_results(precisions, recalls)
 
         # if class mode is enabled, evaluate metrics per class
-        map_per_class_values: Tensor = Tensor([-1])
-        mar_max_dets_per_class_values: Tensor = Tensor([-1])
+        map_per_class_values: Tensor = torch.tensor([-1.0])
+        mar_max_dets_per_class_values: Tensor = torch.tensor([-1.0])
         if self.class_metrics:
             map_per_class_list = []
             mar_max_dets_per_class_list = []
@@ -784,8 +784,8 @@ def compute(self) -> dict:
                 map_per_class_list.append(cls_map.map)
                 mar_max_dets_per_class_list.append(cls_mar[f"mar_{self.max_detection_thresholds[-1]}"])
 
-            map_per_class_values = Tensor(map_per_class_list)
-            mar_max_dets_per_class_values = Tensor(mar_max_dets_per_class_list)
+            map_per_class_values = torch.tensor(map_per_class_list, dtype=torch.float)
+            mar_max_dets_per_class_values = torch.tensor(mar_max_dets_per_class_list, dtype=torch.float)
 
         metrics = COCOMetricResults()
         metrics.update(map_val)


Test patch:
diff --git a/tests/detection/test_map.py b/tests/detection/test_map.py
--- a/tests/detection/test_map.py
+++ b/tests/detection/test_map.py
@@ -99,6 +99,40 @@
     ],
 )
 
+# example from this issue https://github.com/PyTorchLightning/metrics/issues/943
+_inputs2 = Input(
+    preds=[
+        [
+            dict(
+                boxes=torch.Tensor([[258.0, 41.0, 606.0, 285.0]]),
+                scores=torch.Tensor([0.536]),
+                labels=torch.IntTensor([0]),
+            ),
+        ],
+        [
+            dict(
+                boxes=torch.Tensor([[258.0, 41.0, 606.0, 285.0]]),
+                scores=torch.Tensor([0.536]),
+                labels=torch.IntTensor([0]),
+            )
+        ],
+    ],
+    target=[
+        [
+            dict(
+                boxes=torch.Tensor([[214.0, 41.0, 562.0, 285.0]]),
+                labels=torch.IntTensor([0]),
+            )
+        ],
+        [
+            dict(
+                boxes=torch.Tensor([]),
+                labels=torch.IntTensor([]),
+            )
+        ],
+    ],
+)
+
 
 def _compare_fn(preds, target) -> dict:
     """Comparison function for map implementation.
@@ -249,14 +283,13 @@ def _move_to_gpu(input):
 
 @pytest.mark.skipif(_pytest_condition, reason="test requires that torchvision=>0.8.0 is installed")
 @pytest.mark.skipif(_gpu_test_condition, reason="test requires CUDA availability")
-def test_map_gpu():
+@pytest.mark.parametrize("inputs", [_inputs, _inputs2])
+def test_map_gpu(inputs):
     """Test predictions on single gpu."""
     metric = MeanAveragePrecision()
     metric = metric.to("cuda")
-    preds = _inputs.preds[0]
-    targets = _inputs.target[0]
-
-    metric.update(_move_to_gpu(preds), _move_to_gpu(targets))
+    for preds, targets in zip(inputs.preds, inputs.target):
+        metric.update(_move_to_gpu(preds), _move_to_gpu(targets))
     metric.compute()