Repository: pytorch/vision
Pull request: Throw ValueError in draw bounding boxes for invalid boxes
Closes #5517

We also suggest users to convert the boxes, in case they are using some other format.

cc @NicolasHug 

Problem statement:
draw_bounding_boxes should check that (W, H) >= (xmax, ymax) >= (xmin, ymin) >= (0, 0)
### 🚀 The feature

Currently, `draw_bounding_boxes` does not sanity check the input bounding boxes to ensure that `(xmax, ymax) >= (xmin, ymin)`. This can lead to unexpected output that can be difficult to debug if the user, say, inputs boxes in cxcywh format.

I could make a PR for this if needed but would prefer if someone else did it since I have my hands full with a PR I'm working on for PyTorch proper.

### Motivation, pitch

I personally just ran into this problem and I thought something was wrong with my network for a while.

Currently, the box conversion ops do this check, and it seems at least as important to do the same check in `draw_bounding_boxes`.

### Alternatives

_No response_

### Additional context

_No response_

Hints from issue discussion:
I agree with the overall sentiment here. My question is how you got to the situation. Did you know that you had "cxcywh" bounding boxes and missed that `draw_bounding_boxes` requires "xyxy" or did you not know the format before and only started to look into it after you got wrong results?

In the former case, we also could have a `format="xyxy"` parameter on the function, since we know how to convert bounding boxes. In general though, a smoke test after the conversion is still something we could do.

We are currently in the process of introducing new abstractions for bounding boxes. With that, bounding boxes store information about the format and thus a conversion as proposed above would be handled automatically.

https://github.com/pytorch/vision/blob/71d2bb0bc67044f55d38bfddf04e05be0343deab/torchvision/prototype/features/_image.py#L83-L86

In an internal discussion following https://github.com/pytorch/vision/pull/5500#discussion_r816501046, we decided to move this functionality into `torchvision.utils` just before we migrate the new abstractions to the main area.
@pmeier I knew that `draw_bounding_boxes` requires "xyxy", but I got confused about which format my bounding boxes were at that particular point in the code, since I'm using a DETR model where the model outputs cxcywh, but xyxy is used for the GIOU loss, and my dataset uses xyxy format.

The class abstraction definitely sounds like a good idea. I still think though that it would be a good idea to add this check to the "functional" version `draw_bounding_box` API, especially since the conversion ops already do it.
@norabelrose thanks for the suggestion, we'd be happy to review a PR that adds these checks
To avoid code duplication and for convenience.

I suggest we create a small util `_check_box_fmt`. We need these is multiple places. `box_iou`, `generalized_box_iou`, `nms` and `draw_bounding_boxes`.

With this single util we could check all places and raise ValueError suggesting to either check the input or use `box_convert` to change the format. 

cc @NicolasHug @norabelrose 
Let me know if you didn't have bandwidth / need help. 
There was some discussions before about adding these checks in the ops (box_iou, nms etc.). We don't do it because this would require adding a synchronization point of the CUDA streams. It's probably best to ignore the ops for now and stick to `draw_bounding_boxes`
I could look into this if no one else is.

Sure, go ahead!
Go ahead @singhularAdi Feel free to ping Nicolas for review :) 
Note that you should throw a ValueError if the above case occurs 

Edit: 
Wait a sec. I think I know you. You are my college senior. :smile:  World is such a nice circle.

Commit message: Intellij vs ufmt battle

Patch to review:
diff --git a/test/test_utils.py b/test/test_utils.py
index fab8c5fc082..7cff53e98a3 100644
--- a/test/test_utils.py
+++ b/test/test_utils.py
@@ -152,6 +152,7 @@ def test_draw_invalid_boxes():
     img_wrong2 = torch.full((1, 3, 5, 5), 255, dtype=torch.uint8)
     img_correct = torch.zeros((3, 10, 10), dtype=torch.uint8)
     boxes = torch.tensor([[0, 0, 20, 20], [0, 0, 0, 0], [10, 15, 30, 35], [23, 35, 93, 95]], dtype=torch.float)
+    boxes_wrong = torch.tensor([[10, 10, 4, 5], [30, 20, 10, 5]], dtype=torch.float)
     labels_wrong = ["one", "two"]
     colors_wrong = ["pink", "blue"]
 
@@ -167,6 +168,8 @@ def test_draw_invalid_boxes():
         utils.draw_bounding_boxes(img_correct, boxes, labels_wrong)
     with pytest.raises(ValueError, match="Number of colors"):
         utils.draw_bounding_boxes(img_correct, boxes, colors=colors_wrong)
+    with pytest.raises(ValueError, match="Boxes need to be in"):
+        utils.draw_bounding_boxes(img_correct, boxes_wrong)
 
 
 def test_draw_boxes_warning():
diff --git a/torchvision/utils.py b/torchvision/utils.py
index 80627ca5aa2..9484b683772 100644
--- a/torchvision/utils.py
+++ b/torchvision/utils.py
@@ -208,6 +208,10 @@ def draw_bounding_boxes(
         raise ValueError("Pass individual images, not batches")
     elif image.size(0) not in {1, 3}:
         raise ValueError("Only grayscale and RGB images are supported")
+    elif (boxes[:, 0] > boxes[:, 2]).any().item() or (boxes[:, 1] > boxes[:, 3]).any().item():
+        raise ValueError(
+            "Boxes need to be in (xmin, ymin, xmax, ymax) format. Use torchvision.ops.box_convert to convert them"
+        )
 
     num_boxes = boxes.shape[0]
 


Code review comments:
diff_hunk: @@ -208,6 +208,10 @@ def draw_bounding_boxes(
         raise ValueError("Pass individual images, not batches")
     elif image.size(0) not in {1, 3}:
         raise ValueError("Only grayscale and RGB images are supported")
+    elif (boxes[:, 0] > boxes[:, 2]).any().item() or (boxes[:, 1] > boxes[:, 3]).any().item():
path: torchvision/utils.py
text: @author:
Any better idiom than this?

@user1:
SGTM, this is what I would have used.

2 minor Q:

Maybe we don't need the `.item()` part?
Also, do we want this to be `>=` instead of `>` ?

@user1:
as discussed offline the most important thing is to prevent boxes in the wrong format, but degenerate boxes like `xmin == xmax` are not as important. So we're fine keeping `>`