Repository: pytorch/vision

Issue:
Ops to convert `masks` to `boxes`
## 🚀 Feature
A simple `torchvision.ops` to convert Segmentation masks to bounding boxes.

## Motivation

This has a few use-cases.

1. This makes it easier to use semantic segmentation datasets for object detection.
The pipeline can be easier. Also the bounding boxes are represented as `xyxy `in `torchvision.ops` as a convention.
So probably convert masks to `xyxy` format.

2. The other use case is to make it easier in comparing performance of segmentation model vs detection model.
Let's Say that the detection model performs well for segmentation dataset. Then it would be better to go ahead with detection models as it is faster in real-time use-cases than to train a segmentation model.

New Pipeline

```
from torchvision.ops import masks_to_boxes, box_convert

class SegmentationToDetectionDataset(Dataset):
    def __getitem__(self, idx):
          boxes_xyxy = masks_to_boxes(segmentation_masks)

         # Now for any change of boxes to COCO Format.
          boxes_xywh = box_convert(boxes_xyxy, in_fmt="xyxy", out_fmt="xywh")
          return boxes_xywh
```

## Pitch

Port the [masks_to_boxes](https://github.com/ashkamath/mdetr/blob/fdee8c50d7bcf2ad09cc0d6b783a8333720e4048/util/box_ops.py#L63) function from mDeTR.

[masks_to_boxes](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/util/box_ops.py#L64) was also used in DeTR.

## Alternatives

The above function assumes masks of shape `(N, H, W)` -> `num_masks, Height, Width`. A floating tensor.
IIRC, we used a boolean tensor in `draw_segmentation_masks` (After Nicolas refactored). So perhaps we should be using boolean tensor? Though I see no particular use case of this util being only valid for instance segmentation.

## Additional context

I can port this, we perhaps need a few tests to ensure it works fine.
Especially test for float16 overflow.

cc @datumbox @NicolasHug



Discussion:
I think this could be useful, at least as a plotting utils. I'm not sure I fully understand the use of such util within a model though?

> So perhaps we should be using boolean tensor?

Yes the input should be boolean: models have different representations for floating masks as illustrated in the new example, so we can't have a unified approach for floating masks.
I think it should under be `ops` and not `utils` as utils contain utilities which do not do any manipulation of any masks / boxes.

They return the passed image tensors back

This is a operation (of course is non-differentiable) hece suits better to `torchvision.ops`.

The only models which have different representation are instance segmentation models. Is this utility essential for instance segmentation? (I'm not sure)

Thanks for bringing this up @oke-aditya. 

This is a text-book example of utility we want to upstream from DeTR to TorchVision. I think it should be OK placing it in [torchvision.ops.boxes](https://github.com/pytorch/vision/blob/master/torchvision/ops/boxes.py) along with the other box utilities ported already.

> Especially test for float16 overflow.

Yes please, good call. We need to be careful for numeric overflows. See #3383 for addressing a similar issues on other box ops.

Edit: Also since we are porting things from DeTR, let's make sure we give credit by putting a reference to the original code. You can see [examples](https://github.com/pytorch/vision/blob/master/torchvision/ops/boxes.py#L268) of that already in our code.
What do you think @datumbox about the segmentation masks? Should they be kept float as `(num_masks, H, W)` or should we change to adopt bool masks?

I would prefer to keep float masks.
> This makes it easier to use ~~semantic~~ segmentation datasets for object detection.

I think as per your original point, you would have to support `(num_masks, H, W)` if you were to convert a the masks of a specific dataset to boxes for object detection. See how DeTR uses the method on the [CocoPanoptic](https://github.com/facebookresearch/detr/blob/a54b77800eb8e64e3ad0d8237789fcbf2f8350c5/datasets/coco_panoptic.py#L57) dataset.

I also see value to support for bool masks, as this will make it work seamlessly with Nicolas' `draw_segmentation_masks` but it's unclear to me if that should happen on the same method or on a separate. I would personally start by porting DeTR's implementation, adding tests, handling overflows and take it from there.

Adding @fmassa in the discussion as he usually have good intuition around segmentation/detection. 
Has anyone started to work on this? If not, I'm more than happy to port it over.

Resolving patch:
diff --git a/gallery/plot_repurposing_annotations.py b/gallery/plot_repurposing_annotations.py
new file mode 100644
--- /dev/null
+++ b/gallery/plot_repurposing_annotations.py
@@ -0,0 +1,75 @@
+"""
+=======================
+Repurposing annotations
+=======================
+
+The following example illustrates the operations available in the torchvision.ops module for repurposing object
+localization annotations for different tasks (e.g. transforming masks used by instance and panoptic segmentation
+methods into bounding boxes used by object detection methods).
+"""
+import os.path
+
+import PIL.Image
+import matplotlib.patches
+import matplotlib.pyplot
+import numpy
+import torch
+from torchvision.ops import masks_to_boxes
+
+ASSETS_DIRECTORY = "../test/assets"
+
+matplotlib.pyplot.rcParams["savefig.bbox"] = "tight"
+
+####################################
+# Masks
+# -----
+# In tasks like instance and panoptic segmentation, masks are commonly defined, and are defined by this package,
+# as a multi-dimensional array (e.g. a NumPy array or a PyTorch tensor) with the following shape:
+#
+#       (objects, height, width)
+#
+# Where objects is the number of annotated objects in the image. Each (height, width) object corresponds to exactly
+# one object. For example, if your input image has the dimensions 224 x 224 and has four annotated objects the shape
+# of your masks annotation has the following shape:
+#
+#       (4, 224, 224).
+#
+# A nice property of masks is that they can be easily repurposed to be used in methods to solve a variety of object
+# localization tasks.
+#
+# Masks to bounding boxes
+# ----------------------------------------
+# For example, the masks to bounding_boxes operation can be used to transform masks into bounding boxes that can be
+# used in methods like Faster RCNN and YOLO.
+
+with PIL.Image.open(os.path.join(ASSETS_DIRECTORY, "masks.tiff")) as image:
+    masks = torch.zeros((image.n_frames, image.height, image.width), dtype=torch.int)
+
+    for index in range(image.n_frames):
+        image.seek(index)
+
+        frame = numpy.array(image)
+
+        masks[index] = torch.tensor(frame)
+
+bounding_boxes = masks_to_boxes(masks)
+
+figure = matplotlib.pyplot.figure()
+
+a = figure.add_subplot(121)
+b = figure.add_subplot(122)
+
+labeled_image = torch.sum(masks, 0)
+
+a.imshow(labeled_image)
+b.imshow(labeled_image)
+
+for bounding_box in bounding_boxes:
+    x0, y0, x1, y1 = bounding_box
+
+    rectangle = matplotlib.patches.Rectangle((x0, y0), x1 - x0, y1 - y0, linewidth=1, edgecolor="r", facecolor="none")
+
+    b.add_patch(rectangle)
+
+a.set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])
+b.set(xticklabels=[], yticklabels=[], xticks=[], yticks=[])
diff --git a/torchvision/ops/__init__.py b/torchvision/ops/__init__.py
--- a/torchvision/ops/__init__.py
+++ b/torchvision/ops/__init__.py
@@ -1,4 +1,5 @@
-from .boxes import nms, batched_nms, remove_small_boxes, clip_boxes_to_image, box_area, box_iou, generalized_box_iou
+from .boxes import nms, batched_nms, remove_small_boxes, clip_boxes_to_image, box_area, box_iou, generalized_box_iou, \
+    masks_to_boxes
 from .boxes import box_convert
 from .deform_conv import deform_conv2d, DeformConv2d
 from .roi_align import roi_align, RoIAlign
diff --git a/torchvision/ops/boxes.py b/torchvision/ops/boxes.py
--- a/torchvision/ops/boxes.py
+++ b/torchvision/ops/boxes.py
@@ -297,3 +297,35 @@ def generalized_box_iou(boxes1: Tensor, boxes2: Tensor) -> Tensor:
     areai = whi[:, :, 0] * whi[:, :, 1]
 
     return iou - (areai - union) / areai
+
+
+def masks_to_boxes(masks: torch.Tensor) -> torch.Tensor:
+    """
+    Compute the bounding boxes around the provided masks
+
+    Returns a [N, 4] tensor.  Both sets of boxes are expected to be in ``(x1, y1, x2, y2)`` format with
+    ``0 <= x1 < x2`` and ``0 <= y1 < y2``.
+
+    Args:
+        masks (Tensor[N, H, W]): masks to transform where N is the number of
+        masks and (H, W) are the spatial dimensions.
+
+    Returns:
+        Tensor[N, 4]: bounding boxes
+    """
+    if masks.numel() == 0:
+        return torch.zeros((0, 4))
+
+    n = masks.shape[0]
+
+    bounding_boxes = torch.zeros((n, 4), device=masks.device, dtype=torch.int)
+
+    for index, mask in enumerate(masks):
+        y, x = torch.where(masks[index] != 0)
+
+        bounding_boxes[index, 0] = torch.min(x)
+        bounding_boxes[index, 1] = torch.min(y)
+        bounding_boxes[index, 2] = torch.max(x)
+        bounding_boxes[index, 3] = torch.max(y)
+
+    return bounding_boxes


Test patch:
diff --git a/test/assets/labeled_image.png b/test/assets/labeled_image.png
new file mode 100644
Binary files /dev/null and b/test/assets/labeled_image.png differ
diff --git a/test/assets/masks.tiff b/test/assets/masks.tiff
new file mode 100644
Binary files /dev/null and b/test/assets/masks.tiff differ
diff --git a/test/test_masks_to_boxes.py b/test/test_masks_to_boxes.py
new file mode 100644
--- /dev/null
+++ b/test/test_masks_to_boxes.py
@@ -0,0 +1,34 @@
+import os.path
+
+import PIL.Image
+import numpy
+import torch
+
+from torchvision.ops import masks_to_boxes
+
+ASSETS_DIRECTORY = os.path.join(os.path.dirname(os.path.abspath(__file__)), "assets")
+
+
+def test_masks_to_boxes():
+    with PIL.Image.open(os.path.join(ASSETS_DIRECTORY, "masks.tiff")) as image:
+        masks = torch.zeros((image.n_frames, image.height, image.width), dtype=torch.int)
+
+        for index in range(image.n_frames):
+            image.seek(index)
+
+            frame = numpy.array(image)
+
+            masks[index] = torch.tensor(frame)
+
+    expected = torch.tensor(
+        [[127, 2, 165, 40],
+         [2, 50, 44, 92],
+         [56, 63, 98, 100],
+         [139, 68, 175, 104],
+         [160, 112, 198, 145],
+         [49, 138, 99, 182],
+         [108, 148, 152, 213]],
+        dtype=torch.int32
+    )
+
+    torch.testing.assert_close(masks_to_boxes(masks), expected)