## 4 Companies Developing Computer Vision Models for Panoptic Segmentation
### Facebook  
Facebook AI Research (FAIR) released Detectron2 as part of their next-generation library that offers state-of-the-art models for computer vision tasks [31]. It is an open-source framework for object detection and segmentation framework built on top of PyTorch and provides a unified API for performing a variety of tasks such as object detection, instance segmentation, and panoptic segmentation. It provides high-quality implementations of the state-of-the-art algorithms like Mask-RCNN, Densepose estimation (see Figure 12, where the input image to the left is from Google search and the output result to the right  
Figure 11: Subject lifting network architecture [43].  
is extracted using Detectron2 [31]), RetinaNet, and rotated bounding boxes, to name a few, and is designed to be flexible, easy to use, and supports several research projects that have focus on enabling rapid research as it trains faster. It is considered as the successor to Detectron and the Mask-RCNN benchmark. Apart from this, it includes a model zoo with models for object detection, instance segmentation and many others.  
Detectron2 uses a two-stage approach to object detection: the first makes use of the region proposal network (RPN) to generate a set of candidate regions. In the second stage, it uses a Mask R-CNN model to classify and segment the candidate regions. The RPN is a CNN that takes an input image and generates a set of candidate regions and is trained to predict a score for each identified object/region, indicating the likelihood of that object belonging to a certain category, and predicts a bounding box for each category (as seen in Figure 1 where the input image to the left is from Google search and the output to the right is extracted using Detectron2 [31]). The repository for Detectron2 has open-source model weights for algorithms such as instance segmentation, mask-RCNN, panoptic segmentation (PS), and dense pose estimation.  
For PS, Detectron2 has been trained on COCO (common object in context) dataset that has been widely used particularly for visual detection tasks that constitute scene understanding. For training PS, Detectron2 uses 118 K images for training and 5 K images for testing, as the annotations in PS focus on \"stuff\" and \"things\" as discussed in earlier sections. Using this COCO dataset, 80 classes for \"things\" such as a person, an umbrella, a bicycle, etc., and 91 classes for \"stuff\" such as sky, road, pavements, etc., can be detected [44].  
It is seen that PS can be performed only on a handful of datasets, like COCO, Cityscapes, Indian driving, Mapillary vistas, and ADE20k, that have required annotations. Moving forward, the next step in the process is to have an ecosystem where many different datasets are annotated, and PS could then become capable of addressing a wide spectrum of use cases. Training a PS on local machine without GPU support would hinder the performance as there are large images to train. To address this, it is important that the computer vision industry come up with some ways that are easier to implement, support custom datasets, and provide ways to expand the different use cases across multiple domains.