## 5 Publicly Available Datasets and Benchmarks
### Datasets for Panoptic Segmentation  
_COCO 2020 Panoptic Segmentation_  
To push the SOTA in achieving a coherent scene segmentation, the COCO PS aims at unifying the semantic and instance segmentation tasks to address the needs of the current real-world computer vision systems (augmented reality, self-driving vehicles, etc.). As discussed earlier, \"things\" refers to as countable objects (animals, people, tools, etc.) and \"stuff\" refers to regions that belong to the same texture or material (sky, road, grass, pavements, etc.). Earlier Microsoft COCO models evaluated these two tasks separately. COCO PS works by assigning a semantic label and an instance id to each pixel in an image, thereby achieving a dense coherent scene segmentation. This PS task has been a part of the Joint COCO and LVIS recognition challenge workshop at ECCV 2020 [46]. PS utilizes all the annotated 123 k COCO images that are divided into 172 classes, of which 80 belong to the \"thing\" category and 91 to the \"stuff\" category. The panoptic quality (PQ) is used to evaluate the performance of the model. Figure 13: Autonomous driving using panoptic segmentation ([PERSON]) [45]. Cityscapes Panoptic-Semantic Labeling  
Dataset of urban street scenes are captured by a vehicle in 50 German cities [47]. It is widely used to evaluate semantic, instance, and panoptic segmentation tasks. It comprises of around 5000 diverse frames with a high-quality pixel-level annotation. Together it constitutes 30 classes of which 10 classes belong to \"thing\" category (person, car, animals, etc.) and 20 classes belong to \"stuff\" category (sky, grass, and ground), respectively. BDD100K Panoptic Segmentation  
BDD is a largest driving video dataset captured from various cities in the United States and contains 100 K videos that have pixel wise annotations and around 10 tasks for evaluating the exciting progress of image recognition algorithms in context to autonomous driving [48]. Together it constitutes 40 classes of which 10 classes belong to \"thing\" category (particularly non-stationary objects) and 30 classes belong to \"stuff\" category, respectively. This dataset takes into account environmental factors such as the weather, geographic diversity, and is useful for training models such these are less likely to be surprised by new conditions. Mapillary Vistas v2.0 supports semantic, instance, and panoptic segmentation and provides street-level images captured from six different continents. This dataset contains around 25,000 high resolution images with 124 semantic object categories, among which 70 belong to \"thing\" category and 54 belong to the \"stuff\" category [49]. Additionally, it captures various factors that include seasons, weather, time of the day, camera, and viewpoint. Moreover, the annotations consist of polygons and not bitmaps. It is seen that this dataset earlier contained 65 classes in totality, among which 28 were from \"stuff\" classes and 37 from \"thing\" classes, respectively [50]. The dataset was further divided into three sets namely, the training, validation, and testing sets, and moreover, 18,000, 2000, and 5000 images were referred to as the size belonging to each category. Semantic KITTI Panoptic Segmentation  
Semantic KITTI is a 3D point cloud PS dataset that contain street scenes captured with LiDAR sensor or a stereo camera from Karlsruhe, Germany [51], with a 360 degrees field view. It consists of 11 driving sequences with PS labels and these labels use six \"thing\" classes and 16 \"stuff\" class categories. ScanNet  
ScanNet is an RGB-D video dataset that contains 2.5 million views in 1513 scans, annotated with 3D camera poses, surface reconstructions, and instance-level semantic segmentations [52]. It uses 38 \"thing\" classes (items and furniture in rooms) and two \"stuff\" classes (such as walls and floor). Though it is not yet completely developed it covers around 90% of all surfaces.