## 3 Public Datasets for Scene Understanding Methods
On the one hand, urban driving data contain road infrastructure and building facades in the peripheral field of view of the car's sensors. On the other hand, autonomous driving is a leading force for scene understanding and Simultaneous Localization And Mapping (SLAM) methods, which implies that the latest algorithms are often developed and benchmarked with urban driving data. The SemanticKITY dataset contains semantic classes of structural elements: roads, sidewalks, buildings, poles, road signs, and other structures besides the standard traffic unit classes, and became a benchmark for autonomous driving and outdoor SLAM algorithms [74]. The NCTL dataset consists of omnidirectional imagery, 3D LiDAR, planar LiDAR, GPS, and odometry of outdoor and indoor scenes from the University of Michigan campus, captured by an autonomous Segway robot [49]. Unfortunately, the point data are not annotated; therefore, the application for supervised machine learning is not possible without considerable effort to prepare the data. Figure 2: A tree structure to summarize the variety of common dataset configurations for 3D scene understanding tasks. Digital zoom is recommended. Besides autonomous driving, the second most covered spaces in the included datasets are residential indoor scenes. Most real-world indoor datasets were captured with RGB-D devices and were released between 2015 and 2017. This trend can be explained by the price drop of 3D imaging hardware in the mid-2010s, which caused a leap in the development of many successful semantic segmentation methods that use both RGB and depth features [56,83]. Consumer RGB-D cameras, like the Microsoft Kinect [84], are easy to use but not necessarily best suited for capturing large-scale spaces. Their small field of view requires stitching together many frames per scene in postprocessing, which often leads to registration errors and non-complete room geometry, as shown in Figure 4. Almost all indoor datasets have in common that they present fully furnished rooms of residential buildings and office spaces. In most cases, the class annotations focus on furniture besides the main building components (wall, floor, ceiling, door, and window). One of the first large indoor scene understanding datasets was the SUN RGB-D benchmark suite [52], which holds 10K RGB and depth images of residential rooms. Objects and room geometry is annotated in 2D by polygons and in 3D by its boundary boxes and orientation. SceneNN [56] holds comparable content, but the scenes are reconstructed into triangle meshes and possess per-vertex and per-pixel annotation. One of the most impactful 3D datasets in the indoor domain with point-wise semantic class annotations available is ScanNetV2 [62], now in its second generation. The dataset offers RGB-D video streams and 3D camera poses of 1.5K residential scenes captured with low-cost sensor setups and crowd-sourced instance-level semantic annotation. VASAD [22] is a synthetic volume and semantic architectural dataset composed of six buildings. The focus of VASAD is to improve semantic segmentation and volumetric reconstruction for BIM modeling. VASAD's semantic classes only consist of building components, and its authors introduce a method to automatically simulate terrestrial laser scanning (TLS) behavior by raytracing virtual scanlines from iteratively added viewpoints to sample point clouds from mesh-based CAD models. The Stanford 3D Indoor Scene Dataset S3 DIS [59] stands out as the biggest fully annotated indoor point cloud dataset and is heavily used for training and benchmarking in semantic segmentation tasks [85]. The S3 DIS sources from joint 2D-3D semantic data [31] and contains 6 large-scale indoor areas with 271 rooms. These areas show diverse architectural styles and appearances and mainly include office areas, educational and exhibition spaces, restrooms, open spaces, lobbies, stairways, and hallways.