SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jingkang Yang $^{1}$ , Jun Cen $^{2}$ , Wenxuan Peng $^{1}$ , Shuai Liu $^{3}$ , Fangzhou Hong $^{1}$ , Xiangtai Li $^{1}$ , Kaiyang Zhou $^{4}$ , Qifeng Chen $^{2}$ , Ziwei Liu $^{1}$
$^{1}$ S-Lab, Nanyang Technological University
2The Hong Kong University of Science and Technology
$^{3}$ Beijing University of Posts and Telecommunications  $^{4}$ Hong Kong Baptist University
https://github.com/Jingkang50/PSG4D
(a) Visual Input from the 4D Dynamic World
(b) PSG-4D: 4D Panoptic Scene Graph
Figure 1: Conceptual illustration of PSG-4D. PSG-4D is essentially a spatiotemporal representation capturing not only fine-grained semantics in image pixels (i.e., panoptic segmentation masks) but also the temporal relational information (i.e., scene graphs). In (a) and (b), the model abstracts information streaming in RGB-D videos into (i) nodes that represent entities with accurate location and status information and (ii) edges that encapsulate the temporal relations. Such a rich 4D representation serves as a bridge between the PSG-4D system and a large language model, which greatly facilitates the decision-making process, as illustrated in (c).
(c) Reasoning & Planning
We are living in a three-dimensional space while moving forward through a fourth dimension: time. To allow artificial intelligence to develop a comprehensive understanding of such a 4D environment, we introduce 4D Panoptic Scene Graph (PSG-4D), a new representation that bridges the raw visual data perceived in a dynamic 4D world and high-level visual understanding. Specifically, PSG-4D abstracts rich 4D sensory data into nodes, which represent entities with precise location and status information, and edges, which capture the temporal relations. To facilitate research in this new area, we build a richly annotated PSG-4D dataset consisting of 3K RGB-D videos with a total of 1M frames, each of which is labeled with 4D panoptic segmentation masks as well as fine-grained, dynamic scene graphs. To solve PSG-4D, we propose PSG4DFormer, a Transformer-based model that can predict panoptic segmentation masks, track masks along the time axis, and generate the corresponding scene graphs via a relation component. Extensive experiments on the new dataset show that our method can serve as a strong baseline for future research on PSG-4D. In the end, we provide a real-world application example to demonstrate how we can achieve dynamic scene understanding by integrating a large language model into our PSG-4D system.
Corresponding author. Contact: {jingkang001, ziwei.liu}@ntu.edu.sg
The emergence of intelligent agents, autonomous systems, and robots demands a profound understanding of real-world environments [1, 2, 3, 4, 5, 6]. This understanding involves more than just recognizing individual objects - it requires an intricate understanding of the relationships between these objects. In this context, research on Scene Graph Generation (SGG) [7], has sought to provide a more detailed, relational perspective on scene understanding. In this approach, scene graphs represent objects as nodes and their relationships as edges, offering a more comprehensive and structured understanding of the scene [8, 7, 9, 10, 11]. Panoptic Scene Graph Generation (PSG) [12] expands the scope of SGG to encompass pixel-level precise object localization and comprehensive scene understanding, including background elements. Then PSG has been further extended to the domain of videos [13] with the inspiration from Video Scene Graph Generation (VidSGG) [14, 15].
The utility of scene graphs also extends into the realm of 3D perception, introducing the concept of 3D Scene Graphs (3DSG) [16, 17]. 3DSGs offer a precise representation of object locations and inter-object relationships within three-dimensional scenes [18, 19]. Despite these developments, the existing approaches have not fully integrated dynamic, spatio-temporal relationships, particularly those involving human-object and human-human interactions. Consider Figure 1 as an illustrative example. Traditional 3D scene graph methods may recognize the static elements of this scene, such as identifying a booth situated on the ground. However, a more ideal, advanced, and dynamic perception is required for real-world scenarios. For instance, a system should be capable of identifying a dynamic event like a person who has fallen off their bike, so that it could then comprehend the necessity to offer assistance, like helping the person stand up and stabilize their bike.
Therefore, our work takes a significant step towards a more comprehensive approach to sensing and understanding the world. We introduce a new task, the 4D Panoptic Scene Graph (PSG-4D), aiming to bridge the gap between raw visual inputs in a dynamic 4D world and high-level visual understanding. PSG-4D comprises two main elements: nodes, representing entities with accurate location and status information, and edges, denoting temporal relations. This task encapsulates both spatial and temporal dimensions, bringing us closer to a true understanding of the dynamic world.
To facilitate research on this new task, we contribute an extensively annotated PSG-4D dataset that is composed of 2 sub-sets, PSG4D-GTA and PSG4D-HOI. The PSG4D-GTA subset consists of 67 RGB-D videos with a total of 28K frames, selected from the SAIL-VOS 3D dataset [20] collected from the video game Grand Theft Auto V (GTA-V) [21]. The PSG4D-HOI subset is a collection of 3K egocentric real-world videos sampled from the HOI4D dataset [22]. All frames in either of the subset are labeled with 4D panoptic segmentation masks as well as fine-grained, dynamic scene graphs. We believe this dataset will serve as a valuable resource for researchers in the field.
To tackle this novel task, we propose a unified framework called PSG4DFormer. This unified structure encapsulates two primary components: a 4D Panoptic Segmentation model and a Relation model. The 4D Panoptic Segmentation model is designed to accommodate both RGB-D and point cloud data inputs, yielding a 4D panoptic segmentation. This output comprises 3D object masks, which are continuously tracked across temporal dimensions. Then, the Relation model accepts these 3D mask tubes and utilizes a spatial-temporal transformer architecture to delineate long-term dependencies and intricate inter-entity relationships, subsequently yielding a relational scene graph. Through extensive experiments, we demonstrate the effectiveness of the proposed PSG-4D task and the PSG4DFormer model. Our work constitutes a pivotal step towards a comprehensive understanding of dynamic environments, setting the stage for future research in this exciting and crucial area of study.
In summary, we make the following contributions to the community:
- A New Task: We propose a novel scene graph generation task focusing on the prediction of 4D panoptic scene graphs from RGB-D or point cloud video sequences.
- A New Dataset: We provide a PSG-4D dataset, which covers diverse viewpoints: (i) a third-view synthetic subset (PSG4D-GTA) and (ii) an egocentric real-world subset (PSG4D-HOI).
- A Unified Framework: We propose a unified two-stage model composed of a feature extractor and a relation learner. In addition, we offer demo support for both synthetic and real-world scenarios to facilitate future research and real-world applications.
- Open-Source Codebase: We open-source our codebase to facilitate future PSG-4D research.
Scene Graph Generation (SGG) SGG transforms an image into a graph, where nodes represent objects and edges represent relationships [7]. Several datasets [23] and methods, including two-stage [8, 7, 9, 10, 11] and one-stage models [24, 12, 25], have been developed for SGG. Video scene graph generation (VidSGG) extends SGG to videos with notable datasets [14, 15, 26]. Despite progress, limitations remain in SGG and VidSGG due to noisy grounding annotations caused by coarse bounding box annotations and trivial relation definitions. Recent work on panoptic scene graph generation (PSG) [12, 27, 28, 29, 30, 31] has attempted to overcome these issues, and PVSG [13, 32] further extends it into the video domain. This paper presents an extension of PSG into a 4D dynamic world, meeting the needs of active agents for precise location and comprehensive scene understanding.
3D Scene Graph Generation 3D Scene Graphs (3DSGs) offer a precise 3D representation of object locations and inter-object relationships, making them a vital tool for intelligent agents operating in real-world environments [16, 17]. 3DSGs can be categorized into flat and hierarchical structures [33]. The former represents objects and relationships as a simple graph [18, 19], while the latter layers the structures of 3D scenes [34, 35]. Recent 3DSG techniques [19] employ PointNet [36] with 3D object detectors on point clouds or RGBD scans, generating 3D graphs via graph neural networks [18]. Some settings, such as Kimera [37], emphasize pairwise spatiotemporal status to facilitate task planning, while incremental 3DSG necessitates agents to progressively explore environments [38]. However, these graphs largely represent positional relations, lacking dynamic spatiotemporal relations like human-object interactions and human-human relations.
4D Perception Research on 4D perception can be divided by the specific data format they use. The first one is RGB-D video, which can be easily obtained using cheap sensors, e.g. Kinect, and iPhone. With the additional depth data, more geometric and spatial information can be used for reliable and robust detection [39, 40, 41] and segmentation [42, 43, 44, 45]. For RGB-D video, the depth input is usually treated like images. But for point clouds video, 3D or higher dimension convolutions [46, 47, 48, 49] are more commonly used, especially on LiDAR point cloud videos for autonomous driving perception system. In this work, beyond the 4D panoptic segmentation, we focus on more daily scenes and pursue a more high-level and structured understanding of 4D scenes by building 4D scene graphs.
3 The PSG-4D Problem
The PSG-4D task is aimed at generating a dynamic scene graph, which describes a given 4D environment. In this context, each node corresponds to an object, while each edge represents a spatial-temporal relation. The PSG-4D model ingests either an RGB-D video sequence or a point cloud video sequence, subsequently outputting a PSG-4D scene graph  $\mathbf{G}$ . This graph is composed of 4D object binary mask tubes  $\mathbf{M}$ , object labels  $\mathbf{O}$ , and relations  $\mathbf{R}$ .
The object binary mask tubes,  $\mathbf{m}_i\in \{0,1\}^{T\times H\times W\times 4}$ , express the 3D location and extent of the tracked object  $i$  over time  $(T)$  in the case of an RGB-D sequence input, while  $\mathbf{m}_i\in \{0,1\}^{T\times M\times 6}$  is used for point cloud video inputs. Here, 4 denotes RGB-D values, and 6 represents XYZ plus RGB values. M stands for the number of point clouds of interest. The object label,  $o_i\in \mathbb{C}^O$ , designates the category of the object. The relation  $r_i\in \mathbb{C}^R$  represents a subject and an object linked by a predicate class and a time period.  $\mathbb{C}^O$  and  $\mathbb{C}^R$  refer to the object and predicate classes, respectively. The PSG-4D task can be mathematically formulated as:
$$
\Pr (\mathbf {G} \mid \mathbf {I}) = \Pr (\mathbf {M}, \mathbf {O}, \mathbf {R} \mid \mathbf {I}), \tag {1}
$$
where  $\mathbf{I}$  represents the input RGB-D video sequence or point cloud representation.
Evaluation Metrics For evaluating the performance of the PSG-4D model, we employ the R@K and mR@K metrics, traditionally used in the scene graph generation tasks. R@K calculates the triplet recall, while mR@K computes the mean recall, both considering the top K triplets from the PSG-4D model. A successful recall of a ground-truth triplet must meet the following criteria: 1) correct category labels for the subject, object, and predicate; 2) a volume Intersection over Union (vIOU) greater than 0.5 between the predicted mask tubes and the ground-truth tubes. When these criteria are satisfied, a soft recall score is recorded, representing the time vIOU between the predicted and the ground-truth time periods.
Table 1: Illustration of the PSG-4D dataset and related datasets. Unlike the static 3D indoor scenes usually found in 3DSG datasets, the PSG-4D dataset introduces dynamic 3D videos, each annotated with panoptic segmentation. Various 3D video datasets were evaluated as potential sources for PSG-4D, resulting in the creation of two subsets: PSG4D-GTA and PSG4D-HOI. Regarding annotations, PS represents Panoptic Segmentation, BB represents Bounding Box, SS represents Semantic Segmentation, KP represents key points, and PC represents point clouds. TPV represents third-person-view.
Dataset	Type	Scale	View	#ObjCls	#RelCls	Annotation	Year
3DSSG [18]	3DSG	363K RGB-D images, 1482 scans, 478 scenes,	TPV	534	40	3D model, 3D graph	2020
Rel3D [50]	3DSG	27K RGB-D images, 9990 3D Scenes	TPV	67	30	3D model	2020
ScanNet [51]	3D Images	2.5M RGB-D images, 1513 indoor scenes	TPV	20	-	SS, 3D model	2017
Matterport 3D [52]	3D Images	194,400 RGB-D images, 90 building-scale scenes	TPV	40	-	SS, 3D model	2017
Nuscenes [53]	2D Video+PC	1K videos (avg. 20s), 1.3M pointclouds	Vehicle	23	-	3D BB	2020
WAYMO [54]	2D Video+PC	1.2K videos (avg. 20s), 177K pointclouds	Vehicle	20	-	2D BB, 3D BB	2020
Sail-VOS 3D [55]	3D Video	484 videos, 238K RGB-D image, 6807 clips	egocentric	178	-	SS, 3D model	2021
HOI4D [56]	3D Video	4K videos, 2.4M RGB-D image, 610 indoor scenes	egocentric	16	11	PS, KP	2022
EgoBody [57]	3D Video	125 videos, 199K RGB-D images, 15 indoor scenes	egocentric, TPV	36	13	3D model, KP	2022
PSG4D-GTA	PSG4D	67 videos (avg. 84s), 28K RGB-D images, 28.3B pointclouds	TPV	35	43	PS, 4DSG	2023
PSG4D-HOI	PSG4D	2973 videos (avg. 20s), 891K RGB-D images, 282 indoor scenes	egocentric	46	15	PS, 4DSG	2023
4 The PSG-4D Dataset
This section outlines the development of the PSG-4D dataset. We begin by exploring existing datasets that inspired the creation of PSG-4D, followed by a presentation of its statistics, and finally a brief overview of the steps involved in its construction.
4.1 Leveraging Existing Datasets for PSG-4D
Rather than constructing the PSG-4D dataset from the ground up, we sought to evaluate whether currently available datasets could either directly support or be adapted for the PSG-4D task. As shown in Table 1, our initial exploration focused on 3D datasets, including 3D scene graph datasets like 3DSGG [18] and Rel3D [50], along with more conventional 3D datasets such as ScanNet [51] and Matterport 3D [52]. However, while these datasets can be used to reconstruct entire scenes and can generate 3D videos accordingly, the resulting scenes remain static and lack dynamic elements.
We then shifted our focus to video datasets containing 3D information. Autonomous driving datasets such as Nuscenes [53] and WAYMO [54] incorporate point cloud videos, particularly bird's-eye view footage. Nevertheless, the vehicles within these scenes are only captured in 2D video. While this technically constitutes a dynamic 4D scene, it does not align well with the objectives of this study. The dynamic relations in traffic scenarios are relatively limited, and our goal is to develop a visual understanding model for embodied AI [58, 59, 60, 61] that captures 3D scenes from the agent's perspective, not a bird's-eye view.
Another category of 3D videos uses RGB-D sequences as input, which can be easily converted into point clouds. This data format aligns perfectly with the operation of intelligent agents, mimicking human perception, which captures continuous RGB images with depth. Thankfully, recent datasets like SAIL-VOS 3D [55], HOI4D [56], and EgoBody [57] have adopted this approach. While SAIL-VOS 3D uses synthetic data from the GTA game [21], the HOI4D dataset captures egocentric RGB-D videos of simple tasks, such as tool picking. On the other hand, the EgoBody dataset [57] records office activities like conversations, but lacks segmentation annotation and is primarily intended for human pose reconstruction. Despite its wealth of videos, the object interaction in EgoBody is limited. In the medical domain, 4D-OR [60] excels in providing detailed depictions of surgical scenes, showcasing its specialized utility. To cater to a broader spectrum of research applications, we formulated the PSG-4D dataset, integrating the versatile strengths of the SAIL-VOS 3D [55] and HOI4D [56] datasets.
4.2 Dataset Statistics
Figure 2 presents a selection of four video frames, drawn from both the PSG4D-GTA and PSG4D-HOI datasets. Each frame is an RGB-D video with corresponding panoptic segmentation annotations. Underneath each scene, we depict the associated scene graph and statistical word clouds. Annotators constructed these scene graphs as triplets, complete with frame duration. The PSG4D-GTA dataset is
(a) PSG4D-GTA (Synthetic, Third-Person View)
(b) PSG4D-HOI (Real-World, Egocentric)
Figure 2: The Examples and Word Clouds of PSG-4D dataset. The PSG-4D dataset contains 2 subsets, including (a) PSG4D-GTA selected from the SAIL-VOS 3D [20] dataset, and (b) PSG4D-HOI from HOI4D [22] dataset. We selected 4 frames of an example video from each subset. Each frame has aligned RGB and depth with panoptic segmentation annotation. The scene graph is annotated in the form of triplets. The word cloud for object and relation categories in each dataset is also represented.
particularly noteworthy for its composition: it contains 67 videos with an average length of 84 seconds, amounting to 27,700 RGB-D images, 28.3 billion point clouds, and comprises 35 object categories, and 43 relationship categories. This synthetic dataset was captured from a third-person perspective. In contrast, the PSG4D-HOI dataset is compiled from an egocentric perspective, providing a different context for analysis. It includes 2,973 videos with an average duration of 20 seconds, equating to 891,000 RGB-D images across 282 indoor scenes. This dataset includes 46 object categories and 15 object-object relationship categories, offering a diverse range of real-world data for the study. The combination of these two datasets offers a comprehensive understanding of 4D environments due to their complementary nature. A statistical overview of both datasets can be found in the final two rows of Table 1.
4.3 Dataset Construction Pipeline
As outlined in Section 4.1, the PSG4D-GTA is built upon the SAIL-VOS 3D dataset, while the PSG4D-HOI is derived from the HOI4D dataset. To adapt the SAIL-VOS 3D dataset for our purpose, we commenced with a comprehensive review of all 178 GTA videos within the dataset. This stage involved a meticulous elimination process to exclude videos containing NSFW content, resulting in a refined pool of 67 videos. The SAIL-VOS 3D dataset, which is equipped with 3D instance segmentation, required additional annotation for background elements to integrate panoptic segmentation. Leveraging the PVSG annotation pipeline, we employed an event detection method [62] to isolate the key frames. The background elements within these key frames were subsequently annotated using the pre-annotation provided by the SAM model [63]. Upon completion of key frame annotations, the AOT method [64] was utilized to propagate the segmentation across the entire video sequence. The final step involved overlaying the instance segmentation on the stuff segmentation, thereby completing the process. The HOI-4D dataset, devoid of NSFW content, already provides a 4D panoptic segmentation. Consequently, we included all videos from the HOI-4D dataset in the PSG4D-HOI dataset without further modifications.
Upon completion of 4D panoptic segmentation annotation, we proceed to annotate the dynamic scene graph according to the masks. Although HOI4D includes action annotation concerning the person, it doesn't account for interactions between objects. Nevertheless, certain actions such as "pick up" are appropriately considered predicates, and we automatically position the key object in the video to form a subject-verb-object triplet. Once the automatic-annotated dataset is prepared, we ask annotators to review and revise the pre-annotations to ensure accuracy. As SAIL-VOS 3D lacks all kinds of relational annotation, we commence scene graph annotation from scratch. The entire annotation process is diligently executed by the authors around the clock.
5 Methodology
This section details a unified pipeline PSG4DFormer for addressing the PSG-4D problem. As shown in Figure 3, our approach comprises two stages. The initial 4D panoptic segmentation stage aims to segment all 4D entities, including objects and background elements in Figure 3 (a), with the accurate
(a) Frame-Level Panoptic Segmentation
(b) Tracking
(c) Inference  $(\uparrow)$  and Training  $(\downarrow)$  of Relation Model
Figure 3: Illustration of the PSG4DFormer pipeline. This unified pipeline supports both RGB-D and point cloud video inputs and is composed of two main components: 4D panoptic segmentation modeling and relation modeling. The first stage seeks to obtain the 4D panoptic segmentation mask for each object, along with its corresponding feature tube spanning the video length. This is accomplished with the aid of (a) frame-level panoptic segmentation and (b) a tracking model. The subsequent stage (c) employs a spatial-temporal transformer to predict pairwise relations based on all feature tubes derived from the first stage.
temporal association in Figure 3 (b). We extract features for each object and obtain feature tubes according to tracking results for subsequent relation modeling in Figure 3 (c).
5.1 4D Panoptic Segmentation Modeling
As specified in Section 3, given a 3D video clip input, such as an RGB-D sequence of  $\mathbf{I} \in \mathbb{R}^{T \times H \times W \times 4}$  or a point cloud sequence of  $\mathbf{I} \in \mathbb{R}^{T \times M \times 6}$ , the initial stage's goal is to segment and track each pixel non-overlappingly. The model predicts a set of video clips with the output of  $(\mathbf{m}_i, \mathbf{q}_i, p_i)_{i=1}^N$ , where  $\mathbf{m}_i$  denotes the tracked object mask tube,  $\mathbf{q}_i$  denotes the tracked feature tube, and  $p_i$  represents the probability of the object belonging to each category.  $N$  is the number of entities, encompassing things and stuff classes.
Frame-Level Panoptic Segmentation with RGB-D Sequence Given the dual input of RGB and depth images, we adopt a separation-and-aggregation gate (SA-Gate) [65] to efficiently blend information from both modalities. This combined feature set, enriched with data from both inputs, is then fed into a robust Mask2Former [4] for frame-level panoptic segmentation. In the inference stage, at the frame  $t$ , given an RGB-D image  $\mathbf{I}$ , the Mask2Former with SA-Gate directly outputs a set of object query features  $q_{i}^{t} \in \mathbb{R}^{d}, i = 1,\dots ,N$ , each  $q_{i}^{t}$  representing one entity at the frame  $t$ .
Frame-Level Panoptic Segmentation with Point Cloud Sequence Apart from perceiving point cloud sequences directly, 3D point cloud coordinates can be calculated and converted from RGB-D data. This conversion involves computing the Normalized Device Coordinates (NDC) using the depth map and projecting the NDC to world coordinates using the transformation matrix provided. We retain only points with a depth below a defined threshold  $\lambda$ , discarding distant, less relevant elements like far-off mountains. To leverage texture information from the image, point cloud coordinates can be augmented with corresponding RGB values, creating a colorful point cloud representation  $\mathbf{P} \in \mathbb{R}^{M \times 6}$ , where  $M$  is the total number of points in a frame.
We employ DKNet [66], a state-of-the-art indoor segmentation method, as our point cloud segmentation network. It processes input point clouds with a 3D UNet-like [67] backbone and uses sparse convolutions [68] for feature extraction. DKNet localizes instance centroids based on a candidate mining branch and encodes each instance's information into an instance kernel  $k_{i} \in \mathbb{R}^{d}$ . These instance kernels  $\{k_{i}\}_{i=1}^{N}$  are used as the weights of a few convolution layers to obtain the final instance masks.