4D Panoptic Scene Graph Generation
Authors: Jingkang Yang
Jun CEN
Wenxuan Peng
Shuai Liu
Fangzhou Hong
Xiangtai Li
Kaiyang Zhou
Qifeng Chen
Ziwei Liu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Scene Graph Generation
4D Understanding
4D Perception.
TL;DR: We introduce a new task, dataset, and framework for 4D Panoptic Scene Graph Generation.

Abstract: We are living in a three-dimensional space while moving forward through a fourth dimension: time. To allow artificial intelligence to develop a comprehensive understanding of such a 4D environment, we introduce **4D Panoptic Scene Graph (PSG-4D)**, a new representation that bridges the raw visual data perceived in a dynamic 4D world and high-level visual understanding. Specifically, PSG-4D abstracts rich 4D sensory data into nodes, which represent entities with precise location and status information, and edges, which capture the temporal relations. To facilitate research in this new area, we build a richly annotated PSG-4D dataset consisting of 3K RGB-D videos with a total of 1M frames, each of which is labeled with 4D panoptic segmentation masks as well as fine-grained, dynamic scene graphs. To solve PSG-4D, we propose PSG4DFormer, a Transformer-based model that can predict panoptic segmentation masks, track masks along the time axis, and generate the corresponding scene graphs via a relation component. Extensive experiments on the new dataset show that our method can serve as a strong baseline for future research on PSG-4D. In the end, we provide a real-world application example to demonstrate how we can achieve dynamic scene understanding by integrating a large language model into our PSG-4D system.

4D Panoptic Scene Graph Generation
Jingkang Yang', Jun Cen', Wenxuan Peng', Shuai Liu?, Fangzhou Hong',
Xiangtai Li", Kaiyang Zhou', Qifeng Chen², Ziwei Liu E
L-Laa,, Nanyang Technological University The Hong Kong University of Science and Technology
3 Beijing University of Posts and Telecommunications "Hong Kong Baptist University
https: /ttps://github . com/ wt htmm//iinnan55/PSS///
(a) Visual Input from the 4D Dynamic World
(c) Reasoning & Planning
Perceiving
Executing
[krop sonding sraan graph...
[ih unit. steme a man fnl from
wa shwuld go help hin pick up his tike. Where the hike?
G
The like 2 chearl king on the ground.
(b) PSG-4D: 4D Panoptic Scene Graph
Parsing
Band go forward Am the bóa straighten the bike. check whether the man injured
A
en
RED
Exesuttng!
Eaki
adMM
Reasoning
Figure I: Conceptual illustration of PSG-4D. PSG-4D is essentially a spatiotemporal representation capturing not only fine ane-ggained semantics in image pixels (i.e.. panoptic segmentation masks) but also the temporal relational information (i.e., scene graphs). In (a) and (b), the model abstracts information streaming in RGB-D videos into (i) nodes that represent entities with accurate location and status information and (ii) edges that encapsulate the temporal relations. Such a rich 4D representation serves as bridge between the PSG-4D system and a large language model, which greatly facilitates the decision-making process, as illustrated in (c).
Abstract
We are living in diree-dimensional space while moving forward through fourth dimension: time. To allow artificial intelligence develop a comprehensive understanding of such a 4D environment, we introduce 4D Panoptic Scene Graph (PSG-4D), new representation that bridges the raw visual data perceived in a dynamic 4D world and high-level visual understanding. Specifically. PSG-4D abstracts rich 4D sensory data into nodes, which represent entities with precise location and status information, and edges, which capture the temporal relations. To facilitate research this new area, we build richly annotated PSG-4D dataset consisting of 3K RGB-D videos with a total of 1M frames, each of which is labeled with 4D panoptic segmentation masks as well as fine-grained, dynamic scene graphs. To solve PSG-4D, we propose PSG4DFormer, Transformer- -bssdd model that can predict panoptic segmentation masks, track masks along the time axis, and generate the corresponding scene graphs via a relation component. Extensive experiments on the new dataset show that our method can serve as strong baseline for future research on PSG-4D. In the end, provide a real-world application example to demonstrate how we can achieve dynamic scene understanding by integrating large language model into our PSG-4D system.
X Corresponding author: Contact: (jingkang001, zivei liuj-entu. edu.sg
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
The emergence of intelligent agents, autonomous systems, and robots demands a profound under- standing of real-world environments [1, 2, 3, 4, 5, 6]. This understanding involves more than just recognizing individual objects it requires an intricate understanding of the relationships between these objects. In this context, research on Scene Graph Generation (SGG) [7], has sought to provide a more detailed, relational perspective on scene understanding. In this approach, scene graphs represent objects as nodes and their relationships as edges, offering a more comprehensive and structured understanding of the scene [8, 7. 9, 10, 11]. Panoptic Scene Graph Generation (PSG) [12] expands the scope of SGG to encompass pixel-level precise object localization and comprehensive scene understanding, including background elements. Then PSG has been further extended to the domain of videos [13] with the inspiration from Video Scene Graph Generation (VidSGG) [14, 15J.
The utility of scene graphs also extends into the realm of 3D perception, introducing the concept of 3D Scene Graphs (3DSG) [16, 17J. 3DSGs offer precise representation of object locations and inter-object relationships within three-dimensional scenes [18, 19]. Despite these developments, the existing approaches have not fully integrated dynamic, spatio-temporal relationships, particularly those involving human-object and numan-human interactions. Consider Figure as an illustrative example. Traditional 3D scene graph methods may recognize the static elements of this scene, such as identifying a booth situated on the ground. However, more ideal, advanced, and dynamic perception required for real-world scenarios. For instance, a system should be capable of identifying a dynamic event like person who has fallen off their bike, so that it could then comprehend the necessity to offer assistance, like helping the person stand up and stabilize their bike.
Therefore, our work takes a significant step towards more comprehensive approach to sensing and understanding the world. We introduce a new task, the 4D Panoptic Scene Graph (PSG-4D), aiming to bridge the gap between raw visual inputs in dynamic 4D world and high-level visual understanding. PSG-4D comprises two main elements: nodes, representing entities with accurate location and status information, and edges, denoting temporal relations. This task encapsulates both spatial and temporal dimensions, bringing us closer to a true understanding of the dynamic world.
To facilitate research on this new task, we contribute an extensively annotated PSG-4D dataset that composed of 2 sub-sets, PSG4D-GTA and PSG4D-HOL. The PSG4D-GTA subset consists of 67 RGB-D videos with a total of 28K frames, selected from the SAIL-VOS 3D dataset [20] collected from the video game Grand Theft Auto V (GTA-V V) [21]. The PSG4D-HOI subset is a collection of 3K egocentric real-world videos sampled from the HOI4D dataset [22]. All frames in either of the subset are labeled with 4D panoptic segmentation masks as well as fine-grained, dynamic scene graphs. We believe this dataset will serve as valuable resource for researchers in the field.
To tackle this novel task, we propose a unified framework called PSG4DFormer. This unified structure encapsulates two primary components: 4D Panoptic Segmentation model and a Relation model. The 4D Panoptic Segmentation model is designed accommodate both RGB-D and point cloud data inputs. yielding a 4D panoptic segmentation. This output comprises 3D object masks, which are continuously tracked across temporal dimensions. Then, the Relation model accepts these 3D mask tubes and utilizes a -emtiall transformer architecture to delineate long- ann-termm& dependencies and intricate inter-entity relationships, subsequently yielding a relational scene graph. Through extensive experiments, we demonstrate the effectiveness of the proposed PSG-4D task and the PSG4DFormer model. Our work constitutes a pivotal step towards a comprehensive understanding of dynamic environments, setting the stage for future research in this exciting and crucial area of study.
In summary, we make the following contributions to the community:
We propose novel scene graph generation task focusing on the prediction of 4D panoptic scene graphs from RGB-D or point cloud video sequences.
A New Task:
A New Dataset: We provide a PSG-4D dataset, which covers diverse viewpoints: (i) third- view synthetic subset (PSG4D-GTA) and (ii) an egocentric real-world subset (PSG4D-HOI).
A Unified Framework: We propose unified two-stage model composed of feature extractor and a relation learner. In addition, we offer demo support for both synthetic and real-world scenarios to facilitate future research and real-world applications.
Open-Source Codebase: We open-source our codebase to facilitate future PSG-4D research.

---

2 Related Work
Scene Graph Generation (SGG) SGG transforms an image into graph, where nodes represent objects and edges represent relationships [7]. Several datasets [23] and methods, including two- stage [8, 7, 9, 10, 11] and one-stage models [24, 12, 25], have been developed for SGG. Video scene graph generation (VidSGG) extends SGG to videos with notable datasets [14, 15, 26]. Despite progress, limitations remain in SGG and VidSGG due noisy grounding annotations caused by coarse bounding box annotations and trivial relation definitions. Recent work on panoptic scene graph generation (PSG) [12. 27, 28, 29, 30, 31] has attempted overcome these issues, and PVSG [13, 32] further extends it into the video domain. This paper presents an extension of PSG into a 4D dynamic world, meeting the needs of active agents for precise location and comprehensive scene understanding.
3D Scene Graph Generation 3D Scene Graphs (3DSGs) offer precise 3D representation of object locations and inter-object relationships, making them a vital tool for intelligent agents operating in real-world environments [16, 17J. 3DSGs can be categorized into flat and hierarchical structures [33]. The former represents objects and relationships as a simple graph [18, 19), while the latter layers the structures of 3D scenes [34, 35J. Recent 3DSG techniques [19] employ PointNet [36] with 3D object detectors on point clouds or RGBD scans, generating 3D graphs via graph neural networks [18]. Some settings, such Kimera [37], emphasize pairwise spatiotemporal status to facilitate task planning, while incremental 3DSG necessitates agents to progressively explore environments [38]. However, these graphs largely represent positional relations, lacking dynamic spatiotemporal relations like human-object interactions and human- humma relations.
4D Perception Research on 4D perception can be divided by the specific data format they use. The first one is RGB-D video, which can be easily obtained using cheap sensors, e.g. Kinect, and iPhone. With the additional depth data, more geometric and spatial information can be used for reliable and robust detection [39, 40, 41] and segmentation [42, 43, 44, 45J. For RGB-D video, the depth input is usually treated like images. But for point clouds video, 3D or higher dimension convolutions [46, 47, 48, 49] are more commonly used, especially on LiDAR point cloud videos for autonomous driving perception system. In this work, beyond the 4D panoptic segmentation, we focus on more daily scenes and pursue a more high-level and structured understanding of 4D scenes by building 4D scene graphs.
3 The PSG-4D Problem
The PSG-4D task is aimed at generating a dynamic scene graph, which describes a given 4D environment. In this context, each node corresponds to an object, while each edge represents a spatial-temporee relation. The PSG-4D model ingests either an RGB-D video sequence or a point cloud video sequence, subsequently outputting a PSG-4D scene graph G. This graph is composed of 4D object binary mask tubes M, object labels O, and relations R.
The object binary mask tubes, m, E {0, (0,1)7xxx*** tracked object i over time (I)) in the case of an RGB-D sequence input, while m; E {0, 1 X is used for point cloud video inputs. Here, denotes RGB-D values, and 6 represents XYZ plus RGB values. M stands for the number of point clouds of interest. The object label, o E CO, designates the category of the object. The relation ry E C represents subject and an object linked by a predicate class and time period. Co and R refer to the object and predicate classes, respectively. The PSG-4D task can be mathematically formulated as:
express the 3D location and extent of the
Pr (G I) = Pr (M, O, R.
where I represents the input RGB-D video sequence or point cloud representation.
Evaluation Metries For evaluating the performance of the PSG-4D model, we employ the R@ K and mR @ K metrics, traditionally used in the scene graph generation tasks. R@K K calculates the triplet recall, while mR @K computes the mean recall, both considering the top triplets from the PSG-4D model. A successful recall of ground-truth triplet must meet the following criteria: 1) correct category labels for the subject, object, and predicate; 2) a volume Intersection over Union (vIOU) greater than 0.5 between the predicted mask tubes and the ground-truth tubes. When these criteria are satisfied, soft recall score is recorded, representing the time vIOU between the predicted and the tround-truth time periods.
3

---

Table 1: Illustration of the PSG-4D dataset and related datasets. Unlike the static 3D indoor scenes usually found in 3DSG datasets, the PSG-4D dataset introduces dynamic 3D videos, each annotated with panoptic segmentation. Various 3D video datasets were evaluated as potential sources for PSG-4D, resulting in the creation of two subsets: PSG4D-GTA and PSG4D-HOL, Regarding annotations, PS represents Panoptic Segmentation, BB represents Bounding Box, SS represents Semantic Segmentation, KP represents key points, and PC represents point clouds. TPV represents third third- hurddperson--iieee
Datnect
Type
Stale
View
#ChyCle
#RoiCss
Ammortio
Yor
3D8SG (18)
SDSG S08G
167K RGBFD images 1482 scans, 478 spines.
TPV
534
40
30 medel, 3D graph
2020
ReiD (50)
27K RGB-D images, 9900 303 Servica
TPV
67
1D model
2020
ScanNer [S1]
3D Images
2.5M RGB-D insges. 1513 iidoar scenes
TPV
20
3D model
2017
Manerpont32 3D [52]
3D Images
194,400 RGB D images, 90 building scale soctes
TPV
40
SS, 3D model
2017
Nescemee [53]
20 Video=PCC
IK videos (ang. 200. 1.3 poinclouds
Vehicle
30 20 BR, 3D
2020
WAYMO [54]
3D Video=ICC
1.3K videoe (ang 20s. 177K poinchouts
Vehicle
2020
Sul-VOS 3D (55)
Vidko
484 válcos, RGB-D imuge. 6807 clips
epcenitric
mookl)
HOL-D [96]
3D Video
Válcos, 2AM RGB image, 610 Indoct secaes
epocestric
PS,
PgoRodl (57
Imagge.
opoontrac,
PSG4D-GTTA
PSG4D
840,, RGB-B images. 3B poiatchuuds
TPV
4DSG
PSGAD-HOI
PSG4D
videow 891K RGB images. indorr
PS,
2023
4
The PSG-4D Dataset
This section outlines the development of the PSG-4D dataset. We begin by exploring existing datasets that inspired the creation of PSG-4D, followed by presentation of its statistics, and finally brief overview of the steps involved in its construction.
4.1 Leveraging Existing Datasets for PSG-4D
Rather than constructing the PSG-4D dataset from the ground up, we sought to evaluate whether currently available datasets could either directly support be adapted for the PSG-4D task. As shown in Table I, our initial exploration focused on 3D datasets, including 3D scene graph datasets like 3DSGG [18] and Re13D [50], along with more conventional 3D datasets such as ScanNet [51] and Matterport 3D [52]. However, while these datasets can be used to reconstruct entire scenes and can generate 3D videos accordingly, the resulting scenes remain static and lack dynamic elements.
We then shifted our focus to video datasets containing 3D information. Autonomous driving datasets such as Nuscenes [53] and WAYMO [54] incorporate point cloud videos, particularly bird's bird's-eye view footage. Nevertheless, the vehicles within these scenes are only captured 2D video. While this technically constitutes dynamic 4D scene, it does not align well with the objectives of this study, The dynamic relations traffic scenarios are relatively limited, and our goal is to develop a visual understanding model for embodied Al [58, 59, 60, 61] that captures 3D scenes from the agent's 's perspective, not a bird's bird's-eye view.
Another category of 3D videos uses RGB-D sequences input, which can be easily converted into point clouds. This data format aligns perfectly with the operation of intelligent agents, mimicking human perception. which captures continuous RGB images with depth. Thankfully, recent datasets like SAIL-VOS 3D [55]. HOI4D [56]. and EgoBody [57] have adopted this approach. While SAIL- VOS 3D uses synthetic data from the GTA game [21], the HOI4D dataset captures egocentric RGB-D videos of simple tasks, such as tool picking. On the other hand, the EgoBody dataset [57] records office activities like conversations, but lacks segmentation annotation and is primarily intended for human pose reconstruction. Despite its wealth videos, the object interaction in EgoBody is limited. In the medical domain, 4D-OR [60] excels in providing detailed depictions of surgical scenes, showcasing its specialized utility. To cater to a broader spectrum of research applications, we formulated the PSG-4D dataset, integrating the versatile strengths of the SAIL-VOS 3D [55] and HO14D [56] datasets.
4.2 Dataset Statistics
Figure presents selection of four video frames, drawn from both the PSG4D-GTA and PSG4D- HOI datasets. Each frame is an RGB-D video with corresponding panoptic segmentation annotations. Underneath each scene, we depict the associated scene graph and statistical word clouds. Annotators constructed these scene graphs triplets, complete with frame duration. The PSG4D-GTA datasct is

---

promee walking stdr
Lane bin T
road
chair
talking
1ooking
hund-1 mach ast for 30m1
1866-3112
Locket
carry cless
Reachout t putdown Pickup
hodding handhag-
per son ba barrier
bin oud 0U1--3055
0346-2552
bolding
peowed lo perwodd
hand-1 upen
OD Tound
persee persen- tallinn weed
CUP SpOETed
opening
hand-l peldoon bin11
TUTRITIPE
Gr Graspore open
(n) PSG4D.GTA (Synthetic, Third- Person View)
(b) PSG4D-HOI (Real World, Egocentric)
Figure 2: The Examples and Word Clouds of PSG-4D dataset. The PSG-4D dataset contains subsets, including (a) PSG4D-GTA selected from the SAIL-VOS 3D [20] dataset, and (b) PSG4D- HOI from HOI4D [22] dataset. We selected 4 frames of an example video from each subset. Each frame has aligned RGB and depth with panoptie segmentation annotation. The scene graph is annotated in the form of triplets. The word cloud for object and relation categories in each dataset is also represented.
particularly noteworthy for its composition: it contains 67 videos with an average length of 84 seconds, amounting 27, 7,700 RGB-D images, 28.3 billion point clouds, and comprises 35 object categories, and 43 relationship categories. This synthetic dataset was captured from a third-person perspective. In contrast, the PSG4D-HOI dataset is compiled from egocentric perspective, providing different context for analysis. It includes 2,973 videos with an average duration of 20 seconds, equating to 891,000 RGB-D images across 282 indoor scenes. This dataset includes 46 object categories and 15 object-object relationship categories, offering a diverse range of real-world data for the study. The combination of these datasets offers comprehensive understanding of 4D environments due to their complementary nature. A statistical overview of both datasets can be found in the final two rows of Table 1.
4.3 Dataset Construction Pipeline
As outlined in Section 4.1, the PSG4D-GTA is built upon the SAIL-VOS 3D dataset, while the PSG4D-HOI is derived from the HOI4D dataset. To adapt the SAIL-VOS 3D dataset for our purpose, commenced with comprehensive review of all 178 GTA videos within the dataset. This stage involved a meticulous elimination process to exclude videos containing NSFW content, resulting in refined pool of 67 videos. The SAIL-VOS 3D dataset, which is equipped with 3D instance segmentation, required additional annotation for background elements to integrate panoptic segmentation. Leveraging the PVSG annotation pipeline, employed event detection method [62] to isolate the key frames. The background elements within these key frames were subsequently annotated using the pre-annotation provided by the SAM model [63]. Upon completion of key frame annotations, the AOT method [64] was utilized to propagate the segmentation across the entire video sequence. The final step involved overlaying the instance segmentation on the stuff segmentation, thereby completing the process. The HOI-4D dataset, devoid of NSFW content, already provides a 4D panoptic segmentation. Consequently, we included all videos from the HOI-4D dataset in the PSG4D-HES HOI dataset without further modifications.
Upon completion of 4D panoptic segmentation annotation, we proceed annotate the dynamic scene graph according to the masks. Although HOI4D includes action annotation concerning the person, doesn't 't account for interactions between objects. Nevertheless, certain actions such as "pick up" appropriately considered predicates, and we automatically position the key object in the video to form bubject-verb-obiee triplet. Once the automatic-annotal dataset is prepared, we ask annotators to review and revise the pre-annotations to ensure accuracy. As SAIL-VOS 3D lacks all kinds of relational annotation, commence scene graph annotation from scratch. The entire annotation process is diligently executed by the authors around the clock.
5 Methodology
This section details unified pipeline PSG4DFormer for addressing the PSG-4D problem. As shown in Figure 3, our approach comprises two stages. The initial 4D panoptic segmentation stage aims to segment all 4D entities, including objects and background elements in Figure 3 (a), with the accurate
5

---

4D Panoptic Segmentation Model
4D Scene Graph
Relation Model
hond- bomle) mti 1883
hand- grnq sortie-
Option 1: RGB D Input
reah Lube
DiEE arived Graup
Spocaul marmanae
Panopiie
Tootlo
Option 2: Point Cloud Input
Train
Point Cloud Sogmentor
Feature Association
NUFTER
bab
Harum
malk Cobes
ground bruth relatins hand bottlet
(a) Frame-Level Panoptic Segmentation
(b) Tracking
(c) Inference (1) and Training (1) of Relation Model
Figure 3: Illustration of the PSG4DFormer pipeline. This unified pipeline supports both RGB-D and point cloud video inputs and composed of two main components: 4D panoptic segmentation modeling and relation modeling. The first stage seeks to obtain the 4D panoptic segmentation mask for each object, along with corresponding feature tube spanning the video length. This is accomplished with the aid of (a) frame-level panoptic segmentation and (b) tracking model. The subsequent stage (c) employs a spatial-temporal transformer to predict pairwise relations based on all feature tubes derived from the first stage.
temporal association in Figure 3 (b). We extract features for each object and obtain feature tubes according to tracking results for subsequent relation modeling in Figure (c).
5.1 4D Panoptic Segmentation Modeling
As specified in Section 3, given 3D video clip input, such as an RGB-D sequence of E R point cloud sequence of I R x66 the initial stage's goal is segment and track each pixel non-overlappingly, The model predicts a set of video clips with the output of (m; qi, pt) where denotes the tracked object mask tube, q: denotes the tracked feature tube, and p; represents the probability of the object belonging to each category. N the number of entities, encompassing things and stuff classes.
Frame-Level Panoptic Segmentation with RGB-D Sequence Given the dual input of RGB and depth images, we adopt
gate (SA-Gate) [65] to efficiently blend information from both modalities. This combined feature set, enriched with data from both inputs, is then fed into a robust Mask2Former [4] for frame-level panoptic segmentation. In the inference stage, at the frame I, given an RGB-D image I, the Mask2Former with SA-Gate directly outputs set of object query features of E Rd ; I,..., N, each of representing one entity the frame
Frame-Level Panoptic Segmentation with Point Cloud Sequence Apart from perceiving point cloud sequences directly, 3D point cloud coordinates can be calculated and converted from RGB-D data. This conversion involves computing the Normalized Device Coordinates (NDC) using the depth map and projecting the NDC to world coordinates using the transformation matrix provided. We retain only points with depth below a defined threshold \, discarding distant, less relevant elements like far-off mountains. To leverage texture information from the image, point cloud coordinates can be augmented with corresponding RGB values, creating a colorful point cloud representation P E M where M is the total number of points in a frame.
We employ DKNet [66], a state-of-thhe of-thherrt indoor segmentation method, our point cloud segmenta- tion network. It processes input point clouds with a 3D UNet-like [67] backbone and uses sparse convolutions [68] for feature extraction. DKNet localizes instance centroids based on a candidate mining branch and encodes each instance's 's information into an instance kernel k; E IRd. These instance kernels ( are used as the weights of few convolution layers to obtain the final instance masks.

---

Tracking After frame-level panoptic segmentation, we link each frame via using UniTrack [69] for tracking to obtain the final tracked video cubes for each clip for either modality input. Specifically, instead of incorporating an additional appearance model for tracking embedding extraction, we directly utilize the instance kernels {h: )== ] from the segmentation step of DKNet, or object query features {q: J from Mask2Former as the tracking embeddings for the association. We find that the instance kernels exhibit sufficient distinctiveness for tracking purposes, even when dealing with different objects belonging to the same semantic class. This is primarily because each instance kernel is designed to maximize the response for specific instance while suppressing the responses of all other instances, including those with the same semantic class. For a video sequence with the length T, the obtained 4D feature tubes are noted as Q; = (g/)
5.2 Relation Modeling: 4D Scene Graph Generation
The object query tubes Q, and mask tubes m, form a bridge between the first and second stages. These feature tubes first pass through spatial-temporal transformer encoder, which augments them with both global context information from the overall image and global temporal space.
Spatial-Tempores Transformer Encoder To infuse the feature tubes with additional temporal dimension information and characteristics from other objects in the scene, we draw inspiration from the Spatial-Temper Temporal Transformer [70]. A spatial encoder is initially employed. For all objects co-occurring at the same time t, a two-layer transformer encoder is applied to the input, comprising all object features specific to time frame L. The spatially-encoded feature tube updates the object feature tube into {Ø( )- Subsequently, a temporal transformer encoder updates each object feature tube along the temporal dimension T. By leveraging both the spatial and temporal encoders, i=1 ready for relation training.
obtain the final feature tube {+)
Relation Classification Training To train the relation model based the updated query tube, training set for relation training must be constructed. It is worth noting that the relation annotation in the training set is in the form of "object-1 relation object-2",, with the mask tube of both objects annotated. To start, we associate the updated query tube with ground truth objects. For each ground truth tube, we find the most suitable updated query tube by calculating the video Intersection over Union (vIOU) between ground truth mask tubes, and assign the query feature tube to the respective objects. frame-level predicate classification is conducted with the assistance of lightweight fully-connected layer. The inference of the relation classification component simply computes the relation probability between pairs of of and oz.
Experiments
Table 2 presents the results of experiments conducted on the PSG-4D dataset. For RGB-D sequences, an ImageNet-pree perttrne-pretr ResNet- serves as both the RGB and depth encoder. We set the training duration 12 epochs. The DKNet, trained from scratch, requires longer training period of 200 epochs. In the second stage, both spatial and temporal transformer encoders span two layers, and training continues for an additional 100 epochs. Besides the standard PSG4DFormer, we also examine variants with the temporal encoder removed (denoted as "/t") and the depth branch removed (denoted as "/d"). As a baseline, we use the 3DSGG model [18], which employs a GNN model to encode frame-level object and relation information, without considering temporal data.
RGB-D ps. Point Cloud Input Table 2 divided into two sections. The upper part (#1-#3) reports results from point cloud input, while the latter part (#4-#7) details results from the RGB-D sequence. appears that the RGB-D sequence generally yields better results than the point cloud sequence, particularly for the PSG4D-GTV dataset. This could potentially be attributed to the ResNet- 101 backbone used for the RGB-D data, which being pretrained on ImageNet, exhibits robust performance on complex datasets like PSG4D-GTA. Mcanwhile, the PSG4D-HOI dataset seems to offer more consistent scenario with abundant training data, thus narrowing the performance gap between the point cloud and RGB-D methods.
Significance Depth The results in Table also allow us to evaluate the importance of depth in the RGB-D method. Specifically, we designed a variant of PSG4DFormer (marked as "Id")) that
7

---

Table 2: Main Results PSG4D. Experimental results reported both the PSG4D-GTM and PSG4D-HOI datasets. In addition to comparing with traditional 3DSGG methods, we conduct experiments to compare the PSG4DFormer and its variants. This includes version with the temporal encoder removed (denoted as "/t") and one with the depth branch removed (denoted as "/d").
Input Type
PSG4D-GTA
PSGID-HOI
Method
R/mR @0
R/mR @0
@ 100
R/mR@
R/mR@s 50
R/mR 100
Point
#1 3DSGG [18]
1.78
2.16 0.79
0.85
3,46/ 2.19
3.14
1.96/2884
Cloud
PSG4DFormer
2.25 1.03
2.67 1.72
3.04/2.05
3.26/ 2.04
3,16/2.35
4.18 2.64
Sequence
#3 PSG4DFormer
4.33
4.83 2.93
3.22/3.13
5.16/3.10
5.61 / 3395
6.76/4.17
#4 3DSGG [18]
0.99
2.46/ 1.01
3.81/ 1.45
4.23/2.19
4.47 2..1
4.86 2.41
RGB-D Sequence
#5 PSG4DFormer
4.43/1.33
4,89 / 2.42
5.26/2.83
4.44/2.37
4.83 /2.43
5,21 2.84
#6 PSG4DFormer'*
4.40
4.91 / 1.93
5.49
5,49
5.97 / 3,92
6.43 /4.21 6.28 / /497
#7 PSG4DFormer
6.68 /3.31
7.17 3.85
7.22/
5.62 3.65
6.16
doesn't 't utilize the depth branch. In other words, both the RGB encoder and the SA-Gate are removed, turning the pipeline into a video scene graph generation pipeline. The performance of this variant is inferior compared to the original, which highlights the significance of depth information the scene graph generation task.
Necessity of Temporal Attention Table 2 includes two methods that do not utilize temporal attention. Specifically, the 3DSGG baseline learns interactions between static object features using graph convolutional network, while PSG4DFormer' removes the temporal transformer encoders. The results demonstrate that ignoring the temporal component could lead to sub-optimal outcomes, emphasizing the importance of temporal attention in 4D scene graph generation.
7 Real-World World Application
This section illustrates the deployment of the PSG-4D model in real-world application, specifically within a service robot. It extends beyond theoretical concepts and computational models, delving into the practical integration and execution of this cutting-edge technology. As shown in Figure 4, the focus here is to demonstrate how the robot leverages the PSG-4D model (pretrained from PSG4D-HOI, RGB-D input) to interpret and respond to its surroundings effectively.
Interaction with Large Language Models The recent advancements in large language models (LLMs) have displayed their exceptional capabilities in reasoning and planning [71]. LLMs have been utilized as planners in numerous recent studies to bridge different modalities, paving the way for more intuitive and efficient human machine interaction [72]. In this work, we employ GPT-4 [71], as the primary planner. Designed to align with human instruction, GPT-4 communicates with the robot by translating the raw scene graph representations into comprehensible human language. Therefore, the interaction begins with the prompt, "I am service robot. For every 30 seconds, I will give you what I have seen in the last 30 seconds. Please suggest me what I could serve." " Subsequently, every 30 seconds, the robot engages with GPT-4, providing update: "In the past 30s, what I captured is: <from start_ tirtttttmtttte to and_time, object-1 relation object-2>, <... < ..... This enables GPT-4 to analyze the situation and provide appropriate feedback.
Post-Processing for Execution The effective deployment of the PSG-4D model necessitates robust set of predefined actions that the robot can execute. Currently, the action list includes tasks such as picking up litter and engaging in conversation with individuals. After GPT-4 provides its suggestions, it further prompted to select a suitable action from this predefined list for the robot to execute. However, the flexibility of this system allows for the expansion of this action list, paving the way for more complex and varied tasks in the future. To encourage community involvement and the development of fascinating applications, we also release the robot deployment module alongside the PSG4D codebase. The demo robot is priced at approximately $1 1.2K and comes equipped with an RGB-D sensor, microphone, speakers, and robotic arm.

---

(a) The RGB-D sequence that is captured by the robot.
(d) Robot Reaction
(b) PSG-4D Parsing
Reasoning & Planning
am service robot, In the past 30s, what captured is: E= 1. Is there anything could serve?
person-1 drinking from bottle-1
PS040
20.0
person-1 throwing bottle-1
20.0x-30.00 30.0s
Cleanup: Reminder:
bottle-1 ground
Figure Demonstration of Robot Deployed with the PSG-4D Model. The service robot interprets the RGB-D sequence shown in (a), where man is seen drinking coffee and subsequently dropping the empty bottle on the ground. The robot processes this sequence, translating it into a 4D scene graph depicted in (b). This graph comprises set of temporally stamped triplets. with each object associated with a panoptic mask, accurately grounding updates its PSG4D) GPT-4, awaiting feedback and instructions. In this scenario, GPT-4 advises the robot to clean up the discarded bottle and remind the man about his action. This directive translated into robot action, visualized in (d).
3D space. The robot regularly
Conclusion, Challenges, and Outlook
This paper presents a novel and demanding extension to the traditional scene graph generation, the 4D Panoptic Scene Graph Generation, which incorporates the spatio-temporal domain into the framework. We introduce comprehensive framework, the PSG4DFormer, capable of processing both RGB-D and point cloud sequences. The successful deployment of this pipeline in a practical service robot scenario underscores its potential in real-world applications. However, these achievements also highlight the nascent state of this field, emphasizing the necessity for continued advancements to fully exploit the potential of 4D Panoptic Scene Graph Generation.
Challenges Despite encouraging results, we have also revealed several persistent challenges in the realm of 4D Panoptic Scene Graph Generation. Through our demonstration, we found that current models, whether derived from PSG4D-GTA or PSG4D-HOI, can handle only simple scenes and falter when faced with more complex real-world environments. Notably, there exist robust models trained in the 2D world. Finding effective and efficient strategies adapt these models to the 4D domain presents compelling direction for future exploration.
Outlook Future work in this field presents several intriguing trajectories. There is pressing need for more efficient algorithms for 4D Panoptic Scene Graph Generation, which can handle larger and more diverse environments. Equally important is the creation of comprehensive and diverse datasets that would allow more rigorous evaluation and foster advancements in model development. Particularly noteworthy is recent Digital Twin dataset [73], which promises a high level of accuracy and photorealism, aligning seamlessly with the objectives of PSG4D. This dataset will be incorporated the third subset of the PSG4D dataset, readily accessible from our codebase. addition to robotics, as demonstrated by the practical application of PSG4DFormer, we are also exploring its potential as an autonomous player in the GTA game, Actually, our recent endeavor Octopus [58] strives to complete GTA missions by employing visual-language programmer to generate executable action code. In contrast to the previously passive task completion, the application in this paper actively perceives and understands the environment, showcasing shift towards autonomy in robotics. Furthermore, Octopus [58] utilizes 4D scene graph structure to capture environmental information during the disual-languaa programmer training, exemplifying practical application of the PSG4D modality.

---

We eagerly anticipate the future progress in the field of 4D Panoptic Scene Graph Generation and its potential to revolutionize our understanding of weal-world dynamics.
Potential Negative Societal Impacts This work releases a dataset containing human behaviors, posing possible gender and social biases inherently from data. Potential users are encouraged to consider the risks of oversighting ethical issues in imbalanced data, especially in underrepresented minority classes. Nevertheless, all the NSFW content is removed from the dataset.
Acknowledgement
This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP2021-0000 1-002), NTU NAP, and under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, the National Key R&D Program of China under grant number 20222D0161501, as well as cash and in-kind contribution from the industry partner(s).
References
Xiaojian Ma, Silong Yong. Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022.
[2] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson. Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03777 2023. 2
[3] Sonia Raychaudhuri, Tommaso Campari, Unnat Jain, Manolis Savva, and Angel X Chang. Reduce, reuse, recycle: Modular multi-object navigation. arXiv preprint axXiv:2304.0360/0 2023.
[4] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. 2022. 2,
[5] Xiangtai Li, Haobo Yuan, Wenwei Zhang, Guangliang Cheng, Jiangmiao Pang, and Chen Change Loy. Tube-link: A flexible cross tube baseline for universal video segmentation. In ICCV, 2023. 2
[6] Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmiao, Kai Chen, Ziwei Liu, and Chen Change Loy. Transtormer- visual segmentation: survey. arXiv pre-print, 2023. 2
[7] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, 2017. 2,
[8] Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR, 2019. 2,
[9] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, 2018. 2,
[10] Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal. Energy-based learning for scene graph generation. In CVPR, 2021. 2, 3
[11] Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, and Yin Li. Learning to generate scene graph from natural language supervision. In ICCV, 2021. 2,3
[12] Jingkang Yang, Yi 7he Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. European Conference on Computer Vision, pages 178-196. 196. Springer, 2022. 2, 3
[13] Jingkang Yang. Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, and Ziwei Liu. Panoptic video scene graph generation. In CVPR, 2023.
2,
[14] Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection. ACM MM, 2017. 2,
[15] Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user-generated videos. In ICMR, 2019. 3,
[16] Matthew Fisher, Manolis Savva, and Pat Hanrahan. Characterizing structural relationships in scenes using graph kernels. In ACM SIGGRAPH 2011 papers, pages 1-12. 2011. 2,
10

---

[17] Robert F Tobler. Separating semantics from rendering: a scene graph based architecture for graphics applications. The Visual Computer, 27(6-8):687-655, 2011. 2,
[18] Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3961-3970, 2020. 2, 3, 4, 7,8
[19] Shoulong Zhang, Aimin Hao, Hong Qin, et al. Knowledge-inspired 3d scene graph prediction in point cloud, Advances in Neural Information Processing Systems, 34:1 14:186200-1632. 2021. 2, 3
[20] Y.-T. Hu, J. Wang. R. A. Ych, and A. G. Schwing. SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Data. Proc. CVPR, 2021. 2,5
[21] Grand theft auto v. 2014. 2.4
[22] Yunze Liu, Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 21013 2-0222, June 2022. 5,5
[23] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, Michael Bernstein, and Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
[24] Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19486- 19496, 2022.
[25] Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation. arXiv preprint arXiv:2201./ /460, 2022. 3
[26] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Nicbles. Action genome: Actions as compositions of spatio-temporal scene graphs. In CVPR, 2020. 3
[27] Jinghao Wang, Zhengyu Wen, Xiangtai Li, Zujin Guo, Jingkang Yang, and Ziwei Liu. Pair then relation: Pair-net for panoptic scene graph generation. arXiv preprint arXiv:2307. 230.29337,00699, 2023. 3
[28] Chengyang Zhao, Yikang Shen, Zhenfang Chen, Mingyu Ding, and Chuang Gan. Texipsg: Panoptic scene graph generation from textual descriptions. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2839-28500 2023. 3
[29] Zijian Zhou, Miaojing Shi, and Holger Caesar. Hilo: Exploiting high low frequency relations for unbiased panoptic scene graph generation. arXiv preprint arXiv:2303.159% 2023. 3
[30] Julian Lorenz, Florian Barthel, Daniel Kienzle, and Rainer Lienhart. Haystack: A panoptic scene graph dataset to evaluate predicate classes. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 62-70, 2023. 3
[31] Jingkang Yang, Zheng Ma, Qixun Wang. Xiaofeng Guo, Haofan Wang, Ziwei Liu, Wayne Zhang, Xing Xu, and Hai Zhang. The psg challenge: towards comprehensive scene understanding. National Science Review, 1666) 2023.
[32] Xiangtai Li, Wenwei Zhang, Jiangmiao Pang, Kai Chen, Guangliang Cheng, Yunhai Tong, and Chen Change Loy. Video k-net: A simple, strong, and unified bascline for vidco segmentation. In CVPR, 2022.
[33] Jaewon Bae, Dongmin Shin, Kangbeen Ko, Juchan Lee, and Ue- Hwan Kim. A survey on 3d scene graphs: Definition, generation and application. In Robot Intelligence Technology and Applications 7: Results from the 10th International Conference Robot Intelligence Technology and Applications, pages 136-147. Springer, 2023.
[34] Uc-Hwan Kim, Jin-Man Park, Tack-Jin Song, and Jong-Hwan Kim. 3-d scene graph: A sparse and semantic representation of physical environments for intelligent agents. IEEE transactions on cybernetics, 50(12):4921-4933 2019. 3
[35] Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In ICCV, 2019. 3
11

---

[36] Charles R Qi. Hao Su. Kaichun Mo. and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In CVPR, 2017.
[37] Antoni Rosinol, Andrew Violette. Marcus Abate, Nathan Hughes. Yun Chang, Jingnan Shi, Arjun Gupta. and Luca Carlone. Kimera: From slam to spatial perception with 3d dynamic scene graphs. The International Journal of Robotics Research, 40(12-14):1110-5544 2021.
[38] Chung Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scenegraphfusion: Incremental 3d scene graph prediction from rgb-d sequences. In Proceedings of the JEEF/CVE Conference Computer Vision and Pattern Recognition, pages 7515-7525, 2021.
[39] Hema Koppula and Ashutosh Saxena. Learning spatio-temporal structure from rgb-d videos for human activity detection and anticipation. In International conference on machine learning, pages 792-800. PMIR, 2013. 3
[40] Qian Xie, Oussama Remil, Yanwen Guo, Meng Wang, Mingqiang Wei, and Jun Wang. Object detection and tracking under occlusion for object-level rgb-d video segmentation. IEEE Transactions on Multimedia, 20(3):580- 59200-9222222 2017. 3
[41] Guyue Zhang. Jun Liu, Hengduo Li, Qiu Chen, and Larry Davis. Joint human detection and head pose estimation via multistream networks for rgb-d vidcos. IEEE Signal Processing Letters, 24(1 (4(11))1666 1670, 2017. 3
[42] David Weikersdorfer, Alexander Schick, and Daniel Cremers. Depth-adaptive supervoxels for rgb-d vidco segmentation. In 2013 IEEE International Conference on Image Processing, pages 2708-2712. IEEE, 2013. 3
[43] Huazhu Fu, Dong Xu, and Stephen Lin. Object-based multiple foreground segmentation rgbd video. IEEE Transactions on Image Processing, 26(3):141 1428-147444 2017. 3
[44] Steven Hickson, Stan Birchfield, Irfan Essa, and Henrik Christensen. Efficient hierarchical graph-based segmentation of rghd videos. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 344-351, 2014.
[45] Numair Khan, Qian Zhang, Lucas Kasser, Henry Stone, Min H Kim, and James Tompkin. View fiew-consistent 4d light field superpixel segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7811-7819, 2019. 3
[46] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-tempone convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3075-3084, 3019.
[47] Wenwei Zhang, Hui Zhou, Shuyang Sun, Zhe Wang, Jianping Shi, and Chen Change Loy. Robust multi- modality multi-object tracking. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2365-2374, 2019.
[48] Xinshuo Weng, Yongxin Wang. Yunze Man, and Kris M Kitani. Gnn3dmot: Graph neural network for 3d multi-object tracking with 2d-3d multi-feature learning. In P Proceedings of the IEEESCVF Conference on Computer Vision and Puttern Recognition, pages 6499-6508, 2020. 3
[49] Xinshuo Weng, Jianren Wang. David Held, and Kris Kitani. 3d multi-object tracking: A baseline and new evaluation metrics. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020.
pages
[50] Ankit Goyal, Kaiyu Yang. Dawei Yang, and Jia Deng. Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d. Advances in Neural Information Processing Systems, 2020, 4
[51] Angela Dai, Angel Chang, Manolis Savva, Maciej Halher, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5828-5839 2017. 4
[52] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: learning from rgb-d data in indoor environments. arXiv preprint arXiv: 2017.
[53] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11633
2020. 4
12

---

[54] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard. Vijaysai Patnaik, Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference un computer vision and pattern recognition. pages 2446 2254, 2020, 4
[55] Mhan-Ting Hu, Jiahong Wang, Raymond Ych, and Alexander G Schwing. Sail-vos 3d: A synthetic dataset and baselines for object detection and mesh reconstruction from video data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1418-1428, 2021. 4
[56] Yunze Liu, Yun Liu, Che Jiang. Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang. and Li Yi. Hoi4d: A egocentric dataset for category- human- interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013- 21022, 2022. 4
[57] Siwei Zhang, Qianli Ma, Yan Zhang, Zhiyin Qian, Taein Kwon, Mare Pollefeys, Federica Bogo, and Siyu Tang. Egobody: Human body shape and motion of interacting people from head-mounted devices. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VI. pages 180 200. Springer, 2022. 4
[58) Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li. Ziyue Wang, Chencheng Jiang, Haoran Tan, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, al. Octopus: Embodied vision-language programmer from environmental feedback. arXiv preprint
2023. 4,
[59] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable task planning. arXiv preprint arXiv:2307.061333 2023. 4
[60] Ege Özsoy, Evin Pınar Ömek. Ulrich Eck. Tobias Czempiel, Federico Tombari, and Nassir Navab. 4d-or: Semantic scene graphs for or domain modeling. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 475-485. Springer, 2022. 4
[61] Saeid Amiri, Kishan Chandan, and Shiqi Zhang. Reasoning with scene graphs for robot planning under partial observability. IEEE Robotics and Automation Letters, 7(2):5500-5567, 2022. 4
[62] Kiyotaka Oisuji and Yoshinobu Tonomura. Projection detecting filter video cut detection. Proceedings of the first ACM international conference on Multimedia, pages 251-257, 1993. 5
[63] Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint urXiv:2304.02643, 2023. 5
[64] Zongxin Yang. Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. In NeurlPS, 2021. 5
[65] Xiaokang Chen, Kwan- Yee Lin, Jingbo Wang, Wayne Wu, Chen Qian, Hongsheng Li, and Gang Zeng. Bi-directional cross-modality feature propagation with scparation-and and-aggregation gate for rgb-d semantic segmentation. In European Conference on Computer Vision (ECCV), 2020. 6
[66] Yizheng Wu, Min Shi, Shuaiyuan Du, Hao Lu, Zhiguo Cao, and Weicai Zhong, 3d instances as 1d kernels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, P Proceedings, Pari XXIX, pages 235-252. Springer, 2022. 6
[67] Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015. 6
[68] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9224 020229922,220 2018. 6
[69] Zhongdao Wang, Hengshuang Zhao, Ya-Li Li, Shengjin Wang. Philip HS Torr, and Luca Bertinetto. Do different tracking tasks require different appearance models? NeurIPS, 2021. 7
[70] Yuren Cong. Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. Spatial- temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international
conference on computer vision, pages 16372-16382, 2021. 7
[71] OpenAL. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. 8
13

---

[72] Wenlong Huang, P. Abbeel. Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ArXiv, abs/2201.07207, 2022.
[73] Xiaging Pan, Nicholas Charron, Yongqian Yang. Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: A new benchmark dataset for egocentric machine perception. Proceedings of IEEE/CVF International Conference on Computer Vision, pages 20133-201-42 2023. 9
14


Reviews:
[{"review_id": "pCb7Q2qFId", "reviewer": "Reviewer_PY8p", "summary": "This paper proposes a new task, named 4D panoptic scene graph generation (PSG-4D). At the same time, a dataset, including PSG4D-GTA and PSG4D-HOI, is collected for this task, and a benchmark is provided to evaluate the performance of this task. PSG-4D offers a more comprehensive scene understanding, bringing more thought and possibilities for the practical implementation of scene understanding. The paper shows the potential application of PSG-4D in Robot Deployed scenarios by combining it with LLM, which is really meaningful.", "questions": "Please refer to the weaknesses. ", "limitations": "Not apply.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. A new task, dataset, and benchmark are proposed to solve the PSG-4D problem, which provides a more comprehensive scene understanding in the real world.\n2. This paper shows the potential applications of the PSG-4D task, making it appear more practically significant.", "weaknesses": "The sample size of the dataset and the number of relationship labels are somewhat limited, and the recall provided by the benchmark is really low, which may limit its practical application.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for recognizing the contributions of our work on \"4D Panoptic Scene Graph Generation\". We are grateful for your feedback and would like to provide additional information to address your concerns.\n\n**Regarding Weaknesses:**\n\n1. **Sample size of the dataset**:\n    - As a continuous project, we are fully aware of the dataset size limitations and have plans for its expansion. We aim to provide the research community with a growing, high-quality dataset to further the advancements in PSG-4D.\n    - We are currently in the process of incorporating the **[Digital Twin](https://www.projectaria.com/datasets/adt/)** dataset, an egocentric dataset with high-quality panoptic segmentation annotation and RGBD signals, which aligns well with PSG4D requirements. Our vision for the future is to integrate more such high-quality datasets to provide a more comprehensive setting for PSG-4D research.\n2. **Number of relationship labels**:\n    - The design of our relationship labels was to capture comprehensive visual expressions from the data. Based on our dataset, we believe that the existing relationships can aptly describe most scenarios.\n    - Our method of introducing relations begins with a pre-defined relation set, typically derived from the PSG dataset. During the annotation phase, if annotators identify potential new relations, they coordinate with the primary authors to evaluate and decide on their inclusion. This process ensures that our relation set remains both diverse and relevant.\n    - We recognize the potential for encountering novel interactions or relations when testing beyond our current dataset. This recognition points towards an exciting direction for future research: open-vocabulary relation predictions. We hope this challenge invites the community to explore and innovate further in this domain.\n3. **Benchmark recall**: Our goal with introducing PSG4DFormer as a baseline was to provide the community with a starting point. While we acknowledge the recall's current limitations, we believe that the evolving nature of our dataset and the advent of new methodologies will lead to improved recall metrics.\n\nWe hope that the above elucidation addresses your concerns related to the dataset size, the relationship labels, and the recall performance. We remain open to further discussions and clarifications on any aspect of our work."}, {"review_id": "1dXw0pLksr", "reviewer": "Reviewer_2QCE", "summary": "This paper introduces a new task of scene graph generation that predicts 4D panoptic scene graphs from RGB-D video sequences. This paper also provides a high-quality PSG-4D dataset including PSG4D-GTA with third-view synthetic data and PSG4D-HOI with egocentric real-world data. As the third contribution, this paper also proposes a unified two-stage model to showcase real-world scenarios.", "questions": "It may be a bit easier for readers to digest the table if the numbers from the best-performing model can be highlighted.", "limitations": "The authors have adequately addressed the limitations and broader impacts.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The new task introduced by this paper is very important. It will help to enable embodied AI to better understand the 4D dynamic real world with complex object-object relations.\n\n2. As illustrated in Tab. 1, the collected dataset features dynamic scenes with much richer data type than previous datasets, covering both egocentric and third-view data. If released, this new dataset will greatly benefit the community.\n\n3. In section 7, the paper also illustrates how to deploy the PSG-4D model in a real-world application by interacting with large language models. This shows the potential of exploiting the introduced dataset to solve practical problems in the real world.", "weaknesses": "1. The introduced datasets are based on existing datasets (ie, SAIL-VOS 3D and HOI4D datasets). Therefore, they only cover  synthetic data for third-person views. This may cause the imbalanced development of algorithms for egocentric and third-view data.\n\n2. L187-190 briefly mentions how the relation annotations are first automatically prepared and then reviewed and revised. Could the authors elaborate and share a bit more whether more design choices have been made to make the process more efficient or this is a very tedious/time-consuming task (if so, what is the average/total time needed for each scene or all scenes)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate your comprehensive feedback on our paper, \"4D Panoptic Scene Graph Generation.\" Based on your comments, we'd like to further clarify our intentions and the processes involved:\n\n**Regarding Weaknesses:**\n\n1. **Datasets based on existing datasets**:\n    - The motivation behind the development of PSG4D is to set the stage for the potential evolution of comprehensive robot perception or embodied AI. In these future scenarios, the agent, whether operating within an egocentric view in the real world or as a game agent in a virtual environment, would need to perceive and understand its 4D surroundings accurately. Given this goal, the emphasis of PSG4D is not necessarily to ensure a balanced representation of both egocentric and third-person view data in our current work. Instead, it is geared towards shaping the potential next steps of agent and robot vision, where an egocentric view becomes highly pertinent. This approach supports the incorporation of datasets like **[Digital Twin](https://www.projectaria.com/datasets/adt/)**, which is rich in egocentric data.\n2. **Relation Annotations Process**:\n    - Indeed, the relation annotation process is a painstaking endeavor. To streamline and improve the annotation efficiency for SAIL-VOS 3D, we designed a method where annotators first describe the video in a few sentences. This allows them to grasp the essence and primary interactions in the content. By then annotating relations based on these descriptions, we can ensure that the main interactions and relations are captured, while minimizing trivial positional relations that might not be as informative.\n    - The manual nature of this task requires careful attention, especially when determining the start and end frames for each relation. On average, a 60-second video from SAIL-VOS 3D requires about 30 minutes for annotation. For the HOI4D dataset, given that it already contains annotations (albeit with quality variations and missing object-object relations), our annotators focused on refining and supplementing these existing annotations. Typically, a 20-second video from HOI4D takes about 5 minutes to revise.\n\n**Addressing Questions**:\n\n- **Highlighting best-performing model in tables**: We are grateful for this suggestion. In our revised version, we will highlight the leading results for better clarity and ease of understanding.\n\nIn summary, our overarching aim with this work is to bolster the capabilities of future embodied AI systems, ensuring they can navigate and understand dynamic 4D environments, be it real or virtual. We hope our clarifications address your concerns, and we are enthusiastic about the potential positive impact of our work on the AI community."}, {"review_id": "vY43Zysuv3", "reviewer": "Reviewer_CrvQ", "summary": "The author introduces a new task called the 4D Panoptic Scene Graph (PSG-4D), which focuses on predicting 4D panoptic scene graphs from RGB-D video sequences. To support this task, the author provides a unique PSG-4D dataset, encompassing diverse viewpoints through two subsets: a third-view synthetic subset (PSG4D-GTA) and an egocentric real-world subset (PSG4D-HOI), each containing detailed frame labels To address the PSG-4D task, the author proposes a unified two-stage model, the PSG4DFormer, which comprises a feature extractor and a relation learner.", "questions": "See those raised in Weaknesses.", "limitations": "Yes, though I’d still would like to hear from the author about the generalizability (even for a specific domain) and external validity of the proposed dataset and framework.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is generally well written and easy to follow.\n- The new task/datasets seem novel, and well supported by the proposed PSG4DFormer framework.", "weaknesses": "- Less relevant to NeurIPS. Perhaps venues like CVPR/ICCV/SIGGRAPH would be a better fit?\n- While I recognize the dataset is one the major contribution here, the novelty of this work is not entirely clearly to me.\n- Justification for the choice of dataset composition described in Section 4 seems insufficient. Why we choose this specific composition of dataset beyond convenience? Do we know model trained/evaluated on this dataset can generalize nicely to other use cases (or a specific vertical of applications?)\n- The experiments seem to be rather limited as only one baseline, though recent and relevant, is evaluated on this very proposed dataset. Are there smaller, ad-hoc dataset that can be used to evaluate the proposed framework to support its external validity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# \n\nThank you for your constructive feedback. We deeply appreciate the time and effort you invested in reviewing our work, and we would like to address each of your raised concerns.\n\n**1. Relevance to NeurIPS:**\nWhile we understand the viewpoint on the relevance of our work to specific venues, we believe our introduction of a novel task, dataset, and the accompanying framework aligns well with NeurIPS’ emphasis on innovative methods and broader impacts. The 4D application of scene graphs, in essence, tackles a pressing deep learning problem with implications spanning a wide array of AI domains.\n\n**2. Clarity of Novelty:**\nWe would like to elucidate that our proposed PSG4D is not merely an enhancement of existing scene graphs. While 3D scene graphs encapsulate the spatial dimension, PSG4D captures the dynamic 4D world, unearthing new opportunities and challenges. This innovative shift is pivotal as it sets the stage for potential future works, especially in the burgeoning field of embodied AI.\n\nFor instance, PSG4D-GTA is designed with the vision of enhancing game AI agents. Such agents, equipped with the capability to perceive and comprehend their in-game surroundings holistically, can perform actions with a level of sophistication previously unattained. Similarly, PSG4D-HOI was tailored to be compatible with emerging technologies like egocentric smart glasses (e.g., Apple Vision Pro). By offering 4D perception, the AI can assist users in dynamic, real-world environments with unprecedented accuracy. Our ongoing efforts to incorporate the **[Digital Twin](https://www.projectaria.com/datasets/adt/)** dataset resonate with the same vision.\n\n**3. Dataset Composition Justification:**\nOur selection was not predicated solely on convenience. The synthetic subset, PSG4D-GTA, with its richness in scenes, ensures robust training and sets a strong foundation for applications in game AI, as mentioned. On the other hand, PSG4D-HOI, capturing naturalistic human-object interactions, becomes a cornerstone for real-world applications like smart glasses. Their combined strengths balance controlled and realistic scenarios, aiming at a wide applicability spectrum.\n\n**4. Experiments:**\n\nWe sincerely appreciate your suggestion regarding external validation on other datasets. Nonetheless, we wish to emphasize the inherent challenges tied to finding apt scene graph datasets that would align with the novel nature of PSG4DFormer. Most traditional 3D scene graph datasets primarily focus on static settings, devoid of dynamic objects, which essentially contrasts what PSG4DFormer is specifically designed for.\n\nWe concede that verifying our methodology across various datasets would indeed strengthen our assertions. However, our primary intention with this submission is to pioneer a new task, furnish the community with robust datasets and benchmarks, and set the groundwork for future research. The novelty and ambition of our work meant we ventured into territories with limited available data that fit the exact criteria our model was built for.\n\nIn recognizing the importance of varied datasets for holistic model validation, we're currently making strides in incorporating **[Digital Twin](https://www.projectaria.com/datasets/adt/)**, hopeful that its inclusion will bolster the validity of all PSG4D methodologies.\n\nOur aspiration is that our introduction will spur the community's innovation in areas like robot perception and game/embodied AI development, as elaborated earlier. As the field matures, we are optimistic about more comprehensive benchmarks and comparative studies."}, {"review_id": "GCMxvhemlH", "reviewer": "Reviewer_z3XZ", "summary": "This paper introduces the concept of the 4D Panoptic Scene Graph (4D-PSG) as an approach for achieving a high-level visual understanding of 4D scenes. Additionally, the authors have developed two datasets annotated with 4D-PSG information. Furthermore, the authors propose a PSG4DFormer method for constructing the 4D-PSG from video data. The utilization of the 4D-PSG has the potential to enhance a robot's ability to comprehend its environment and appropriately respond to it.\n", "questions": "None\n", "limitations": "The authors have discussed the societal impact.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well written and easy to follow.\n2. This paper contributes to the open source community they would release dataset, codebase and robot module.", "weaknesses": "1. The novelty of this work is limited. The 4D-PSG is a simple extension from the existing 3DSG. The proposed PSG4DFormer is also a simple combination of the 4D Panoptic Segmentation Model and the Relation Model.\n2. Although the authors mentioned the potential application of the 4D-PSG, no experiments are provided to substantiate this claim. The authors really need to add these experiments. Also, a strong baseline should be provided for comparison.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We sincerely appreciate the time you have taken to review our work and provide constructive insights.\n\n**1. Novelty Concerns:**\nWe understand your concerns about the perceived linear progression from 3DSG to our proposed PSG4D. However, it's crucial to elucidate that the evolution from 3DSG to PSG4D is not just a dimensional increment but a paradigm shift in scene understanding. While 3DSG provides a hierarchical representation of static three-dimensional environments, PSG4D delves into dynamic realms, capturing the fluidity of real-world interactions over time.\n\nFor instance:\n\n- **Dynamic Interactions:** While 3DSG may depict a person next to a bicycle, PSG4D can elucidate sequences such as a person parking a bicycle, then tying their shoelaces, and eventually walking away. Such representations are pivotal for systems requiring predictive or reactive capabilities.\n- **Temporal Relations:** Objects and entities don't exist in isolation. They interact, and these interactions vary over time. Consider two people meeting; 3DSG would represent them standing together, but PSG4D can provide a timeline: Person A walking towards Person B, both shaking hands, and then engaging in a conversation. Such granularity is crucial for applications in surveillance, human-robot interaction, or even cinematic content generation.\n- **Real-world Ambiguities:** Dynamic scenes introduce challenges like occlusion, rapid motion, and interaction complexities. PSG4D not only captures entities but also traces these entities through the ambiguities, ensuring consistent scene understanding.\n\nThe transition to a 4D dynamic world from a static 3D realm indeed paves the way for numerous new opportunities and tasks, making this extension substantive and not merely incremental.\n\n**2. Experiments and Baselines:**\nWhile we recognize the need for further experimental demonstrations, it's essential to highlight our existing work with baselines and experimental variations.\n\n- **3DSGG as a Baseline:** We have indeed incorporated 3DSGG as a fundamental baseline. Specifically, 3DSGG leverages the same panoptic segmentation as our model but diverges in the second stage. Instead of the temporal aspect, it uses a Graph Neural Network (GNN) to encode frame-level object and relation information, inherently missing out on the dynamic temporal nuances.\n- **Model Variations for Analysis:** As elucidated in our discussions (refer Table 2), we did not merely present PSG4DFormer in isolation. We included variations of our model, such as PSG4DFormer/t, which removes the temporal transformer encoders. The comparative outcomes with these models underscore the pivotal role of temporal attention in 4D scene graph generation.\n- **Future Work:** We envision our provided baselines of 3DSGG and the newly introduced PSG4DFormer as foundational building blocks. We are optimistic that they will catalyze further research, fostering advancements in scene graph generation that leverage both spatial and temporal dimensions.\n\nWe genuinely hope these clarifications underline our commitment to rigorous experimentation and our intention to advance the field both through our current contributions and by laying the groundwork for subsequent research.\n\nThank you once again for your consideration and valuable insights."}, {"review_id": "maJL3LLUM8", "reviewer": "Reviewer_4B6f", "summary": "This paper offers a novel perspective on understanding a 4D environment via the introduction of 4D Panoptic Scene Graph (PSG-4D). PSG-4D transforms 4D sensory data into nodes, representing entities with precise location and status data, and edges that capture temporal relationships. The authors have developed a high-quality PSG-4D dataset, which includes 3K RGB-D videos with a total of one million frames. Each frame comes with 4D panoptic segmentation masks and detailed, dynamic scene graphs. In order to solve PSG-4D, they propose PSG4DFormer, a model capable of predicting panoptic segmentation masks, tracking masks across the time axis, and generating corresponding scene graphs via a relation component. Extensive experiments demonstrate that their method forms a robust baseline for PSG-4D. Furthermore, the paper provides a real-world application example demonstrating how dynamic scene understanding can be achieved by incorporating a large language model into the PSG-4D system.", "questions": "Please refer to the weaknesses.", "limitations": "The authors mentioned potential challenges and societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is significant for several reasons: it formulates a new problem, annotates a novel dataset, and provides simple baselines. These aspects could advance future research and encourage applications in robotic systems. Even though the research appears to be a logical progression from panoptic scene graph generation and panoptic video scene graph generation, the introduction of a new data modality provides potential for a broader range of applications. SGG in 4D space is an area that has not been extensively explored, to my knowledge.\n2. The dataset selection and construction pipeline is detailed and appears to be reasonable.\n3. The paper is well articulated and easy to comprehend. The incorporation of the large language model (LLM) in the proposed application seems to be an intriguing and viable addition.", "weaknesses": "1. I have a significant concern with the Experimental section of the paper. The task of Scene Graph Generation is intricate, particularly when extended to a 4D scenario. In building a pipeline to solve this problem, it is reasonable to use various components each serving a unique role. However, it remains unclear which component holds the most promise for future research focus. While the ablation study in Table 2 gives us insights into the depth map data and naïve temporal modeling, it would be valuable for the authors to provide more detailed analysis and insights.\n2. In Line 112, there is a minor formatting error: 'M' should be formatted as an equation rather than plain text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The submission presents a representation, named 4D Panoptic Scene Graph Generation, which extends the 3D scene graph into the temporal domain. 3D scene graphs have been attracting interest as one solution to filling the gap between lower-level information that is often extracted from sensory data, like images, and higher-level information that is needed for performing action. An extension to incorporate the temporal domain is naturally useful. The paper includes a number of relevant components in this area, including a dataset, baselines, and demonstration of applications. 

All reviewers are positive. The paper has unanimous acceptance recommendations. The rebuttal was also considered and generally appreciated by the reviews. The less positive reviewers cited issues with novelty and relevance to NeurIPS compared to vision conferences. The AC has considered them and judged the extension to the temporal domain presents significant enough challenges to not view the submission as too incremental. 

As the paper poses to be the first to introduce the 4D extension, the main question for the future impact of the paper, and generally the proposed direction, is how far the utility can be pushed and experimentally demonstrated.
Decision: Accept (spotlight)