SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tracking After frame-level panoptic segmentation, we link each frame via using UniTrack [69] for tracking to obtain the final tracked video cubes for each clip for either modality input. Specifically, instead of incorporating an additional appearance model for tracking embedding extraction, we directly utilize the instance kernels  $\{k_i\}_{i=1}^N$  from the segmentation step of DKNet, or object query features  $\{q_i\}_{i=1}^N$  from Mask2Former as the tracking embeddings for the association. We find that the instance kernels exhibit sufficient distinctiveness for tracking purposes, even when dealing with different objects belonging to the same semantic class. This is primarily because each instance kernel is designed to maximize the response for a specific instance while suppressing the responses of all other instances, including those with the same semantic class. For a video sequence with the length  $T$ , the obtained 4D feature tubes are noted as  $Q_i = \{q_i^t\}_{t=1}^T$ .
5.2 Relation Modeling: 4D Scene Graph Generation
The object query tubes  $Q_{i}$  and mask tubes  $\mathbf{m}_i$  form a bridge between the first and second stages. These feature tubes first pass through a spatial-temporal transformer encoder, which augments them with both global context information from the overall image and global temporal space.
Spatial-Temporal Transformer Encoder To infuse the feature tubes with additional temporal dimension information and characteristics from other objects in the scene, we draw inspiration from the Spatial-Temporal Transformer [70]. A spatial encoder is initially employed. For all objects co-occurring at the same time  $t$ , a two-layer transformer encoder is applied to the input, comprising all object features specific to time frame  $t$ . The spatially-encoded feature tube updates the object feature tube into  $\{\tilde{q}_i^t\}_{i=1}^N$ . Subsequently, a temporal transformer encoder updates each object feature tube along the temporal dimension  $T$ . By leveraging both the spatial and temporal encoders, we obtain the final feature tube  $\{\hat{q}_i^t\}_{i=1}^N$ , ready for relation training.
Relation Classification Training To train the relation model based on the updated query tube, a training set for relation training must be constructed. It is worth noting that the relation annotation in the training set is in the form of "object-1 relation object-2", with the mask tube of both objects annotated. To start, we associate the updated query tube with ground truth objects. For each ground truth tube, we find the most suitable updated query tube by calculating the video Intersection over Union (vIOU) between ground truth mask tubes, and assign the query feature tube to the respective objects. A frame-level predicate classification is conducted with the assistance of a lightweight fully-connected layer. The inference of the relation classification component simply computes the relation probability between pairs of  $\hat{q}_i^t$  and  $\hat{q}_j^t$ .
6 Experiments
Table 2 presents the results of experiments conducted on the PSG-4D dataset. For RGB-D sequences, an ImageNet-pretrained ResNet-101 serves as both the RGB and depth encoder. We set the training duration to 12 epochs. The DKNet, trained from scratch, requires a longer training period of 200 epochs. In the second stage, both spatial and temporal transformer encoders span two layers, and training continues for an additional 100 epochs. Besides the standard PSG4DFormer, we also examine variants with the temporal encoder removed (denoted as “/t”) and the depth branch removed (denoted as “/d”). As a baseline, we use the 3DSGG model [18], which employs a GNN model to encode frame-level object and relation information, without considering temporal data.
RGB-D vs. Point Cloud Input Table 2 is divided into two sections. The upper part (#1-#3) reports results from point cloud input, while the latter part (#4-#7) details results from the RGB-D sequence. It appears that the RGB-D sequence generally yields better results than the point cloud sequence, particularly for the PSG4D-GTA dataset. This could potentially be attributed to the ResNet-101 backbone used for the RGB-D data, which being pretrained on ImageNet, exhibits robust performance on complex datasets like PSG4D-GTA. Meanwhile, the PSG4D-HOI dataset seems to offer a more consistent scenario with abundant training data, thus narrowing the performance gap between the point cloud and RGB-D methods.
Significance of Depth The results in Table 2 also allow us to evaluate the importance of depth in the RGB-D method. Specifically, we designed a variant of PSG4DFormer (marked as “/d”) that
Table 2: Main Results on PSG4D. Experimental results are reported on both the PSG4D-GTA and PSG4D-HOI datasets. In addition to comparing with traditional 3DSGG methods, we conduct experiments to compare the PSG4DFormer and its variants. This includes a version with the temporal encoder removed (denoted as “/t”) and one with the depth branch removed (denoted as “/d”).
Input Type	Method	PSG4D-GTA	PSG4D-HOI
R/mR@20	R/mR@50	R/mR@100	R/mR@20	R/mR@50	R/mR@100
Point Cloud Sequence	#1 3DSGG [18]	1.48 / 0.73	2.16 / 0.79	2.92 / 0.85	3.46 / 2.19	3.15 / 2.47	4.96 / 2.84
#2 PSG4DFounder/t	2.25 / 1.03	2.67 / 1.72	3.14 / 2.05	3.26 / 2.04	3.16 / 2.35	4.18 / 2.64
#3 PSG4DFounder	4.33 / 2.10	4.83 / 2.93	5.22 / 3.13	5.36 / 3.10	5.61 / 3.95	6.76 / 4.17
RGB-D Sequence	#4 3DSGG [18]	2.29 / 0.92	2.46 / 1.01	3.81 / 1.45	4.23 / 2.19	4.47 / 2.31	4.86 / 2.41
#5 PSG4DFounder/t	4.43 / 1.34	4.89 / 2.42	5.26 / 2.83	4.44 / 2.37	4.83 / 2.43	5.21 / 2.84
#6 PSG4DFounder/d	4.40 / 1.42	4.91 / 1.93	5.49 / 2.27	5.49 / 3.42	5.97 / 3.92	6.43 / 4.21
#7 PSG4DFounder	6.68 / 3.31	7.17 / 3.85	7.22 / 4.02	5.62 / 3.65	6.16 / 4.16	6.28 / 4.97
doesn't utilize the depth branch. In other words, both the RGB encoder and the SA-Gate are removed, turning the pipeline into a video scene graph generation pipeline. The performance of this variant is inferior compared to the original, which highlights the significance of depth information in the scene graph generation task.
Necessity of Temporal Attention Table 2 includes two methods that do not utilize temporal attention. Specifically, the 3DSGG baseline learns interactions between static object features using a graph convolutional network, while PSG4DFormer $^{t}$  removes the temporal transformer encoders. The results demonstrate that ignoring the temporal component could lead to sub-optimal outcomes, emphasizing the importance of temporal attention in 4D scene graph generation.
7 Real-World Application
This section illustrates the deployment of the PSG-4D model in a real-world application, specifically within a service robot. It extends beyond theoretical concepts and computational models, delving into the practical integration and execution of this cutting-edge technology. As shown in Figure 4, the focus here is to demonstrate how the robot leverages the PSG-4D model (pretrained from PSG4D-HOI, RGB-D input) to interpret and respond to its surroundings effectively.
Interaction with Large Language Models The recent advancements in large language models (LLMs) have displayed their exceptional capabilities in reasoning and planning [71]. LLMs have been utilized as planners in numerous recent studies to bridge different modalities, paving the way for more intuitive and efficient human-machine interaction [72]. In this work, we employ GPT-4 [71], as the primary planner. Designed to align with human instruction, GPT-4 communicates with the robot by translating the raw scene graph representations into comprehensible human language. Therefore, the interaction begins with the prompt, "I am a service robot. For every 30 seconds, I will give you what I have seen in the last 30 seconds. Please suggest me what I could serve." Subsequently, every 30 seconds, the robot engages with GPT-4, providing an update: "In the past 30s, what I captured is: <from start_time to end_time, object-1 relation object-2>, <...>, <...>." This enables GPT-4 to analyze the situation and provide appropriate feedback.
Post-Processing for Execution The effective deployment of the PSG-4D model necessitates a robust set of predefined actions that the robot can execute. Currently, the action list includes tasks such as picking up litter and engaging in conversation with individuals. After GPT-4 provides its suggestions, it is further prompted to select a suitable action from this predefined list for the robot to execute. However, the flexibility of this system allows for the expansion of this action list, paving the way for more complex and varied tasks in the future. To encourage community involvement and the development of fascinating applications, we also release the robot deployment module alongside the PSG4D codebase. The demo robot is priced at approximately $1.2K and comes equipped with an RGB-D sensor, microphone, speakers, and a robotic arm.
(a) The RGB-D sequence that is captured by the robot.
(b) PSG-4D Parsing
Figure 4: Demonstration of a Robot Deployed with the PSG-4D Model. The service robot interprets the RGB-D sequence shown in (a), where a man is seen drinking coffee and subsequently dropping the empty bottle on the ground. The robot processes this sequence, translating it into a 4D scene graph depicted in (b). This graph comprises a set of temporally stamped triplets, with each object associated with a panoptic mask, accurately grounding it in 3D space. The robot regularly updates its PSG4D to GPT-4, awaiting feedback and instructions. In this scenario, GPT-4 advises the robot to clean up the discarded bottle and remind the man about his action. This directive is translated into robot action, as visualized in (d).
(d) Robot Reaction
8 Conclusion, Challenges, and Outlook
This paper presents a novel and demanding extension to the traditional scene graph generation, the 4D Panoptic Scene Graph Generation, which incorporates the spatio-temporal domain into the framework. We introduce a comprehensive framework, the PSG4DFormer, capable of processing both RGB-D and point cloud sequences. The successful deployment of this pipeline in a practical service robot scenario underscores its potential in real-world applications. However, these achievements also highlight the nascent state of this field, emphasizing the necessity for continued advancements to fully exploit the potential of 4D Panoptic Scene Graph Generation.
Challenges Despite encouraging results, we have also revealed several persistent challenges in the realm of 4D Panoptic Scene Graph Generation. Through our demonstration, we found that current models, whether derived from PSG4D-GTA or PSG4D-HOI, can handle only simple scenes and falter when faced with more complex real-world environments. Notably, there exist robust models trained in the 2D world. Finding effective and efficient strategies to adapt these models to the 4D domain presents a compelling direction for future exploration.
Outlook Future work in this field presents several intriguing trajectories. There is a pressing need for more efficient algorithms for 4D Panoptic Scene Graph Generation, which can handle larger and more diverse environments. Equally important is the creation of comprehensive and diverse datasets that would allow more rigorous evaluation and foster advancements in model development. Particularly noteworthy is a recent Digital Twin dataset [73], which promises a high level of accuracy and photorealism, aligning seamlessly with the objectives of PSG4D. This dataset will be incorporated as the third subset of the PSG4D dataset, readily accessible from our codebase. In addition to robotics, as demonstrated by the practical application of PSG4DFormer, we are also exploring its potential as an autonomous player in the GTA game. Actually, our recent endeavor Octopus [58] strives to complete GTA missions by employing a visual-language programmer to generate executable action code. In contrast to the previously passive task completion, the application in this paper actively perceives and understands the environment, showcasing a shift towards autonomy in robotics. Furthermore, Octopus [58] utilizes a 4D scene graph structure to capture environmental information during the visual-language programmer training, exemplifying a practical application of the PSG4D modality.
We eagerly anticipate the future progress in the field of 4D Panoptic Scene Graph Generation and its potential to revolutionize our understanding of real-world dynamics.
Potential Negative Societal Impacts This work releases a dataset containing human behaviors, posing possible gender and social biases inherently from data. Potential users are encouraged to consider the risks of overseeing ethical issues in imbalanced data, especially in underrepresented minority classes. Nevertheless, all the NSFW content is removed from the dataset.
This study is supported by the Ministry of Education, Singapore, under its MOE AcRF Tier 2 (MOE-T2EP20221-0012), NTU NAP, and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, the National Key R&D Program of China under grant number 2022ZD0161501, as well as cash and in-kind contribution from the industry partner(s).
[1] Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes. arXiv preprint arXiv:2210.07474, 2022. 2
[2] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023. 2
[3] Sonia Raychaudhuri, Tommaso Campari, Unnat Jain, Manolis Savva, and Angel X Chang. Reduce, reuse, recycle: Modular multi-object navigation. arXiv preprint arXiv:2304.03696, 2023. 2
[4] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022. 2, 6
[5] Xiangtai Li, Haobo Yuan, Wenwei Zhang, Guangliang Cheng, Jiangmiao Pang, and Chen Change Loy. Tube-link: A flexible cross tube baseline for universal video segmentation. In ICCV, 2023. 2
[6] Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Guangliang Cheng, Pang Jiangmiao, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-based visual segmentation: A survey. arXiv pre-print, 2023. 2
[7] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, 2017. 2, 3
[8] Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR, 2019. 2, 3
[9] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, 2018. 2, 3
[10] Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal. Energy-based learning for scene graph generation. In CVPR, 2021. 2, 3
[11] Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, and Yin Li. Learning to generate scene graph from natural language supervision. In ICCV, 2021. 2, 3
[12] Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, and Ziwei Liu. Panoptic scene graph generation. In European Conference on Computer Vision, pages 178-196. Springer, 2022. 2, 3
[13] Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, and Ziwei Liu. Panoptic video scene graph generation. In CVPR, 2023. 2, 3
[14] Xindi Shang, Tongwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detection. In ACM MM, 2017. 2, 3
[15] Xindi Shang, Donglin Di, Junbin Xiao, Yu Cao, Xun Yang, and Tat-Seng Chua. Annotating objects and relations in user-generated videos. In ICMR, 2019. 2, 3
[16] Matthew Fisher, Manolis Savva, and Pat Hanrahan. Characterizing structural relationships in scenes using graph kernels. In ACM SIGGRAPH 2011 papers, pages 1-12. 2011. 2, 3
[72] Wenlong Huang, P. Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ArXiv, abs/2201.07207, 2022. 8
[73] Xiaqing Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20133-20143, 2023. 9