3D Question Answering for City Scene Understanding
Authors: name: Penglei Sun; name: Yaoxian Song; name: Xiang Liu; name: Xiaofei Yang; name: Qiang Wang; name: Tiefeng Li; name: Yang Yang; name: Xiaowen Chu
Published: 2024-07-24 16:22:27

Abstract: 3D multimodal question answering (MQA) plays a crucial role in scene
understanding by enabling intelligent agents to comprehend their surroundings
in 3D environments. While existing research has primarily focused on indoor
household tasks and outdoor roadside autonomous driving tasks, there has been
limited exploration of city-level scene understanding tasks. Furthermore,
existing research faces challenges in understanding city scenes, due to the
absence of spatial semantic information and human-environment interaction
information at the city level.To address these challenges, we investigate 3D
MQA from both dataset and method perspectives. From the dataset perspective, we
introduce a novel 3D MQA dataset named City-3DQA for city-level scene
understanding, which is the first dataset to incorporate scene semantic and
human-environment interactive tasks within the city. From the method
perspective, we propose a Scene graph enhanced City-level Understanding method
(Sg-CityU), which utilizes the scene graph to introduce the spatial semantic. A
new benchmark is reported and our proposed Sg-CityU achieves accuracy of 63.94
% and 63.76 % in different settings of City-3DQA. Compared to indoor 3D MQA
methods and zero-shot using advanced large language models (LLMs), Sg-CityU
demonstrates state-of-the-art (SOTA) performance in robustness and
generalization.