3D Question Answering with Scene Graph Reasoning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:14:26

Abstract: 3DQA has gained considerable attention due to its enhanced spatial understanding capabilities compared to image-based VQA. However, existing 3DQA methods have explicitly focused on integrating text and color-coded point cloud features, thereby overlooking the rich high-level semantic relationships among objects.
In this paper, we propose a novel graph-based 3DQA method termed 3DGraphQA, which leverages scene graph reasoning to enhance the ability to handle complex reasoning tasks in 3DQA and offers stronger interpretability.
Specifically, our method first adaptively constructs dynamic scene graphs for the 3DQA task. Then we inject both the situation and the question inputs into the scene graph, forming the situation-graph and the question-graph, respectively. 
Based on the constructed graphs, we finally perform intra- and inter-graph feature propagation for efficient graph inference: intra-graph feature propagation is performed based on Graph Transformer in each graph to realize single-modal contextual interaction and high-order contextual interaction; inter-graph feature propagation is performed among graphs based on bilinear graph networks to realize the interaction between different contexts of situations and questions.
Drawing on these intra- and inter-graph feature propagation, our approach is poised to better grasp the intricate semantic and spatial relationship issues among objects within the scene and their relations to the questions, thereby facilitating reasoning complex and compositional questions.
We validate the effectiveness of our approach on ScanQA and SQA3D datasets, and expand the SQA3D dataset to SQA3D Pro with multi-view information, making it more suitable for our approach. Experimental results demonstrate that our 3DGraphQA outperforms existing methods.

Main paper: 
 related work: 
 In this section, we give a brief review on recent advances in 3D Question Answering, 3D Captioning and Visual Grounding, as well as Graph-based Visual Question Answering.3D Question Answering (3DQA) refers to answering questions related to 3D scenes, distinguishing it from VQA, which aims to answer questions related to 2D images. A few benchmarks have been devised in the field including SQA3D [20] and ScanQA datasets [3, 35] for the evaluation. Depending on the output answering content, we categorize previous 3DQA work into two types: those that solely provide answers and those that simultaneously provide answers along with object information.For the methods that solely provide answers, Ye et al. [35] proposed 3DQA-TR, a transformer-based 3DQA method, that adopts a 3D-linguistic Bert to first encode the point cloud into tokens, then these tokens, along with the text encoding of the question, are jointly fed into the Bert for training and inference. The output answers are generated by the decoder of the Bert model. Zhou et al. [33] proposed the Chat-3D method, where they first segment each object in the scene. Then, they encode each object individually and fed them into a large language model (GPT) for answer prediction. Leveraging the powerful knowledge priors of GPT, this method can thus obtain open-ended answer results. Ma et al. [20] proposed a zero-shot 3DQA method, called SQA3D, which first inputs the point cloud scene into the pre-trained Scan2Cap model to obtain dense captions of the scene. Subsequently, these dense captions are combined with the context description and the question, to feed into the GPT-3 to obtain the answer.For the methods that simultaneously provide answers along with object information, Azuma et al. [3] proposed ScanQA, which consists of 3D and language encoding layers, three-dimensional and language fusion layers, and object localization and language decoding layers. In the fusion layer, it mainly references the 2D MCAN [37] , using transformer blocks to capture the relationship between objects and questions. This method facilitates answering questions and 3D object localization. Delitzas et al. [10] investigated Multi-CLIP, a pre-training multi-modal CLIP-based architecture that embeds the 3D scene features to their corresponding captions and multi-view images in the CLIP space. This method facilitates the tasks of question answering and referred object localization, and 3D situated question answering.In this paper, to address the interpretability and complex reasoning problems in the field, we propose a novel scene graph-based method for 3D question answering. We also extend the SQA3D dataset to SQA3D Pro dataset with multi-view information, enabling more accurate reasoning.Witnessing the popularity of the vision-language models in image domains, many researchers have exploited the 3D-language models for 3D scene tasks including 3D captioning [36] and 3D visual grounding. Chen et al. [6] proposed ScanRefer, a novel method for 3D object localization using natural language. They also developed ScanRefer dataset, which is the first large-scale scene-linguistic Figure 2 : The overall architecture of our 3DGraphQA framework: our method takes the 3D scene context, the situation description and the question as input, ultimately yielding answers to the questions along with object positions and orientations.The key premise of our approach is that the scene graph and graph reasoning model can effectively facilitate complex reasoning tasks in the vision-language domain. dataset. Subsequently, Achlioptas et al. [1] presented ReferIt3D dataset, which contains fine-grained multi-instance 3D object labels to a scene, aiming to facilitate the task of identifying the instance of the same fine-grained object class in a 3D scene. Based on the two datasets, a large body of work has been dedicated to 3D caption and 3d visual grounding. For example, Chen et al. introduced Scan2Cap [7] for the task of dense scene caption generation. This method utilizes PointNet++ [25] as its foundation to generate textual descriptions for objects in the scene. Following their work, Yuan et al. [38] further proposed X-Tran2Cap, which leverages 2D priors for optimization. Wu et al. [39] proposed EDA, which offers a decoupling approach during text encoding, achieving better alignment between scenes and text. Zhao et al. [39] introduced 3DVG-Transformer, a transformer-based 3D visual grounding method that exploits the proposal relations among objects in 3D scene. Cai et al. [5] and Chen et al. [8] offered unified architectures for joint 3D captioning and visual grounding.This work also falls within the realm of multi-modal 3D-language tasks. In this work, our primary focus is 3D question answering.Visual Question Answering (VQA) is a computer vision task that involves both image comprehension and text recognition. Its goal is to equip machines with the capability to comprehend image content and respond to questions about it using natural language. Tremendous efforts have been devoted to the task, with the central challenge lying in how to capture dense semantic interactions and reasoning within and across modalities of images and text. To alleviate this issue, some researchers have attempted to introduce Graph Neural Networks (GNN) into VQA tasks. These works have demonstrated that by integrating graph structural information, such as concept graphs and scene graphs, significant improvements can be made in the interpretability and effectiveness of VQA tasks, thereby enhancing the answer prediction accuracy. For example, Will et al. [21] utilize Graph Convolutional Networks (GCN) to model the semantic correlations between objects in the image scene and learn graph structure representations relevant to the questions. Zhu et al. [40] propose an object-aware graph learning module guided by questions, suggesting that differences provide more information when modeling semantic relationships. To capture complex semantic and spatial relationships in images, many researchers [18, 28, 31, 32] have exploited to use scene graphs to enhance the multi-model image-text learning, facilitating the complex reasoning regarding to the semantic and spatial relationship among objects in 3D scenes.In this paper, we pioneer the graph-based 3DQA method by exploiting scene graph reasoning. Compared with 2D graph-based VQA, our graph to 3D scene is dynamic, depending on the situation and the question setting. 
method: 
 In this section, we provide a detailed introduction to our 3DGraphQA, which employs graph reasoning to address the task of 3D question answering. As illustrated in the Fig. 2 , 3DGraphVQA takes the point cloud 𝑃 of the 3D scene context, the situation description 𝑠, the question 𝑞, as well as multi-view images 𝐼 , as input, aiming to generate accurate answer from the answer set 𝑎 = {𝑎 1 , • • • , 𝑎 𝑀 }, and corresponding to the bounding box 𝑏𝑏𝑜𝑥 of the question, and predicting This module aims to encode the input language and scene images into features. Specifically, for the input situation description 𝑠 and question 𝑞, we utilize GloVe [23] , followed SQA3D, to encode the situation descriptions and questions. These embeddings are then fed into a biLSTM [15] for sequence modeling. Subsequently, the situation features 𝐹 𝑠 and the question features 𝐹 𝑞 are obtained by projecting the outputs of the LSTM through GELUs activation [16] non-linear layers. As for the scene image input, we employ the CILP model [26] , a multi-modal pretrained model based on contrastive learning of images and text, to achieve image feature denoted as 𝐹 𝐼 .To construct the scene graph, we first need to encode the point cloud of the scene. Given the point cloud𝑃 = {𝑝 1 , • • • , 𝑝 𝑁 } ∈ 𝑅 𝑁 ×3of the scene as input, where 𝑁 denotes the number of points in the scene, our method employs VoteNet [24] , which is a point cloud object detection network based on PointNet++ [25] , to extract various object proposals and obtain their features. We assume the number of extracted object proposals is denoted by 𝐾, and the features of the object proposals are represented by𝐹 𝑃 = {𝐹 1 , • • • , 𝐹 𝐾 },where 𝐹 𝑖 refers to the features of the 𝑖-th object. The object bounding boxes are denoted by 𝐵 = {𝑏 1 , • • • , 𝑏 𝐾 }, where 𝑏 𝑖 denotes the position information of the boundaries, containing the position information of a center and eight corners. In the following, we will use the relative positional relationships between each object in the scene to construct the scene graph. Initial Graph Construction. We use the 𝐾 extracted objects in the scene as vertices to construct the initial fully connected undirected graph. Let the graph be represented as 𝐺 = (𝑉 , 𝐸) with 𝐾 nodes 𝑣 𝑖 ∈ 𝑉 and edge element 𝑒 𝑖,𝑗 ∈ 𝐸 denotes the edge between the vertices 𝑣 𝑖 and 𝑣 𝑗 . We associate the feature set 𝐹 𝑃 to 𝑉 , seen as node features, and the elements in the edge set are initially set 1s, leading to a fully connected graph of 𝐺. Because there are potential relationships between every two objects in the fully connected graph, and no prior knowledge is used during the learning process, the weights can be learned and adjusted during the subsequent graph-based feature fusion module.Pruned Graph with Prior-Knowledge. Based on the fully connected graph with uniform weights, which may not effectively measure the spatial relationships between objects in the scene. To overcome this issue, we utilize prior knowledge to prune the graph. Since there are explicit relationships between objects in the scene that can be utilized, we can prune non-existent edges using some prior-knowledge to transform the fully connected graph into a locally connected graph. Specifically, we observed that the performance improves when objects are not connected to themselves compared to when they are connected, so we assign a weight of 0 to the diagonal positions in the adjacency matrix. Then, when establishing local connectivity, we found that most questions are related to objects surrounding the current target object. Therefore, we set a neighborhood range (e.g. k-nearest-neighbor) as the receptive field of the current target and use the bounding boxes of each object as the receptive field to construct this locally connected graph, which can be defined as follows:EQUATIONwhere ẽ𝑖,𝑗 ∈ Ẽ is the edge element of pruned edge set Ẽ, 𝑘𝑛𝑛(•) represents the k-nearest-neighbor, which is calculated based on the absolute distances between the center of the bound box of an object and the center of the bound box of other surrounding nodes. The remaining positions of the adjacency matrix are set to 0s. Therefore, the connectivity graph can be defined as: G = (𝑉 , Ẽ). Additionally, we dynamically set the weights of edges based on the proximity of bounding box relative positions. To ensure the robustness of the graph in multimodal fusion, we also randomly add some explicit unidirectional edges to the graph.Situation-Graph and Question-Graph To better capture the interaction between the situation and question inputs with the 3D scene objects, we investigate injecting the features of perspective images, situation descriptions, and questions into the constructed scene graph. Specifically, considering that the features of individual objects carry local scene information, we connect the encoded image with the features of graph nodes to enhance the global perception capability of target nodes.Additionally, in 3D scene question answering tasks, there exist different scenes, situation descriptions, and questions, and even the same scene may have different scenarios with varying questions. Therefore, in designing multi-modal feature fusion, it is crucial to address how the adaptive relationship between situation descriptions and questions can be effectively injected into the graph structure and how the weights between multi-modal graph nodes can be learned reasonably. Therefore, we first construct the situation graph 𝐺 𝑠 = (𝑉 𝑠 , Ẽ) with the feature set of 𝑉 𝑠 is updated to [𝐹 𝑃 ∥𝐹 𝐼 ∥𝐹 𝑆 ] from the initial 𝐹 𝑃 , and the question graph is similarly defined as 𝐺 𝑞 = (𝑉 𝑞 , Ẽ) with the feature set of 𝑉 𝑞 is updated to [𝐹 𝑃 ∥𝐹 𝑞 ] from the initial 𝐹 𝑃 , where ∥ denotes the concatenation operation.Within the situation-graph and the question-graph, and between the situation-graph and the question-graph, there exist complex semantic and spatial relationships. To comprehend these relationships, we perform feature fusion in both intra-graph and inter-graph.Intra-graph Fusion. Intra-graph feature fusion aims to capture complex relations among nodes of the same graph via message propagation. In this work, we adopt the Graph Transformer [12] to facilitate the task, which includes the attention mechanism and the positional encoding, benefiting the global perception while avoiding over-smoothing.Figure 3 illustrates the architecture of graph transformer, which consists of two essential modules: a multi-head self-attention module (MHA) and a feed-forward network (FFN), akin to vanilla Transformer [29] . Without loss of generality, given the situation-graph 𝐺 𝑠 = (𝑉 𝑠 , Ẽ), we use 𝑋 ′ = [𝐹 𝑃 ∥𝐹 𝐼 ∥𝐹 𝑆 ] to denotes the node features with 𝑥 ′ 𝑖 ∈ 𝑋 ′ represents the feature of node 𝑣 𝑖 , and ẽ𝑖,𝑗 ∈ Ẽ denotes the edge between 𝑣 𝑖 and 𝑣 𝑗 . The graph transformer method first computer the Laplacian eigenvectors [4] 𝛿 of the graph 𝐺 𝑠 and uses them as node positional information to inject into the node feature: 𝑋 = [𝑋 ′ ∥𝛿], where ∥ is the concatenation operation. The graph transformer takes the 𝑋 and Ẽ as inputs and passes them through blocks of MHA and FFN modules. Let v𝑖 be one neighborhood node of 𝑣 𝑖 , the neighborhood feature set can accordingly be defined as X , the specific one head attention and FFN are defined as follows:EQUATIONEQUATIONEQUATIONEQUATIONwhere 𝑊 ℎ 𝑄 , 𝑊 ℎ 𝐾 , 𝑊 ℎ 𝑉 , 𝑊 ℎ 𝑂 , 𝑊 Ẽ represents the weight matrices of the Query (𝑋𝑊 𝑄 ), Key ( X𝑊 𝐾 ), Value ( X𝑊 𝑉 ), Output (𝑋𝑊 𝑂 ) and the edge set for the ℎ-th head, and 𝐻 is the number of attention head. 𝑊 1 and 𝑊 2 are the learnable parameters that constitute the FFN with the inclusion of residual connections and normalization. The node features from the final layer will be passed to MLP to compute the output features.Given the predefined situation-graph and question-graph, our method passes these graphs through the Graph Transformer for intra-graph feature propagation, and outputs the updated graph with updated node and edge representations.Inter-graph Fusion. For inter-graph feature propagation, our method refers to bilinear graph networks (BAN) [13] , as is illustrated in Figure 4 . The original BAN work conducts message passing between image-graph and question-graph in their work. Here, we extend this work to our scene-graph case by analogously representing the learned situation-graph and question-graph as image-graph. To be specific, our method first concatenate the situation-graph and the question-graph, forming an initial joint embedding of scene objects and situation descriptions, and questions. We note that the joint embedding is aware of the words from the situation and question descriptions with objects, but lacks the interaction between questions and the situation description regarding the given scene. Hence, our method then computes the fused embedding between the vectors of the question words and the joint embedding, where the question word vectors are obtained through the text encoder in section 3.1. Finally, the fused embedding (ℎ 𝑙,𝑖 in Figure 4 ) is further updated through an MLP to explore their complex interactions. Following the BAN [13] , our method also stacked the modules multiple times (see the arrow in Figure 4 ) by reusing the fused feature output (𝑜 𝑙,𝑖 in Figure 4 ) as the input of question vectors recursively.After obtaining the fused features, we concatenate the output features of BAN module and input them into an MLP to predict a candidate answer from the answer set 𝑎. To generate multiple answers, we employ a decoding mechanism similar to ScanQA [3] and calculate the final score using a binary cross-entropy loss function.To obtain the object proposals, we feed the graph node features, originating from the question-graph and are processed by the Graph Transformer, into an MLP through max-pooling for decoding to obtain the target object proposals. During training, our method calculates the cross-entropy loss by comparing the decoded features with the ground truth bounding box features for each target object. To reach the localization and rotation of scene objects, a similar approach is employed where we also feed the graph node features, originating from the question-graph and are processed by the Graph Transformer, into an MLP through max-pooling for decoding. During training, we use the mean squared error (MSE) loss to compare the predicted scene position (𝑠 𝑝𝑜𝑠 ) and orientation (𝑠 𝑟𝑜𝑡 ) obtained from SQA3D with the ground truth preset values.Our loss function is defined similarly to SQA3D, which includes the answering loss, the position loss, the perspective loss, the location loss, and the object detection loss. We set the final loss as a simple linear combination of these losses, computed as:EQUATIONwhere 𝜆 1 , 𝜆 2 , 𝜆 3 and 𝜆 4 are the weighting factors for the loss functions. In our experiments, we set all these hyper-parameters to 1s. 
sqa3d pro dataset.: 
 In this section, we describe our proposed SQA3D Pro dataset, which is an extension of the SQA3D dataset with additional multi-view situation information. We note that the SQA3D dataset is currently a publicly available dataset for 3D scene question answering, where various situations from a first-person perspective are predefined for each scene in ScanNet and provided in the form of textual descriptions. Unfortunately, the overhead views and videos provided in the SQA3D dataset only correspond to the global semantic information of the entire scene and do not perfectly correspond to the first-person perspective situations. Inspired by the ScanQA dataset, which incorporates multi-view image information of the entire scene into the model training process, resulting in improved performance. Similarly, for the SQA3D dataset, integrating multi-view information from each first-person perspective into the training process is expected to enhance the effectiveness of the method. Therefore, we also adopt the strategy of incorporating multi-view images to complement the image information of the first-person perspective situations in the SQA3D dataset, Table 1 : The question answering accuracy on the SQA3D dataset. In the method column: "w/o s" denotes the method without situation descriptions, "+ pos" indicates the provision of object position information in addition to textual answers. resulting in the creation of the SQA3D Pro dataset, as illustrated in the Figure 5 .When supplementing the multi-view images, we select four viewpoints: the front view, the back view, the left view, and the right view. Particularly, if any of these views have missing information or are facing a wall, we exclude them from the dataset. Since the SQA3D dataset comprises 650 indoor scenes from the ScanNet dataset, with 6.8k unique situations and 20.4k situation descriptions, our dataset provides approximately 23k additional multi-view images for the 6.8k unique situations in the SQA3D dataset. 
experiments: 
 In this section, we first describe the experimental setup of our approach, including the datasets, implementation details, and evaluation metrics. Subsequently, we discuss the experimental results of our approach. Finally, we conduct ablation experiments to validate the performance of the main modules in our model.Datasets. In this work, we validate our method on the ScanQA dataset [3] and SQA3D dataset [20] . The ScanQA dataset [3] contains 800 3D scenes, 41,363 questions, and 58,191 answers, which is built upon ScanRefer [6] and Scan2Cap [7] . We follow the training, validation, and testing set configurations as described in ScanQA [3] . The SQA3D dataset [20] is designed for embodied scene understanding by integrating situation understanding and situated reasoning. It was constructed using 650 scenes from ScanNet [9] , comprising 6.8k unique situations, 20.4k descriptions, and 33.4k diverse reasoning questions related to these situations.Implementation. We implement our model using the PyTorch library, and all experiments are performed on a single Nvidia RTX 4090 GPU. We utilize the ADAM optimizer to train our model, with an initial learning rate of 0.0001, and a batch size of 16. Our model is trained for 80 epochs, after 40 epochs, the learning rate is reduced to 0.00002 for convergence. Additionally, when the number of edges in the graph structure is high, the batch size will also be decreased to 14.Evaluation Metrics. On the ScanQA dataset [3] , we employ the same evaluation metrics as the work [3] , which include EM@1 and EM@10, where EM denotes the exact match and EM@K represents the percentage of predictions that exactly match any ground truth The qualitative comparison results between our method and SQA3D. From the figure, our approach achieves higher accuracy in answering "what"-related questions, and can provide accurate bounding boxes for relevant objects. In contrast, SQA3D's answers are often inaccurate and fall short in providing bounding boxes which serve as auxiliary aids for understanding.answer among the top 𝐾 predicted answers. Meanwhile, we utilize BLEU-4, ROUGE, METEOR, and CIDEr as the sentence-level evaluation metrics. On the SQA3D dataset [20] , accuracy is determined by the average accuracy of answers obtained for six predefined questions. Similarly, our SQA3D Pro dataset also employs EM@1 and EM@10 as the evaluation metrics, consistent with SQA3D.In this section, we compare the performance of 3DGraphVQA on the SQA3D dataset [20] and the ScanQA dataset [3] with various SOTA methods, including SQA3D [20] , ScanQA [3] , CLIP-PVQA [22] , Multi-CLIP [10] and 3D-VisTA [41] . The results are presented in Table 1 and Table 2 .We begin by conducting experiments on the SQA3D dataset. Our method uses the complete reasoning architecture described in Section 3. We set up both question and scene graphs, and both undergo graph reasoning using Graph Transformer. Additionally, in the fusion process, we use a bilinear graph network [13] for intergraph fusion. As shown in Table 1 , SQA3D performs better on Is and Can questions, but gets poorer results on What questions. This is because SQA3D has a simpler reasoning process, making it easier to answer binary questions with fewer answer options. However, due to the comprehensive graph reasoning of our approach, we achieve the best results on What and Which questions. Particularly, since we also utilize the SQA3D Pro dataset as supplemental image modality for the current scenario, our method also achieved improved performance on the SQA3D dataset. In Figure 6 we also illustrate the qualitative comparison results between our method and SQA3D.We then conduct experiments on the ScanQA dataset. ScanQA is a question-answering dataset containing only scenes and specific questions, and no prior work has constructed dynamic scene graphs on ScanQA. To better facilitate graph reasoning, we set up only one question graph for graph reasoning. Since we only have one graph, there is no need for inter-graph fusion learning. However, we still aim to achieve appropriate multimodal fusion of text and graph during the multimodal fusion process. To achieve this, we refer to BUTD [2] , which is a method that combines bottom-up and top-down attention mechanisms. The bottom-up attention focuses on the visual features themselves using a feed-forward attention mechanism, while the top-down attention relies on textual features to predict attention distribution in visual features. By inputting the node features after graph reasoning and the encoded question description, we enable the question features to receive more attention from matching graph nodes. As shown in Table 2 , we have achieved the SOTA performance, especially in EM@10, where our method exceeds other SOTA methods by a large margin.In this section, we conduct various ablation experiments to validate the effectiveness of different modules in our pipeline including scene graph construction, graph reasoning, as well as bilinear graph networks.Scene Graph Construction. Given that 3DGraphVQA is an endto-end network architecture where target features and bounding boxes are dynamically acquired, we need to dynamically construct graph nodes and edges. Specifically, for the construction of graph nodes, our method directly takes the objects as the graph nodes and incorporates the features of each object as node features in the graph.For the construction of graph edges, our method initially experiments with unconnected graphs or fully connected graphs. In the unconnected graph setup, we only inject the node features into the graph reasoning process. Conversely, in the fully connected graph setup, all edge weights of the bounding nodes were set to 1s and injected into the graph reasoning process along with the node features. In addition, considering the strong correlation between the problem design of the SQA3D dataset and the surrounding objects of the current perspective, we set edge nodes based on the prior information of each target bounding box. Here, we rank the objects closest to the current graph node's bounding box, and the top-ranked objects are selected within the neighborhood range, meaning the edge weight between this target and the current graph node is set to 1; otherwise, it is set to 0. We set the neighborhood range to 2, 4, 6, 8, and 10. As shown in Figure 7 , the best performance is achieved when the neighborhood range is set to 8 targets. On the contrary, when using unconnected or fully connected graphs, the effect of both is relatively inferior.Moreover, as we employed a dual-graph design for inter-graph processing, consisting of situation-graph and question-graph, we also compare the results with those obtained from a single-graph design. In this setup, we concatenate the situation description feature and question encoding feature into the node features as a single fusion graph. As shown in the table 3, we demonstrated that simultaneous inter-graph processing with situation-graph and question-graph yields superior results.Table 3 : The ablation study of our 3DGraphQA. We show the EM@1 and EM@10 results on SQA3D dataset. Graph Reasoning Methods. There are various methods for reasoning over graph structures in the graph reasoning field. In this work, during the graph reasoning process, our method adopt the former constructed graphs as inputs to the Graph Transformer. Additionally, we conduct ablation experiments with different numbers of layers in the Graph Transformer, i.e. selecting 5 and 10 layers for comparison. As shown in the table, employing Graph Transformer with 10 layers outperformed the other setup.Bilinear Graph Networks. Finally, we compared the results with and without the module of bilinear graph networks. When discarding bilinear graph networks, we concatenated the node features of the scene graph and question graph together and directly decoded the features. The results demonstrated that without the presence of bilinear graph networks, the accuracy of the answers decreased, which validates the effectiveness of the bilinear graph networks. 
conclusion: 
 In this paper, we present 3DGraphQA, a novel graph-based 3D question answering method. The key premise of our method is that graph reasoning can facilitate the complex relation reasoning between 3D scene objects. To accomplish this goal, our method first constructs the scene graph based on the scene and injects the situation description and question description separately to form the situation-graph and the question-graph. Subsequently, for the intra-graph and inter-graph feature fusion of the situationgraph and the question-graph, we propose intra-graph message passing based on Graph Transformer and inter-graph message passing based on the Bilinear Attention Network (BAN). Experimental results demonstrate the effectiveness of our proposed approach. 
: 
 3DQA serves as an extension of the VQA task, since both tasks require the analysis and understanding of multimodal visual and textual content, followed by inference to obtain answers. However, compared to 2D VQA tasks, reasoning within authentic 3D environments allows for the avoidance of spatial ambiguities inherent in 2D data, thereby requiring the acquisition of accurate geometric information and object relationships. Moreover, 3D scenes typically encompass a greater number of objects and entail more intricate interrelationships among them.Many efforts have been made to address the challenges of 3DQA. For example, Ye et al. [35] proposed 3DQA-TR, a transformer-based 3DQA model, that leverages two separate encoders to encode the appearance and geometry information and then adopts a 3D-linguistic Bert [11] to fuse multi-modal information including appearance, geometry and linguistic question for 3DQA. Azuma et al. [3] developed a baseline model based on their constructed ScanQA dataset, which primarily comprises 3D and language encoders, 3D and language fusion, and 3D object localization and 3D QA modules. It can be observed that these methods predominantly focus on aligning and fusing text and point cloud features. Though significant progress has been made, these approaches still exhibit limitations in interpretability and handling complex semantic and spatial relationship reasoning. For example, consider the question posed in Figure 1 . " What is behind me that you can sleep on or sit on?" or "How many backpacks are on the ottoman behind me to the left side?" not only requires detecting objects in geometric scenes but also understanding the complex semantic and spatial relationships between scene objects, questions and situations. This paper is inspired by graph-based 2D VQA methods [18, 28, 31, 32] , which have demonstrated the effectiveness of enhancing the interpretability and reasoning capabilities for complex questions in VQA. However, directly transferring the graph structure from VQA to 3DQA is not feasible, because 2D scene graphs only carry local information, whereas our 3D scene graphs can encapsulate both global scene information and support the generation of end-toend dynamic graphs. Depending on the situation and the question setting, different scene graphs can be constructed for the same scene. For example, the situation: I play the piano, and the situation: I am sitting on the armchair facing the desk. require different scene graphs of the scene for the question: What is behind me that you can sleep on or sit on?.In this paper, we propose a novel graph-based 3DQA method, termed 3DGraphQA, exploring using the graph reasoning to enhance the performance of 3DQA task. Specifically, in the graph construction stage, our method utilizes VoteNet [24] to detect objects in the scene based on the specified situational setting. The detected objects and their features act as nodes in the graph and are optimized using prior knowledge, thereby forming the initial scene graph. Note that as the situation setting changes, the final scene graph also changes accordingly, making the scene graph dynamic. Our method injects the input situation description and question description separately into the scene graph to obtain the situationgraph and the question-graph; In the graph feature fusion stage, to facilitate the high-order contextual interaction within the graph and the cross-modal contextual interaction between the situation-graph and the question-graph, our method investigates graph transformer to realize intra-graph contextual interaction, and adopts bilinear graph networks to realize inter-graph contextual interaction.Our method is experimentally validated on the ScanQA and SQA3D datasets, both of which are built upon the ScanNet dataset. Moreover, to further explore the potential of our method, we also develop the SQA3D Pro dataset, which is an extension of the SQA3D dataset with additional multi-view situation information, drawing inspiration from ScanQA dataset. We show that the SQA3D Pro dataset provides rich information for the scenes and facilitates a better understanding of the current scene for scene graph reasoning, enabling performance improvement. The experimental results have shown that our proposed method achieves SOTA performance on these datasets.In summary, our contributions are listed as follows:• We propose a novel graph-based 3DQA method, which exploits dynamic scene graphs to facilitate the 3DQA tasks. • We introduce a Graph Transformer-based model for intragraph feature fusion, enabling contextual interactions between the scene objects and the question, and between the scene objects and the situation description.  
 
 Appendix:  


Decision: All reviewers are positive about this submission and most of the concerns have been addressed in the rebuttal. But there are still some concerns remain. For example, in-depth analysis and reasonable ablation experiments are still required. The authors are encouraged to include the rebuttal in the final version.

Reviews: review: paper_summary: paper_summary: The paper presents 3DGraphQA, a graph-based method developed to enhance the capabilities of 3D Question Answering (3DQA) , which traditionally rely heavily on text and basic point cloud features. By introducing dynamic scene graphs that utilize both situational and question-specific graphs, 3DGraphQA advances the processing of spatial relationships and complex reasoning tasks. The system employs Graph Transformers for intra-graph feature propagation and bilinear graph networks for inter-graph interactions, which enable richer semantic understanding. This paper conducetd experiments on two 3D QA datasets SQA3D and ScanQA, and improvements were made by integrating multi-view information in the SQA3D Pro dataset.
strength_weakness: strength_weakness: - 3DGraphQA enhances the ability to interpret complex questions by constructing dynamic scene graphs that integrate both situational and question-specific graphs, employing intra-graph and inter-graph feature propagation techniques to deepen semantic relationship understanding.
- The proposed SQA3D Pro dataset provides additional multi-view images for the SQA3D dataset.
- This paper is well-written and easy to follow.
- Experimental results demonstrate that 3DGraphQA significantly outperforms existing methods.
limitations: limitations: - In the experiment on SQA3D, 3DGraphQA used an additional dataset SQA3D to achieve the new sota, where there may be unfair comparisons.
- In Equation 6, the final loss is a combination of several losses. Line 616. "We set all these hyperpara-meters to 1", why is the value is 1s?
- In Table 3, which method does GAT refer to? I didn't find a definition. Why do the first two lines compare the performance of Graph Tansformer single Graph and Graph Tansformer without BAN, what is the performance of the baseline method?
paper_summary: paper_summary: This paper focuses on the problem of the rich high-level semantic relationships among objects. To this end, This study presents a novel graph-based method, 3DGraphQA, which utilizes scene graph reasoning to improve the interpretability and performance of 3DQA tasks. Specifically, 3DGraphQA constructs dynamic scene graphs that adapt to specific situational settings, using these graphs to model and propagate features within and across contexts of situations and questions. It enables the model to grasp intricate semantic and spatial relationships among objects and their relation to the questions.
strength_weakness: strength_weakness: 1. Interpretability: By leveraging scene graph reasoning, the proposed method enhances the interpretability of the model’s decisions, making it easier to understand how conclusions are reached. (But need more visualization to reach this point)

2. Comprehensive evaluation: The method is validated on multiple datasets, including the newly developed SQA3D Pro, demonstrating its robustness and effectiveness across different contexts and question types.
limitations: limitations: 1. The results are not strong enough. In Tab.2, the 3DGraphQA with multi-view input is not shown to be superior compared with 3D-Vista.

2. The model is trivial. The bilinear graph transformer-based model for intra-graph features and inter-graph features is common in VQA tasks such as [1].

3. Lack of detailed error analysis: The paper does not provide an in-depth error analysis or discuss potential failure modes of the proposed method. Understanding where and why the model fails is crucial for further improvement and practical deployment.

4. Data collection constraints: The requirement for multi-view images in the SQA3D Pro dataset might not be feasible in all real-world applications due to constraints in data collection and processing. This could limit the method's applicability in scenarios where such data is unavailable.

5. Complexity: The training process involves multiple components (VoteNet, Graph Transformers, bilinear graph networks), which might complicate the model's implementation and tuning. This complexity could pose challenges for researchers and practitioners attempting to replicate or extend the work.

[1] Li, Hao, et al. "Joint learning of object graph and relation graph for visual question answering." 2022 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2022.
paper_summary: paper_summary: This paper proposes a graph-based 3DQA method called 3DGraphQA, which enhances the handling of complex reasoning tasks and interpretability in 3DQA by leveraging scene graph reasoning. The method constructs dynamic scene graphs and utilizes intra- and inter-graph feature propagation to capture semantic and spatial relationships among objects and their connections to the questions.
strength_weakness: strength_weakness: - The use of scene graph reasoning allows the proposed method to handle complex reasoning tasks more effectively.
- By performing feature propagation within and between graphs, the proposed method effciently captures both single-modal and cross modal contextual interactions, improving understanding of semantic and spatial relationships.
- An extension dataset SQA3D Pro was built upon the existing SQA3D dataset, while providing additional multi-view situation information.
- Experimental results demonstrate effectiveness of the proposed methods on several benchmarks.
limitations: limitations: - The construction and reasoning over dynamic scene graphs, along with the use of graph transformers and bilinear graph networks, likely result in high computational complexity, which should be discussed.
- The method relies on the initial accuracy of object detection using votenet, errors in this initial stage might propagate through the pipeline.
- While the dynamic construction of scene graphs allows for adaptability of different questions and situations, it also introduces challenges in maintaining consistency and stability in graph structure across different scenes.
paper_summary: paper_summary: In this paper, the author attempts to introduce multi-view situation information to 3D Question Answering. To this end, the author proposes a dual-branch graph-based framework to enhance interpretability and handle complex semantic and spatial relationship reasoning. Experimental results show that the proposed method set a new record of 3DQA.
strength_weakness: strength_weakness: 1. Introducing a graph-based approach to address complex relations in 3DAQ is reasonable in this topic.
2. Different components of this framework are well organized.
limitations: limitations: 1. Table 1, lacks an analysis of the question why the proposed method has poor performance on “Is“ and “Can“ questions since it can reason complex questions like “What“ or “Which“. The results in Table 2 have the same problem.
2. Although the proposed method achieved promising results, it is better to provide sufficient analysis or comparison in the Experimental Results section.
3. In the ablation study subsection, the influence of each component is not well explained. Whether such a module works well with the proposed framework?