3D-Aware Visual Question Answering about Parts, Poses and Occlusions
Authors: Xingrui Wang
Wufei Ma
Zhuowan Li
Adam Kortylewski
Alan Yuille
Venue: NeurIPS 2023 poster (2023)
Keywords: VQA
reasoning
3D scene understanding
analysis-by-synthesis
neural modular network
neuro-symbolic reasoning
TL;DR: We introduce the task of 3D-VQA where questions query the object parts, 3D poses and occlusions, and create a dataset Super-CLEVR-3D. A modular model with 3D generative scene parsing is proposed to solve the task.

Abstract: Despite rapid progress in Visual question answering (\textit{VQA}), existing datasets and models mainly focus on testing reasoning in 2D.  However, it is important that VQA models also understand the 3D structure of visual scenes, for example to support tasks like navigation or manipulation.  This includes an understanding of the 3D object pose, their parts and occlusions.   In this work, we introduce the task of 3D-aware VQA, which focuses on challenging questions that require a compositional reasoning over the 3D structure of visual scenes.   We address 3D-aware VQA from both the dataset and the model perspective.   First, we introduce Super-CLEVR-3D, a compositional reasoning dataset that contains questions about object parts, their 3D poses, and occlusions.   Second, we propose PO3D-VQA, a 3D-aware VQA model that marries two powerful ideas: probabilistic neural symbolic program execution for reasoning and deep neural networks with 3D generative representations of objects for robust visual recognition.  Our experimental results show our model PO3D-VQA outperforms existing methods significantly, but we still observe a significant performance gap compared to 2D VQA benchmarks, indicating that 3D-aware VQA remains an important open research area.

3D-Aware Visual Question Answering about Parts, Poses and Occlusions
Xingrui Wang' Wufei Ma Zhuowan Li Adam Kortylewsk Alan Yuille!
Johns Hopkins University 2 Max Planck Institute for Informatics University of Freiburg (xwang378, wma27, zli110, ayuillelJ0jhu.ee akortyle@mpi- inf .mp.dee
Abstract
Despite rapid progress in Visual question answering (VQA), existing datasets and models mainly focus on testing reasoning in 2D. However, it is important that VQA models also understand the 3D structure of visual scenes, for example to support tasks like navigation or manipulation. This includes an understanding of the 3D object pose, their parts and occlusions. In this work, we introduce the task of 3D- aware VQA, which focuses on challenging questions that require a compositional reasoning over the 3D structure of visual scenes. We address 3D-aware VQA from both the dataset and the model perspective. First, we introduce Super-CLa CLEVR-3D, a compositional reasoning dataset that contains questions about object parts, their 3D poses, and occlusions. Second, we propose PO3D-VQA a 3D-aware VQA model that marries powerful ideas: probabilistic neural symbolic program execution for reasoning and deep neural networks with 3D generative representations objects for robust visual recognition. Our experimental results show model PO3D-VQA outperforms existing methods significantly, but we still observe a significant performance gap compared to 2D VQA benchmarks, indicating that 3D-aware VQA remains an important open research area. The code available at https://github. http://gtttuo..cooo//gguuuuuaeae www//ingruwaang////00000 Aware-
Introduction
Visual question answering (VQA) is challenging task that requires an in-depth understanding of vision and language, as well as multi-modal reasoning. Various benchmarks and models have been proposed to tackle this challenging task, but they mainly focus on 2D questions about objects, attributes, or 2D spatial relationships. However, is important that VQA models understand the 3D structure of scenes, in order to support tasks like autonomous navigation and manipulation.
An inherent property of human vision is that comprehensive understanding of the 3D structure in images. For example, humans can answer the questions shown in Fig. [ which ask about the object parts, their 3D poses, and occlusions. However, current VQA models, which often rely on 2D bounding boxes to encode a visual scene 52 25 struggle to answer such questions reliably (as can seen from our experiments). We hypothesize this is caused by the lack of understanding of the 3D structure images.
naturally answer questions that require
In this work, introduce the task of 3D-aware VQA, where answering the questions requires compositional reasoning over the 3D structure of the visual scenes. More specifically, we focus on challenging questions that require multi-step reasoning about the object-part hierarchy, the 3D poses of the objects, and the occlusion relationships between objects or parts.
Wufee contributed to develop the 3D algorithm for multi-objects pose estimation. 1 ZZuowaan contributed to dataset construction, manuscript writing and conceptualizing the framework.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Q: What is the name of brown part the large rubber thing? Wheel
Part
Q: What is the material of the trunk that belongs to the same object as
the purple part? Metallic
Q: Which direction the double bus is facing? Left
3D Pose
Q: What is the color of the small object which faces to the right? Truck
Occlusion is the bumper of purple SUV occluded?
What the size of the aeroplane whose wing occluded? Small
filterjarroplane]
obj_to_parc
filter_ (iter_par[wwnn]
filter_ filter_occlude occlludee
part_to _o_obj
query_size
What size aeroplane whose wing occluded?
Figure 1: Examples from Super-CLEVR-3DD We introduce the task of 3awawere VQA, which requires 3D understanding of the image, including the parts, 31D poses, and occlusions.
We address the challenging 3D-aware VQA task from both the dataset and the model perspective. From the dataset perspective, we introduce Super-CLEVR-
which extends the Super-CLEVE dataset [32 with 3D-aware questions. Given the visual scenes from Super-CLEVR that contain randomly placed vehicles of various categories, we define set of 3D-aware reasoning operations and automatically generate 3D questions based on these operations. Fig. [ [howws examples of the images, questions and the underlying 3D operations for the questions. From the model perspective, we introduce VQA, VQA model that marries two powerful ideas: probabilistic neural symbolic program execution for reasoning and deep neural network with 3D generative representations of objects for robust visual scene parsing. Our model first recovers a 3D scene representation from the image and program from the question, and subsequently executes the program the 3D scene representation to obtain an answer using a probabilistic reasoning process that takes into account the confidence of predictions from the neural network. We refer our system as PO3D-VQA, which stands for Parts, Poses, and Occlusions in 3D Visual Question Answering.
On Super- Cieer-CEEVR-3.... we experiment with existing representative models, their variants, and our model PO3D-VQA. The results show that our model outperforms existing methods significantly, leading to an improvement in accuracy of more than 11%, which shows the advantage of the generative 3D scene parser and the probabilistic neural symbolic reasoning process. Moreover, further analysis on questions with different difficulty levels reveals that the improvements of our model are even greater on harder questions with heavy occlusions and small part sizes. Our results indicate that reliable 3D understanding, together with the modular reasoning procedure, produces a desirable 3D-aware VQA model.
In summary, our contributions are follows. (1) We introduce the challenging task of 3D-aware VQA and propose the Super-CIEVRR331 dataset, where 3D visual understanding about parts, 3D poses, and occlusions are required. (2) We propose a 31D-aware neural modular model PO3D-VQA that conducts probabilistic reasoning in a step-wise modular procedure based on robust 3D scene parsing, (3) With experiments, we show that 3D-aware knowledge and modular reasoning are crucial for 3D-aware VQA, and suggest future VQA methods take 3D understanding into account.
2 Related Work
Visual Question Answering (VQA). Rapid progress has been made in VQA in both the datasets
and the models. To solve the challenging VQA datasets [15
61
17 45
with real images, multiple
models are developed including two-stream feature fusion 2 14
28
55
23
30 or transformer-
However, the real datasets are shown to suffer from spurious 26 (27]. Alternatively, synthetic datasets like CLEVR (24) and Super- CLevr- 32 are developed to study the compositional reasoning ability of VQA systems, tasks [34
based pretraining [48 36
31
59
25
correlations and biases 42
16
41 15
which are also extended to study other iision-ndd-laguugge synthetic datasets promote the development of neural modular methods B
29
53
58
6 47
20 The
54
40
22 where the
reasoning is done in a modular step-by step manner. It is shown that the modular methods have
nice properties including interpretability, data efficiency [54] 40 better robustness (32 and strong performance on synthetic images [54). However, most existing methods rely on region features
2 [9] extracted using 2D object detectors (46) for image encoding, which is not 3D-aware. We
follow the works the synthetic dataset and enhance the modular methods with 3D understanding.

---

VQA in 3D. Multiple existing works study VQA under the 3D setting, such as SimVQA 81. SQA3D 39 SDMV-VQA [9. CLEVR-3D [5T], ScanQA [52] 3DQA [52], and EmbodiedQA [3] which
focus on question answering on the 3D visual scenes like real 3D scans 39 51 5 52
simulated 3DD
environments [3], or multi-view images [9]] PTR (20 is a synthetic VQA dataset that requires part-based reasoning about physics, analogy and geometry. Our setting differs from these works because we focus on 3D in the questions instead of 3D in the visual scenes, since our 3D-aware questions explicitly query the 3D information that can be inferred from the 2D input images.
3D scene understanding. One popular approach for scene understanding is to use the CLIP features pretrained on large-scale text-image pairs and segment the 2D scene into semantic regions I0 43 However, these methods lack a 3D understanding of the scene and cannot be used to answer 3D- related questions. Another approach is to adopt category-level 6D pose estimation methods that can locate objects in the image and estimate their 3D formulations. Previous approaches include classification-based methods that extend Faster R-CNN model for 6D pose estimation (60, [8 and compositional models that predicts 6D poses with silllyss-----ythssiii [38]. We also notice the huge progress of 3D vision language foundation models, which excel in multiple 3D vision-language
understanding tasks [9 37 21). Still, we focus on the reasoning with compositional reasoning that brings more interpretability and robustness 322
3 Super-CLEVR-33 Dataset
To study 3D-aware VQA, we propose the Super-CLEVR-3D dataset, which contains questions explicitly asking about the 3D object configurations of the image. The images are rendered using scenes from the Super-CLEVR dataset [2], which is VQA dataset containing synthetic scenes of randomly placed vehicles from 5 categories (car, plane, bicycle, motorbike, bus) with various of sub-types (e.g. different types of cars) and attributes (color, material, size). The questions are generated by instantiating the question templates based on the image scenes, using a pipeline similar to Super-CLEVE In Super-CLEVRR-3. three types of 3D-aware questions are introduced: part questions, 3D pose questions, and occlusion questions. the following, we will describe these three types of questions, and show the new operations we introduced for our 3D-aware questions about object parts, 3D poses, and occlusions. Examples of the dataset are shown in Fig. [
Part questions. While in the original Super-CLEVR dataset refers to objects using their holistic names or attributes, objects are complex and have hierarchical parts, as studied in recent works 33 11 20). Therefore, we introduce part-based questions, which use parts to identify objects (e.g. "which vehicle has red door") or query about object parts (e.g. "what color is the door of the car"). To enable the generation of bart-based questions, we introduce two new operations into the reasoning programs: part_ to_ obojjectt (.), which find the objects containing the given part, and abject_to_paat (-), which select all the parts of the given object. We also modify some existing operations (i.e. filter, query and unique), enabling them to operate on both object-level and pert-level. With those reasoning operations, we collect 9 part-based templates and instantiate them with the image scene graph to generate questions.
3D pose questions. Super-CLEVR-33 direction the car facing in"", "n"", or the pair-wiss wise pose relationships between objects (e.g. "which object has vertical direction with the red car"). The pose for an individual object (e.g. "facing left") can be processed in similar way as attributes like colors, so we extend the existing attribute- related operations like filter and query to have them include pose as well. For pair-wise pose relationship between objects, we add three operations, i.e. same_pose, apposite__ort and vertical_pose to deal with the three types of pose relationships between objects. For example, opposite __toeeeeeee___e (-) returns the objects that are in the opposite pose direction with the given object. 17 templates are collected to generate 3D pose questions.
asks questions about the 3D poses of objects (e.g. "which
Occlusion questions. Occlusion questions ask about the occlusion between entities (i.e. objects or parts). Similar to 3D poses, occlusion can also be regarded as either an attributes for an entity (e.g. "which object is occluded"),, or as relationship between entities (e.g. "which object occludes the car door"). We extend the autribute-relate operations, and introduce new operations to handle the pair-wise occlusion relationships: silttr_occludee occluded, relate, -oclluiin which finds the entities that are occluded by the given entity, and relate gellte_occluded occluded which finds the entities that are occluding the given entity. Using these operations, 35 templates are collected to generate the occlusion questions.
which filters the entities that are being
3

---

4 Method
Image
Answer: small
3D-aware scene representations (0, P,A .H.S3
Execution
query_
part_to_4k [bj
Scene Parsing
fitor pocruecdddedd
①
3D-NMS
greedy proposal generation
attribute/part prediction
Larget Teat.
Sherlwing!
bus
aeroplane
compare
2
cbi,
car
#tterlplane]
reconssructed
Question:
What the size of the plane whose wing accluded?
Kore
Figure 2: An overview of our model PO3D-VOR VQA. The image is parsed into 3D-aware scene represen- tations (blue box) using our proposed scene parser based on the idea of render-and-compare (green box). The question is parsed into program composed of reasoning operations (orange box). Then the operations are executed on the 3D-aware scene representations to predict the answer.
In this section, introduce PO3D-VQA, which is parse-then-execute modular model for 3D-aware VQA. The overview of our system is shown in Fig. 2 We first parse the image into a scene graph representation that is aware of 3D information like object parts, 3D poses and occlusion relations, then we parse the question into reasoning program and execute the program on the derived scene representations in probabilistic manner. In Sec. [41 we define the scene representation required; in Sec. we describe how we parse the image into the scene representation based on multi-class 6D pose estimation model with non-trivial extensions; in Sec, 4.3 we describe how the question is executed on the derived scene representation to predict the answer.
4.1 3D-wware scene representation
Given input image 7, parse it into 3D-aware scene representation R that contains the objects (O) with attributes (44), the parts (P) with attributes (AP), the hierarchical relationships between objects and parts (H), and the occlusion relationships between them (S). The attributes include the 3D poses and locations of objects or parts, as well as their colors, materials, and sizes. The scene representation
{0, P, A AP, H,S) S) is comprehensive and therefore we can directly execute the symbolic reasoning module on this representation without taking into account the image any further.
In more detail, objects represented as matrix O R containing the probability scores of each object being a certain instance, where n is the number objects in the given image and Notj is the number of all possible object categories in the dataset (i.e. vocabulary size of the objects). Similarly, parts
P E RpxNNet where p is the number of parts in the image and Nrr is the vocabulary size of the object parts. The object-part hierarchy is represented by if the object contains the part j or Hęj O otherwise. The attributes A E RaxNNu and AP E RPxNNa containing probability scores of each object or part having a certain attribute or the value bounding box. Here Now is the number of attributes including the 3D poses, location coordinates, colors, materials and sizes. Occlusion relationships represented by S E [g(+t))x where each element Süj represents the score of object (or part) i being occluded by object j.
represented
binary matrix H E R where H.j
4.2 Multi-class 6D Scene Parsing
While most existing VQA methods 2 [59 encode the image using pretrained object detectors like Dester-RCNy (66), we build our 6D-aware scene parser a different way, based on the idea of mallyis-bb-synnheen through inverse rendering [4] which has the following advantages: first, the model prediction more robust [49] as the render-and-compare process can naturally integrate robust reconstruction loss to avoid distortion through occlusion; second, while the object parts

---

(1)
diribike bicyle
(b)
[c]
(d)
(II)
bus
aeroplane
bicycle
motorbike
Figure 3: Visualization of intermediate steps in our parser. Given image (a), per-category feature activation maps (shown in II) are computed through render-and- sonder-and-oopare Then the category- wise competition (3D-NMS) is performed (results shown in b) and post-filtering step is taken to remove mis-detected objects (c). Based on the pose estimation results (d), we project the 3D object mesh back onto the image to locate parts and occlusions(e).
are usually very challenging for Faster- to detect due to their small size, they can be much easier located using the 3D object shape, by first finding the object and estimating its 3D pose, and subsequently locating the parts using the 3D object shape (as shown in our experimental evaluation).
However, observe two open challenges for applying existing 6D pose estimators that follow (a) these pose estimators assume that the object class is the scene parser must learn to estimate the object class jointly with the pose; and (b) the scenes in Super- Eiper-CLEVR-3D are very dense, containing multiple close-by objects that occlude each other. order address these two challenges, introduce several improvements over (38] that enable it to be integrated into a 3D-aware VQA model.
render-and-compare approach 38 49] known, but in Super-CLEVR--S
In the following, first describe neural meshes 449 38), which proposed in prior work for pose estimation of single objects following an extend this method complex scenes with densely located and possibly occluded objects to obtain coherent scene representation, including object parts and attributes.
by--ynthesyiisss approach. Subsequently, we
Preliminaries. Our work builds on and significantly extends Neural Meshes 38 that were introduced for 6D pose estimation through inverse rendering. The task is to jointly estimate the 6D pose (2D location, distance to the camera and 3D pose) of objects in an image. An object category is represented with a category-level mesh [49] M, = (un [R
J==1 and neural texture T, E Nxc on the surface of the mesh My. where is the dimension of the feature and y is the object category. Given the object 3D pose in camera view a, we can render the neural mesh model Og = [My, y Tg into a feature map with soft rasterization [35] Fy (a) ROO,, a:). Following prior work in pose estimation [49 formulate the render-an conder-and-oee com-coee process as an optimization the likelihood model:
p(F F Oy, Og. B)
II p(Je 1g, SEFG
P(' B) iE8g
II
where FG and BG are the set of foreground and background locations on the 2D feature map and fa the feature vector of F location Here the foreground and background likelihoods are modeled as Gaussian distributions.
To train the feature extractor 4 the neural texture {T,) and the background model B jointly, utilize the EM-type learning strategy as originally introduced for keypoint detection CoKe Specifically. the feature extractor is trained using stochastic gradient descent while the parameters of the generative model {T, } and B are trained using momentum update after every gradient step in the feature extractor, which was found to stabilize training convergence.
At inference time, the object poses a can be inferred by minimizing the negative log- liglliillhood w.r.t. the 3D pose using gradient descent [38]
Multi-object competition with 3D-NMS. We extend Neural Meshes to predict the 6D object pose and class label in complex multi-object scenes. In particular, we introduce ODD-Non-Maximung Suppression (3D-NMS) into the maximum likelihood inference process. This introduces a competition between Neural Meshes of different categories in explaining the feature map. In contrast classical
5

---

2D-NMS, our 3D-NMS also takes into account the distance of each object to the camera and hence naturally enables reasoning about occlusions of objects in the scene.
We denote the 6D pose as 7 = { I), where x = {a, {8} represents the 3D object pose a and object distance the camera 3, and I is the 2D object location in the feature map. We first detect the 61)
poses of each object category independently and apply 2D-NMS such that for each 2D location l' in a neighborhood defined by radius r, the predicted 6D pose {x, I yields the largest activation:
max P 2.
s.t. p(F 1) p(F 1z),,, VI' E {(' 0 11' - 11 <}
(2)
We enable multi-category, 6D pose estimation by extending this formulation to 3D non-maximum suppression (3ID-MSS). Using y/ to represent the set of all object categories, we model the category label y from a generative perspective:
max p(F F 1,,,y) s.l. p(F F Ix, 1,,,,))
p I,, V'' E (1 00 11 - -1
(3)
and ((F 1, p 21,1,1)) AA tyEY
(4)
Dense scene parsing with greedy proposal generation. Typically, object detection in complex scenes requires well chosen thresholds and detection hyperparameters. Our render-and-compare approach enables us to avoid tedious hyperparameter tuning by adopting a greedy approach to maximize the model likelihood (Eq. ) using a greedy proposal strategy. In particular, we optimize the likelihood greedily by starting from the object proposal that explains away the most parts of the image with highest likelihood, and subsequently update the likelihood of the overlapping proposals taking into account, that at every pixel in the feature map only one object can be visible [56]. Formally, given list of objects proposals for (Oy.i: {]_ (with predicted category label y and 6D) pose a), we first order the object proposals based on their likelihood score s = p( F o, B) such that
8, S 83 for < j. Based on the ordering, we greedily update the 6D pose a and the corresponding proposal likelihood for object 0j by masking out the foreground regions of previous objects with
1 isis S j 1. In this way, we can largely avoid missing close-by objects or duplicated detection.
Part and attribute prediction. Given the predicted location and pose of each object, we project the object mesh back onto the image to get the locations for each part. To predict the attributes for the objects and parts, we crop the region containing the object or part from the RGB image, and train an additional CNN classifier using the cropped patches to predict the attributes (color, size, material) and the line-grained classes (i.e. different sub-types of cars) of each patch using a cross-entropy loss. The reason why this additional CNN classifier needed instead of re- usiug the features from the 6D pose estimator is that the pose estimation features are learned to be invariant to scale and texture changes, which makes it unsuitable for attribute prediction.
Finally, we post-process the located objects using the fine-grained CNN classifier. We compare the category labels predicted by the 6D pose estimator with the ones predicted by the CNN classifier, and remove the objects for which these two predictions do not agree. This post-filtering step helps with the duplicated detections that cannot be fully resolved with the 3D-NMS.
Post-filtering.
Summary. Fig. 2 provides an overview of our scene parser and Fig. 3 visualize the intermediate results. With the idea of render-ande annder--nd-oopprr (shown in the green box of Fig. 2) the model first computes an activation map for each possible object category (Fig. II) Next, to infer the category for each object, the category-wise competition 3D-NMS is performed (Fig. B)) and a filterilt step is taken to remove mis-detected objects (Fig. 3D.. Fig. 31 shows the 6D pose estimation results. To predict parts, we project the 3D object mesh back onto the image to locate parts based projected objects (Fig. (pp). In this way, the input image can be parsed into 3D-aware representation, which is ready for the question reasoning with program execution.
4.3 Program execution
After the 3D-aware scene representations are predicted for the given image, the question is parsed into a reasoning program, which is then executed on the scene representation to predict the answer. model is trained to parse the question into its corresponding program. Like P-NSVQA (2), each operation in the program is executed on the scene representation in a probabilistic way. In the following, we describe the execution of the new operations introduced.
The question parsing follows previous work [54], where a LSTM
peruencc-to-sqqueer
The part-related operators are implemented by querying the object-part hierarchy matrix H, so that the object containing given part (part_to _t__t___oo__ to__tjeeooetteee object) and the parts belonging to the given object

---

(object_too pect_tt_part)) can be determined. The pose-related operators are based on the estimated 3D pose in the object attributes A". For the filter and query operations regarding pose, the 3D poses are quantified into four direction (left, right, front, back). For the pair-wise pose relationships, the azimuth angle between two objects is used to determine the same/opposite/erticaal directions. The occlusion-related operations are implemented by querying the occlusion matrix S. Based on the occlu- sion scores Syy representing whether entity i being occluded by entity j. we can compute the score of one entity being occluded Σ, Süj (filter_ _ccludeee) find the entities that occlude a given entity (relate_occluded) or find the entities that are occluded by a given entity (relate_occluded)
Experiments
5.1
Evaluated methods
We compare our model with three representative VQA models: FiLM 44, mDETR 225] and PNSVQA [32]. Additionally, we introduce a variant of PNSVQA, PNSVQA+Projcctio,, to analyze the benefit of our generative 6D pose estimation approach.
FiLM 44 Feature-wise Linear Modulation is a representative two-stream feature fusion method. The FiLM model merges the question features extracted with GRU [1] and image features extracted with CNN and predicts answers based on the merged features.
mDETR (25] mDETR is a pretrained text-guided object detector based on transformers. The model is pretrained with 1.3M image and text pairs and shows strong performance when finetuned on downstream tasks like referring expression understanding or VQA.
PNSVQA (32) PNSVQA is a SoTA neural symbolic VQA model. It parses the scene using MaskR- CNN I8 and an attribute extraction network, then executes the reasoning program on the parsed visual scenes with taking into account the uncertainty of the scene parser. To extend PNSVQA to the 3D questions in Super-CLEVR-33 3D, we add a regression head in the attribute extraction network to predict the 3D posefor each object; parts are detected in a similar way as objects by predicting 2D bounding boxes; the part-object associations and occlusions are computed using intersection-over union: part belongs an intersected object if the part label matches the object label, otherwise it is occluded by this object.
PNSVQA+Projection Similar with NSVQA, this model predicts the 6D poses, categories and attributes using MaskRCNN and the attribute extraction network. The difference is that the parts and occlusions are predicted by projecting the 3D object models onto the image using the predicted 6D pose and category (same with how we find parts and occlusions in our model). This model helps us ablate the influence of the two components in model, i.e. 6D pose prediction by render-and- compare, and part/occlusion detection with mesh projection.
5.2 Experiment setup
dataset shares the same visual scenes with Super-CLEVR dataset. We re-render the images with more annotations recorded (camera parameters, parts annotations, occlusion maps). The dataset splits follow the Super-CLEV training. for validation, and 5k for testing. For question generation, we create templates for part questions, 17 templates for pose questions, 35 templates for occlusion questions (with and without parts). For each of the three types, 8 to 10 questions are generated for each image by randomly sampling the templates. We ensure that the questions are not ill-posed and cannot be answered by taking shortcuts, i.e. the questions contain no redundant reasoning steps, following the no-redundancy setting in (32 More details including the list of question templates can be found in the Appendix.
Dataset. Our
dataset, where we have 20k images for
Implementation details. We train the 6D pose estimator and CNN attribute classifier separately. We train the 6D pose estimator (including the contrastive feature backbone and the nerual mesh models for each of the 5 classes) for 15k iterations with batch size 15, which takes around hours on NVIDIA RTX A5000 for each class. The attribute classifier, which is ResNet50, is shared for objects and parts. It is trained for 100 epochs with batch size 64. During inference, it takes 22s for 6D pose estimation and 10s for object mesh projection for all the objects in one image. During inference of the 6D pose estimator, we assume the theta During 3D NMS filtering, we choose the radius r as 2, and we also filter the object proposals with threshold of 15 on the score map.
5.3 Quantitative Results
We trained our model and baselines on Super-CI.EVR-311 training split, reporting answer accuracies on the test split in Tab. [ Accuracies for each question type are detailed separately.
7

---

Table I: Model accuracies on the Super-CLEVR-3D testing split, reported for each question type, Le. questions about parts, 3D poses, occlusions between objects, occlusions between objects and parts.
Mean
Part
Pose
Occ.
Part+Oec.
FiLM 45
50.53
38.24
67.82
51.41
44.66
mDETR 25
55.72
41,52
71.76
64.99
50.47
PNSVQA 32
64.39
50.61
87.78
65.80
53.35
PNSVOA+Projection
68.15
56.30
86.70
70.70
58.90
PO3D-VQA (Ours)
75.64
71.85
86.40
76.90
67.40
Comparison with baselines. First, among all the baseline methods, the neural symbolic method PNSVQA performs the best (64.4% accuracy), outperforming the end-to-end methods mDETR and FiLM by a large margin (> 8%). This shows the advantage of the step-wise modular reasoning procedure, which agrees with the findings in prior works that the modular methods excel on the simulated benchmarks that require long-trad reasoning. Second, our model achieves 75.6% average accuracy, which significantly outperforms all the evaluated models. Especially, comparing our PO3D-VQA with its 2D counterpart NSVQA, we see that the injection of 3D knowledge brings large performance boost of 11 %,, suggesting the importance of the 3D understanding.
Comparison with PNSVQA variants. By analyzing the results of PNSVQA variants (PNSVQA, PNSVQA+Projection and our (O3D-VQAA, we show (a) the benefit of estimating object 31) poses using our
method over regression and (b) the benefit of object-part structure knowledge. First, by detecting part using 3D model projection, PNSVQA results by 4%, which indicates that locating parts based on objects using the object-part structure knowledge is beneficial. Second, by estimating object 6D poses with our generative render- and-compare method, our PO3D- VQA outperforms PNSVQA+Projection by 7% (from 68.2% to 75.6%), showing the advantage of our render-amm munder-and-com -nd---ndddrrreeee model. Moreover, looking at the per-type results, we find that the improvement of our PO3D- VQA most significant on the part-related questions (21 % improvement over PNSVQA) and part- with-occlusion questions (14%), while the accuracy on pose-related questions does not improve. The reason that part and occlusion predictions require precise pose predictions for accurate mesh projection, while the pose questions only require rough pose to determine the facing direction.
DSSV@++Projection improves the
5.4
Analysis and discussions
To further analyze the advantage PO3D-VQA over other PNSVQA variants, we compare the models questions of different difficulty levels. It is shown that the benefit model is the most significant on hard questions. In Fig. we plot the relative accuracy drop questions with different occlusion ratios and questions with different part sizes.
3 of each model on
PHSVDA-Proje:tion
0.0
0.0
0.0
0.2
0.2
0.2
8-04
0.4
0.4
0.8
0.8
0.8
1.0
0
10
15
20
25
30
1.0
max
300
150
100
50
20
1.0
max
300
150
100
20
(a) Pose wrt. Occlusion Ratio
(b) Part Part Size
[c] Part Occlusion Part
Figure 4: Analysis on questions different difficulty levels. The plots show the relative accuracy drop of models, pose questions w.r.t. different occlusion ratios (a), on part questions w.r.t. different part sizes (b), and on part+occlusion questions w.r.t. different part sizes (c).
Questions with different occlusion ratios. We sort pose-related questions into different sub-groups based on their occlusion ratios and evaluate the models on each of the sub- groups. The occlusion ratio r of a question is the minimum of occlusion ratios for all the objects in reasoning trace. We choose r from 0% to 30%, in increment of 5%. The results are shown is Fig. 4 (a). Our PO3D-VQA much more robust occlusions compared to the other methods: while the performances of all
'Relative accuracy drop means the ratio of absolute accuracy drop and the original accuracy. For example, if model's accuracy drops from 50% to 45%, its relative accuracy drop 10%.
8

---

the three models decrease as the occlusion ratio increases, the relative drop of ours is much smaller than others. The results show that our render-and-conee and-compare scene parser is more robust heavy occlusions compared with the discriminative methods.
Questions with different part sizes. Questions about small parts harder than the ones about larger parts. We sort the questions into different part size intervals (s, where the largest part that the question refers has an area (number of pixels occupied) larger than and smaller than t. We compare the models on the part questions and the part+occlusion questions with different part sizes in Fig. (b) and (c). In (b), the accuracy drop of PO3D- VQA is smaller than PNSVoA+Projection and PNS VQA when parts get smaller. In (c), PNSVON ANNVOA+Projcctioon is slightly better than our model and they are both better than the original PNSVQA.
In summary, by sorting questions into different difficulty levels based on occlusion ratios and part sizes, show the advantage of our PO3D-VQA on harder questions, indicating that model is robust to occlusions and small part sizes.
Qualitative results. Fig. shows examples of predictions for our model and PNSVQA variants. In (a), the question asks about occlusion, but with a slight error the pose prediction, PNSVOA+Projection misses the occluded bus and predicts the wrong answer, while our model is correct with accurate pose. In (b), the question refers to the heavily occluded minivan that is difficult to detect, but our model gets the correct prediction thanks to its robustness to occlusions.
Ours
PNSVOA-Projection
(a)
Q: What the material gray object that is occluded? rubber
Ours: rubber
x PNSvQA+Praj: metal
x PNSVQA: metal
(b)
Which direction is the minivan facing?
A: left
left
X PNSVQA +Proj.: right
PNSVQA: front
Figure 5: Examples models' predictions. Our model (a) predicts the object pose accurately and (b) is robust to heavy occlusions. Red boxes are for visualization only.
Limitations and failure cases. Due to the difficulties of collecting real images with compositional scenes and 3D) annotations, our work is currently limited by its synthetic nature. For PO3D VQA, it sometimes fails detect multiple objects if they are from the same category and heavily overlap (see Appendix D for more visualizations). 3D NMS can effectively improve the dense scene parsing results when objects are from different categories, but conceptually it is limited when objects are from the same category. However, 6D pose estimation in dense scenes is a challenging problem, whereas many current works on 6D pose estimation are still focusing on simple scenes with single objects 38 57).
6 Further Discussion
In this section, we discuss two meaningful extensions of our work: the incorporation of z-direction questions and the application of our model to real-world images.
Z-direction questions. While the proposed Super-CLEVR-31 dataset has been designed with 3D-aware questions, all objects within it are placed on the same surface. Introducing variability in the z direction can further enrich dataset with more comprehensive 3D spatial relationships.
We consider the scenario where aeroplane category, is different elevations, introducing the z dimension into the spatial relationships (see Fig- (). This allowed us to formulate questions that probe the model's understanding of height relationships and depth perception. We create a subset containing 100 images and 379 questions and test our PO3D-VQA model directly on it without retraining the 6D

---

parser. On this dataset, our PO3D model achieves 90.33% aceuracy on height relationship questions and 78.89% on depth-related questions, suggesting that our model can successfully handle questions about height. As the baseline models only use the bounding box to determine the spatial relationship between objects, they are not able to determine the height relationships.
Height question: There is blue object that is below the biplans; what shape it? Answer: Tandem
Height question: How many objects are above the shiny bicycle?
Answer: 2
Depth question: the biplane closer than the motorbike? Answer: Yes
Depth question: Does the truck have greater distance than the shiny bus?
Answer: Yes
Figure 6: Example images and questions of objects with different elevations.
Extension to real-world images While our PO3D-VQA model has demonstrated impressive perfor- mance the synthetic Super-CLEVR-35 dataset, essential research direction is extending it to real images or other 3D VQA datasets (such as GQA and FE-3DGQA). However, it's not trivial to these real-world problems, and primary challenge is the lack of 3D annotations
truly evaluate and the highly articulated categories (like the human body) in these datasets.
However, we show that our PO3D-VQA model can, in principle, work on realistic images. We generate several realistic image samples manually using the vehicle objects (c.g. car, bus, bicycle) from ImageNet with 3D annotation (sce Fig. 7 and real-im real-image background. In this experiment, the pose estimator is trained on the PASCAL3D+ dataset, and is used to predict the poses of objects from the image before pasting, shown in (b). The attribute (color) prediction module is trained on Super-CI CLEVR-3D and the object shapes are predicted by a ResNet trained on ImageNet. Our model correctly predict answers to questions about the object pose, parts, and occlusions, e.g. "Which object is occluded by the mountain bike".
does mountain (b1)
(82)
(b2)
[Pose]
Which direction does race car face Right
[Part] Q: What is the color the fin that belongs to the seroplane? Our: Red
W.Wa''ss the shape of the object that has wing? Our. Aeroplane
[Dcc.] Q: Which object is occluded by the mountain bike? Our: Troleybus
a: What the color the accluded object? Our: Red
(c1)
(o2)
Figure 7: Examples results on realistic images. Given a realistic image (al, a2), our model can successfully estimate the 6D poses of objects (b1, b2) and answer the 3D-aware questions (cl, c2),
Conclusion
In this work, we study the task of 3D-aware VQA. We propose the Super-CLEVR-311 dataset containing questions explicitly querying 3D understanding including object parts, 3D poses, and occlusions. To address the task, 3D-aware neural symbolic model PO3D- VQD proposed, which enhances the probabilistic symbolic model with robust 3D scene parser based on analysis- by-synthesis. With the merits of accurate 3D scene parsing and symbolic execution, our model outperforms existing methods by large margin. Further analysis shows that the improvements are even larger on harder questions. With the dataset, the model, and the experiments, we highlight benefit of symbolic execution and the importance of 3D understanding for 3D-aware VQA.
A Acknowledgements
We thank the anonymous reviewers for their valuable comments. We thank Qing Liu, Chenxi Liu, Elias Stengel-Eskin, Benjamin Van Durme for the helpful discussions on early version the project. This work supported by Office of Naval Research with grants N00014-23-1-266 N00014-21- 2812. A. Kortylewski acknowledges support via his Emmy Noether Research Group funded by the German Science Foundation (DFG) under Grant No.468670075.
10

---

References
[1] Aishwarya Agrawal, Dhruv Batra, and Devi Parikh. Analyzing the behavior of visual question answering models, arXiv preprint 1606.066600073333 2016.
[2] Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEFF. conference on computer vision and pottern recognition, pages 6077-6086, 2018.
[3] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39-48, 2016.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEFE international conference on computer vision, pages 2425-2433, 2015.
[5] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19129-19 19139, 2022.
[6] Dzmitry Bahdanau, Harm de Vries, Timothy J O' O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv: 057085883 2019.
[7] Yutong Bai, Angtian Wang, Adam Kortylewski, and Alan Yuille. Coke: Contrastive learning for robust keypoint detection. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 65-74, 2023.
[8] Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. Rubi: Reducing unimodal biases for visual question answering. Advances neural information processing systems, 32, 2019.
[9] Paola Cascante- Bonilla, Hui Wu, Letao Wang, Rogerio S Feris, and Vieente Ordonez. Simvqa: Exploring simulated environments for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5056-5066, 2022.
[10] Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. arXiv preprint arXiv:2301.04926 2023.
[11] Xianjie Chen, Roozheh Mottaghi, Xianhai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can:
Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1971-1978, 2014.
[12] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint 1406v:140..1077 2014.
[13] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra, Embodied question answering. In Proceedings of the IEEE conference computer vision and pottern recognition, pages 10, 2018
[14] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint 1606. 1606.6664/..00 01847, 2016.
[15] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding Visual Question Answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[16] Vipul Gupta, Zhuowan Li, Adam Kortylewski, Chenyu Zhang, Yingwei Li, and Alan Yuille. Swapmix: Diagnosing and regularizing the over-reliance on visual context in visual question answering. arXiv preprint 2204
2022.
[17] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz, grand challenge: Answering visual questions from blind people. In P Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3608 3617,,7,,44 2018
[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pages
2017.
11

---

[19] Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. arXiv preprini WrXiv:2333.11327, 2023.
[20] Yining Hong. Li Yi, Josh Tenenbaum, Antonio Torralba, and Chuang Gan. Ptr: A benchmark for part-based conceptual, relational, and physical reasoning. Advances in Neural Information Processing Systems, 34, 2021.
[21] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. arXiv preprint arXiv: 12981, 2023.
[22] Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pages 53-69, 2018.
[23] Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. 2018.
[24] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2017.
[25] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnacve, Ishan Misra, and Nicolas Carion. addetr-modllate detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[26] Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Roses are red, violets are blue... but should vqa expect them to? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2776-2785, 2021.
[27] Corentin Kervadec, Theo Jaunet, Grigory Antipov, Moez Baccouche, Romain Vuillemot, and Christian Wolf. How transferable are reasoning patterns in vqa? 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4205-4214, 2021.
[28] Jin-Hwa Kim, Jachyun Jun, and Byoung-Taie Zhang. Bilincar attention networks. Advances in Neural Information Processing Syxtems, 31, 2018.
[29] Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. Clevr-dialog: A diagnostic dataset for multi-round reasoning in visual dialog, arXiv preprint vrXiv:1903.0316: 2019.
[30] Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2019.
[31] Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre me-training for vision langugg tasks, ECCV 2020, 2020.
[32] Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan Yuille. Super-clevr: A virtual benchmark diagnose domain robustness in visual reasoning. arXiv preprint arXiv:2212.0022. 2022.
[33] Qing Liu, Adam Kortylewski, Zhishuai Zhang, Zizhang Li. Menggi Guo, Qihao Liu, Xiaoding Yuan, Jiteng Mu, Weichao Qiu, and Alan Yuille. Learning part segmentation through unsupervised domain adaptation from synthetic vehicles. In CVPR, 2022.
[34] Runtao Liu, Chenxi Liu, Yutong Bai, and Alan L Yuille. Clevr-ref+: Diagnosing visual reasoning with referring expressions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition, pages 4 4185-4194, 2019.
[35] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image- based 3d reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7708-7717, 2019.
[36] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-languaal tasks. Advances in neural information processing systems, 32, 2019.
[37] Tiange Luo, Chris Rockwell, Honglak Lee, and Justin Johnson. Scalable 3d captioning with pretrained models, arXiv preprint arXiv:2306.0722 07279, 2023.
12

---

[38] Wufei Ma, Angtian Wang, Alan Yuille, and Adam Kortylewski. Robust category-level 6d pose estimation with Moars--o--fine rendering of neural features. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings. Part IX. pages 492-508. Springer. 2022.
[39] Xiaojian Ma, Silong Yong. Zilong Zheng. Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. Sqašd: Situated question answering in 3d scenes. In International Conference on Learning Representations, 2023.
[40] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu, The Neuro-Symbole Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In International Conference on Learning Representations. 2019.
[41] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition ( (CVPR), pages 12700-12710, June 2021.
[42] Yulci Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xiansheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look language bias. 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12695- 2021.
[43] Songyou Peng. Kyle Genova, Chiyu "Max" Jiang. Andrea Tagliasacchi, Mare Pollefeys, and Thomas Funkhouser. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023.
[44] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reason- ing with a general conditioning layer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[45] Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. Advances in neural information processing systems, 28, 2015.
[46] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-can: Towards real-timm object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[47] Leonard Salewski, A Koepke, Hendrik Lensch, and Zeynep Akata, Clevr-x: A visual reasoning dataset for natural language explanations. In International Workshop on Extending Explainable Al Beyond Deep Models and Classifiers, pages 69-88. Springer, 2022.
[48] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint / 1908.0749, 2019.
[49] Angtian Wang, Adam Kortylewski, and Alan Yuille. Nemo: Neural mesh models of contrastive features for robust 3d pose estimation. In International Conference on Learning Representations, 2021.
[50] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A beuchmark for 3d object detection in the wild, In IEEE winter conference on applications of computer vision, pages 75-82. IEEE, 2014.
[51] Xu Yan, Zhihao Yuan, Yuhao Du, Yinghong Liao, Yao Guo, Zhen Li, and Shuguang Cui, Clevr3d: Compositional language and elementary visual reasoning foor question answering in 3d real-world scenes. arXiv preprint 2rXiv:2112.1169) 2021.
[52] Shuquan Ye, Dongdong Chen, Songfang Han, and Jing Liao. 3d question answering. IEEE Transoctions Visualization and Computer Graphics, 2022.
[53] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arxiv:1910.01442, 2019.
[54] Kexin Yi. Jiajun Wu, Chuang Gan, Antonio Torralba. Pushmeet Kohli, and Joshua B Tenenbaum. Neural- Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In Advances in Neural Information Processing Systems (NIPS), 2018.
[55] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6281-6290, 2019.
[56] Xiaoding Yuan, Adam Kortylewski, Yihong Sun, and Alan Yuille. Robust instance segmentation through reasoning about oulti-object occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 11141-11150 2021.
13

---

[57] Yanjie Ze and Xiaolong Wang. Category-level 6d object pose estimation in the wild: A semi-superrised learning approach and dataset. Advances Neural Information Processing Systems, 35:27469- 27483, 2022.
[58] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5317-5327, 2019.
[59] Pengehuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision-langues models. CVPR 2021, 2021.
[60] Xingyi Zhou, Arjun Karpur, Linjie Luo, and Qixing Huang. Starmap calegory-agnostic keypoint and viewpoint estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 918-334, 2018
[61] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4995-5004, 2016.
14

---

A Dataset Details
A.1 Part list
In Super- CLEVR-3D, the parts of each objects are listed in Tab.
A.2 Question templates
Part Questions collect 9 part-based templates when generating the part-based questions, as shown in Tab. 4 In the table, <attribute> means one attribute from shape, material, color or size to be queried, <object> (or <object 1>, <object 2>) means one object to be filtered with combination of shape, material, color, and size. Different from the pose and occlusion question, we don't 't query the size of the object.
3D Pose questions We design 17 3D pose-based templates in question generation (as shown in 5) The 17 templates consist of: template of the query of the pose: 4 questions of the query of shape, material, color, size, where the pose the filtering conditions; 12 templates about the query of shape, material, color, size, where the relationship of the pose is the filtering condition.
Occlusion Questions There 35 templates in the occlusion question generation shown in table which consists of occlusion of objects and occlusion of parts.
The occlusion of objects consists of occlusion status and occlusion relationship. For the occlusion status of the object, there are 4 templates to query the shape, color, material, and size respectively. There are 2 occlusion relationships of objects (occluded and occluding), and each of them has templates.
Similarly, we then create template about occlusion status and occlusion relationship for the parts. The only difference between object and part is that the parts only have attributes to be queried: shape (name), material and color.
A.3 Statistics
As result, a total of 448988 part questions, 314,986 pose questions, and 228,397 occlusion questions and 314,988 occlusion questions with parts.
In Fig. show the distributions of all attributes of objects including categories, colors, sizes, and materials
Categories
Colors
Meterials
16, SE%
12 23%
12 22.52%
32.67%
41.ER
40 00%
12.5%%
19 10%
netorbãe broAT
12.57%
$1.39%
mebi
goeen
Figure 8: Distributions for all the attributes of objects including categories, colors, sizes, and materials
B Implementation details for the baselines
The FiLM and mDETR are trained with default settings as the official implementation. FiLM is trained for 100k iterations with batch size 256. mDETR is trained for 30 epochs with batch size 64 using 2 GPUs for both the grounding stage and the answer classification stage.
For P-NSVQA, we first train MaskRCNN for 30k iterations with batch size 16 to detect the objects and parts, then train the attribute extraction model (using Res50 backbone) for 100 epochs with batch size 64. Different fully connected(FC) layers used for different type of question: the
15

---

part questions and occlusion questions have 4 FC layers for the shape, material, color, and size classification (as the parts also have size annotations in the dataset when generating scene files, but they are meaningless in the question answering). The pose question includes pose prediction of an object, so we add new FC layer with output dimension to predict the rotations, followed by an MSE loss during training. For different types of questions (part, pose and occlusion), the MaskRCNN and attribute extraction model are trained separately.
In the PNSVOA-Projection baseline, we first train MaskRCNN to detect all of the objects and predict their 3D pose (azimuth, elevation and theta) without category labels in the scene. This MaskRCNN is trained with batch size and iteration 15000. We use an SGD optimizer with learning rate of 0.02, momentum of 0.9 and weight decay 0.0001. Then, we use the same setting as our PO3D- VQA to train a CNN to classify the attributes of objects and parts.
C Detailed results of Analysis
As an extension for section 5.4 in main paper, here we include the numerical value of accuracy and drop for the pose, part, occlusion + part question with reference to occlusion ratio part size. The result shown Tab. 7 Tab. and Tab,
D Failure cases
Examples of failure cases of 1O3D-VQA described in Section 5.4 in main paper. In (a) and (b), PO3D-VQA misses the bicycle behind when two bicycles have a heavy overlap, the same for the two motorbikes (c) and (d).
(d)
Figure 9: Failure cases of our PO3D-VQA (a) and (c) is the input images with the objects missed by the model. (b) and (c) the re-projection results from the model.
16

---

Table 2: List of objects and parts.
shape
part list
airliner
left door, front wheel, fin, right engine, propeller, back left wheel, left engine, back right wheel, left tailplane, right door, right tailplane, right wing, left wing front wheel, fin, propeller, left tailplane, right tailplane, right wing, left wing left door, front wheel, fin, right engine, propeller, back left wheel, left engine, back right wheel, left tailplane, right tailplane, right wing, left wing
biplane
jet
fighter
fin, right engine, left engine, left tailplane, right tailplane, right wing, left wing left handle, brake system, front wheel, left pedal, right handle, back wheel, saddle, carrier, fork, right crank arm, front fender, drive chain, back fender, left crank arm, side stand, right pedal rearlight, front wheel, back wheel, fork, front fender, back fender
utility bike
tandem bike
road
left handle, brake system, front wheel, left pedal, right handle, back wheel, saddle, fork, right crank arm, drive chain, left crank arm, right pedal
bike
mountain
left handle, brake system, front wheel, left pedal, right handle, back wheel, saddle, fork, right crank arm, drive chain, left crank arm, right pedal
bike
articulated
left tail light, front license plate, front right door, back bumper, right head light, front left. wheel, left mirror, right tail light, back right door, back left wheel, back right wheel, back license plate, front right wheel, left head light, right mirror, trunk, mid right door, roof left tail light, front license plate, front right door, front bumper, back bumper, right head light, front left wheel, left mirror, right tail light, back left wheel, back right wheel, back license plate, mid left door, front left door, front right wheel, left head light, right mirror, trunk, mid right door, roof
bus
double
bus
regular bus
left tail light, front license plate, front right door, front bumper, back bumper. right head light, front left wheel, left mirror, right tail light, back right door, back left wheel, back right wheel, back license plate, front right wheel, left head light, right mirror, trunk, mid right door, roof
school
left tail light, front license plate, front right door, front bumper, back bumper, right head light, front left wheel, left mirror, right tail light, back left wheel, back right wheel, back license plate, mid left door, front right wheel, left head light, right mirror, roof front left door, left tail light, left head light, back right wheel, right head light, front bumper, right mirror, front license plate, front right wheel, back bumper, left mirror, back left wheel, right tail light, hood, trunk, front left wheel, roof, front right door
bus
truck
suv
front left door, left tail light, left head light, back left door, back right wheel, right head light, front bumper, right mirror, front right wheel, back bumper, left mirror, back left wheel, right tail light, hood, trunk, front left wheel, back right door, roof, front right door front left door, left tail light, left head light, back left door, back right wheel, right head light, front bumper, right mirror, front license plate, front right wheel, back bumper, left mirrrr, back left wheel, right tail light, hood, trunk, front left wheel, back right door, roof, front right door, back license plate
minivan
sedan
front left door, left tail light, left head light, back left door, back right wheel, right head light, front bumper, right mirror, front license plate, front right wheel, back bumper, left mirror, back left wheel, right tail light, hood, trunk, front left wheel, back right door, roof, front right door, back license plate
wagon
front left door, left tail light, left head light, back left door, back right wheel, right head light, front bumper, right mirror, front license plate, front right wheel, back bumper, left mirror, back left wheel, right tail light, hood, trunk, front left wheel, back right door, roof, front right door, back license plate
chopper
left handle, center headlight, front wheel, right handle, back wheel, center taillight, left mirror, gas tank, front fender, fork, drive chain, left footrest, right mirror, windscreen, engine, back fender, right exhaust, seat, panel, right footrest left handle, center headlight, front wheel, right handle, back cover, back wheel, center taillight, left mirror, front cover, fork, drive chain, right mirror, engine, left exhaust, back fender, seat, panel
scooter
cruiser
left handle, center headlight, right headlight, right taillight, front wheel, right handle, back cover, back wheel, left taillight, left mirror, left headlight, gas tank, front cover, front fender, fork, drive chain, left footrest, license plate, right mirror, windscreen, left exhaust, back fender, right exhaust, seat, panel, right footrest left handle, front wheel, right handle, back cover, back wheel, gas tank, front cover, front fender, fork, drive chain, left footrest, engine, right exhaust, seat, panel, right footrest
dirtbike
17

---

Table 4: Templates of parts questions
Templates
Count
What is the <attribute> the <part> of the <object>?
What is the <attribute> the <object> that has <part>?
What is the <attribute> the <part 1> that belongs the same object the <part 2>?
Table 5: Templates of pose questions
Templates
Count
Which direction the <object> is facing?
What is the <attribute> of the <object> which face to the <D>?
4
What is the <attribute> of the <object 1> that faces the same direction as a <object 2> What is the <attribute> of the <object 1> that faces the opposite direction as a <object 2>
4
What is the <attribute> of the <object 1> that faces the vertical direction a <object 2>
4
Table 6: Templates of occlusion questions
Templates
Count
cobject> that occluded?
What the cattribute> the <object 1> that occluded by the <object 2>
4
What the <atte ibute> cobject 1> that occludes the cobject 2>?
Is the <part> the <object> occluded? Which part of the <ohjact> occluded?
What the <attribute> the <object> whose <part> is occluded?
4
What the <attribute>
<part> which belongs to an occluded <object>?
What the <attr lbute> the spart which belongs the <objects whose spart 2> is occluded?
Is the spart> the cobject 1> occluded by the <object
What the sbutrebute of the <object 15 whese spart> occluded by the <object
What <attz ibute> <part>
belongs to cobject which occluded by the cobject 2> What the <attr tbute> the <part which helongs to the same object whose spart. 2> occluded by the cobjact 2>?
Table 7: Accuracy value and relative drop for pose questions occlusion ratio
Occlusion Ratio
10
15
25
30
Accuracy
87.43
74.09
74.09
63.16
62.01
60.33
58.52
PNSVQA
Drop
0.00%
15.26%
15.26%
27.76%
29.08%
31.00%
33.07%
Accuracy
86.30
74.61
67.20
66.78
60.26
56.52
$5.56
PNSVQA Projection
Drop
0.00%
13.54%
22.13%
22.62%
30.17%
34.51%
35.63%
Accuracy
86.43
86.05
84.32
75.00
79.44
73.22
67.98
Ours
Drop
0.00%
0.44 %
2.44 %
13.22%
8.09 %
15.28 %
21.35%
Table 8: Accuracy value and relative drop for occlusion + part wrt. part size
Part Size
max
300
150
100
50
20
Accuracy
58.18
54.98
54.05
52.09
45,20
21.28
PNSVQA
Drop
0.00%
5.49%
7.10%
10.47%
22.3 %
63.43%
Accuracy
61.85
50.64
56.77
53.97
55.29
45.83
PNSVQA + Projection
Drop
0.00%
18.1 %
8.20%
12.74%
10.60 %
25.89 %
Accuracy
81.68
75.32
77.20
71.54
67.00
53.19
Ours
Drop 0.00%
7.78 %
5.49 %
12.41 %
17.97%
34.88%
18

---

Table 9:
Accuracy value and relative drop for part wrt. part size
Part Size
max
100
20
Accuracy
57.31
51.00
37.50
44.18
40.85
29.73
PNSVQA
Drop
0.00%
11.02%
34.57%
22.92%
28.73%
12%
Accuracy
58.89
57.54
42.64
43.20
46.73
38.67
PNSVQA + Projection
Drop
0.00%
2.30%
27.60%
26.65%
20.65 %
34.34%
Accuracy
64.04
64.80
60.16
57.03
49.05
55.41
Ours
Drop
0.00%
-1.19%
6.06 %
10.94 %
23.41% %
13.48 %
19


Reviews:
[{"review_id": "T3QXAT6dme", "reviewer": "Reviewer_sJrg", "summary": "This paper is seeks to improve the VQA models' understanding of 3D structure of images, particularly parts, poses, and occlusions.  There are two main contributions: \n1. Super-CLEVR-3D: a dataset that contains questions about parts, poses, and occlusions.\n2. PO3D-VQA: a 3D aware VQA model that combines nuerosymbolic program execution for reasoning and 3D generative representations. \n\nExperimental settings: the proposed model is tested on the proposed dataset Super-CLEVR-3D.", "questions": "1. Has jointly training the pose estimator and attribute classifier been explored? Could training them jointly limit the erroneous pose prediction instances?\n2. How is the performance on the original Super-CLEVR dataset or on the CLEVR dataset?\n3. Does the method generalize well to cases where parts/poses/occlusions are not mentioned? Can the proposed model be reliably used for VQA outside of the Super-CLEVR setting?\n4. Can this model be used for real-world images with questions about parts/poses/occlusions?", "limitations": "Limitations are discussed [line 348]. Discussions along the lines of Q2/3/4 above would also be useful when describing the limitations.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The dataset is a good contribution for studying 3D vision-language reasoning.\n2. The proposed model, PO3D-VQA is well-explained and motivated, along with the addition of a 6D parsing module and 3D NMS for better scene understanding.\n3. The proposed model is a good proof-of-concept for the combination of neurosymbolic program execution and features learned using deep learning. \n4. Relevant baselines are used and the experimental setting is soundand well explained.", "weaknesses": "1. The proposed model is only tested on the Super-CLEVR-3D dataset\n2. There aren't any questions about \"z\" direction i.e. above/below relationships as the objects in [31] are always on the same surface.  Because of this it is questionable to call the dataset \"3D\" as only 2D relationships are covered. The dataset is missing templates/questions about depth, distance between objects etc.\n4. Super-CLEVR-3D dataset only 5 categories, all of which are forms of ‘vehicles’. This could also lead to an overestimation of the object detection module that is used in the model. \n5. The method could be tested on several other 3D aware datasets: for instance GQA [Hudson et al CVPR 2019], or this work https://arxiv.org/abs/2209.12028. Also see Q2, Q3, Q4 for more questions about evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the detailed discussions and the appreciation of our work. We would like to address the concerns and questions you raised: \n\n**Q0 & A0: Please refer to the general response for generalization to real images.**\n\n**Q1: There aren't questions about “z” direction (“above/below” relationships) as the objects are always on the ground.**\n\nA1: We thank the reviewer for the suggestion. As suggested, we create new images where some airplanes are flying in the air (while the cars are kept on the ground plane). Examples of the images are shown in Fig-2 of the rebuttal PDF. For the new images, we ask “z-questions” about height relationships and depth. To be more specific, the height relationship asks about “above/below” relationships that can be used to query the objects (e.g. “What shape is the cyan thing above the blue shiny car?”); for the depth question we add the comparison of depth between two objects where the size and bounding box location is not sufficient for the prediction (e.g. “Is the aeroplane closer to the camera than the school bus?”). In this way, we create a subset containing 100 images and 379 questions and test our PO3D model on this subset.\n\nWe test the PO3D-VQA model directly on the new dataset without retraining. On this dataset, our PO3D model achieves 90.33% accuracy on height relationships questions and 78.89% accuracy on the distance-based question suggesting that our model is not limited to objects on the ground plane and can successfully handle questions about height. As the baseline models only use the bounding box to determine the spatial relationship between objects, they are not able to determine the height relationships.\n\n\n**Q2: Super-CLEVR-3D dataset only 5 categories, all of which are forms of vehicles.**\n\nA2: Our model is trained from 3D CAD models that have detailed part annotations. Such well-annotated CAD models are difficult to obtain, therefore we are limited to the object categories covered in the UDA-Part dataset [32]. While there are large-scale 3D object models like Objaverse [A] defining and annotating parts on these meshes is challenging and time-consuming. However, we argue that the 5 categories are already very challenging for existing models, as suggested in [31, 32]. In our work, we not only improve the performance greatly over the existing methods but also extend the dataset with different types of questions, which is a non-trivial contribution.\n\n**Q3: Could jointly training the pose estimator and attribute classifier bring better results?**\n\nA3: Thanks for the suggestion. We agree that it’s a valuable direction to study if the pose estimator can benefit from the joint training. However, there remains a non-trivial technical gap that the pose estimation process is an iterative render-and-compare process that cannot be easily integrated with an attribute classification head. Based on the significant amount of effort required, we leave this as future work.\n\n\n**Q4: How is the performance on the original Super-CLEVR dataset or on the CLEVR dataset?**\n\nA4: We test our model on the original Super-CLEVR dataset. The accuracy of the PO3D-VQA model is 88.40%, which is lower than the SOTA methods (95% PNSVQA). Given that the original dataset primarily focuses on 2D questions, precision in 2D location becomes crucial. While MaskRCNN can train the detection directly from the bounding boxes supervision, our 6D pose estimator is trained to solve 6D pose but not explicitly designed for 2D detection. In Fig-5 of the rebuttal pdf, we show that our model will miss some objects especially when multiple objects are too close or overlapping.  As the mesh-based 6D pose estimator detects the object's class by class (Fig-3 II of the main paper), it will fail to parse all the objects from one feature map of one category in such cases.  We will add this limitation analysis to the paper.\n\n**Q5: Does the method generalize well to cases where parts/poses/occlusions are not mentioned or VQA outside of the Super-CLEVR setting?**\n\nA5: First, the original Super-CLEVR dataset does not contain parts/poses/occlusions questions. Our model achieves 88.40% accuracy on the original Super-CLEVR, suggesting that the model generalizes well to this setting. Second, as mentioned in the general response, the model works reasonably well on more realistic images, which indicates that our model is not limited to the Super-CLEVR setting.\n\n[A] Objaverse: A Universe of Annotated 3D Objects. arXiv 2212.08051"}, {"review_id": "LwiO8X7IVi", "reviewer": "Reviewer_HP3a", "summary": "This work introduces a framework to Visual Question Answering (VQA) that incorporates understanding of the 3D structure of scenes, a significant leap from traditional 2D-based models. To evaluate the task, they proposed a new dataset, Super-CLEVR-3D, designed specifically for 3D-aware VQA, containing questions that necessitate compositional reasoning regarding the 3D object parts, poses, and occlusions. To tackle these queries, they also put forth a new model, PO3D-VQA, combining probabilistic neural symbolic program execution for reasoning and deep neural networks for robust 3D scene parsing. Experiments show the proposed PO3D-VQA model outperforms existing techniques, especially on more complex questions, underscoring the need for 3D understanding in future VQA research. Despite the improvements, the noticeable performance gap compared to 2D VQA benchmarks indicates that 3D-aware VQA remains a critical area of exploration.\n\n\n\n\n", "questions": "Please address the concerns raised above. ", "limitations": "It seems the current framework cannot correct itself if the perception model generated wrong outputs.\n\n", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I believe the studied direction, 3D-aware 2D-VQA is important for our community, especially when we want to tackle real-world tasks. Both the proposed VQA dataset and the framework, PO3D-VQA, takes a good attempt towards the big goal. \n\nAll the used techniques as shown in method and Figure 2 are sound and composed in a reasonable way. Overall, the paper is easy to follow.\n\nResults in Table 1 demonstrated the strength of the proposed method, which yield significant improvements, and also indicate there are more to do in the future. Some ablation studies are included in Section 5.4. ", "weaknesses": "As the paper claimed a 3D-VQA dataset contribution, I am curious how the model transfer performance from the dataset to real-images. For example, taking a picture with multiple real 3d models  and run the model to check the sim-to-real performance. \n\nAlso, an odd part is that the current motivation for this paper is 3D navigation and manipulation, but few of the defined problems are related to navigation or manipulation. \n- Can the proposed model accurately locate parts of 3D objects that can help manipulate? \n- Are there questions directly related to navigation and manipulation in 3D space?\n\nThe current question-answers in Figure1 in my mind is more related to previous VQA while less related to 3D VQA that really matters. \n\nBesides, there are no failure cases understanding and analysis. For the proposed pipeline, the failed cases should include both the failure of perception module and the failure of reasoning module. Providing failure cases can help people understand the limitation of the proposed system, while not weaken the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. Here we address the questions:\n\n**Q0 & A0: Please refer to the general response for generalization to real images.**\n\n**Q1: About 3D navigation and manipulation: Are there questions directly related to navigation and manipulation in 3D space?**\n\nA1: There are no questions about navigation and manipulation in our dataset and we will clarify this in our final paper. In the submission, we talked about 3D navigation and manipulation because this is one of the motivations as well as the long-term goals of our work. By introducing the 3D VQA we make an initial step in that direction by designing 3D questions that the current 2D baseline models are not able to answer well. We also note that for manipulation and navigation, we would need to annotate parts that are useful for these interactions, which is a direction that we plan to pursue in future work.\n\n**Q2: Can the model accurately locate parts of 3D objects that can help manipulate?**\n\nA2: In our 3D VQA setting, the model can locate the part accurately in most cases. In Fig.3 of the rebuttal PDF, we visualize the part localization results predicted by our model in the SuperCLEVR-3D dataset. But as we stated in Q1, our 3D VQA dataset is just a first step, and the parts needed for 3D manipulation or navigation might be different from those annotated in our dataset. \n\n**Q3: The current question-answers in Figure1 in my mind is more related to previous VQA while less related to 3D VQA that really matters.**\n\nA3: As stated in our paper, the question in the dataset requires a comprehensive understanding of the 3D structure of scenes, like the 3D poses, occlusion relationship, and the hierarchical relationship between objects and parts. From our experiment, we show that the 2D baseline methods cannot handle this question without the 3D understanding. In this rebuttal, we add the z-direction relationship and depth-related questions on new images, where the 2D models are not able to determine the precise height (in the z-direction) and depth (see Fig-2 in rebuttal PDF).\n\n**Q4:  Failure case analysis: does the errors come from the perception module or the reasoning module?**\n\nA4: A4: We check the failure cases and provide two visualizations in Fig-5 of the rebuttal PDF. We observe that most of the failure cases are due to the errors of the 6D pose estimator, typically when multiple objects from the **same class** are overlapping. As the mesh-based 6D pose estimator detects objects and their 6D poses by class (Fig-3 II of the main paper), it may fail to parse all objects from the same category when they are too close to each other. 3D NMS can effectively improve the dense scene parsing when objects are from different categories, but conceptually it cannot help when objects are from the same categories in a dense scene. We argue that 6D pose estimation in dense scenes is still a challenging problem and many current works on 6D pose estimation still focus on simple scenes with single objects (Ma et al. 2022; Yu et al. 2014; Yang et at. 2022).\n\n**Q5: It seems the current framework cannot correct itself if the perception model generates wrong outputs.**\n\nA5: As our reasoning module is a probabilistic neuro-symbolic method (P-NSVQA [31]), it allows the model to output correct answers even when the vision module makes errors. We also introduce new probabilistic reason functions for filtering poses and occlusion in the reasoning module. As studied in the [31], with the same perception module, P-NSVQA can outperform the NSVQA in all test sets, which means it revises the errors from the vision predictions. We believe that the probabilistic reasoning module can also help our 3D perception model to revise some errors. But as with most modular VQA models, the errors from the vision model can inevitably impact the final model's accuracy and performance, which is also good for interpretability.\n\n[1] Ma, Wufei, et al. \"Robust category-level 6d pose estimation with coarse-to-fine rendering of neural features.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Xiang, Yu, Roozbeh Mottaghi, and Silvio Savarese. \"Beyond pascal: A benchmark for 3d object detection in the wild.\" IEEE winter conference on applications of computer vision. IEEE, 2014.\n\n[3] Ze, Yanjie, and Xiaolong Wang. \"Category-level 6d object pose estimation in the wild: A semi-supervised learning approach and a new dataset.\" Advances in Neural Information Processing Systems 35 (2022): 27469-27483."}, {"review_id": "C2Neid1CMZ", "reviewer": "Reviewer_SUrx", "summary": "1, The paper extends 2D VQA to the task of 3D-aware VQA, which requires the understanding of the 3D structure of visual scenes and includes knowledge of 3D object pose, parts, and occlusions.\n2. Introducing the Super-CLEVR-3D dataset, which contains questions about object parts, their 3D poses, and occlusions.\n3. The proposed model, PO3D-VQA, is a 3D-aware VQA model that has two key techniques: probabilistic neural symbolic program execution, and deep neural networks paired with 3D generative object representations for robust visual recognition.", "questions": "1. What type of parts are included in the part questions?\n2. Is there a threshold for an object to be considered occluded?", "limitations": "1. As described in the paper, the work is currently limited by synthetic scenes.\n2. As described in the paper, the method is sensitive to pose prediction errors.", "rating": 9, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Proposed an interesting 3D-aware VQA task.\n2. Introduced a valuable Super-CLEVR-3D dataset.\n3. Developed a novel  PO3D-VQA method, that significantly outperforms prior methods.\n4. Detailed mathematical explanation of the components of the methods.\n5. Comprehensive analysis and discussions.", "weaknesses": "1. Lack of some statistics about the constructed Super-CLEVR-3D dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful and positive feedback on our paper. Please find our responses below:\n\n**Q0 & A0: Please refer to the general response for generalization to real images.**\n\n**Q1: Lack of some statistics about the constructed Super-CLEVR-3D dataset.**\n\nA1: We include the dataset statistics, as well as the list of question templates and the object-part list in the supplementary materials. We will revise and include important statistics in the main paper, especially the distribution of the questions, objects and attributes (classes, shape, material, color, size), parts, and occlusion ratios of objects and parts. Here we list the distribution of all attributes of objects in these tables.\n\n|           | car     | bus     | motorbike | aeroplane | bicycle  |\n|-----------|---------|---------|-----------|-----------|----------|\n| Percentage | 23.56%  | 19.78%  | 19.64%    | 16.96%    | 20.05%   |\n\n|           | red     | brown   | cyan    | green   | purple  | blue    | gray    | yellow  |\n|-----------|---------|---------|---------|---------|---------|---------|---------|---------|\n| Percentage| 12.28%  | 12.48%  | 12.57%  | 12.41%  | 12.57%  | 12.67%  | 12.49%  | 12.52%  |\n\n|           | small   | large   |\n|-----------|---------|---------|\n| Percentage| 55.34%  | 44.66%  |\n\n\n|           | metal   | rubber  |\n|-----------|---------|---------|\n| Percentage| 50.04%  | 49.96%  |\n\n**Q2: What type of parts are included in the part questions?**\n  \nA2: Thanks for bringing this up and we will include more details in our revised paper. The object parts are mined from the UDA-Part dataset [32], where each 3D object model is annotated with parts. We cleaned the UDA-Part annotations by removing the noisy annotations and the extremely small parts which can hardly be seen when rendered onto images. The final list of parts for each object type is included in the supplementary materials.\n\n**Q3: Is there a threshold for an object to be considered occluded?**  \n\nA3: Yes, there is a threshold. When generating the questions, we consider an object to be “not occluded” if the number of the occluded pixels is zero; we consider an object to be ”occluded” when the number of occluded pixels is larger than 15 (with 640 by 480 image resolution). We will include the threshold in the implementation details.\n"}, {"review_id": "3QOuUn5l0p", "reviewer": "Reviewer_NU9U", "summary": "This paper introduces the concept of 3D-aware Visual Question Answering (VQA) and presents a new dataset called \"Super-CLEVR-3D\". It also proposes a model called \"PO3D-VQA\" that combines probabilistic neural symbolic program execution with deep neural networks using 3D generative representations of objects. The experimental results show that the proposed model outperforms existing methods, but there is still a significant performance gap compared to 2D VQA benchmarks, highlighting the need for further research in 3D-aware VQA.", "questions": "Have you considered analyzing the failure cases of existing 2D VQA models on 3D-aware questions? This could provide insights into the limitations of 2D reasoning and highlight the unique strengths of the 3D-aware approach.\n\nHave you evaluated the performance of the model on larger-scale scenes or real-world datasets? If not, what challenges do you anticipate in scaling up the model to such scenarios?", "limitations": "Please refer to the weakness part. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ Originality: The authors propose a new dataset, Super-CLEVR-3D, that extends an existing 2D dataset with 3D-aware questions, introducing a novel benchmark for evaluating 3D-aware VQA models. The paper presents the PO3D-VQA model, which combines probabilistic neural symbolic program execution with deep neural networks using 3D generative representations of objects, providing a novel approach for addressing the 3D-aware VQA task.\n\n+ Quality: Thorough experimental results demonstrate the superiority of the proposed model, PO3D-VQA, over existing methods.\n\n+ Clarity: The paper provides clear motivation, defines the task of 3D-aware VQA, and describes the proposed dataset and model in a detailed and comprehensible manner.\n\n+ Significance: The paper addresses the importance of understanding the 3D structure of visual scenes in VQA, introduces a new dataset, and showcases improvements in accuracy, advancing the field of VQA.\n", "weaknesses": "- The paper addresses the VQA problem in 3D scenes but only takes images as input. Why not use point cloud or multi-view images which are more suitable for 3D scenario?\n\n- Where are the ground truth information of pose and occlusion come from? Are they included in the Super-CLVER dataset?\n\n- The model design of PO3D-VQA is somewhat weak. It looks like the combination of Neural Meshes and P-NSVQA.\n\n- I wonder about the performance of only using language and using oracle object representation ( ground truth class and pose label). In this way, we can show the reasoning ability of the model, or just accurate object detection is needed to solve this task.\n\n- How the proposed model compared with the scene graph-based method, since the method first parses the image as scene representations. What is the advantage of the neural symbolic method against deep graph networks?\n\n- There is no limited discussion on dataset limitations: While the Super-CLEVR-3D dataset is introduced as an extension of an existing dataset, the paper does not extensively discuss the limitations or potential biases of the dataset. Providing insights into the dataset construction process, potential challenges, and potential mitigations would strengthen the validity of the findings. And there is no limited scalability discussion: While the experimental results show improvements over existing methods, the paper does not thoroughly discuss the scalability of the proposed model. Understanding how the model's performance scales with larger and more complex scenes would be valuable for assessing its practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the time and effort you put into reviewing our paper. Here we address the questions you raised.\n\n**Q1:  Why not use point cloud or multi-view images, instead of single-view images, as input?**\n\nA1: We would like to highlight the focus of our paper: 3D-aware VQA from 2D images, i.e. answering questions about 3D information that can be inferred from 2D images. We chose to use 2D images instead of 3D data because (1) In real-world applications, it is much harder to obtain point cloud or multi-view data compared with single-view 2D images. Therefore, understanding 3D information from 2D images is an important ability of computer vision models. (2) While point cloud or multi-view images may provide rich information, the three types of questions that we focus on in this paper, i.e. pose, parts, and occlusions, can be inferred from single-view images. Therefore, we argue that it is important and valuable to understand 3D information from 2D images. While 3D input data can be an extension of future work, this is beyond the scope of this paper. We would add this to the revision of the paper. \n\n\n**Q2: Where does the ground truth information of pose and occlusion come from? Are they included in the Super-CLEVR dataset?**\n\nA2: (a) The pose information is provided in scene annotation files of the Super-CLEVR dataset, which annotates the position, rotation (pose), and attributes of each object in the scene. (b) For the occlusion annotations, while they are not directly provided in the Super-CLEVR dataset, they can be easily computed by rerendering the scene based on the scene annotation files. To get the occlusion annotation, we render each object in the scene separately to get single-object images for each object which is guaranteed to be unoccluded, we compare the unoccluded masks in these rendered single-object images with the object masks of the multi-object images provided in Super-CLEVR. If the single-object mask differs from the multi-object by a threshold (15 pixels), we consider the object/part to be occluded. We will add the description to the revised paper.\n\n**Q3: The model design of PO3D-VQA is somewhat weak. It looks like the combination of Neural Meshes and P-NSVQA.**\n\nA3: We would like to politely disagree and argue that the model is not a simple concatenation of neural meshes and P-NSVQA. As discussed in Sec 4.2, non-trivial technical improvements have been made. Existing mesh-based pose estimation models [36, 47] are for single-class images: i.e., all objects in the image are from the same class and the object class is known during inference. However, in VQA settings, the scene contains multiple objects from multiple classes, posing a greater challenge to correctly classify the object and accurately solve the 6D pose. Therefore, we adopt the analysis-by-synthetic pipeline used in Neural Meshes and significantly extend the model by introducing including 3D-NMS, greedy proposal generation, and post-filtering are proposed in this paper to address the multi-class and  multi-object problem. In our paper, we also discuss the advantage of our method over the naive method that uses object detectors like FasterRCNN to detect and classify the object category. \n\nMoreover, besides the technical contributions, our work is the first to integrate 3D geometric neural meshes with VQA, which is shown to be effective and achieves strong results compared with existing methods. We would like to highlight this contribution and suggest the importance of 3D geometry in VQA.\n\n**Q4: Performance using oracle object representation, in order to show the reasoning ability of the model.**\n\nA4: Thank you for the suggestion. We replace the perception with the oracle object representation provided in the annotations and run the reasoning module (P-NSVQA) based on it. The model achieves 99% using the oracle representations, suggesting the strong ability of the symbolic reasoning module.\n\n**Q5: What is the advantage of the neural symbolic method against deep graph networks?**\n\nA5: We agree that graph models are widely adopted in VQA, which has been shown in previous works (Hu et al., 2019). However, compared with these models, neural symbolic methods like P-NSVQA or NSVQA have been shown to be more effective and achieve state-of-the-art performance on both the CLEVR dataset and the SuperCLEVR dataset. In addition, the modular symbolic methods have better interpretability (step-by-step reasoning), data efficiency, and robustness in the out-of-distribution settings, as shown in previous works [38, 31, 51].\n\n**Q6: Have you evaluated the performance of the model on larger-scale scenes or real-world datasets? If not, what challenges do you anticipate in scaling up the model to such scenarios?**\n\nA6: Please refer the the **general response** about results on the real images, as well as challenges for doing experiments on large-scale datasets.\n\n**Q7: Discussion of failure cases of baseline model.**\n\nA7: We show such examples in Figure 5 of the main paper, also here we provide two more examples in Fig-4 of the rebuttal PDF. The 2D models are hard to locate the small parts and determine if an object is occluded when the occlusion area is small. For the PNSVQA-Project method, it will make similar mistakes as the pose estimator is not precise enough.  \n\n**Q8: Discussion of the dataset limitations.**\n\nA8: Regarding potential biases in the dataset - As the original Super-CLEVR dataset is designed to study VQA domain generalization, the distribution of the objects/attributes, question redundancy, and the compositionality of concepts are balanced. We built our Super-CLEVR-3D on the $default$ version of Super-CLEVR, where the concepts are well-balanced. We will include more statistics about the dataset distribution in the supplementary materials.\n\n[1] Hu, Ronghang, et al. \"Language-conditioned graph networks for relational reasoning.\" Proceedings of the IEEE/CVF international conference on computer vision. 2019."}]

Author rebuttal:
# General Response

We thank all the reviewers for the feedback. We are glad that all four reviews are positive and acknowledge both of our contributions: (1) The Super-CLEVR-3D dataset, i.e. a new VQA dataset that introduces an important new task (HP3a, NU9U) that requires reasoning over object parts, 3D poses, and occlusions, and (2) the originality of our proposed PO3D-VQA model (NU9U, SUrx, sJrg) for solving these tasks. We are happy that the reviewers highlighted the significance of our experimental improvements over the baselines (HP3a, SUrx, NU9U), and the clear motivation and good writing quality of the paper.

Here we provide a general response about how our proposed PO3D-VQA model can be extended beyond Super-CLEVR-3D to datasets that contain real-world imagery.

First, **we agree that our work should be extended to real images or other 3D VQA datasets as an important research topic, and we have started doing this in the current ongoing work**. However, we note that extending such a 3D-aware VQA to datasets is non-trivial. For the real-world data, existing real-world VQA datasets (like GQA and FE-3DGQA suggested by reviewer-sJrg) lack 3D annotations to objects and 3D-related question-answers (questions in existing VQA datasets refer mostly to the 2D space), which makes it hard to train and test our model on them. To enable 3D-aware VQA on these datasets, we need to generate additional 3D annotations on them. Moreover, real VQA datasets contain a large variety of object classes with many being highly articulated (e.g. humans and animals). We are working on annotating these datasets with 3D and scaling up the number of object classes in our PO3D-VQA model, which could allow us to train and test our model, but this requires a significant amount of effort and time.

However, to show that our PO3D-VQA model can, in principle, work on realistic images, we tested it on several more realistic image samples that were generated with objects from real datasets that had 3D annotations given (see Fig-1 of rebuttal file). This small preliminary experiment shows that our model can successfully answer 3D-related questions on realistic images. 

To give more details on this experiment: The example images are manually created using the vehicle objects (e.g. car, bus, bicycle) from ImageNet. In this experiment, the pose estimator is trained on the PASCAL3D+ dataset, which can successfully predict the poses of objects in the image, as shown in (b). The attribute (color) prediction module is trained on Super-CLEVR-3D and the object shapes are predicted by a ResNet trained on ImageNet. Our model can correctly predict answers to questions about the object pose, parts, and occlusions, e.g. “Which object is occluded by the mountain bike”. 

From the provided examples, we hope to demonstrate that the difficulty of annotation makes it hard to use vision algorithms that are 3D-aware, not just our model. Additionally, we also conduct experiments on new types of questions with more complex scenes as asked by some reviewers. We hope these can help to address the reviewers’ questions about our synthetic dataset and our proposed 3D-aware VQA model.


Meta-review: Four experts reviewed this paper with all accepted recommendations. The area chairs agree that this work makes a very important contribution by introducing a new 3D-aware VQA dataset. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. It would be also nice if the authors could include some discussions on how to make progress on this challenge by leveraging the recent success of 3D foundational models (e.g. [1, 2, 3] ) in the final version.

[1] 3D Concept Learning and Reasoning from Multi-View Images. CVPR 2023

[2] Scalable 3D Captioning with Pretrained Models.

[3] 3D-LLM: Injecting the 3D World into Large Language Models
Decision: Accept (poster)