SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Summary. Fig. 2 provides an overview of our scene parser and Fig. 3 visualize the intermediate results. With the idea of render-and-compare (shown in the green box of Fig. 2), the model first computes an activation map for each possible object category (Fig. 3II). Next, to infer the category for each object, the category-wise competition 3D-NMS is performed (Fig. 3b) and a post-filtering step is taken to remove mis-detected objects (Fig. 3c). Fig. 3d shows the 6D pose estimation results. To predict parts, we project the 3D object mesh back onto the image to locate parts based on projected objects (Fig. 3e). In this way, the input image can be parsed into a 3D-aware representation, which is ready for the question reasoning with program execution.
4.3 Program execution
After the 3D-aware scene representations are predicted for the given image, the question is parsed into a reasoning program, which is then executed on the scene representation to predict the answer. The question parsing follows previous work [54], where a LSTM sequence-to-sequence model is trained to parse the question into its corresponding program. Like P-NSVQA [32], each operation in the program is executed on the scene representation in a probabilistic way. In the following, we describe the execution of the new operations we introduced.
The part-related operators are implemented by querying the object-part hierarchy matrix  $H$ , so that the object containing a given part (part_to_object) and the parts belonging to the given object
(object_to_part) can be determined. The pose-related operators are based on the estimated 3D pose in the object attributes  $A^o$ . For the filter and query operations regarding pose, the 3D poses are quantified into four directions (left, right, front, back). For the pair-wise pose relationships, the azimuth angle between two objects is used to determine the same/opposite/vertical directions. The occlusion-related operations are implemented by querying the occlusion matrix  $S$ . Based on the occlusion scores  $S_{ij}$  representing whether entity  $i$  being occluded by entity  $j$ , we can compute the score of one entity being occluded  $\sum_{j} S_{ij}$  (filter_occludee), find the entities that occlude a given entity (relate_occluded), or find the entities that are occluded by a given entity (relate_occluded).
5 Experiments
5.1 Evaluated methods
We compare our model with three representative VQA models: FiLM [44], mDETR [25], and PNSVQA [32]. Additionally, we introduce a variant of PNSVQA, PNSVQA+Projection, to analyze the benefit of our generative 6D pose estimation approach.
FiLM [44] Feature-wise Linear Modulation is a representative two-stream feature fusion method. The FiLM model merges the question features extracted with GRU [12] and image features extracted with CNN and predicts answers based on the merged features.
mDETR [25] mDETR is a pretrained text-guided object detector based on transformers. The model is pretrained with 1.3M image and text pairs and shows strong performance when finetuned on downstream tasks like referring expression understanding or VQA.
PNSVQA [32] PNSVQA is a SoTA neural symbolic VQA model. It parses the scene using MaskR-CNN [18] and an attribute extraction network, then executes the reasoning program on the parsed visual scenes with taking into account the uncertainty of the scene parser. To extend PNSVQA to the 3D questions in Super-CLEVR-3D, we add a regression head in the attribute extraction network to predict the 3D posefor each object; parts are detected in a similar way as objects by predicting 2D bounding boxes; the part-object associations and occlusions are computed using intersection-overunion: a part belongs to an intersected object if the part label matches the object label, otherwise it is occluded by this object.
PNSVQA+Projection Similar with NSVQA, this model predicts the 6D poses, categories and attributes using MaskRCNN and the attribute extraction network. The difference is that the parts and occlusions are predicted by projecting the 3D object models onto the image using the predicted 6D pose and category (same with how we find parts and occlusions in our model). This model helps us ablate the influence of the two components in our model, i.e. 6D pose prediction by render-and compare, and part/occlusion detection with mesh projection.
5.2 Experiment setup
Dataset. Our Super-CLEVR-3D dataset shares the same visual scenes with Super-CLEVR dataset. We re-render the images with more annotations recorded (camera parameters, parts annotations, occlusion maps). The dataset splits follow the Super-CLEVR dataset, where we have 20k images for training, 5k for validation, and 5k for testing. For question generation, we create 9 templates for part questions, 17 templates for pose questions, 35 templates for occlusion questions (with and without parts). For each of the three types, 8 to 10 questions are generated for each image by randomly sampling the templates. We ensure that the questions are not ill-posed and cannot be answered by taking shortcuts, i.e. the questions contain no redundant reasoning steps, following the no-redundancy setting in [32]. More details including the list of question templates can be found in the Appendix.
Implementation details. We train the 6D pose estimator and CNN attribute classifier separately. We train the 6D pose estimator (including the contrastive feature backbone and the nerval mesh models for each of the 5 classes) for 15k iterations with batch size 15, which takes around 2 hours on NVIDIA RTX A5000 for each class. The attribute classifier, which is a ResNet50, is shared for objects and parts. It is trained for 100 epochs with batch size 64. During inference, it takes 22s for 6D pose estimation and 10s for object mesh projection for all the objects in one image. During inference of the 6D pose estimator, we assume the theta is 0. During 3D NMS filtering, we choose the radius  $r$  as 2, and we also filter the object proposals with a threshold of 15 on the score map.
5.3 Quantitative Results
We trained our model and baselines on Super-CLEVR-3D's training split, reporting answer accuracies on the test split in Tab. 1. Accuracies for each question type are detailed separately.
Table 1: Model accuracies on the Super-CLEVR-3D testing split, reported for each question type, i.e. questions about parts, 3D poses, occlusions between objects, occlusions between objects and parts.
Mean	Part	Pose	Occ.	Part+Occ.
FiLM [44]	50.53	38.24	67.82	51.41	44.66
mDETR [25]	55.72	41.52	71.76	64.99	50.47
PNSVQA [32]	64.39	50.61	87.78	65.80	53.35
PNSVQA+Projection	68.15	56.30	86.70	70.70	58.90
PO3D-VQA (Ours)	75.64	71.85	86.40	76.90	67.40
Comparison with baselines. First, among all the baseline methods, the neural symbolic method PNSVQA performs the best (64.4% accuracy), outperforming the end-to-end methods mDETR and FiLM by a large margin ( $>8\%$ ). This shows the advantage of the step-wise modular reasoning procedure, which agrees with the findings in prior works that the modular methods excel on the simulated benchmarks that require long-trace reasoning. Second, our model achieves 75.6% average accuracy, which significantly outperforms all the evaluated models. Especially, comparing our PO3D-VQA with its 2D counterpart NSVQA, we see that the injection of 3D knowledge brings a large performance boost of 11%, suggesting the importance of the 3D understanding.
Comparison with PNSVQA variants. By analyzing the results of PNSVQA variants (PNSVQA, PNSVQA+Projection, and our PO3D-VQA), we show (a) the benefit of estimating object 3D poses using our analysis-by-synthesis method over regression and (b) the benefit of object-part structure knowledge. First, by detecting part using 3D model projection, PNSVQA+Projection improves the PNSVQA results by  $4\%$ , which indicates that locating parts based on objects using the object-part structure knowledge is beneficial. Second, by estimating object 6D poses with our generative render-and-compare method, our PO3D-VQA outperforms PNSVQA+Projection by  $7\%$  (from  $68.2\%$  to  $75.6\%$ ), showing the advantage of our render-and-compare model. Moreover, looking at the per-type results, we find that the improvement of our PO3D-VQA is most significant on the part-related questions ( $21\%$  improvement over PNSVQA) and part-with-occlusion questions ( $14\%$ ), while the accuracy on pose-related questions does not improve. The reason is that part and occlusion predictions require precise pose predictions for accurate mesh projection, while the pose questions only require a rough pose to determine the facing direction.
5.4 Analysis and discussions
To further analyze the advantage of PO3D-VQA over other PNSVQA variants, we compare the models on questions of different difficulty levels. It is shown that the benefit our model is the most significant on hard questions. In Fig. 4, we plot the relative accuracy drop  ${}^{3}$  of each model on questions with different occlusion ratios and questions with different part sizes.
(a) Pose wrt. Occlusion Ratio
(b) Part wrt. Part Size
Figure 4: Analysis on questions of different difficulty levels. The plots show the relative accuracy drop of models, on pose questions w.r.t. different occlusion ratios (a), on part questions w.r.t. different part sizes (b), and on part+occlusion questions w.r.t. different part sizes (c).
(c) Part + Occlusion wrt. Part Size
Questions with different occlusion ratios. We sort pose-related questions into different sub-groups based on their occlusion ratios and evaluate the models on each of the sub-groups. The occlusion ratio  $r$  of a question is the minimum of occlusion ratios for all the objects in its reasoning trace. We choose  $r$  from  $0\%$  to  $30\%$ , in increment of  $5\%$ . The results are shown in Fig. 4 (a). Our PO3D-VQA is much more robust to occlusions compared to the other two methods: while the performances of all
${}^{3}$  Relative accuracy drop means the ratio of absolute accuracy drop and the original accuracy. For example,if a model's accuracy drops from 50% to 45%, its relative accuracy drop is 10%.
the three models decrease as the occlusion ratio increases, the relative drop of ours is much smaller than others. The results show that our render-and-compare scene parser is more robust to heavy occlusions compared with the discriminative methods.
Questions with different part sizes. Questions about small parts are harder than the ones about larger parts. We sort the questions into different part size intervals  $(s, t)$ , where the largest part that the question refers to has an area (number of pixels occupied) larger than  $s$  and smaller than  $t$ . We compare the models on the part questions and the part+occlusion questions with different part sizes in Fig. 4 (b) and (c). In (b), the accuracy drop of PO3D-VQA is smaller than PNSVQA+Projection and PNSVQA when parts get smaller. In (c), PNSVQA+Projection is slightly better than our model and they are both better than the original PNSVQA.
In summary, by sorting questions into different difficulty levels based on occlusion ratios and part sizes, we show the advantage of our PO3D-VQA on harder questions, indicating that our model is robust to occlusions and small part sizes.
Qualitative results. Fig. 9 shows examples of predictions for our model and PNSVQA variants. In (a), the question asks about occlusion, but with a slight error in the pose prediction, PNSVQA+Projection misses the occluded bus and predicts the wrong answer, while our model is correct with accurate pose. In (b), the question refers to the heavily occluded minivan that is difficult to detect, but our model gets the correct prediction thanks to its robustness to occlusions.
(a)
Q: What is the material of the gray object that is occluded? A: rubber
(b)
Q: Which direction is the minivan facing? A: left
Figure 5: Examples of models' predictions. Our model (a) predicts the object pose accurately and (b) is robust to heavy occlusions. Red boxes are for visualization only.
PNSVQA+Projection
Limitations and failure cases. Due to the difficulties of collecting real images with compositional scenes and 3D annotations, our work is currently limited by its synthetic nature. For PO3D-VQA, it sometimes fails to detect multiple objects if they are from the same category and heavily overlap (see Appendix D for more visualizations). 3D NMS can effectively improve the dense scene parsing results when objects are from different categories, but conceptually it is limited when objects are from the same category. However, 6D pose estimation in dense scenes is a challenging problem, whereas many current works on 6D pose estimation are still focusing on simple scenes with single objects [38, 50, 57].
6 Further Discussion
In this section, we discuss two meaningful extensions of our work: the incorporation of z-direction questions and the application of our model to real-world images.
Z-direction questions. While the proposed Super-CLEVR-3D dataset has been designed with 3D-aware questions, all objects within it are placed on the same surface. Introducing variability in the z direction can further enrich our dataset with more comprehensive 3D spatial relationships.
We consider the scenario where aeroplane category, is in different elevations, introducing the  $z$  dimension into the spatial relationships (see Fig. 6). This allowed us to formulate questions that probe the model's understanding of height relationships and depth perception. We create a subset containing 100 images and 379 questions and test our PO3D-VQA model directly on it without retraining the 6D
parser. On this dataset, our PO3D model achieves  $90.33\%$  accuracy on height relationship questions and  $78.89\%$  on depth-related questions, suggesting that our model can successfully handle questions about height. As the baseline models only use the bounding box to determine the spatial relationship between objects, they are not able to determine the height relationships.
Height question: There is a blue object that is below the biplane; what shape is it? Answer: Tandem
Depth question: Is the biplane closer than the red motorbike? Answer: Yes
Height question: How many objects are above the shiny bicycle? Answer: 2
Depth question: Does the truck have a greater distance than the shiny bus? Answer: Yes
Figure 6: Example images and questions of objects with different elevations.
Extension to real-world images While our PO3D-VQA model has demonstrated impressive performance on the synthetic Super-CLEVR-3D dataset, an essential research direction is extending it to real images or other 3D VQA datasets (such as GQA and FE-3DGQA). However, it's not trivial to truly evaluate it on these real-world problems, and a primary challenge is the lack of 3D annotations and the highly articulated categories (like the human body) in these datasets.
However, we show that our PO3D-VQA model can, in principle, work on realistic images. We generate several realistic image samples manually using the vehicle objects (e.g. car, bus, bicycle) from ImageNet with 3D annotation (see Fig. 7) and real-image background. In this experiment, the pose estimator is trained on the PASCAL3D+ dataset, and is used to predict the poses of objects from the image before pasting, as shown in (b). The attribute (color) prediction module is trained on Super-CLEVR-3D and the object shapes are predicted by a ResNet trained on ImageNet. Our model can correctly predict answers to questions about the object pose, parts, and occlusions, e.g. "Which object is occluded by the mountain bike".
(a1)
(b1)
[Post] Q: Which direction does the mountain bike face to? Our: Left Q: Which direction does the race car face to? Our: Right [Part] Q: What is the color of the fin that belongs to the aeroplane? Our: Red Q:What's the shape of the object that has a wing? Our: Aeroplane [Occ.] Q: Which object is occluded by the mountain bike? Our: Trolleybus Q: What is the color of the occluded object? Our: Red (c1) (c2)
Figure 7: Examples of results on realistic images. Given a realistic image (a1, a2), our model can successfully estimate the 6D poses of objects (b1, b2) and answer the 3D-aware questions (c1, c2).
(a2)
(b2)
7 Conclusion
In this work, we study the task of 3D-aware VQA. We propose the Super-CLEVR-3D dataset containing questions explicitly querying 3D understanding including object parts, 3D poses, and occlusions. To address the task, a 3D-aware neural symbolic model PO3D-VQA is proposed, which enhances the probabilistic symbolic model with a robust 3D scene parser based on analysis-by-synthesis. With the merits of accurate 3D scene parsing and symbolic execution, our model outperforms existing methods by a large margin. Further analysis shows that the improvements are even larger on harder questions. With the dataset, the model, and the experiments, we highlight the benefit of symbolic execution and the importance of 3D understanding for 3D-aware VQA.
We thank the anonymous reviewers for their valuable comments. We thank Qing Liu, Chenxi Liu, Elias Stengel-Eskin, Benjamin Van Durme for the helpful discussions on early version of the project. This work is supported by Office of Naval Research with grants N00014-23-1-2641, N00014-21-1-2812. A. Kortylewski acknowledges support via his Emmy Noether Research Group funded by the German Science Foundation (DFG) under Grant No.468670075.
[57] Yanjie Ze and Xiaolong Wang. Category-level 6d object pose estimation in the wild: A semi-supervised learning approach and a new dataset. Advances in Neural Information Processing Systems, 35:27469-27483, 2022.
[58] Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5317-5327, 2019.
[59] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision-language models. CVPR 2021, 2021.
[60] Xingyi Zhou, Arjun Karpur, Linjie Luo, and Qixing Huang. Starmap for category-agnostic keypoint and viewpoint estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 318–334, 2018.
[61] Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4995–5004, 2016.
A Dataset Details
A.1 Part list
In Super-CLEVR-3D, the parts of each objects are listed in Tab. 2
A.2 Question templates
Part Questions we collect 9 part-based templates when generating the part-based questions, as shown in Tab. 4. In the table, <attribute> means one attribute from shape, material, color or size to be queried, <object> (or <object 1>, <object 2>) means one object to be filtered with a combination of shape, material, color, and size. Different from the pose and occlusion question, we don't query the size of the object.
3D Pose questions We design 17 3D pose-based templates in question generation (as shown in table 5). The 17 templates consist of: 1 template of the query of the pose; 4 questions of the query of shape, material, color, size, where the pose is in the filtering conditions; 12 templates about the query of shape, material, color, size, where the relationship of the pose is the filtering condition.
Occlusion Questions There are 35 templates in the occlusion question generation as shown in table 6, which consists of occlusion of objects and occlusion of parts.
The occlusion of objects consists of occlusion status and occlusion relationship. For the occlusion status of the object, there are 4 templates to query the shape, color, material, and size respectively. There are 2 occlusion relationships of objects (occluded and occluding), and each of them has 4 templates.
Similarly, we then create a template about occlusion status and occlusion relationship for the parts. The only difference between object and part is that the parts only have 3 attributes to be queried: shape (name), material and color.
A.3 Statistics
As a result, a total of 314,988 part questions, 314,986 pose questions, and 228,397 occlusion questions and 314,988 occlusion questions with parts.
In Fig. 8, we show the distributions of all attributes of objects including categories, colors, sizes, and materials
Figure 8: Distributions for all the attributes of objects including categories, colors, sizes, and materials
B Implementation details for the baselines
The FiLM and mDETR are trained with default settings as in the official implementation. FiLM is trained for 100k iterations with batch size 256. mDETR is trained for 30 epochs with batch size 64 using 2 GPUs for both the grounding stage and the answer classification stage.
For P-NSVQA, we first train a MaskRCNN for 30k iterations with batch size 16 to detect the objects and parts, then train the attribute extraction model (using Res50 backbone) for 100 epochs with batch size 64. Different fully connected(FC) layers are used for a different type of question: the
part questions and occlusion questions have 4 FC layers for the shape, material, color, and size classification (as the parts also have size annotations in the dataset when generating scene files, but they are meaningless in the question answering). The pose question includes pose prediction of an object, so we add a new FC layer with 1 output dimension to predict the rotations, followed by an MSE loss during training. For different types of questions (part, pose and occlusion), the MaskRCNN and attribute extraction model are trained separately.
In the PNSVQA+Projection baseline, we first train a MaskRCNN to detect all of the objects and predict their 3D pose (azimuth, elevation and theta) without category labels in the scene. This MaskRCNN is trained with batch size 8 and iteration 15000. We use an SGD optimizer with a learning rate of 0.02, momentum of 0.9 and weight decay 0.0001. Then, we use the same setting as our PO3D-VQA to train a CNN to classify the attributes of objects and parts.
C Detailed results of Analysis
As an extension for section 5.4 in main paper, here we include the numerical value of accuracy and drop for the pose, part, occlusion + part question with reference to occlusion ratio or part size. The result is shown in Tab. 7, Tab. 9 and Tab. 8.
D Failure cases
Examples of failure cases of our PO3D-VQA, as described in Section 5.4 in main paper. In (a) and (b), PO3D-VQA misses the bicycle behind when two bicycles have a heavy overlap, the same for the two motorbikes in (c) and (d).
(b)
Figure 9: Failure cases of our PO3D-VQA. (a) and (c) is the input images with the objects missed by the model. (b) and (c) is the re-projection results from the model.
Table 4: Templates of parts questions
Templates	Count
What is the <attribute> of the <part> of the <object>?	3
What is the <attribute> of the <object> that has a <part>?	3
What is the <attribute> of the <part 1> that belongs to the same object as the <part 2>?	3
Table 5: Templates of pose questions
Templates	Count
Which direction the <object> is facing?	1
What is the <attribute> of the <object> which face to the <0>?	4
What is the <attribute> of the <object 1> that faces the same direction as a <object 2>	4
What is the <attribute> of the <object 1> that faces the opposite direction as a <object 2>	4
What is the <attribute> of the <object 1> that faces the vertical direction as a <object 2>	4