SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xingrui Wang $^{1}$  Wufei Ma $^{1*}$  Zhuowan Li $^{1\dagger}$  Adam Kortylewski $^{2,3}$  Alan Yuille $^{1}$ $^{1}$  Johns Hopkins University  $^{2}$  Max Planck Institute for Informatics  $^{3}$  University of Freiburg  
{xwang378, wma27, zli110, ayuille1}@jhu.edu akortyle@mpi-inf.mpg.de
Despite rapid progress in Visual question answering (VQA), existing datasets and models mainly focus on testing reasoning in 2D. However, it is important that VQA models also understand the 3D structure of visual scenes, for example to support tasks like navigation or manipulation. This includes an understanding of the 3D object pose, their parts and occlusions. In this work, we introduce the task of 3D-aware VQA, which focuses on challenging questions that require a compositional reasoning over the 3D structure of visual scenes. We address 3D-aware VQA from both the dataset and the model perspective. First, we introduce Super-CLEVR-3D, a compositional reasoning dataset that contains questions about object parts, their 3D poses, and occlusions. Second, we propose PO3D-VQA, a 3D-aware VQA model that marries two powerful ideas: probabilistic neural symbolic program execution for reasoning and deep neural networks with 3D generative representations of objects for robust visual recognition. Our experimental results show our model PO3D-VQA outperforms existing methods significantly, but we still observe a significant performance gap compared to 2D VQA benchmarks, indicating that 3D-aware VQA remains an important open research area. The code is available at https://github.com/XingruiWang/3D-Aware-VQA.
Visual question answering (VQA) is a challenging task that requires an in-depth understanding of vision and language, as well as multi-modal reasoning. Various benchmarks and models have been proposed to tackle this challenging task, but they mainly focus on 2D questions about objects, attributes, or 2D spatial relationships. However, it is important that VQA models understand the 3D structure of scenes, in order to support tasks like autonomous navigation and manipulation.
An inherent property of human vision is that we can naturally answer questions that require a comprehensive understanding of the 3D structure in images. For example, humans can answer the questions shown in Fig. 1, which ask about the object parts, their 3D poses, and occlusions. However, current VQA models, which often rely on 2D bounding boxes to encode a visual scene [2, 59, 25] struggle to answer such questions reliably (as can be seen from our experiments). We hypothesize this is caused by the lack of understanding of the 3D structure images.
In this work, we introduce the task of 3D-aware VQA, where answering the questions requires compositional reasoning over the 3D structure of the visual scenes. More specifically, we focus on challenging questions that require multi-step reasoning about the object-part hierarchy, the 3D poses of the objects, and the occlusion relationships between objects or parts.
*Wufei contributed to develop the 3D algorithm for multi-objects pose estimation.
†Zhuowan contributed to dataset construction, manuscript writing and conceptualizing the framework.
Figure 1: Examples from Super-CLEVR-3D. We introduce the task of 3D-aware VQA, which requires 3D understanding of the image, including the parts, 3D poses, and occlusions.
Part Q: What is the name of the brown part of the large rubber thing? Wheel Q: What is the material of the trunk that belongs to the same object as the purple part? Metallic
3D Pose
Q: Which direction the double bus is facing? Left
Q: What is the color of the small object which faces to the right? Truck
Q: Is the bumper of the purple SUV occluded? No
Q: What is the size of the aeroplane whose wing is occluded? Small
filter[aeroplane]
obj_to_part
filter_part[wing]
filter_occludee
part_to_obj
query_size
What is the size of the aeroplane whose wing is occluded?
We address the challenging 3D-aware VQA task from both the dataset and the model perspective. From the dataset perspective, we introduce Super-CLEVR-3D, which extends the Super-CLEVR dataset [32] with 3D-aware questions. Given the visual scenes from Super-CLEVR that contain randomly placed vehicles of various categories, we define a set of 3D-aware reasoning operations and automatically generate 3D questions based on these operations. Fig. 1 shows examples of the images, questions and the underlying 3D operations for the questions. From the model perspective, we introduce PO3D-VQA, a VQA model that marries two powerful ideas: probabilistic neural symbolic program execution for reasoning and a deep neural network with 3D generative representations of objects for robust visual scene parsing. Our model first recovers a 3D scene representation from the image and a program from the question, and subsequently executes the program on the 3D scene representation to obtain an answer using a probabilistic reasoning process that takes into account the confidence of predictions from the neural network. We refer to our system as PO3D-VQA, which stands for Parts, Poses, and Occlusions in 3D Visual Question Answering.
On Super-CLEVR-3D, we experiment with existing representative models, their variants, and our model PO3D-VQA. The results show that our model outperforms existing methods significantly, leading to an improvement in accuracy of more than  $11\%$ , which shows the advantage of the generative 3D scene parser and the probabilistic neural symbolic reasoning process. Moreover, further analysis on questions with different difficulty levels reveals that the improvements of our model are even greater on harder questions with heavy occlusions and small part sizes. Our results indicate that a reliable 3D understanding, together with the modular reasoning procedure, produces a desirable 3D-aware VQA model.
In summary, our contributions are as follows. (1) We introduce the challenging task of 3D-aware VQA and propose the Super-CLEVR-3D dataset, where 3D visual understanding about parts, 3D poses, and occlusions are required. (2) We propose a 3D-aware neural modular model PO3D-VQA that conducts probabilistic reasoning in a step-wise modular procedure based on robust 3D scene parsing. (3) With experiments, we show that 3D-aware knowledge and modular reasoning are crucial for 3D-aware VQA, and suggest future VQA methods take 3D understanding into account.
2 Related Work
Visual Question Answering (VQA). Rapid progress has been made in VQA [4] in both the datasets and the models. To solve the challenging VQA datasets [15, 61, 17, 45] with real images, multiple models are developed including two-stream feature fusion [2, 14, 28, 55, 23, 44, 30] or transformer-based pretraining [48, 36, 31, 59, 25]. However, the real datasets are shown to suffer from spurious correlations and biases [42, 16, 41, 1, 15, 26, 27]. Alternatively, synthetic datasets like CLEVR [24] and Super-CLEVR [32], are developed to study the compositional reasoning ability of VQA systems, which are also extended to study other vision-and-language tasks [34, 29, 53, 58, 6, 47, 20]. The synthetic datasets promote the development of neural modular methods [3, 54, 40, 22], where the reasoning is done in a modular step-by-step manner. It is shown that the modular methods have nice properties including interpretability, data efficiency [54, 40], better robustness [32] and strong performance on synthetic images [54]. However, most existing methods rely on region features [2, 59] extracted using 2D object detectors [46] for image encoding, which is not 3D-aware. We follow the works on the synthetic dataset and enhance the modular methods with 3D understanding.
VQA in 3D. Multiple existing works study VQA under the 3D setting, such as SimVQA [8], SQA3D [39], 3DMV-VQA [19], CLEVR-3D [51], ScanQA [52], 3DQA [52], and EmbodiedQA [13], which focus on question answering on the 3D visual scenes like real 3D scans [39, 51, 5, 52], simulated 3D environments [9, 13], or multi-view images [19]. PTR [20] is a synthetic VQA dataset that requires part-based reasoning about physics, analogy and geometry. Our setting differs from these works because we focus on 3D in the questions instead of 3D in the visual scenes, since our 3D-aware questions explicitly query the 3D information that can be inferred from the 2D input images.
3D scene understanding. One popular approach for scene understanding is to use the CLIP features pretrained on large-scale text-image pairs and segment the 2D scene into semantic regions [10, 43]. However, these methods lack a 3D understanding of the scene and cannot be used to answer 3D-related questions. Another approach is to adopt category-level 6D pose estimation methods that can locate objects in the image and estimate their 3D formulations. Previous approaches include classification-based methods that extend a Faster R-CNN model for 6D pose estimation [60, 38] and compositional models that predicts 6D poses with analysis-by-synthesis [38]. We also notice the huge progress of 3D vision language foundation models, which excel in multiple 3D vision-language understanding tasks [19, 37, 21]. Still, we focus on the reasoning with compositional reasoning that brings more interpretability and robustness [32].
3 Super-CLEVR-3D Dataset
To study 3D-aware VQA, we propose the Super-CLEVR-3D dataset, which contains questions explicitly asking about the 3D object configurations of the image. The images are rendered using scenes from the Super-CLEVR dataset [32], which is a VQA dataset containing synthetic scenes of randomly placed vehicles from 5 categories (car, plane, bicycle, motorbike, bus) with various of sub-types (e.g. different types of cars) and attributes (color, material, size). The questions are generated by instantiating the question templates based on the image scenes, using a pipeline similar to Super-CLEVR. In Super-CLEVR-3D, three types of 3D-aware questions are introduced: part questions, 3D pose questions, and occlusion questions. In the following, we will describe these three types of questions, and show the new operations we introduced for our 3D-aware questions about object parts, 3D poses, and occlusions. Examples of the dataset are shown in Fig. 1.
Part questions. While in the original Super-CLEVR dataset refers to objects using their holistic names or attributes, objects are complex and have hierarchical parts, as studied in recent works [33, 11, 20]. Therefore, we introduce part-based questions, which use parts to identify objects (e.g. "which vehicle has red door") or query about object parts (e.g. "what color is the door of the car"). To enable the generation of part-based questions, we introduce two new operations into the reasoning programs: part_to_object(\cdot), which find the objects containing the given part, and object_to_part(\cdot), which select all the parts of the given object. We also modify some existing operations (i.e. filter, query and unique), enabling them to operate on both object-level and part-level. With those reasoning operations, we collect 9 part-based templates and instantiate them with the image scene graph to generate questions.
3D pose questions. Super-CLEVR-3D asks questions about the 3D poses of objects (e.g. "which direction is the car facing in"), or the pair-wise pose relationships between objects (e.g. "which object has vertical direction with the red car"). The pose for an individual object (e.g. "facing left") can be processed in a similar way as attributes like colors, so we extend the existing attribute-related operations like filter and query to have them include pose as well. For pair-wise pose relationship between objects, we add three operations, i.e. same Pose, opposite Pose and vertical Pose, to deal with the three types of pose relationships between objects. For example, opposite Pose(\cdot) returns the objects that are in the opposite pose direction with the given object. 17 templates are collected to generate 3D pose questions.
Occlusion questions. Occlusion questions ask about the occlusion between entities (i.e. objects or parts). Similar to 3D poses, occlusion can also be regarded as either an attribute for an entity (e.g. "which object is occluded"), or as a relationship between entities (e.g. "which object occludes the car door"). We extend the attribute-related operations, and introduce new operations to handle the pair-wise occlusion relationships: filter_occludee which filters the entities that are being occluded, relate_occluding which finds the entities that are occluded by the given entity, and relate_occluded which finds the entities that are occluding the given entity. Using these operations, 35 templates are collected to generate the occlusion questions.
4 Method
Figure 2: An overview of our model PO3D-VQA. The image is parsed into 3D-aware scene representations (blue box) using our proposed scene parser based on the idea of render-and-compare (green box). The question is parsed into a program composed of reasoning operations (orange box). Then the operations are executed on the 3D-aware scene representations to predict the answer.
In this section, we introduce PO3D-VQA, which is a parse-then-execute modular model for 3D-aware VQA. The overview of our system is shown in Fig. 2. We first parse the image into a scene graph representation that is aware of 3D information like object parts, 3D poses and occlusion relations, then we parse the question into a reasoning program and execute the program on the derived scene representations in a probabilistic manner. In Sec. 4.1, we define the scene representation required; in Sec. 4.2, we describe how we parse the image into the scene representation based on a multi-class 6D pose estimation model with non-trivial extensions; in Sec. 4.3, we describe how the question is executed on the derived scene representation to predict the answer.
4.1 3D-aware scene representation
Given an input image  $I$ , we parse it into a 3D-aware scene representation  $R$  that contains the objects  $(O)$  with attributes  $(A^o)$ , the parts  $(P)$  with attributes  $(A^p)$ , the hierarchical relationships between objects and parts  $(H)$ , and the occlusion relationships between them  $(S)$ . The attributes include the 3D poses and locations of objects or parts, as well as their colors, materials, and sizes. The scene representation  $R = \{O, P, A^o, A^p, H, S\}$  is comprehensive and therefore we can directly execute the symbolic reasoning module on this representation without taking into account the image any further.
In more detail, objects are represented as a matrix  $O \in \mathbb{R}^{n \times N_{obj}}$  containing the probability scores of each object being a certain instance, where  $n$  is the number of objects in the given image and  $N_{obj}$  is the number of all possible object categories in the dataset (i.e. vocabulary size of the objects). Similarly, parts are represented as  $P \in \mathbb{R}^{p \times N_{prt}}$ , where  $p$  is the number of parts in the image and  $N_{prt}$  is the vocabulary size of the object parts. The object-part hierarchy is represented by a binary matrix  $H \in \mathbb{R}^{n \times p}$ , where  $H_{ij} = 1$  if the object  $i$  contains the part  $j$  or  $H_{ij} = 0$  otherwise. The attributes  $A^o \in \mathbb{R}^{n \times N_{att}}$  and  $A^p \in \mathbb{R}^{p \times N_{att}}$  containing probability scores of each object or part having a certain attribute or the value of bounding box. Here  $N_{att}$  is the number of attributes including the 3D poses, location coordinates, colors, materials and sizes. Occlusion relationships are represented by  $S \in \mathbb{R}^{(n + p) \times n}$ , where each element  $S_{ij}$  represents the score of object (or part)  $i$  being occluded by object  $j$ .
4.2 Multi-class 6D Scene Parsing
While most existing VQA methods [2, 59] encode the image using pretrained object detectors like Faster-RCNN [46], we build our 6D-aware scene parser in a different way, based on the idea of analysis-by-synthesis through inverse rendering [49] which has the following advantages: first, the model prediction is more robust [49] as the render-and-compare process can naturally integrate a robust reconstruction loss to avoid distortion through occlusion; second, while the object parts
(II)
bus
Figure 3: Visualization of intermediate steps in our scene parser. Given an image (a), per-category feature activation maps (shown in II) are computed through render-and-compate. Then the category-wise competition (3D-NMS) is performed (results shown in b) and a post-filtering step is taken to remove mis-detected objects (c). Based on the pose estimation results (d), we project the 3D object mesh back onto the image to locate parts and occlusions(e).
are usually very challenging for Faster-RCNN to detect due to their small size, they can be much easier located using the 3D object shape, by first finding the object and estimating its 3D pose, and subsequently locating the parts using the 3D object shape (as shown in our experimental evaluation).
However, we observe two open challenges for applying existing 6D pose estimators that follow a render-and-compare approach [38, 49]: (a) these pose estimators assume that the object class is known, but in Super-CLEVR-3D the scene parser must learn to estimate the object class jointly with the pose; and (b) the scenes in Super-CLEVR-3D are very dense, containing multiple close-by objects that occlude each other. In order to address these two challenges, we introduce several improvements over [38] that enable it to be integrated into a 3D-aware VQA model.
In the following, we first describe neural meshes [49, 38], which were proposed in prior work for pose estimation of single objects following an analysis-by-synthesis approach. Subsequently, we extend this method to complex scenes with densely located and possibly occluded objects to obtain a coherent scene representation, including object parts and attributes.
Preliminaries. Our work builds on and significantly extends Neural Meshes [38] that were introduced for 6D pose estimation through inverse rendering. The task is to jointly estimate the 6D pose (2D location, distance to the camera and 3D pose) of objects in an image. An object category is represented with a category-level mesh [49]  $M_y = \{v_n \in \mathbb{R}^3\}_{n=1}^N$  and a neural texture  $T_y \in \mathbb{R}^{N \times c}$  on the surface of the mesh  $M_y$ , where  $c$  is the dimension of the feature and  $y$  is the object category. Given the object 3D pose in camera view  $\alpha$ , we can render the neural mesh model  $O_y = \{M_y, T_y\}$  into a feature map with soft rasterization [35]:  $F_y(\alpha) = \Re(O_y, \alpha)$ . Following prior work in pose estimation [49] we formulate the render-and-compare process as an optimization of the likelihood model:
$$
p \left(F \mid O _ {y}, \alpha_ {y}, B\right) = \prod_ {i \in \mathcal {F} \mathcal {G}} p \left(f _ {i} \mid O _ {y}, \alpha_ {y}\right) \prod_ {i \in \mathcal {B} \mathcal {G}} p \left(f _ {i} ^ {\prime} \mid B\right) \tag {1}
$$
where  $\mathcal{F}\mathcal{G}$  and  $\mathcal{B}\mathcal{G}$  are the set of foreground and background locations on the 2D feature map and  $f_{i}$  is the feature vector of  $F$  at location  $i$ . Here the foreground and background likelihoods are modeled as Gaussian distributions.
To train the feature extractor  $\Phi$ , the neural texture  $\{T_y\}$  and the background model  $B$  jointly, we utilize the EM-type learning strategy as originally introduced for keypoint detection in CoKe[7]. Specifically, the feature extractor is trained using stochastic gradient descent while the parameters of the generative model  $\{T_y\}$  and  $B$  are trained using momentum update after every gradient step in the feature extractor, which was found to stabilize training convergence.
At inference time, the object poses  $\alpha$  can be inferred by minimizing the negative log-likelihood w.r.t. the 3D pose  $\alpha$  using gradient descent [38].
Multi-object competition with 3D-NMS. We extend Neural Meshes to predict the 6D object pose and class label in complex multi-object scenes. In particular, we introduce 3D-Non-Maximum-Suppression (3D-NMS) into the maximum likelihood inference process. This introduces a competition between Neural Meshes of different categories in explaining the feature map. In contrast to classical
2D-NMS, our 3D-NMS also takes into account the distance of each object to the camera and hence naturally enables reasoning about occlusions of objects in the scene.
We denote the 6D pose as  $\gamma = \{x, l\}$ , where  $x = \{\alpha, \beta\}$  represents the 3D object pose  $\alpha$  and object distance to the camera  $\beta$ , and  $l$  is the 2D object location in the feature map. We first detect the 6D poses of each object category independently and apply 2D-NMS such that for each 2D location  $l'$  in a neighborhood defined by radius  $r$ , the predicted 6D pose  $\{x, l\}$  yields the largest activation:
$$
\max  _ {x} p (F \mid x, l) \text {s . t .} p (F \mid x, l) > p (F \mid x, l ^ {\prime}), \forall l ^ {\prime} \in \left\{l ^ {\prime} \mid 0 <   | l ^ {\prime} - l | <   r \right\} \tag {2}
$$
We enable multi-category 6D pose estimation by extending this formulation to a 3D non-maximum suppression (3D-NMS). Using  $\mathcal{V}$  to represent the set of all object categories, we model the category label  $y$  from a generative perspective:
$$
\max  _ {x} p (F \mid x, l, y) \text {s . t .} p (F \mid x, l, y) > p (F \mid x, l ^ {\prime}, y), \forall l ^ {\prime} \in \left\{l ^ {\prime} \mid 0 <   | l ^ {\prime} - l | <   r \right\} \tag {3}
$$
$$
a n d \quad p (F \mid x, l, y) > p (F \mid x, l, y ^ {\prime}), \forall y ^ {\prime} \neq y \in \mathcal {Y} \tag {4}
$$
Dense scene parsing with greedy proposal generation. Typically, object detection in complex scenes requires well chosen thresholds and detection hyperparameters. Our render-and-compare approach enables us to avoid tedious hyperparameter tuning by adopting a greedy approach to maximize the model likelihood (Eq. (1)) using a greedy proposal strategy. In particular, we optimize the likelihood greedily by starting from the object proposal that explains away the most parts of the image with highest likelihood, and subsequently update the likelihood of the overlapping proposals taking into account, that at every pixel in the feature map only one object can be visible [56]. Formally, given a list of objects proposals  $\{o_i = (O_{y,i},\alpha_{y,i})\}_{i = 1}^k$  (with predicted category label  $y$  and 6D pose  $\alpha$ ), we first order the object proposals based on their likelihood score  $s = p(F|o_i,B)$  such that  $s_i\leq s_j$  for  $i < j$ . Based on the ordering, we greedily update the 6D pose  $\alpha_{j}$  and the corresponding proposal likelihood for object  $o_j$  by masking out the foreground regions of previous objects  $o_i$  with  $1\leq i\leq j - 1$ . In this way, we can largely avoid missing close-by objects or duplicated detection.
Part and attribute prediction. Given the predicted location and pose of each object, we project the object mesh back onto the image to get the locations for each part. To predict the attributes for the objects and parts, we crop the region containing the object or part from the RGB image, and train an additional CNN classifier using the cropped patches to predict the attributes (color, size, material) and the fine-grained classes (i.e. different sub-types of cars) of each patch using a cross-entropy loss. The reason why this additional CNN classifier is needed instead of re-using the features from the 6D pose estimator is that the pose estimation features are learned to be invariant to scale and texture changes, which makes it unsuitable for attribute prediction.
Post-filtering. Finally, we post-process the located objects using the fine-grained CNN classifier. We compare the category labels predicted by the 6D pose estimator with the ones predicted by the CNN classifier, and remove the objects for which these two predictions do not agree. This post-filtering step helps with the duplicated detections that cannot be fully resolved with the 3D-NMS.