Visual Question Answering From Another Perspective: CLEVR Mental Rotation Tests
Authors: Christopher Beckham
Martin Weiss
Florian Golemo
Sina Honari
Derek Nowrouzezahrai
Christopher Pal
Venue:  (2021)
Keywords: vqa
clevr
contrastive learning
3d
inverse graphics

Abstract: Different types of \emph{mental rotation tests} have been used extensively in psychology to understand human visual reasoning and perception. Understanding what an object or visual scene would look like from another viewpoint is a challenging problem that is made even harder if it must be performed from a single image. 3D computer vision has a long history of examining related problems. However, often what one is most interested in is the answer to a relatively simple question posed in another visual frame of reference -- as opposed to creating a full 3D reconstruction. 
Mental rotations tests can also manifest as consequential questions in the real world such as: does the pedestrian that I see, see the car that I am driving?
We explore a controlled setting whereby questions are posed about the properties of a scene if the scene were observed from another viewpoint. To do this we have created a new version of the CLEVR VQA problem setup and dataset that we call CLEVR Mental Rotation Tests or CLEVR-MRT, where the goal is to answer questions about the original CLEVR viewpoint given a single image obtained from a different viewpoint of the same scene. Using CLEVR Mental Rotation Tests we examine standard state of the art methods, show how they fall short, then explore novel neural architectures that involve inferring representations encoded as feature volumes describing a scene. Our new methods use rigid transformations of feature volumes conditioned on the viewpoint camera. We examine the efficacy of different model variants through performing a rigorous ablation study. Furthermore, we examine the use of contrastive learning to infer a volumetric encoder in a self-supervised manner and find that this approach yields the best results of our study using CLEVR-MRT.

Under review as a conference paper at ICLR 2021
VISUAL QUESTION ANSWERING FROM ANOTHER PERSPECTIVE: CLEVR MENTAL ROTATION TESTS
Anonymous authors Paper under double-blind review
ABSTRACT
Different types of mental rotation tests have been used extensively in psychology understand human visual reasoning and perception. Understanding what an object or visual scene would look like from another viewpoint is challenging problem that is made even harder if it must be performed from a single image. 3D computer vision has a long history of examining related problems. However, often what one is most interested in is the answer to relatively simple question posed in another visual frame of reference as opposed to creating a full 3D reconstruction. Mental rotations tests can also manifest as consequential questions in the real world such as: does the pedestrian that I see, see the car that am driving? We explore a controlled setting whereby questions are posed about the properties of a scene if the scene were observed from another viewpoint. To do this we have created a new version of the CLEVR VQA problem setup and dataset that we call CLEVR Mental Rotation Tests or CLEVR-MRT, where the goal is to answer questions about the original CLEVR viewpoint given a single image obtained from a different viewpoint of the same scene. Using CLEVR Mental Rotation Tests we examine standard state of the art methods, show how they fall short, then explore novel neural architectures that involve inferring representations encoded as feature volumes describing a scene. Our new methods use rigid transformations feature volumes conditioned on the viewpoint camera. We examine the efficacy of different model variants through performing a rigorous ablation study. Furthermore, we examine the use of contrastive learning to infer a volumetric encoder in a self-supervised manner and find that this approach yields the best results of our study using CLEVR-MRT.
INTRODUCTION
Psychologists have employed mental rolation tests for decades (Shepard & Metzler 1971)
powerful tool for devising how the human mind interprets and (internally) manipulates three di- mensional representations of the world. Instead of using these test probe the human capacity for mental 3D manipulation, we are interested here in: a) understanding the ability of modern deep neural architectures to perform mental rotation tasks, and b) building architectures better suited 3D inference and understanding.
Recent applications of concepts from 3D graphics to deep learning, and vice versa, have led to promising results. We are similarly interested in leveraging models of 3D image formation from the graphics and vision communities to augment neural network architectures with inductive biases that improve their ability to reason about the real world. Here we measure the effectiveness of adding such biases, confirming their ability to improve the performance of neural models on mental rotation tasks. Concepts from inverse graphics can be used to guide the construction of neural architectures designed to perform tasks related to the reverse of the traditional image synthesis processes: namely, taking 2D image input and inferring 3D information about the scene. For instance, 3D reconstruction in computer vision (Furukawa & Hernández) 2015) can be realized with neural-based approaches that output voxel (Wu et al. 2016 Nguyen-Phuuue Phuoc et al. 2019), mesh (Wang et al. 2018 8) point cloud (Qi et al. 2017) representations of the underlying 3D scene geometry. Such inverse graphics methods range from fully-diie differentiable graphics pipelines (Kato et al. 2018) to implicit neural-based approaches with learnable modules designed to mimic the structure of certain components of the forward graphies pipeline (Yao al. 2018 Thies et al. 2019). While inverse rendering is potentially

---

Under review as a conference paper at ICLR 2021
an interesting and useful goal in itself, many computer vision systems could benefit from neural architectures that demonstrate good performance for more targeted mental rotation tasks.
In our work here standard benchmark for visual question-and and-answering (VQA) through answering questions with respect to another viewpoint. We the the Compositional Language and Elementary Visual Reasoning (CLEVR) Diagnostic Dataset JJhnsson et al.) 2017) as the starting point for our work. While we focus on this well known benchmark, many analogous questions of practical interest exist. For example, given the camera viewpoint of a (blind) person crossing the road, can we infer if each of the drivers of the cars at an intersection can see this blind person crossing the street? As humans, we are endowed with the ability to reason about scenes and imagine them from different viewpoints, even if we have only seen them from one perspective. As noted by others, therefore seems intuitive that we should encourage the same capabilities in deep neural networks (Harley et al. (209)). In order to answer such questions effectively, some sort of representation encoding 3DD information seems necessary to permit inferences to be drawn due to change in the orientation and position of the viewpoint camera. However, humans clearly do not have access to error signals obtained through re- re-rendering scenes, but able to perform such tasks. To explore these problems in controlled setting, we adapt the original CLEVR setup in which a VQA model is trained to answer different types of questions about scene consisting of various types and colours of objects. While images from this dataset are generated through the rendering of randomly generated 3D scenes, the diree-dimensional structure of the scene is never fully exploited because the viewpoint camera never changes, We call our problem formulation and data set CLEVR-MRT, as it is a new Mental Rotation Test version of the CLEVR problem setup. In CLEVR-MRT alternative views of scene are rendered and used as the input to perception pipeline that must then answer question that was posed with respect another (the original CLEVR) viewpoint P This gives rise to a more difficult task where the VQA model must learn how to map from its current viewpoint to the viewpoint that is required to answer the question. This can be seen Figure Fb.) Figure aa) depicts a real world situation and analogy where the answers to similar types of questions may help different types of systems make consequential decisions, e.g. intelligent intersections, cars, robots, or navigation assistants for the blind. The fact that MRT' applications motivated us to create the controlled setting of CLEVR-MRTS depicted in Figure (b).
interested in exploring neural "mental rotation" by adapting well known
classical tool of Psychology and the link to these different practical
(a) (I.fft) view of street corner. (Middle) CLEVR-like representation of the scene with abstractions buildings, cars and pedestrians. (Right) The same virtual scene from another viewpoint, where questions concerning the relative positions of objects after mental rotation could be of significant practical interest.
(b) Random views of an example scene in CLEVR-MRYS The center image is the 'canonical" view, which is the unseen point of view for which questions must be answered using only one of the other views as input. Figure I: (a) A real-world example where the ability to perform mental rotations can be of practical utility. (b) Images from the CLEVR-MES dataset.
Using our new mental rotation task definition and our CLEVR-MRT dataset, we examine a number of new graphiss inspired neural architectures. We examine models that use the FILM (Feature-wise Linear Modulation) technique (Perez et al. (017)) for VQA, which delivers competitive performance using contemporary -fate-of-the-arr convolutional network techniques. We observe that such methods fall short for this more challenging MRT VQA setting. This motivates us to create architectures that involve inferring latent feature volume that subject to rigid 3D) transformations (rotations and translations). in a manner that has been examined in 3D generative modelling techniques such spatial transformers (Jaderberg et al. 2015) as well as HoloGAN
Dataset and code will be available at https:tpp//......... http://///////g.t///////o//////a
2

---

Under review as a conference paper at ICLR 2021
Nguyen- Ngguyen-Puuoc Phuoc al., 2019). This can either be done through the adaptation of pre-trained 2D encoder network, i.e. an ImageNet-basee feature extractor as in Section [.2.1 or through training our encoder proposed here, which is obtained through the use of contrastive learning as in Section 2.2.3 In the case of the latter model, we leverage the InfoNCE loss (Oord et al. 2018) to minimise the distance between different views of the same scene in metrie space, and conversely the opposite for views of different scenes altogether. However, rather than simply using stochastic (2D) data augmentation policy create positive pairs (e.g. random crops, resizes, and pixel perturbations), we leverage the fact that we have access to many views of each scene at training time and that this can be seen as a data augmentation policy operating in 3D. This in turn can be leveraged learn an encoder that can map 2D views to 3D latent space without assuming any extra guidance such as camera extrinsics.
METHODS
We begin here by describing simple and strong baseline methods as well as upper bound estimates used to evaluate the performance of different techniques on this dataset. We then present our new approach to learning 3D features and two different ways to address this task.
2.1 FILM BASELINES
The architecture we use is very similar to that proposed by FILM (Perez et al. 2017), in which pre-trained ResNet-101 classifier on ImageNet extracts features from the input images which are then fed to a succession of FILM-modulated residual blocks using the hidden state output from the GRU. As sanity check to ensure our models are adequately parameterised the simplest baseline to run is one where images in the dataset are filtered to only contain canonical views. In this setting, we would expect the highest validation performance since the viewpoint given is the same as the canonical viewpoint (the viewpoint wrt which the question must be answered). The second and third baselines to run are ones where we use the full dataset, with and without conditioning on the viewpoint camera via FILM, respectively. This is illustrated in Figure 2 where we can see the viewpoint camera also being embedded before being concatenated to the question embedding and passed through the subsequent FILM blocks. Note that in the case of the canonical baseline, the viewpoint and canonical view is the same thing, so camera conditioning is necessary.
RESNET-101
FILM RESBLOCK
FILM RESBLOCK,
X
0g
0g f
FILM
FLM
VEWPONT
VIEWPOINT CAMERA
"How many
grean sphanes shiny gold thing?"
GRU
QUEESTION BUT POSED FROM THE CANONICAL VIEMOONT
The pipeline of our FILM baselines (Perez et al. 2017). An input view (which is randomly sampled from a scene) is fed through a pre-trained (on ImageNet) ResNet-101 model produce a high-dimensional stack of feature maps. The dotted border on the Re0 indicate it is frozen during training. The question embedding is fed through GRU which outputs an embedding vector of the sentence. The viewpoint camera is run through its own embedding embed & (c) before being concatenated this vector (the dotted line indicates that this stage is optional, depending on what baseline is run). The resulting feature maps are fed through FlLouuudllteedd residual blocks using the final embedding vector. (Please see the supplementary materials section for more details.)
Figure 2:
If we let S denote scene consisting of all of its camera views (images) X, the camera c, the question q, and its corresponding answer y, we can summarise this as the following:
3

---

Under review as a conference paper at ICLR 2021
S (X, q: c.)) D (sample scene)
Cgu GRU
x ~ (sample random view)
y = FILM& (h, [Egru ecam))
h (esNet(x))
Lds ((y,8),
Ceam embed (c)
where the encoder (a ResNet) is frozen and we do not update its parameters during training.
2.2 LEARNING 3D FEATURE REPRESENTATIONS FROM SINGLE VIEW IMAGES
So far we have been operating in 2D, based the pre-trained ResNet-101 ImageNet encoder which outputs nigh-dimensional stack of feature maps (a 3D tensor). To work in 3D, we would either need somehow transform the existing encoding into 4D tensor (a stack of 3D feature cubes) or use completely different encoder altogether which can output 3D volume directly. Assuming we already had such volume, we can manipulate the volume in 3D space directly by having undergo any rigid transformation that is necessary for the question to be answered. In Section 2.2. we illustrate a simple technique which simply takes the existing ImageNet encoder's features and runs it through learnable post-processing' block to yield 3D volume, and in Section 2.2.3 propose a sulf--upe contrastive approach to learn such an encoder from scratch.
2.2.1 PROJECTING 2D FEATURES INTO 3D FEATURES
To exploit the power of pre-trained representations, we start with a pre-trained ResNet encoder and transform its stack of feature maps through an additional set of 2D convolution blocks using the "post-processor' shown in Figure 3 right before reshaping the 3D tensor into 4D. In other words, learn module which maps from stack of feature maps h to a stack of feature cubes hn. Since the post-processor is learnable module through which the FILM part of the pipeline is able to backpropagate through, 3D feature volumes h'. Through pack-propagation it learns to perform well when manipulated with camera controllable FILM operations either is or, more interestingly, when also subjected rigid 3D transformations as we will see shortly in Section 2.2.2
be seen as learning
appropriate set of transforms that construct
RESNET-101
POSTRROC
ENCODER
Figure 3: The pre-trained ResNet encoder outputs a stack of feature maps h of dimensions (1024, 14, 14), in Fig. 2 but now a post-processing module postproc, (h), e.g. set of 2D convolutions, processes the feature stack and reshapes into 4D tensor h' of dimensions (64, 16, 14, 14), i.e., stack of feature cubes. This entire block (inside the grey border) is the new encoder.
2.2.2 3D CAMERA CONTROLLABLE FILM
In lieu conditioning the camera with FILM (as seen in Figure 2 with embed we can also condition on output translation and rotation parameters (0.,, Oy, Oz, En Ey tz) which are then used to construct a 3D rotation and translation matrix O Therefore, we can write out the 3D FILM pipeline as:
h' = pestoroc,,.encoder (x))
y = FILM (K &
0 2 θ n. 0 i, Ey, tx) embed (c) h := transform(hi h:; P(θ.. 0y 0,, Ez. Eyy t.))
Las E(y y), (2)
where now minimise fats with respect to our usual parameters ò and the post-processor parameters v). and P(.) is function that produces a rigid transform matrix from its parameters (via Euler angles). This is illustrated in Figure Note that in this figure have illustrated that we can still perform the camera do rigid transforms, modulate the
camera conditioning via efbed(flm) i.e.
4

---

Under review as a conference paper at ICLR 2021
ENCODER g
FILM RESBLOCK
FILM RESBLOCK
0y
0=
X
ENIEED
MENFOITT
VENPONT CAMERA
ENIEED
"How many
green spheres of gold thing?"
GRU
QUESTION BUT POSID FROM THE CANONICAL VEWPOIN
Figure 4: The 3D version of the FILM pipeline proposed in Section 2.2 The encoder can be either the 2D-to-3D formulation in Section 2.2.1 (with the ResNet-101 inside frozen but the postprocessor block learnable, i.e. Figure k) or the contrastive encoder in Section 2.2.3 (whcch has no post-processor and is completely frozen). A camera encoder embed
(c) is trained to map the camera coordinates of the scene to a transformation matrix which is used to transform the resulting 4D volume via an explicit rotation and translation, and/or it can be embedded and concatenated with the GRU embedding via embed (the dotted lines for both indicate that either/or are optional). This volume then fed to FILM- moodulaad ResBlocks (which internally compute 3D convolutions since we are dealing with a volume) to produce the answer.
&
GRU embedding, or both. For the sake of brevity, Equation 2 only shows the where the camera for rigid transforms via embed Also note that we cannot directly use the raw camera parameters c = (0z. yy. 0z. tz, ty. t₂) to construct the rigid transform is because these are relative to world coordinates. Finally, in the next section (Section 2.2.3), we will show that we can replace the encoder and its postprocessor (Figure 3 with a contrastive encoder trained from scratch, without the use of postprocessor since it directly outputs a latent volume.
2.2.3 LEARNING 3D CONTRASTIVE ENCODERS
In Section 1.2.1 the encoder proposed was an adaptation of pre-trainee rre-trained ImageNet classifier backbone to output latent volumes. Here we propose the training of an encoder from scratch in an unsupervised manner, via the use of contrastive learning as demonstrated in Chen et al. (2020). Conceptually it is simple: given two random views of the same scene X1 and X2. the goal of the encoder enc, infer latent volume from each of those views h1 and h2. To enforce the notion of similarity between these volumes (and conversely the opposite for an X2 that is not from the same scene as x1), both these volumes are reduced down to dense codes (through the contrastive loss is applied. This is shown in Figure []] Unlike Chen et al. (2020) however, our main goal here infer latent volumes for downstream tasks, rather than latent codes.
extra encoder enc.) Z1 and Z2, where
ENCODER (H)
(X)"
(X!l Tww
[ER! 17
(Z) Jr
HTTDNNT
ENCODER (H)
FACOOIR
[X.
(M S
(66.1)x000 x90)
Figure 5: The contrastive-based encoder, inspired from (Chen et al. 2020). We sample two sets of minibatches (1) and x where the i'th instance in each set comprises a positive pair (different views of the same scene). The H encoder generates a 3D volume for each view, and an additional Z encoder convolves this down to summarisation vector over which the contrastive loss is applied.
Let us denote (1) and (2) to be minibatches of images, with subscripts for individual examples in
the minibatch (e.g. x (")) We will assume that (x(1) (1)
x!2 (2)) correspond to the same scene if j.
otherwise they are different. The InfoNCE loss (Oord al. 2018) defined
H (Tc,,((((())) ( ench (T(X (1) enc, (H (() enc, (nccHHH)) (2))
5

---

Under review as a conference paper at ICLR 2021
NNCE 2 a 11 Σ
exp(sim (z.") (2(z)//))
NCE: where (()) log
Ei=1 exp(sim( (Z. _ )
where T(-) is some stochastic data augmentation operator (e.g. random crops, flips, colour pertur- bations) which operates on a per-example basis in the batch. This loss also contains a temperature term r, which is a hyperparameter to optimise (in practice, we found softmax loss). Since a large number of negative examples is needed to learn good features, we train this encoder on an 8-GPU setup with a combined batch size of 2048. Note that in Chen al. (2020), the contrastive loss is enforced between stochastic 2D data augmentations of the same image i.c. they contrast T T(X(1) (1) and ((x(2)) (with ( = x())). We refer to this as 2D only data augmentation, since the contrastive learner does not ever contrast between two different views of a scene. In the case of Equation 3 since X() * x(2) we call this 2D + 3D data augmentation, and if T is the identity function then we refer to this as 3D only data augmentation. When training of this encoder has converged, we freeze it and use it in place of the pre-trained ImageNet encoder and postprocesor as originally shown in Figure 3 as well as the same pipeline described in Figure and Equation 2
0.1 to produce the lowest
RELATED WORK
Several extensions of the CLEVR dataset exist, though mainly through NLP-based extensions such as evaluating systematic generalisation (Bahdanau et al. 2019), adding dialogue (Kottur et al. 2019). ) and robust captioning of changes between scenes (Park et al., 2019). In terms of visual- based extensions, Yi et al. (2019) proposed temporal version of CLEVR which looks at VQA in the context of causal and counterfactual reasoning. Concurrent to our work, a version of CLEVR has recently been proposed (Qiu et al. 2020) in the context of reinforcement learning. where an agent is trained to perform viewpoint selection scene consisting of a large occluder object in the center to accentuate occlusions. However, the main difference is that dataset decouples the camera viewpoint from the viewpoint from which the question must be answered. Furthermore, this dataset has relatively limited question and scene variability (for instance, focusing on only two types of questions and the same occluding object in the center). Lastly, in our work we do not assume the VQA model is an agent that is able to vary its active viewpoint to better answer the question instead, our model must learn to 'imagine' what the same scene should look like from another perspective, conditioned only on single view.
scene to be able to answer the question, with each
Using rigid transforms to infer latent 3D volumes was loosely inspired by HoloGAN (Nguyen Phuoc et al., 2019). Here, they use a GAN to map randomly sampled noise vector to 3D latent volume before subsequently rendering using a neural renderer. Several works Sgguyen-Phuoe et al. 2018 Sitzmann et al. 2018: Lombardi et al., 2019) condition on images and camera poses to learn voxels that represent the input, with options to re-render from different camera poses. These methods, however, assume that the camera poses are known and evaluated on single-scene settings. Conversely, our dataset consists of tens of thousands of scenes, which makes any re-rendering task (i.e. autoencoding) significantly more difficult due to the need to reconstruct well all scenes, all while having a larger computational footprint than decoder-less approaches. Neural rendering is also not limited to just voxel representations; other ways of encoding 3D data can be used such point clouds (Qi et al. 2017), meshes (Wang et al.) 2018 Kato et al. 2018), surfels (Rajeswar et al. 2020). latent codes (Eslami et al. 2018). or even in the weights of a neural network (Mildenhall et 2020
Other VQA models have been proposed, e.g., MAC (Hudson & Manning 2018) proposes a memory and attention-based reasoning architecture for more interpretable VQA. While this could in principle be modified to leverage 3D volumes, FILM serves as a simpler architectural choice for analysis. As for learning encoders, contrastive learning is currently state-of-the of-the-art in learning self-supervised representations that are competitive with their supervised variants (Oord et al. 2018 Bachman et al., 2019 Chen et al. 2020, He et al.) 2020). Tian et al. (2019) explored contrastive learning of scenes, though the multiview aspect in this setting was applied to different sensory views, rather than camera views. HHaleyy et al. (2019) explored the use of contrastive learning on 2.5D video (i.c. RGB + depth) to predict novel views, with the goal of learning 3D object detectors in semi-supervised manner.
The CLEVR dataset (Johnson et al., 2017) is a VQA dataset consisting of a range of synthetic 3D shapes laid out canvas. The dataset consists of range of questions designed to test various aspects of visual reasoning such as counting (c.g. 'how many red cubes are in this scene?"), spatial relationships (e.g. "what colour is the cylinder to the left of the big brown cube?') and comparisons

---

Under review as a conference paper at ICLR 2021
(e.g. "are there equal number of blue objects red ones?"). In recent years however, proposed techniques have performed extraordinarily well on the dataset (Perez et al. 2017: Hudson & Manning 2018), which has inspired us to explore VQA in more difficult contexts. The ongiinal CLEVR dataset provided one image for each scene. CLEVR-MRT contains 20 images generated for each scene holding a constant altitude and sampling over azimuthal angle. To ensure that the model would not have any clues as to how the view had been rotated, we replaced the asymmetrical "photo backdrop" canvas of the CLEVR dataset with large plane and centered overhead lighting. To focus on questions with viewpoint dependent answers, we filtered the set of questions to only include those containing spatial relationships (e.g. 'is X to the right of Y'). From the original 90 question templates, only 44 contained spatial relationships. In total, the training validation split consists of 45,600 scenes, each containing roughly questions for a total of 455,549 questions. 5% of these scenes were set aside for validation. For the test set, 10,000 scenes were generated with roughly 5 questions each, for total of 49,670 questions. Figure [b) shows an example of one of these scenes.
RESULTS AND ANALYSIS
For each experiment, we perform a sweep over many of the hyperparameters (which are detailed in the appendix) find the experiment which performs the best, according to validation set accuracy. We then select the best-performing experiment and run repeats of with varying seeds (3-6, depending on the overall variance), for total of N runs. The validation set accuracy reported is the mean over these N runs, and similarly for the test set. It is worth noting that for our 3D FILM experiments, some runs appeared to hit undesirable local minima, exhibiting much lower validation accuracies, which we conjecture is due to a 'domain mismatch' between using a encoder that was initially pre-trained for ImageNet classification and our CLEVR dataset, (This appears to be supported by the fact that these outliers do not exist when we use our pre-trained contrastive encoder, in Table 2) To deal with these outliers, we instead compute the mean/stdev over only the top three runs out of the N 6 we originally trained.
Table 1: Table of results for experiments run using pre-trained ResNet-101 encoder. The Upper bound model is baseline model where only canonical views are given as input, and is expected to have the highest performance since it does not have to answer questions from another random viewpoint as input. For the columns shown: 3D? refers to whether we are using 2D) latents or 3D latents (the difference between Figs 2 and 4) camera (embed) refers to embedding the camera coordinates and concatenating with the question embedding: camera (rotation) refers to using the camera to map to a rigid transform of the volume (shown in Figure The result denoted with (in small text) indicates the same experiment but with the postprocessor frozen after random weight initialisation. See Supp. 6.3 for more MAC baseline. Accuracies shown percentages. For all
3D FILM, projection results, the mean/stdev is computed over the top 3 performing models (out of 6).
Method
3D?
camera (embed)
camera (rotation)
valid acc. (%)
test acc. (%)
Majority class
24.72 + 0.00
24.75 1: 0.00
GRU-only
49.38 0.40
Upper bound (canon. views only)
X
94.19 = 0.39
94.24 0.40
MAC (Hudson Manning
2018)
X
/
70.96 0.97
X
X
X
70.63 + 0.19
69.60 + 0.09
2D FILM (Sec
2.1 Fig
X
83.95 1.21
83.68 + 1.21
X
68.19 = 1.87
3D FILM, projection (Sec 2.2.1
88.82 3.04
86.36 + 3.46
Fig 4
X
92.80 0.30
90.86 + 0.87
(1 98 + 1.4))
89 83 = 1.36
89.68 1.34
Our results for the FILM baselines (Section 2.1 and using 2D-to-3D projections (Section 2.2. are shown in Table [ What find surprising that the 2D baseline without camera conditioning is able to achieve a decent accuracy of roughly 70%. On closer inspection the misclassifications do not appear to be related to how far away the viewpoint camera is from the canonical, with misclassified
7

---

Under review as a conference paper at ICLR 2021
Table 2:
Table of results for experiments run with the contrastive encoder described in Section 2.2.3 For the columns shown: Data aug refers to what data augmentation scheme was used, for stochastic data augmentation operator TÜ (2D contrast T(X,) and T(X;), 3D contrast X, and X2. 2D+3D = contrast T(Xi) and T(x2): NCE accuracy is how well the contrastive encoder is able to distinguish between pairs of views that belong to the same/different scene; the remaining
columns denote the FILM task, as described in Table [
Contrastive pre- vre-training stage
FILM stage
Data aug
NCE accuracy (valid)
camera (embed)
camera (rotation)
valid acc. (%)
test acc. (%)
X
59.78 + 0.23
59.14 + 0.43
2D
0.1
9.13
x
/
59.29 + 0.44
58.57 + 0.53
/
X
57.42 1 0.26
56.74 + 0.31
3D
0.1
99.72
X
/
57.63 1 0.21
57.10 + 0.33
X
65.15 + 4.63
63.70 = 3.74
2D + 3D
0.1
98.14
X / 87.49 + 0.78
86.01 + 0.69
points being distributed more or less evenly around the scene. Given that the question-only baseline ("GRU-only') ) is able achieve accuracy significantly greater than that of the majority class baseline (x 25% versus x 50%), it seems like it is likely exploiting statistical regularities between the actual question itself and the scene. If we add camera conditioning via FILM (that is, appending the camera embedding to the GRU's embedding) then we achieve a much greater accuracy of 83.68 + 1.21. Furthermore, our results demonstrate the efficacy of using rigid transforms with 3D volumes, achieving the highest accuracy of 92.80 = 0.30 on the test set. If we take the same experiment and freeze the postprocessor (denoted by the small symbol), then we achieve a much lower accuracy of 69 %. This is to be expected, considering that any camera information that is forward- propaaate will contribute gradients back to the postprocessing parameters in the backward propagation, effectively giving the postprocessor supervision in the form of camera extrinsics. (However, we will soon show just as good performance for our contrastive encoder experiments in Table which did not utilise any camera extrinsics in the pre-training phase, and postprocessor in the FILM stage.) Finally, the last row of the table shows that if one uses the camera for both rigid transforms and embedding, the mean test accuracy is roughly the same as the igiidtransform--nif variant (90.86 + 0.87 vs 89.68 + 1.34). This appears to suggest that simply performing rigid rotations of the volume is sufficient by itself for good performance.
In Table [ we perform an ablation on the type of data augmentation used during the contrastive pre-training stage (described in Section 2.2.3) and find that 3D data augmentation essential for the encoder to distinguish whether a pair of views come from the same scene or not, as shown in the 'NCE accuracy' column. However, both 2D and 3D data augmentation necessary in order for the FILM task to yield the best results, as seen in the last row. Similar to Table [ utilising the viewpoint camera for rigid transforms produces the best results, with 86.01 0.69 test accuracy. While the best result of Table [ does have much higher variance (86.60 = 7.57), we consider our contrastive 2D + 3D experiment to yield our best result due to its almost negligible variance.
Our results show that our best performing formulation (either 2D-to-3D or contrastive) performs on average only 8 % less than the canonical baseline's 94%, which can be seen as a rough upper bound on generalisation performance. While we obtained promising results, it may not leave lot of room to improve on top of our methods, and we identified some ways in which the dataset could be made more difficult. One of them is removing the viewpoint camera coordinates and instead placing a sprite in the scene showing where the canonical camera is. This means that the model has an additional inference task it has to perform, which inferring the 3D position of the canonical viewpoint from a 2D marker. Another idea is to allow some variation in the clevation of the viewpoint camera. While this can accentuate the effects of occlusion (if the camera is allowed to go lower than its default elevation), it also provides for more grounded dataset since occlusions are commonplace in real-world datasets. We examine the latter here, generating a version of CLEVR-ME
where the camera elevation is allowed vary, and with both small and large objects present in the scene (small objects were not present in the original CLEVR-MEE dataset). Concretely, the default elevation in the original dataset was 30", whereas now it is randomly sampled from Uniform (20, 30). An example
scene of this dataset is shown in Figure 6
8

---

Under review as a conference paper at ICLR 2021
Figure 6: Left-most image is the canonical view. The rest are randomly sampled views, of varying azimuth and elevation. Compared to Figure Tb occlusions here are more prevalent.
See Table below. Please note that hyperparameter tuning for these experiments are still ongoing, with particular emphasis placed on improving the 3D FILM with camera embedding (third row) as well as the canonical upper bound (first row).
Table 3:
Select experiments from Table but trained on CLEVR-MRT. For all experiments shown in this table, the mean/stdev is computed over the top three runs (out of six in total).
Method
3D?
camera (embed)
camera (rotation)
valid acc. (%)
test acc. (%)
Upper bound (canon. views only)
X
90.00 = 0.23
89.37 + 0.19
X
67.26 0.78
2D FILM (Sec 2.1
Fig
x
/
79.69 = 2.05
79.14 + 2.35
65.49 1.46
65.10 1.67
3D FILM, projection (See 2.2. 2.2c1.2.1 Fig 4
V
X
86.92 = 2.00
86.89 + 2.04
V
89.98 = 0.59
89.91 + 0.73
CONCLUSIONS AND BROADER IMPACTS
We address the problem of answering a question from a single image, posed in a reference frame that is different to the one of the viewer. We illustrate the difficulties here in controlled setting, proposing a new CLEVR dataset and exploring a 3D FILM- architecture that operates directly on latent feature volumes (using camera conditioning via FILM via direct rigid transforms the volume). We propose two techniques to train volumetric encoders: with 2D>to-3D projection of ImageNet features, using a self-supervised contrastive approach. In the latter case, we showed that the use of combined 2D+3D data augmentation was crucial to learning volumetric encoder, without the use of camera extrinsics nor a postprocessor. Through rigorous ablations, we demonstrated that performing 3D FILM was the most effective for CLEVR-MRT especially when the volume can be subjected rigid transformations in order to answer the question.
Endowing intelligent embodied systems with the ability to answer questions regarding properties of a 3D visual scene with respect to the perspective of another agent could make such systems safer. In the case of autonomous vehicles, better control decisions could eventually be made. If such systems are adversarial in nature, negative outcomes could arise to the adversaries of such systems.
REFERENCES
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp. 15535-15545, 2019.
Dzmitry Bahdanau, Harm de Vries, Timothy J O' Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv:19/2.05783, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 2002:20000.05709 2020.
SM Ali Eslami. Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu. Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 36016394): 2018.

---

Under review as a conference paper at ICLR 2021
Yasutaka Furukawa and Carlos Heráández. Multi-view stereo: A tutorial. Foundations and Trends in Computer Graphics and Vision, 91-2):1-148, 2015.
Adam W Harley, Shrinidhi K Lakshmikanth, Fangyu Li, Xian Zhou, Hsiao- Yu Fish Tung, and Katerina Fragkiadaki. Learning from unlabelled videos using contrastive predictive neural 3d mapping. arXiv preprint arXiv: (ekiv:1900.. 1906 1906.037644, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9733 2020.
Drew A. Hudson and Christopher D. Manning. Compositional attention networks for machine reasoning. CoRR, abs/1 803 000000 2018. httptpp////axxx http://aaixxv...oaaaaa/// wttp://rrg/bbb....o.ooo
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Advances in neural information processing systems, pp. 2017-2025, 2015.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2901-291111 2017.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3907-3916, 2018.
Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. Clevr-dialog: A diagnostic dataset for multi-round reasoning in visual dialog. arXiv preprint 1WXXv:1903.03166 2019.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Transactions on Graphics (TOG), 38(4):65, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. arXiv preprint arXiv:2003.0888 08934, 2020.
Thu Nguyen-Phuoe, Chuan Li, Stephen Balaban, and Liang Yang. Rendemet: A deep convolu- tional network for differentiable rendering from 3d shapes. CoRR, abs/ htw/1880.065755 2018. URL
http:://arxiw.orr//aass. 06677
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. CoRR, abs/1904 1904001366..6 2019.
URL htt.thttp//axiitt
org al0p///////99........ .03326
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEE international conference on computer vision, pp. 4624 4633, 2019.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. CoRR, abs/1779.07871, 2017. URL
7/arxiv. wttpv//.rr/aaabb//bb/aaab
07871
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep leamiing point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 652-660,, 2017.
Yue Qiu, Yutaka Satoh, Ryota Suzuki, Kenji Iwata, and Hirokatsu Kataoka. Multi-view visual question answering with active viewpoint selection. Sensors, 20(8):2281, 2020.
10

---

Under review as a conference paper at ICLR 2021
Sai Rajeswar, Fahim Mannan, Florian Golemo, Jéröme Parent- Parent-Lévesque, David Vazquez, Derek Nowrouzezahrai, and Aaron Courville. Pix2shape: Towards unsupervised learning of scenes from images using view-based representation. International Journal of Computer Vision, pp. 1-16, 2020.
Roger N Shepard and Jacqueline Metzler. Mental rotation of three-dimensional objects. Science, 171 (3972):701-703, 1971.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. CoRR, abs/1812.01024, 2018.
URL http:://axxi...orr///// http://www...raaa/////////// abs//81s/2111
01024]
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions Graphics (TOG), 38(4):1-12, 1-1,, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 0et.49996.0849 2019.
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating mesh models from single rgb images. ECCV, 2018.
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a probabilis- tic latent space of object shapes via 3d information processing systems, pp. 82-90, 2016.
aevessrrraaaaaa modeling. In Advances in neural
Shunyu Yao, Tzu-Ming Harry Hsu, Jun-Yan Zhu, Jiajun Wu, Antonio Torralba, William T. Free- man, and Joshua B. Tenenbaum. 3d-aware scene manipulation via inverse graphics. CoRR, abs/1 00211888..0955 2018. URL http: arxiv. org/ abs/ 1180//10.000 09351
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442,,200 2019.


Reviews:
[{"review_id": "dIecMfK5HH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper studies visual question answering focusing on answering questions in a reference image of a different viewpoint. They propose a new dataset CLEVR-MRT drawing motivation from the well-known visual reasoning dataset CLEVR to illustrate the idea in which they have full control of the changes of viewpoints in an image. They then propose to use a volumetric encoder to represent 3D image features of an image via either 2D-to-3D projection or a contrastive-based encoder and further adapt an existing method (FiLM) to handle 3D tensors. Experiments on the CLEVR-MRT show that the use of the 2D features and 3D features of an image is complementary to each other.\n\nComments (Technical, Major Flaws of this paper): \n\n(1) The idea of addressing VQA in multi-view settings is reasonable but it is not entirely new. My main concern is at the limitations of a synthetic dataset in a controlled setting where the relations between objects are limited compared to real data. In addition, I believe that given enough such generated question-answer pairs with associated programs, models may possibly learn to decode the generation procedure under the hood instead of learning the actual semantic meanings of languages and the relations between objects.\n\n(2) Since there are no statistics about the newly introduced dataset, it is hard to judge the empirical results in the paper. As pointed out by many previous studies (e.g. Hudson, D.A., et al., 2019; Le, T.M., et al., 2020), models' performance seems to converge on CLEVR given enough training data. Having that said, existing methods easily fail if we reduce the number of training instances. As for the CLEVR-MRT, even without any information about the viewpoints, the baseline models could achieve more than 70% accuracy on the proposed dataset. It seems that the dataset is too simple that the model could have good performance without knowing the camera parameters. This leads to concerns about the validity of the proposed dataset. Please address these points.\nReferences:\n - Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2020). Dynamic Language Binding in Relational Visual Reasoning. In IJCAI 2020.\n - Hudson, D. A., & Manning, C. D. (2018). Compositional attention networks for machine reasoning. In ICLR 2019.\n\n(3) For those who are not familiar with the CLEVR dataset, briefly explaining the procedure to generate the dataset and its variants might be helpful. \n\n(4) Given a question related to the object positions, there may exist many different views that provide the same answer. Let's take the question \"How many green spheres to the left of the shiny gold thing?\" in Figure 4 as an example. There are many views in the scene that provide the correct answer \"1\" for this question.   Without restricting the variance of the camera view (as in [1]), how can we ensure the model to infer the correct viewpoint?\n\nSome typos:\n- (1): 1. Introduction: We use the the Compositional -> We use the Compositional\n- (2): 2.1 FILM Baseline: the viewpoint and canonical view is the same thing -> the viewpoint and the canonical view are the same thing\n- (3): Figure 2: The dotted border on the ResNet-101 indicate -> The dotted border on the ResNet-101 indicates\n- (4): Conclusion: In the case of an autonomous vehicles -> In the case of autonomous vehicles", "extra_scores": {}, "extra_text": {}}, {"review_id": "_BrD2XsKL1h", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n### Overall\n\nAuthors extend CLEVR dataset so as to consider multiple viewpoints, and evaluate current neural network models in that setting. They also update a standard approach to introduce camera viewpoint information in the network so it can better answer visual question from the canonical scene frame even from other perspectives.\n\n### Positive aspects\n\n* Authors provide a study on a important topic of Computer Vision: understanding multiple views of a same scene. They do such study on a hard task, which is VQA. Actually, authors provide a more complex version of a simple VQA dataset (*simple* because it is synthetic and has very well established domain limits).\n* Authors evaluate different training frameworks (supervised and unsupervised from scratch).\n* Authors provided accuracy values for pretraining with NCE, which can be helpful.\n* Results seem to be promising.\n* In general, text is well written and easy to read.\n* It is interesting that even a frozen pretrained network provides good results in such visually different dataset. Although, it was nice that authors trained an encoder from scratch.\n* Code already available!\n\n### Weak aspects and suggestions\n\n* The problem is interesting, though my main concern is regarding the novelty and contribution of the paper. It seems to be an adaptation of CLEVR dataset, and an adaptation of the FILM model. In addition, authors use camera viewpoint information to ease the identification of the scenes. I have mixed feelings in using such specific kind of information in the model, because in a real world scenario we don't have access to them. I might be wrong, but maybe it is possible to insert a module in their approach to estimate the camera parameters, so as the network itself could learn to predict how viewpoints work and how scenes change with that. I think this could be done by adding such parameters as target information some of the models. For instance, the unsupervised architecture could be trained to predict whether the scenes are the same, but also the camera parameters. Apologies if I miss something here.\n\n* The proposed architecture seems to be basically an adaptation of the FILM model considering camera viewpoint information.\n\n* FILM (2018) is the best performing approach in CLEVER to date? There are more recent approaches that could be used in the results section as baselines.\n\n* It is unclear what happened to the spatial-related questions. They were removed of the dataset?\n\n* Results are promising, although why do they have such high variance? (7-8% of variance is not negligible by any means); considering that for some experiments it is likely that 2D FILM provides similar performance than 3D one. A statistical test might help to verify whether such results are statistically significant or not.\n\n* Font size for all images should be quite larger. It is hard to read in the current size.\n\n* Figure of the post processor does not help much. Authors could detail a little bit more what is inside that $postproc_w$ box.\n\n* *\"Since the post-processor is a learnable module through which the FILM part of the pipeline is able to backpropagate through, it can be seen as learning an appropriate set of transforms that construct 3D feature volumes h0.\"* I suggest rewriting this sentence, it is very confusing.\n\n* *\"While we obtained great results, it may not leave a lot of room to improve on top of our methods,\"* This sentence is odd. The sentence \"we obtained great results\" can be written in a more objective and scientific way (avoid the usage of adjectives). Another important aspect is: often it is easy to provide first large steps in a task (ImageNet for instance), although it gets much harder to improve on that when results are good (AlexNet vs ResNet, see the performance difference). Another aspect: maybe authors made the task too easy and should have explored more challenging scenarios.\n\n* *\"and we identified some ways in which the dataset could be made more difficult\"* Those ideas to make the task more challenging are indeed important. Why authors did not perform experiments in such scenarios? It does not seem very hard to generate such datasets.\n\n* Is it possible to visualize and understand what the postproc module does? It would be nice to visually explain the $h'$ (64, 16, 14, 14) tensor represents.\n\n* There could be some qualitative analysis.\n\n* The dataset extension seems to be a large portion of the work. I think it could have a separate section with more details.\n\n### Additional questions\n\n* What happens if other conditioning camera information strategy is used? For instance, simply concatenating or using other simpler fusion techniques. FILM would perform much better than other simpler approaches?\n\n* *\"ResNet outputs... feature maps h of dimensions (1024,14, 14)\"* Is this correct? I believe Resnet101 outputs (2048, 14, 14) feature maps.\n\n* *\"in practice, we found $\\tau = 0.1$ to produce the lowest softmax loss.\"* Which ones you have tested? Why $\\tau$ is 1.0 in Table 2?\n\n* *\"Another idea is to allow the viewpoint camera’s elevation to change. \"* That is true. Or even the distance from the camera. Why did authors decide not to include such examples in this work?\n\n* *\"This is to be expected, considering that any camera information that is forward-propagated will contribute gradients back to the postprocessing parameters in the backward propagation, effectively giving the postprocessor supervision in the form of camera extrinsics.\"*. Can authors support/prove this claim?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "P-OFBR52the", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores the problem of visual question answering from another perspective. Similar to VQA, a system is provided with a scene and a question. However, the difference is that the question needs to be answered from a viewpoint different from the one provided. Hence, the system needs to perform “mental rotation”. The paper creates a new dataset called CLEVR Mental Rotation Tests which is based on the prior CLEVR dataset. The paper also studies the efficacy of various supervised and self-supervised models on the proposed dataset.\n\n#### Strong points:\n- The problem of asking questions related to “mental rotation” seems interesting.\n- The paper shows that contrastive pre-training could be useful for the task, which is an interesting result.\n\n#### Weak points: \n- Although the problem seems interesting, I am unclear about the usefulness of the proposed dataset. The paper says that “many computer vision systems could benefit from neural architectures that demonstrate good performance for more targeted mental rotation tasks.” To justify this claim it gives the following example, “given the camera viewpoint of a (blind) person crossing the road, can we infer if each of the drivers of the cars at an intersection can see this blind person crossing the street?”. This could potentially be a useful scenario, however, the dataset proposed is different from the example as the camera viewpoint is provided as part of the input and not inferred from the question. The paper does not provide justification or evidence of how the current setup (i.e. with camera viewpoint) is useful. In particular, it would be nice if the paper could further explain how the current setup is better than solving the “view rendering” and VQA problems separately.\n\n- The dataset seems to be too simple for the mental rotation tests. It is unclear if in the future the dataset would be useful in distinguishing which models are better. As the paper shows that “2D baseline without camera conditioning” already achieves 70% accuracy. As far as I could understand, even without knowing which view to look at, a model could achieve 70% accuracy indicating that there is a lot of bias in the dataset. Moreover, simply adding camera embedding with the question to a 2D baseline (Table 1, 2D FILM with camera), already performs close to the best 3D model and upper bound. (Please clarify if my understanding is wrong.)\n\n- The paper is poorly organized and hard to follow. For example, one of the contributions of the work is the CLEVR-MTR dataset, however, there is no clear section in the main paper describing the details of how the dataset. Instead, the information about the dataset is scattered in the introduction and related work. Another example is that the paper moves into talking about the method (Section 2) without defining the task concretely. It is only from the figure that one notices that the camera viewpoint is part of the input. From the examples provided in the introduction, the reader is under the impression that the camera viewpoint has to be inferred from the question itself. Similarly, it's hard to parse what the training signal for each baseline is. Does a baseline use the rendered image from the other view during training?\n \n#### Minor Comments:\n- The figures and tables are interspersed with the text making the paper harder to read. It might be better to place the figures and tables at the end of the beginning of the page so that the captions are separated from the main text.\n- Many equations like some parts of equation 1 and equation 2 might not be necessary as they don’t seem to contribute to understanding the paper. In many places, it seems like a simple intuitive explanation would be sufficient.\n- Similarly, Figure 3 might not be necessary.\n- The figures are unclear and hard to understand. For example, is the canonical viewpoint part of the input? If not, Figure 2 and Figure 4 could be changed to make it more clear.\n- Is this line correct, “If we add camera conditioning via FILM (that is, appending the camera embedding to the GRU’s embedding) then we achieve a much greater accuracy of 69.60 ± 0.09.” Should its value be 83.68 ± 1.21 as indicated in the table?\n \n#### Overall Recommendation:\nAlthough the problem could potentially be useful, the current dataset seems to be not so useful and over-simplified. Moreover, I found the paper not well-organized and hard to understand even after multiple reads. I feel the paper can be improved a lot and hence recommend rejection for the current version.\n\n#### Post Rebuttal\n(Copying from the discussion below)\n\nI would like to thank the author(s) for their response. After going over them, I am still not very confident about the paper would stick to my initial assessment. Following are my primary concerns:\n\n\"We note that there is a distinction between wanting to see something from another point of view, versus wanting to answer a question from another point of view. The former is where re-rendering is appropriate, but we do not make the claim that this alternative (view rendering + VQA) performs better or worse empirically.\"\n\nI understand the distinction. But the issue still remains. Why is the out-of-the-box \"view rendering + VQA\" solution insufficient? Is there any empirical justification for it? If not its hard to see the value in the current setup. A potential way to address this could be to run a simple out-of-the-box \"view rendering + VQA\" baseline.\n\n\"(2) R3 and R4’s concern about camera information being provided to the model and its potential infeasibility in practice: In real world settings, camera rigs can and do have knowledge about where they are situated in the world, for instance using SLAM or GPS coordinates. In that case, it is not unreasonable for e.g. an autonomous vehicle to answer queries by performing rotations and/or translations of its current viewpoint.\"\n\nThe concern was not about the viewpoint of the observer but the new viewpoint from which the question has to be answered. Also, the location of the new viewpoint need not be converted into float and appended to the question. It could be expressed in natural language. For example \"viewpoint of the driver in the other car\" like in the example provided by the paper. In the current setup the information about this viewpoint is provided in terms of exact coordinates, which makes the setup less interesting and not so practical.\n\nAlthough the authors improved some of the figures, the latest version of the paper does not seem to address other clarity concerns like a clear section for the dataset; organization of text and figures; removing unnecessary equations", "extra_scores": {}, "extra_text": {}}, {"review_id": "HpiDQBdAgH5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros: \n\n1. The paper presents an interesting idea to learn mental rotations using a variation of the CLEVR-VQA dataset. The contributions are - the creation of this synthetic CLEVR-Mental Rotation dataset for targeting this problem and a model that encodes questions and viewpoint information to produce answers via FiLM based encoders and 3D volume encoder. \n2. The results in Table 1 and Table 2 show improvements with respect to the baselines using their final model but there is still some concern in the improvements on their ablations. \n3. The paper is well written and easy to understand. \n\nCons: \n1. The motivation of why we need to learn mental rotations is not very clearly expressed, the practical examples given in the introduction are not sufficient. Does the model really learn these mental rotations from a simple spatial VQA task? This should be evaluated in the experiments either using activation maps or by visualizing intermediate 3D encodings. \n2. Is the model trained on all views for a single question-view pair or any one random viewpoint is sampled during mini batch training ? Does the rotation of a scene done over the complete 360 degree ? How do you decide how much to rotate to generate a viewpoint ?\n3. The self supervised learning of 3D volumes is an interesting idea, but it's use case in this particular problem is very weakly motivated both in experiments and theory. Why is this method better than the method discussed in Section 2.2.1? What is 3D data augmentation and how is it different from 2D data augmentation? \n4. There is a large variance in some experiments in Table 1. Is it due to the camera transformation embedding? It will be good to discuss the reasons why this is in Table 1 and not in Table 2.  \n5. Although the models developed are used in a very different problem setting with minor contributions, still a large part of the methods seem to be derived from the literature. \n6. The final results in Table 2 though argued are better due to small variance but more extensive experiments need to be performed to show the benefits of the self-supervised pre-training over the traditional encoder approach.\nMinor: What is the value of t (tau) used in Eq 3 ? In Table 2 it shows 1.0, but in the text it’s discussed as 0.1. Is this a typo or both of them are supposed to be different, if yes why ?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper was reviewed by 4 experts in the field. The reviewers raised their concerns on lack of novelty, unconvincing experiment, and the presentation of this paper, While the paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.

Decision: Reject