Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
### Overall

Authors extend CLEVR dataset so as to consider multiple viewpoints, and evaluate current neural network models in that setting. They also update a standard approach to introduce camera viewpoint information in the network so it can better answer visual question from the canonical scene frame even from other perspectives.

### Positive aspects

* Authors provide a study on a important topic of Computer Vision: understanding multiple views of a same scene. They do such study on a hard task, which is VQA. Actually, authors provide a more complex version of a simple VQA dataset (*simple* because it is synthetic and has very well established domain limits).
* Authors evaluate different training frameworks (supervised and unsupervised from scratch).
* Authors provided accuracy values for pretraining with NCE, which can be helpful.
* Results seem to be promising.
* In general, text is well written and easy to read.
* It is interesting that even a frozen pretrained network provides good results in such visually different dataset. Although, it was nice that authors trained an encoder from scratch.
* Code already available!

### Weak aspects and suggestions

* The problem is interesting, though my main concern is regarding the novelty and contribution of the paper. It seems to be an adaptation of CLEVR dataset, and an adaptation of the FILM model. In addition, authors use camera viewpoint information to ease the identification of the scenes. I have mixed feelings in using such specific kind of information in the model, because in a real world scenario we don't have access to them. I might be wrong, but maybe it is possible to insert a module in their approach to estimate the camera parameters, so as the network itself could learn to predict how viewpoints work and how scenes change with that. I think this could be done by adding such parameters as target information some of the models. For instance, the unsupervised architecture could be trained to predict whether the scenes are the same, but also the camera parameters. Apologies if I miss something here.

* The proposed architecture seems to be basically an adaptation of the FILM model considering camera viewpoint information.

* FILM (2018) is the best performing approach in CLEVER to date? There are more recent approaches that could be used in the results section as baselines.

* It is unclear what happened to the spatial-related questions. They were removed of the dataset?

* Results are promising, although why do they have such high variance? (7-8% of variance is not negligible by any means); considering that for some experiments it is likely that 2D FILM provides similar performance than 3D one. A statistical test might help to verify whether such results are statistically significant or not.

* Font size for all images should be quite larger. It is hard to read in the current size.

* Figure of the post processor does not help much. Authors could detail a little bit more what is inside that $postproc_w$ box.

* *"Since the post-processor is a learnable module through which the FILM part of the pipeline is able to backpropagate through, it can be seen as learning an appropriate set of transforms that construct 3D feature volumes h0."* I suggest rewriting this sentence, it is very confusing.

* *"While we obtained great results, it may not leave a lot of room to improve on top of our methods,"* This sentence is odd. The sentence "we obtained great results" can be written in a more objective and scientific way (avoid the usage of adjectives). Another important aspect is: often it is easy to provide first large steps in a task (ImageNet for instance), although it gets much harder to improve on that when results are good (AlexNet vs ResNet, see the performance difference). Another aspect: maybe authors made the task too easy and should have explored more challenging scenarios.

* *"and we identified some ways in which the dataset could be made more difficult"* Those ideas to make the task more challenging are indeed important. Why authors did not perform experiments in such scenarios? It does not seem very hard to generate such datasets.

* Is it possible to visualize and understand what the postproc module does? It would be nice to visually explain the $h'$ (64, 16, 14, 14) tensor represents.

* There could be some qualitative analysis.

* The dataset extension seems to be a large portion of the work. I think it could have a separate section with more details.

### Additional questions

* What happens if other conditioning camera information strategy is used? For instance, simply concatenating or using other simpler fusion techniques. FILM would perform much better than other simpler approaches?

* *"ResNet outputs... feature maps h of dimensions (1024,14, 14)"* Is this correct? I believe Resnet101 outputs (2048, 14, 14) feature maps.

* *"in practice, we found $\tau = 0.1$ to produce the lowest softmax loss."* Which ones you have tested? Why $\tau$ is 1.0 in Table 2?

* *"Another idea is to allow the viewpoint camera’s elevation to change. "* That is true. Or even the distance from the camera. Why did authors decide not to include such examples in this work?

* *"This is to be expected, considering that any camera information that is forward-propagated will contribute gradients back to the postprocessing parameters in the backward propagation, effectively giving the postprocessor supervision in the form of camera extrinsics."*. Can authors support/prove this claim?

review_title: Official Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper explores the problem of visual question answering from another perspective. Similar to VQA, a system is provided with a scene and a question. However, the difference is that the question needs to be answered from a viewpoint different from the one provided. Hence, the system needs to perform “mental rotation”. The paper creates a new dataset called CLEVR Mental Rotation Tests which is based on the prior CLEVR dataset. The paper also studies the efficacy of various supervised and self-supervised models on the proposed dataset.

#### Strong points:
- The problem of asking questions related to “mental rotation” seems interesting.
- The paper shows that contrastive pre-training could be useful for the task, which is an interesting result.

#### Weak points: 
- Although the problem seems interesting, I am unclear about the usefulness of the proposed dataset. The paper says that “many computer vision systems could benefit from neural architectures that demonstrate good performance for more targeted mental rotation tasks.” To justify this claim it gives the following example, “given the camera viewpoint of a (blind) person crossing the road, can we infer if each of the drivers of the cars at an intersection can see this blind person crossing the street?”. This could potentially be a useful scenario, however, the dataset proposed is different from the example as the camera viewpoint is provided as part of the input and not inferred from the question. The paper does not provide justification or evidence of how the current setup (i.e. with camera viewpoint) is useful. In particular, it would be nice if the paper could further explain how the current setup is better than solving the “view rendering” and VQA problems separately.

- The dataset seems to be too simple for the mental rotation tests. It is unclear if in the future the dataset would be useful in distinguishing which models are better. As the paper shows that “2D baseline without camera conditioning” already achieves 70% accuracy. As far as I could understand, even without knowing which view to look at, a model could achieve 70% accuracy indicating that there is a lot of bias in the dataset. Moreover, simply adding camera embedding with the question to a 2D baseline (Table 1, 2D FILM with camera), already performs close to the best 3D model and upper bound. (Please clarify if my understanding is wrong.)

- The paper is poorly organized and hard to follow. For example, one of the contributions of the work is the CLEVR-MTR dataset, however, there is no clear section in the main paper describing the details of how the dataset. Instead, the information about the dataset is scattered in the introduction and related work. Another example is that the paper moves into talking about the method (Section 2) without defining the task concretely. It is only from the figure that one notices that the camera viewpoint is part of the input. From the examples provided in the introduction, the reader is under the impression that the camera viewpoint has to be inferred from the question itself. Similarly, it's hard to parse what the training signal for each baseline is. Does a baseline use the rendered image from the other view during training?
 
#### Minor Comments:
- The figures and tables are interspersed with the text making the paper harder to read. It might be better to place the figures and tables at the end of the beginning of the page so that the captions are separated from the main text.
- Many equations like some parts of equation 1 and equation 2 might not be necessary as they don’t seem to contribute to understanding the paper. In many places, it seems like a simple intuitive explanation would be sufficient.
- Similarly, Figure 3 might not be necessary.
- The figures are unclear and hard to understand. For example, is the canonical viewpoint part of the input? If not, Figure 2 and Figure 4 could be changed to make it more clear.
- Is this line correct, “If we add camera conditioning via FILM (that is, appending the camera embedding to the GRU’s embedding) then we achieve a much greater accuracy of 69.60 ± 0.09.” Should its value be 83.68 ± 1.21 as indicated in the table?
 
#### Overall Recommendation:
Although the problem could potentially be useful, the current dataset seems to be not so useful and over-simplified. Moreover, I found the paper not well-organized and hard to understand even after multiple reads. I feel the paper can be improved a lot and hence recommend rejection for the current version.

#### Post Rebuttal
(Copying from the discussion below)

I would like to thank the author(s) for their response. After going over them, I am still not very confident about the paper would stick to my initial assessment. Following are my primary concerns:

"We note that there is a distinction between wanting to see something from another point of view, versus wanting to answer a question from another point of view. The former is where re-rendering is appropriate, but we do not make the claim that this alternative (view rendering + VQA) performs better or worse empirically."

I understand the distinction. But the issue still remains. Why is the out-of-the-box "view rendering + VQA" solution insufficient? Is there any empirical justification for it? If not its hard to see the value in the current setup. A potential way to address this could be to run a simple out-of-the-box "view rendering + VQA" baseline.

"(2) R3 and R4’s concern about camera information being provided to the model and its potential infeasibility in practice: In real world settings, camera rigs can and do have knowledge about where they are situated in the world, for instance using SLAM or GPS coordinates. In that case, it is not unreasonable for e.g. an autonomous vehicle to answer queries by performing rotations and/or translations of its current viewpoint."

The concern was not about the viewpoint of the observer but the new viewpoint from which the question has to be answered. Also, the location of the new viewpoint need not be converted into float and appended to the question. It could be expressed in natural language. For example "viewpoint of the driver in the other car" like in the example provided by the paper. In the current setup the information about this viewpoint is provided in terms of exact coordinates, which makes the setup less interesting and not so practical.

Although the authors improved some of the figures, the latest version of the paper does not seem to address other clarity concerns like a clear section for the dataset; organization of text and figures; removing unnecessary equations
review_title: Official Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

The paper studies visual question answering focusing on answering questions in a reference image of a different viewpoint. They propose a new dataset CLEVR-MRT drawing motivation from the well-known visual reasoning dataset CLEVR to illustrate the idea in which they have full control of the changes of viewpoints in an image. They then propose to use a volumetric encoder to represent 3D image features of an image via either 2D-to-3D projection or a contrastive-based encoder and further adapt an existing method (FiLM) to handle 3D tensors. Experiments on the CLEVR-MRT show that the use of the 2D features and 3D features of an image is complementary to each other.

Comments (Technical, Major Flaws of this paper): 

(1) The idea of addressing VQA in multi-view settings is reasonable but it is not entirely new. My main concern is at the limitations of a synthetic dataset in a controlled setting where the relations between objects are limited compared to real data. In addition, I believe that given enough such generated question-answer pairs with associated programs, models may possibly learn to decode the generation procedure under the hood instead of learning the actual semantic meanings of languages and the relations between objects.

(2) Since there are no statistics about the newly introduced dataset, it is hard to judge the empirical results in the paper. As pointed out by many previous studies (e.g. Hudson, D.A., et al., 2019; Le, T.M., et al., 2020), models' performance seems to converge on CLEVR given enough training data. Having that said, existing methods easily fail if we reduce the number of training instances. As for the CLEVR-MRT, even without any information about the viewpoints, the baseline models could achieve more than 70% accuracy on the proposed dataset. It seems that the dataset is too simple that the model could have good performance without knowing the camera parameters. This leads to concerns about the validity of the proposed dataset. Please address these points.
References:
 - Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2020). Dynamic Language Binding in Relational Visual Reasoning. In IJCAI 2020.
 - Hudson, D. A., & Manning, C. D. (2018). Compositional attention networks for machine reasoning. In ICLR 2019.

(3) For those who are not familiar with the CLEVR dataset, briefly explaining the procedure to generate the dataset and its variants might be helpful. 

(4) Given a question related to the object positions, there may exist many different views that provide the same answer. Let's take the question "How many green spheres to the left of the shiny gold thing?" in Figure 4 as an example. There are many views in the scene that provide the correct answer "1" for this question.   Without restricting the variance of the camera view (as in [1]), how can we ensure the model to infer the correct viewpoint?

Some typos:
- (1): 1. Introduction: We use the the Compositional -> We use the Compositional
- (2): 2.1 FILM Baseline: the viewpoint and canonical view is the same thing -> the viewpoint and the canonical view are the same thing
- (3): Figure 2: The dotted border on the ResNet-101 indicate -> The dotted border on the ResNet-101 indicates
- (4): Conclusion: In the case of an autonomous vehicles -> In the case of autonomous vehicles
review_title: The idea is not entirely new and details are missing 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros: 

1. The paper presents an interesting idea to learn mental rotations using a variation of the CLEVR-VQA dataset. The contributions are - the creation of this synthetic CLEVR-Mental Rotation dataset for targeting this problem and a model that encodes questions and viewpoint information to produce answers via FiLM based encoders and 3D volume encoder. 
2. The results in Table 1 and Table 2 show improvements with respect to the baselines using their final model but there is still some concern in the improvements on their ablations. 
3. The paper is well written and easy to understand. 

Cons: 
1. The motivation of why we need to learn mental rotations is not very clearly expressed, the practical examples given in the introduction are not sufficient. Does the model really learn these mental rotations from a simple spatial VQA task? This should be evaluated in the experiments either using activation maps or by visualizing intermediate 3D encodings. 
2. Is the model trained on all views for a single question-view pair or any one random viewpoint is sampled during mini batch training ? Does the rotation of a scene done over the complete 360 degree ? How do you decide how much to rotate to generate a viewpoint ?
3. The self supervised learning of 3D volumes is an interesting idea, but it's use case in this particular problem is very weakly motivated both in experiments and theory. Why is this method better than the method discussed in Section 2.2.1? What is 3D data augmentation and how is it different from 2D data augmentation? 
4. There is a large variance in some experiments in Table 1. Is it due to the camera transformation embedding? It will be good to discuss the reasons why this is in Table 1 and not in Table 2.  
5. Although the models developed are used in a very different problem setting with minor contributions, still a large part of the methods seem to be derived from the literature. 
6. The final results in Table 2 though argued are better due to small variance but more extensive experiments need to be performed to show the benefits of the self-supervised pre-training over the traditional encoder approach.
Minor: What is the value of t (tau) used in Eq 3 ? In Table 2 it shows 1.0, but in the text it’s discussed as 0.1. Is this a typo or both of them are supposed to be different, if yes why ?

review_title: The authors propose to learn mental rotations via a synthetic CLEVR-Mental Rotation dataset based on VQA

Meta-review: This paper was reviewed by 4 experts in the field. The reviewers raised their concerns on lack of novelty, unconvincing experiment, and the presentation of this paper, While the paper clearly has merit, the decision is not to recommend acceptance. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.

Decision: Reject