Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper describes a new model architecture for machine reasoning. In contrast
to previous approaches that explicitly predict a question-specific module
network layout, the current paper introduces a monolithic feedforward network
with iterated rounds of attention and memory. On a few variants of the CLEVR
dataset, it outperforms both discrete modular approaches, existing iterated
attention models, and the conditional-normalization-based FiLM model. 

So many models are close to perfect accuracy on the standard CLEVR dataset that
I'm not sure how interesting these results are. In this respect I think the
current paper's results on CLEVR-Humans and smaller fractions of synthetic CLEVR
are much more exciting.

On the whole I think this is a strong paper. I have two main concerns. The
largest is that this paper offers very little in the way of analysis. The model
is structurally quite similar to a stacked attention network or a particular
fixed arrangement of attentive N2NMN modules, and it's not at all clear based on
the limited set of experimental results where the improvements are actually
coming from. It's also possible that many of the proposed changes are
complementary to NMN- or CBN-type models, and it would be nice to know if this
is the case.

Secondarily, the paper asserts that "our architecture can handle
datasets more diverse than CLEVR", but runs no experiments to validate this. It
seems like once all the pieces are in place it should be very easy to get
numbers on VQA or even a more interesting synthetic dataset like NLVR.

Based on a sibling comment, it seems that there may also be some problems with
the comparison to FiLM, and I would like to see this addressed.

On the whole, the results are probably strong enough on their own to justify
admitting this paper. But I will become much more enthusiastic about if if the
authors can provide results on other datasets (even if they're not
state-of-the-art!) as well as evidence for the following:

1. Does the control mechanism attend to reasonable parts of the sentence?

Here it's probably enough to generate a bunch of examples showing sentence
attentions evolving over time.

2. Do these induce reasonable attentions over regions of the image?

Again, examples are fine.

3. Do the self-attention and gating mechanisms recover the right structure?

In addition to examples, here I think there are some useful qualitative
measures. It should be possible to extract reasonable discretized "reasoning
maps" by running MST or just thesholding on the "edge weights" induced by
attention and gating. Having extracted these from a bunch of examples, you can
compare them to the structural properties of the ground-truth CLEVR network
layouts by plotting a comparison of sizes, branching factors, etc.

4. More on the left side of the dataset size / accuracy curve. What happens if
   you only give the model 7000 examples? 700? 70?

Fussy typographical notes:

- This paper makes use of a lot of multi-letter names in mathmode. These are
  currently written like $KB$, which looks bad, and should instead be
  $\mathit{KB}$.

- Variables with both superscripts and subscripts have the superscripts pushed
  off to the right; I think you're writing these like $b_5 ^d$ but they should
  just be $b_5^d$ (no space).

- Number equations and then don't bother carrying subscripts like $W_3$, $W_4$
  around across different parts of the model---this isn't helpful.

- The superscripts indicating the dimensions of parameter matrices and vectors
  are quite helpful, but don't seem to be explained anywhere in the text. I
  think the notation $W^{(d \times d)}$ is more standard than $W^{d, d}$.

- Put the cell diagrams right next to the body text that describes them (maybe even
  inline, rather than in figures). It's annoying to flip back and forth.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a recurrent neural network for visual question answering. The recurrent neural network is equipped with a carefully designed recurrent unit called MAC (Memory, Attention and Control) cell, which encourages sequential reasoning by restraining interaction between inputs and its hidden states. The proposed model shows the state-of-the-art performance on CLEVR and CLEVR-Humans dataset, which are standard benchmarks for visual reasoning problem. Additional experiments with limited training data shows the data efficiency of the model, which supports its strong generalization ability.

The proposed model in this paper is designed with reasonable motivations and shows strong experimental results in terms of overall accuracy and the data efficiency. However, an issue in the writing, usage of external component and lack of experimental justification of the design choices hinder the clear understanding of the proposed model.

An issue in the writing
Overall, the paper is well written and easy to understand, but Section 3.2.3 (The Write Unit) has contradictory statements about their implementation. Specifically, they proposed three different ways to update the memory (simple update, self attention and memory gate), but it is not clear which method is used in the end.

Usage of external component
The proposed model uses pretrained word vectors called GloVE, which has boosted the performance on visual question answering. This experimental setting makes fair comparison with the previous works difficult as the pre-trained word vectors are not used for the previous works. To isolate the strength of the proposed reasoning module, I ask to provide experiments without pretrained word vectors.

Lack of experimental justification of the design choices
The proposed recurrent unit contains various design choices such as separation of three different units (control unit, read unit and memory unit), attention based input processing and different memory updates stem from different motivations. However, these design choices are not justified well because there is neither ablation study nor visualization of internal states. Any analysis or empirical study on these design choices is necessary to understand the characteristics of the model. Here, I suggest to provide few visualizations of attention weights and ablation study that could support indispensability of the design choices.

review_title: Looks good but needs clarification

final_score: rating: 7: Good paper, accept
review_content: Summary: 
The paper presents a new model called Compositional Attention Networks (CAN) for visual reasoning. The complete model consists of an input unit, a sequence of the proposed Memory, Attention and Composition (MAC) cell, and an output unit. Experiments on CLEVR dataset shows that the proposed model outperforms previous models.

Strengths: 
— The idea of building a compositional model for visual reasoning and visual question answering makes a lot of sense, and, I think, is the correct direction to go forward in these fields.
— The proposed model outperforms existing models pushing the state-of-the-art.
— The proposed model is computationally cheaper and generalizes well with less training data as compared to existing models.
— The proposed model has been described in detail in the paper.

Weaknesses: 
— Given that the performance of state-on-art on CLEVR dataset is already very high ( <5% error) and the performance numbers of the proposed model are not very far from the previous models, it is very important to report the variance in accuracies along with the mean accuracies to determine if the performance of the proposed model is statistically significantly better than the previous models.
— It is not clear which part of the proposed model leads to how much improvement in performance. Ablations studies are needed to justify the motivations for each of the components of the proposed model.
— Analysis of qualitative results (including attention maps, gate values, etc.) is needed to justify if the model is actually doing what the authors think it should do. For example, the authors mention an example on page 6 at the end of Section 3.2.2, but do not justify if this is actually what the model is doing.
— Why is it necessary to use both question and memory information to answer the question even when the question was already used to compute the memory information? I would think that including the question information helps in learning the language priors in the dataset. Have the authors looked at some qualitative examples where the model which only uses memory information gives an incorrect answer but adding the question information results in a correct answer?
— Details such as using Glove word embeddings are important and can affect the performance of models significantly. Therefore, they should be clearly mentioned in the main paper while comparing with other models which do not use them.
— The comparisons of number of epochs required for training and the training time need fixed batch sizes and CPU/GPU configurations. Is that true? These should be reported in this section.
— The authors claim that their model is robust to linguistic variations and diverse vocabulary, by which I am guessing they are referring to experiments on CLEVR-Humans dataset. What is there in the architecture of the proposed model which provides this ability? If it is the Glove vectors, it should be clearly mentioned since any other model using Glove vectors should have this ability.
— On page 6, second paragraph, the authors mention that there are cases which necessitate the model to ignore current memories. Can the authors show some qualitative examples for such cases?
— In the intro, the authors claim that their proposed cell encourages transparency. But, the design of their cell doesn’t seem to do so, nor it is justified in the paper.

Overall: The performance reported in the paper is impressive and outperforms previous state-of-the-art, but without proper statistical significance analysis of performance, ablation studies, analysis of various attention maps, memory gates, etc. and qualitative results, I am not sure if this work would be directly useful for the research community.
review_title: Needs more analysis and qualitative evaluation

Meta-review: PROS:
1. Good results on CLEVER datasets
2. Writing is clear
3. The MAC unit is novel and interesting.
4. Ablation experiments are helpful

CONS:
The authors overstate the degree to which they are doing "sound" and "transparent" reasoning.  In particular statements such as "Most neural networks are essentially very large correlation engines that will hone in on any statistical, potentially spurious pattern that allows them to model the observed data more accurately. In contrast, we seek to create a model structure that requires combining sound inference steps to solve a problem instance." I think are not supported.  As far as I can tell, the authors' do not show that the steps of these solutions are really doing inference in any sound way

I also found the interpretability section to be a bit unconvincing.  The reviewers and I discussed this and there was some attempt to assess what the operations were actually doing but it is not clear how the language and the image attention are linked.

I wonder whether the learned control activations are abstract and re-used across problems the way that the accompanying functional solution's primitives are.  Have you looked at how similar the controls are across problems which are identical except for a different choice of attributes?  To me, one of the hallmarks of a truly "compositional" solution is one in which the pieces are re-used across problems, not just that there is some sequence of explicit control activations used to solve each individual problem.
Decision: Accept (Poster)