Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
The paper proposes a novel model architecture for the visual question answering task in the CLEVR dataset. The main novelty of the proposed model lies in its problem specific differentiable forking mechanism that is designed to encode complex assumptions about the data structure in the given problem. The proposed model is also applied on the task of solving Reverse Polish Notation (RPN) expression. On the CLEVR dataset, the proposed model beats the state of the art by 0.7% with ~5 times fever learning parameters and in about ~1/2 as many epochs. For the RPN task, the proposed model beats an LSTM baseline by 0.07 in terms of L1 error with ~10 times fewer parameters. 
	
Strengths:
1.	The proposed model is novel and interesting.
2.	The performance of the proposed model on the “Count” questions in the CLEVR dataset is especially better than existing models and is worth noting.
3.	The discussion on the tradeoff between tacking difficult problems and using the knowledge of program structure is engaging. 

Weaknesses:
1.	The paper writing about the model architecture can be improved. As of now, it is not easy to follow.
2.	The motivation behind the proposed model for the CLEVR task has been explained example of one type of questions – “How many objects are red or spheres?”. It is not clear how the proposed model is better than existing models (in terms of model architecture) for other types of questions.
3.	The motivation behind the proposed model for the RPN task is not strong enough. It is not clear why is machine learning needed for the RPN task? Is the idea that we do not want to use domain knowledge about which symbols correspond to operations vs. which correspond to numbers? Or is there more to it?
4.	The proposed model needs to use the information about program structure. It would be good if authors could comment on how can the proposed model be used to answer natural language questions about images, such as, those in the VQA dataset (Antol et al., ICCV 2015).
5.	The paper does not have any qualitative examples for either of the two tasks. Qualitative examples of successes and failures would be helpful to better position proposed model against existing models.

Overall: The experimental results look good, however, the proposed model needs to be better motivated. The paper writing, especially the “DDR Architecture” section needs improvement to make it easy to follow. A discussion on how the existing model can be adapted for natural language questions would be good to add.

review_title: Motivation behind the proposed model needs improvement, paper writing about model architecture not easy to follow.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
The paper presents a generic dynamic architecture for CLEVR VQA and Reverse Polish notation problems. Experiments on CLEVR show that the proposed model DDRprog outperforms existing models, but it requires explicit program supervision. The proposed architecture for RPN, called DDRstack outperforms an LSTM baseline.

Strengths:
— For CLEVR VQA task, the proposed model outperforms the state-of-the-art with significantly less number of parameters.
— For RPN task, the proposed model outperforms baseline LSTM model by a large margin.

Weaknesses:
— The paper doesn’t describe the model clearly. After reading the paper, it’s not clear to me what the components of the model are, what each of them take as input and produce as output, what these modules do and how they are combined. I would recommend to restructure the paper to clearly mention each of the components, describe them individually and then explain how they are combined for both cases - DDRprog and DDRstack. 
— Is the “fork” module the main contribution of the paper? If so, at least this should be described in detail. So, if no fork module is required for a question, the model architecture is effectively same as IEP?
— Machine accuracy is already par with human accuracy on CLEVR and very close to 100%. Why is this problem still important? 
— Given that the performance of state-on-art on CLEVR dataset is already very high ( <5% error) and the performance numbers of the proposed model are not very far from the previous models, it is very important to report the variance in accuracies along with the mean accuracies to determine if the performance of the proposed model is statistically significantly better than the previous models or not.
— In Figure 4, why are the LSTM32/128 curves different for Length 10 and Length 30 till subproblem index 10? They are both trained on the same training data, only test data is of different length and ideally both models should achieve similar accuracy for the first 10 subproblems (same trend as DDRstack).
— Why is DDRstack not compared to StackRNN?
— Can the authors provide training time comparison of their model and other/baseline models? Because that is more important than the number of epochs required in training.
— There are only 3 curves (instead of 4) in Figure 3.
— In a number of places, the authors are referring to left and right program branches. What are they? These names have not being defined formally in the paper.

Overall:
I think the research work in the paper is interesting and significant, but given the current presentation and level of detail in the paper, I don’t think it will be helpful for the research community. By proper restructuring of paper and adding more details, the paper can be converted to a solid submission.
review_title: Presentation of the approach is not clear

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Summary: This paper leverages an explicit program format and proposes a stack based RNN to solve question answering. The paper shows state-of-the art performance on the CLEVR dataset.

Clarity:
- The description of the model is vague: I have to looking into appendix on what are the Cell and Controller function.
- The authors should also improve the intro and related work section. Currently there is a sudden jump between deep learning and the problem of interest. Need to expand the related work section to go over more literature on structured RNN.

Pros:
- The model is fairly easy to understand and it achieves state-of-the-art performance on CLEVR.
- The model fuses text and image features in a single model.

Cons:
- This paper doesn’t mention recursive NN (Socher et al., 2011) and Tree RNN (Tai et al., 2015). I think they have fairly similar structure, at least conceptually, the stack RNN can be thought as a tree parser. And since the push/pop operations are static (based on the inputs), it’s no more different than encoding the question structure in the tree edges.
- The IEP (Cells) module (Johnson et al., 2017) seems to do all the heavy-lifting in my opinion. That’s why the proposed method only uses 9M parameters. The task isn’t very challenging to learn because all the stack operations are already given. Table 1 should note clearly which methods use problem specific parsing information to train and which use raw text. Based on my understanding of FiLM at least, they use raw words instead of groundtruth parse trees. So it’s not very surprising that the proposed method can outperform FiLM (by a little bit).
- I don’t fully agree with the title - the stack operations are not differentiable. So whatever network that outputs the stack operation cannot be jointly learned with gradients. This is based on the if-else statements I see in Algorithm 1.

Conclusion:
- Since the novelty is limited and it requires explicit program supervision, and the performance is only on par with the state-of-the-art (FiLM), I am not convinced that this paper brings enough contribution to be accepted. Weak reject.

References:
- Socher, R., Lin, C., Ng, A.Y., Manning, C.D. Parsing Natural Scenes and Natural Language with Recursive Neural Networks. The 28th International Conference on Machine Learning (ICML 2011).
- Tai, K.S., Socher, R., Manning C.D. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. The 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015).
review_title: Good performance but requires explicit program supervision to slightly beat the state of the art

Meta-review: The reviewers generally agree that the DDRprog method is both novel and interesting, while also seeing merit in outperformance of related methods in the empirical results. However, There were a lot of complaints about the writing quality, the clarity of the exposition, and unclear motivation of some of the work.  The reviewers also noted insufficient comparisons and discussions regarding relevant prior art, including recursive NNs, Tree RNNs, IEP, etc.  While the authors have made substantial revisions to the manuscript, with several additional pages of exposition, reviewers have not raised their scores or confidence in response.
Decision: Reject