Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This works applies neural module network to reading comprehension that requires symbolic reasoning. There are two main contributions: (1) the authors designed a set of differentiable neural modules for different operations (for example, arithmetics, sorting, and counting) that is required to perform reasoning over a paragraph of text. These modules can be compositionally combined to perform complex reasoning. And the parameters of each module (which can be viewed as executor of each operation) are learned jointly with the parser that generates thee program composed of those modules. (2) To overcome the challenge of weak supervision, the authors proposed to use auxiliary loss (information extraction loss, parser supervision, intermediate output supervision). The model is evaluated on a subset of DROP, and outperforms the state-of-the-art models. Ablation studies supported the importance of the auxiliary losses.

Strength:

(1) The problem of applying symbolic reasoning over text is important and very challenging. This work has explored a promising direction that applies NMN, which achieved good results in VQA, to QA tasks that requires reasoning, specifically, a subset of the DROP dataset.

(2) The result, although preliminary, seems promising. The design of the modules seems intuitive and the introduction of auxiliary tasks to alleviate the problem of weak supervision is well motivated and works reasonably well. 

I am leaning towards rejection because:

(1) The main concern is that the paper, in its current form, seems incomplete. It is understandable that the type of datasets that requires reasoning is not very common nowadays, so only DROP is used for evaluation. However, the current evaluation is only on a subset of DROP, which seems unsatisfying.  

The paper argues that "Our model possesses diverse but limited reasoning capability; hence, we try to automatically extract questions in the scope of our model based on their first n-gram". However, results on the full dataset seems necessary for evaluating the potential of NMN approach over text. Even if the result is negative, it is still good to know the cause of the failure. For example, does the difficulty come from unstable training or does it come from insufficient coverage of the modules. 

(2) There are several modules introduced in the paper, but there isn't much analysis of them during the experiments. For example, what are some good and bad samples that uses each type of operations. 

(3) Since the modules are learned jointly with the parser, it is good to check whether the learned modules are indeed performing the intended operation instead of just adding more capacity to the model. For example, it might help to show a few examples that demonstrates the "compare-num-lt" is actually performing the comparisons. This can support the interpretability claim of the proposed model.  


Minor issues:

The complexities of some modules seem large. For example, "compare-num-lt" needs to enumerate all the pairs of numbers, which is quadratic. And the complexity of "find-max-num" depends on the choice of n, which could be large (although it is chosen to be 3 in this work). 

It is stated that "Our model performs significantly better than the baseline with less training data, showing the efficacy of explicitly modeling compositionality." However, the comparison with MTMSN using less training data seems a bit unfair since the proposed model is given more supervision (question parse supervision and intermediate module output supervision). Maybe a better argument is that by explicitly modeling compositionality, it is easier to add such extra supervisions than black box models like MTMSN. 

For "count", why is the attention scaled using values [1, 2, 5, 10] first?

In summary, I do like the main idea and the paper has merits, but it requires more evaluation and analysis to be accepted. I am willing to increase my score if more contents are added and I look forward to seeing it in a more complete form.

===================================

Update after author response:

Thanks for the clarification and adding the content, I have updated my score accordingly. However, I still believe the impact of this paper will be much larger if the evaluation can be more complete, e.g., evaluating over the full DROP dataset or even include some other datasets. In the current form, it looks borderline. 

Selecting a subset (~22.7% of DROP dataset) based on the design of the proposed model ("heuristically chosen based on their first n-gram such that they are covered by our designed modules"), and compare to other models, which can actually handle a broader set of questions, only on the selected subset seems incomplete and raises concerns about how generally applicable the proposed model is. For example, since the proposed model is handling some types of questions better, it would be good to show that it can be combined with other models to get a better overall result. 
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Paper Claims

The paper offers a new deep learning approach to symbolic reasoning over text. They propose using Neural Module Networks to perform explicit reasoning steps that are nevertheless differentiable. The process is separated into a semantic parsing of the question, and a resolution using MNMs. Auxiliary tasks improve performance and enable using a BERT pretrained model as a seed. The proposed model's performance surpasses previous SOTA on several question types. 

Decision

I'm in favor of accepting this paper because it tackles an extremely challenging and important problem in a novel and successful way. Reasoning has progressed more slowly than other NLP domains, and answering complex multi-reasoning-step questions is a good way to tackle the core of the problem. Regarding the approach, I find the mix of explicit reasoning steps, deep learning modeling, and even heuristics for some of the data preparation, powerful and effective. I see this as a useful step to advance the body of work in this space -- despite not being the desired end-result, it is nevertheless very instructive of what might work for at least some parts of the larger, AI-complete reasoning problem. Also, the paper is clear, well-written, and well-motivated.

Further details on Decision

I'm more than satisfied with the breadth of question types tackled here, and correspondingly the variety of modules. There's extensive, valuable work in designing these modules and making them work together. As the authors point out, it appears that other types of questions will require more intricate modules (or some other means), and I suspect that predetermined modules will not be what generalizes eventually. Nevertheless, the NMN approach taken here can be a stepping stone to further understanding how to tackle symbolic reasoning in a deep neural network. It will be instructive in designing a more ambitious, generalizable model. 

The auxiliary supervision tasks appear to be essential to obtaining the results, most notably the unsupervised loss for IE. I think this area has room for further improvement, but what is achieved in the paper is sufficient for publication. In particular, the writing of heuristics is a very specific solution targeting specific types of question and this will not scale to the full scope of natural language questions, and much less to all reasoning. Discussion of how to expand on them, scale them (automatic discovery, some other means?), etc. would be very welcome, as it is the main weakness of the paper.

I also think the methodology is sound and the results are obtained in a reasonable and mostly reproducible way.

This is truly great work that deserves to be published, discussed, and expanded upon.



review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a model and a training framework for question answering which requires compositional reasoning over  the input text, by building executable neural modules and training based on additional auxiliary supervision signals.

I really like this paper and the approach taken: tackling complex QA tasks is an important topic and current state-of-the-art methods rely heavily on lexical similarities as mentioned in the paper. I think learning differentiable programs is a good direction to address this space.

However in IMHO, the paper as it stands is premature for publication, the primary reason being the lack of strong experimental evidence that where the strength of this approach is compared to other methods compared. To be specific, the results in Table 2 are very close between MTMSN and the BERT-based model proposed and it's not clear if the difference is because (1) the model is generally better; (2) this is a subset of the dataset that this model performs better; (3) this is because of the additional supervision signals provided (e.g. the results of Fig 2a without the aux-sup is almost the same as MTMSN) and if we provided similar auxiliary supervision for other models they would equally do well; (4) due to lack of reporting variance and error-bars across runs we see a small increase which may not be significant; ...

Again, the paper is very interesting, but I don't think it's clear and thorough to experimentally prove that the overall approach is working better.

review_title: Official Blind Review #1

Meta-review: This work extends the previously introduced NMN for VQA for handling reasoning over text using symbolic reasoning components that can perform counting, sorting etc and can be compositionally combined. Moreover, to successfully train the model, the authors introduce a simple unsupervised auxiliary loss for training the IE components as well heuristically incorporating inductive biases in the behaviour on couple of components. All reviews agreed that this is a challenging topic and an interesting approach to symbolic reasoning over text. At the same time, reviewers did point that experiments are borderline thin, since the authors start with DROP and drop questions that are not particularly suited for symbolic reasoning, resulting in a substantially smaller dataset. Despite the fact that the experiments could probably be stronger, I’m recommending acceptance cause this topic is very interesting and this is a good paper to raise discussions at ICLR,
Decision: Accept (Poster)