Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: One recent work that comes to mind from ACL 2019: Leveraging Language Models for Commonsense Reasoning (Rajani et al 2019). In that work, they also have human annotators provide explanations (extending the CommonsenseQA dataset), and they show that by training with these explanations, inference is improved even without them. They also train a language model to generate the explanations, and they show that the language model generated explanations improve performance further at inference time. Seems like a reasonable reference to contrast the more structured approach to using explanations like Srivasta et al (2017), Hancock et al (2018), and this work.

I find the method summary beginning with "Human explanations are first converted to machine-actionable logical forms by a semantic parser" until the end of that paragraph to be unnecessary. Actually, as I read it, I find myself asking a lot of questions that get answered below. So I would prefer scrapping that method summary and just getting straight into the Explanation Parsing.

"indicates the the logical form matches" redundant 'the'

I can't find a definition for LF(E) anywhere, and yet LF(E) is present in many tables. I see that E is mentioned to be the explanations, but this is only in the caption of Table 2 even though the symbol is first used in the first paragraph of Section 4.1. I'm assuming LF logical forms applied directly to explanations, but this should be stated explicitly. Can you elaborate on why it is so dominant on precision In Table 6 and 7 of the Appendix, but low on recall, rather than just saying this is expected in Section 4.1? 

"For keyword query q, we directly encode it into vector z_q by bi-LSTM and attention." Can you elaborate on how z_q is constructed? Is it the final state of a forward LSTM concatenated with the final state of a backward LSTM?

Why is it so essential that you study the setting in which explanations are low-resource? I'm curious to see what would happen with more explanations.

I am surprised that none of the modules or compared methods include any architecture that use a Transformer or a form of contextualized word vectors (McCann et al 2017, Peters et al 2017, Devlin et al 2018). Is there an explanation for this?

I would prefer to see a larger suite of tested tasks given that each of these datasets is quite small. Would any other tasks from benchmarks like GLUE or SuperGLUE be amenable to your approach? The tasks you've chosen limit the scope of this work and leaves the question of whether it would generally improve across a greater variety of tasks, especially tasks that have seen significant improvement using new methods. Your claim would be much stronger if the explanations were shown to be helpful even to pretrained models like BERT when fine-tuned for a specific task. In particular, it would be interesting to see how the benefits of explanations vary for different kinds of tasks and for different training set sizes. 

"In the real world, a more realistic problem is that, with limited human-power, should we just annotate more labels or spend time explaining existing annotations."" I think that you mean "should we" makes it sound like this is a question, but there is no question mark, and it makes more sense as a statement. I propose "...human-power, there is a question of whether it is more valuable to ..."

"Explanations prove to be an very efficient form for data annotation." should be "a very" not "an very".

" a vital rule" should be "a vital role"

I find the first paragraph of Section 4.3 quite abstruse and bare in its explanation of the method used.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Labeling sentences for NLP requires a lot of human effort. In order to tackle this problem, the system proposed in this paper, called NMET, aims at labeling sentences by exploiting the explanations given by humans. First it converts the explanations into logical formulas. This logical formulas are then exploited for partitioning the dataset into two datasets: labeled dataset and unlabeled dataset. Then, NMET relaxes the logical formulas for labeling unlabeled examples by exploiting a neural architecture that uses four modules to deal with different types of predicates. 

The paper is pretty clear and well-written. It can be understood even by non-experts. The proposed approach seems technically sound and pretty novel. Moreover, the experimental results show that the proposed system achieves better performances than traditional label-only supervised models.
The only concern that I see in the paper is that the Deterministic Function Module is not explained very well. It is not clear to me its purpose.

[Minor]
Page 3
“the the logical form”

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: 
This paper explores using natural language explanations as auxiliary training data for NLP tasks. It first transforms natural language expressions into a logical form through CCG, and then use a neural module network architecture to label data instances. Experimental analyses are conducted on two tasks -- relation extraction and sentiment analysis, showing that the proposed approach outperforms previous work that incorporates explanations as training data.

Overall, the paper addresses an important issue of how to utilize human explanations as additional supervision source for NLP tasks and shows promising results. Hence, I believe the paper is above the acceptance threshold, and recommend for weak acceptance.

I had concerns on the cost of collecting human explanations, since they are non-trivial to collect. However, the authors provided convincing arguments regarding the data annotation cost in their response, so I do not think this is a major limitation of the method.

However, I would also like to note that the paper has a few limitations. The proposed is based on semantic parsing of the natural language explanations into logical forms and is therefore inherently limited by the representation power of symbolic and logical representations. In the two examples shown in Figure 1, the human explanations are very simple and have limited variety, so it is relatively easy to represent them in logical forms. However, on many NLP tasks (such as question answering), the human explanations (in natural language) may often be complicated and difficult to be represented in CCG. Therefore, it is unclear whether the proposed approach can be easily generalized to other tasks.

review_title: Official Blind Review #1

Meta-review: This paper proposing a framework for augmenting classification systems with explanations was very well received by two reviewers, and on reviewer labeling themselves as "perfectly neutral". I see no reason not to recommend acceptance.
Decision: Accept (Poster)