Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper is aimed at tackling a general issue in NLP: Hard-negative training data (negative but very similar to positive) can easily confuse standard NLP model. To solve this problem, the authors first applied distant supervision technique to harvest hard-negative training examples and then transform the original task to a multi-task learning problem by splitting the original labels to positive, hard-negative, and easy-negative examples. The authors consider using 3 different objective functions: L1, the original cross entropy loss; L2, capturing the shared features in positive and hard-negative examples as regularizer of L1 by introducing a new label z; L3, a three-class classification objective using softmax.
This authors evaluted their approach on two tasks: Text Classification and Sequence Labeling. This implementation showed improvement of performance on both tasks.

Strenghts:
+ the paper proposes a reasonable way to try to improve accuracy by identifying hard-negative examples
+ the paper is well written, but it would benefit from another round of proofreading for grammar and clarity

Weaknesses:
- performance of the proposed method highly depends on labels of hard-negative examples. The paper lacks insight about a principled way to label such examples, the costs associated with such labeling, and impacts of the labeling quality on accuracy. The experiments are not making a convincing case that similar improvements could be obtained on a larger class of problems.
- The objective function L3 is not well justified.
- It would be important to see if the proposed method is also beneficial with the state of the art neural networks on the two applications. 
- Table 3 (text classification result) does not list baselines.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors propose a novel approach to leverage Distant Supervision for discriminating between positive examples and "negative examples that share salient features with the positive ones." In spite of its simplicity, the method appears to be quite promising.

The main feedback for the authors is to describe "early & in detail" the distant supervision techniques used in the experiments. The paper would be greatly improved by adding:
- an intuitive paragraph in the intro that explains a concrete example of DS high level, but with enough details for the reader to grasp the idea)
- adding a new section right after related work (and before the current "3. Models") in which you present in great detail (and with concrete & complete examples) the two main DS techniques that are used in the experiments; with that solid understanding 


Other comments:
- for sake of simplicity & understand-ability, you should avoid the use of the term "soften positive examples"  in the abstract
- avoid using the term "unbelievable" (one in abstract & twice in intro"
- the last paragraph before "Conclusion" seems to refer to an earlier version of Figure 4, which, in its current form, does NOT 
have S(pos,E) or the word "interesting"
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes to improve performance of NLP tasks by focusing on negative examples that are similar to positive examples (e.g. hard negatives). This is achieved by regularizing the model using extra output classifiers trained to classify examples into up to three classes: positive, negative-easy, and negative-hard. Since those labels are not provided in the original data, examples are classified using heuristics (e.g. negative examples that contain a lot of features predictive of a positive class will be considered as hard-negative examples), which are used to provide distant supervision. This general approach is evaluated on phrase classification tasks, one information extraction task, and one MRCQA task.

Although the proposed approach is interesting, this paper has several weaknesses (i) the method is not sufficiently justified or analyzed; (ii) there are missing links with previous work (notably on domain adversarial training); (iii) experimental setting is rather weak.

1) About the justification of the approach:
1.1) I feel like the proposed are not intuitively justified enough. They point out that "L_2 can be thought as an objective to capture the shared features in positive examples and hard-negative examples". Why would that be good from an intuitive perspective ?
1.2) L_3 is forcing the model to group all the hard-negative examples together. Do you have an intuition why would that be useful ?
1.3) What happens if the model overfits the hard negative examples in the training set ? This would mean that it has captured some features that can distinguish positive / negative label. Why would L_3 help in that case ?

2) About related approaches:
2.1) How does this method relate to domain adversarial training applied to positive and hard-negatives and adversarial examples in general ?
2.2) Would similar performance be obtained by virtual adversarial training for example ?

3) About the experimental setting:
3.1) The performance reported is well below the use of recent work on these datasets and recent models such as BERT. Would these improvements carry over to bigger architectures ?
3.2) In SST, the paper say they use BERT large, but the baseline performance (81.5) is well below BERT large performance in the original paper (94.9, https://arxiv.org/pdf/1810.04805.pdf). Why the mismatch ?
3.3) What's the proportion of hard-negative examples mined for training and test set ? While the heuristics used seem reasonable, without those numbers, it is impossible to know if the heuristics truly predict hard-negative examples.
3.4) Does the performance gain comes from better predicting hard-negative examples in the test set ? One could analyze the performance per error type (i.e. true positive, false negative, false positive (easy), false positive (hard)) with the baseline model and of the various proposed regularizing tasks (e.g. L_2 and L_3, both in training and test).
3.5) The heuristics are used to pick what should be adversarial examples, but there is no mentions of this concept in the text. Oversampling those adversarial examples could, potentially, improve the performance of the baseline model. It would be interesting to try this.
3.6) If possible, it would be good to add standard deviation to the results obtained running multiple runs.
3.7) The visualization sub-section is anecdotal and not especially illuminating, and its text seems to refer to a different example than the figures ("interesting" is not in the figures.)

Minor points:

- Section 3 (Models) may be made shorter, the models used are utterly simple. This could free up space for more experiments.
- In the tables, simply adding the name of the used model to the "L1" rows would be clearer.
- The description of the pipelined results in Section 4.1 does not match the results shown in the table.
- The citations are not well integrated with the text (\citep vs \cite), and the formatting of CRF changes in the last paragraph of Section 3.2.

review_title: Official Blind Review #1

Meta-review: This paper proposes a way to handle the hard-negative examples (those very close to positive ones) in  NLP, using a distant supervision approach that serves as a regularization.   The paper addresses an important issue and is well written; however, reviewers pointed put several concerns, including testing the approach on the state-of-art neural nets, and making experiments more convincing by testing on larger problems.
 

Decision: Reject