Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This article deals with the problem of partially labeled dataset: if some entities are missing, how SOTA approaches are going to behave? To answer this question, the authors degrade CoNLL classical dataset by masking a pourcentage of the labeled data. Then, they wonder which part of the missing performance is due to the lack of labels and which part is due to the incorrect labelling of discarded supervision.
The experiments are well explained and interesting on synthetic dataset. Then the authors propose a new cross entropy loss to test their hypothesis on real data by sampling high confidense negative samples as ground truth. It is a way of performing distillation on the model using negative sampling.

Consistant & relevant work that deserves to be publised in ICLR.

* We wonder what would give a classical distillation process on this task. [even if relevant comparison are made with results from the litterature]

* Given the architecture, we wonder what is the detailed learning procedure: it seems clear that the network is first trained on the real ground truth and then refined using the distillation loss. Section 3.1 is a little bit short on this point.

* Regarding the model, equation (4) is not discussed nor analysed.

* The approach is rather simple but elegant.
review_title: a good idea supported by consistent experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper investigates the unlabeled entity problem, which is generally observed in the manual annotation setting and distant supervision as well. The unlabeled problem is important and some existing works focus on solving the problems using partial CRF setting or data selector. The main observation of this paper lies in two aspects: 1) comparison between the reduction of annotated entities or treating unlabeled entities as negative instances.  Most interestingly, the authors show the observed difference between pre-trained language models and LSTM-based models.  Based on the observations, they propose a general approach to eliminate the misguidance brought by unlabeled entities and such a simple design shows good performances.  

The Paper is overall well written and easy to follow. But  I still have a few questions and want to get answers from authors. 

Questions:

1) The first question is about 4.2 Training via Negative sampling on page 5. I am not quite sure about the procedure. Negative instance candidates are randomly selected from original sentences.  You use \hat{y}, which is a subset of randomly selected span to replace a missed entity set defined in Eq. (2)?

2) Could you expand more about Equation 8 to add more details?

3) The unlabeled entity problem is most serious in the distant supervision setting. However, the distant supervision setting suffers from entity ambiguation and unlabeled entity problem simultaneously.  How do you think your design to tackle entity ambiguation problem? Moreover, in the distant supervision experiment in Table  3, how will you model compare with other distant supervision models like AutoNER?


review_title: Generally good, a few questions 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper focuses on the unlabeled entity problem in NER, where the entities of a sentence are incomplete annotated. Since some entities may not be annotated, the performance of models can be degraded. This paper analyzes the performance degradation by evaluating synthetic datasets and finds that all the unlabeled entities are treated as negative instances is the main factor of the performance degradation. To alleviate the performance degradation, this paper proposes a negative sampling approach that considers only a small subset of unlabeled entities in order to reduce the impacts of unlabeled entities. The experimental results show that the proposed method achieves better performances compared to previous studies on real-world datasets and achieves competitive performances compared to the state-of-the-art methods on well-annotated datasets.

Strong points:
+ The analysis using two proposed metrics to find causes of the performance degradation of the unlabeled entity problem is quite interesting.
+ The idea of using the negative sampling under the data imbalance situation, especially in NER, seems reasonable.
+ The paper is well-written and easy to follow. It provides comprehensive experiments on several datasets.

Concerns: 
1. The key concern about this paper is the missing of some references. There are some previous studies related to the unlabeled entity problem. Particularity, this problem is quite related to or almost the same as the “Incomplete annotations” at [1] or “Partially annotated training data” at [2]. In addition, the unlabeled entity problem has a little relevance to the data imbalances [3] if we assume that all unlabeled spans are negative instances. Therefore, the proposed method should be compared with these previous studies experimentally and/or theoretically.
[1] Jie et al., Better Modeling of Incomplete Annotations for Named Entity Recognition, NAACL 2019.
[2] Mayhew et al., Named Entity Recognition with Partially Annotated Training Data, CONLL 2019.
[3] Li et al., Dice Loss for Data-imbalanced NLP Tasks, ACL 2020.
2. In the experiments on the real-word dataset, the proposed method only compared with weak baselines. The reviewer suggests the authors to evaluate with the state-of-the-art NER methods such as BERT-MRC or BERT-Biaffine Model.
3. At the inference time, every span should be input to the MLP layer to obtain the predicted score. That is, the proposed method takes more time (O(n^2)) to infer new sentences compared to other methods (O(n)). Furthermore, this paper uses heuristic(s) when the spans of inferred entities intersects. However, there is no discussion about them.

Minor comments:
1. In the section 3.2, the figures and the contents about the figures are inconsistent. That is, the middle parts of the figures are for the change in misguidance rates and the right-hand ones are for the change of the erosion rates.
2. For batching with sampling, it would be better to provide more details about it.
3. It would be nice to show a model convergence or loss convergence graph because this paper uses random sampling.
4. It would be better to show the correlation in numbers between the f1 scores and the proposed rates.
review_title: A plausible approach but a weak comparison

final_score: rating: 7: Good paper, accept
review_content: This paper conducted an empirical analysis on the unlabeled entity problem in the NER task. It concluded that there are two reasons to affect the NER model's performance:  the reduction of annotated entities, and treating unlabeled entities as negative instances. Experiments showed that the latter reason gave a much more negative impact on the NER models. 

This empirical study was conducted on the synthetic datasets which were extracted from two English NER datasets using negative sampling. The way of constructing synthetic datasets should be explained clearly (Section 3.1). It is not clear how does the negative sampling applied on the dataset (in sentence level? entity level? token level?). 
review_title: Practical, interesting results

Meta-review: This paper studies the unlabeled entity problem in NER. Specifically, performance degradation in training of NER models due to unlabeled entities. It analyzes the reason through evaluation on synthetic datasets and finds that it is due to the fact that all the unlabeled entities are treated as negative examples. To cope with the problem, it proposes a negative sampling method which considers the use of only a small subset of unlabeled entities. Experimental results show that the proposed method achieves better performances than the baselines on real-world datasets and achieves competitive performances compared with the state-of-the-art methods on well-annotated datasets.

Pros
•	The paper is clearly written.
•	The proposed method appears to be technically sound.
•	Experimental results support the main claims.
•	The findings in the paper are useful for the field.

Cons
•	Novelty of the work might not be enough.

The authors have addressed some clarity and reference issues pointed out by the reviewers in the rebuttal.  Discussions have been made among the reviewers.

Decision: Accept (Poster)