Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The authors presented an interesting paper which tries to solve a practically important question. Biomedical and tech industries usually hire human labelers for machine learning tasks, whose labels are usually noisy. Therefore, it is important to have a metric that can distinguish the performance of models with noisy labels.

In this paper, the authors proposed to measure the model performance based on the ratio of the discrepancy between the model prediction and the labeler, with the discrepancy between the labelers. The authors showed that in binary classification settings, their proposed metric can be reduced to simple to analyze quantities. The authors demonstrated the performance of their proposal in synthetic data as well in two real-world medical image datasets.

Overall, the paper is well motivated with a reasonable amount of novelty. The numerical experiments are well conducted, but I am not totally convinced by their results.

Detailed comments:
Significance: The paper is trying to address a practically important issue in machine learning. 
Novelty: The mathematical formulation of the metric consists of simple sums and averages, which in itself is not novel. However, the authors' choice of using such formulations to assess model performance is novel.
Clarity: The authors could simplify their notations, and could periodically remind the readers about the notations. For example, I was stuck by the sigma^2 notation in (6) (undefined) and the m notation in (7) defined in the past page in small texts. It would help me a lot if the authors reminded me the definition of those notations when they appeared.
Numerical Experiments: This is my biggest complain. I wasn't able to conclude that the authors proposal is better than majority voting based on Figures 1b and 2b. To me they look qualitatively the same. Is there any reason that the discrepancy ratio is superior to the majority vote? In addition, I didn't see whether the curves in Figures 1 and 2 are an average of numerous simulated samples. if not, the authors should use the average to mitigate randomness; and if the curves are averages, then the distribution of the metrics should be described, because I think ultimately, we don't care about whether the average black curve is above the average red curve, but the chance of black curve being above the red curve. 
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposed to evaluate model performance when the ground truth labels were not available and noisy labels provided by multiple uncertain experts were provided instead. The proposed evaluation metric, called discrepancy ratio, is defined as the ratio between the average model-annotator discrepancy and the average annotator-annotator discrepancy. It can be applied to compare 1) the relative performance of different models; and 2) the relative performance of average annotators and the model. Experimental results on the MNIST classification task showed the proposed metric can compare model performance under different noise levels and is robust w.r.t. the number of annotators and a single annotator's strictness. Experimental results on a real-world medical image classification was also presented. 

This work is well motivated. In real-world applications, such as medical image classifications, obtaining ground-truth labels is difficult or even infeasible, only multiple noisy labels are available to evaluate the model performance. Therefore, the setup of this problem seems novel (besides Rajpurkar et al. 2018).

In the experimental results on MNIST classification, for Naive Evaluation and Majority Vote, as the author pointed out, the gap between three models (model 1.0, model 0.9, model 0.8) becomes narrower when the label swap probability increases, indicating it's becoming more difficult to differentiate model improvements. However, this is also true for the proposed discrepancy ratio, as is shown in Figure 2(b). Therefore, the discrepancy ratio didn't overcome those "drawbacks" associated with these two baselines, correct? In other words, in the high-noise regime, it would be natural to observe narrower gaps between those models. 

In MNIST classification, the performances gap of those three models evaluated by Naive Evaluation and Majority Vote didn't decrease rapidly as the noisy level increases. This is consistent with the constant gap between their ground truth accuracies (gap=10% between 100%, 90%, 80%). However, for the proposed discrepancy ratio, the gap between these three models decreases rapidly when the noise level increases from 0 to 0.4. This is an undesirable behavior since it over-estimate the performance gap in the low-noise regime. Since most real-world datasets could belong to this low-noise regime, this rapid change of performance gaps seems problematic.

In section 4.1.4, the claim that "It is in this regime that the model performance can be said to be better than the average human performance" seems questionable. When the ratio is less than 1, it simply means the annotator-model agreement is larger than the annotator-annotator agreement. What's the definition of the "average human performance" in this context?

In the work, multiple assumptions were made regarding different annotators. First, multiple annotators are assumed to be conditionally independent given the ground-truth label. Second, each annotator should provide high-quality labels. Third, different experts should have similar expertise. However, in medical applications, any of these assumptions could be violated. For example,1) multiple experts could provide similar labels given their similar knowledge background. It's more likely to observe multiple groups of experts, where experts in the same group are similar. 2), experts could provide random labels when they are extremely uncertain 3) different experts could have varying levels of expertise. Could the author provide some ideas on how to relax some of these assumptions?

Comments after reading authors' reply
-----------------------------------------------------
I would like to thank the authors for their detailed reply to my questions. The key novelty of this work, i.e., evaluating model performance (in terms of comparing different models and model vs average annotator performance) in the presence of noisy labels, should be appreciated. I'll raise my score to 6: Weak Accept. In the future work, it would be interesting to apply the proposed metric (or develop new metrics) to study whether any of the assumptions (conditional independence, reasonably high accuracy of each individual expert, different expert accuracies) can be relaxed.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Summary: This paper proposes a novel metric "discrepancy ratio" for evaluating the performance of a model where the ground truth for each data point comes from many expert-yet-imperfect annotators. The authors suggested that this problem has remained largely unexplored. The proposed metric is intuitive and is easy-to-use. The authors suggested that this metric can be used for many applications. For example, to evaluate if a model is better than the annotators on average, to evaluate the annotator, to compare between models. 

========================================================
Comments on clarity:

The writing is good overall and I am able to understand the contribution and the key idea of this paper.

Questions and comments on clarity:

1. While MPD is defined with an argument (Y, Y'), the definitions annotator discrepancy, model discrepancy and discrepancy ratio ignore the inputs of the function. It might be better to keep the arguments of the function for clarity. When people want to adopt the discrepancy ratio in their work and there are many ratios to discuss, then the proposed definition can be used conveniently. 

2. In Eq. (5) \sigma seems to be undefined. I believe it is a variance but it is kinder to readers to explicitly state it (I found the authors explicitly stated it in Appendix B).

3. In Eq. (8), is it intentional to have $i$ exists over m and y as written in the paper?

4. Does the discussion of the simplicity of the discrepancy ratio hold for other \delta, i.e., not the squared loss?

5. In Section 4.1 (experiment with MNIST), did the authors binarize the data or simply discuss the multi-class problem? Because this follows section 3.3 which dedicates the whole page to discuss binary classification. It may be a bit sudden to move to multi-class classification and an additional sentence to clarify this might help.

6. I could not understand what is "single annotator strictness in Figures 1d and 2d. It is not explained if I did not miss it.

7. I think we should not resize the figure/table caption. The text size of captions are much smaller than the main text.

8. Is PLAX binary classification? (I understood as yes from Figure (3a) but I think it's also nice to clarify it to avoid confusion (also for MNIST)).

9. What is the y-axis for figure 3a?

10. I think reporting the performance of the mean-per-class discrepancy ratio without defining it in the main text (it is defined in a natural language (English) in Appendix E) makes the main body of the paper not self-contained. I strongly suggest to either (1) remove mean-per-class discrepancy ratio entirely from the main body or (2) define it mathematically in the main body. Also, I don't see reporting mean-per-class discrepancy ratio adds values to a paper for current experiments because the reported mean-per-class ratio is very similar to that of discrepancy ratio (Figures 3b (almost overlap) and Table 2 (same trend and the values are very close to the discrepancy ratio for all cases)).

11. Can we use F1 to evaluate whether a model is better than the average human performance?

Minor points on clarity, which may just be a personal preference of a reviewer:

1. The authors may consider using \begin{definition} when defining three key values: the mean pairwise deviation (MPD), annotator discrepancy, and model discrepancy. So the readers can easily look up when getting lost in definitions. And it may help to make a definition more precise (e.g., given label Y, data points, MPD is defined as follows).

2. Eqs (1), (2), (3) may be easier to understand if we put a denominator out of the summation as much as we can. For example, for Eq. 2, N can be outside of the two sums, A^i(A^i-1) can be between the two sums. 

========================================================
Significance: 
The problem that this paper addresses is highly important. I like the idea and I am convinced that discrepancy ratio is potentially useful for the community. And it also opens many possibilities to analyze the discrepancy ratio and to find potential applications for it.  

========================================================
Other comments:

On the limitation:
I am satisfied that the authors clearly discuss the limitations of the proposed method and I agree with the discussion.

On the experiments: 
1. Why we use model 1.0, 0.95, 0.9 for Figures 1c, 1d, 2c, 2d but 1.0, 0.9, 0.8 for Figures 1a, 1b, 2a, 2b? Perhaps full results with many models should also be reported for completeness.

2. Isn't it more natural to show a graph that has x-axis as label swap probability in the same figure for comparison? For example, Figure 1 only report performance with respect to each measure (Naive Evaluation, Majority Vote, Relative F1, Discrepancy ratio) with respect to label swap probability. We can fix the number of annotators to 9 for relative F1 and discrepancy ratio, to validate whether it also becomes increasingly difficult to differentiate model improvements or not for them. Then after suggesting that Naive evaluation is not good and other metrics look fine. Majority vote cannot evaluate that the model is better than human average performance, which we want to know. Then Figure 2 highlights the contribution of the discrepancy ratio and shows that it outperforms relative F1 in some perspectives.

3. For the MNIST benchmark dataset, it is possible to show the performance on the test ground truth too. And I think it is interesting to compare different models with the same label noise rate and confirm that discrepancy ratio is successful to evaluate the best model with respect to the ground truth accuracy. It would be very impressive to see if the best model with respect to the discrepancy ratio can successfully be the best also on the ground truth test set.

On space usage:
I feel there are many important things to be included in the main body but it is instead in the appendix. On the other hand, I think there are several parts that can be shortened in the main body. And it seems that the binary classification example (3.3) also consumed a lot of space. The derivation may be omitted and make it a Theorem and then let's suggest the readers to read the appendix for the proof if they are interested. Then we can have more space to add important components in the main body of the paper.

========================================================
Decision:
Although there are issues on the clarity of the paper, I believe these issues are not too difficult to fix in the final version. The idea is interesting and potentially impactful. For these reasons, I vote a weak accept for this paper.


Update: I would like to thank the authors for the clarification. I have read the rebuttal.
I read through the updated version of the paper. I believe that although there is still room for improvement, the contribution of this paper is sufficient as an important step towards solving this difficult yet highly relevant problem. This paper also stimulates several future directions. The authors clearly stated the limitations of the paper, which is highly useful for researchers to improve this work or find a better solution.  The clarity of the paper is also improved. For these reasons, I increased my score from weak accept to accept.
review_title: Official Blind Review #3

Meta-review: This paper tackles an interesting problem: "How should we evaluate models when the test data contains noisy labels?". This is a particularly relevant question in the medical imaging domain where expert annotators often disagree with each other. The paper proposes a new metric "discrepancy ratio" which computes the ratio how often the model disagrees with humans to how often humans disagree with each other. The paper shows that under certain noise models for the human annotations the discrepancy ratio can exactly determine when a model is more accurate than humans, whereas commonly used baselines such as comparing with the majority vote do not have this property. Reviewers were satisfied with the author rebuttal, particularly with the clarification that the goal of the metric is to accurately determine when model performance exceeds that of human annotators, and not to better rank models. The metric should be quite useful, assuming users are cautious of the limitations described by the authors.
Decision: Accept (Poster)