Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Learning with noise labels is a hot topic now due to the reason that deep learning algorithms often require large-scale supervised training samples and labelling a large amount of data is costly. However, almost all of the existing methods assume that the label noise is instance-independent. It either depends on the clean classes or is completely random. This paper studies the instance-dependent label noise, which is more realistic and applicable, but difficulty to address. The authors target to solve this problem. A feasible solution would contribute to the community a lot.

The challenging part for dealing with instance-dependent label noise is to learn the instance-dependent label flip function, which is hard to learn by only exploiting noisy data without any assumption. Few existing papers proposed assumptions to make the flip function learnable. This paper also introduces an assumption that the confidence score for each data is given. To me, this is also a strong assumption. Although the authors have provided examples of how to collect confidence score, collecting confidence scores may not be easy for many specific problems. 

Given the confidence score, the authors proposed to learn the flip function. The clean class posterior can be inferred by using the noisy class posterior and the label flip function. So, how to learn the instance-dependent flip function is an essential step. Some technical contribution has been made to learn the flip function. In this section, the authors further assume that the off-diagonal entries of the flip matrix are independent of instance. After seeing the explanation, I personally agree that the assumption is reasonable for some cases. However, I found that in the experiments, the authors synthesized label noise where the off-diagonal entries depend on the instance. The proposed method also shows its advantages on this case. Can you explain this?

The experiment parts show the effectiveness of the proposed method both on synthetic data and real-world data. Overall, the paper is well-motivated and well-written. 

I have another concern that the obtained confidence score may not accurate. Is the proposed method sensitive to the confidence scores?

review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: --- Overall ---

I think the problem of incorporating information about labeling uncertainty (when such information is available) is an interesting and important problem; however, I think this paper contains a crucial misunderstanding of the definition of calibration (as defined in Guo et al. 2017), is missing some important parts of the literature, and does not provide adequate convergence guarantees for the proposed method.

--- Major comments ---

1. In section 3.1, the authors substitute the confidence score r_x for the conditional probability P(Y=y|\bar{Y}=y,X=x); however, even if the confidence scores are perfectly calibrated, these quantities are not necessarily equal (or even particularly close). As defined in Guo et al. (2017), a confidence score is well calibrated if the probability that a prediction is correct given the confidence score is equal to the confidence score. Using their notation: P(Y=\hat{Y}|\hat{P}=p) = p. Importantly, this is not conditioned on \hat{Y} or X. One way to see that these two quantities are not equal is to observe that there are many possible confidence score functions that satisfy the above definition whereas the conditional probability defined above is a unique function. In particular, if the confidence score function is a constant equal to the accuracy of the model, then the confidence score is well calibrated, but clearly not equal to the conditional probability above.

2. The other scenario considered by the authors is when there are multiple annotators; however, there is substantial literature on learning from multiple noisy annotations which the authors do not review. I suggest the authors start with "Modeling annotator expertise: Learning when everybody knows a bit of something" by Yan et al. (2010)  (and the citations therein) which also considers the instance dependent noise case.

3. Under what conditions does the proposed algorithm converge and to what does it converge to?

4. The proposed method relies on several assumptions that are scattered throughout the description of the algorithm. I highly recommend making these assumptions clear near the beginning of the paper. Specifically, my understanding is that the main assumptions are:

i. Confidence scores r_x are available for each instance and r_x \approx P(Y=y|\bar{Y}=y,X=x).
ii. Y _|_ X | Y\neq\bar{Y}, \bar{Y} = y
iii. Anchor points are available on some portion of the data.

Beyond the presentation, I find this to be a fairly strong set of assumptions, particularly the first assumption. 

--- Minor comments ---

1. I really appreciated the synthetic example demonstrating the potential pitfalls of the small loss approach; however, I would spend a bit more time clearly explaining the small loss approach so that readers understand why it fails and how you are solving it's problems.

2. Also in the synthetic example, the authors state that covariate shift leads poor accuracy, however, I think this point would be stronger if demonstrated instead of just asserted.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper focuses on instance-dependent label noise problem, which is a new and important area in learning with noisy labels. The authors propose confidence-scored instance-dependent noise (CSIDN) to overcome strong assumptions on noise models. They clearly define confidence scores and justify their availability. To solve CSIDN model, they propose instance-level forward correction with theoretical guarantees. Their experiments on both synthetic and real-world datasets show the advantage of this algorithm.

Pros:

1. This paper is clearly written and well-structured in logic. For example, in Section 2, they introduce from class-conditional noise to instance-dependent noise first, which paves the way for confidence-scored instance-dependent noise. This make readers easy to follow the main contribution, namely the new noise model.

2. This paper pushes the knowledge boundary of learning with noisy labels, since it focuses on more realistic and challenge topic "instance-dependent label noise". The authors leverage the idea of confidence scores, and propose confidence-scored instance-dependent noise (CSIDN). Compared to previous solutions, CSIDN is a tractable instance-dependent noise model, which enjoys several benefits, such as multi-class classification, rate-identifiability and unbound-noise.

3. This paper proposes an algorithm to solve CSIDN inspired by forward correction called instance-level forward correction (ILFC). Their algorithm has been verified in both synthetic datasets and real-world datasets. The empirical results show the advantage of ILFC.

(Minor) cons:

1. Section 3 is a bit dense in understanding the estimation of transition matrix. The authors are encouraged to polish this section. 

2. Although ILFC outperform CT and LQ in real-world datasets, the authors need to add the reasults of MAE and FC to more thoroughly verify the performance of ILFC.
review_title: Official Blind Review #1

Meta-review: While two reviewers  rated this paper as an accept, reviewer 3 strongly believes there are unresolved issues with the work as summarized in their post-rebuttal review. This work seems very promising and while the AC will recommend rejection at this time, the authors are strongly encouraged to resubmit this work.
Decision: Reject