Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies instance-dependent noise (IDN) problems. It proposes a single-parameter (confidence) model for the noise generating process. Moreover, *instance-confidence embedding* (ICE) method is employed in the training process. The experimental results on various image and text classification tasks confirm the effectiveness of the proposed method.
main_review: Overall this work has enough novel and significant contributions, but some issues with the experiments weaken the work. The single-parameter noise model and the use of ICE to learn with IDN are novel to my best knowledge, though aspects of these ideas exist in other literature. The paper does a good job describing the proposed method, but the experiment section lacks some important details.

---
Strengths:

- The single-parameter noise model proposed in this work is somewhat novel, and the idea is relatively simple (which is a good thing). This simplicity allows easier estimation (smaller estimation error), though it may have larger approximation error, compared with methods that model general noise transition function $T(x)$. But it indeed has a good balance of both approximation error and estimation error, in the middle of CCN (where $T(x)$ is a constant matrix) and general $T(x)$. This noise model is well-motivated.

- The use of ICE to overcome the issue where $T(x)$ could vary significantly for two adjacent instances is interesting and seems beneficial.

---
Weaknesses:

1. Lack of theoretical results. Variational approximation in this paper is only used to justify that the proposed single-parameter noise model leads to a valid lower bound. In fact, one can use any noise model and still get a valid lower bound using the same justification. The paper does not provide any theoretical analysis of the gap. Also, the connection with CCN is not clear. It looks like the proposed noise model does not include CCN as a special case.

2. In the experiments, it is not clear how that clean labels and noise are generated. The authors only mentioned that they followed a similar way used in Zhang et al. (2021a). However, the exact details are missing in the paper and the appendix. I am kindly asking the authors to provide this information. Some specific questions:

- In Zhang et al. Section 4, they first trained a model to approximate the posterior class probability $\eta(x)$ using the whole data set. Then they sampled $y_x \sim \eta(x)$ for each instance $x$. Instead of using the original labels, they used the sampled labels as clean labels. Was this paper following the same approach?

- In Zhang et al. Section 4, several types of noise generation are considered. Which type did this paper use? If for different data sets different types were used, please list them accordingly.

- In Table 1 of this paper, how was the overall noise rate controlled? If this paper followed Zhang et al. by multiplying the noise function by some constant factor, please specify the constants. Please also list the controlled noise rate for each data set.

3. This paper did not compare with the method in Zhang et al. (2021a) without any justification, even though the paper used Zhang et al.'s setting for noise generation. Please add the method in Zhang et al. to Table 1, or provide explanation why you chose not to compare with Zhang et al.

4. In Section 3.2, one important assumption is that the noisy posterior class probability ($q$) and the clean posterior class probability ($p$) share the same argmax. This is quite a strong assumption, because effectively, with this assumption, one can train directly on the noisy examples using standard methods and predict correctly (in the Bayes optimal sense). In fact, such assumption is not necessary for good predictions. For example, in CCN setting, if the noise transition matrix is invertible, one can train a good classifier using noisy examples and predict well w.r.t. the clean distribution. In Zhang et al., no such assumption was made.  A related question: how is this assumption enforced in the noise generation for the experiments?

summary_of_the_review: Overall this work has enough novel and significant contributions (single-parameter noise model and ICE). But it has a strong assumption (argmax preserving) and lacks theoretical justifications. More importantly, the experiment section lacks detailed descriptions and did not compare with Zhang et al. (2021a). These shortcomings make me skeptical about the effectiveness of the proposed method. Therefore, I vote for a weak reject.

---

After rebuttal:

I raised my score to 6. See my reply below for more details.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presented an instance-confidence embedding model (ICE), which is a variational approximation of the instance-dependent noise (IDN). Given the observation that most columns of the IDN transition matrix have only limited influence on the class-posterior estimation, this paper uses a single-scalar confidence parameter to model the noise and uses a simpler transformation to transfer clean class posterior to noisy class posterior. The suggested method captures instance-specific noise information and thereby improves classification performance when compared to previous methods based on the class-conditional noise (CCN) assumption.
main_review: Strength: 
The suggested method captures instance-specific noise information and thereby improves classification performance when compared to previous methods based on the class-conditional noise (CCN) assumption.


Concerns:
1.	For an instance X with label Y=i, how do you define $T_{kj}(x)$, where $k \neq i$.
2.	At the bottom of page 2, matrix-valued function $T$ is introduced. Is it different from the transition matrix $T(x)$?
3.	Why do you need $K\times K$ parameters to model the transition relation for each instance X while the $p$ and $q$ you defined are both $\in \Delta^{K-1}$?
4.	The previous works employed some assumptions or are based on domain-specific knowledge but they fully model the IDN. In your method, the X is replaced by one postulated variable C learned by embedding an index parameter. Is this assumption strong?
5.	The authors claim that most columns of the IDN transition matrix have only limited influence on the class-posterior estimation and the transition function should be argmax-preserving function. Does it mean this method can only deal with label noise with a low noise rate?
6.	$h(\boldsymbol{p} ; 0)=\boldsymbol{u}$, where $\boldsymbol{u} \in \Delta^{K-1}$. Does it mean $u$ is uniformly sampled front the rest $K-1$ classes expect the true class? Why is that?
7.	In Eq. (8), what is $u_i$? Is it the $i-th$ element of $u$, then what is $u$?
8.	Eq. (9) needs to be elaborated. 
9.	The choices of $h$ and $g$ seem come from nowhere and need more justification. Why can this method work?
10.	Important baselines are missing. All the IDN methods the authors mentioned in the introduction part were not considered as baselines. 
11.	Some grammatical issues should be fixed.
summary_of_the_review: Overall, the proposed method is too heuristic without enough justification. The experiments are not sufficient.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces ICE, an instance-confidence embedding approach to learn with a challenging noisy label setting: instance-dependent noise (IDN). Considering the sparse property of IDN, the authors propose to use instance embedding by equipping a trainable parameter to each data instance. Experiments results on various image datasets and text data show the effectiveness of the proposed method with the presence of label noise.
main_review: $\textbf{Strengths}$
* **Novelty** This work provides a novel and interesting approach to using only a single-scalar confidence parameter to approximate the IDN transition matrix.
* **Ambiguous/Noisy-label detection** The 1-D instance embedding is potentially beneficial for detecting ambiguous or wrongly labeled instances. 
* **Extensive empirical validation** The authors test the effectiveness of ICE in multiple image classification datasets and text classification tasks.

$\textbf{Weaknesses}$
* **A single-scalar V.S. $K\times K$ noise transition matrix** Admittedly, the $K\times K$ noise transition matrix is usually of a sparse pattern. Will a single-scalar miss crucial information? Or in other words, could authors explain why this approximation captures necessary/useful noisy-label information at the instance level?

* **The intuition of $g$** It would be better if the authors could provide me with more intuitions of $g$ in the role of confidence score.

* **IDN-based baselines**  Learning with instance-dependent noisy labels has become a popular topic in recent years. And several approaches have been proposed to mitigate the impact of IDN. However, the majority of baselines adopted in Table 1 are designed for the CDN model. To further validate the effectiveness of ICE, it would be better if the authors could compare with some more IDN-based methods, for example, [1]-[5].

* **Real-world noisy benchmarks** Evaluating the rationality of (synthetic) instance-dependent label noise is non-trivial. Real-world noisy labels are commonly assumed to follow the IDN pattern. It would be better if the authors could test the performances on real-world noisy benchmarks, especially some small-scale benchmarks such as Animal-10N [6], Controlled Noisy Web Labels [7], or CIFAR-10N, CIFAR-100N [8].


$\textbf{Reference}$

[1] Part-dependent Label Noise: Towards Instance-dependent Label Noise. (NeurIPS, 2020)

[2] Learning with Feature-Dependent Label Noise: A Progressive Approach. (ICLR, 2021)

[3] Learning with Instance-Dependent Label Noise: A Sample Sieve Approach. (ICLR, 2021)

[4] A Second-Order Approach to Learning With Instance-Dependent Label Noise. (CVPR, 2021)

[5] Confidence Scores Make Instance-dependent Label-noise Learning Possible. (ICML, 2021)

[6] https://dm.kaist.ac.kr/datasets/animal-10n/.

[7] https://ai.googleblog.com/2020/08/understanding-deep-learning-on.html.

[8] http://noisylabels.com/.
summary_of_the_review: This is an overall novel and interesting paper. While I still have some concerns regarding the single-scale confidence parameter and the intuition of the instance-confidence embedding function $g$. Besides, I am not fully convinced of the effectiveness of ICE by referring to the baseline selections. I am willing to increase my score if some of my above concerns are well addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes ICE (Instance-confident embedding) method for learning with IDN (instance-dependent label noise). ICE approximates IDN by using a single-scalar confident parameter for each sample based on the assumption that $P(Y|x)$ is close to a one-hot vector. ICE is incorporated into the loss function for efficient learning process under label noise. Authors conduct experiments on both image and text classification to verify the effectiveness  of ICE and perform gradient analysis for further understanding the method.

main_review: Strength:
1. This paper is well written and easy to follow. The motivation is strong and interesting. It is of great importance to approximate IDN with a simple method since IDN is generally hard to model. This paper proposes a simple way to approximate IDN by using single-scalar confident parameter.

Concerns:
1. I am quite confused by the part of embedding. The embedding is built on the sample index instead of the sample (feature) itself. I think it does not follow the definition of IDN. $C$ in Equation (8) (9) should be dependent on the feature rather than the feature index. 

2. Many methods have been proposed in the literature to deal with IDN (A1, A2, A3, A4, A5). However, in Table 1, the authors do not compare any of them. Besides, the performance gain of ICE is minor compared to GCE on CIFAR10 and CIFAR100. 

3. ICE is similar to label smoothing in the formulation. Authors are encouraged to perform a comparison in the experiments. 

4. Why only perform 40 epochs on CIFAR10 on CIFAR100. It does not follow the convention. Does the network begin to overfit label noise as training proceeds?

A1: Learning with bounded instance- and label-dependent label noise. ICML 2020

A2: Part-dependent label noise: Towards instance-dependent label noise. NeurlPS 2020

A3: Confidence scores make instance-dependent label-noise learning possible. ICML2021

A4: Learning with feature-dependent label noise: A progressive approach. ICLR2021

A5: Learning with instance-dependent label noise: A sample sieve approach. ICLR2021
summary_of_the_review: This paper has a strong motivation. However, it seems that the proposed approach (ICE) does not strongly connect to IDN and the performance gain is minor compared to existing approaches. Authors should provide more explanations of why embedding on the index can approximate IDN and perform more comparisons.
review_title: null

Meta-review: The paper proposes a new method for the problem of learning under instance-dependent noise (IDN). The idea is to construct a variational approximation to the ideal training objective, which involves learning a single scalar C(x) per instance. In turn, each such scalar is treated as an additional parameter to be learned by the network.

Reviewers generally found the basic idea of the proposal to be interesting and novel, with the response clarifying some initial questions on the design of the network to learn C(x). The paper is also well-written, and presents experiments on image and text classification benchmarks. Some concerns were however raised:

(1) _Limited theoretical justification_. There is limited formal analysis of when the proposed method can work well.

(2) _Lack of comparison to IDN baselines_. The original submission did not include any IDN baselines as comparison. The revision included results of the method of (Zhang et al., '21a), which is on-par or better than the proposed method; it seems that this baseline really ought to have been included in the original submission, but it is appreciated that these have been added. A related concern was the marginal gains over the GCE method on the CIFAR datasets.

(3) _Sufficiency of learning a single parameter_. The paper learns a single scalar per sample. Several reviewers were unsure on the sufficiency of this parameter to capture the underlying noise distribution.

For (1), the authors acknowledge theoretical analysis as an important future direction. This is perfectly reasonable, but does then require weighting more any issues with the the conceptual and empirical contributions of the paper.

For (2), the response clarified that most of these operate either in the binary setting, or require auxiliary information. This is a valid motivation for the present work; it would however be more compelling to include results in a binary setting, to better understand the strengths and weaknesses compared to existing proposals. The response also clarified the present method does not claim to improve upon state-of-the-art performance, but rather proposes a simple method which has additional applications (as shown in Appendix E). This is a reasonable claim; however, to my taste, there is insufficient discussion of the PLC method (Zhang et al., '21a), and what new conceptual information the present work offers.

For (3), the response argued that the present results already demonstrate the efficacy of using a single parameter, and that using multiple parameters can be studied in future work. One reviewer was not convinced of the efficacy being shown in some of the results in Appendix E. It could strengthen the work if there is an empirical analysis of when the single parameter assumption starts to break down; e.g., perhaps under increasing levels of CCN noise?

Overall, the paper has interesting ideas and some nice analyses. At the same time, there was clear scope for improvement in the original submission. This was partially addressed in the revision, but given that several domain experts retain reservations (particularly in regards to comparisons against prior IDN works), it is encouraged that the authors incorporate the above comments for a future version of the paper.
Decision: Reject