Confidence Difference Reflects Various Supervised Signals in Confidence-Difference Classification

Authors: Yuanchao Dai, Changchun Li, Ximing Li
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Confidence-Difference Classification, Binary Classification, Weakly Supervised Learning

Abstract:
Training a precise binary classifier with limited supervision in weakly supervised learning scenarios holds considerable research significance in practical settings. Leveraging pairwise unlabeled data with confidence differences has been demonstrated to outperform learning from pointwise unlabeled data. We theoretically analyze the various supervisory signals reflected by confidence differences in confidence difference (ConfDiff) classification and identify challenges arising from noisy signals when confidence differences are small. To address this, we partition the dataset into two subsets with distinct supervisory signals and propose a consistency regularization-based risk estimator to encourage similar outputs for similar instances, mitigating the impact of noisy supervision. We further derive and analyze its estimation error bounds theoretically. Extensive experiments on benchmark and UCI datasets demonstrate the effectiveness of our method. Additionally, to effectively capture the influence of real-world noise on the confidence difference, we artificially perturb the confidence difference distribution and demonstrate the robustness of our method under noisy conditions through comprehensive experiments.

Decision: reject
Reviewer scores: 5, 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1729239299983

thread_type:
official_review

<thread_object>
creator: reviewer_r1j9

<content>
summary:
This paper deals with confidence-difference classification, a weakly supervised binary classification problem. In order to mitigate the noise contained in the confidence differences, a novel risk estimator using consistency regularization is employed to improve performance. Extensive experiments on benchmark datasets validate the effectiveness of the proposed method.

strengths:
- The problem studied, i.e., confidence-difference classification with noise, is an interesting and promising problem.
- The motivation that different confidence difference values convey different information is valid and interesting, which has not been explored in the literature.
- The proposed consistency regularization term is valid, which can improve the performance when the confidence difference is small for some training data.
- The experiments are comprehensive and validation studies are also conducted to validate the effectiveness of the proposed method.
- The paper is clearly written and easy to understand.

weaknesses:
- One of my concerns is whether the noise is more pronounced when the confidence difference is small. When the confidence difference is large, noise can also be introduced. Although it does not affect the algorithm, which I think can work well even if no noise is introduced in the experiments. I think this point needs clarification and the paper can be polished as well. I think a more reasonable story is to use some terms like "clear" and "ambiguous" to replace data with and without noise. Some data pairs are more clear (larger confidence difference absolute values) and their labels can be (+1,-1) or (-1,+1) with a high probability. Some data are ambiguous (small confidence differences) and we cannot tell if they are both positive or negative. This story may be more reasonable from my point of view. To handle different types of data, the consistency regularization term is introduced to improve learning from ambiguous data pairs.

- The descriptions in Section 4.2 are not so clear. I cannot understand how the data is generated, such as the meaning of $c_{non-outlier}$. 

- There are some unclear notations and expressions that should be examined. For example, $\ell$ is missing in line 235. In line 241 it should be $\hat{g}$ instead of $R(\hat{g})$. In line 42-43, it seems to apply to general supervised learning instead of weak supervised learning? In line 89 it should read "difference". Also, more discussion should be added after Theorem 1. The space for the post text can be reduced. Also, the "\_" notation for the methods can be replaced with "-".

questions:
- Why do larger confidence differences contain less noise?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730103345146

thread_type:
official_review

<thread_object>
creator: reviewer_wt35

<content>
summary:
The authors study a challenging weakly supervised task called confidence difference (ConfDiff) classification, where only the posterior probability differences for pairwise data points are accessible during training. To handle the influence of noise in ConfDiff, the authors introduce a method called CRCR, which partitions the training data into two subsets with different ConfDiff levels and applies a different strategy to each subset. Additionally, theoretical analyses of the proposed methods and experimental results on multiple benchmark datasets are provided.

strengths:
1. This article treats ConfDiff as a soft label problem. It partitions the dataset using a threshold and applies different strategies to handle data with high and low ConfDiff levels. The effectiveness of this method in combating noise is verified through experiments.
2. The authors strengthen the analysis by conducting ablation studies to clarify the contributions of dataset partitioning and the consistency regularization term. These experiments significantly enhance the clarity and depth of the results, making it easier to understand the individual impact of each component.
3. Theoretical analyses are provided to demonstrate the theoretical guarantees of the proposed methods.

weaknesses:
1.	There is a lack of descriptions of Theorem 1.
2.	It is recommended that the authors provide a detailed explanation of 'consistency' within the text.
3.	The manuscript requires careful revision to address various issues that currently detract from its clarity and academic rigor. For example, please consider revising the following sections of the manuscript to enhance clarity and accuracy:

* line 014: Consider revising the phrase ‘’significant research significance‘’ to avoid redundancy and enhance clarity. Perhaps ‘’considerable practical significance‘’ would be more appropriate.
* line 155 Eq. (3): The notation [L(x, x), L(x, x)] may need to be corrected to [L(x, x) + L(x, x)].
* Line 174: It might be appropriate to ‘substitute the form of the loss function from Equation 5 into Equation 3.’.
* Line 210: Please clarify the meaning of $\theta(D^S)$ and $\theta(D^C)$.
* Line 241: the term $R(\hat{g}{CRCR})$ should be corrected to $\hat{g}_{CRCR}$.
* Table 3, line 387: It appears there is a labeling error, as CRCR_RelLU (0.930) is noted to significantly outperform ConfDiffABS (0.940), which seems counterintuitive. Please verify and correct.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730479794645

thread_type:
official_review

<thread_object>
creator: reviewer_u74g

<content>
summary:
This paper considers weakly-supervised binary classification from noisy confidence difference annotations. It builds on the existing work of risk-estimation based approach and further analyze and modify the terms in the risk estimator. For the proposed risk estimator, error bound is derived and thorough experiments are conducted.

strengths:
- The paper is overall well written. Phrases are generally easy to understand.
- The mathmetical prensentation is sound, and definitions are clear and easy to follow.
- Experiments are thoroughly conducted, with implementation codes attached.
- The background of the problem setting and related methods are well summarized. Authors show a good level of understanding of the problem.
- The problem of tackling noise in the confdiff weakly supervised settings is of high practical importance, and also a theoretically interesting problem.

weaknesses:
- The paper lacks a clear story of presentation of the motivation. There seems exist multiple concepts of "noise". At least there are two different noises: the noise of generated conf diff resulting inaccurate $c$, or the noise of model learned by $|c|<0.5$. They are not clearly described, thus resulting weak motivation for the proposed method.
  - Phrases such as "distribution influenced by real-world noise", "perturb the confidence difference distribution to better fit real-world scenarios", "we observe that the confidence difference values utilized for training in ConfDiff classification are frequently noisy, rather than exact difference between the posterior probabilities of two samples." clearly indicate the motivation is about the noise of the generated conf diffs.
  - At the same time, phrases such as "result in one of the data points being estimated in the opposite direction, introducing noise.
This leads to inaccurate predictive directions even in the absence of noise." seems to talk about another level of noise, noise of the model learned by $|c|<0.5$.
- The important part of consistency regularization term takes a too small part of the whole paper. The motivation and intuition of the modification and how the modification behaves if not clearly demonstrated.
- The other part of contribution, consistency risk of $D^S$ is not well motivated, thus seems irrelevant to the story.
- For Eq. 5, "general form of many commonly used losses" is suddenly introduced without further explanation, such as what specific loss functions belong to this form, or does not belong to this form.
- Too much space is spent on background methods, such as Section 2.

questions:
- For Figure 1
  - What does proportion (x-axis) being more than 1.0 mean?
  - How to control the proportion with a fixed $\pi$? I assume the generation process is that two data points are first i.i.d. generated and then the conf diff is consequently calculated. Does changing the proportion mean that there is a selection mechanism exists during the data generation process, such resulting a skewed data distribution?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730626010536

thread_type:
official_review

<thread_object>
creator: reviewer_u1nv

<content>
summary:
The paper studies a special type of weakly-supervised learning known as confidence difference learning. This method leverages confidence differences between unlabeled data pairs to improve classifier training under noisy real-world conditions. By incorporating a noise generation technique and a risk estimation framework that includes consistency risk and regularization, ConfDiff classification demonstrates enhanced robustness and outperforms traditional methods in experiments on benchmark and UCI datasets. Theoretical analyses providing error bounds for the risk estimations further support the method's effectiveness.

strengths:
1. The main theoretical contribution of this paper improved over [1] seems to be the incorporation of consistency regularization and its effect in error bound; based on my understanding, $C_{g}$  bounded the differences between $x$ and $x'$, so that under authors' setup, if the prediction of these data points is close enough, then the perceived generalization error should decrease, which is sensible.

2. After a very coarse examination, the proof of this paper seems to be correct.

3. Encourage instances with smaller confidence differences to produce similar outputs that seem intuitive and sensible, both theoretically and empirically.

[1] Binary classification with confidence difference, NeurIPS 2023.

weaknesses:
1. It appears that the ConfDiff learning requires the classifier to be perfectly calibrated, which is usually infeasible in reality. I think it would be useful to incoperate the results on how different level of calibration error will influence the performance of ConfDiff learning.

2. The implications of the theoritical results in this paper are not discussed, the authors simply present the bound as is, but fail to elaborate any new insights or messages we can draw from the error bounds - what are the dominant terms in the error bound? How is this related to the noisy supervision signal? What properties of your proposed method is related to this error bound? How this bound improves upon the existing bounds? Given the current form of the paper, the audience can only have a vague guess of the above questions.

3. This paper appears to be unclear in many details, why negative risk can lead to overfitting? 

4. Suppse we are given a supervision signal $\tilde{c}(x,x') < 0.5$ (smaller confidence difference), then by fitting this objective, wouldn't the $R_{CD}$ inherently encourages $x,x'$ to be similar? Hence making $R_{CRCR}$ trivial? 

Minor issues:

1. From line 210-212, shouldn't the $c(x,x')$ be $\tilde{c}(x,x')$ instead?

questions:
How the noise is being defined and formulated in this paper is still a bit puzzling to me, I recommend the authors to improve the writing of the Section 3.1, and maybe considering adding some concrete examples ton make their points.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732288849172

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
This section provides a general response to the comments on "**the motivation of this paper**" and "**the analysis for Theorem 1**". We would like to thank all of you for your constructive comments, which are very helpful for improving the paper. We have revised our paper carefully. And the detailed point-by-point responses to the comments are given in the corresponding sections for each reviewer.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733277534212

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Area Chair,

We would like to provide a summary following the authors' rebuttal. We sincerely thank all the reviewers for their thorough evaluations and valuable feedback. We are especially grateful to Reviewer u74G for the opportunity to clarify potential issues and for being willing to **update the rating** from 3 to 5. We also appreciate Reviewers wT35 and R1j9 for recognizing our paper's technical soundness and providing positive ratings **6** and **8** prior to the rebuttal, respectively.

We would like to express our sincere gratitude to Reviewer u1NV for their thoughtful review. However, we note that they provided lower scores. In our rebuttal, we carefully addressed the reviewer's feedback and greatly appreciate the two additional questions raised after receiving our response. We have provided detailed replies to these questions but have yet to receive feedback from the second round of review. Consequently, we have not had the opportunity for further discussion with the reviewer. Nevertheless, we are confident that we have thoroughly addressed their concerns.

Regarding the probability calibration concern raised by Reviewer u1NV, we would like to clarify the following:

This issue arises solely at the experimental setup and is unrelated to our method.

Our experiments follows the setup outlined in [1], and motivated by the probability calibration problem, we introduce varying proportions of noise to the posterior probabilities to simulate the effects of calibration errors.

Furthermore, regarding the concern of consistency regularization, we would like to clarify that it is introduced to address the problem of incorrect prediction directions arising from small confidence differences in $R_{CD}$. Our optimization objective targets the classifier's output rather than the confidence difference.

More details are available in our rebuttal response to the reviewers.

According to the input we provided, most reviewers indicated that their concerns were addressed. We sincerely appreciate the reviewers for recognizing the innovation and potential impact of our work.

Thank you!

Sincerely,
The Authors

[1] Wei Wang, Lei Feng, Y uchen Jiang, Gang Niu, Min-Ling Zhang, and Masashi Sugiyama. Binary classification with confidence difference. Advances in Neural Information Processing Systems, 36, 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work enhances binary classification in weakly supervised learning by leveraging pairwise confidence differences and proposing a consistency regularization-based risk estimator to mitigate noisy supervision. The paper has received mixed feedback from the reviewers. We acknowledge and appreciate the authors' efforts in thoroughly addressing the reviewers' comments and conducting additional experiments to resolve some concerns. However, after a comprehensive review of the feedback and the authors' responses, several issues remain unresolved. These include the lack of a discussion on the implications of the theoretical results, the absence of an analysis of dominant terms in the error bound, and an unclear narrative in presenting the motivation for the work. Based on these considerations, the paper is not recommended for acceptance at ICLR'25.