Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies a weakly supervised learning setting, in which one has limited access to the *confidence labels* of the training examples. Previous work in this line includes the setup with pointwise confidence scores (Pconf), and the setup where pairwise comparisons of the confidence scores are available (Pcomp). This work presumes more fine-grained information than the latter: Unlabeled data pairs with confidence difference (ConfDiff). This learning setup is then formulated as empirical risk minimization and a corresponding unbiased risk estimator is constructed, together with an estimation error bound.


strength_and_weaknesses: ### Strength  

- The paper is technically well formulated. The proposed learning setting was rigorously set up and an unbiased risk estimator is derived for the empirical risk minimization. 
- The paper is well written and easy to follow. Even people outside this particular field should be able to grasp the general idea proposed therein. 

### Weaknesses

- In my opinion, the biggest weakness of this paper is its setting. Is the setting realistic? The paper doesn't provide sufficient motivation in the introduction; it also lacks realistic experimental setup to support the ConfDiff setting. 
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. As far as I can tell, the reproducibility is high (although I didn't read all the proofs.)

Regarding the novelty, while I do think the ConfDiff setting is new, I am doubtful if it is a realistic or useful setting in practice, as mentioned above.
summary_of_the_review: As mentioned earlier, my key issue of this paper is that I don't think the ConfDiff setting is realistic. 

In terms of the level of information required in training, ConfDiff sits in between Pconf and Pcomp, in theory. In practice, however, I don't see a situation where one obtains the exact difference between two confidence scores, **without** first estimating the pointwise confidence scores. Even the experiments in the paper have to first do the point estimation. 

It was mentioned in the paper that *the confidence difference is given by annotators in real-world applications*. Has such annotation procedure ever actually applied in the real world? We have to realize that it is extremely hard for a human annotator to give exact confidence difference between two examples. I would say the annotation settings of Pconf and Pcomp are, comparably speaking, more realistic. In the case of former, each annotation provides more information. (Also I'd say the annotator might have to do pointwise estimates first before giving exact confidence differences). In the case of latter, the annotation is much simpler for the annotator as it is only a qualitative paired comparison.

In any case, I think the paper should motivate the proposed setting better, ideally with some real-world applications.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to learn from observing probability differences between pairs of labels, in a binary classification setting. The premise is that obtaining such information is sometimes easier than obtaining single probability estimate, and richer than simple qualitative comparisons. 

The paper demonstrates that such information is sufficient to converge towards the optimal risk, deals with the case of corrupted estimates, and also propose a correction to avoid the loss becoming negative through a ReLu correction. 

Some experiments are done, in particular to compare with a qualitative feedback. 
strength_and_weaknesses: +: the paper is clearly written, and I think the proofs are correct (I am less familiar with generalisation bounds, so I cannot give strong guarantees for Theorem 3 and the next ones, but the reasoning seems ok to me). 

+: As far as I know, the proposed framework is original

-: it is not clear at all to me, despite the two high-level examples given in the introduction, that providing accurate probability differences for pairs of data is much easier/simpler than providing single probability estimates. While I would agree that providing qualitative comparison (Pcomp framework) is easier, it is not so clear for probability differences. And the argument put forward later on that different experts may disagree on probabilistic estimates is equally true for probabilistic differences. Do we have actual evidence (and not illustrative example, that I have a hard time really figuring out) that this is indeed the case in some applications? Maybe the authors could elaborate a bit more on their examples to provide a fuller story, where the estimates are explicitly mentioned?

-: while Pcomp and Pconf can be adapted easily to multi-class setting, it seems intuitively harder to adapt the current proposal to more than two classes, both from a theoretical perspective but also from a practical perspective (it seems very demanding and hard to elicit probability differences over all pair of classes). Could some comments be provided with respect to that? 

-: I do not understand why the framework is not compared, in Table 1 (or in appendices tables) to both Pconf and the framework where data are fully observed (rather than using various Pcomp variants).

-(minor): I wonder why in the test non-binary data sets were turned into binary data sets. UCI and other repositories contain plenty of native binary data sets, and it is unclear why those particular data sets were chosen, and why the separation was chosen the way it was? 
clarity,_quality,_novelty_and_reproducibility: Overall the paper is clear and well-written.
summary_of_the_review: The paper proposes a new learning framework considering as supervision differences of probabilities. While the theoretical analysis is interesting, it is unclear how practical this framework is, nor how realistic is the assumption that we can have good estimates of probability differences. It is also unclear how the framework could extend beyond binary classification.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposed a novel learning problem for binary classifications, namely Confidence-Difference (ConfDiff) classification, in which the learner is only given unlabeled data pairs $x,x'$ equipped with confidence difference specifying the difference in the probabilities of being positive $P(y'=1|x') - P(y=1|x)$. The authors proposed to solve the problem via ERM by constructing an unbiased risk estimator. Estimation error bound was derived, and the robustness of the risk estimator was analyzed. Experiments demonstrated effectiveness of the proposed method.
strength_and_weaknesses: **Strong points.**

The proposed learning problem is interesting. The proposed method is backed by strong theoretical analyses and empirical results. The paper is well-written and easy to understand.

**Weak points.**

The concern I have is that whether the proposed learning problem is a valid weakly supervised learning paradigm. Confidence labels (posterior class probabilities $P(Y=1|X)$) certainly contains more information than the usual labels. On the other hand, unlabeled data pairs and differences somewhat constitute a weaker learning problem. Thus, to me, it is not entirely clear whether learning with unlabeled data pairs and confidence differences is a weaker or stronger learning problem.

Also, in the experiments, Pcomp was compared against. However, as mentioned in Section 2.3, in Pcomp classification, the learner only knows unlabeled pairs, and within each pair $x,x'$, which one is more likely to have positive label than the other (whether $P(y'=1|x') > P(y=1|x)$). Clearly, ConfDiff classification has access to more information. Thus, it is expected that ConfDiff can outperform Pcomp. Nevertheless, it is okay to use Pcomp as a baseline.
clarity,_quality,_novelty_and_reproducibility: The paper is well-organized and well-written. I enjoyed reading it. Overall it is novel at large. The reproducibility is good.
summary_of_the_review: I vote for a weak accept as I think the paper overall meets the standard, and its contributions outweigh its shortcomings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses a learning setting where data come as unlabeled pairs decorated with the difference between their probability for each of them to be positive data, a setup called ConfDiff. The authors provide unbiased estimators of the risk (depending on the loss functions) that are used to derive learning algorithm.

+ paper well written
+ the topic of getting away from the traditional labelled training data is key, as getting labels may be expensive
+ the used technique: recovering an unbiased estimator of the risk and then minimizing it is relevant

- the motivation and applicability of the learning setting are not obvious
- the used technique is a "traditional" technique encountered when dealing with noisy, positively-unlabeled, or confidence-rated labels

The authors haven't discussed "structural" results as those coming from the seminal PAC literature making the connection between learnable classes of concepts like those learnable under classification noise, statistical queries, class-conditional classification noise, constant-partition classification noise... Those are general results making reductions from one "non-usual" setup of learning to the other (see Kearns, 1998, Blum, 2000, Ralaivola et al, 2006).
justification_for_why_not_higher_score: The authors fail to provide a better motivation for the setup, which henceforth seems a bit artificial.

justification_for_why_not_lower_score: The technical material is correct and the write-up is good.
Decision: Reject