Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed a method to learn with PU data which quantifies the epistemic uncertainty of an ensemble of networks and selects which examples to pseudo-label based on their predictive uncertainty. The authors propose to use the pseudo-labeling technique based on the uncertainty of the prediction, combined with early stopping.
main_review: Trivial motivation:

In Abstract, the authors say that "two-steps procedures are vulnerable to incorrectly estimated pseudo-labels" and to mitigate this issue they propose this method. It is well known that many important PU learning methods are not two-step, such as nnPU.

Limited novelty:

First, I am not convinced that the proposed method is novel. The key techniques used are transferred from some existing methods (from Sec.3.2-3.6).
Second, I don't understand the meaning of designing an algorithm like this. For example, it uses an unbiased risk (Kiryo et al., 2017), which can already guarantee the consistency of learning. But complicating the learning procedure makes this loss lose its original advantage, and an additional regularization method has to be added. And it labels the instance according to the epistemic uncertainty (Eq.4-6). It is unclear why this measure is used and what the advantages are. The algorithm design is almost at random, at least from the current description. In addition, I think the complex algorithm design does not result in significant performance improvements.

Minor concerns:

- What is Eq.(3) for?
- The writing of the paper can be improved, e.g., "two-steps procedures"->"two-step procedures"

summary_of_the_review: The novelty is limited and the contributions are incremental.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors proposed to use pseudo-labels based on high confidence predictions to improve the classification performance of PU learning. Experiments show the effectiveness of the proposed method.
main_review: Strengths
+ A simple uncertainty-aware pseudo-labeling framework for PU learning is proposed.
+ The proposed method is general and can be integrated into any loss function for PU learning.
+ This paper is easy to follow and the logic is clear. 

Weaknesses
+ The novelty of this paper may not be enough. Specifically, the pseudo-labeling techniques have been well studied in a lot of weakly surprised learning scenarios such as label noise learning and semi-supervised learning. 
+ The motivation of the metric used to quantify the uncertainty is not clear. The authors should give more explanations of the advantage of the proposed quantified metric. 
+ The notations in Section 3.3 (which is the most important section) may cause confusion. For example, is it a sample x_i or example x_i? Is \hat{p}_{i} a vector or scalar? How about \hat{p}_{ik}?
+ Only four baseline methods are used in experiments. I think it is better to add more.
summary_of_the_review: I think the research problem is interesting. However, I think the technical novelty is marginally significant, and the motivation of the proposed metric may not be clear.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes PUUPL, an uncertainty-aware pseudo-label selection method for positive-unlabeled (PU) learning. To improve the performance of pseudo-labeling, the authors suggest using the epistemic uncertainty (the difference between the entropy of the mean prediction and the mean of entropies of each prediction). In the experiments, PUUPL outperformed the existing state-of-the-art PU learning methods.
main_review: The idea of using the epistemic uncertainty in PU learning is interesting. However, it is not clear the technical novelty and it lacks a simple baseline against uncertainty-aware pseudo-labeling.

---

What is the difficulty to introduce uncertainty-aware pseudo-labeling techniques to PU learning? It seems that simply applying uncertainty-aware pseudo-labeling technique results in the proposed method. That is, the technical novelty of the proposed method seems not strong. If the authors can clarify non-trivial technical contributions, it would help understand this paper well.

Comparison with uncertainty-unaware pseudo-labeling techniques helps understand the superiority of the proposed method. It is expected to illustrate that considering uncertainty is important when we use pseudo-labeling in PU learning. For example, nnPU with ordinary pseudo-labeling techniques could be compared with nnPU with the proposed uncertainty-aware pseudo-labeling.

In Section 3.2, there is the sentence "we assume cleanly labeled (positives and negatives) validation and test sets (see Section 3.6) as usual in literature." I checked the cited papers quickly, but could not find such a setting. If negative samples are available, we can use them (e.g., a half of them) in training. And, it might result in better performance thanks to negative samples. So, the assumption sounds not practical. Thus, the results reported in the experiments are not good evidence of the effectiveness of the proposed method. But, there is a chance that I miss the sentences supporting the authors' assumption. If the authors can show the sentences describing the assumption about the validation set, I will confirm it.

=== After rebuttal ===  
Thank you for your responses.  
The idea of using the epistemic uncertainty in PU learning is interesting. So, I hope that the authors will resolve my and the others' concerns in the next venue.
summary_of_the_review: This paper introduces uncertainty-aware pseudo-labeling techniques to PU learning. Although the idea is interesting, the technical novelty and the performance improvement seem marginal. In addition, according to the paper, it assumes cleanly-labeled validation data (we can access negative samples) are available, which is a strange assumption in PU learning.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the PU learning problem. It proposes a two-step approach that can estimate the pseudo-label uncertainty so that more reliable pseudo-labels can be assigned, which improves the predictive performance. The proposed estimation method is different from previous methods. 
main_review: This paper studies the PU learning problem. It proposes a two-step approach that can estimate the pseudo-label uncertainty so that more reliable pseudo-labels can be assigned, which improves the predictive performance. The proposed estimation method is different from previous methods. 

My main concern is that two key assumptions are made in the paper, which are not realistic. The first one is that a positive and negative labeled validation set is available and used as the validation set for early stopping. The second is that the proportion of positive and negative samples is known. Both are problematic. The paper justifies the assumptions by listing some prior work that made one or the other assumption. However, I don’t think the fact that some prior work using these assumptions makes the assumptions right. In a real-life situation, these numbers are not available. There are many approaches that do not make any of the assumptions, but they are not compared, e.g., the recent system PAN, “Predictive adversarial learning from positive and unlabeled data,” AAAI-2021. 

Prior work has used several techniques to score the unlabeled set and then label them. Since the paper that proposed the two-step approach, “partially supervised classification of text documents,” ICML-2002, several approaches have been proposed to pseudo-label the unlabeled set. I understand that the proposed approach is different and probably more advanced than those older approaches, but it will be more complete to have an experimental comparison with some earlier approaches to show the advantage of the proposed method. 

The paper is easy to understand. 

summary_of_the_review: Although the proposed estimation method is different from existing ones, my main concern is that the two assumptions are unrealistic. They actually run against the very setting of PU learning. Without the two assumptions, the proposed method would have problems. The paper should also compare with some earlier methods.
review_title: null

Meta-review: This paper proposes a new loss-agnostic PU learning method based on uncertainty-aware pseudo-label selection.
I would like to thank the authors for their feedback to the initial reviews, which clarified many uncertain issues and improved our understanding of the current paper.
Nevertheless, even if the pseudo-labeling technique was applied to PU learning for the first time, given that it is a common practice in many weakly supervised learning tasks, the technical novelty is rather limited.

Therefore I cannot recommend acceptance of this paper.
Decision: Reject