Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed to reduce the bias of classifiers learned on PU data by a heuristic mixup technique that partially selects the unlabelled instances and mixes them up with the positive instances around the decision boundary learned with PU data. The experimental results demonstrate the effectiveness of the heuristic mixup technique.
main_review: Strengths
+ The motivation of this paper is strong, and the research problem is interesting. The authors found a phenomenon that the decision boundary learned with PU data tends to shift to the positive side compared to the boundary learned with PN data. Because shifting of the decision boundary leads to the bias of learned classifiers, the authors try to reduce the bias by exploiting a heuristic mixup technique.
+ The proposed method has strong empirical performance. The experimental results on different datasets show that the proposed method can consistently outperform the state-of-the-art PU learning methods.
+ This paper is well structured and easy to follow. 

Weaknesses
+ It seems that “directional boundary” is not commonly used in existing papers. To avoid confusion, it is better to add a specific definition or change to other words. I assume that it is the same as the “decision boundary”.   
+ To select the marginal pseudo-negative instance, predictive scores can be different by employing different learning models. It is better to add some discussion that what should pay attention to when choosing the learning model for estimating predictive scores.
+ In the abstract, “For the unlabelled instances with ambiguous predictive results…”. The word “ambiguous” is not clear. I think that the authors should high-levelly explain the word “ambiguous”.
summary_of_the_review: This paper has a strong motivation as I mentioned above. I like the idea of using a heuristic mixup technique for reducing bias in PU learning. I think the idea that adds heuristic to mixup could be extended to other weakly supervised machine learning tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies an interesting weakly supervised binary classification problem called positive and unlabeled (PU) learning. The authors propose a novel PU learning method inspired by the boundary deviation phenomenon observed in experiments. Specifically, a new mixup method is proposed, which selects the mixup partners for unlabeled examples heuristically to obtain more correct supervised signals. Extensive empirical results, including ablation study and sensitiveness analysis, are provided to evaluate the proposal.
main_review: The idea that achieves data augmentation and supervision correction by refining mixup is interesting and easy to implement. This paper is well-organized and well-motivated. Extensive experiments on several datasets and ablation studies prove the effectiveness of the proposed method and its components. Exhaustive discussion of related work is presented. 

Concerns:
1.	The method is evaluated on several image classification datasets. It will be better to do experiments on practical PU learning problems, such as product recommendation and medical diagnosis, as mentioned in the paper. The data distributions of these practical problems may be different from image dataset. Thus, the conclusions may be different. 
2.	MixPUL is a recently proposed PU learning method, which also applies mixup. It may be better to compare with MixPUL and discuss the difference.
3.	What is the main difference between positive and unlabeled (PU) learning and anomaly detection?
4.	Does the proposed method also work well with Cutmix[1] augmentation?

[1] Cutmix: Regularization strategy to train strong classifiers with localizable features, Yun et al 2019

summary_of_the_review: The paper proposes a new positive-unlabeled learning method which achieves data augmentation and supervision correction simultaneously by refining mixup strategy. The method is well-motivated by empirical findings. Extensive experiments and ablation study demonstrate the effectiveness of the proposed method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a variant of the mixup technique for positive-unlabeled learning. Based on the observation that the learned PU boundary tends to deviate towards the positive side, the authors suggest selecting samples between the learned PU and supervised boundaries. The proposed P3MIX method and its variant improve the classification performance of PU learning.
main_review: The ideas of marginal pseudo negative instances and candidate mixup pool are interesting. Since PU learning is different from ordinary supervised learning, it would be reasonable to develop a specific mixup approach for PU learning. 

---

It is not really clear to me that when to use the proposed early-learning regularization. If possible, it is nice to write when to use early-learning regularization (and also, pseudo-negative instance correction) in Algorithm 1 or create Algorithm 2 including the techniques in Section 2.2 (Robustness).

It is expected that the proposed method would be compared with one of the PU learning methods, e.g., nnPU, with the ordinary mixup technique. Probably, we can respectively augment P and U data by a mixup technique and then use the existing PU learning methods. This approach can be regarded as a simple baseline against the proposed method. It is expected that such a simple baseline would be included in the experiments.

In Section 3.4, it is reported that \beta \in {0.8, 0.85, 0.9, 0.95} is better on the basis of the experiments. But, it is not clear the relation between the class-prior probability and \beta. When we set \beta=1/(2\pi) and the sigmoid loss is used, minimization of Eq. (1) is equivalent to minimization of the objective function of uPU. It is also known that a rough class-prior estimation is sufficient when the true-class prior is known to be large (du Plessis et al., NeurIPS2014). The suggested \beta might not be generalized to the settings other than the settings of this paper. That is, the suggest \beta might correspond to the rough estimation of the class-prior, by chance. To support the effectiveness of the suggested \beta, it would be necessary to show the comprehensive experiments to illustrate the relation between \beta and the class-prior.

In Section 2.1, it is not obvious the rule of assigning the generated samples into \hat{X}\_p or \hat{X}\_u. Since this part is very important in the proposed method, without explicitly-written rules, it would be difficult to reproduce the results from this paper.

In Figure 1, it seems that "disambiguation-free" is labeled as "discrimination-free."
summary_of_the_review: This paper proposes a mixup method specialized for PU learning. The idea of marginal pseudo-negative instance estimation is interesting. In the current manuscript, there are however several unclear points and it lacks a simple baseline in the experiments. Having such a simple baseline, the advantage of the proposed method will become more clear.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors focus on the problem of positive and unlabeled learning. They show an interesting phenomenon, where the learned PU boundary tends to deviate the supervised boundary towards the positive side when treating unlabeled examples as pseudo-negative examples. The phenomenon may imply there are a number of marginal pseudo-negative examples that are more likely to be positive but labeled as negative. Based on this, the paper proposes a PU learning approach building on a novel heuristic mixup technique, which can achieve both data augmentation and supervision correction. They also present many empirical results to show the superior performance comparing with SOTA PU learning methods.
main_review: Strengths:
1. The paper is well written and easy-to-follow.
2. The motivation and the proposed method are clearly described. Especially, the observed phenomenon and the key idea of correcting marginal pseudo-negative examples with a heuristic mixup technique is interesting, and may potentially be useful for semi-supervised learning.
3. The proposed PU learning approach does not require explicit computation of a class prior.
4. The experiments are well-conducted, and comprehensively compared to recent SOTA methods. Ablations and sensitivity analysis are also shown. 

Weakness:
1. Does the proposed approach rely on the "selected completely at random" (SCAR) assumption? 
2. Showing sensitivity analysis on \alpha would be better.
3. Why is the size of the candidate mixup pool fixed as 100? Bigger candidate mixup pool, better performance?
4. Could the authors release the code?

summary_of_the_review: The problem is interesting and the proposed method is promising. I vote for accepting this paper.
review_title: null

Meta-review: Mixup is very helpful when the training sample is scarce or has weak supervision. The paper studies how to adapt mixup to positive and unlabeled (PU) learning, a representative weakly supervised learning problem. By studying the specific properties of PU learning, the authors propose the concept of marginal pseudo-negative instances, which are more likely to be positive but actually annotated by negative. A novel mixup variant has been proposed for PU learning by mixuping the marginal pseudo-negative instances with the positive instance around the classification boundary. The effectiveness has been empirically shown.
Decision: Accept (Poster)