Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies positive and unlabeled (PU) learning, which apparently is an important problem in machine learning. In many existing researches, the PU classifiers assumes that the irreducibility assumption holds on input data, which, though, may not always be true in many real-world applications. As a result, the learn classifier might not work well due to estimation bias on data prior. To address this problem, this paper proposes to construct a transformed the probability distributions of input data by a re-grouping operation, such as the positive prior cannot be a component (or called as a support) of the negative prior. Good experiment results have been obtained to support the proposed method.
main_review: +++Strength:
1. PU learning is apparently an important machine learning problem, which has the potential to be used in many real-world applications.
2. The analyses on the prior studies appear reasonable.
3. Experiments have been conducted to support the proposed method and the results appear good.


---Weakness:
1. While the proposed probability distribution transformation makes sense to avoid relationship between the prior of positive data and that of negative data, the proposed method appear pretty simple. This might not be a weakness, but methodology is not a strength of this paper.
2. A big gripe of this paper is its writing. There exist too many grammatical/syntactical mistakes. Here I list few in the below.

Page 2: "To let these distributional-assumption-free methods can be used to identify class-prior $\pi$, $\kappa^*$ must ...". This sentence can be definitely written in a better way.

Page 2: "Because the irreduciblility assumption is impossible to check without making any assumption on $P_n$. Thereby, in ...". This is an incomplete sentence, which is a mistake.

Page 3: "... by creating a new auxiliary distribution $P_{p'}$ always guaranteeing the irreduciblility assumption..." Here using an attributive clause is much better than using a present participle.

Page 4: "... i.e., there exists a set can achieve the minimum 0, ..." This sentence is simply wrong.

3. As we all know, deep learning has become the main stream of machine learning in the past decade, which is particularly true in ICLR. Many research on PU learning using deep neural networks have been proposed and demonstrated good performance. As a result, without comparing to such methods, it is harder for reader to appreciate the effectiveness of the paper.
summary_of_the_review: This paper studies positive and unlabeled (PU) learning by constructing a transformed the probability distributions of input data by a re-grouping operation, such as the irreduciblility assumption holds on the data to the subsequent classifier. Overall, this paper is clearly written. But because of the following concerns, also mentioned above, I suggest the authors to further improve this paper before publication:
1. Please proofread this manuscript carefully to improve the writing.
2. More comprehensive experiments to compare deep neural network based PU learning should be added to convince the readers.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the prior $\pi$ in PU learning. More specifically, it studies the estimation method of the prior $\pi$ in a more general case, i.e., without the irreducibility assumption. When unsatisfied, such an assumption will often lead to the overestimation of $\pi$. However, current class-prior estimation methods are usually based on the irreducibility assumption. The authors propose a new CPE problem based on a new auxiliary distribution that always satisfies the irreducibility requirement. The technique is called regrouping, which transforms the true distributions to  Both theoretical and experimental results are included in the paper, showing the validness of the proposed ReCPE method. 
main_review: **Strengths**
- The paper is well-written and easy to follow. The mathematical contents are not hard to parse and examples are given. 
- This work is well-motivated and novel. The paper addresses the overestimation problem of $\pi$ in PU learning by proposing a regrouping technique.
- Both the theoretical and experimental proofs are given. They help support the authors' claims.
- Although the gain is small in absolute value, the large number of repeat experiments makes it statistically significant (and statistical tests are included). 

**Weaknesses and Clarifications**
- The authors exclusively compare the estimation of $\pi$. While I understand that the method is aimed at estimating the prior, I think that it would be beneficial if the authors can provide the PU learning results with different priors (fixed, CPE, ReCPE, etc. ). A simple nnPU would suffice. 
- Many "irreducibility" has a typo ("irreduciblility").

summary_of_the_review: The authors give an insightful demonstration of limitations of existing CPE methods, and also give feasible solutions to the limitations. Theoretical proofs are given, and experiment results show significant superiority of the proposed method compared to existing CPE methods. So I believe this work would be beneficial to the community and therefore give a score of 8.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies a relatively little-concerned problem, that is, the problem of class-prior estimation(CPE) in PU scenes. The authors concluded that the existing CPE methods are based on a critical assumption that the support of the positive data distribution cannot be contained in the support of the negative data distribution. However, in actual scenarios, this assumption may not be satisfied. It is also difficult to prove that a certain data set satisfies this assumption. Existing CPE methods will systematically overestimate the class prior when the data does not meet the critical assumption. To remove the assumption to make CPE always valid, a strategy called Regrouping CPE (ReCPE), which builds an auxiliary probability distribution, are proposed so that the support of the positive data distribution is never contained in the support of the negative data distribution. Theoretically, this method can give a more accurate estimate regardless of whether the dataset meets the assumption. The authors also proved the effectiveness of the method through a series of experiments.
main_review: strengths:
1.Compared with the previous class-prior estimation methods, the authors consider the practical problems in the actual scene, that is, the distribution of the data set generally does not meet the ideal conditions. This is a relatively practical research topic.
2.In general, this article is well written, which is reflected in the clear organizational structure and concise and easy-to-understand presentation. The summary of related work is very informative.
3.The experimental design is generally reasonable, and the experiments with real -world data sets give more substantial results.
weaknesses:
1.Why is case-control PU learning more general than censoring PU learning? There is no clear explanation about this issue, please give a more detailed explanation in the part Introduction.
2.Regarding the main research scenarios in this article, that is, "assuming that the support of the positive class-conditional distribution is contained in the support of the negative class-conditional distribution", the article gives a theoretical definition. Can the author give a more intuitive explanation? Please give some more examples of this in actual problems. In what scenarios will such distributed data appear?
3.Why choose the most likely positive sample among the unlabeled samples? If the support of the positive class-conditional distribution is contained in the support of the negative class-conditional distribution, then intuitively these contained positive samples should look more like a negative sample.
4.In the ReCPE algorithm, the author proposes to find the most unlikely negative data among the unlabeled samples. The way to achieve this step is to train a binary classifier with the unlabeled sample and positive sample by treating unlabeled sample as a negative sample. When there are many positive samples in unlabeled samples, the "least unlikely negative sample" selected by the binary classifier obtained by this method is unreliable. Thereby the reliability of the next steps cannot be guaranteed.
5.The author mentioned that 10% was chosen as the value of the hyperparameter p in all experiments of the data set, but this approach seems to be contrary to Theory 1. This approach is too casual, I am worried that the new class-conditional distribution, which is obtained by transporting the probability mass of the set A from the negative class to the positive class, are really guaranteed to satisfy the irreduciblility assumption?
6.Why assume that the positive class-priorπis close to the new positive class-priorπ’? If the support of the positive class-conditional distribution is strongly contained in the support of the negative class-conditional distribution, The two class-priors should be quite different.
7.In general, although this research scenario is very meaningful, the ReCPE method proposed in this article is too strategic. It only proposes solutions intuitively, and there seems to be no strong guarantee in theory. The authors may be able to strengthen this article from this perspective.
Minor comments: The spelling of "real word" in Note 2 on page 7 is incorrect.


summary_of_the_review: This article focuses on practical issues, and its writing style is clear and easy to read. The structure of the article is reasonable, and the experiments are full and practical.
However, there are many parts that need to be improved in the method part, which are listed in Main review. In general, although this research scenario is very meaningful, the ReCPE method proposed in this article is too strategic. It only propose solutions intuitively, and there seems to be no strong guarantee in theory.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of class-prior estimation (CPE) in positive-unlabeled (PU) learning. The authors consider that the existing methods usually fail if the data distribution dissatisfies the irreducibility assumption. To address this problem, the authors introduce a method named Regroping CPE (ReCPE) which tries to transform the original data distribution to an auxiliary data distribution, such that the produced distribution always guaranteeing the irreducibility assumption. 

A practice implementation is proposed by firstly training a binary classifier and then picking some samples whose outputs are mostly dissimilar to the negative examples as the pseudo positive samples. The proposed methods can be considered as a pre-processing method and used together with any CPE method. The experiments on synthetic data show the intuition of the proposed ReCPE and the results on real-world data show the effectiveness of the proposed method when it is combined with other CPE methods.
main_review: Strengths:

1. This paper introduces a particular situation in which the irreducibility assumption cannot be satisfied, which has not been studied (even considered) in previous PU learning literature.

2. The idea of "regrouping" is novel. The authors give a very simple and intuitive example in Subsection 3.1 as the motivation of the proposed method, which is very interesting and helps the readability of this paper. In Subsection 3.3, some theoretical results are introduced and these results can help the readers to understand the proposed "regrouping' idea.

3. The experimental results on synthetic data validate the correctness of the theoretical part and show the effectiveness of the proposed method in "reducibility" setting.

Weaknesses:

1. The experiments on real-world data use same  hyper-parameter $p$ across all datasets. Considering that this paper actually aims to address the hyper-parameter (class-prior $\pi$) selection problem and the proposed method introduces another hyper-parameter $p$ (which seems to have significant impact to the performance), the parameter sensitivity analysis is needed in experiment. 

2. More related work about "irreducibility" can be introduced before Section 3. I don't know is there any solution for the irreducibility problem in mixture proportion estimation after reading the whole paper.

Other comments:

1. Is the indicator function really needed in "Convergence analysis" part? It seems that $\mathbb{1}_{\{h(x)=1\}}$ is equivalent to $h(x)$.

2. Can you explain that why "In this case, regrouping will generally not change $P_p$ but only increase number of examples in $P_p$".

3. Typos: "diffferent" --> "defferent", "the trained classifier a h" --> "the trained classifier h".
summary_of_the_review: This paper considers a particular PU learning setting where the irreducibility assumption is not satisfied, and gives a new method to address this problem. The proposed method can be combined with any class-prior estimation method and the experimental results show positive effect of the proposed practical implementation. This paper is well-motivated and novel, and it will be better with the weaknesses addressed.
review_title: null

Meta-review: This paper received a majority vote for acceptance from reviewers and me. I have read all the materials of this paper including manuscript, appendix, comments and response. Based on collected information from all reviewers and my personal judgement, I can make the recommendation on this paper, *acceptance*. Here are the comments that I summarized, which include my opinion and evidence.

**Research Motivation and Problem**

This paper is well motivated by the agnostic of CPE assumption that the support of the positive data distribution cannot be contained in the support of the negative data distribution. To tackle this problem, the authors built an auxiliary probability distribution such that the support of the positive data distribution is never contained in the support of the negative data distribution.

**Technical Contribution**

The technical part is simple and clear. The regrouping idea is also easy to implement. The theoretical justification is a good complement of the proposed ReCPE algorithm.

**ReCPE does not affect its base if the assumption already holds**

The authors employed the synthetic datasets to verify this point. This is a plus.

**Experimental Results**

The authors demonstrated their ReCPE algorithm can be used as a booster on seven base PU classifiers.

**Presentation**

The presentation has been much improved with the guidance of one reviewer. But I found two extra minor ones. (1) "PU Learning" -> "PU learning" at the beginning of the second paragraph on Page 1. (2) Two typos in "9 real word datasets." on Page 7 -> "9 real-world dataset.", where the footnote should be placed after the period.

**Layout**

(1) Too many lines in Table 1. It is suggested to remove the horizontal lines among the same dataset, (2) Appendix should go after the main manuscript, rather than a separate file.

No objection from reviewers was raised to again this recommendation.
Decision: Accept (Poster)