Title: Learning from positive and unlabeled examples -Finite size sample bounds
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learning Theory

PU Learning

Domain Adaptation
Abstract: PU (Positive Unlabeled) learning is a variant of supervised classification learning in which the only labels revealed to the learner are of positively labeled instances. PU learning arises in many real-world applications. Most existing work relies on the simplifying assumption that the positively labeled training data is drawn from the restriction of the data generating distribution to positively labeled instances and/or that the proportion of positively labeled points (a.k.a. the class prior) is known apriori to the learner. This paper provides a theoretical analysis of the statistical complexity of PU learning under a wider range of setups. Unlike most prior work, our study does not assume that the class prior is known to the learner. We prove upper and lower bounds on the required sample sizes (of both the positively labeled and the unlabeled samples).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides sample complexity bounds in a general *positive unlabled* learning setting. Here the learner has access to two different distributions (from potentially known families of distributions). one generating the unlabeled data and the second generating positively labeled data. They study agnostic and realizable PAC learning variants, which previously appeared in the literature. Their main contributions are novel upper bounds when the class distribution is not known (which appears to be a standard assumption in this line of work), novel tighter lower bounds in various settings, and a novel combinatorial parameter, the *claw number*, allowing some lower bounds on the number of required unlabeled examples.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* summarizes and improves various previous results on PU learning.\n\nThe main weakness I see that this paper would benefit greatly from a deeper discussion with related work and if the results would be put into a broader context. See my questions below. **When these are addressed appropriately, I am happy to raise my score.**\n\nSome more smaller weaknesses:\n* some of the proposed/improved bounds are still not tight. While the claw number is interesting, it's unclear whether it gives the optimal tight bounds. See also questions below.\n* it's sometimes not fully clear when results are stated for general concept classes and when they focus on particular domains ($X=[0,1]^k$) and/or assume large margin etc. \n\nSmall comments:\n* there are appear to be various smaller formatting issues. The table 1 too large (and consider using booktabs and removing the vertical lines). Some typos. Usage of dashes seems wrong.\n\nQUESTIONS:\nYour proposition 4 strongly reminds me of learning with one-sided error [7]. There it is the same situation that learning is possible iff the intersection-closure has finite VC dimension. Is there any relationship here?\n\nPlease compare the claw number to existing combinatorial parameters beyond VC Dimension (e.g., [1,2], even invariants from convex geometry like Helly, Carathéodory, Radon numbers [3] etc). Are there any known related results? This could provide more context on the relevance of the claw number and perhaps even lead to tighter results.\n\nThe claw number based lower bounds are not fully matching the upper bounds, in particular, there is a remaining gap of $h\\le d$. Do you think the upper bound can be improved from vc to claw number or vice versa? Or is the gap in general unavoidable (using these parameters)?\n\nI think it is very interesting that you manage to get lower bound on the number of unlabeled examples. In related settings like semi-supervised [5,6] and and transductive learning [4] it is commonly known that unlabeled samples do not really in general.\n\n\n\n[1] Bousquet, Olivier, et al. \"Proper learning, Helly number, and an optimal SVM bound.\" COLT 2020.\n[2] Hanneke, Steve. \"The Star Number and Eluder Dimension: Elementary Observations About the Dimensions of Disagreement.\" COLT 2024.\n[3] Kay, David, and Eugene W. Womble. \"Axiomatic convexity theory and relationships between the Carathéodory, Helly, and Radon numbers.\" Pacific Journal of Mathematics 38.2 (1971): 471-485.\n[4] Tolstikhin, I., & Lopez-Paz, D. (2016). Minimax lower bounds for realizable transductive classification. arXiv preprint arXiv:1602.03027.\n[5] Balcan, M. F., & Blum, A. (2010). A discriminative model for semi-supervised learning. Journal of the ACM (JACM), 57(3), 1-46.\n[6] Darnstädt, M., & Simon, H. U. (2011). Smart PAC-learners. Theoretical Computer Science, 412(19), 1756-1766.\n[7] Natarajan, Balaubramaniam Kausik. \"On learning boolean functions.\" Proceedings of the nineteenth annual ACM symposium on Theory of computing. 1987\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical analysis of PU learning. First, it identifies different PU learning settings, including cases with covariate shift of positive data. Then, it provides a detailed analysis of the sample complexity of these settings. The paper offers new insights into understanding PU learning with distribution shifts.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The theoretical analysis is comprehensive and well-executed.\n\n- The studied problem, i.e., the sample complexity of different PU learning settings, is important to the literature.\n\n- The presented theoretical results are interesting and novel, as shown in Table 1.\n\n## Weaknesses\n\n- The paper is novel for the PU learning literature. However, most of the analysis seems dependent on Ben-David and Urner (2012). Therefore, the authors should justify how their proposed setting differs significantly from that of Ben-David and Urner (2012). Additionally, could the authors discuss the differences with [1]?\n\n- The theoretical analysis of the paper is good and comprehensive. The study of different PU learning settings is also valid and sound. However, I am not familiar with the sample complexity analysis of transfer learning in Ben-David and Urner (2012), so it may be difficult for me to justify the soundness of the proposed theory.\n\n- Since the paper contains many theoretical analyses, it is important to write them in a clearer way so that readers can understand them.\n\n- The proposed assumptions need further discussion and justification. Additionally, it is unclear how the theoretical analysis can be applied to real-world applications. There are no experimental results in the paper.\n\n[1] Coudray O, Keribin C, Massart P, et al. Risk bounds for positive-unlabeled learning under the selected at random assumption[J]. Journal of Machine Learning Research, 2023, 24(107): 1-31.\n\nQUESTIONS:\nPlease see \"weaknesses\".\n\nLIMITATIONS:\nThe limitations should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a comprehensive theoretical analysis of the statistical complexity of PU learning. In contrast to most prior work, this study does not require the class prior, α, to be known by the learner. The paper explores both the realizable and agnostic learning paradigms across four distinct data-generating settings: Selected Completely at Random (SCAR), Selected at Random (SAR), Positive Covariate Shift (PCS), and Arbitrary Positive Distribution Shift (APDS) .The main contributions include: 1. Providing nearly tight upper and lower sample complexity bounds for the SCAR and SAR settings. 2. Establishing the first finite-sample generalization guarantees for the PCS and APDS settings.3. Introducing a novel combinatorial parameter, the \"claw number,\" to characterize the lower bound on unlabeled sample complexity.\n4. In the agnostic setting with an unknown α, proving that a learner can achieve an error bound proportional to the optimal error and showing that this proportionality factor is theoretically tight.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The most significant contribution of this work is its systematic treatment of the unknown class prior setting. This addresses a long-standing bottleneck in PU learning theory and brings the theoretical analysis closer to real-world application scenarios.\n2. The paper is impressively comprehensive, systematically analyzing settings from the well-studied SCAR to the more challenging PCS and APDS. By providing both upper bounds (algorithmic guarantees) and lower bounds (inherent problem difficulty) for several key settings, the analysis is remarkably complete and convincing.\n3. The introduction of the \"claw number\" is an interesting conceptual innovation. Furthermore, providing the first finite-sample guarantees for the PCS and APDS settings, where such results were previously lacking, represents a significant theoretical breakthrough.\n\nWeaknesses:\n1. To achieve theoretical guarantees in more challenging settings like PCS, the paper introduces new, strong assumptions, such as the γ-margin assumption. This leads to a sample complexity for unlabeled data that is exponential in the data dimensionality.\n2. This is a purely theoretical paper with no experimental section. While the theoretical contributions are self-contained and sufficient, the absence of experiments on synthetic or real-world datasets makes it difficult to assess the practical performance of the proposed algorithms (e.g., Algorithm 1), their sensitivity to hyperparameters, and the tightness of the theoretical bounds in practice.\n\nQUESTIONS:\nThe exponential dependency on dimensionality in your bound for Algorithm 1 seems to be a significant practical barrier. Could you comment on the tightness of this bound? Are there avenues for improvement, or could you outline specific scenarios where this complexity would be manageable?\n\nLIMITATIONS:\nAs noted, the results for challenging distribution shifts rely on a margin assumption, which limits the applicability of these specific conclusions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study revisits a basic issue in PU learning. The authors define PU learnability and discuss under which conditions we can obtain a desirable classifier. Their analysis investigates each setting in PU learning, including those considered by Elkan and Noto (2008) and du Plessis et al. (2015), and provides a comprehensive framework.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors address a fundamental problem in PU learning. Although their problem setup is interesting, I have several questions regarding their results. I list the points below:\n- I believe the problem setup in Elkan and Noto (2008) and its subsequent works differs from that in du Plessis et al. (2015) and related studies. The former is referred to as censoring PU learning, while the latter is known as case-control PU learning. The difference is not related to knowledge of the class prior. Even when the class prior is known, the data-generating processes are distinct. It seems that the authors' setting largely follows that of du Plessis et al. (2015). If they were to follow the setting in Elkan and Noto (2008), we would usually assume access only to a single dataset from which labels can be observed. For an explanation of this difference, see Niu et al. (2016).\n- Theorem 14 appears to be somewhat trivial. The Bayes classifier is usually expressed as $p(y = 1 \\mid x) = \\alpha, p(x \\mid y = 1) / p(x)$, where $p(x \\mid y = 1)$ is the class-conditional density of $x$, $p(x)$ is the marginal density, and $\\alpha$ is the class prior. This density ratio $p(x \\mid y = 1) / p(x)$ can be estimated via density-ratio estimation methods. Consequently, it is rather obvious that we can obtain the result stated in Theorem 14, because we can recover the ordering of $p(y = 1 \\mid x)$; the exact classification boundary is then determined by the class prior. For the relationship between density-ratio estimation and PU learning, see Kato and Teshima (2021).\n\nReference:\n- Niu, du Plessis, Sakai, Ma, and Sugiyama (2016). Theoretical Comparisons of Positive-Unlabeled Learning against Positive-Negative Learning. NeurIPS.\n- Kato and Teshima (2021). Non-Negative Bregman Divergence Minimization for Deep Direct Density Ratio Estimation. ICML.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}