Title: Probably Approximately Precision and Recall Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: PAC learning

multi-label learning

sample complexity
Abstract: *Precision* and *Recall* are fundamental metrics in machine learning tasks where both accurate predictions and comprehensive coverage are essential, such as in multi-label learning, language generation, medical studies, and recommender systems. 
A key challenge in these settings is the prevalence of one-sided feedback, where only positive examples are observed during training—e.g., in multi-label tasks like tagging people in Facebook photos, we may observe only a few tagged individuals, without knowing who else appears in the image. To address learning under such partial feedback, we introduce a Probably Approximately Correct (PAC) framework in which hypotheses are set functions that map each input to a set of labels, extending beyond single-label predictions and generalizing classical binary, multi-class, and multi-label models. Our results reveal sharp statistical and algorithmic separations from standard settings: classical methods such as Empirical Risk Minimization provably fail, even for simple hypothesis classes. We develop new algorithms that learn from positive data alone, achieving optimal sample complexity in the realizable case, and establishing multiplicative—rather than additive—approximation guarantees in the agnostic case, where achieving additive regret is impossible.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAs traditional learning paradigms fail when only positive labels are observed, a new PAC-learning framework is presented to optimize precision and recall losses under partial feedback. This setup in practice applies to multi-label learning tasks, such as tagging people in photos, where only some tagged names are observed, but not the full set of people.  The paper shows that optimizing precision is fundamentally more difficult than optimizing recall in this setting because of the absence of negative feedback. Two algorithms are presented, the maximum likelihood-based algorithm, and the surrogate loss minimization algorithm, which are shown to achieve upper and lower bounds. Two settings are analyzed: i. Realizable Setting, when the true hypothesis is in class H; ii. Agnostic Setting, when the true hypothesis not necessarily in H.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n•\tOriginality: The paper tackles a novel and important learning setting, which is learning under one-sided (positive-only) feedback. This is highly relevant in real-world applications like multi-label learning and recommender systems. The formalization of this setup into a new PAC-learning framework is a meaningful and original contribution.\n•\tSignificance: By highlighting fundamental differences between learning precision and recall under partial supervision, the work sheds light on overlooked aspects of supervised learning theory. In particular, the hardness of estimating precision loss without negative labels is both practically and theoretically significant.\n•\tTheoretical Quality: The paper demonstrates a high level of rigor in constructing new learning guarantees. It provides both upper and lower bounds, separating realizable and agnostic settings, and analyzes both scalar-loss and Pareto-loss objectives. These contributions form a solid theoretical foundation.\n•\tAlgorithmic Contributions: Two learning algorithms, based on maximum likelihood estimation and surrogate loss minimization, are clearly described and motivated. They show that learning from only positive feedback is tractable in some settings, but fundamentally limited in others.\n•\tClarity of High-Level Ideas: Despite the complex theoretical nature, the paper does a good job of motivating the problem and building intuition through illustrative examples (e.g., the indistinguishable hypotheses with vastly different precision losses).\nWeaknesses:\n•\tClarity for Non-Theorists: While the high-level concepts are well-motivated, the technical sections, especially the proof sketches may be hard to parse for readers not familiar with PAC learning theory or sample complexity analysis. A bit more intuitive commentary around key lemmas and theorems would help make the results more broadly accessible.\n•\tNo Empirical Results: The paper is entirely theoretical. Although this is acceptable for NeurIPS theory tracks, some small-scale synthetic experiments (even toy data) could help reinforce the practical relevance and intuition behind the proposed hardness results or algorithmic strategies.\n•\tAssumptions on Output Size Knowledge: Some results depend on knowing or bounding the output size n_G^{target}, which may be non-trivial in practice. The practical implications of this assumption are not fully explored.\n•\tPractical Impact Path Not Fully Developed: Although the authors claim relevance to real-world domains, there is a gap between the theory and potential practical implementation guidance. It would be helpful to bridge that more directly, even if via discussion or future work.\n\nQUESTIONS:\n1.\tOn Practical Impact and Real-World Feasibility: Could the authors elaborate more on how their algorithms might be adapted to real-world tasks (e.g., recommender systems or tagging applications)? If possible, implementation on some toy datasets, including some commentary on feasibility under partial observability assumptions. These would strengthen the paper’s significance.\n2.\tClarification on Output Size Assumptions: Several results depend on the assumption that the output size n_G^{target} is bounded or known. In many practical applications, this is not directly available. Could the authors discuss how their methods behave when this size is unknown or only loosely bounded?\n3.\tIntuition and Use Cases for Scalar vs. Pareto Loss: The distinction between scalar-loss and Pareto-loss objectives is theoretically clear. Could the authors clarify what kind of applications might naturally benefit more from optimizing for one versus the other?\n4.\tLower Bounds Interpretation: The paper proves lower bounds like \\alpha ≥1.05 for the scalar loss and impossibility of achieving additive error for Pareto loss. Could the authors comment on whether these lower bounds are tight in any known cases? Are there scenarios where existing heuristics approach these bounds?\n5.\tImproving Accessibility: While the main ideas are well-explained, could the authors consider adding more intuitive commentary or diagrams near the formal results to guide non-theory readers (e.g., explaining the significance of each key theorem in words, or linking them to the motivating examples)?\n\nLIMITATIONS:\nAuthors only discussed the theoretical limitations of the paper. As this is almost purely a theoretical paper, the authors did not discuss any societal impact aspects of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on a learning setting where the learning data consists of partial feedback and the goal is to learn to predict a set of labels. Specifically, it considers two settings, realizable setting and an agnostic setting, depending on whether the ground truth hypothesis is in the hypothesis class. It further presents the theoretical results on the learnability for both settings, including both positive and negative results.\n\nSTRENGTHS AND WEAKNESSES:\nThe problem of learning from partial labels considered in this work is interesting since such scenarios can happen when labeling is costly. However, a few concerns arise when reading this work:\n- the assumption that the hypothesis class is finite weakens the theoretical results in this work since it doesn't hold in practice. \n- missing related work: the problem setting is closely related to the PU learning setting [1,2] where only positive labels are observed.\n- in this work, it compares with ERM a lot but it's unclear to me how ERM is defined in this work and why ERM would fail in this case. In the paragraph from Line 121, the authors claim that the reason ERM would fail is because two hypotheses can have the same recall loss but different precision loss, which is further illustrated by Example 1. However, in example 1, recall losses are not impossible to compute since it requires the ground truth label set {u_1, ..., u_n}, which is not accessible since the training datapoints only reveal one label, while only precision loss is possible. Further, precision loss can actually distinguish the two hypotheses in this case, so ERM could possibly work. Can the authors elaborate more on this?\n- Theorem only proves about PAC learnability but not the efficiency of the proposed algorithm. For the first algorithm maximum likelihood, the idea is standard and not really a technical contribution; also, it doesn't seem that the optimization problem can be efficiently solved.\n- the definitions of G' and G'' in algorithm 2 are unclear. Can the authors provide an example to show how this vector is computed?\n- minor: at line 189, should the domain be X*Y instead of X^2?\n\n[1] Bekker, Jessa, and Jesse Davis. \"Learning from positive and unlabeled data: A survey.\" Machine Learning 109.4 (2020): 719-760.\n[2] Xu, Yixing, et al. \"Multi-Positive and Unlabeled Learning.\" IJCAI. 2017.\n\nQUESTIONS:\nIn addition to those raised in [Strengths And Weaknesses]:\n- why obtaining an unbiased estimation of precision loss is not possible in this setting (line 216)? My understanding is that as long as the sample is large enough, all the target labels would be observed.\n- at line 100, it is mentioned that in the partial feedback setting, it is impossible to estimate precision loss while later the scalar-loss objective is defined as the combination of recall and precision loss. Can the authors clarify on what metrics can be computed exactly and if there are any approximations?\n- what's the computational complexity of algorithm 1?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel PAC learning framework for precision-recall optimization under partial feedback, where algorithms observe only positive examples during training (e.g., a single randomly sampled label from the ground-truth set) but must predict complete label sets at test time. The authors demonstrate that classical approaches like Empirical Risk Minimization provably fail in this setting, even for simple hypothesis classes, due to the inability to estimate precision loss without negative examples. They propose two new algorithms for the realizable case achieving optimal $O(\\log|H|/\\epsilon)$ sample complexity, and establish fundamental limitations in the agnostic setting where additive approximation guarantees are impossible, requiring multiplicative approximation factors instead.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis paper proposes an interesting problem: how can models optimize to achieve good precision and recall when only partial positive labels are available, which is meaningful in real-world applications.\n\nWeaknesses:\nThe paper has certain relevance to \"Revisiting Pseudo-Label for Single-Positive Multi-Label Learning\". That paper proves the learnability under single-positive multi-label learning through ERM and proposes two assumptions (the assumption of pseudo-label accuracy and the assumption of data distribution sampling). The authors should provide some discussion regarding that paper. In addition, the related work section should include discussion of articles related to multi-label learning with missing labels, such as [1-5].\n[1] Revisiting Pseudo-Label for Single-Positive Multi-Label Learning\n[2] Multi-Label Learning from Single Positive Labels\n[3] Label-Aware Global Consistency for Multi-Label Learning with Single Positive Labels\n[4] One Positive Label is Sufficient: Single-Positive Multi-Label Learning with Label Enhancement\n[5] Large Loss Matters in Weakly Supervised Multi-Label Classification\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nthe authors adequately addressed the limitations and potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies PAC learning for the scenario where the hypotheses are set functions that map each input to a set of labels, instead of traditional single-label predictions. The authors consider realizable and agnostic settings. For realizable setting, they prove the sample complexity upper bound $O(\\frac{\\log|H|/\\delta}{\\epsilon})$ to achieve both $\\epsilon$ precision and recall rate. They give two algorithms to achieve this rate: one is maximum likelihood estimation and the other is minimizing surrogate loss. In the agnostic setting, they prove that achieving additive error is impossible by providing two lower bounds. Instead, they prove the sample complexity to achieve a multiplicative excess risk. The algorithm achieving this rate is the adapted version of  the realizable setting algorithms to the agnostic case.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\nThis paper for the first time studies the PAC learnability of precison and recall losss, which is very novel and useful, since in practice, people care more about PR rate of an algorithm rather than the overall misclassification rate. The theory is neat and interesting, and the algorithms are intuitive and natural---the authors show that the traditional ERM will fail but MLE is a PAC learner for precision and recall loss. They further provide impossibility results for agnostic setting, which match with my intuition that one cannot do well both in precision and recall rate.\n\nWeaknesses:\nThe authors claim that the additive excess risk bound is not achievable, and a multiplicative factor is unavoidable. However, the constant in lower bound and upper bound is rather large (1.05 vs 5). Hence I am curious what will be the optimal constant. Is that achievable by refining your techniques?\n\nQUESTIONS:\nPlease see the weakness part.\n\nLIMITATIONS:\nThis is a theory paper so I did not see potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}