Title: Marginal-Nonuniform PAC Learnability
Venue: NeurIPS 2025
Accepted: yes
Keywords: statistical learning theory

nonuniform learning

PAC learning

universal learning

VC dimension
Abstract: We revisit the classical model of nonuniform PAC learning, introduced by Benedek and Itai [1994], where generalization guarantees may depend on the target concept (but not on the marginal distribution). In this work, we study a complementary variant, which we call *marginal-nonuniform learning*. In this setting, guarantees may depend on the marginal distribution over the domain, but must hold uniformly over all concepts. This captures the intuition that some data distributions are inherently easier to learn from than others, allowing for a flexible, distribution-sensitive view of learnability. Our main result is a complete characterization of the achievable learning rates in this model, revealing a trichotomy: exponential rates of the form $e^{-n}$ arise precisely when the hypothesis class is finite; linear rates of the form $d/n$ are achievable when a recently introduced combinatorial parameter, the VC-eluder dimension $d$, is finite; and arbitrarily slow rates may occur when $d = \infty$. Additionally, in the original (concept-)nonuniform model, we show that for all learnable classes linear rates are achievable. We conclude by situating marginal-nonuniform learning within the landscape of universal learning, and by discussing its relationship to other distribution-dependent learning paradigms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of marginal-nonuniform PAC learning. Here there is a known class C of functions on a space X. One wants to (realizably) PAC learn an unknown function f in C on an unknown distribution D on X, but in a way where the rate at which the error converges to 0 in terms of samples is allowed to depend on the marginal distribution D.\n\nIn particular, they say that you can learn with rate R (for some function R going monotonly to 0) if there is an algorithm A so that after receiving n samples (x,y) where x ~ D and y = f(x) for some f in C returns a hypothesis h with error over C at most O_D( R( Omega_D(n) ) ). In other words, when run with any fixed marginal D, the error goes to 0 at a rate with the same functional form as R.\n\nThe main result of this paper is a classification of achievable rates. In particular:\n1) In some trivial cases, one can learn in <= 1 sample and take R = 0.\n2) Otherwise, if C is finite, one learns with exponential rate R(n) = e^{-n}.\n3) Otherwise, if C has finite VC dimension, one learns with rate R(n) = 1/n.\n4) Finally, if C has infinite VC dimension, then there is no rate R(n) that goes to 0 so that we can learn with rate R.\n\nThey also show that in case (3) the learning rate can be taken more precisely to be bounded by O(d/n) where d is the VC-eluder-dimension.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The basic idea of marginal-nonuniform learning is a nice one and this paper essentially solves some version of it.\n* The final classification is clean and not entirely expected.\n* The proofs are easy to follow.\n* Generally the background is well explained.\n\nWeaknesses:\n* The technical definition of this kind of marginal-nonuniform learning in terms of rate functions R is a bit complicated and not especially natural.\n* The results here are all relatively easy and do not seem to introduce technical tools that are likely to have broader applicability. (Though the VCE stuff has some more complicated ideas)\n\nComments:\n* Line 91: *What* is a strict version of the realizability assumption? My initial read was that you were claiming that the function er was.\n* Definition 2:\n - Isn't h_n^ more naturally the (randomized) output of an algorithm rather than the algorithm itself?\n - What do you mean by \"a faster rule than rate R\"?\n\nQUESTIONS:\n* Is the proof of Theorem 16 essentially just contained in the citations given? If not, what is the proof?\n* What *was* previously known about this problem?\n* What in general do you mean by considering fine-grained rates?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the classical model of non-uniform PAC learning, where the error convergence rate includes constants depend on the true concept or the data distribution. The main conclusion of this paper is that when considering marginal-nonuniform learning (case that the constant allows to be dependent on feature distribution but is required to be uniform across concepts), the optimal rate is a trichonomy, characterized by the VC-eluder dimension. This reveals some inclusions regarding uniform learning at rate $1/n$ and some non-uniform rates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The problem is fundamental in machine learning theory. The paper is generally well-written. The problem investigated in the paper is clearly proposed and defined, the symbols and proofs are standard that easy to follow. The results are clear, with proper comparison and discussions to the results in related topics.\n\nWeaknesses: I only have some suggestions:\n  - Current draft seems somewhat \"heavy\" in math. The definitions of \"different learning cases\", \"trichonomy rates\" and notions of \"at rate\", \"not faster than\", all require large text space. Besides, the authors provide most of the proofs in main text. As a paper with page limit of 9, the main body of the paper is full of definitions, theorems and proofs, sometimes hard to follow. Is it possible to reduce the appearance of similar definitions or theorems, and put more proofs into the appendix?\n  - While it is definitely OK to prove some of the results in this paper by adapting from [Hanneke and xu, 2024], it would be better if the authors could discuss how the proof techniques (rather than setting) in this paper differ from that of previous works. \n  - The cross-marks in Fig.1 may incur some mis-understandings that the corresponding regions are empty, while the marked areas are proven to be non-empty. It would be better if they could be replaced by some other symbols.\n  - The last section includes 8 paragraphs of discussions, while some of them are similar and some of them seems a bit minor. Fewer paragraphs of discussions with better organization of the topics may be better.\n\nQUESTIONS:\n- I am concerning the significance and applicability of theoretically studying the non-uniform settings of learning \\emph{in the present days}. Specifically, casting at its greatest generality, what conclusions can be drawn from research in this direction, and what problems can be solved? How does resolving these issues promote the study of machine learning theory or assist in solving real-world machine learning problems? I appreciate that this is a fundamental research area that worth to investigate, but to be honest, such concern affect my score to some extent (and sadly, I can hardly tell a clear criteria of how this affects). I would appreciate it if the authors could share some opinions at this point. \n - In line 106-107 you've mentioned that \"there are families of classes such that $VCE(\\mathcal{H})=1$\", while $VC(\\mathcal{H})$ is unbounded\", while in line 135-136 you said that $1\\leq VCE(\\mathcal{H}) <\\infty$ is equivalent to $|\\mathcal{H}|=\\infty$ and $VC(\\mathcal{H})<\\infty$. Do these two statements mean that while there exists classes $\\mathcal{H}$ with VCE-dimension $1$ and *arbitrarily large* VC-dimension, there does not exist a class $\\mathcal{H}$ with VCE-dimension $1$ and *infinite* VC-dimension?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores one of the four fundamental notions of PAC learnability that has not been thoroughly studied: marginal-nonuniform learnability, where the generalization rate may depend on the marginal distribution but must be consistent across all concepts. The authors present a clear trichotomy of possible learning rates under this framework: 1) exponential rate for finite classes, 2) linear rates determined by the VC-eluder dimension, and 3) arbitrarily slow rates when the dimension is unbounded. The results are accompanied by multiple matching lower bounds, a detailed analysis of the linear case, and a contextualization within the broader landscape of universal learning. The paper is purely theoretical and offers useful characterizations of how distribution dependence affects learnability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper examines a natural variation of nonuniform PAC learning that is both conceptually interesting and technically well-defined.\n- The main accomplishment is a precise trichotomy, providing a comprehensive classification of learning rates based on a single combinatorial parameter, the VC-euler dimension.\n- The presentation is clear, especially in the abstract and introduction, and the proofs use existing tools from prior work transparently.\n- The paper connects to established frameworks in universal and concept-non-uniform learning, effectively placing its contribution within the existing literature.\n\nWeaknesses:\n- The formulation of marginal-nonuniform learning is not entirely new. It originates from the taxonomy introduced by Ben-David et al. (1995), and the paper does not distinguish itself in terms of foundational definitions.\n- Many of the theoretical claims, including several lemmas and various example-based separations, directly build on existing results in the literature, which seem to require only modest adjustments rather than significant innovation.\n- The fine-grained analysis of the linear rate regime using the VC-eluder dimension appears to be the most technically challenging part of the paper (e.g., Lemmas 10-12). It would be helpful if the authors more clearly highlighted where genuine technical novelty appears.\n\nQUESTIONS:\nPlease take a moment to review the comments and suggestions in the sections above. Let me know if anything seems missing, misunderstood, or could benefit from further clarification. Additionally,\n- Could the authors elaborate on whether the results are applicable in agnostic/non-realizable settings?\n- What common assumptions could cause the VC-eluder dimension to be significantly smaller than the VC dimension?\n- Are there specific examples that match the theoretical findings to illustrate the idea that \"some data distributions are inherently easier to learn from than others\"?\n\nLIMITATIONS:\nThis work is purely theoretical and has minimal negative impact on society. The authors have already identified some gaps in their results and future directions, such as active learning settings, which should help address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper formalizes \"marginal-nonuniform learning\" as a variant of the PAC model that allows for distribution-specific learning rates that hold uniformly over all possible target concepts. Specifically, a concept class is learnable at rate R if for every marginal distribution over examples, there exists a (distribution-dependent) constant c such that every concept in the class is learnable with error at most ~R(cn) using n samples. This ordering of quantifiers makes it a relaxation of distribution-free PAC learning (uniform), a strengthening of universal learning, and complementary to (concept)-nonuniform learning.\n\nThe paper proves the following results. The main result is a trichotomy of rates for marginal-nonuniform learning in terms of the VC-eluder (VCE) dimension of the target class. Moreover, in the case where the learning rate is linear, it scales as VCE(H)/n. The paper then revisits concept-nonuniform learning, for which it shows that the one-inclusion graph algorithm achieves a linear rate for countable unions of VC classes. Finally, the paper discusses the relationships between uniform, non-uniform, and universal learning and presents examples of classes separating these notions.\n\nSTRENGTHS AND WEAKNESSES:\n(+) The paper studies a mathematically natural learning model that cleanly fills in a gap between previously studied ones.\n\n(+) The main trichotomy result and characterization via VCE dimension is sharp and satisfying.\n\n(+) Proofs are conceptually simple and do a nice job intuitively illuminating why the corresponding statements are true.\n\n(-) While the question of understanding learning rates for marginal-nonuniform learners is new, it's debatable how new the model of marginal-nonuniform learning itself is. As acknowledged in the paper, it's related to distribution-specific and distribution-family learning. But even more closely related questions about the difference between uniform PAC learning and learnability with respect to known, but arbitrary, marginal distributions have been studied in the past. For example, in Section 7.2 of the arXiv version of \"A General Characterization of the Statistical Query Complexity,\" Feldman showed a strong separation between distribution-independent SQ complexity and the max over all distributions of distribution-specific SQ complexity.\n\n(-) The paper's statements and proofs very closely track those of recent prior work of Hanneke and Xu on universal learning rates. As such, there isn't much in the way of new technical ideas or tools developed in this paper.\n\nQUESTIONS:\n* Does Theorem 16 (or its proof) suggest a characterization of the fine-grained rate of concept-nonuniform learnability of countable unions of VC classes?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the question marginal-nonuniform learnability, requiring learners to depend only non-uniformly on the marginal, but uniformly on the concept. Thus this notion contrast standard non-uniform learning, which requires guarantees to be uniform for all marginals but non-uniform over concepts. These notions are in-between notion of uniform PAC learning, which provides uniform rates for all concepts and distributions and universal learning, which allow for non-uniform guarantees for both concept and marginal. \n\nThey relate this learnability to a recently introduced complexity measure - the VC-eluder dimension. In particular they show that:\n- A class is marginal non-uniformly learnable, with exponential rates if and only if it is finite\n- A class is marginal non-uniformly learnable with linear rates if and only if the class has finite VC-eluder dimension\n- A class has at best arbitrarily slow non-uniform rates for marginal non-uniform learnability if it has infinite VC-eluder dimension. \n\nThey also show improved rates for concept-nonuniform learning. \n\nTheir results rely on established methods within the literature of universal learning rates.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nI overall enjoyed the presentation of the paper. I enjoyed the use of color to highlight different quantifiers as well as the picture separating between different cases.\nThere is a good overview of the literature, highlighting many important results. \nThe results discuss a relevant non-uniform learnability notion and it provides novel results.\nI enjoyed the discussion section. \n\nOverall I enjoyed reading the paper and found it insightful and well-rounded.\n\n\nWeaknesses:\n-In contrast to the universal learning rates, this work does not provide a full characterization, as there is no distinction between arbitrarily slow rates and a class not being non-uniform marginally learnable. In universal learning every class (with sufficient measurability conditions) is universally learnable at least with arbitrarily slow rates. As the authors mention, in previous work it has been shown that this is not the case for marginal non-uniform learning. However there is no clear characterization of what distinguishes marginal non-uniformly learnable classes from not marginal non-uniformly learnable classes. I think this should be highlighted more, as the presentation as trichotomy which is akin to the universal learning rates paper from Bousquet et al 2021 (and follow-up works), where the trichotomy gives a more complete characterization (as there is no fourth case of unlearnable classes)\n- I wish there was a bit more intuition given for cases in which marginal non-uniform learnability is distinct from universal learning. There are some examples mentioned from the literature, but it would be nice if they could be discussed in more detail. How do the complexity measures infinite Littlestonetree/VCL-tree and VC-eluder dimension relate? A lot of the examples from old work are merely stated, which makes sense for space constraints. In order to understand the intuition of this paper, it might help though to rewrite those examples and relate them explicitly to those complexity measures. \n- The insight of this paper heavily relies on bringing together insights from previous work and the ideas within this work itself aren't as novel or ground-breaking. (I don't view this as a reason to reject the paper, as it was still insightful to read the paper)\n- As is often the case for results on universal learning rates the results rely on the realizable case, which is arguably a very strong assumption. (However, as this is a general drawback of this line of work I would not count it as heavily against this work in particular)\n\n\nOverall I feel that this was a decent paper and I enjoyed reading it. The characterization and discussion is a good addition to the literature and fits well within the NeurIPS program.\n\nQUESTIONS:\nOne downside of non-uniform learning approaches as opposed to uniform PAC learning approaches is that non-uniform learning guarantees do not provide any error/confidence guarantees for a learned classifier. \nIs this something you can still overcome in the marginal non-uniform setting?\n\nCan you say anything about learnability in the agnostic case?\n\nLIMITATIONS:\nThe authors provide proofs for all of their statements. Limitations and assumptions have been acknowledged in the main text of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-24"}