Title: Conservative classifiers do consistently well with improving agents: characterizing statistical and online learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: PAC learning

online learning

sample complexity

learning with agents
Abstract: Machine learning is now ubiquitous in societal decision-making, for example in evaluating job candidates or loan applications, and it is increasingly important to take into account how classified agents will react to the learning algorithms. The majority of recent literature on strategic classification has focused on reducing and countering deceptive behaviors by the classified agents, but recent work of Attias et al. identifies surprising properties of learnability when the agents genuinely improve in order to attain the desirable classification, such as smaller generalization error than standard PAC-learning. In this paper we characterize so-called learnability with improvements across multiple new axes. We introduce an asymmetric variant of minimally consistent concept classes and use it to provide an exact characterization of proper learning with improvements in the realizable setting. While prior work studies learnability only under general, arbitrary agent improvement regions, we give positive results for more natural Euclidean ball improvement sets. In particular, we characterize improper learning under a generative assumption on the data distribution. We further show how to learn in more challenging settings, achieving lower generalization error under well-studied bounded noise models and obtaining mistake bounds in realizable and agnostic online learning. We resolve open questions posed by Attias et al. for both proper and improper learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper continues the study of the setting known as \"learning with improvements.\" Here, agents (i.e. instances of the example space) can react to a published classifier by moving to a new instance which is positively classified by the published classifier. Compared to \"strategic classification\", in the \"learning with improvements\" setting,  agents can genuinely improve in order to attain the desirable classification. The authors provide several contributions. They provide an exact characterization of proper learning with improvements in the realizable setting in terms of a variant of minimally consistent concept classes. In the improper setting, the authors characterize learnability with improvements under a mild generative assumption on the data distribution. Finally, the authors also provide results under label noise as well as in the online setting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper is well written and easy to follow\n- The problem setting is clean and practically relevant\n- In my opinion, the results are interesting and present a clear contribution to the learning theory community.\n\nWeaknesses:  No major weaknesses stand out to me. That said, I would appreciate it if the authors could address my questions/suggestions below, as there are a few places where I believe that readability can be improved.\n\nQUESTIONS:\nSuggestions: \n1) I think it would be good to provide some more intuition about when the loss in Equation (2.1) is 1, especially in terms of false positives and false negatives. In particular, it seems like the loss is 1 as long as there exists a false positive in $\\Delta_h(x)$ or h(x) is a false negative. \n2) In lines 140-146, I think it would be helpful to define \"least consistent\" explicitly in terms of the support of a hypothesis $h \\in H$ defined as $\\supp(h) = \\{x: h(x) = 1\\}.$ That is, for a realizable sample $S$, a hypothesis $g \\in H$ is least consistent, if for every consistent hypothesis $h \\in H$, we have that $\\supp(g) \\subseteq \\supp(h)$. \n3) In lines 140-146 and Definition 3.3, I think it would be good to give some examples of classes that are minimally and nearly minimally consistent.\n4) $\\Delta_G$ in Theorem 6.1 needs to be defined (I think it's the max out degree)\n\nQuestions: \n1) Theorem 3.4 seems to be a qualitative characterization of proper learning with improvements. Is there a quantitative characterization? That is, what is the optimal sample complexity for learning with improvements? Is there a dimension that characterizes this? \n2) Perhaps I'm mistaken, but when learning with label-noise, the goal is still to obtain small population error with respect to the true labels (not the noisy labels). It's just that the labels in the training data are noisy. This makes me confused about the noisy loss function defined in Line 274. Why is the loss here in terms of the noisy labels? What does getting a loss of $1$ here mean?\n3) In Lines 193-194, are you claiming that proper learning is without loss of generality if $\\Delta$ can be anything? Are you claiming that there exists a choice of $\\Delta$ such that a class $H$ is improperly learnable with improvements if and only if it is nearly minimally consistent?\n4) The algorithms for online learning are deterministic. While this is fine in the realizable setting, in the agnostic setting, randomization is key to removing the multiplicative factor in front of OPT. Is there a natural way to randomize Algorithm 3 and remove the $\\Delta$ factor in front of OPT? If not, what is the barrier?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* This paper extends the “learning with Improvements” framework (Attias et al., ICML 2025) to more general settings. \n* In the learning with improvements setting, feature-label pairs represent agents, which respond to deployed classifiers. Given classifier $h$, agents represented by a feature vector $x$ move to a point in their “improvement set” $x’\\\\in\\\\Delta(x)$ if their classification can be made positive ($h(x)=0$, $h(x’)=1$), and stay in place otherwise. True label $f^*(x’)$ is a function of the “improved” feature vector $x’$, and the classifier’s performance is measured according to its post-response accuracy, with worst-case tie breaking. A learning algorithm has access to a finite sample from a base distribution, and the goal is to prove statistical learnability.\n* The main theoretical result of Section 3 is a characterization: A class $\\\\mathcal{H}$ is PAC-learnable with improvements if and only if it is nearly minimally consistent - a variation of the known notion of minimal consistency, which characterizes PAC learnability with one-sided error.\n* Section 4 explores improper learnability when the improvement functions are the set of euclidean balls. The main positive result gives a learnability guarantee that depends on distribution coverability, and the main negative result provides a lower bound for the required training set size in the general case.\n* Section 5 explores the learnability of linear separators under bounded label noise, relating sample complexity of upper bounds for learning with improvements in this setting to bounds of PAC learnability in the bounded noise model.\n* Finally, Section 6 explores learning with improvements in the online discrete-graph setting, providing positive results for realizable and agnostic settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Significantly extends a set of recently published results in a meaningful way.\n* Presentation is clear and easy to follow. Helpful intuition for proofs is provided.\n* Relation to adjacent work is clearly explained, both formally and intuitively.\n\nWeaknesses\n* No empirical evaluation.\n* Possible societal impact is not discussed.\n\nQUESTIONS:\n* Is it possible to extend the characterization result in Section 3 to the non-realizable case?\n* How would the results change if the agent is not assumed to make the \"worst case\" choice among all points in $\\\\Delta(x)$?\n* Are there classes $\\\\mathcal{H}$ which do allow for proper learning under ball improvement sets?\n* What are the possible societal implications of the results, or in which way results can be extended to increase societal impact?\n\nLIMITATIONS:\nRelation to adjacent work is discussed, and questions that arise from the analysis are stated clearly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies PAC-learnability in the learning with improvement setting, where the agent can improve their feature vectors after observing the hypotheses given by the learner. The authors provide new conditions for which concept class, data distribution is learnable under different conditions, and provide the corresponding learning algorithms. See the \"Strengths And Weaknesses\" for more discussion on the results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths And Weaknesses\n---\n\nSignificance/novelty of the results:\n\nThis paper provides new and significant results for understanding PAC-learnable with impartments, including conditions for learnable concepts, learnable distributions, and more general settings. More specifically:\n\nWithout any assumption on the improvement function of the agents, the authors show that the necessary and sufficient condition for PAC-learnability with improvement is the concept class being nearly minimally consistent.\n\nAssuming the agents can only improve their decisions over a ball, the authors further show that there exists an algorithm that can learn any agent distribution if the conditional distribution of positive instances is recoverable. \n\nThe authors also consider the more general setting with label noise, and e construct optimal algorithms for learning linear separators. Finally, they also consider online learning with improvements for the discrete graph model, and provide algorithms with near-optimal mistake bounds.\n\n\nPresentation:\n\nThis paper is clearly written and though not very easy to follow. It would be great to shortly discuss/define things like minimally consistent in the introduction. \n\n\nQuestions and comments:\n\n\nFor the setting in ball improvement set discussed in section 4, if X is bounded, how is the improvement set defined for the points at the surface? Is it the interaction between X and Delta(x)? \n\nIn Definition 4.1,  D is (epsilon, beta, N) coverable if … radius r. why (epsilon, beta, N) coverable does not depend on r? (e.g., (epsilon, beta, N, r) coverable). Also, why do Theorems 4.2 and 4.3 not depend on r? What if r is infty then? \n\n\nTheorem 3.4 shows a necessary and sufficient condition for PAC-learnable with improvements. However, it seems to be transforming from one definition to another. While it is nice, I wonder if the authors could discuss what it implies? How does nearly minimally consistent help understanding PAC-learnable with improvements?\n\nPersonally, I think this paper would fit COLT or ALT more, but NeurIPS is also suitable.\n\nQUESTIONS:\nThe questions are given in \"Strengths And Weaknesses\".\n\nLIMITATIONS:\nPlease see \"Strengths And Weaknesses\" for more discussion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning in settings where agents can take improvement actions that beneficially alter their features after observing a published classifier. These improvements are assumed to reflect genuine increases in quality, as opposed to adversarial manipulation commonly studied in strategic classification. The paper builds on prior work by Attias et al. (2025) and investigates several learning settings under this assumption, including proper PAC learning, improper learning with geometric constraints, learning under label noise, and online learning on discrete improvement graphs.\n\nKey contributions include a characterization of proper PAC learnability via a condition termed nearly minimal consistency, sample complexity results for improper learning under Euclidean improvement sets with a coverability assumption, algorithms for noise-robust learning achieving Bayes optimality under specific distributional assumptions, and mistake-bound guarantees for online learning in realizable and agnostic settings. The paper also draws connections to other models in the literature, including PAC learning with one-sided error and strategic classification.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Addresses a learning setting distinct from adversarial manipulation by considering beneficial improvements.\n2. Provides formal characterizations and bounds in multiple learning paradigms.\n3. Extends prior work and resolves some theoretical questions.\n\nWeaknesses\n1. Lack of Game-Theoretic Modeling Clarity:\nThe paper does not explicitly present the learner-agent interaction as a Stackelberg game, nor does it characterize the agent’s best-response behavior given a published classifier.\nThere is no analysis or modeling of how agents decide to take improvement actions, or under what utility assumptions they operate.\n\n2. Limited Connection to Strategic Learning Literature:\nStrategic learning frameworks often distinguish between gaming and improvement actions. This distinction is not discussed, nor is there a clear explanation of when or why the improvement model applies instead of a more general strategic model.\n\n3. Illustrative Examples Missing:\nThere are no concrete examples or visual illustrations of key ideas (e.g., how classifiers behave under improvements, what consistency conditions mean, or how improvements affect learning performance), which makes some theoretical claims hard to interpret.\n\n4. No Empirical Evaluation:\nThe paper does not include experiments on real or synthetic data. As a result, it is unclear whether the theoretical findings lead to effective or robust learning performance in practice.\nThere is no evidence provided to support the practical efficiency, robustness, or scalability of the proposed algorithms.\n\n5. It is difficult to tell the difficulty/novelty of the proof techniques\n\nQUESTIONS:\n1. Can you formalize the interaction between the learner and agents using a Stackelberg framework, and explicitly define agent best responses?\n\n2. Please clarify how your model compares to strategic classification—can agents both game and improve, or are improvements enforced by assumption?\n\n3. Can you add examples that illustrate how your algorithms behave on simple input spaces (e.g., linearly separable data under improvements).\n\n4. Can you provide experimental results on synthetic or real datasets to validate your theoretical claims?\n\nLIMITATIONS:\nnot beyond the weaknesses and questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}