Title: Learning with Statistical Equality Constraints
Venue: NeurIPS 2025
Accepted: yes
Keywords: Machine learning

Constrained optimization

Constrained learning

Fairness

Duality
Abstract: As machine learning applications grow increasingly ubiquitous and complex, they face an increasing set of requirements beyond accuracy. The prevalent approach to handle this challenge is to aggregate a weighted combination of requirement violation penalties into the training objective. To be effective, this approach requires careful tuning of these hyperparameters (weights), involving trial-and-error and cross-validation, which becomes ineffective even for a moderate number of requirements. These issues are exacerbated when the requirements involve parities or equalities, as is the case in fairness and boundary value problems. An alternative technique uses constrained optimization to formulate these learning problems. Yet, existing approximation and generalization guarantees do not apply to problems involving equality constraints. In this work, we derive a generalization theory for equality-constrained statistical learning problems, showing that their solutions can be approximated using samples and rich parametrizations. Using these results, we propose a practical algorithm based on solving a sequence of *unconstrained*, *empirical* learning problems. We showcase its effectiveness and the new formulations enabled by equality constraints in fair learning, interpolating classifiers, and boundary value problems.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article studies the statistical learning problem with statistical equality constraints. It establishes a generalization bound (Thm 3.1) for this problem and discusses clearly and explicitly how this result is complementary to known properties of inequality-constrained statistical optimization problems. It introduces a primal-dual algorithm for the problem that alternates between the solution of unconstrained, empirical learning problems and the optimization of Lagrange parameters via an oracle (Assumption 6). The empirical dual value is related to the average of the Lagrangian iterates (Thm., 4.1), without solving the primal recovery problem, and with respect to the oracle. Toward experiments, the oracle is replaced by single alternating updates. The effectiveness of this practical algorithm is analyzed in an application to practical instances of the problem in which fairness is expressed in the form of statistical equality constraints.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The problem is interesting and important.\n- The theoretical results are plausible, the high-level reasoning in the main article is correct and convincing. (I have not checked the proofs in detail).\n- Although the article is an immediate continuation of [11,32], the authors are explicit about this fact (e.g. in Line 98) and explain why this continuation is non-trivial (notably in Remark 2.1).\n- The empirical method is convincing; the empirical results are interesting and insightful.\n- The article is well-written.\n\nWeaknesses:\n- I am not sure whether the case of deterministic (hard) equality constraints is merely a special case. The authors should discuss this. If special attention is needed, I suggest to use a more specific title \"Learning with Statistical Equality Constraints\".\n\nQUESTIONS:\n- While I understand both (P) and (P-DP), it took me a bit to see how (P-DP) can be put in the form (P). Would you like to introduce the $g_i$ that does this instead of using the Iverson bracket in (P-DP)?\n- Define the image of $f_\\theta$ consistently as $\\mathcal{Y}$ (as in Line 52) or $\\mathbb{R}^K$ (as in Line 51).\n\nLIMITATIONS:\nThe article communicates explicitly all assumptions made by the authors. This includes, in particular, the important assumption of uniform convergence (Assumption 5). This assumption is satisfied in many relevant cases, including families with finite VC-dimension.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of optimization with equality constraints, where the duality and generalization results obtained for inequality-constrained learning apply to these problems. This is motivated by the practical needs that machine learning systems with goals such as in fairness tends to require equality constraints instead of a bound. This paper developed a generalization theory for equality constrained learning tasks:\n- the paper derived additional regularity conditions under which duality and sensitivity results for non-convex equality-constrained optimization problems are obtained\n- the paper then propose a practical algorithm based on solving a sequence of unconstrained, empirical learning problems, and characterized the approximation and generalization error of its solutions\n- experiments using examples in group fairness show the effectiveness of the proposed solution\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- this paper studied a practical but less explored formulation with the equality constraints instead of inequality constraints, the problem is well-motivated from practical applications especially fairness\n- the derivation of the error bound is intuitive via the duality formulation, and the bound itself reflects the different drivers of the error, including the sensitivity on constraints / formulation, model capacity, sample size etc\n- experimental results show that with small tolerance, inequality formulation achieves similar results as the proposed DP, however the difference becomes non-negligible when the tolerance parameter is high\n\nWeakness:\n- The algorithm in section 4 assumes the access to an oracle optimizer. Although the experiments show that single alternating updates with an optimizer eg Adams work relatively well, the results are only based on simple setting and one dataset. The experimental support should be strengthened with more diverse setups and complex data.\n\nQUESTIONS:\n- there is a lack of any discussions on the optimality of the provided error bound: under what scenarios it is tight  / lose?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper addresses the problem of learning under equality constraints. It proposes a theoretical framework for non-convex problems and proves strong duality and generalization bounds under a set of relatively strong assumptions. Based on this, the authors introduce a practical primal-dual algorithm that reduces the problem to a sequence of unconstrained optimizations.

STRENGTHS AND WEAKNESSES:
**Strengths**

- The paper are clearly written.
- The paper fills a theoretical gap by extending constrained learning theory from inequality to equality constraints.

**Weaknesses**

- Would Assumption 1 still hold if multiple potentially conflicting fairness constraints are imposed simultaneously? Have the authors considered or tested such settings?
- It is unclear how robust the theory is when the assumption only approximately holds or fails in more constrained or imbalanced scenarios.
- The algorithm requires solving a full learning problem at each dual update, which can be costly with large models or many constraints.

QUESTIONS:
please see weakness section

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theory and algorithm for equality constrained learning, addressing the limitations of existing approaches to inequality constraints with an application to fair machine learning. The authors dualized a learning problem with inequality and equality constraints, investigated the generalization bound, and proposed a dual ascent algorithm to solve the problem. Moreoverm the authors demonstrated that the proposed algorithm works in practice, even though its theoretical properties were analyzed under the assumption of access to an oracle optimizer, which is not guaranteed in real applications.\n\nSTRENGTHS AND WEAKNESSES:\nstrengths\n- Provides a theory and algorithm for equality constrained learning which has been underexplored. \n- Provides theoretical results (though I am not fully confident in their correctness, as they are outside my area of expertise) and a numerical algorithm that works in practice.\n\nweaknesses\n- The major contribution of this paper is to develop a theory and algorithm for equality constrained learning. But the examples, applications, and experiments seem biased solely toward fair machine learning. The authors insisted in the introduction that many important requirements are expressed as equality constraints. Why did the authors focus on fair machine learning? I doubt that fair machine learning is the best beneficiary of this work (please see limitations). It would have been better if the authors presented more diverse examples and empirically demonstrated the advantages of applying equality constraints in those examples.\n- Figure 1 shows that the approximate algorithm with a small tolerance for the inequality constraints works as good as the exact algorithm with equality constraints. If we are able to achieve a sufficiently good approximation by choosing an appropriate tolerance for the inequality constraint, why do we need an exact algorithm? This example does not support the claim \"small values $\\epsilon > 0$ lead to numerical issues and worse generalization bounds (although it is true). An example that the inequality with small tolerance cannot approximate the exact one will be more convincing. \n- In summary, I’m not entirely convinced that developing theory and optimization algorithms for equality constrained learning is important, however, I do believe this line of work is valuable and will offer useful contributions.\n\nQUESTIONS:\n- For my major concerns, please see the weaknesses above.\n- In Remark 2.1, for the claim that 'small values $\\epsilon > 0$ lead to numerical issues and worse generalization bounds arising from large Lagrange multipliers,' the authors refer to Theorem 3.1. However, it is not immediately clear to me how Theorem 3.1 supports this claim.\n- There are some minor formatting issues. It would be better to use consistent terms for references. For example, consistently using either the full term or the abbreviation (e.g., theorem vs. thm, section vs. sec, assumption vs. ass, lemma vs. lem) to avoid confusion.\n\nLIMITATIONS:\nAlthough the examples given in the paper seem decent, I doubt fair machine learning is the best example to motivate equality constrained learning. When developing fair models, the goal is usually to balance overall performance with fairness across sensitive groups, rather than enforcing strict equality constraints at all costs. In many cases, it is hard to justify compromising the performance of a better-performing group (often the majority) to improve fairness. This raises a concern that if equality constraints are applied, we may end up with trivial or counterproductive solutions that lower performance for all groups. I think this is why recent algorithmic fairness work shifted to encourage a better performance-fairness trade-off instead.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}