SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Prop. 3 already provides a more computationally amenable definition of resilient relaxation, given that Lagrange multipliers are often computed while solving  $(\tilde{\mathbf{P}}_{\mathbf{u}})$ , e.g., when using primal-dual methods. Yet, an even more straightforward formulation of Def. 1 is obtained by rearranging (2) as  $0 \in \partial (\tilde{\mathbf{P}}^{\star} + h)(\mathbf{u}^{\star})$ , as we did in the proof of Prop. 1, which suggests that  $\mathbf{u}^{\star}$  is related to the minimizers of  $\tilde{\mathbf{P}}^{\star} + h$ . This is formalized in the following proposition (see proof in appendix D.4).
Proposition 4. A relaxation  $\mathbf{u}^{\star}$  satisfies the resilient equilibrium (2) if and only if it is a solution of
$$
\tilde {\mathrm {P}} _ {\mathrm {R}} ^ {\star} = \min  _ {\phi \in \mathcal {F}, \mathbf {u} \in \mathbb {R} _ {+} ^ {m}} \quad \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D} _ {0}} \left[ \ell_ {0} (\phi (\mathbf {x}), y) \right] + h (\mathbf {u})
$$
subject to  $\mathbb{E}_{(\mathbf{x},y)\sim \mathcal{D}_i}\Big[\ell_i\big(\phi (\mathbf{x}),y\big)\Big]\leq u_i,\quad i = 1,\ldots ,m.$
The corresponding minimizer  $\phi^{\star}$  is a resilient solution of the functional learning problem  $(\tilde{\mathbf{P}}_{\mathbf{u}})$ .
Prop. 4 shows that it is possible to simultaneously find a resilient relaxation  $\mathbf{u}^{\star}$  and solve the corresponding resilient learning problem. Indeed, a resilient solution of a constrained learning problem can be obtained by incorporating the relaxation cost in its objective. This is reminiscent of first-phase solvers found in interior-point methods used to tackle convex optimization problems [39, Chap. 11]. Note, once again, that this is not the same as directly adding the constraints in the objective as penalties or regularizations. Indeed, recall from Def. 1 that the resilient relaxation balances the marginal effects of  $h$  on  $\tilde{\mathbb{P}}^{\star}$  and not  $\ell_0$ . Before using Prop. 4 to introduce a practical resilient learning algorithm and its approximation and generalization properties (Sec. 4), we use these equivalent formulations to relate resilient learning to classical learning tasks.
3.3 Relation to Classical Learning Tasks
(Un)constrained learning: Both traditional unconstrained and constrained learning can be seen as limiting cases of resilient learning. Indeed, if  $h \equiv 0$ ,  $\mathbf{u}$  has no effect on the objective of (P-RES). We can then take  $u_{i} = B$  for  $i = 1, \dots, m$ , which reduces (P-RES) to an unconstrained learning problem (recall that all losses are  $[-B, B]$ -valued). On the other hand, if  $h$  is the indicator function of the non-negative orthant (i.e.,  $h(\mathbf{u}) = \mathbf{0}$  for  $\mathbf{u} \preceq \mathbf{0}$  and  $h(\mathbf{u}) = \infty$ , otherwise), then it must be that  $\mathbf{u} = \mathbf{0}$  in (P-RES) as long as this specification is feasible. Neither of these relaxation costs satisfy the conditions from Def. 1, since they are not componentwise increasing or differentiable, respectively. Still, there exists valid relaxation costs that approximate these problems arbitrarily well (see Appendix D.5).
Penalty-based methods: Rather than the constrained formulations in (P) or  $(\tilde{\mathrm{P}}_{\mathbf{u}})$ , requirements are often incorporated directly into the objective of learning tasks using fixed penalties as in
$$
\underset {\phi \in \mathcal {F}} {\text {m i n i m i z e}} \mathbb {E} _ {\mathcal {D} _ {0}} [ \ell_ {0} (\phi (\mathbf {x}), y) ] + \sum_ {i = 1} ^ {m} \gamma_ {i} \mathbb {E} _ {\mathcal {D} _ {i}} [ \ell_ {i} (\phi (\mathbf {x}), y) ], \tag {4}
$$
where the fixed  $\gamma_{i} > 0$  represent the relative importance of the requirements. It is immediate from Prop. 3 that resilient learning with a linear relaxation cost  $h(\mathbf{u}) = \sum_{i}\gamma_{i}u_{i}$  is equivalent to (4) as long as  $\mathbb{E}_{\mathcal{D}_i}\left[\ell_i\big(\phi^{\star}(\mathbf{x}),y\big)\right]\geq 0$ . From Def. 1, this is the same as fixing the marginal effect of relaxations on the perturbation function.
Soft-margin SVM: Linear relaxation costs are also found in soft-margin SVM formulations, namely
$$
\underset {\theta \in \Theta , \mathbf {u} \in \mathbb {R} _ {+} ^ {m}} {\text {m i n i m i z e}} \quad \frac {1}{2} \| \theta \| ^ {2} + \gamma \sum_ {i = 1} ^ {m} u _ {i} \tag {PI}
$$
subject to  $1 - y_{i}\theta^{T}\mathbf{x}_{i}\leq u_{i},\quad i = 1,\ldots ,m.$
Though written here in its parametrized form, the hypothesis class underlying (PI) (namely, linear classifiers) is convex. It can therefore be seen as an instance of (P-RES) where each loss  $\ell_{i}$  represent a classification requirement on an individual sample. Soft-margin SVM is therefore a resilient learning problem as opposed to its hard-margin version, where  $u_{i} = 0$  for  $i = 1,\dots ,m$ .
4 Resilient Constrained Learning Algorithm
We defined the resilient equilibrium  $\mathbf{u}^{\star}$  in (2) and the equivalent resilient learning problems  $(\tilde{\mathrm{P}}_{\mathbf{u}^{\star}})$  and (P-RES) in the context of the convex functional space  $\mathcal{F}$ . Contrary to  $(\mathrm{P_u})$ , that are defined on the finite dimensional  $\mathcal{F}_{\theta}$ , these problems are not amenable to numerical solutions. Nevertheless, we have argued that as long as  $\mathcal{F}_{\theta}$  is a good approximation of  $\mathcal{F}$ , the values of  $(\mathrm{P_u})$  and  $(\tilde{\mathrm{P}}_{\mathbf{u}})$  are close. We use this idea to obtain a practical primal-dual algorithm (Alg. 1) to approximate solutions
Algorithm 1 Resilient Constrained Learning  $(\eta, \eta_u, \eta_\lambda > 0; \theta^{(0)} \in \Theta; \pmb{\lambda}^{(0)}, \mathbf{u}^{(0)} \in \mathbb{R}_+^m)$
for  $t = 1,\dots ,T$
$$
\begin{array}{l} \left\{ \begin{array}{l} \theta_ {1} = \theta^ {(t - 1)} \\ \theta_ {n + 1} = \theta_ {n} - \eta \nabla_ {\theta} \Big [ \ell_ {0} \big (f _ {\theta_ {n}} (\mathbf {x} _ {n, 0}), y _ {n, 0} \big) + \sum_ {i = 1} ^ {m} \lambda_ {i} \ell_ {i} \big (f _ {\theta_ {n}} (\mathbf {x} _ {n, i}), y _ {n, i} \big) \Big ], n = 1, \ldots , N \\ \theta^ {(t)} = \theta_ {N + 1} \end{array} \right. \\ \mathbf {u} ^ {(t)} = \left[ \mathbf {u} ^ {(t - 1)} - \eta_ {\mathbf {u}} \left(\nabla h \left(\mathbf {u} ^ {(t - 1)}\right) - \boldsymbol {\lambda} ^ {(t - 1)}\right) \right] _ {+} \\ \lambda_ {i} ^ {(t)} = \left[ \lambda_ {i} ^ {(t - 1)} + \eta_ {\lambda} \left(\frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {i} \left(f _ {\theta^ {(t - 1)}} (\mathbf {x} _ {n, i}), y _ {n, i}\right) - u _ {i} ^ {(t - 1)}\right) \right] _ {+}, \quad i = 1, \dots , m \\ \end{array}
$$
of (P-RES) (and thus  $(\tilde{\mathbf{P}}_{\mathbf{u}^{\star}})$ ). The main result of this section (Thm. 1) establishes how good this approximation can be when using only samples from the  $\mathcal{D}_i$ .
Explicitly, consider a set of  $N$  i.i.d. sample pairs  $(\mathbf{x}_{n,i},y_{n,i})$  drawn from  $\mathcal{D}_i$  and define the parametrized, empirical Lagrangian of the resilient learning problem (P-RES) as
$$
\hat {L} _ {\theta} (\theta , \boldsymbol {\lambda}; \mathbf {u}) = h (\mathbf {u}) + \frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {0} \left(f _ {\theta} \left(\mathbf {x} _ {n, 0}\right), y _ {n, 0}\right) + \sum_ {i = 1} ^ {m} \lambda_ {i} \left(\frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {i} \left(f _ {\theta} \left(\mathbf {x} _ {n, i}\right), y _ {n, i}\right) - u _ {i}\right). \tag {5}
$$
The parametrized, empirical dual problem of (P-RES) is then given by
$$
\hat {\mathrm {D}} _ {\mathrm {R}} ^ {\star} = \max  _ {\boldsymbol {\lambda} \in \mathbb {R} _ {+} ^ {m}} \min  _ {\theta \in \Theta , \mathbf {u} \in \mathbb {R} _ {+} ^ {m}} \hat {L} _ {\theta} (\theta , \boldsymbol {\lambda}; \mathbf {u}). \tag {D-RES}
$$
The gap between  $\hat{D}_{\mathrm{R}}^{\star}$  and the optimal value  $\tilde{P}_{\mathrm{R}}^{\star}$  of the original problem  $(\mathbf{P_u})$  can be bounded under the following assumptions.
Assumption 3. The loss functions  $\ell_i$ ,  $i = 0 \dots m$ , are  $M$ -Lipschitz continuous.
Assumption 4. For every  $\phi \in \mathcal{F}$ , there exists  $\theta^{\dagger} \in \Theta$  such that  $\mathbb{E}_{\mathcal{D}_i}\left[|\phi (\mathbf{x}) - f_{\theta^\dagger}(\mathbf{x})|\right] \leq \nu$ , for all  $i = 0, \dots, m$ .
Assumption 5. There exists  $\xi(N, \delta) \geq 0$  such that for all  $i = 0, \dots, m$  and all  $\theta \in \Theta$ ,
$$
\left| \mathbb {E} _ {\mathcal {D} _ {i}} \left[ \ell_ {i} \left(f _ {\theta} (\mathbf {x}), y\right) \right] - \frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {i} \left(f _ {\theta} \left(\mathbf {x} _ {n, i}\right), y _ {n, i}\right) \right| \leq \xi (N, \delta)
$$
with probability  $1 - \delta$  over draws of  $\{(\mathbf{x}_{n,i},y_{n,i})\}$
Although the parameterized functional space  $\mathcal{F}_{\theta}$  can be non-convex, as is the case for neural networks, Ass. 4 requires that it is rich in the sense that the distance to its convex hull is bounded. When  $\xi$  exists for all  $\delta >0$  and is a decreasing function of  $N$ , Ass. (5) describes the familiar uniform convergence from learning theory. Such generalization bounds can be derived based on bounded VC dimension, Rademacher complexity, or algorithmic stability, to name a few [41].
We can now state the main result of this section, whose proof is provided in Appendix C.
Theorem 1. Consider  $\hat{\mathbf{P}}_{\mathrm{R}}^{\star}$  and  $\mathbf{u}^{\star}$  from (P-RES) and  $\hat{\mathsf{D}}_{\mathbb{R}}^{\star}$  from (D-RES). Under Ass. 1-5, it holds with probability of  $1 - (3m + 2)\delta$  that
$$
\left| \tilde {\mathrm {P}} _ {\mathrm {R}} ^ {\star} - \hat {\mathrm {D}} _ {\mathrm {R}} ^ {\star} \right| \leq h \left(\mathbf {u} ^ {\star} + \mathbb {1} \cdot M \nu\right) - h \left(\mathbf {u} ^ {\star}\right) + M \nu + (1 + \Delta) \xi (N, \delta). \tag {6}
$$
Thm. 1 suggests that resilient learning problems can be tackled using (D-RES), which can be solved using saddle point methods, as in Alg. 1. Even if the inner minimization problem is non-convex, dual ascent methods can be shown to converge as long as its solution can be well approximated using stochastic gradient descent [38, Thm. 2], as is often the case for overparametrized NNs [42-44]. A more detailed discussion on the algorithm can be found in Appendix E. Next, we showcase its performance and our theoretical results in two learning tasks.
We now showcase the numerical properties of resilient constrained learning. As illustrative case studies, we consider federated learning under class imbalance and invariance constrained learning. Additional experimental results for both setups are included in Appendix F and H. We also include ablations on the hyperparameters of the method, including dual and perturbation learning rates, and the choice of the perturbation cost function  $h$ .
5.1 Heterogeneous Federated Learning
Federated learning [45] entails learning a common model in a distributed manner by leveraging data samples from different clients. Usually, average performance across all clients is optimized under the assumption that data from different clients is identically distributed. In practice, heterogeneity in local data distributions can lead to uneven performance across clients [46, 47]. Since this may be undesirable, a sensible requirement in this setting is that the loss of the model is similar for all clients.
Let  $\mathfrak{D}_c$  be the distribution of data pairs for Client  $c$ , and  $R_{c}(f_{\theta}) = \mathbb{E}_{(\mathbf{x},y)\sim \mathfrak{D}_{c}}[\ell (f_{\theta}(\mathbf{x}),y)]$  its statistical risk. We denote the average performance as  $\overline{R} (f_{\theta}):= (1 / C)\sum_{i = 1}^{C}R_{c}(f_{\theta})$ , where  $C$  is the number of clients. As proposed in [18] heterogeneity issues can be tackled by imposing a proximity constraint between the performance of each client  $R_{c}$ , and the loss averaged over all clients  $\overline{R}$ . This leads to the constrained learning problem
$$
\min  _ {\boldsymbol {\theta} \in \Theta}. \quad \bar {R} (f _ {\boldsymbol {\theta}}) \tag {P-FL}
$$
$$
s. t o \quad R _ {c} (f _ {\boldsymbol {\theta}}) - \overline {{R}} (f _ {\boldsymbol {\theta}}) - \epsilon \leq 0, \quad c = 1, \dots , C,
$$
where  $\epsilon > 0$  is a small (fixed) positive scalar. It is ready to see that this problem is of the form in (P); see Appendix F. Note that the constraint in (P-FL) is an asymmetric (and relaxed) version of the equality of error rates common in fairness literature.
As shown by [18] this problem can be solved via a primal-dual approach in a privacy-preserving manner and with a negligible communication overhead, that involves sharing dual variables. However, because the heterogeneity of the data distribution across clients is unknown a priori, it can be challenging to specify a single constraint level  $\epsilon$  for all clients that results in a reasonable trade-off between overall and individual client performance. In addition, the performance of all clients may be significantly reduced by a few clients whose poor performance make the constraint hard to satisfy. The heuristic adopted in [18] is to clip dual variables that exceed a fixed value.
In contrast, we propose to solve a resilient version of (P-FL) by including a relaxation  $u_{c}, c = 1, \ldots, C$  for each constraint and a quadratic relaxation cost  $h(\mathbf{u}) = \alpha \| \mathbf{u} \|_2^2$ . The resilient version of problem (P-FL) can also be solved in a privacy preserving manner as long as  $h(\mathbf{u})$  is separable in each of the constraint perturbations  $u_{c}$ . In that case  $u_{c}$  can be updated locally by client  $c$ .
Following the setup of [18], heterogeneity across clients is generated through class imbalance. More specifically, samples from different classes are distributed among clients using a Dirichlet distribution. Experimental and algorithmic details along with privacy considerations are presented in Appendix F.
Constraint relaxation and relative difficulty: Our approach relaxes constraints according to their relative difficulty. In the context of class imbalance, the majority classes exert a stronger influence on the overall objective (average performance). Therefore, the overall performance  $\overline{R}$  tends to favor smaller losses in the majority classes rather than in the minority ones. As a result, meeting the proximity constraint in (P-FL) for clients whose datasets contain a higher fraction of training samples from the minority classes can deteriorate performance. In Figure 2(left), we demonstrate that the constraint is effectively relaxed more for these clients.
Controlling the performance vs. relaxation trade-off: Through the choice of the relaxation cost function, we can control the trade-off between relaxing the equalization requirements from (P-FL) and performance. To illustrate this, we perform an ablation on the coefficient  $\alpha$  in the quadratic relaxation cost  $h(\mathbf{u}) = \alpha \| \mathbf{u}\| _2^2$ . As sown in Figure 2(middle) smaller values of  $\alpha$  enable better performance at the cost of larger relaxations. As  $\alpha$  increases, the relaxations vanish and the problem approaches the original constrained problem. In this manner, the resilient approach enables navigating this trade-off
Figure 2: (Left) Constraint relaxation and relative difficulty for federated learning under heterogeneous class imbalance across clients (crosses). We plot the perturbation  $u_{c}$  against the fraction of the dataset of client  $c$  from the minority class, which is associated to how difficult it is to satisfy the constraint, since minority classes typically have higher loss. (Middle) Relaxation cost parameter  $(h(\mathbf{u}) = \alpha \| \mathbf{u}\| _2^2)$  vs. final training loss and perturbation norm. (Right) Constraint violations on train and test sets
by changing a single hyperparameter. Still, the optimal relaxation for each client is determined by their local data distribution, i.e., the relative difficulty of the constraint. In Appendix F.4 we also perform an ablation on the cost function by changing the perturbation norm.
Constraint violation and generalization: Relaxing stringent requirements not only makes the empirical problem easier to solve, but it can also lead to a better empirical approximation of the underlying statistical problem. As shown in Figure 2(right), the resilient approach has a smaller fraction of clients that are infeasible at the end of training. In addition, when constraints are evaluated on the test set, larger constraint violations are observed for some (outlier) clients in the hard constrained approach  $(u_{c} = 0)$ . In Appendix F.4 we show that this holds across different problem settings. In addition, we also illustrate the fact that this is partly due to generalization issues, i. e., that overly stringent requirements can harm generalization. This observation is in line with the constrained learning theory developed in [7, 8].
Comparing client performances: Our method should sacrifice the performance of outliers—which represent hard to satisfy constraints—in order to benefit the performance of the majority of the clients. In order to showcase this, we sort clients according to their test accuracy  $\mathrm{Acc}_{[1]} \geq \mathrm{Acc}_{[2]} \geq \ldots \geq \mathrm{Acc}_{[c]}$ . We report the fraction of clients in the resilient method that outperform equally ranked clients for the constrained baseline [18] ( $\mathrm{Acc}_{[c]}^{\mathrm{res}} \geq \mathrm{Acc}_{[c]}^{\mathrm{const}}$ ), as well as the average and maximum increases and decreases in performance. As shown in Table 1, the majority of clients achieve a small increase in performance, while a small fraction experiences a larger decrease. In Appendix F.4 we include plots of ordered accuracies to further illustrate this.
Dataset	Imbalance Ratio	Improved over constrained (%)	Max Improvement	Max Deterioration
CIFAR10	10	77	1.5	10.0
20	79	2.1	9.1
F-MNIST	10	92	4.8	23.3
20	94	2.6	19.4
Table 1: Changes in accuracy for equally ranked clients for the resilient method compared to the constrained baseline [18]. We report the fraction of equally ranked clients which improved their accuracy, along with the mean and maximum change across all clients that improve (imp.) and decrease (dec.) their accuracy, respectively. Performance improves for most clients, although at the cost of a decrease in performance for a few outliers
5.2 Invariance Constrained Learning
As in [21], we impose a constraint on the loss on transformed inputs, as a mean of achieving robustness to transformations which may describe invariances or symmetries of the data. Explicitly,
$$
\min  _ {\boldsymbol {\theta} \in \Theta}. \quad \mathbb {E} _ {(\mathbf {x}, y) \sim \mathfrak {D}} [ \ell (f _ {\boldsymbol {\theta}} (\mathbf {x}), y) ] \tag {P-IC}
$$
$$
\mathrm {s . t o} \quad \mathbb {E} _ {(\mathbf {x}, y) \sim \mathfrak {D}} \left[ \max  _ {g \in \mathcal {G} _ {i}} \ell (f _ {\boldsymbol {\theta}} (g \mathbf {x}), y) \right] - \epsilon \leq 0, \qquad i = 1, \ldots , m,
$$
Figure 3: (Left) Constraint Relaxation and Relative difficulty for Synthetically invariant datasets. Bars denote the perturbation  $\mathbf{u}_i$  associated with different transformations, and the x-axis shows the synthetic invariance of the dataset. (Right) Sensitivity (Dual variables) for Synthetically invariant datasets. Bars correspond to the values of dual variables (for all constraints) at the end of training. We compare the resilient and constrained approach across different synthetic datasets.
where  $\mathcal{G}_i$  is a set of transformations  $g: \mathcal{X} \to \mathcal{X}$  such as image rotations, scalings, or translations. The constraint can be approximated by sampling augmentations using MCMC methods (see [12, 21] for details). However, it is challenging to specify the transformation sets  $\mathcal{G}_i$  and constraint levels  $\epsilon_i$  adequately, since they depend on the invariances of the unknown data distribution and whether the function class is sufficiently expressive to capture these invariances. Therefore, we propose to solve a resilient version of problem (P-IC) using  $h(\mathbf{u}) = \alpha \| \mathbf{u} \|_2^2$  as a perturbation cost.
We showcase our approach on datasets with artificial invariances, following the setup of [23]. Explicitly, we generate the synthetic datasets, by applying either rotations, translations, or scalings, to each sample in the MNIST [48] and FashionMNIST [49] datasets. The transformations are sampled from uniform distributions over the ranges detailed in Appendix H.
Constraint relaxation and relative difficulty As shown in Figure 3(left) the relaxations  $(\mathbf{u})$  associated with synthetic invariances are considerably smaller than for other transformations. This indicates that when the transformations in the constraint correspond to a true invariance of the dataset, the constraint is easier to satisfy. On the other hand, when the transformations are not associated with synthetic invariances of the dataset, they are harder to satisfy and thus result in larger relaxations. This results in smaller dual variables, as shown in Figure 3 (right).
Resilience Improves Performance The resilient approach is able to handle the misspecification of invariance requirements, outperforming in terms of test accuracy both the constrained and unconstrained approaches. We include results for MNIST in Table 2 and F-MNIST in Appendix H. In addition, though it was not designed for that purpose, our approach shows similar performance to the invariance learning method Augerino [22].
Dataset	Method	Rotated (180)	Rotated (90)	Translated	Scaled	Original
MNIST	Augerino	97.78 ± 0.03	96.38 ± 0.00	94.65 ± 0.01	97.53 ± 0.00	98.44 ± 0.00
Unconstrained	94.49 ± 0.12	96.25 ± 0.13	94.64 ± 0.20	97.47 ± 0.03	98.45 ± 0.06
Constrained	94.55 ± 0.18	96.90 ± 0.07	93.74 ± 0.07	97.92 ± 0.15	98.74 ± 0.08
Resilient	95.38 ± 0.18	97.19 ± 0.09	95.21 ± 0.15	98.20 ± 0.04	98.86 ± 0.02
Table 2: Classification accuracy for synthetically invariant MNIST. We use the same invariance constraint level  $\epsilon_{i} = 0.1$  for all transformations. We include the invariance learning method Augerino [22] as a baseline.
This paper introduced a method to specify learning constraints by balancing the marginal decrease in the objective value obtained from relaxation with the marginal increase in a relaxation cost. This resilient equilibrium has the effect of prioritizing the relaxation of constraints that are harder to satisfy. The paper also determined conditions under which this equilibrium exists and provided an algorithm to automatically find it during training, for which approximation and statistical guarantees were derived. Experimental validations showcased the advantages of resilient constrained learning for classification with invariance requirements and federated learning. Future work includes exploring different relaxation costs and applications to robustness against disturbances and outliers.
The work of A. Ribeiro and I. Hounie is supported by NSF-Simons MoDL, Award 2031985, NSF AI Institutes program, Award 2112665, and NSF HDR TRipods Award 1934960. The work of Dr. Chamon is supported by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy (EXC 2075-390740016).
A Constrained learnability and relaxations
In this section, we illustrate how constraints affect learnability, an issue that can be exacerbated by relaxing their specification either too much or too little. Resilient constrained learning, on the other hand, progressively adapts to the underlying problem, striking a balance between modifying the learning task and improving the objective value that can overcome these issues.
Let us begin by stating what we mean by constrained learnability (see [38, Def. 4] for details).