SlowGuess/NeurIPS_2023_OCR huggingface dataset:
First, we will show sufficient conditions for  $\mathbf{u}^{\star}\preceq \mathbb{1}\epsilon$  and  $\mathbf{u}^{\star}\succeq \mathbf{b} - \mathbb{1}\epsilon$ , in terms of  $\partial \tilde{\mathsf{P}}^{\star}$  and  $\nabla h$  evaluated at  $1\epsilon$  and  $\mathbf{b} - 1\epsilon$ , respectively. For any  $\mathbf{u}\in \mathbb{R}_+^m$  the strong duality of  $\tilde{\mathsf{P}}^{\star}$  (lemma 2) implies that  $\partial \tilde{\mathsf{P}}^{\star}(\mathbf{u})$  is non empty and bounded. Also, the convexity of  $\tilde{\mathsf{P}}^{\star}$  (assumption 1) implies that for all  $\mathbf{p_u}\in \partial \tilde{\mathsf{P}}^{\star}(\mathbf{u})$  and  $\mathbf{p_{u}^{\star}}\in \partial \tilde{\mathsf{P}}^{\star}(\mathbf{u}^{\star})$  by (31) from proposition (3) it holds that
$$
\left\langle \mathbf {p} _ {\mathbf {u} ^ {\star}} - \mathbf {p} _ {\mathbf {u}}, \mathbf {u} ^ {\star} - \mathbf {u} \right\rangle \geq 0 \tag {43}
$$
Analogously, due to the convexity of  $h$  we have that
$$
\left\langle \nabla h (\mathbf {u} ^ {\star}) - \nabla h (\mathbf {u}), \mathbf {u} ^ {\star} - \mathbf {u} \right\rangle \geq 0 \tag {44}
$$
Adding (43) and (44) we obtain
$$
\left\langle \left[ \mathbf {p} _ {\mathbf {u} ^ {\star}} + \nabla h (\mathbf {u} ^ {\star}) \right] - \left[ \mathbf {p} _ {\mathbf {u}} + \nabla h (\mathbf {u}) \right], \mathbf {u} ^ {\star} - \mathbf {u} \right\rangle \geq 0 \tag {45}
$$
(45) holds for any  $\mathbf{p}_{\mathbf{u}^{\star}} \in \partial \tilde{\mathsf{P}}^{\star}(\mathbf{u}^{\star})$ . Since  $\mathbf{u}^{\star}$  achieves the resilient equilibrium, by definition  $-\nabla h(\mathbf{u}^{\star}) = \mathbf{p}_{\mathbf{u}^{\star}} \in \partial \tilde{\mathsf{P}}^{\star}(\mathbf{u}^{\star})$ , which implies that for all  $\mathbf{p}_{\mathbf{u}} \in \partial \tilde{\mathsf{P}}^{\star}(\mathbf{u})$
$$
\left\langle \left[ \mathbf {p} _ {\mathbf {u}} + \nabla h (\mathbf {u}) \right], \mathbf {u} ^ {\star} - \mathbf {u} \right\rangle \leq 0 \tag {46}
$$
Finally, by choosing particular values of  $\mathbf{u}$  in (46) we obtain the desired conditions.
$(\mathbf{u}^{\star}\preceq \mathbf{1}\epsilon)$  : Letting  $a = \epsilon$  ,(46) implies that if there exists
$$
\mathbf {p} _ {\mathbb {1} \epsilon} \in \partial \tilde {\mathrm {P}} ^ {\star} (\mathbb {1} \epsilon) \text {s u c h t h a t} \mathbf {p} _ {(\mathbb {1} \epsilon)} + h (\mathbb {1} \epsilon) \succ 0 \text {t h e n} \mathbf {u} ^ {\star} \preceq \mathbb {1} \epsilon . \tag {47}
$$
$(\mathbf{u}^{\star}\succeq \mathbb{1}\mathbf{b} - \mathbb{1}\epsilon)$  : Letting  $a = \mathbf{b} - \mathbb{1}\epsilon$  , (46) implies that if there exists
$$
\mathbf {p} _ {(\mathbf {b} - \mathbb {1} \epsilon) _ {+}} \in \partial \tilde {\mathbf {P}} ^ {\star} ((\mathbf {b} - \mathbb {1} \epsilon) _ {+}) \text {s u c h t h a t} \mathbf {p} _ {(\mathbf {b} - \mathbb {1} \epsilon) _ {+}} + h ((\mathbf {b} - \mathbb {1} \epsilon) _ {+}) \prec 0 \text {t h e n} \mathbf {u} ^ {\star} \succeq (\mathbf {b} - \mathbb {1} \epsilon) _ {+}. \tag {48}
$$
Let  $h(\mathbf{u}) = \alpha \| \mathbf{u} \|_2^2$ ,  $\alpha \in \mathbb{R}_+$ , which has  $\nabla h(\mathbf{u}) = 2\alpha \mathbf{u}$ . We will now show that there exist coefficients  $\bar{\alpha}$  and  $\underline{\alpha}$  such that the associated costs  $\bar{h}$  and  $\underline{h}$  satisfy the conditions (47) and (48).
$(\underline{h})$  : Let  $\underline{\alpha} >\frac{\|\mathbf{p}_{1\epsilon}\|_{\infty}}{2\epsilon}$  , then
$$
\nabla \underline {{h}} (\mathbb {1} \epsilon) _ {i} = 2 \underline {{\alpha}} \epsilon > [ - \mathbf {p} _ {\mathbb {1} \epsilon} ] _ {i} \text {f o r a l l} i = 1, \dots , m
$$
which satisfies (47) and thus  $\mathbf{u}^{\star}\preceq \mathbb{1}\epsilon$
$(\bar{h})$  : Let  $\bar{\alpha} <  \min_{i}\frac{[-\mathbf{p}_{(\mathbf{b} - \mathbb{1}\epsilon)_{+}}]^{i}}{2(\mathbf{b}_{i} - \epsilon)}$  , then
$$
\nabla \bar {h} ((\mathbf {b} - \mathbb {1} \epsilon) _ {+}) _ {i} = 2 \bar {\alpha} (\mathbf {u} _ {i} - \epsilon) <   [ \mathbf {p} _ {\mathbb {1} (\mathbf {b} _ {i} - \epsilon)} ] _ {i} \text {f o r a l l} i = 1, \dots , m
$$
which implies that  $\bar{h}$  satisfies (48) and thus  $\bar{\mathbf{u}}^{\star} \succeq \mathbf{b} - \mathbb{1}\epsilon$ . Since the definition of  $\mathbf{b}$  and  $\epsilon$  ensures that  $\min_{i} \frac{[-\mathbf{P}_{(\mathbf{b} - \mathbb{1}\epsilon)_+}]_i}{2(\mathbf{b}_i - \epsilon)} > 0$ , there exists  $\bar{\alpha} > 0$  that satisfies the above condition.
E Primal and Dual Updates
In section 4 we introduced the unconstrained problem (D-RES) that approximates our original problem  $(\mathrm{P_u})$  as the saddle point of the empirical Lagrangian  $\hat{L}_{\theta}$ :
$$
\hat {L} _ {\theta} (\theta , \boldsymbol {\lambda}; \mathbf {u}) = h (\mathbf {u}) + \frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {0} \left(f _ {\theta} \left(\mathbf {x} _ {n, 0}\right), y _ {n, 0}\right) + \sum_ {i = 1} ^ {m} \lambda_ {i} \left(\frac {1}{N} \sum_ {n = 1} ^ {N} \ell_ {i} \left(f _ {\theta} \left(\mathbf {x} _ {n, i}\right), y _ {n, i}\right) - u _ {i}\right). \tag {49}
$$
As described in Algorithm 1 he saddle point can be obtained by alternating the minimization and the maximization of the Lagrangian with respect to the primal variables and dual variables respectively. We now describe the updates of both primal and dual variables in greater detail. For clarity, we
describe the updates using gradient descent and sub-gradient ascent for primal and dual variables, respectively. However, nothing precludes our method from using other optimization algorithms, such as Adam [54] or SGD with nesterov momentum.
The optimization is done via gradient descent and  $\mathbf{u}$  and stochastic gradient descent for the primal variables  $\theta$
$$
\theta^ {(t + 1)} = \theta^ {(t)} - \eta_ {\theta} d \hat {\theta} ^ {(t)} \tag {50}
$$
$$
\mathbf {u} ^ {(t + 1)} = \mathbf {u} ^ {(t)} - \eta_ {\mathbf {u}} d \mathbf {u} ^ {(t)},
$$
and stochastic subgradient ascent for the dual variables
$$
\boldsymbol {\lambda} ^ {(t + 1)} = \boldsymbol {\lambda} ^ {(t)} + \eta_ {\boldsymbol {\lambda}} d \hat {\boldsymbol {\lambda}} ^ {(t)}. \tag {51}
$$
In order to obtain the gradients of the Lagrangian with respect to the primal variables, observe that the minimization of the Lagrangian can be separated into two parts that each depend on only one primal variable.
$$
\hat {\mathcal {L}} (\theta , \boldsymbol {\lambda}, \mathbf {u}) = \hat {\mathcal {L}} _ {\theta} (\theta , \boldsymbol {\lambda}) + \hat {\mathcal {L}} _ {\mathbf {u}} (\boldsymbol {\lambda}, \mathbf {u}) \tag {52}
$$
where  $\hat{\mathcal{L}}_{\phi}(\phi ,\pmb {\lambda}) = \hat{\mathcal{L}} (\phi ,\pmb {\lambda};0)$  and  $\hat{\mathcal{L}}_{\mathbf{u}}(\pmb {\lambda},\mathbf{u})$  is
$$
\hat {\mathcal {L}} _ {\mathbf {u}} (\boldsymbol {\lambda}, \mathbf {u}) = h (\mathbf {u}) - \boldsymbol {\lambda} ^ {\top} \mathbf {u}. \tag {53}
$$
The gradient with respect to  $\mathbf{u}$  is obtained from (53)
$$
d \mathbf {u} = \frac {\partial \hat {\mathcal {L}} (\boldsymbol {\lambda} , \mathbf {u})}{\partial \mathbf {u}} = \nabla h (\mathbf {u}) - \boldsymbol {\lambda}. \tag {54}
$$
Solving for the optimal  $\theta$  is similar to solving a regularized empirical risk minimization problem. The gradient with respect to  $\theta$  is estimated from  $\hat{\mathcal{L}} (\theta ,\lambda ;0)$  using a minibatch of  $B$  samples as
$$
d \hat {\theta} = \frac {1}{B} \sum_ {n = 1} ^ {B} \left(\nabla_ {\theta} \ell_ {0} \left(f _ {\theta} \left(\mathbf {x} _ {n}\right), y _ {n}\right) + \sum_ {i = 1} ^ {m} \nabla_ {\theta} \ell_ {i} \left(f _ {\theta} \left(\mathbf {x} _ {n}\right), y _ {n}\right)\right). \tag {55}
$$
The constraints evaluated at the optimal Lagrangian minimizers are supergradients of the corresponding Lagrange multipliers [55]. In the same manner, stochastic supergradient ascent is done by updating the dual variable  $\lambda$  with the super-gradient estimated using a batch of  $B$  samples.
$$
\hat {d} \boldsymbol {\lambda} _ {i} = \frac {1}{B} \sum_ {n = 1} ^ {B} \ell_ {i} \left(f _ {\theta} \left(\mathbf {x} _ {n}\right), y _ {n}\right) - \mathbf {u} _ {i} i = 1 \dots m. \tag {56}
$$
F Heterogenous Federated Learning Experiments
F.1 Formulation
In this section, we show that the constrained problem  $P$ -FL can be re-written as a constrained learning problem of the form of  $P$ , and then introduce its resilient version,
Let  $\mathfrak{D}_i$  be the distribution of data pairs for Client  $i$ , and  $R_{i}(f_{\theta}) = \mathbb{E}_{(\mathbf{x},y)\sim \mathfrak{D}_{i}}[\ell (f_{\theta}(\mathbf{x}),y)]$  its statistical risk. We denote the average performance as  $\overline{R} (f_{\pmb{\theta}})\coloneqq (1 / C)\sum_{i = 1}^{C}R_{i}(f_{\pmb{\theta}})$ , where  $C$  is the number of clients. As proposed in [18] heterogeneity issues in federated learning can be tackled by imposing a proximity constraint between the performance of each client  $R_{i}$ , and the loss averaged over all clients  $\overline{R}$ . This leads to the constrained learning problem:
$$
\min  _ {\theta \in \Theta}. \quad \bar {R} (f _ {\theta}) \tag {P-FL}
$$
$$
s. t o \quad R _ {i} (f _ {\boldsymbol {\theta}}) - \bar {R} (f _ {\boldsymbol {\theta}}) - \epsilon \leq 0, \quad i = 1, \dots , C,
$$
where  $C$  is the number of clients and  $\mathfrak{D}_i$  is the distribution of data pairs for client  $i$ , and  $\epsilon$  is a small (fixed) positive scalar.
In order to show that problem  $P$ -FL can be re-written as a constrained learning problem of the form of  $\mathbf{P}$ , we introduce a random variable  $c$  uniformly distributed over client indices  $\{1, \dots, C\}$ , i.e.,
$$
P (c = i) = \frac {1}{C} \text {f o r a l l} i = 1, \dots , C
$$
Then, we construct a mixture distribution  $\overline{\mathfrak{D}}$  for data pairs such that
$$
(\mathbf {x}, y) | c = i \sim \mathfrak {D} _ {i}.
$$
Then we can re-write the average risk  $R(f_{\pmb{\theta}})$  as the expected loss under  $\overline{\mathfrak{D}}$ :
$$
\begin{array}{l} \mathbb {E} _ {\overline {{\mathfrak {D}}}} \left[ \ell \left(f _ {\boldsymbol {\theta}} (\mathbf {x}), y\right) \right] = \sum_ {i = 1} ^ {C} \frac {1}{C} \mathbb {E} _ {\mathfrak {D} _ {i}} \left[ \ell \left(f _ {\boldsymbol {\theta}} (\mathbf {x}), y\right) \right] \\ = R _ {i} \left(f _ {\boldsymbol {\theta}}\right), \\ \end{array}
$$
where the first inequality holds by law of total expectation.
Let  $\ell_i(f_\theta(\mathbf{x}), y) = (C\mathbb{1}(c = i) - 1)\ell(f_\theta(\mathbf{x}), y) - \epsilon$ , where  $\mathbb{1}(c = i)$  is the indicator function for client  $i$ .
In the same manner, we can re-write the  $i - th$  constraint  $R_{i}(f_{\pmb{\theta}}) - \overline{R} (f_{\pmb{\theta}}) - \epsilon$  as the expectation of the loss  $\ell_i$  under  $\overline{\mathfrak{D}}_i$ :
$$
\begin{array}{l} \mathbb {E} _ {\overline {{\mathfrak {D}}}} \left[ \ell_ {i} (f _ {\boldsymbol {\theta}} (\mathbf {x}), y) \right] = \mathbb {E} _ {\mathfrak {D} _ {i}} \left[ \ell (f _ {\boldsymbol {\theta}} (\mathbf {x}), y) \right] - \epsilon - \sum_ {j = 1} ^ {C} \frac {1}{C} \mathbb {E} _ {\mathfrak {D} _ {j}} \left[ \ell (f _ {\boldsymbol {\theta}} (\mathbf {x}), y) \right] \\ = R _ {i} (f _ {\boldsymbol {\theta}}) - \bar {R} (f _ {\boldsymbol {\theta}}) - \epsilon . \\ \end{array}
$$
Then, we can re-write  $(P$ -FL) as
$$
\begin{array}{l} \mathsf {P} ^ {\star} = \min  _ {\theta \in \Theta} \quad \mathbb {E} _ {\overline {{\mathcal {D}}}} \left[ \ell \left(f _ {\theta} (\mathbf {x}), y\right) \right] \\ \begin{array}{l} \theta \in \Theta \\ \text {s u b j e c t t o} \end{array} \begin{array}{l} \mathbb {E} _ {\overline {{\mathcal {D}}}} \left[ \ell_ {i} \left(f _ {\theta} (\mathbf {x}), y\right) \right] \leq 0, i = 1, \dots , m. \end{array} \tag {PC-FL'} \\ \end{array}
$$
which is in the form of (P).
As motivated in section 5.1, we then propose to solve the resilient version of  $(P$ -FL):
$$
P ^ {\star} = \min  _ {\boldsymbol {\theta} \in \Theta} R (f _ {\boldsymbol {\theta}}) + h (\mathbf {u})
$$
(PR-FL)
$$
\begin{array}{l} s. t o \mathbb {E} _ {(\mathbf {x}, y) \sim \mathfrak {D} _ {i}} [ \ell (f _ {\boldsymbol {\theta}} (\mathbf {x}), y) - R (f _ {\boldsymbol {\theta}}) - \epsilon ] \leq \mathbf {u} _ {i}, \\ i = 1, \ldots , C. \\ \end{array}
$$
In the next section we derive the primal dual algorithm that enables us to tackle (PR-FL) in a distributed, privacy preserving manner.
F.2 Resilient FL Algorithm
In this section we show that the resilient problem (PR-FL) can be tackled in a distributed and privacy preserving manner, as long as  $h$  is additively separable in each component. Explicitly, we assume that
$$
h (\mathbf {u}) = \sum_ {i} h _ {i} \left(\mathbf {u} _ {i}\right)
$$
which holds for the quadratic cost used throughout the experimental section. We thus derive a distributed version of the resilient primal-dual algorithm presented in Section 4 (Algorithm 1).
We begin by writing the empirical Lagrangian associated to problem (PR-FL):
$$
\begin{array}{l} \hat {\mathcal {L}} (\theta , \boldsymbol {\lambda}, \mathbf {u}) = h (\mathbf {u}) + \frac {1}{C} \sum_ {i = 1} ^ {C} \frac {1}{N _ {i}} \sum_ {n _ {i} = 1} ^ {N _ {i}} \left[ \ell \left(f _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {n _ {i}}\right), y _ {n _ {i}}\right) \right] (57) \\ + \lambda_ {i} \left[ \frac {1}{N _ {i}} \sum_ {n _ {i} = 1} ^ {N _ {i}} \left[ \ell \left(f _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {n _ {i}}\right), y _ {n _ {i}}\right) \right] - \mathbf {u} _ {i} \right] (58) \\ = \sum_ {i = 1} ^ {C} \frac {1}{C} (1 + \lambda_ {i} - \bar {\lambda}) \frac {1}{N _ {i}} \sum_ {n _ {i} = 1} ^ {N _ {i}} [ \ell \left(f _ {\boldsymbol {\theta}} \left(\mathbf {x} _ {n _ {i}}\right), y _ {n _ {i}}\right) ] - \lambda_ {i} \mathbf {u} _ {i} + h _ {i} \left(\mathbf {u} _ {i}\right), (59) \\ \end{array}
$$
with  $\bar{\lambda} = \frac{1}{C}\sum_{i = 1}^{C}\lambda_{i}$
Note that in (59) we have used the assumption that the perturbation cost is additively separable. This enables us to update the perturbation for each client locally, and thus the resilient formulation does not incur in any additional communication costs (with respect to the constrained problem in [18]). That is, clients do not need to communicate  $\mathbf{u}_i$  during optimisation. The server need only compute and communicate to all clients the average dual variable  $\bar{\lambda}$  and the average loss  $R(f_{\theta})$ . Therefore, Homomorphic Encryption techniques can be leveraged to compute these averages without revealing the values of individual dual variables  $\lambda_i$  and local losses to the server. In the same manner, Homomorphic Encryption could be used to aggregate the relaxation cost  $h(\mathbf{u}) = \sum_{i=1}^{X} h_i(\mathbf{u}_i)$  at the server without loss of privacy.
$\theta$  update. For a fixed  $\lambda$  and  $\mathbf{u}$ , the minimization of  $\mathcal{L}$  with respect to  $\theta$  is equivalent to a re-weighted version of the standard unconstrained FL objective:
$$
\min _ {\theta \in \Theta} \hat {\mathcal {L}} (\boldsymbol {\theta}, \boldsymbol {\lambda}, \mathbf {u}) \Longleftrightarrow \min _ {\theta \in \Theta} \frac {1}{C} \sum_ {i = 1} ^ {C} \left(1 + \lambda_ {i} - \bar {\lambda}\right) \frac {1}{N _ {i}} \sum_ {n _ {i} = 1} ^ {N _ {i}} \left[ \ell (f _ {\boldsymbol {\theta}} (\mathbf {x} _ {n _ {i}}), y _ {n _ {i}}) \right].
$$
Therefore,  $\theta$  can be updated using any FL solver. We thus refer to the update on  $\theta$  as a subroutine ClientUpdate  $\left(\{w_i\}_{i=1}^C, \theta^t\right) \to \theta^{t+1}$ , which can be implemented using, for example FedAVG [56] or FedPD [57].
$\mathbf{u}$  update. For a fixed  $\lambda$  and  $\theta$ , the minimization of  $\hat{\mathcal{L}}$  with respect to  $\mathbf{u}$  is can be carried independently by each client, since the objective is additively separable:
$$
\min  _ {\mathbf {u} \in \mathcal {U}} \hat {\mathcal {L}} (\theta , \boldsymbol {\lambda}, \mathbf {u}) \Longleftrightarrow \min  _ {\mathbf {u} \in \mathcal {U}} \sum_ {i = 1} ^ {C} \lambda_ {i} \mathbf {u} _ {i} + h _ {i} (\mathbf {u} _ {i})
$$
We can thus employ gradient descent (or any other optimization algorithm) locally at each client.
$\lambda$  update. For a fixed perturbation  $\mathbf{u}$  and model  $\theta$ , we perform dual update on  $\lambda$  by taking dual ascent steps of the following equivalent objective,
$$
\max  _ {\boldsymbol {\lambda} \in \mathbb {R} _ {+} ^ {N}} \hat {\mathcal {L}} (\boldsymbol {\theta}, \boldsymbol {\lambda}, \mathbf {u}) \Longleftrightarrow \max  _ {\boldsymbol {\lambda} \in \mathbb {R} _ {+} ^ {N}} \frac {1}{C} \sum_ {i = 1} ^ {C} \lambda_ {i} \left(\frac {1}{N _ {i}} \sum_ {n _ {i} = 1} ^ {N _ {i}} \left[ \ell (f _ {\boldsymbol {\theta}} (\mathbf {x} _ {n _ {i}}), y _ {n _ {i}}) \right] - R (f (\boldsymbol {\theta})) - \mathbf {u} _ {i}\right).
$$
If server then computes and communicates the average of the client's losses  $R(f(\theta))$ , the updates to each multiplier  $\lambda_{i}$  can be made locally by each client as in 56.
As stated before, these update steps can be applied in alternated manner to find a solution of the empirical dual problem, as as described in Algorithm 2.
Algorithm 2 Resilient Federated Learning
Initialize  $\theta (0),\lambda (0)$ $\mathbf{u}(0)$  , and  $0 <   \eta \ll 1$
for  $t = 1\dots T$
Compute weights: for all  $i \in [N]$ ,  $w_i = 1 + \lambda_i - \bar{\lambda}$ , with  $\bar{\lambda} = \frac{1}{N} \sum_{i=1}^{N} \lambda_i$ ;
Theta Update:  $\theta^t\gets \mathrm{ClientUpdate}\Big(\{w_i\}_{i = 1}^N,\theta^{t - 1}\Big);$
Perturbation Update:  $\mathbf{u}_i(t) = [\mathbf{u}_i(t - 1) - \eta_{\mathbf{u}}d\mathbf{u}_i(t - 1)]_+$
Dual Update:  $\pmb{\lambda}_i(t) = \left[\pmb{\lambda}_i(t - 1) + \eta_{\pmb{\lambda}}\left[\frac{1}{N_i}\sum_{j = 1}^{N_i}\ell (f_{\pmb{\theta}}(\mathbf{x}_j),y_j) - \mathbf{u}_i(t - 1)\right]\right]_+$
F.3 Experimental Setup
As in [18] we create heterogeneity between clients through class imbalance. Unless stated otherwise, we use three minority classes (Class Labels  $\{0,2,4\}$ ), and simulate the phenomenon of class imbalance by keeping only  $\rho = 10\%$  of the data data belonging to the minority classes in the training set.
We follow the implementation from [58], in which a Dirichlet prior (over the ten classes) is sampled independently for each client. One client at a time, we sample without replacement according to each client's prior. Once a class runs out of samples, the subsequent clients do not own samples of that class.
We also create a test set for each client with the same class balance as the train set. Test sets are then sampled without replacement for each client (but with replacement among different clients, i.e., test sets overlap across clients).
We use the same small neural network architecture as in [18, 58]. Namely, a CNN model consisting of 2 convolutional layers with  $645 \times 5$  filters followed by 2 fully connected layers with 384 and 192 neurons. For both the constrained and resilient formulations, we use FedAVG [56] to update  $\theta$ , using 5 communication rounds for each primal update, as in [18].
In all experiments we use 100 clients, and all clients participate from each communication round. We run 500 communication rounds in total. We set  $\epsilon = 0.1$ , dual learning rate  $\eta_{\lambda} = 0.1$ , local learning rate  $\eta_{\theta} = 5 \times 10^{-2}$  and use no data augmentation in all experiments. In the resilient learning formulation, unless stated otherwise, we use a quadratic penalty on the perturbation  $h(u) = \| u\|_2^2$  and a perturbation learning rate  $\eta_{\mathbf{u}} = 0.1$ .
All of the plots in section 5 correspond to CIFAR10 [59]. Results for FashionMNIST [49] can be found on F.4.4.
F.4 Additional Results
F.4.1 Performance Relaxation trade off.
As we have already shown, through the choice of the relaxation cost function, we can control the trade-off between relaxing requirements and performance. Figure 4 shows that changing the ablation on the coefficient  $\alpha$  in the quadratic relaxation cost  $h(\mathbf{u}) = \alpha \| \mathbf{u}\| _2^2$  not only results in larger relaxations, but that these larger relaxations effectively lead to clients with higher losses.
Figure 4: Train loss for the worst client and averaged over all clients for different values of the perturbation cost coefficient  $\alpha$ .
F.4.2 Sensitivity to Problem Specification
The resilient approach is less sensitive to the specification of the constraints. Dual variables indicate the sensitivity of the objective with respect to constraint perturbations. As shown by Figure 5 the resilient approach yields smaller dual variables, respectively of the tolerance  $\epsilon$  in the constraint specification. In this context, setting the constraint levels a priori requires knowledge about the heterogeneity of the client local distributions, which in our setup depends on the imbalance of the dataset. As shown by figure 6 the resilient approach yields smaller dual variables, irrespectively of the tolerance irrespective of the percentage of samples of minority classes that are kept, whereas the dual variables grow for more imbalanced and thus harder to satisfy scenarios.
Figure 5: Dual variables after training with respect to constraint specification  $\epsilon$ .
Figure 6: Dual variables at the end of training with respect to the percentage of minority samples  $\rho$  kept in the dataset. Smaller  $\rho$  means more imbalance, and thus results in harder to satisfy requirements.
F.4.3 Constraint violation across setups.
Relaxing stringent requirements not only makes the empirical problem easier to solve, but it can also lead to a better empirical approximation of the underlying statistical problem. That is, overly stringent requirements can result not only in large dual variables, but can also harm generalization. Figure 7 shows that more stringent constraint specifications can lead to poor constraint satisfaction in the test set. This improvement in constraint satisfaction is associated with smaller generalization gaps for the constraints were observed for the resilient approach, as shown in Figure 8.
F.4.4 Test Accuracy.
In this section we aim to give quantitative performance metrics. We include a constrained baseline [18] and another method for federated learning under class imbalance [60] for comparison.
We first compare approaches in terms of the objective, by averaging the test accuracy over all clients. As shown in Table 3, the average accuracy of the resilient approach is overall similar to the constrained approach and the baseline [60].
In order to assess how the distribution o performance among clients varies we also report spread metrics for the test accuracy across clients. As shown in Table 3, Resilient learning has (generally) less spread in the interquartile range and higher maximum spread than its constrained counterpart.
F.4.5 Comparing client performances
We sort clients according to their test accuracy  $\mathrm{Acc}_{[1]} \geq \mathrm{Acc}_{[2]} \geq \ldots \geq \mathrm{Acc}_{[c]}$ . We plot the accuracy of equally ranked clients for the constrained baseline [18] ( $\mathrm{Acc}_{[c]}^{\mathrm{res}} \geq \mathrm{Acc}_{[c]}^{\mathrm{const}}$ ). As shown in Figures 9 and 10, in all setups the majority of clients achieve a small increase in performance, while a small fraction experiences a larger decrease. For higher imbalance ratios, this is more pronounced.
Figure 7: Constraint violation (evaluated no the test set) for Resilient and constrained learning using different (left) constraint specifications  $\epsilon$ , and (right) fraction of minority classes.
Figure 8: Largest generalization error across clients for different constraint specifications e.
F.5 Ablation on perturbation cost function  $h$ .
In all experiments, we have used a quadratic perturbation cost function  $h = \alpha \| u\| _2^2$  In order to assess the impact of the choice of cost the function, we run our algorithm using as a cost  $\| u\|_{\beta}$  with  $\beta = 1,2,4,\infty$ , for fashion MNIST in both the federated and invariant setting.