SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As shown in Table 4, both the mean and spread statistics are similar for  $\beta = 1,2,4$ , i.e. our approach is not overly sensitive to the choice of cost function in this experimental setup. However,  $\beta = \infty$  does show a substantially different behaviour. Penalizing the only the largest perturbation results in a smaller range while attaining a higher IQR and lower mean accuracy. That is, it reduces the worst client test accuracy, but the performance of most clients deteriorates, both in terms of its average and spread. The infinity norm is related to Rawlsian (i.e. minimax) formulations which have been proposed in the context of fairness see e.g. [14].
G Ablation on dual and resilient learning rates
We also perform an ablation on  $\eta_u, \eta_\lambda$ , the perturbation and dual learning rates, over a small grid of 12 values. In this particular setup, we find that the performance of the algorithm is not overly sensitive to this choice. We also observe that the rates that were used in the paper ( $\eta_u = 0.1, \eta_\lambda = 2$ ) are not optimal in this case, and thus further improvements in performance could be obtained through
Dataset	Imb. Ratio	Mean	IQR	Range
[60]	[18]	Ours	[60]	[18]	Ours	[60]	[18]	Ours
F-MNIST	10	92.5	92.6	93.4	2.2	3.5	2.4	64.1	28.6	50.6
20	94	93.8	94.4	2.2	2.7	1.7	50.2	28.6	45.8
CIFAR10	10	81.3	81.5	81.5	8.1	8.7	8.7	49.4	35.8	46.4
20	83.4	82.4	82.6	8.6	8.4	7.9	44.5	32	41.1
Table 3: Client accuracy spread metrics for different setups. IQR denotes interquantile range and range denotes the maximum minus the minimum accuracy, both computed across 100 clients.
Figure 9: Test Accuracy for equally ranked clients for the resilient and constrained baseline [18] in CIFAR10.
Figure 10: Test Accuracy for equally ranked clients for the resilient and constrained baseline [18] in CIFAR10.
more extensive hyperparameter tuning. However, the main aim of our numerical experiments is to validate empirically the properties of our approach.
H Invariance Constrained Learning Experiments
H.1 Experimental setup.
We showcase our approach on datasets with artificial invariances, following the setup of [23]. Explicitly, we generate the synthetic datasets, by applying either rotations, translations or scalings, to each sample in the MNIST [48] and FashionMNIST [49] datasets. The transformations are sampled from uniform distributions over the ranges detailed in Table 6. We use the same MLP and CNN architectures and hyperparameters as [23], except that we use only 6 augmentations per sample instead of 31 during training.
For each transformation set (rotations, translations and scalings), we constraint the expected loss over samples augmented with the transformations sampled from uniform distributions over the ranges detailed in Table 7. Note that there is a mismatch between the distribution used to generate the
β	Mean Acc.	IQR	Max Range
1.0	93.3	2.5	49.5
2.0	93.4	2.4	50.6
4.0	93.4	2.6	45.6
∞	92.7	3.8	28.1
Table 4: Cost function ablation for the heterogeneous federated learning in fashion-MNIST using 100 clients, 3 minority classes, an imbalance ratio of  $\rho = 10$  and dirichlet allocation with parameter  $d = 0.3$ . We report the mean, interquartile range and range (maximum value minus minimum value) test accuracy across clients.
ηu\ηλ	0.1	0.5	1	2
0.1	81.4	81.6	81.8	81.8
0.5	81.4	80.9	81.4	81.1
1	81.6	81.6	81.6	81.7
data and that used in the constraints. That is, except for the fully rotated dataset, the constraints are larger than the true transformation range used to construct the synthetic dataset (Table 6). The purpose of this experiments is to showcase that the resilient approach can relax constraints associated to transformation sets that do not describe symmetries or invariances of the data and can thus hinder performance. We use the same transformation sets and constraint specification  $(\epsilon)$  for all synthetic datasets.
Table 5: Dual and resilient learning rate ablation in Heterogenous federated learning setting. We report mean Test Accuracy for CIFAR100 using 100 clients, 3 minority classes, an imbalance ratio of  $\rho = 10$  and dirichlet allocation with parameter  $d = 0.3$ .
Synthetic invariance	Parameter	Distribution
Full Rotation	Angle in radians.	U[-π/2, π/2]
Partial Rotation	Angle in radians.	U[-π, π]
Translation	Translation in pixels.	U[-8, 8]2
Scale	Exponential Scaling factor.	U[-log(2), log(2)]
Table 6: Sampling parameters for transformations used to obtain synthetically invariant datasets, from [23]
Constraint Set	Parameter	Range
Rotations	Angle in radians.	[−π, π]
Translation	Translation in pixels.	[−16, 16]2
Scale	Exponential Scaling factor.	[−1.5, 1.5]
H.2 Results
H.2.1 Ablation on perturbation cost.
As in the case of federated learning (presented in section 4) perform an ablation on the cost function  $h$  using  $\| u\|_{\beta}$  with  $\beta = 1,2,4,\infty$ .
As shown in table 8,  $\beta = 1$  showed slightly better performance in terms of average test accuracy across all invariant settings. As discussed in Section 3.3,  $\beta = 1$  recovers penalty based methods, i.e. it is equivalent to setting a fixed penalization coefficient. Nonetheless, the dynamics of our algorithm are different and can thus lead to different solutions.
Table 7: Transformation sets  $\mathcal{G}_i$  used as invariance constraints. All sets are used simultaneously, with the same constraint level  $(\epsilon_i)$  for all datasets (0.1).
β	Partially Rotated	Translated	Scaled
1.0	86.08 ± 0.38	86.85 ± 0.20	85.02 ± 0.46
2.0	85.37 ± 0.17	86.65 ± 0.25	84.92 ± 0.39
4.0	85.23 ± 0.20	86.64 ± 0.11	84.65 ± 0.68
∞	82.94 ± 0.17	85.47 ± 0.19	83.35 ± 0.49
Table 8: Cost function ablation for invariant fashion-MNIST datasets. We compute the mean and standard deviation in test accuracy across three independent runs.
H.2.2 Performance on F-MNIST
The resilient approach is able to handle the misspecification of invariance requirements, outperforming in terms of test accuracy both the constrained and unconstrained approaches. In addition, though it was not designed for that purpose, our approach shows similar performance to the invariance learning method Augerino [22].
Dataset	Method	Rotated (180)	Rotated (90)	Translated	Scaled	Original
F-MNIST	Augerino	85.28 ± 0.54	81.48 ± 0.49	81.13 ± 0.77	83.17 ± 0.46	90.09 ± 0.20
Unconstrained	77.94 ± 0.06	81.57 ± 0.36	79.23 ± 0.17	82.99 ± 0.18	90.20 ± 0.23
Constrained	84.96 ± 0.12	85.66 ± 0.32	83.61 ± 0.10	86.49 ± 0.09	91.02 ± 0.02
Resilient	85.57 ± 0.26	86.48 ± 0.15	85.06 ± 0.23	87.26 ± 0.14	91.55 ± 0.31
Table 9: Classification accuracy for synthetically invariant F-MNIST. We use the same invariance constraint level  $\epsilon_{i} = 0.1$  for all datasets and transformations. We include the invariance learning method Augerino [22] as a baseline.