SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. PRBM is a translation transformation with parameters  $\bar{z}_i^p$ , s.t.  $\forall \boldsymbol{x}_i \in \mathcal{U}_i, \boldsymbol{z}_i = \boldsymbol{z}_i^g + \bar{\boldsymbol{z}}_i^p$ , where  $\boldsymbol{z}_i = \mathcal{F}'(\boldsymbol{x}_i) \in \tilde{\mathcal{U}}_i'$  and  $\boldsymbol{z}_i^g = \mathcal{F}^g(\boldsymbol{x}_i) \in \tilde{\mathcal{U}}_i^g$ . In other words,  $\forall \boldsymbol{z}_i^g \in \tilde{\mathcal{U}}_i^g$ ,  $\exists! \boldsymbol{z}_i \in \tilde{\mathcal{U}}_i'$ . Therefore, we have  $\operatorname*{Pr}_{\tilde{\mathcal{U}}_i^g}[\{\boldsymbol{z} \in \mathcal{Z}\}] = \operatorname*{Pr}_{\tilde{\mathcal{U}}_i'}[\{\boldsymbol{z} \in \mathcal{Z}\}]$  and the same applies to the pair of  $\tilde{\mathcal{U}}^g$  and  $\tilde{\mathcal{U}}'$ , i.e.,  $\operatorname*{Pr}_{\tilde{\mathcal{U}}^g}[\{\boldsymbol{z} \in \mathcal{Z}\}] = \operatorname*{Pr}_{\tilde{\mathcal{U}}'}[\{\boldsymbol{z} \in \mathcal{Z}\}]$ . Then the subtraction of the probability on each side is also equal, i.e.,
$$
\operatorname * {P r} _ {\tilde {\mathcal {U}} _ {i} ^ {g}} \left[ \{\boldsymbol {z} \in \mathcal {Z} \} \right] - \operatorname * {P r} _ {\tilde {\mathcal {U}} ^ {g}} \left[ \{\boldsymbol {z} \in \mathcal {Z} \} \right] = \operatorname * {P r} _ {\tilde {\mathcal {U}} _ {i} ^ {\prime}} \left[ \{\boldsymbol {z} \in \mathcal {Z} \} \right] - \operatorname * {P r} _ {\tilde {\mathcal {U}} ^ {\prime}} \left[ \{\boldsymbol {z} \in \mathcal {Z} \} \right].
$$
$\forall h^{\prime}\in \mathcal{H},h^{g} = h^{\prime}\circ \mathrm{PRBM}\in \mathcal{H},$  so  $\forall z^a\in \mathcal{Z}$  if  $h^g (z^a) = 1$  , then  $h^\prime (z^b) = 1$  , where  $z^{b} = z^{a} + \bar{z}_{i}^{p}$  Therefore, we have
$$
\operatorname * {P r} _ {\tilde {\mathcal {U}} _ {i} ^ {g}} \left[ \mathcal {Z} _ {h ^ {g}} \right] - \operatorname * {P r} _ {\tilde {\mathcal {U}} _ {i} ^ {g}} \left[ \mathcal {Z} _ {h ^ {g}} \right] = \operatorname * {P r} _ {\tilde {\mathcal {U}} _ {i} ^ {f}} \left[ \mathcal {Z} _ {h ^ {f}} \right] - \operatorname * {P r} _ {\tilde {\mathcal {U}} ^ {f}} \left[ \mathcal {Z} _ {h ^ {f}} \right],
$$
where  $\mathcal{Z}_{h^g} = \{\pmb {z}\in \mathcal{Z}:h^g (\pmb {z}) = 1\}$ $h^g\in \mathcal{H}$  and  $\mathcal{Z}_{h'} = \{\pmb {z}\in \mathcal{Z}:h'(\pmb {z}) = 1\}$ $h^\prime \in \mathcal{H}$ . According to Definition 1, we have
$$
\begin{array}{l} d _ {\mathcal {H}} \left(\tilde {\mathcal {U}} ^ {\prime}, \tilde {\mathcal {U}} _ {i} ^ {\prime}\right) = 2 \sup _ {h ^ {\prime} \in \mathcal {H}} \left| \operatorname * {P r} _ {\tilde {\mathcal {U}} _ {i} ^ {\prime}} \left[ \mathcal {Z} _ {h ^ {\prime}} \right] - \operatorname * {P r} _ {\tilde {\mathcal {U}} ^ {\prime}} \left[ \mathcal {Z} _ {h ^ {\prime}} \right] \right| \\ = 2 \sup  _ {h ^ {g} \in \mathcal {H}} \left| \Pr_ {\tilde {\mathcal {U}} _ {i} ^ {g}} \left[ \mathcal {Z} _ {h ^ {g}} \right] - \Pr_ {\tilde {\mathcal {U}} ^ {g}} \left[ \mathcal {Z} _ {h ^ {g}} \right] \right| \\ = d _ {\mathcal {H}} \left(\tilde {\mathcal {U}} ^ {g}, \tilde {\mathcal {U}} _ {i} ^ {g}\right) \\ \leq d _ {\mathcal {H}} \left(\tilde {\mathcal {U}}, \tilde {\mathcal {U}} _ {i}\right). \\ \end{array}
$$
C Detailed Settings
We create the datasets for each client using six public datasets: Fashion-MNIST (FMNIST)<sup>3</sup>, Cifar100<sup>4</sup>, TinyImageNet<sup>5</sup> (100K images with 200 labels) and AG News<sup>6</sup> (a news classification dataset with four labels, more than 30K samples per label). The MDL is calculated through the public code<sup>7</sup>. We run all experiments on a machine with two Intel Xeon Gold 6140 CPUs (36 cores), 128G memory, eight NVIDIA 2080 Ti GPUs, and CentOS 7.8.
C.2 Hyperparameters of DBE
For hyperparameter tuning, we use grid search to find optimal hyperparameters, including  $\kappa$  and  $\mu$ . Specifically, grid search is performed in the following search space:
$\kappa$  : 0, 0.001, 0.01, 0.1, 1, 5, 10, 20, 50, 100, 200, 500
$\mu$  : 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
In this paper, we set  $\kappa = 50, \mu = 1.0$  for the 4-layer CNN,  $\kappa = 1, \mu = 0.1$  for the ResNet-18, and  $\kappa = 0.1, \mu = 1.0$  for the fastText. We only set different values for the hyperparameters  $\kappa$  and  $\mu$  on different model architectures but use identical settings for one architecture on all datasets. Different models exhibit diverse capabilities in both feature extraction and classification. Given that our proposed DBE operates by integrating itself into a specific model, it is crucial to tune the parameters  $\kappa$  and  $\mu$  to adapt to the feature extraction and classification abilities of different models.
As for the criteria for hyperparameter tuning,  $\kappa$  and  $\mu$  require different tunning methods according to their functions. Specifically,  $\mu$  is a momentum introduced along with the widely-used moving average technology in approximating statistics, so for the model architectures that originally contain statistics collection operations (e.g., the batch normalization layers in ResNet-18) one can set a relatively small value by tuning  $\mu$  from 0 to 1 with a reasonable step size. For other model architectures, one can set a relatively large value for  $\mu$  by tuning it from 1 to 0. The parameter  $\kappa$  is utilized to regulate the magnitude of the MSE loss in MR. However, different architectures generate feature representations with varying magnitudes, leading to differences in the magnitude of the MSE loss. Thus, we tune  $\kappa$  by aligning the magnitude of the MSE loss with the other loss term.
<sup>3</sup>https://pytorch.org/vision/stable/datasets.html#fmnist
<sup>4</sup>https://pytorch.org/vision/stable/datasets.html#cifar
<sup>5</sup>http://cs231n.stanford.edu/tiny-imagenet-200.zip
<https://pytorch.org/text/stable/datasets.html#ag-news
<sup>7</sup>https://github.com/willwhitney/reprieve
D.1 Convergence
Figure 4: The training loss and test accuracy curve of FedAvg+DBE on FMNIST dataset using the 4-layer CNN in the practical setting.
Recall that our objective is
$$
\min  _ {\boldsymbol {\theta} _ {1}, \dots , \boldsymbol {\theta} _ {N}} \mathbb {E} _ {i \in [ N ]} \left[ \mathcal {L} _ {\mathcal {D} _ {i}} \left(\boldsymbol {\theta} _ {i}\right) \right], \tag {11}
$$
and its empirical version is  $\min_{\theta_1,\dots,\theta_N}\sum_{i=1}^N\frac{n_i}{n}\mathcal{L}_{\hat{\mathcal{D}}_i}(\pmb{\theta}_i)$ . Here, we visualize the value of  $\sum_{i=1}^N\frac{n_i}{n}\mathcal{L}_{\hat{\mathcal{D}}_i}(\pmb{\theta}_i)$  and the corresponding test accuracy during the FL process. Figure 4 shows the convergence of FedAvg+DBE and its stable training procedure. Besides, we also report the total iterations required for convergence on Tiny-ImageNet using ResNet-18 in Table 8. Based on the findings from Table 8, we observe that the utilization of DBE can yield a substantial reduction from 230 to 107 (more than  $50\%$ ) in the total number of communication iterations needed for convergence, as compared to the original requirements of FedAvg.
D.2 Model-Splitting in ResNet-18
In the main body, we have shown that DBE improves the per-layer MDL and accuracy of FedAvg no matter how we split the 4-layer CNN. In Table 7, we report the per-layer MDL and accuracy when we consider model splitting in ResNet-18, a model deeper than the 4-layer CNN. No matter at which layer, we split ResNet-18 to form a feature extractor and a classifier, DBE can also reduce MDL and improve accuracy, showing its general applicability.
Table 7: The MDL (bits, ↓) of layer-wise representations, test accuracy (\%, ↑), and the number of trainable parameters (↓) in PRBM when adding DBE to FedAvg on Tiny-IIImageNet using ResNet-18 in the practical setting. The “B”, “CONV”, “POOL”, and “FC” means the “block”, “convolution block”, “average pool layer”, and “fully connected layer” in ResNet-18 [29], respectively.
Metrics	MDL	Accuracy	Param.
CONV→B1	B1→B2	B2→B3	B3→B4	B4→POOL	POOL→FC	Logits
Original (FedAvg)	4557	4198	3598	3501	3445	3560	3679	19.45	0
CONV→DBE→B1	4332	4050	3528	3407	3292	3347	3493	19.96	16384
B1→DBE→B2	4527	4072	3568	3456	3361	3451	3560	19.50	16384
B2→DBE→B3	4442	4091	3575	3474	3326	3411	3520	19.55	8192
B3→DBE→B4	4447	4073	3511	3414	3259	3346	3467	20.72	4096
B4→DBE→POOL	4424	4030	3391	3304	3284	3511	3612	39.99	2048
POOL→DBE→FC	4432	4035	3359	3298	3209	3454	3594	42.98	512
D.3 Distinguishable Representations
As our primary goal is to demonstrate the elimination of representation bias rather than improving discrimination in Figure 3 (main body), we present the t-SNE visualization for our largest dataset in experiments, Tiny-ImageNet (200 labels). Given that the 200 labels are distributed around the chromatic circle, adjacent labels are assigned
similar colors, resulting in Figure 3 (main body) being indistinguishable by the label. Using a dataset AG News with only four labels for t-SNE visualization can clearly show that the representations extracted by the global feature extractor are distinguishable in Figure 5.
Figure 5: t-SNE visualization for the representations extracted by the global feature extractor on AG News (four labels) in FedAvg+DBE. We use color and shape to distinguish labels and clients, respectively.
D.4 A Practical Scenario with New Participants
To simulate a practical scenario with new clients joining for future FL, we perform method-specific local training for 10 epochs on new participants for warming up after their local models are initialized by the learned global model (or client models in FedFomo). Since FedAvg, Per-FedAvg, and FedBABU do not generate personalized models during the FL process, we fine-tune the entire global model on new clients for them to obtain test accuracy. Specifically, using Cifar100 and 4-layer CNN, we conduct FL on 80 old clients  $(\rho = 0.5$  or  $\rho = 0.1)$  and evaluate accuracy on 20 new joining clients after warming up. We utilize the data distribution depicted in Figure 9. According to Table 8, FedAvg shows excellent generalization ability with fine-tuning. However, DBE can still improve FedAvg by up to  $+6.68$  with more stable performance for different  $\rho$ .
Table 8: The total iterations for convergence and the averaged test accuracy  $(\% ,\uparrow)$  of pFL methods.
Items	Iterations	New Participants	Local Epochs
ρ = 0.5	ρ = 0.1	1	5	10
Per-FedAvg [22]	34	48.66	48.36	95.10	93.92	93.91
pFedMe [67]	113	41.20	38.39	97.25	97.44	97.32
Ditto [47]	27	36.57	45.06	97.47	97.67	97.64
FedPer [3]	43	39.86	42.39	97.44	97.50	97.54
FedRep [20]	115	38.75	35.09	97.56	97.55	97.55
FedRoD [14]	50	50.10	51.73	97.52	97.49	97.35
FedBABU [61]	513	48.60	42.29	97.46	97.57	97.65
APFL [21]	57	38.19	45.16	97.25	97.31	97.34
FedFomo [89]	71	27.50	27.47	97.21	97.17	97.22
APPLE [52]	45	—	—	97.06	97.07	97.01
FedAvg	230	52.52	49.44	85.85	85.96	85.53
FedAvg+DBE	107	57.62	56.12	97.69	97.75	97.78
D.5 Large Local Epochs
We also conduct experiments with more local epochs in each iteration on FMNIST using the 4-layer CNN, as shown in Table 8. All the pFL methods perform similarly with the results for one local epoch, except for Per-FedAvg, which degenerates around 1.18 in accuracy (\%).
Table 9: The test accuracy  $(\%)$  on the HAR dataset.
Methods	Accuracy
FedAvg	87.20±0.27
SCAFFOLD	91.34±0.43
FedProx	88.34±0.24
MOON	89.86±0.18
FedGen	90.82±0.21
Per-FedAvg	77.12±0.17
pFedMe	91.57±0.12
Ditto	91.53±0.09
FedPer	75.58±0.13
FedRep	80.44±0.42
FedRoD	89.91±0.23
FedBABU	87.12±0.31
APFL	92.18±0.51
FedFomo	63.39±0.48
APPLE	86.46±0.35
FedAvg+DBE	94.53±0.26
D.6 Real-World Application
We also evaluate the performance of our DBE in a real-world application. Specifically, we apply DBE to the Internet-of-Things (IoT) scenario on a popular Human Activity Recognition (HAR) dataset [2] with the HAR-CNN [81] model. HAR contains the sensor signal data collected from 30 users who perform six activities (WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STANDING, LAYING) wearing a smartphone on the waist. We show the results in Table 9, where FedAvg+DBE still achieves superior performance.
E Broader Impacts
The representation bias and representation degeneration naturally exist in FL under statistically heterogeneous scenarios, which are derived from the inherently separated local data domains on individual clients. In the main body, we show the general applicability of our proposed DBE to representative FL methods. More than that, DBE can also be applied to other practical fields, such as the Internet of Things (IoT) [26, 31, 59] and digital health [15, 16]. Furthermore, introducing the view of knowledge transfer into FL sheds light on this field.
Although FL comes along for privacy-preserving and collaborative learning, it still suffers from privacy leakage issues with malicious clients [12, 93] or under attacks [23, 53]. We design DBE based on FL to improve generalization and personalization abilities, and we only modify the local training procedure without affecting the downloading, uploading, and aggregation processes. Thus, the DBE-equipped FL methods still suffer from the originally existing privacy issues like the original version of these FL methods when attacks happen. It requires future work to devise specific methods for privacy-preserving enhancement.
G Data Distribution Visualization
We illustrate the data distributions (including training and test data) in our experiments here.
(b) Cifar100
Figure 6: The data distributions of all clients on FMNIST, Cifar100, and Tiny-ImageNet, respectively, in the pathological settings. The size of a circle represents the number of samples.
(c) Tiny-ImageNet
(a) FMNIST
Figure 7: The data distributions of all clients on FMNIST, Cifar100, and Tiny-ImageNet, respectively, in the practical settings  $(\beta = 0.1)$ . The size of a circle represents the number of samples.
(b) Cifar100
(a)  $\beta = 0.01$
Figure 8: The data distribution on all clients on Tiny-ImageNet in three additional practical settings. The size of a circle represents the number of samples. The degree of heterogeneity decreases as  $\beta$  in  $Dir(\beta)$  increases.
(b)  $\beta = 0.5$
(c)  $\beta = 5$
Figure 9: The data distributions of all clients on Cifar100 in the practical setting  $(\beta = 0.1)$  with 100 clients, respectively. The size of a circle represents the number of samples.