SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\gamma_{3} = (1 + \frac{\Delta_{1}}{\|(\mathbf{D}_{u}^{\intercal}\mathbf{X}^{\intercal}\boldsymbol{\alpha}^{\intercal}\boldsymbol{\alpha}\mathbf{X}\mathbf{D}_{u})\|_{F}^{2}})^{-\frac{1}{4}}(1 + \frac{\Delta_{1}}{\|(\mathbf{D}_{v}^{\intercal}\mathbf{X}^{\intercal}\boldsymbol{\alpha}^{\intercal}\boldsymbol{\alpha}\mathbf{X}\mathbf{D}_{v})\|_{F}^{2}})^{-\frac{1}{4}}.$
As Assumption 2.6 holds, it becomes
$$
\begin{array}{l} \frac {\gamma_ {1} \gamma_ {2} \gamma_ {3}}{\sqrt {c}} \frac {\left\| \mathbf {D} _ {u} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {v} \right\| _ {F}}{\left\| \left(\mathbf {D} _ {u} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {u}\right) \right\| _ {F} ^ {\frac {1}{2}} \left\| \left(\mathbf {D} _ {v} ^ {\intercal} \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X D} _ {v}\right) \right\| _ {F} ^ {\frac {1}{2}}} \\ = \frac {\gamma_ {1} \gamma_ {2} \gamma_ {3}}{\sqrt {c}} \frac {\left\| \mathbf {D} _ {u} ^ {\intercal} \mathbf {D} _ {v} \right\| _ {F} \left\| \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X} \right\| _ {F}}{\left\| \mathbf {D} _ {u} ^ {\intercal} \right\| _ {F} ^ {\frac {1}{2}} \left\| \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X} \right\| _ {F} ^ {\frac {1}{2}} \left\| \mathbf {D} _ {u} \right\| _ {F} ^ {\frac {1}{2}} \left\| \mathbf {D} _ {v} ^ {\intercal} \right\| _ {F} ^ {\frac {1}{2}} \left\| \mathbf {X} ^ {\intercal} \boldsymbol {\alpha} ^ {\intercal} \boldsymbol {\alpha} \mathbf {X} \right\| _ {F} ^ {\frac {1}{2}} \left\| \mathbf {D} _ {v} \right\| _ {F} ^ {\frac {1}{2}}} \tag {55} \\ = \frac {\gamma_ {1} \gamma_ {2} \gamma_ {3}}{\sqrt {c}} \frac {\| \mathbf {D} _ {u} ^ {\intercal} \mathbf {D} _ {v} \| _ {F}}{\| \mathbf {D} _ {u} \| _ {F} \| \mathbf {D} _ {v} \| _ {F}} \\ = \frac {\gamma_ {1} \gamma_ {2} \gamma_ {3}}{\sqrt {c}} \mathbf {c o s} (\mathbf {D} _ {u}, \mathbf {D} _ {v}). \\ \end{array}
$$
$$
\mathcal {S} \left(\mathbf {Z} _ {u}, \mathbf {Z} _ {v}\right) = \frac {\gamma_ {1} \gamma_ {2} \gamma_ {3}}{\sqrt {c}} \cos \left(\mathbf {D} _ {u}, \mathbf {D} _ {v}\right). \tag {56}
$$
Specifically, we have  $\gamma_{2} = \sqrt{1 - \frac{\Delta}{\gamma_{1}^{2}\gamma_{3}^{2}}\frac{1}{\cos^{2}(\mathbf{D}_{u},\mathbf{D}_{v})}}$ , and since  $\Delta$  are small, with Taylor expansion,  $\gamma_{2} \approx 1 - \frac{1}{2}\frac{\Delta}{\gamma_{1}^{2}\gamma_{3}^{2}}\frac{1}{\cos^{2}(\mathbf{D}_{u},\mathbf{D}_{v})}$ . The term  $\frac{1}{\cos^2(\mathbf{D}_u,\mathbf{D}_v)}$  causes non-linearity in the relation between CCA and filter subspace similarity.
Model training of Federated Learning. In each experiment we have 100 clients in total and sample a ratio  $r = 0.1$  of all the clients on every round. All models are randomly initialized and trained for  $T = 100$  communication rounds for the CIFAR datasets. At each round, the client executes 15 epochs of SGD with momentum to train the local model, the learning rate is 0.01 and momentum is 0.9. Accuracies are computed by taking the average local accuracies for all users at the final communication round. As shown in the Table 3, we have different settings for CIFAR-10 and CIFAR-100. For example, (100, 2) means 100 clients with 2 classes on each client. For each method, the training takes about 12 hours on Nvidia RTX A5000.
Comparison with other FL approaches. We compare our approach by evolving shared atom coefficients with various personalized federated learning methods and federated learning methods with local finetuning. Among these methods, FedPer [2] and FedRep[8] have the similar ideas by learning shared global representation and personalized local heads. Ditto [27] and FedProx [29] induce global regularization to improve the model performance. We also compare our method with FedAvg [34]. FedRep [8] approaches the common knowledge with shared representation. The codes
Table 3: Compare accuracy with different approaches
CIFAR-10	CIFAR-10
(# client, # classes per client)	(100, 5)	(100, 20)	(100, 2)	(100, 5)	(1000, 2)
FedAvg	82.39	62.92	86.37	70.63	86.12
FedProx	80.77	59.7	85.90	69.94	84.83
FedPer	81.46	62.52	81.74	68.24	81.74
FedRep	72.98	37.71	80.55	67.3	82.98
Local	81.21	49.25	90.24	72.05	97.80
Ours	81.03	52.13	83.37	65.63	82.54
are adapted from  $\dagger$ . We evaluate the test accuracy on CIFAR-10 and CIFAR-100 with different FL setting. As shown in Table 3, our method achieves comparable performance among different methods.
Fine-tuning models for ensemble. We select 3 models with different similarity measures for ensemble. For feature-based similarity methods, we randomly select 1000 examples from CIFAR-100 dataset. The fully-connected layer of each model is fine-tuned on the user's local data with 100 epochs. The fine-tuning takes about 12 hours on Nvidia RTX A5000. After fine-tuning, the accuracy is measured on local test data, with the predictions of current model and 3 selected models.
A.3 Extra Experiments
Representation dependency on filter atoms. We first validate the dependency of deep features on filter atoms in Proposition 2.1 with a simple experiment. The model  $\mathcal{F}$  here is a 2-layer CNN with coefficient  $\alpha$  and atom  $\mathbf{D}$  generated from normal distribution  $\mathcal{N}(0,1)$ . The input sample  $\mathbf{X}$  is also generated from normal distribution  $\mathcal{N}(0,1)$ . Figure 8(a) shows the relation between  $\| \mathbf{Z}_u - \mathbf{Z}_v\|_F$  and  $\| \mathbf{D}_u - \mathbf{D}_v\|_F$  by fixing coefficient  $\alpha$  and input sample  $\mathbf{X}$  and randomly varying filter atoms  $\mathbf{D}$ . All the points are below the line which is the bound provided by Proposition 2.1, reflecting that the representation variations are dominated by filter atoms.
Correlation between probing-based and filter subspace-based methods. In addition, we empirically verify that CCA and filter subspace similarity have a strong correlation with AlexNet. In this experiment, 10 tasks are generated from CIFAR100 [23] with 10 classes in each task. Only the filter atoms of each task are trained while the atom coefficients are fixed. We calculate CCA and filter subspace similarity among 45 pairs of models. The correlation between CCA and filter subspace similarity is 0.8638 which is shown in Figure 9(b). Similarly, the correlation between CKA and filter subspace similarity is also reported in Figure 9 (Table). These results clearly show that the proposed filter subspace similarity has high linear relationship with popular probing-based similarities, which agrees with Theorem 2.5 and Theorem 2.7.
Figure 6: The shared coefficients and user-specific atoms represent common knowledge and personalized information. The filter subspace similarity is used to calculate the relations among users. Users with heterogeneous data result in lower similarity, as illustrated in a similarity matrix.
$\dagger$ https://github.com/lgcollins/FedRep
(b) 50th epoch
(a)
Figure 8: (a) The change of features  $\| \mathbf{Z}_u - \mathbf{Z}_v\| _F$  is bounded by the change of atoms  $\| \mathbf{D}_u - \mathbf{D}_v\| _F$ . (b) The channel decorrelation leads to a higher correlation between CCA and filter subspace similarity. And the correlation can reach 0.985 with  $\beta = 3\times 10^{-3}$ , which means a near linear relation between CCA and filter subspace similarity.
Figure 7: Similarity matrices that show relations among 120 users in FL with our filter subspace similarity through the training process.
(b)
Effect of channel decorrelation. We further design a regularization term  $\beta \sum_{i\neq j}(\mathbf{Z}_u^\intercal \mathbf{Z}_u)^2_{ij}$  to approach  $(\mathbf{Z}_u^\intercal \mathbf{Z}_u)_{ii}\gg (\mathbf{Z}_u^\intercal \mathbf{Z}_u)_{ij}$  in Assumption. 2.6. As shown in Figure 8(b), the correlation between CCA and filter subspace similarity keeps increasing as  $\beta$  increases. The correlation reaches 0.985 when  $\beta = 3\times 10^{-3}$ , indicating a near-linear relationship, which is aligned with Theorem 2.7.
Similar representations across datasets. Similar to [21], we can use filter subspace similarity to compare networks trained on different datasets. In Figure 10(a), we show that pairs of models that are both trained on CIFAR-10 and CIFAR-100 have high atom-based similarities. Models learned on two datasets respectively still show high similarity. In contrast, similarities between trained and untrained models are significantly lower.
Limitation of probing-based methods. As shown in Figure 10(b), to illustrate sensitivity of probing-based similarities to probing data, we perform a simple regression task with data,  $\{(x_{i} = 0,y_{i},z_{i})\}_{i = 1}^{n}$ , where  $z_{i} = f(x_{i},y_{i}) + \epsilon_{i}$  and  $y_{i},\epsilon_{i}\sim \mathcal{N}(0.5,0.1)$ . Two NN models  $\mathcal{F}_1$  and  $\mathcal{F}_2$
(a)
Figure 9: (a) Correlation between Grassmann similarity and filter subspace similarity; (b) Correlation between CCA and filter subspace similarity. (Table) Correlation between filter subspace similarity and other approaches.
Correlation
CCA [43]	0.8638
CKA [21]	0.7358
Grassmann Distance [35]	0.8793
Figure 10: (a) Using filter subspace similarity, models trained on different datasets (CIFAR-10 and CIFAR-100) are similar among themselves, but they differ from untrained models. (b) Illustration of limitations of probing-based similarities. Input data from "red"  $\left(\left(x_{i} = 0,y_{i}\right)\right)$  and "blue"  $\left(\left(x_{i}^{\prime} = y_{i},y_{i}^{\prime} = 0\right)\right)$  are orthogonal. Since two models are learned on "red" data, their similarity should be 1, which can be faithfully indicated by our atom similarity. However, probing-based similarities will become 0 with the "blue" probing data.
(b) 2nd conv
(c) 3rd conv
(a) 1st conv
(d) 4th conv
(e) 5th conv
Figure 11: Similarity of AlexNet with atoms from different time point during the training.
with the same initialization and atom coefficients are trained for their different atoms to learn  $\mathcal{F}:(X,Y)\to Z$ . It is can be simply found that the filter subspace similarity of  $\mathcal{F}_1$  and  $\mathcal{F}_2$  is 1 and the probing-based similarity is also 1 with the same  $\{(x_i = 0,y_i)\}$  as the probing data. However, if we choose  $\{(x_i' = y_i,y_i' = 0)\}$  as the probing data, then the probing-based similarities directly become  $\mathbf{0}$  as the data are now orthogonal to model parameters.
A.4 Training dynamics.
We investigate the training dynamics of AlexNet [24] and VGG [50] separately on CIFAR-100 [23] and ImageNet [47]. The details of training dynamics of models with atoms from different time point during the training are shown in Figure 11 and Figure 12. Moreover, we examine the similarity between the two participated models shared the same initialization trained only with atoms on two different tasks. The results is shown in Figure 13 and Figure 14. The difference is less on the first few layers, but more on the middle layers. It reflects the middle layer is more critical than other layers, which is aligned with previous work [39].
(a) 1st conv
(e) 5th conv
Figure 12: Similarity of VGG with atoms from different time point during the training.
(b) 2nd conv
(f) 6th conv
(c) 3rd conv
(g) 7th conv
(d) 4th conv
(h) 8th conv
(a) Model 0 vs. 1
(b) Model 0 vs. 2
(c) Model 0 vs. 3
(d) Model 0 vs. 4
(g) Model 0 vs. 7
Figure 13: Similarity of AlexNet trained on different tasks during the training.
(e) Model 0 vs. 5
(h) Model 0 vs. 8
(f) Model 0 vs. 6
(i) Model 0 vs. 9
(d) Model 0 vs. 4
(e) Model 0 vs. 5
(f) Model 0 vs. 6
(g) Model 0 vs. 7
(h) Model 0 vs. 8
(i) Model 0 vs. 9
Figure 14: Similarity of VGG trained on different tasks during the training.