SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Different from the prior works [2, 28] for federated DG adopting the usual setting where the number of clients equals the number of source domains, in our main paper, we introduce a more practical experimental setting for federated DG where the source data is distributed to more clients than the number of source domains. For a comparison with them in the same setting, we also provide additional experimental results in the setup with number of clients = number of source domains. Table 7 shows the results on PACS and Office-Home datasets with three clients in a single-domain data distribution scenario. It is confirmed from the results that our StableFDG also achieves better performance compared to the existing works [2, 28] in this simple setting.
Methods	Art	Cartoon	Photo	Sketch	Avg.
CCST [2]	81.25	73.34	95.21	80.27	82.52
FedSR [28]	83.20	76.00	93.80	81.90	83.70
StableFDG	83.01	79.31	94.85	79.76	84.23
(a) Results on PACS dataset.
Methods	Art	Clipart	Product	Real	Avg.
CCST [2]	59.05	50.06	72.97	71.67	63.56
FedSR [28]	57.93	50.45	73.33	75.51	64.31
StableFDG	57.19	57.94	72.76	72.16	65.01
(b) Results on Office-Home dataset.
D Effect of Each Component of StableFDG
As mentioned in the main manuscript, we provide further ablation studies on the effect of each component in StableFDG using VLCS dataset. Table 8 shows that each component individually brings performance gain compared to FedAvg. Using both strategies achieves greater performance gains, confirming that the two proposed schemes work in a complementary fashion.
Table 7: Results in a single-domain data distribution scenario with three clients.
Methods	Caltech	Labelme	Pascal	Sun	Avg.
FedAvg [26]	93.65 ± 1.59	61.10 ± 2.08	72.55 ± 0.90	65.40 ± 0.28	73.18 ± 0.27
StableFDG (only style)	98.19 ± 0.81	58.93 ± 1.16	75.19 ± 0.67	68.69 ± 0.73	75.25 ± 0.20
StableFDG (only attetnion)	94.10 ± 1.22	62.02 ± 1.55	72.26 ± 0.78	66.16 ± 2.58	73.64 ± 0.89
StableFDG (both)	98.50 ± 0.17	60.07 ± 0.79	74.40 ± 1.88	69.43 ± 1.11	75.61 ± 0.71
E Ablation Studies on Style-Based Learning
E.1 Randomness in style sharing
In our style based learning, style sharing among clients is performed at random. However, one can think of the strategy where client  $n$  receives the style information  $\Phi_{n'}$  that has the largest distance with its own style information  $\Phi_{n'}$  in the style space. Table 9 shows the corresponding result using PACS dataset in a multi-domain data distribution setup. Interestingly, it can be seen that the random selection adopted in this paper performs better, since most of the users generally tend to receive the same style statistics when using the largest distance strategy.
Table 8: Effect of each component on VLCS dataset in a multi-domain data distribution scenario. The reported results indicate (mean ± 95% confidence interval) over 3 random trials.
Methods	Art	Cartoon	Photo	Sketch	Avg.
Random sharing (current manuscript)	83.97 ± 1.25	79.10 ± 0.45	96.27 ± 0.36	75.67 ± 0.58	83.75 ± 0.23
Large distance	83.01 ± 1.43	78.33 ± 0.51	96.19 ± 0.76	74.63 ± 0.65	83.04 ± 0.30
Table 9: Random sharing vs receiving the style that has the largest distance in style space. The reported results indicate (mean ± 95% confidence interval) over 3 random trials.
E.2 Effect of the number of shared styles
We also compare the effect of number of styles received at each client in Table 10 using PACS dataset in a multi-domain data distribution setup. The performance increases as the number of received styles increases, with small additional communication load (the vector length of the style information is 128, which is negligible compared to the number of model parameters, which is 11,180,103).
Methods	Art	Cartoon	Photo	Sketch	Avg.
No style sharing	83.16 ± 0.18	78.54 ± 1.11	95.56 ± 0.92	74.64 ± 1.06	82.97 ± 0.36
Receive 1 style (current manuscript)	83.97 ± 1.25	79.10 ± 0.45	96.27 ± 0.36	75.67 ± 0.58	83.75 ± 0.23
Receive 3 styles	84.55 ± 1.20	78.67 ± 0.59	95.75 ± 0.58	76.77 ± 0.27	83.94 ± 0.07
Table 10: Effect of the number of shared styles. The reported results indicate (mean  $\pm$ $95\%$  confidence interval) over 3 random trials.
E.3 Effect of  $k$ -means++ in style shifting
In Section 3.1, we utilized the  $k$ -means++ as a tool for facilitating our key idea in style shifting, which is to effectively balance between the original source domain and the new source domain for better generalization;  $k$ -means++ plays a role to select the  $B/2$  styles that are similar to the remaining  $B/2$  styles in the mini-batch. By doing so, the model can explore new styles while not losing the performance on the original styles. To see this effect, we compare  $k$ -means++ vs. random sampling when selecting  $B/2$  samples to be shifted, in Table 11. The results show that strategically selecting the  $B/2$  samples to be shifted achieves better performance especially in the Sketch domain (1.89% gain) that has a large style gap with other domains. We believe that these results motivate and justify our design choice.
Methods	Art	Cartoon	Photo	Sketch	Avg.
Shifting B/2 random styles	83.69	79.61	95.99	73.78	83.27
Shifting B/2 styles via k-means++	83.97	79.10	96.27	75.67	83.75
Table 11: Effect of  $k$ -means++ in style shifting on PACS dataset.
E.4 Effect of class-balanced oversampling
In our main paper, we performed the class-balanced oversampling in the feature space to alleviate the class-imbalance issue during style exploration. To confirm the effectiveness of the class-balanced oversampling, we compare it with random oversampling under the same condition where only the style exploration is applied without other components. Table 12 shows the results on Office-Home dataset, where the class distribution is highly imbalanced in a multi-domain data distribution scenario. It can be seen that our class-balanced oversampling achieves higher performance over the simple random sampling, which validates the efficacy of mitigating the class imbalance problem in FL clients.
Methods	Art	Cartoon	Photo	Sketch	Avg.
Random oversampling	56.21	54.43	69.37	72.29	63.08
Class-balanced oversampling	57.20	53.15	71.2	74.23	64.10
Table 12: Effect of the class-balanced oversampling on Office-Home dataset in a multi-domain data distribution scenario.
E.5 Effect of the operation probability in style-based learning
We conduct additional ablation studies on the probability value (defined as  $p$  here) utilized to control the operation of the style sharing/shifting and style exploration modules. Larger  $p$  means that our scheme is more likely to be activated. In Table 13, we provide results on various probability values in a single-domain data distribution scenario using PACS dataset. From the results, it is confirm that for all  $p$  values, the proposed StableFDG outperforms existing baselines, demonstrating that our scheme can work well with an arbitrarily chosen probability  $p$ . In detail, when  $p$  ranges from 0.3 to 0.7, the high performance is maintained while the performance decreases at both extreme probabilities  $(p = 0.1$  and 0.9). Therefore, it is recommended for practitioners to select the  $p$  in an appropriate range, avoiding extreme cases.
Methods	Art	Cartoon	Photo	Sketch	Avg.
MixStyle [39]	79.10	76.30	90.10	60.63	76.53
DSU [21]	80.43	75.70	92.60	69.87	79.65
StableFDG (p=0.1)	82.70	78.30	95.30	75.30	82.90
StableFDG (p=0.3)	84.50	79.50	96.00	75.70	83.93
StableFDG (p=0.5)	83.10	79.50	96.40	76.00	83.75
StableFDG (p=0.7)	82.40	78.10	95.70	76.30	83.13
StableFDG (p=0.9)	81.40	78.80	95.90	73.60	82.70
E.6 Where to apply the style module
For implementation, style-based learning is applied only in the 1st, 2nd, 3rd blocks among 4 residual blocks in ResNet-18. Note that at the output of the 4th block, label information is dominant rather than style information, which results in degraded performance when style-based schemes are applied. This is confirmed by our new experiments in the table below. It can be seen from the results that if we consider the 4th residual block to apply our style-based learning, the performance gets degraded. This result confirms the intuition that style-based learning should be conducted at the earlier layers where style information is preserved.
Table 13: Effect of the probability value in our style-based learning on PACS in a single-domain data distribution scenario.
Methods	Art	Cartoon	Photo	Sketch	Avg.
Style exploration at 1st, 2nd, 3rd layers (main manuscript)	84.10	78.57	95.40	72.73	82.70
Style exploration at 1st, 2nd, 3rd, 4th layers	82.99	78.54	94.13	73.35	82.25
F Ablation Studies on Attention-based Feature Highlighter
F.1 Effect of adopting both cross-attention and self-attention
In our main paper, the similarity metric in equation (6) adopts cross-attention, while the metric in equation (8) combines cross-attention and self-attention. When applying only the cross-attention-based metric in equation (6), we found that the similarity value could become low even when the two samples belong to the same class, in special cases. We handled this issue by adding the self-attention component as in equation (8). Intuitively, by doing this, the attention module is learning to extract the important features across images (via cross-attention), and within the image (via self-attention). Table 15 compares the performance of our StableFDG when using (i) self-attention alone, (ii) cross-attention alone (equation (6)), and (iii) both self and cross attentions at the same time (equation (8)), confirming the advantage of using self-attention and cross-attention together.
Table 14: Ablation experiments on applying style-based learning at different layers (PACS dataset).
Methods	Clipart	Infograph	Painting	Quickdraw	Real	Sketch	Avg.
StableFDG (with self-attention alone)	61.77	24.88	48.28	14.15	59.78	52.41	43.55
StableFDG (with cross-attention alone)	61.24	24.97	50.55	14.67	61.12	50.44	43.83
StableFDG (with self + cross)	62.58	24.12	52.23	14.87	60.60	52.50	44.48
Table 15: Effects of similarity metrics using DomainNet dataset in a multi-domain data distribution scenario.
F.2 Comparison using the same model size
Our attention module requires  $0.44\%$  of additional model parameters to perform the attention-based learning. For a fair comparison to see the effect of our attention-based learning, we consider a
different baseline with the same model size but without attention-based learning. Specifically, the baseline computes the attention score map using only additional convolutional operations and take the weighted average of the feature  $z_{i}$  based on the attention score map. Table 16 shows the results using PACS dataset in a single-domain data distribution scenario. The results demonstrate that our attention-based learning achieves performance improvements on all four domains while playing a key role in capturing essential parts of the features.
Methods	Art	Cartoon	Photo	Sketch	Avg.
Baseline (same model size)	78.88	69.03	91.74	60.77	75.11
Attention-based learning	79.54	72.48	92.51	64.71	77.31
F.3 Effect of attention in a centralized setup
Now we provide answer to the following question: Instead of the FL setup we focused on, can attention provide benefits in the centralized DG setup? Table 17 shows the results with/without attention module in a centralized setup using PACS dataset. The results show that attention still provides performance improvements in the centralized setup by learning domain-invariant features, although the gain is slightly lower than the gain in the FL setup as shown in Table 3 of the main manuscript. These results indicate that the proposed attention-based learning indeed captures the domain-invariant characteristics of samples, while the scheme provides more benefits in the FL setup where each client is prone to overfitting due to lack of data.
Table 16: Ablation study on attention-based learning using PACS dataset in a single-domain data distribution scenario.
Methods	Art	Cartoon	Photo	Sketch	Avg.
StableFDG (centralized setup, without attention)	84.15	79.45	96.21	77.09	84.23
StableFDG (centralized setup, with attention)	85.02	79.65	96.38	78.45	84.88
Table 17: Effect of proposed attention-based feature highlighter in a centralized DG setup using PACS dataset.
G Other Implementation Details
Our code is built upon the official code of [39] and [1]. During the local update process, we use SGD as an optimizer with a momentum of 0.9 and a weight decay of  $5e^{-4}$ . For PACS, Office-Home and VLCS, the learning rate is set to 0.001 and the cosine annealing is used as a scheduler. For Digits-DG, we set the learning rate to 0.02 and the learning rate is decayed by 0.1 every 20 steps. For our attention-based feature highlighter, at least two samples are required to be in the mini-batch of every client. When this condition is not met, additional samples are extracted from the corresponding client's local dataset to facilitate cross-attention.
More detailed description on oversampling: Let  $s^n \in \mathbb{R}^{B \times C \times H \times W}$  be a mini-batch of features in client  $n$  at a specific layer, obtained after Steps 1 and 2 in the main manuscript. Now given a fixed oversampling size, we oversample the features in the mini-batch to obtain  $\tilde{s}^n$ , so that the concatenated mini-batch  $\hat{s}^n = [s^n, \tilde{s}^n]$  becomes class-balanced as much as possible. Consider a toy example where the number of samples for classes  $a, b, c$  are 3, 2, 1, respectively in the mini-batch  $s^n$ . In this example, if the oversampling size is 3, we randomly choose one data point from class  $b$  and two data points from class  $c$  (in this case, the same data point is selected for two times with duplication) to obtain  $\tilde{s}^n$ , so that the concatenated mini-batch  $\hat{s}^n = [s^n, \tilde{s}^n]$  becomes class-balanced. If the oversampling size is 1, we oversample one data point in class  $c$  to make the concatenated mini-batch to be balanced as much as possible. If the oversampling size is 6, we oversample 1, 2, 3 samples from classes  $a, b, c$ , respectively to construct  $\tilde{s}^n$ . The concatenated mini-batch  $\hat{s}^n = [s^n, \tilde{s}^n]$  is utilized for style-based learning and updating the model. This process not only mitigates the class-imbalance issue in each client but also provides a good platform for style exploration by oversampling the features. In our
work, we reported the results with oversampling size of  $B$  (which is equal to the mini-batch size), while the effect of the oversampling size is also reported in Fig. 4 of the main manuscript: A larger oversampling size leads to a better performance, and more importantly, our StableFDG outperforms the baseline even without any oversampling.
H Visualization of Attention Score Maps
Finally in Fig. 6, we visualize the attention score maps of different input images during testing. The results confirm the effectiveness of our attention-based feature highlighter to focus on the important parts of each image from the unseen domain.
Figure 6: Visualization of attention score maps of each input image.
I Pseudo Code Algorithm of Stable FDG
Algorithm 1 summarizes the overall process of our StableFDG.
Algorithm 1 StableFDG
Input: Initialized model  $\mathbf{w}_0$ , Output: Global model  $\mathbf{w}_T$   
1: for each global round  $t = 0,1,\dots,T-1$  do  
2:  
3: Stage 1. Model download and style sharing  
4: The server samples a set of participating clients  $M_t$  and sends the global model  $\mathbf{w}_t$  to the clients in  $M_t$   
5: for each device  $n \in M_t$  do  
6: Compute style information  $\Phi_n$  according to the Step 1 in Sec. 3.1  
7: Transmit  $\Phi_n$  to the server  
8: end for  
9: The server shares  $\{\Phi_n\}_{n \in M}$  to the clients in  $M_t$  according to the Step 1 in Sec. 3.1  
10:  
11: Stage 2. Local updates and model aggregation  
12: for each device  $n \in M_t$  do  
13: for local epoch  $= 1,2\dots,E$  do  
14: (i) Style shifting according to the Step 2 in Sec. 3.1,  
15: (ii) Style exploration according to the Step 3 and 4 in Sec. 3.1  
16: (iii) Attention based weighted averaging according to Sec. 3.2  
17: Loss computation and model update  
18: end for  
19: The server updates the global model by aggregating the client models  
20: end for  
21:  
22: end for