StableFDG: Style and Attention Based Learning for Federated Domain Generalization
Authors: Jungwuk Park
Dong-Jun Han
Jinho Kim
Shiqiang Wang
Christopher Brinton
Jaekyun Moon
Venue: NeurIPS 2023 poster (2023)
Keywords: Federated Learning
Domain Generalization
TL;DR: We propose a style and attention based learning method tailored to federated domain generalization.

Abstract: Traditional federated learning (FL) algorithms operate under the assumption that the data distributions at training (source domains) and testing (target domain) are the same. The fact that domain shifts often occur in practice necessitates equipping FL methods with a domain generalization (DG) capability. However, existing DG algorithms face fundamental challenges in FL setups due to the lack of samples/domains in each client’s local dataset. In this paper, we propose StableFDG, a style and attention based learning strategy for accomplishing federated domain generalization, introducing two key contributions. The first is style-based learning, which enables each client to explore novel styles beyond the original source domains in its local dataset, improving domain diversity based on the proposed style sharing, shifting, and exploration strategies. Our second contribution is an attention-based feature highlighter, which captures the similarities between the features of data samples in the same class, and emphasizes the important/common characteristics to better learn the domain-invariant characteristics of each class in data-poor FL scenarios. Experimental results show that StableFDG outperforms existing baselines on various DG benchmark datasets, demonstrating its efficacy.

StableFDG: Style and Attention Based Learning for Federated Domain Generalization
Jungwuk Park KAIST eavertm@kiist.ac
Dong-Jun Han" Purdue University han7620purdue edu
Jinho Kim
SK Hynix j jinho123 finho123.kimssk. kikkkiikkkk. com
Shiqiang Wang IBM Research
Christopher G. Brinton Purdue University cgb@purdue .dd
Jackyun Moon KAIST
wangshi quus. ibm. com
jmoonOkaist.cm
Abstract
Traditional federated learning (FL.) algorithms operate under the assumption that the data distributions at training (source domains) and testing (target domain) are the same. The fact that domain shifts often occur in practice necessitates equipping FL methods with a domain generalization (DG) capability. However, existing DG algorithms face fundamental challenges in FL setups due to the lack of sam- ples/domains in each client's 's local dataset. In this paper, we propose StableFDG, a style and attention based learning strategy for accomplishing federated domain generalization, introducing two key contributions. The first is style-based learn- ing, which enables each client explore novel styles beyond the original source domains in its local dataset, improving domain diversity based on the proposed style sharing, shifting, and exploration strategies. Our second contribution is attention- feature highlighter, which captures the similarities between the features of data samples in the same class, and emphasizes the important/common characteristics to better learn the domain-invarin data-poor FL scenarios. Experimental results show that StableFDG outperforms existing baselines on various DG benchmark datasets, demonstrating its efficacy.
characteristies of each class in
Introduction
Federated learning (FL) has now become a key paradigm for training machine learning model using local data of distributed clients [26] 19 Without directly sharing each client's 's data to the third party, FL enables the clients construct a global model via collaboration. However, although FL has achieved remarkable success, the underlying assumption of previous works is that the data distributions during training and testing are the same. This assumption is not valid in various scenarios with domain shifts; for example, although the FL clients only have data samples that belong to the source domains (e.g., images on sunny days and rainy days), the trained global model should be also able to make reliable predictions for the unseen target domain (e.g., images on snowy days). Therefore, practice, FL. methods have to equipped with a domain generalization (DG) capability.
Given the source domains in the training phase, DG aims to construct model that has a generalization capability to predict well on unseen target domain. Various DG methods have been proposed in centralized setup 39 21 36 38
16 15] 23) However, directly applying centralized DG schemes FL. can potentially restrict the model performance since each client has limited numbers of data samples and styles in its local dataset. The local models are unable to capture the characteristics due lack of data/styles in individual clients.
Authors contributed equally to this work.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Although several researchers have recently focused on DG for FL 25 2 35| 28), they still do not directly handle the fundamental issues that arise from the lack of data and styles (which represent domains) in individual FI. clients. These performance limitations become especially prevalent when federating complex DG datasets having large style shifts between domains or having backgrounds unrelated to the prediction of class, as we will see in Sec. f Despite the practical significance of federated DG, this field is still in an early stage of research and remains a great challenge.
Contributions. In this paper, we propose StableFDG, a style and attention based learning strategy tailored to federated domain generalization. StableFDG tackles the fundamental challenges in federated DG that arise due to the lack of data/styles in each FL client, with two novel characteristics:
We first propose style- learning, which exposes the model of each FL client to various styles beyond the source domains in its local dataset. Specifically, we (i) design style-sharing method that can compensate for the missing styles in each client by sharing the style statistics with other clients; (ii) propose style-shifting strategy that can select the best styles to be shifted to the new style to balance between the original and new styles; and (iii) develop style- exploration further expose the model a wider variety of styles by extrapolating the current styles. Based on these unique characteristics, our style-based learning handles the issue of the lack of styles in each FL. client, significantly improving generalization capability.
We also propose an allention-bused feature highlighter, which enables the model to focus only on the important/essential parts of the features when making the prediction. Our key contribution here is to utilize an attention module to capture the similarities between the features of data samples in the same class (regardless of the domain), and emphasize the important/common characteristics to better learn the data-por poor-ppor&ro FL scenarios where models are prone to overfitting to small local datasets, our new strategy provides advantages for complicated DG tasks by removing background noises that are unrelated to class prediction and focusing on the important parts.
features. Especially in
The two suggested schemes work in a complementary fashion, each providing one necessary compo- nent for federated DG: our style- learning improves domain diversity, while the attention-based feature highlighter learns domain- invariant characteristics of each class. Experiments on various FL setups using DG benchmarks confirm the advantage of StableFDG over (i) the baselines that directly apply DG methods to FL and (ii) the baselines that are specifically designed for federated DG.
2 Related Works
Federated learning. FL enables multiple clients to train a shared global model or personalized models without directly sharing each client's data with the server or other clients. FedAvg (26) is a well-known early work that sparked interest in FL in the machine learning community. Since then, various FL strategies have been proposed to handle the communication burden issue [31 9. data
heterogeneity issue 20 12]. adversarial attack issue (3 29 7) and personalization issue 3 [8] 5)
However, existing FL methods do not have generalization capabilities to predict well on an arbitrary unseen domain. In other words, most prior FL methods are not able to handle the DG problem.
Domain generalization. DG is one of the emerging fields in the Al community due to its significance in practical applications. Existing DG strategies based on domain alignment [17 24 23], meta- 4
[39 21 36 38 have shown great success in a centralized setup where the whole dataset is accessible during training. Recently, style-augmentation methods 39 21 36) including MixStyle 3D] and DSU (21 are receiving considerable attention due to their high compatibility with various tasks and model architectures. However, although existing DG solutions work well in a centralized setup, they face challenges in FL scenarios; in data-poor FL setups, prior works achieve relatively low performance due the lack of data samples domains in each client, resulting in compromised generalization capabilitics. The applications of meta-lcarning or domain alignment methods could be also limited when domain labels are not accessible in each client. Compared to these prior works focusing on a centralized DG setup, we develop style and attention based DG strategy tailored to FL. The advantages of our methodology against these baselines are confirmed via experiments in Sec. 4
learning [6
15 37) and
Federated domain generalization. Only few recent works 25 intersection of FL. and DG. Based on the training set distributed across clients, the goal of these works is to construct global model that is generalizable to unseen target domain. In (2), the authors
35 28
have focused on the

---

proposed to share the style statistics across different clients that could be utilized during local updates. However, this method does not utilize the styles beyond the clients' domains and shows limited performance in specific datasets where the data samples are not well separated in the style space. Moreover, increases the computation and memory costs for generating new images in each client using a pretrained style transfer. In our work, we handle these issues via style exploration and the attention-base based feature highlighter to train the model with novel styles while capturing the important knowledge of each class. In [25]. the authors proposed to exchange distribution information among clients based on Fourier transform, especially targeting image segmentation tasks for medical data. The authors of [35] proposed strategy for federated DG based on the Nomain-invariaan feature extractor and an ensemble of domain- -omain-specific classifiers. Two regularization losses are developed in 28 aiming to learn simple representation of data during client-side local updates.
28 improve DG performance, the authors do not directly handle the issues that arise from limited styles and data in each client. Compared to these works, we take an orthogonal approach based on style and attention based learning to effectively learn style-invariant features while capturing common knowledge of classes. Experimental results in Sec. reveals that our scheme outperforms existing ideas tackling federated DG in practical data distribution setups.
Although these prior works 25
Proposed StableFDG Algorithm
Problem formulation. We consider FL setup with N clients distributed over the network. Let S, = =(x", )e= be the local dataset of the n-th client, which consists of p. pairs of data sample and the corresponding label y. Here, each client
have data samples from either single or multiple source domains in its local dataset S,,- Previous works on FL. focus on constructing a global model that predicts well on the overall dataset S S2, SN) or personalized models that work well on individual local datasets Sn,. In contrast to the conventional FL setup, given the overall dataset (or source domains) S, the goal of this work is construct shared global model that has a generalization capability to predict well on any unseen target domain T.
Background. Let E RC*H*XX XW be the feature of a sample which is obtained specific layer of the neural network. Here, C, H. W denote the dimensions of channel, height, width, respectively. Given the feature of a specific data sample, the channel-wise mean sus) RC and the channel-wise standard deviation (s) E IRC can be written as
H p(s) HW ΣΣ Sch,, (s)c
H W ΣΣ
1 HW (schh, -c())))
(s))c)?,
(1)
respectively. These variables are known as style statistics as they contain style information of an image in CNNs [10]. Based on these style statisties, various atyle-augmentttion schemes such as MixStyle (39 or DSU (21) have been proposed in the literature targeting a centralized setup.
Overview of approach. Fig. provides an overview of the problem setup and our StableFDG algorithm. As in conventional FL, the training pro- cess consists of multiple global rounds, which we index t = 1, 1,2,... .., T. In the beginning of round t, a selected set of clients download the current global model w, from the server. Before local training begins, each client n computes its own style information t = [un, on, En (1),, En) using its local dataset according to Q which will be clarified soon. This information is sent the server, and the server shares these information with other clients to compensate for the lack of styles or domains in each client. During the local update process, each client selectively shifts the styles of the original data in the mini-batch to the new style (received from the server) via adaptive instance normalization (AdaIN) [10), to improve domain diversity (inner box in Fig. 2b). After this style sharing and shifting process, each client performs style exploration via feature-level oversam- pling to further expose the model to novel styles beyond the current source domains of each client (outer box in Fig. 2b). Finally, at the output of the feature extractor, we apply our intention-based
Ste shainn
Limoeen domain
Local updane
shiting
sxporraion (sec 51)
weighted averginng (Ses. 3.2)
Source domain
Sounce donáin
Source dorran
Overview of our problem setup and algorithm for federated domain generalization. Each client can have single source domain as described in Fig. [ or even multiple source domains in its local dataset.
Figure 1:
3

---

Style exploration
Style sharing and shifting
New styles (from another client)
client)
Origraal styles cient
Selective shift
Nowly genersled styles biteed on siyle sugmenaaion
Saye
A Samples selected vis e-mennse+
Samples shifeed to now styles
New styles genembed vie style replonstion [nvenemmpee festumes)
MisStyle
DSU
(a) Existing style-based DG baselines
(b) Proposed style-based learning for federated DG
Figure 2: Proposed style-based learning strategy (Sec. 3.1): Compared to existing style-based DG methods that rely on interpolation or style shift near the original style of each sample, our style-based learning (i) effectively utilizes other FL clients' styles based on style sharing and shifting, and (ii) enables the model to further explore a wider style space via feature-level oversampling and extrapolation, handling the issue of comain-limited FL settings.
feature highlighter to extract common/important feature information within each class and emphasize them for better generalization (Fig. B). When local updates are finished, the server aggregates the client models and proceeds the next round.
In the following, we first describe our style-based learning in Sec. 3.1 which determines how the style information is shared and utilized in each FL client, and how style exploration is conducted, to overcome the lack of styles in each client. In Sec. 3.2] we show how attention mechanism is utilized capture the essential characteristics of each class for better generalization in data-poor FL setups.
3.1 Style-based Learning for Federated DG
The main components of our style-based learning are style-sharing, selective style-shifting, and style-exploration. each having its own role to handle the style limitation problem at each client. As illustrated in Fig. this fundamental challenge of federated DG is not directly resolvable using existing aylleaugmeenttion DG methods that can only explore limited areas the style-space.
Step 1: Style information sharing, Based on the data samples in its local dataset Sn, at a specific layer of the model, each client n computes the average of channel-wise means, fmns and the variance of the channel-wise means, E,, (1), as follows:
Pn 1
Pm Σ p(s?), (n(m)=
Pro 1=1
where xe E RRCHHW is the feature of the i-th sample in the n-th client's local dataset, and the square operation (.)2 works in an element- wise manner. Similarly, the average and the variance of channel- wise standard deviations are computed as o, P.. EiT (s²)). Σ,
Din (a,, )) respectively. Here, Un and on represent the center of the style statistics of client n, while E,, (μ) and En, (a) show how these style statistics of client n are distributed around the center. Now we define
4 [um, On (n), (n)
the style information representing the domain identity of the neth client in the style-space. Each client n sends 2n to the server, and the server randomly shares the style information to other clients in one-to-one manner; client receives Φ,,' that belongs to client n' (n & n') without overlapping with other clients. By compensating for the lack styles in each FL client, this style-sharing process the first step that provides opportunity for each model to get exposed to styles (blue region in Fig. (bb) beyond the client's original styles (orange region in Fig. 2b).
Step 2: Selective style shifting. Suppose client received E., from the server. Now the question is, how should each client utilize this additional style information during training to improve domain/style diversity? Our idea is to selectively shift the styles of the samples from the original style to the

---

new style to effectively balance between the original/new source domains. To this end, given a mini-batch with size B, each client runs k-means++ with k B/2 in the style-space for one iteration and selects B/2 cluster centers. This enables each client to choose the B/2 styles that are similar to the remaining B/2 styles, so that the model can get exposed to new styles while not losing the performance on the original styles. The selected B/2 samples keep their original styles, while for the remaining B/2 samples, we shift the style of their feature 5 to the new style via AdaIN [[O]
as f(s (o + Eg En (o)( (=m(())
[(s) + (Hep' + En where f(s) is the new feature shifted
from s and Ey. -NO0, (x)),,, 6, -,,,,,,,,))) (n)) the values sampled from normal distributions. Then, the mini-batch applied with new styles in Pn is forwarded to the next layer. The inner box in Fig. 2b shows how style shifting is performed in client based on the new style information @n.-
Overall, based on the shared style statistics in Step I, our style shifting in Step 2 balances between the original source domain and the new source domain via k-means++ for better generalization, which cannot be achieved by previous methods in Fig. 2a that rely on interpolation or style-shift near the original style. In the following, we describe our style exploration that can further resolve the style-limitatin problem based on feature-level oversampling and extrapolation.
Step 3: Feature-level oversampling. Let 8 E BB be mini-hatch of features in client at specific layer, obtained after Steps and 2 above. Here, we oversample the features by the mini- batch size B in as balanced as possible up to size B. For example, suppose that the number of samples for classes a, b, are 3, 2, 1, respectively in the mini-batch. Using oversampling size of 6, we oversample by 1, 2, 3 data points for classes a, b, c, respectively, to balance the mini-batch
manner; the samples belonging to minority classes are compensated
terms of classes. Based on this, we obtain new oversampled mini-batch 8" with size B, and concatenate it with the original mini-batch as follows: 8" = [s" 3"] This not only mitigates the class-imbalanne issue in each client but also provides good platform for better style exploration; the oversampled features are utilized to explore a wider style-space beyond the current source domains, as we will describe in Step 4. The oversampling size can be adjusted depending on the clients' computation/memory constraints.
Step 4: Style exploration. In order to further expose the model to wider variety of styles, we transfer the styles of tensors in 3 (i.e., the set of oversampled features) to novel styles beyond the style of each client. Let 8 be the feature of i-th sample in the oversampled mini-batch 5 We obtain the new styles by extrapolating the original styles in h around the average of channel-wise mean ut (8") and the average of standard deviations o (5") computed on the concatenated mini-batch s" [s".8" as
Nnew (57) = (x(5)) + a. (M(37) - 14 (s")), Dnew (8.) o (3,)) (a())) (s"))))
where a is the exploration level. We perform AdaIN to shift the style of 8 to the new style statistics (new (B.') and onew (8 If a = the styles remain unchanged, and a increases, the styles are shifted farther from the center. The outer box in Fig. 2b describes the concept of our style exploration.
Step 5: Style augmentation. After style exploration, we apply existing dyyleaaugmentation methods during training. In this work, we mix the style statistics of the entire samples in 8" to generate diverse domains as in 39 as (ne = X ((87) + (1 A) (((7) and Tnew (8.)
X o())) ) + (1 A)- o(8"), ), where S and Sy arbitrary two samples in s" and À is mixing parameter sampled from the beta distribution. Below, we wish to highlight two important points.
Remark 1 (Privacy). It is already well-known that there is an inherent clustering of samples based on their domains in the style-space, regardless of their labels [39]. This indicates that label information is not contained in the style statistics, resolving privacy issues. Note that some prior works on federated DG [2 2 also adopt sharing style information between clients, but in different ways (see Sec. 2
Remark 2. By enabling the model to explore a wider region in the style space based on the exploration level a, our style exploration in Step is especially beneficial when the target domain is significantly far from the source domains (e.g., Sketch domain in PACS dataset). Existing style- basee- DG methods (e.g., MixStyle, DSU) are ineffective in this case as they can explore only some limited areas near the original styles in each client (Fig. 2a vs. Fig. 2b). which leads to performance degradation especially in data-poor FL scenarios. It turns out that our scheme has significant performance improvements
over these baselines even with a rather arbitrarily chosen a, as we will see in Sec.
5

---

3.2 Attention-based Feature Highlighter
In this subsection, we describe the second component of our solution, the attention- feature highlighter, which operates at the output of the feature extractor to tackle the remaining challenge of federated DG: limited generalization in each client due to the lack of data. To handle this issue, start from our key intuition that the images from the same class have inherently common characteristics regardless of domains to which they belong (e.g.. Fig. B) Based on this insight, we take advantage of attention find the essential characteristics of images in the same class and emphasize them, which turns out to be very effective in data-poor FL. scenarios.
Attention score. Consider the i-th data sample of client n.. Given the feature = RCHW obtained from the output of the feature extractor, we flatten to a two-dimensional tensorop X; E RC with size of (C, HW). where we omit the client index for notational simplicity. Now consider another X5 (j & i) in a mini-batch that belongs to the same class as X,, where the domains of X, and Xj can be either same different. Inspired by the attention mechanism [3]. measure the spatial similarity S " HW between X, and X, as follows:
S = Sim(X, s, Xy) = x7xx))
where 0g E BR 02 E RRdxC are the learnable parameters trained to extract important information in each class from the given samples and d is the embedding size of queries Q
0g X, and
Then, we reshape S E Hww to S, E R |/ the (m, nt-th spatial feature (E "=" of S, represents the similarity between the (m, n) -h spatial feature of the key feature X, and the overall spatial feature of the query feature Xj. Then by taking the mean of obtain the
keys K = P,X, x
Key sample i (Class: cleck)
Style shifting
Feature
Roslingg
Classlier
A
Training loss
Common//mgortnt part from same class
Midull
Avsnoging
S, along the first dimension, attention score as, which represents how the spatial feature at each location of key X, is similar to the overall features of query Xy: a, mean(S,) E _#*W A higher score indicates a higher similarity. Finally, we normalize the attention score through softmax function so that the total sum is one: a, < softmax(o.) E "
Festure
Important parts emphasized
Query sample (Class: aborm clock)
Figure 3: Proposed attention- feature high- lighter (Sec. (S2c. Our attention brccetion-basedd learning cap- tures important characteristics within each class (re- gardless of the domain) for better generalization.
Attention-based weighted averaging. Based on the attention score, we take the weighted average of E XC generate an attention feature A(z;) R in which the important parts of the key image, having common characteristics with the query image (in the same class), are emphasized:
H Σ> Σ 1=1w=
A(z).
(w))).
As shown in Fig. 3. this attention feature is concatenated with the last embedding feature (e.g.. after the global average pooling in ResNet) before the classifier, and it goes through the classifier to compute the loss during training. One important thing to note that 0, and 6₂ are trained so that the common features of query and key images become close in the embedding space while unimportant factors such as backgrounds are effectively distanced (i.e., less emphasized).
When implementing attention based weighted averaging in practice, instead of directly adopting equation (6). we modify the similarity function using the query of its own
Simmixe (X,, Xy) 0g,XX ( + 0,x,
(0 (0xx,),
(
2
to avoid performance degradation when there is little commonality between key and query images. In other words, StableFDG takes advantage of both coos-attention and self-attention, enabling the model to extract and learn important characteristics across images (via cross-attention), and within the image (via self-attention). A more detailed analysis on ( can be found in Appendix.

---

Inference. During testing, given new test sample, we compute the spatial similarity of the test sample itself as Sim(X,, X;) based on self-attention, take the weighted average, and concatenate the features to make prediction.
Remark 3. It is interesting to note that we adopt self-attention during testing. This is because Og and 0g are trained to capture the essential characteristics of each class, which enables the attention module to effectively find the important parts of individual samples via self-attention (e.g.. see heat map visualization in Fig. 3). Moreover, with only 0.44% of additional model parameters for the attention module, this strategy turns out to be very effective in learning Nomain-invariinn characteristics of classes in data-poor FL setups, handling the key challenge of federated DG.
3.3 StableFDG
Finally, we put together StableFDG. In each FL round, the clients first download the global model from the server and perform style sharing, shifting, and exploration according to [311 which are done in the early layers of CNNs where the style information is preserved. Then, at the output of the feature extractor, attention-based weighted averaging is applied according to Sec. [3.2] These two components have their own roles and work in a complementary fashion handle the challenging DG problem in FL; our style-based strategy is effective in improving the domain diversity, while our iitention-based method can directly capture the domain-invariant characteristics of each class. After the local update process, the server aggregates the client models and proceeds to the next round.
Remark 4 (Computational complexity). The computational complexity of StableFDG depends on the oversampling size in Sec. Sec.B.I and the attention module size in Sec. 3.2 which could be controlled depending the resource constraints of clients. We show in Sec. 4 that StableFDG achieves the state-of-the-ar performance with (i) minimal oversampling size and (ii) negligible cost of attention module. A more detailed discussion on the computational complexity is in Appendix.
Experimental Results
4.1 Experimental Setup
Datasets. We consider five datasets commonly adopted in DG literature: PACS 14) VLCS 6
Digits-DG [88] Office-Home [33], and DomainNet (30). PACS consists of 7 classes from 4 different domains, while VLCS contains 4 domains with 5 classes. Digits-DG composed of 4 different digit datasets, MNIST [3], MNIST-M 8) SVHN (27). SYN (8. each corresponding to a single domain. Office-Home consists of 65 classes from 4 domains, while DomainNet has 345 classes from 6 domains. The DomainNet results are reported in Appendix.
Data partitioning for FL. When evaluating the model performance, we follow the conventional leave-one- -om-annno- protocol where one domain is selected as a target and the remaining domains are utilized as sources. Compared to the centralized setup. in FL. the source domains are distributed across the clients. We consider a setup with N = 30 clients and distribute the training set into two different ways: single-domain data distribution and multi-domain data distribution scenarios. In a single-domain setup, we let each client to have training data that belong to a single source domain. Since there are three different source domains during training (except DomainNet), the training data of each domain is distributed across 10 clients uniformly at random. In multi-domain distribution setup, each client can have multiple domains, but the domain distribution within each client is heterogeneous. For each domain, we sample the heterogeneous proportion from Dirichlet distribution with dimension N 30 and parameter 0.5, and distribute the train samples of each domain to individual clients according to the sampled proportion. In Appendix, we also report the results with N = 3 following the settings of prior works in federated DG
28
Implementation. Following (39], we utilize ResNet-18 18 pretrained on ImageNet as a backbone while the results on ResNet-50 are reported in Sec. 4.3 The exploration level a is set to for all experiments regardless of datasets. For our attention module, we set the embedding size d of queries Q and keys K to 30, where Q and K matrices are extracted from the output of the last residual block using x 1 convolution such that the channel size of each output is 30. When the attention module is applied, the input dimension of the classifier becomes twice as large since we concatenate the attention feature with the last embedding feature before the classifier. The number of additional model parameters for the attention module is only 0.44% of the entire model. FL. is performed for 50 global rounds and we trained the local model for 5 epochs with a mini-bar mitc- size of 32. Among
7

---

PACS
VLCS
Methods
Art
Cartoon
Photo
Sketch
Avg.
Caltech
LabelMe
Pascal
Sun
Avg.
FedAvg
73.67
70.87
90.27
59.30
70.05
69.90
73.25
FedBN 22
78.42
70.9
90.96
54.07
73.59
94.81
$8.59
72.06
70.36
73.96
MixStyle
39
79.10
76.30
90.
76.53
60.40
72.10
69.93
74.41
DSU 21
80.43
75.70
92.60
69.87
79.65
96.13
58.77
71.80
71.87
74.64
CCST
71.35
72.40
88.65
64.10
74.13
92.50
61.20
68.20
66.50
72.
FodDG
25
71.20
71.40
90.70
59.20
73.13
$7.5
72.8
69.8
73.85
FedSR
28
76:40
71.25
93.25
60.55
75.36
92.10
60.50
70.75
71.65
73.75
(ours)
84.10
78.57
95.40
82.70
70.27
75.30
(a) PACS and VL.CS datasets.
Office-Home
Digits-DG
Methods
Art
Clipart
Product
Real
Avg.
MNIST
MNIST.M
SVHN
SYN
Avg.
FodAve XD
57.27
48.23
72.77
74.60
63.22
98.05
70.95
68.95
86.40
81.09
FedBN 22
$7.56
48.13
72.65
74.57
63.23
97.33
72.68
71.77
85.36
81.79
MixStyle
$6.05
62.95
DSU 21
58.55
52.60
71.60
73.15
63.98
98.10
75.60
70.47
85,80
82.49
CCST 2
51.3
51.75
70.2
70.3
60.89
95.10
62.80
56.60
74.90
72.35
FeiDG
57.6
48.1
72.55
74.33
63.15
97.97
72.13
71.03
87.87
82.25
FedSR 28
57.8
48.1
72.1
74.2
63.05
98.00
73.00
68.50
86. 70
81.55
StableFDG (ours)
57.57
54.30
72.33
74.97
64.79
97.23
74.53
72.95
85.85
82.64
(b) Office-Homee and Digits- DG datasets.
Table I: Main result 1 (single-domain data distribution): Each client has one source domain in its local data. The proposed StableFDG achieves the best generalization, underscoring its effectiveness.
total of N 30 clients, 10 clients participate in each global round. All reported results are averaged over three random seeds.
Where apply style-based modules. Inspired by 39
apply our style-based modules with a probability of 0.5 at specific layers. In particular, style sharing and shifting are executed at the first residual block of ResNet with probability 0.5, while the style exploration module is performed the first or second or third residual blocks independently with probability 0.5 after style sharing/shifting,
21
Baselines. (i) FL baselines: First, consider FedAvg (26] and FedBN (22], which are the basic FL baselines not specific to DG. (ii) DG baselines applied to each FL client: We apply MixStyle [39) also
during the local update process of each client and aggregate the model via FedAvg. Similarly, apply DSU [21] at each client and then perform FedAvg to compare with our work. (iii) Federated DG baselines: Among DG schemes tailored FL, implement the recently proposed CCST 2]. FedDG (25), FedSR [2 and evaluate the performance. For a fair comparison, we reproduced all the baselines in accordance with experimental setup. The augmentation level in CCST, hyperparameter that controls the amount of augmented images, is set to 3 as in the original paper (2). The hyperparameterss in FedSR, that control the regularization losses, tuned achieve the best performance in our setup. Except for these, we adopted the parameter values in the original papers.
4.2 Main Experimental Results
Single-domain data distribution. Table shows our results in single-domain data distribution setup. We have the following observations. Compared to the previous results provided in the centralized DG works 39 21). the performance of each method is generally lower. This is due the limited numbers of styles and data samples in each FL client, which restricts the generalization performance of individual client models. It can be FedAvg and FedBN that do not tackle the DG problem. The proposed StableFDG achieves the best average accuracy for all benchmark datasets, where the gain especially large in PACS having large shifts between domains. In contrast to our scheme, the prior works 22 25 28 targeting federated DG show marginal performance gains relative FedAvg our practical experimental setup with (i) more clients (which results in less data in each client) and (ii) partial client participations.
that most of the baselines perform better than
Multi-domain data distribution. In Table 2, report the results in multi multi-domain data distribution scenario. Compared to the results Table I most of the schemes achieve improved performance in Table 2 This is because each client has multiple source domains, and thus providing better platform for each client model to gain generalization ability. The proposed StableFDG still performs the best, demonstrating the effectiveness of style and attention based learning strategy for federated DG.

---

Offic-Home
VLCS
Methods
Art
Clipart
Product
Real
Avg.
||
Caltech
LabelMe
Pascal
Sun
FedAvg
57.70
75.33
63.55
61.10
65.40
FedRN
22
57.07
48.32
72.31
74.57
63.07
94.14
62.61
69.89
69.04
73.97
MixStyle
19
55.87
52.03
71.10
74.20
63.30
60.77
73.90
66.73
74.15
DSU 21
58.60
52.80
71.63
74.00
64.26
96.87
60.23
72.97
68.97
74.76
CCST
52.2
52.2
70.6
72.3
61 83
96.70
60.40
71.40
65.00
73.38
FeIDG
26
57.9
73.2
75.0
63. 68.68
60.7
72.4
67.3
74.15
FedSR
28
58.1
48.2
72.5
75.4
63.55
92.60
60,80
72.15
68.30
73.46
StablePDG (ouns)
57.87
73.10
75.00
75.60)
(a) Office-Horee Home and VL.CS datasets.
Art
Cartoon
Photo
Sketch
Avg.
FodAvg
26
74.87
74.53
95.30
63.37
77.02
FedBBN
77.00
76.83
95.45
67.01
79.07
MixStyle
80.07
77.53
96.23
67.40
80.31
DSU 21
80.53
76.30
95.37
70.93
80.78
FedDG
25
75.35
75.85
95.68
61.05
76.98
FedSR
73.83
74.83
95.53
66.03
77.56
StableFDG (ours)
83.97
79.10
96.27
75.67
83.75
(b) PACS dataset.
Table 2: Main result 2 (multi-domain data distribution): Each client has multiple source domains in its local dataset. The results consistent with the single-domain scenario in Table [
4.3 Ablation Studies and Discussions
Effect of each component. To see the ef- feet of each component of StableFDG, in Table 3 we apply our style-based learn- ing and attention-based learning one-by- one in multi-domain data distribution setup using PACS. We compare our re- sults with style-augmentation DG base-
Methods
Photo
Avg.
MixSiekk
96.23
80.31
DSU 21
80.53
76.30
95.37
70.93
80.78
SubleFDG conly style) 82.62
79.01
95.57
74,47
82.92
SubieFDG (only attemiion 79.98
78.58
95.75
71.35
81.41
Table 3: Effect of each component of StableFDG.
lines, MixStyle (39] and DSU (2I) By applying only our style-based learning, StableFDG already outperforms prior augll-augmentttoon methods. Furthermore, by adopting only one of the proposed components, our scheme performs better than all the baselines in Table Additional ablation studies using other datasets are reported in Appendix.
Effect of hyperparameters. In DG setups, it is generally impractical to tune the hyperpa- rameter using the target domain, because there is no information on the target domain during training. Hence, we used fixed exploration level a = 3 throughout all experiments without tuning. Fig. we observe how the hyperpa- rameters affect the target domain performance on PACS. In the first plot Fig. if a is too small, the performance is relatively low since model is not able to explore novel styles beyond the client's 's source domains. ar is too large, the performance could be slightly degraded because the model would explore too many redundant styles. The overall results show that StableFDG still performs better than the basclines with an arbitrarily chosen a, which is a significant ad vdnataee of scheme in the DG setup where hyperparameter tuning is challenging. The second plot of Fig. 4 shows how the oversampling size (introduced in Step 3 of Sec. 3.1 affects the DG performance. StableFDG still outperforms the baseline with mini- mal oversampling size, indicating that other components of our solution (style sharing/shifting and attention-bal based components) already strong enough. The size of oversampling depending on the clients' computation/meemory constraints, with the cost of improved generalization.
84
Average acouracy
Avergge sccuracy
82
82
80
80
StabbeFDG FDG M Mistyle
StaoleFOG Mixstyle
28
78
70
76
10
Exploration level
Oversamping
Figure 4: Effects of exploration level a (left) and oversampling size (right) StableFDG.
be determined
Performance in a centralized setup. Although our scheme is tailored to federated DG, the ideas of StableFDG can be also utilized in centralized setup. Table 4a we study the effects of our style and attention based strategies in centralized DG setting using PACS, while the other settings

---

Mathods
Art
Cartooa
Photo
Sketch
Avg.
Methods
Art
Castoon
Photoo
Sketch
Ang.
MixStyle
$1.61
78.85
96.77
72.29
82.28
MixStylc
87.51
51.12
91.48
08.39
83.03
DSU 21
78.84
79.56
95.21
79.59
83.25
DSU
21
86.48
81.22
97.21
73.99
84.73
StableFDG
78.35
Stable DG
87.70
(a) Performance in a centralized DG setup.
(b) Performance using ResNet-50.
Table 4: The applicability of StableFDG in centralized DG setup (Table 4ap and performance using a larger model (Table 4b on the PACS dataset.
the same as in the FL setup. The results demonstrate that the proposed ideas are not only specific data-poor FL scenarios but also have potentials to be utilized in centralized DG settings.
Performance with ResNet-50. In Table 4b also conduct experiments using ResNet-50 PACS dataset in the multi-domain data distribution scenario. Other settings are exactly the same as in Table 2 The results further confirm the advantage of StableFDG with larger models.
Attention score visualization. To gain intuitive understanding of the effect of our attention- -aseed learning, in Fig. M we visualize the score maps obtained via our attention module at testing. The score maps interpolated that it has the same size as the original image. A warmer color indicates higher value. It be seen that attention module highlights important parts of each class even in the presence unrelated backgrounds.
Figure 5: Visualization of attention score maps of cach input image (left: bed, right: radio).
Additional results. Other implementation details, comprehensive ablation studies for each compo- nent, discussions on complexity, and results on DomainNet dataset are in Appendix.
Conclusion
Despite the practical significance, the field of federated domain generalization still in the early stage of research. In this paper, proposed StableFDG, new training strategy tailored to this unexplored area. Our style-bused strategy enables the model to get exposed to various novel styles beyond each client's source domains, while our attention- method captures and emphasizes the important/common characteristics of each class. Extensive experimental results confirmed the advantage of our StableFDG for federated domain generalization with data-poor FL clients.
Limitations and future works. StableFDG requires 0.45 % more communication load compared to FedAvg for sharing the attention module and style statistics, which is the cost for better DG performance. Further developing our idea to tailor to centralized DG and extending our attention strategy to segmentatioon/dteettion DG tasks are also interesting directions for future research.
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF-2019R 111A2222061 135), by IITP funds from MSIT of Korea (No. 2020-0-00626), by the National Science Foundation (NSF) under grants CNS-2146171 and SPPS-2313100 239 and by the Defense Advanced Research Projects Agency (DARPA) under grant D22AP00168-00.
References
[1] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3286-3295, 2019.
[2] Junming Chen, Meirui Jiang, Qi Dou, and Qifeng Chen. Federated domain generalization for image recognition via cross-client style transfer. Proceedings of the IEFF/CVF Winter Conference Applications of Computer Vision, pages 361-370, 2023.
10

---

[3] Yuyang Deng, Mohammad Mahdi Kamani, and Mehrdad Mahdavi. Adaptive personalized federated learning, arXiv preprint wrXiv:2003.13466 2020.
[4] Yingjun Du, Jun Xu, Huan Xiong, Qiang Qiu, Xiantong Zhen, Cees GM Snoek, and Ling Shao. Learning to learn with variational information bottleneck for domain generalization. In European Conference on Computer Vision, pages 200-216. Springer, 2020.
[5] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostie meta-learning approach. Advances in Neural Information Processing Systems, 33:3557-3555 1366877588,,. 2020.
[6] Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pages 1657-1664, 2013.
[7] Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks (Byzantine-Robbl } federated learning. In 29th USENIX Security Symposium (USENIX Security 20), pages 1605-1622, 2020.
[8] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[9] Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: A communication- efficient algorithm for federated learning. In International Conference Muchine Learning, pages 3973-3983. PMLR, 2020.
[10] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference computer vision, pages 1501-1510, 2017.
[11] Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14(1-2):1-2) 2021.
[12] Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132-5143. PMLR, 2020.
[13] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-2)) 1998.
[14] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pages 5542-5550, 2017.
[15] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-leamnig for domain generalization. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
[16] Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 1446-1455, 2019.
[17] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400-5409, 2018
[18] Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning, pages 6357-6368. PMLR, 2021.
[19] Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Chal- lenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-60, 2020.
11

---

[20] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems, 2:429 450, 2020.
[21] Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Lingyu Duan. Uncertainty modeling for out-of-distribution generalization. In International Conference Learning Representations, 2022.
[22] Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non-id features via local batch normalization. In International Conference on Learning Representations.
[23] Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[24] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 624 224-639,200 2018.
[25] Quande Liu, Cheng Chen, Jing Qin, Qi Dou, and Pheng-Ann Heng. Feddg: Federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In Proceedings of the IEEF/CVE Conference on Computer Vision and Puttern Recogni- tion. pages 1013-1023, 2021.
[26] Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-eficiie learning of deep networks from decentralized data. In Artificial Intelligence and Statistics, pages 1273-1282, 2017.
[27] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[28] A Tuan Nguyen, Philip Torr, and Ser-Nam Lim. Fedsr: A simple and effective domain generalization method for federated learning. In Advances in Neural Information Processing Systems, 2022.
[29] Jungwuk Park, Dong-Jun Han, Minseok Choi, and Jackyun Moon. Sageflow: Robust federated learning against both stragglers and adversaries. Advances in Neural Information Processing Systems, 34:840-88 100-551,2220 2021.
[30] Xingchao Peng. Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference computer vision, pages 1406-1415, 2019.
[31] Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: communicatioon-eficcien federated learning method with periodic averaging and quantization. In International Conference on Artificial Intelligence and Statistics, pages 2021- - 2031. PMLR, 2020.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[33] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018-5027, 2017.
[34] Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. Advances in Neural Information Processing Systems, 33:16070-16084, 2020.
12

---

[35] Guile Wu and Shaogang Gong, Collaborative optimization and aggregation for decentralized do- main generalization and adaptation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 0484-6493 2021.
[36] Yabin Zhang, Minghan Li, Ruihuang Li, Kui Jia, and Lei Zhang. Exact feature distri- bution matching for arbitrary style transfer and domain generalization. arXiv preprint 2733v:2203.07740, 2022.
[37] Yuyang Zhao, Zhun Zhong, Fengxiang Yang, Zhiming Luo, Yaojin Lin, Shaozi Li, and Nicu Sebe. Learning to generalize unseen domains via memory-based multi-source meta-learning for person re-identification. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 6277-6286, 2021.
[38] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In European conference on computer vision, pages 561-578, Springer, 2020.
[39] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. International Conference on Learning Representations, 2021.
13

---

A Results on DomainNet Dataset
To demonstrate the effectiveness of our StableFDG larger dataset, performed experiments DomainNet dataset in a single-domain data distribution scenario. To this end, we utilize ResNet-50 pretrained ImageNet. The number of global rounds and mini-batch size set 60 and 64, respectively. The remaining settings are the same as those in our main paper. The results in Table 5 show that our StableFDG consistently outperforms not only the centralized DG works but also the prior works on federated DG even with more complex dataset.
Methods
Clipart
Inforgraph
Painting
Quickdraw
Real
Sketch
Avg.
FedAvg
26
61.52
24.75
50.83
12.08
60.00
49.92
43.18
FedBN 22
60.23
24.40
50.43
11.99
$9.57
49.66
42.71
MixStyle
61.39
24.33
51.41
13.07
$7.90
$1.40
43,25
DSU 21
62.43
24.30
51.87
13.65
58.75
52.40
43,90
FedDG 25
62.49
23.71
48.42
12.45
61.44
49.11
42,94
FedSR [28
61.91
25.37
50.54
11.59
62.03
50.13
43,60
StableFDG
62.58
24.12
52.23
14.87
60.60
52.50
44.48
Table 5; Results on DomainNet dataset in a single-domain data distribution scenario.
B Discussion on Computational and Communication Costs
T compares the communication, computation, and average accuracy of different schemes on PACS in a multi-domain data distribution scenario. ResNet- 18 is adopted as our main manuseript. We first compare the uplink communication load of each client in specific global round. Compared to FedAvg that only transmits the model each round, our scheme requires additional communication burden for transmitting the style statistics and the attention module, which are negligible. We also compare the computation time by measuring the time required for local update each client using an GTX 1080 Ti GPU. CCST ( and FedDG 225 require large computation due to the increased amounts of data samples or multiple backpropagations for meta training. Our scheme requires additional computation caused by style exploration, attention module update, etc., which are the costs for better generalization to the unseen domain.
Methods
Communication load
Computation time
Achievable average accuracy
FedAvg 26
44,98 MB
4.57 sec
77.02 %
CCST 2
44,98 MB
9.03
79.00
FedDG 25
44.98 MB
22.44
76.98
FedSR 28
44.98 MB
4.59 sec
77.56
StableFDG
45.00 MB
7.39 sec
83.75
Table 6: Computation and communication cost comparison.
C Experiments with Three Clients
for federated DG adopting the usual setting where the number of clients equals the number of source domains, in our main paper, we introduce a more practical experimental setting for federated DG where the source data is distributed to more clients than the number of source domains. For a comparison with them in the same setting, we also provide additional experimental results the setup with number of clients number of source domains. Table 7 shows the results on PACS and Office- Home datasets with three clients in single-domain data distribution scenario. is confirmed from the results that our StableFDG also achieves better performance compared to the existing works 2
Different from the prior works 22 28
28 in this simple setting.
14

---

Methods
Art
Cartoon
Photo
Sketch
Avg.
CCST 2
81.25
73.34
95.21
80.27
82.52
FedSR 28
83.20
76.00
93.80
81.90
83.70
StableFDG
83.01
79.31
94.85
79.76
84.23
(a) Results on PACS dataset.
Methods
Art
Clipart
Product
Real
Avg.
CCST 2
59.05
50.06
72.97
71.67
63.56
FedSR (2
57.93
50.45
73.33
75.51
64.31
StableFDG
57.19
57.94
72.76
72.16
65.01
(b) Results on Office-Home dataset.
Table 7: Results in single-domain data distribution scenario with three clients.
D Effect of Each Component of StableFDG
As mentioned in the main manuscript, we provide further ablation studies on the effect of each component in StableFDG using VLCS dataset. Table g shows that each component individually brings performance gain compared to FedAvg. Using both strategies achieves greater performance gains, confirming that the two proposed schemes work in complementary fashion.
Methods
Caltech
Labelme
Pascal
Sun
Avg.
FedAvg 26
93.65 1.59
61.10 2.08
72.55 - 0.90
65.40 = 0.28
73.18 + 0,27
StableFDG (only style)
98.19 + 0.81
58.93 = 1.16
75.19
+ 0.67
68.69 = 0,73
75.25 + 0.20
StableFDG (only attetnion)
94.10 + 1.22
62.02 + 1.55
72.26
+ 0.78
66.16 + 2.58
73.64 + 0.89
StableFDG (both)
98.50 + 0.17
60.07 0.79
74.40 + 1.88
69.43 + 1.11
75.61 + 0.71
Table Effect each component on VLCS dataset multi-domain data distribution scenario.
The reported results indicate (mean = 95% confidence interval) over 3 random trials.
E
Ablation Studies on Style-Based Learning
E.1
Randomness in style sharing
In our style based learning, style sharing among clients is performed random. However, one can think of the strategy where client receives the style information Φ, that has the largest distance with its own style information Φ,, in the style space. Table shows the corresponding result using PACS dataset in multi-domain data distribution setup. Interestingly. it can be seen that the random selection adopted in this paper performs better, since most of the users generally tend to receive the same style statistics when using the largest distance strategy.
Methods
Art
Cartoon
Photo
Sketch
Avg.
Random sharing (current manuscript)
83.97
1.25
79.10
0.45
96.27
0.36
75.67
0.58
83.75 + 0.23 83.04 0.30
Large distance
83.01
1.43
78.33
0.51
96.19
0.76
74.63
0.65
Table 9: Random sharing receiving the style that has the largest distance in style space. The reported results indicate (mean 95% confidence interval) over random trials.
E.2 Effect of the number of shared styles
We also compare the effect number of styles received at each client in Table TQuuing PACS dataset in multi-domai data distribution setup. The performance increases the number of received styles increases, with small additional communication load (the vector length of the style information 128, which is negligible compared to the number of model parameters, which is 1 1100
15

---

Methods
Art
Carioon
Photo
Sketch
Avg.
No style sharing Receive style (current manuscript)
83.16 +: 0.18
78.54 1 1.11
95.56 1: 0.92
74.64 +: 1.06
82.97 1: 0.36
83.97 + 1.25
79.10
+ 0.45
96.27
= 0.36
75.67 = 0,58
83.75 = 0.23
Roceive 3 styles
84.55
78.67
95.75
76.77
83.94 =
Table 10: Effect of the number of shared styles. The reported results indicate (mean + 95% confidence interval) over random trials.
E.3 Effect of k-means++ in style shifting
In Section 3.1, we utilized the k-means++ as a tool for facilitating our key idea in style shifting. which is to effectively balance between the original source domain and the new source domain for better generalization; k-means++ plays a role select the B/2 styles that are similar to the remaining B/2 styles in the mini-batch. By doing so, the model can explore new styles while not losing the performance on the original styles. To see this effect, we compare k-means++ vs. random sampling when selecting B/2 samples to be shifted, in Table [1] The results show that strategically selecting the B/2 samples be shifted achieves better performance especially in the Sketch domain (1.89% gain) that has a large style gap with other domains. We believe that these results motivate and justify our design choice.
Methods
Art
Cartoon
Photo
Sketch
Avg.
Shifting B/2 random styles
83.69
79.61
95.99
73.78
83.27
Shifting B/2 styles via +
83.97
79.10
96.27
75.67
83.75
Table 11: Effect of k-means++ in style shifting on PACS dataset.
E.4 Effeec of class-balanced oversampling
In our main paper, performed the class-balanced oversampling in the feature space to alleviate the class-imbalance issue during style exploration. To confirm the effectiveness of the class- balanced oversampling, compare with random oversampling under the same condition where only the style exploration is applied without other components. Table 12 shows the results on Office- Home dataset, where the class distribution is highly imbalanced in multi-domain data distribution scenario. can be seen that our class-balanced oversampling achieves higher performance over the simple random sampling, which validates the efficacy of mitigating the class imbalance problem in FL clients.
Methods
Art
Cartoon
Photo
Sketch
Avg.
Random oversampling 56.21
54.43
69.37
72.29
63.08
Class-balanced oversampling
57.20
53.15
71.2
74.23
64.10
Table 12: Effect of the class-balanced oversampling on Office-Home dataset in data distribution scenario.
E.5 Effect of the operation probability in style-based learning
We conduct additional ablation studies on the probability value (defined as here) utilized to control the operation of the style sharing/shifting and style exploration modules. Larger p means that our scheme is more likely to be activated. In Table 13 we provide results on various probability values in single-domain data distribution scenario using PACS dataset. From the results, it confirm that for all values, the proposed StableFDG outperforms existing baselines, demonstrating that our scheme can work well with an arbitrarily chosen probability p. In detail, when ranges from 0.3 to 0.7, the high performance is maintained while the performance decreases at both extreme probabilities (p = 0.1 and 0.9). Therefore, it recommended for practitioners to select the p in an appropriate range, avoiding extreme cases.
16

---

Methods
Art
Cartoon
Photo
Sketch
Avg.
MixStyle 39
79.10
76.30
90.10
60.63
76.53
DSU 21
80.43
75.70
92.60
69.87
79.65
StableFDG (p 0.1)
82.70
78.30
95.30
75.30
82.90
StableFDG (p = 0.3)
84.50
79.50
96.00
75.70
83.93
StableFDG (p 0.5)
83.10
79.50
96,40
76.00
83.75
StableFDG (p 0.7)
82.40
78.10
95.70
76.30
83.13
StableFDG (p 0.9)
81.40
78.80
95,90
73.60
82.70
Table 13: Effect of the probability value in our style-based learning on PACS in single-domain data distribution scenario.
E.6 Where to apply the style module
For implementation, style-based learning is applied only in the 1st, 2nd, 3rd blocks among residual blocks in ResNet-11. Note that at the output of the 4th block, label information is dominant rather than style information, which results in degraded performance when style-based schemes are applied. This is confirmed by new experiments in the table below. It can be seen from the results that if we consider the 4th residual block to apply our style-based learning, the performance gets degraded. This result confirms the intuition that style-based learning should be conducted at the carlier layers where style information is preserved.
Methods
Art
Cartoon
Photo
Sketch
Avg.
Style exploration at 1st, 2nd, 3rd layers (main manuscript)
84.10
78.57
95.40
72.73 73.35
82.70 82.25
Style exploration at 1st, 2nd, 3rd, 4th layers
82.99
78.54
94.13
Table 14: Ablation experiments applying style-based learning different layers (PACS dataset).
F
Ablation Studies on Attention-based Feature Highlighter
F1
Effect of adopting both cross-attention and self-attention
In our main paper, the similarity metric in equation (6) adopts cross-attention, while the metric in equation (8) combines cross-attention and self-attention. When applying only the cross-attention- based metric equation (6). we found that the similarity value could become low even when the two samples belong to the same class, in special cases. We handled this issue by adding the self-attention component as in equation (8). Intuitively, by doing this, the attention module is learning to extract the important features across images (via cross-attention), and within the image (via self-attention). Table 15 compares the performance of our StableFDG when using (i) self attention alone, (ii) cross-attention
alone (equation (6)), and (iii) both self and cross attentions at the same time (equation ( ), confirming the advantage of using self-attention and cross-attention together.
Methods
Clipurt
Infograph
Painting
Quickdraw
Real
Sketch
Avg.
StableEDG (with self-attention alonc)
61.77
24.88
48.28
14.15
59.78
52.41
43.55
StableFDD (with cross atlention alone)
61.24
14.67
43.83
StableFDG (with self + cross)
62.58
24.12
$2.23
14.87
60.60
52.50
44,48
Table 15: Effects of similarity metrics using DomainNet dataset in multi-domain data distribution scenario.
F.2 Comparison using the same model size
Our attention module requires 0.44% of additional model parameters to perform the attention-based learning. For fair comparison to see the effect of our attention-bad based learning, consider
17

---

different baseline with the same model size but without inrention-basee learning. Specifically, the baseline computes the attention score map using only additional convolutional operations and take the weighted average of the feature 2; based on the attention score map. Table 16 shows the results using PACS dataset in a single-domain data distribution scenario. The results demonstrate that our iiiention-based learning achieves performance improvements on all four domains while playing key role in capturing essential parts of the features.
Methods
Art
Cartoon
Photo
Sketch
Avg.
Baseline (same model size) Attention-based learning
78,88
69.03
91.74
60.77
75.11
79.54
72.48
92.51
64.71
77.31
Table 16: Ablation study on attention-base learning using PACS dataset in single-domain data distribution scenario.
F.3
Effect of attention in centralized setup
Now we provide answer to the following question: Instead of the FL. setup we focused on, can attention provide benefits the centralized DG setup? Table 17 shows the results with/without attention module in a centralized setup using PACS dataset. The results show that attention still provides performance improvements in the centralized setup by learning although the gain is slightly lower than the gain in the FL setup as shown in Table 3 of the main manuscript. These results indicate that the proposed attention-based learning indeed captures the domain-invarian characteristics of samples, while the scheme provides more benefits in the FL setup where each client is prone to overfitting due to lack of data.
-invaiatn features,
Methods
Art
Cartoon
Photo
Sketch
Avg.
StableFDG (centralized setup, without attention)
84.15
79.45
96.21 96.38
77.09
84.23 84.88
StableFDG (centralized setup, with attention)
85.02
79.65
78.45
Table 17: Effect of proposed attention-based feature highlighter in centralized DG setup using PACS dataset.
G Other Implementation Details
Our code is built upon the official code of 39 and M. During the local update process, we use SGD as an optimizer with a momentum of 0.9 and a weight decay of 5e For PACS, Office-Home and VLCS, the learning rate is set to 0.001 and the cosine annealing is used as scheduler. For Digits-DG, we set the learning rate to 0.02 and the learning rate is decayed by 0.1 every 20 steps. For our attention-based feature highlighter, at least two samples are required be in the mini-batch of every client. When this condition is not met, additional samples are extracted from the corresponding client's local dataset facilitate cross-attention.
More detailed description on oversampling: Let 8" E BB be a mini-batch of features in client specific layer, obtained after Steps 1 and 2 in the main manuscript. Now given fixed oversampling size, we oversample the features in the mini-batch to obtain g", so that the concatenated mini-batch g [s" 3"] becomes class-balanced as much as possible. Consider a toy example where the number of samples for classes a, 6, are 3, 2, 1, respectively in the mini-batch s" In this example, if the oversampling size is 3, we randomly choose one data point from class and two data points from class e (in this case, the same data point is selected for two times with duplication) to obtain s". so that the concatenated mini-batch 8* [ becomes is 1, we oversample one data point in class c to make the concatenated mini-batch to be balanced as much as possible. the oversampling size 6, we oversample 1, 3 samples from classes a, b, e, respectively to construct g". The concatenated mini-batch g = [s": g"] is utilized for style-based learning and updating the model. This process not only mitigates the class-imbalance issue in each client but also provides good platform for style exploration by oversampling the features. In
If the oversampling size
18

---

work, we reported the results with oversampling size of B (which is equal to the mini- size), while the effect of the oversampling size also reported Fig. the main manuscript: A larger oversampling size leads to a better performance, and more importantly, our StableFDG outperforms the baseline even without any oversampling.
H Visualization of Attention Score Maps
Finally in Fig. visualize the attention score maps of different imput images during testing. The results confirm the effectiveness of our attention- based feature highlighter to focus on the important parts of each image from the unseen domain.
Figure 6: Visualization of attention score maps of each input image.
I Pseudo Code Algorithm of Stable FDG
Algorithm summarizes the overall process of our StableFDGG
Algorithm StableFDG
Input: Initialized model Wo. Output: Global model Wr
1: for each global round / = 1.... do
2:
3: Stage 1. Model download and style sharing
4: The server samples set of participating clients M, and sends the global model w, to the clients in Mr
5:
for each device n E M, do
6: Compute style information D, according the Step in Sec. 3.1
7: Transmit 4,, to the server
8: end for
9: The server shares { he a to the clients in M, according to the Step in Sec, 3.1
10:
11: Stage 2. Local updates and model aggregation
12:
for each device M, do
13:
for local epoch 1, 2..., E do
(i) Style shifting according the Step in Sec. 3.1,
15:
(ii) Style exploration according to the Step 3 and 4 in Sec. 3.1 (iii) Attention based weighted averaging according to Sec. 3.2
17:
Loss computation and model update
end for
19: The server updates the global model by aggregating the client models
end for
21:
end for
19


Reviews:
[{"review_id": "6Gy1m11tWj", "reviewer": "Reviewer_i1eF", "summary": "This paper presents a novel domain generation method in the FL framework. The authors make two key contributions to enhance domain generalization in FL. Firstly, they introduce style-based learning, which empowers individual clients to explore new styles by leveraging both their own local statistics and those of other clients. This approach promotes domain diversity. Secondly, the authors propose an attention-based feature highlighter that captures similarities among the features of data samples within the same class. This component enables better learning of domain-invariant characteristics. To validate the efficacy of their proposed method, extensive experiments were conducted. The results demonstrate the superiority of the approach compared to existing methods on various benchmark datasets.", "questions": "1) In this paper, statistics are recalculated at each round. Instead, could the running mean and variance of Batch Normalization (BN) be used as a substitute? Also, I am curious about the relationship between style augmentation and BN.\n2) The attention-based feature highlighter combines self-attention and cross-attention for feature refinement. However, it remains unclear why this specific method is tailored to Federated Domain Generalization (DG). Further explanation is needed to clarify this aspect.\n3) During training, both self-attention and cross-attention are used in Eq. (7), but during testing, only self-attention is employed. It would be beneficial to confirm if using self-attention alone during training is not sufficient.", "limitations": "Due to page constraints, some answers to certain questions were provided in the supplementary material, particularly in Supplementary Appendix E. It would be beneficial to include relevant content from the supplementary material into the main body of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses an important practical problem within the FL setup. It provides comprehensive coverage of related works and is well-written for easy understanding. The extensive experiments conducted effectively demonstrate the effectiveness of the proposed method. The supplementary material is valuable.", "weaknesses": "The adaptation of mixing styles and exploration to FL frameworks is straightforward. Also, the attention-based feature highlighter can be viewed as a variation of cross-attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the time and efforts, and providing helpful comments. Our responses are given below.\n### **Style-based learning**\nWe would like to first clearly differentiate the concept of existing style-augmentation [ICLR'21], [ICLR'22] and our style-based learning, and show that the proposed approach is based on key innovations that have not been considered in prior works, making the solution not straightforward. As shown in Fig. 2a of our paper, prior style-augmentation DG methods are constrained to generate new styles only near the original source domains, which leads to performance degradation especially in data-poor FL setups.\n\nWe overcome these limitations by taking a different angle of attack: we intentionally and strategically expose the model to a wider style-space to effectively tackle DG in data-poor FL setups (as shown in Fig. 2b in the paper). The key advantages are: (i) the generated styles become more diverse beyond the styles each client has, and (ii) the generated styles can have *better chances to cover more diverse target domains* during training. We achieve this by developing our key components: (i) feature-level oversampling (that generates samples that will explore new styles) and (ii) parameter $\\alpha$ that controls the exploration level. This is a new perspective in DG/FL proposed by our work to tackle the limitations of prior works, and has not been considered in the literature. Extensive experiments in 5 benchmark datasets also confirm the advantage of our approach.\n### **Cross attention in StableFDG**\nCompared to existing works adopting attention, we would like to maintain that our paper provides a new direction by showing that *attention can indeed capture the domain-invariant characteristics* to tackle DG, in data-poor FL setups. We had to tackle several research questions to develop an attention strategy tailored to DG: First, how should we utilize attention to effectively capture important characteristics within each class, regardless of domains? We came up with the solution of strategically mixing cross and self attention, which enables the model to learn important characteristics across images (via cross-attention), and within the image (via self-attention). The next challenge was on how to design an inference strategy. Cross-attention was not applicable during inference because it is challenging to choose two samples from the same class for cross-attention without label information. Hence, we adopted self-attention during inference, which still provides StableFDG with significant performance advantages because our trained attention module is already able to capture key characteristics within the image.\n\nMoreover, as shown in our new experiments in **Table R1** above, adopting self attention solely during training is not the best option. This indicates that *different samples in the same class indeed share some common characteristics regardless of domains*, confirming the advantage of using both cross and self attention for training. \n\nOverall, we feel that (i) the idea of capturing domain-invariant characteristics in DG via attention, (ii) our training strategy (mixed cross, self-attention) and inference strategy (self-attention), and (iii) superior performance validated via experiments deserve merits. We would be happy to address any remaining concerns the reviewer might have.\n\n### **Comparison with BN**\nWe appreciate the comment. It has been revealed in recent studies [ICCV’17], [ICLR'21] that per-instance feature statistics (as in Instance Normalization [CVPR'17]) can represent style information of an image. Style augmentation schemes utilize these per-instance feature statistics to generate new styles to tackle DG. On the other hand, while the batch axis is also considered, per-batch feature statistics are obtained in BN. After normalizing the features using the comprehensive feature statistics, the BN layer shifts the features of all samples (instances) within a batch to the same distribution (according to learnable $\\gamma$ and $\\beta$). However, when the data distributions at training and testing are different as in DG setups, the performance of BN is degraded due to mismatch in feature statistics. Overall, BN is not able to fully capture the style of each instance and is susceptible to domain shifts. Our new experiments in **Table R2** above demonstrate this.\n\n### **Why our attention is tailored to FL+DG**\nThanks for this comment. In FL, each device typically has a limited number of samples, and thus models are prone to overfitting to each local dataset during local updates. Our attention strategy effectively learns domain-invariant features in such data-poor FL setups while mitigating the risk of overfitting to irrelevant feature information unrelated to each class. This is achieved by extracting common characteristics of samples in the same class and emphasizing them, while removing unimportant parts (e.g. background noises) that cause overfitting. Table 3 in the paper shows that applying our attention scheme alone already outperforms baselines.\n\nThe follow-up question that the reviewer might be interested in is: Instead of the FL setup we focused on, can attention provide benefits in the centralized DG setup? Our new experiments in **Table R3** indicate that the proposed attention-based learning indeed captures the domain-invariant characteristics of samples, while the scheme provides more benefits in the FL setup where each client is prone to overfitting due to lack of data.\n \nAgain, we appreciate the reviewer for the helpful comments. Your comments are clear and to the point, and we tried to clarify all the issues you raised: how/why our approach has new components, discussions on BN with new experiments, why attention is especially beneficial in FL, discussions on cross and self-attention with new experiments. In case there are remaining questions/concerns, we hope to be able to have an opportunity to further answer them."}, {"review_id": "Piey0jD6Ss", "reviewer": "Reviewer_fAoj", "summary": "In this paper, the authors present StableFDG, where a style-based strategy enables the model to reach a variety of new styles beyond each client's source domain, and an attention-based approach captures and emphasizes the important/common features of each category.", "questions": "See the weakness.", "limitations": "The authors could have elaborated more fully on their new experimental setup.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The StableFDG method proposed in the article has achieved good experimental results.\nThe experimental details in the article are particularly well articulated and richly illustrated.\nThe essay writing is well-organized and readable.", "weaknesses": "The idea of enabling the model to reach a variety of new styles beyond the source domain of each client has been proposed by other method [1], and what the authors have done is more of an incremental improvement.\nIn addition, the authors in a related work evaluate [1] to improve the computational and memory costs, but it itself takes up more memory than [1].\n [1] Junming Chen, Meirui Jiang, Qi Dou, and Qifeng Chen. Federated domain generalization for image recognition via cross-client style transfer. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 361–370, 2023.\n\nThe method proposed in section 3.1 of the article is more like a combination of the methods mentioned in the authors' text.\n\nIn the \"Where to apply style-based modules\" subsection, the authors directly follow the way other methods apply modules, where more detailed ablation experiments could be done or the benefits or reasons for such application could be explained in the article.\n\nThe authors propose a new experimental setup, are the other methods in the paper based on this experimental setup, and what are the advantages of such a new setup over the previous ones?\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s constructive suggestions. We make all of the raised points clearer below, with additional experiments as needed.\n### **Differentiation from [1] and novelty of Sec. 3.1**\nOur style-based learning methodology in Sec 3.1 has three key components: (i) style sharing, (ii) selective style shifting, (iii) style exploration. [1] only shares some common idea with our first component (i.e., style sharing) in that each client shares some style information with others. Even in this component, the approaches employed in the two methods are widely different. We elaborate on three key points of differentiation from [1], and the novelty of Sec. 3.1 more generally, below.\n\n**(1) Style observation limitations in [1]:** It is important to clarify that [1] does not allow the model to observe *the styles that do not exist in the concatenated datasets of all FL clients*. To see this, consider a toy example with 2 clients, where the first client has style-information of $\\mu_1$ and the second client has $\\mu_2$ in its local dataset. [1] enables both clients to observe $\\mu_1$ and $\\mu_2$ (by sharing styles), but does not enable them to observe other styles beyond $\\mu_1$ and $\\mu_2$, limiting the style variability during training. StableFDG tackles this issue via style exploration combined with selective style shifting, which are the new perspectives in DG proposed by our work. As shown in Fig. 2b of our paper, style exploration exposes the model to various *novel styles that do not exist in the overall original dataset of the system*. This provides StableFDG with the following two key innovations: (a) by exposing the model to various novel styles beyond $\\mu_1$ and $\\mu_2$, the domain diversity of the source domains increases; (b) the model has better chances to cover diverse target domains during training since it can *explore a wider region in the style-space* (e.g., $\\mu= \\mu_1+\\alpha(\\mu_1-\\mu_2$) via extrapolation). This leads to a significant performance improvement obtained by StableFDG (e.g. 8.57% gain on PACS in Table 1 of the manuscript).\n\n**(2) Computation/storage requirements:** [1] incurs more computation/storage compared to StableFDG because [1] directly generates *new data samples* in each client. On the other hand, our style sharing requires a much lower cost as it does not require generating the image itself. *Note that in Table 2 of the supplementary material, the first column indicates communication cost, not storage.* This very small 0.45\\% additional communication burden is the fair cost for achieving significantly improved performance (4.75% gain over [1]) with reduced computation time.\n\n**(3) Attention-based feature highlighter:** Beyond our style-based learning in Sec. 3.1, the proposed attention-based feature highlighter (in Sec. 3.2) is another central contribution of our work. This handles the fundamental issues faced by DG in data-poor FL scenarios, by capturing the important/common characteristics of the samples in the same class. We stress that this approach has not been considered before, and we are one of the first to show that cross-attention can be strategically utilized to tackle the DG problem.\n### **Where to apply the style module**\n**(1) Applied in layers where the style information is preserved:** Style-based learning is applied only in the 1st, 2nd, 3rd blocks among 4 residual blocks in ResNet-18. Note that at the output of the 4th block, label information is dominant rather than style information, which results in degraded performance when style-based schemes are applied. This is confirmed by our new experiments in **Table R4** above.\n\n**(2) Applied probabilistically:** Given the 1st, 2nd, 3rd block layers, style-based learning is applied probabilistically at one of the layers: It could be operated at the 1st, 2nd, 3rd, or none of the layers. This allows the model to see various combinations of style shift/exploration compared to the case where it is always applied at a fixed layer or none of the layers. This is confirmed by our new experiments in **Table R5** above.\n### **New experimental setup**\nWe first highlight that **experimental results in the conventional setup have been given in Section C of Supplementary Material**, to provide a fair comparison with the baselines [1], [27] in their settings. The results confirm the advantage of StableFDG in the same setting as prior works.\n\nPrior works [1], [27] considered a setup with N=3 clients, assuming that a single source domain is allocated to each client. We will represent this setup as (N=3, single). In our manuscript, we considered a setup with N=30, and considered both single-domain and multi-domain data distribution setups, e.g., (N=30, single) and (N=30, multi), which come from the following two motivations: (i) The original setup with only 3 clients does not well-reflect the practical FL scenario with a large number clients. This motivated us to consider more clients (N=30). (ii) The original setup does not consider each client having multiple domains in its local dataset. This motivated us to additionally consider the multi-domain setup.\n\nOverall, given the results with the original setting (in supplementary material), we believe that providing results with the new setup should be seen not as a weakness but as a strength of our presentation, because the results (i) further strengthen the validity of our method and (ii) reflect practical scenarios that are tailored to large-scale, heterogeneous FL settings.\n\nAgain, thank you for your time and efforts. Your raised concerns made us think carefully, and we feel we have managed to clarify all the issues raised: we clarified the contributions of Sec. 3.1 compared to [1], provided intuitions/ablations on where to apply of style modules, and clarified our experimental setup. We hope that these are sufficient grounds for you to reconsider your rating. We would appreciate further opportunities to answer any remaining concerns you might have."}, {"review_id": "wvdDpYmGXm", "reviewer": "Reviewer_wPUB", "summary": "The authors observe that existing domain generalization (DG) algorithms face the issue in the setting of federated learning (FL) because of the lack of samples in each client's local datasets/domains. They propose StableFDG, which utilizes both style-based learning which allows the clients to explore styles different from that of the original source domain, and an attention-based feature highlighter, which captures the similarities and common characteristics of features of samples of the same class. The authors conduct their evaluation on datasets such as Office-Home and Digits-DG.", "questions": "+ Have you considered style transfer approaches to boost your current style-based learning? Style transfer approaches also strive to extract the domain representation (style); would this field help or hinder the performance of StableFDG?", "limitations": "- The authors argue that, because the label information is not included in the style parameters, the proposed approach doesn't have privacy issues. However, style statistics/parameters, derived from the domains, still contain domain knowledge. It's not clear and not elaborated why the privacy issue is resolved simply because the labels aren't shared.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+ The evaluation is thorough, considering most of the popular benchmarks including the more challenging ones such as DomainNet.\n+ The visuals are helpful and illustrative for readers to understand the proposed method", "weaknesses": "- It would seem that the work could greatly benefit from works in the field of style transfer, which also involves extracting domain knowledge. It would be better for the authors to remark on the connections between their style sharing and exploration with style transfer.\n\n- It is not clearly explained what would happen after the style information is obtained by Equation 3. It's evident that this piece of information is used for style-shifting, etc., but how does such an operation relate to the intuition/big picture behind the approach?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the positive comments and feedback. We are also thankful for the suggestions to  discuss  connections  between our style-based learning and existing style transfer. Our responses to the comments raised by the reviewer are given below.\n\n \n### **Intuition behind equation 3**\n\nIn a high-level, the style information (in equation 3) of other clients provide an opportunity for each client to train the model with the style information that is not in its local dataset. Fig. 2b in our manuscript provides insights into this. Without other clients' style information, a specific client will have only the styles within the orange region of Fig. 2b, which represents the styles in the client's local dataset. However, by utilizing other client's style information in equation 3 (blue region in Fig. 2b), the client is now able to expand the region in the style space to the blue region, by conducting style shifting to the blue area. As a result, the client's model is able to explore a wider style space when conducting style exploration, which leads to a better generalization. More detailed descriptions on how the parameters in equation 3 are utilized, are given in Lines 171-174 of our main manuscript.\n\n \n### **Connections with style transfer literature**\n\nRelated to the style transfer literature, our style-based learning can be  actually viewed as  a scheme that conducts  style transfer to the new styles (e.g., to the red points in Fig. 2b) based on the style statistics $\\mu$ and $\\sigma$, via Adaptive Instance Normalization (AdaIN) [ICCV'17].  Compared to  the style transfer methods that require a style transfer model for generating new images, AdaIN can conduct style transfer at feature-level during forward propagation in real-time, inspired by the instance normalization. Our style-based learning is built upon AdaIN as it  is especially beneficial  in FL settings as clients can perform style transfer without requiring an additional style-transfer model; the clients can directly conduct style transfer at feature-level during forward propagation (without generating images itself), using the style statistics via instance normalization.  \n\nHowever, as can be seen from prior style transfer works [ICCV'21],  AdaIN may not achieve the best style transfer quality compared to  the style transfer methods that use an additional model, which is the cost for achieving lower computation and lower latency. Hence, we believe that if the FL clients can have large computational powers and storage spaces to conduct style transfer  using an additional model (and thus can possibly improve the quality of style transfer), and if a well-trained style transfer network is given, the achievable accuracy   could get improved with the cost of increased computation and delay. However, how our style-based learning should be designed/modified tailored to this new strategy, is currently open.\n\nFinally, we would like to note that the authors of [WACV'23] propose to directly generate new images at clients using a style-transfer network. However, as can be seen in Section B of our Supplementary Material, this scheme requires more computation time for processing more images, and more importantly, it achieves lower performance as the scheme is not able to explore enough styles (please refer to Table 1 and 2 in our main manuscript comparing CCST vs. StableFDG). Our style-based learning handles these issues by conducting style transfer via AdaIN instead of using an additional style-transfer network, and by exploring a wider style space via proposed style exploration.\n\n[ICCV'17]  Arbitrary style transfer in real-time with adaptive instance normalization\n\n[ICCV'21]  Domain-Aware Universal Style Transfer \n\n[WACV'23] Federated domain generalization for image recognition via cross-client style transfer \n\n \n### **Domain knowledge and privacy**\n\nWe appreciate the reviewer for pointing this out, which is correct: Although the  style parameters do not contain label information, they contain domain knowledge. Our original  intention was that   label information could be critical for privacy leakage as they directly   provide the third party to know  the classes of their images (e.g., types of diseases, types of objects in the image), while the   domain knowledge causes less privacy issue (e.g., weather information of the image, style information  of the image such as sketch, photo). However, we agree that there could be some applications where these domain information can incur privacy issue. In such applications, StableFDG  can be  conducted without  style sharing and shifting; each client directly performs style exploration using its local dataset, while skipping style sharing and shifting processes. In the table below, we show the performance of our StableFDG with and without style sharing/shifting, and compare them with the baselines: Although  the performance of StableFDG is slightly degraded when style sharing/shifting are not conducted,  it still outperforms all the baselines, demonstrating its advantage in applications where domain knowledge contain private information.\n\n* StableFDG with and without style sharing on PACS:\n|Methods             | Art | Cartoon | Photo | Sketch | Avg |\n| :---- | :--: | :----: |:---: |:---: |:--: |\n| MixStyle  |    80.07 | \t77.53 | \t96.23 | \t67.40 | \t80.31\n| DSU    |   80.53 | \t76.30 | \t95.37 | \t70.93 | \t80.78\n| StableFDG (w/o style sharing)    |  83.53  |  \t78.53  |  \t96.30\t  |  75.27\t  |  83.41\n| StableFDG (with style sharing)   | 83.97\t| 79.10| \t96.27| \t75.67| \t83.75\n\nOverall, we agree with the reviewer’s comments and made efforts to discuss them in our response. Specifically, we tried to connect our work with other style transfer works, and provided discussions on how StableFDG should be modified domain knowledge contain private information. We hope to know if you are satisfied with our response. We would be happy to answer any remaining concerns you might have."}, {"review_id": "T1m7JC3YcF", "reviewer": "Reviewer_hWZZ", "summary": "This paper proposes a style and attention based learning strategy for federated domain generalization which consists of two stages. The first stage shares style information among clients and then explorates and augments styles, and the second stage highlights the essential characteristics of images with attention mechanism. Experiments  show that the proposed method achieves a new state of the art.", "questions": "- Why can eq.7. avoid performance degradation when there is little commonality between key and query images ?\n- How is oversampling implemented. What if the dataset itself is not class balanced ?", "limitations": "The paper discusses its limitation in sec. 5.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The style-based learning strategy is novel and intuitive.\n- The paper is well written and easy to follow\n- The proposed method is technically sound with superior results on the 5 datasets.", "weaknesses": "The detail of oversampling is not fully covered in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive comments and  feedback. In particular, we  appreciate the reviewer’s acknowledgment on the novelty of our work.  Below, we reply to the comments raised by the reviewer.    \n \n### **[Question 1] Insights into equation (7)**\n\nWe appreciate this comment. The similarity metric in equation 5 adopts cross-attention, while  the metric in equation 7  combines cross-attention and self-attention.  When applying only the cross-attention-based metric in equation 5, we found that  the similarity value could become low even when the two samples belong to the same class,   in special cases. We handled this issue by adding the self-attention component as in equation 7. Intuitively, by doing this, the attention module is   learning to extract the important features across images (via cross-attention), and within the image (via self-attention). The table below compares the performance of our StableFDG  when using equations 5 and 7 for similarity computations, confirming the advantage of using  self-attention and cross-attention together. In accordance with the reviewer's comment, we will make this clearer in the revised  manuscript.\n\n* Effect of similarity metrics:\n|Methods             | Art | Cartoon | Photo | Sketch | Avg |\n| :---------------- | :------: | :------: |:------: |:------: |:------: |\n| StableFDG (similarity metric in equation 5)       |   83.09 |  78.60 |  95.87 |  74.71 |  83.07 \n| StableFDG (similarity metric in equation 7)        |   **83.97** |   **79.10**  |  **96.27** |  **75.67** |  **83.75** \n\n \n### **[Question 2] How oversampling is implemented & the case with imbalanced dataset**\n\n**How feature-level oversampling is implemented:** Let $s^n\\in \\mathbb{R}^{B \\times C \\times H \\times W}$ be a mini-batch of features in client $n$ at a specific layer,  obtained after   Steps 1 and 2 in the main manuscript.     Now given a fixed oversampling size, we oversample the features in the mini-batch to obtain  $\\tilde{s}^n$, so that the concatenated mini-batch $\\hat{s}^n =  [s^n, \\tilde{s}^n ]$ becomes class-balanced as much as possible. Consider a toy example where the number of samples for classes  $a$, $b$, $c$ are 3, 2, 1, respectively in the mini-batch $s^n$. In this example, if the oversampling size is 3, we randomly choose   one data point from class $b$\n and two data points from class  $c$ (in this case, the same data point is selected for two times with duplication) to obtain $ \\tilde{s}^n$, so that the concatenated mini-batch $\\hat{s}^n =  [s^n, \\tilde{s}^n ]$ becomes class-balanced. If the oversampling size is 1, we oversample one  data point in class $c$ to make the concatenated mini-batch to be balanced as much as possible. If the oversampling size is 6, we oversample 1, 2, 3 samples from classes  $a$, $b$, $c$, respectively to construct $\\tilde{s}^n$.  The concatenated mini-batch $\\hat{s}^n =  [s^n, \\tilde{s}^n ]$ is utilized for    style-based learning and updating the model.  This  process not only mitigates the class-imbalance issue in each client but also provides a good platform for  style exploration by oversampling the  features. In our work, we reported the results with oversampling size of $B$ (which is  equal to the mini-batch size), while  the effect of the oversampling size is also reported in Fig. 4 of the main manuscript: A larger oversampling size leads to a better performance, and more importantly, our StableFDG outperforms the baseline even without any oversampling.\n\n\n**Class-imbalanced dataset:** The main reason of conducting feature-level oversampling is to generate new style statistics for our style exploration process. At the same time, as described in our answer to the previous question above, our feature-level oversampling mitigates the class-imbalance issue in each client, for class-imbalanced datasets.  \n The table below  shows the results on Office-Home dataset, where the class distribution is highly imbalanced in a multi-domain data distribution scenario (this result was already given in Section E.4 of the supplementary material). It can be seen that our class-balanced oversampling achieves higher performance over the simple random   oversampling, which validates the efficacy of mitigating the class imbalance problem in FL clients.\n\n* Effect of feature-level oversampling:\n|Methods             | Art | Cartoon | Photo | Sketch | Avg |\n| :---------------- | :------: | :------: |:------: |:------: |:------: |\n| Random oversampling    |    56.21| 54.43| 69.37| 72.29 | 63.08\n| Class-balanced oversampling   |   57.20 |  53.15 | 71.2 | 74.23 | **64.10**\n\n\n\n\nAgain, thank you for your  time and efforts for reviewing our paper, and providing insightful comments. If there are any more clarifications we could provide, we would be grateful if you could let us know."}]

Author rebuttal:
We appreciate all reviewers for providing constructive comments, which have greatly helped us to improve the paper. 

Due to the limited content we can provide in each response, we would like to share additional experimental results that **Reviewer i1eF** and **Reviewer fAoj** suggested here. For the other reviewers (Reviewer wPUB \& Reviewer hWZZ), all results are provided in our response corresponding to each reviewer.

&nbsp;


### Tables \& References for **Reviewer i1eF**

* **Table R1:** Using cross + self attention together vs. using self attention alone (DomainNet dataset)
>| Methods    |  Clipart |  Inforgraph |  Painting |  Quickdraw |  Real | Sketch |  Avg| 
| :---- | :--: | :----: |:---: |:---: |:---: |:---: |:--: |
| StableFDG (with self attention alone)  |  61.77 | **24.88** |  48.28 |  14.15 |  59.78  |  52.41 |  43.55
| StableFDG (with self + cross attention)  |  **62.58**|  24.12 | **52.23**|  **14.87** |  **60.60** |  **52.50** |**44.48**
>
>$\rightarrow$ It can be seen that considering cross attention together improves  the performance compared to the scheme that uses self-attention alone. This indicates that *different samples in the same class indeed share some common characteristics regardless of the domain*,  confirming  the advantage of adopting  cross-attention  to   learn domain-invariant features.  By taking advantage of both cross and self attention, StableFDG enables the model to extract and learn important characteristics across images (via cross-attention), and within the image (via self-attention). 

* **Table R2:** Effects of sharing style-information vs. BN features in StableFDG (PACS dataset)
>|Methods             | Art | Cartoon | Photo | Sketch | Avg |
| :---- | :--: | :----: |:---: |:---: |:--: |
| StableFDG (sharing feature statistics of BN)  |  82.84|  78.16|  94.70|   72.43|  82.03
| StableFDG (sharing style-information $\Phi_n$)  |  **84.10**|  **78.57**|  **95.40**|  **72.73**|  **82.70**
>
>$\rightarrow$ The performance drops for *all domains* when using feature statistics of BN instead of the style information $\Phi_n$. Here, the BN feature statistics are only used during feature sharing for the baseline, while other parts remain the same for a fair comparison.

* **Table R3:** Effect of proposed attention-based feature highlighter in a centralized DG setup (PACS dataset)
>|Methods             | Art | Cartoon | Photo | Sketch | Avg |
| :---- | :--: | :----: |:---: |:---: |:--: |
| StableFDG (centralized setup, without attention)  | 84.15| 79.45| 96.21| 77.09|84.23
| StableFDG (centralized setup, with attention)  |  **85.02**|  **79.65**|   **96.38**|  **78.45**|  **84.88**
>
>$\rightarrow$ The results show that  attention still provides performance improvements in the centralized setup by learning domain-invariant features, although the gain is slightly lower than the gain in the FL setup as shown in Table 3 of the main manuscript. These results indicate that the proposed attention-based learning indeed captures the domain-invariant characteristics of samples, while the scheme provides more benefits in the FL setup where each client is prone to overfitting due to lack of data.


[ICLR'21] Zhou et al., ``Domain generalization with mixstyle,'' ICLR 2021.

[ICRL'22]  Li et al., ``Uncertainty modeling for out-of-distribution generalization,'' ICLR 2022.


[ICCV’17] Huang et al., “Arbitrary style transfer in real-time with adaptive instance normalization,” ICCV 2017.

[CVPR'17] Ulyanov et al., ``Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis,'' CVPR 2017.

&nbsp;

### Tables \& References for **Reviewer fAoj**

* **Table R4:** Ablation experiments on applying style-based learning at different layers (PACS dataset)
>|Methods             | Art | Cartoon | Photo | Sketch | Avg |
| :---- | :--: | :----: |:---: |:---: |:--: |
| Style exploration at 1st,  2nd,  3rd layers (main manuscript) |  84.10	  |  78.57	  |  95.40  |  72.73  | 82.70
| Style exploration at 1st, 2nd,  3rd,  4th layers    | 82.99|	78.54|	94.13	| 73.35 |	82.25
>
>$\rightarrow$ If we consider the 4th residual block to apply our style-based learning, the performance gets degraded. This result confirms the insight that style-based learning should be conducted at the earlier layers where style information is preserved.

* **Table R5:** Ablation experiments on applying style-based learning probabilistically (PACS dataset)
>|Methods             | Art | Cartoon | Photo | Sketch | Avg |
| :---- | :--: | :----: |:---: |:---: |:--: |
| Style exploration (none of the layers) |  79.30	 |  76.24	 |  92.87	 |  68.48 |79.22
| Style exploration (fixed first layer)   |  74.56| 	73.72| 	90.60	|   62.75   | 	75.41
|Style exploration (probabilistic, main manuscript)   | **84.10**	|**78.57**	|**95.40**|**72.73**|	**82.70**
>
>$\rightarrow$ It can be seen that applying style exploration probabilistically achieves the best performance. This confirms the intuition that the probabilistic approach will let the model to  see various combinations of style shift and exploration (including the original style of the image), compared to the case where style exploration is always applied at a fixed layer or none of the layers. 

  [1] Chen et al., ``Federated domain generalization  for image recognition via cross-client style transfer,'' WACV 2023.

  [27]  Nguyen et al., ``Fedsr: A simple and effective domain generalization method for federated learning,'' NeurIPS 2022.

Meta-review: Four experts reviewed the paper, and all recommended Borderline/Weak Accept. Reviewer hWZZ was especially excited about the work for its technical novelty and superior results. The other reviewers were not impressed with the paper's technical novelty, but they liked the experiments and found the rebuttal convincing. Hence, the decision is to recommend the paper for acceptance. The authors are encouraged to incorporate the reviewers' suggestions and the rebuttal into the revision.
Decision: Accept (poster)