FedFA:  Federated Feature Augmentation
Authors: Tianfei Zhou
Ender Konukoglu
Venue: ICLR 2023 poster (2023)
Keywords: federated learning
feature augmentation

Abstract: Federated learning is a distributed paradigm that allows multiple parties to collaboratively train deep models without exchanging the raw data. However, the data distribution among clients is naturally non-i.i.d., which leads to severe degradation of the learnt model. The primary goal of this paper is to develop a robust federated learning algorithm to address feature shift in clients’ samples, which can be caused by various factors, e.g., acquisition differences in medical imaging. To reach this goal, we propose FedFA to tackle federated learning from a dis- tinct perspective of federated feature augmentation. FedFA is based on a major insight that each client’s data distribution can be characterized by statistics (i.e., mean and standard deviation) of latent features; and it is likely to manipulate these local statistics globally, i.e., based on information in the entire federation, to let clients have a better sense of the underlying distribution and therefore alleviate local data bias. Based on this insight, we propose to augment each local feature statistic probabilistically based on a normal distribution, whose mean is the original statistic and variance quantifies the augmentation scope. Key to our approach is the determination of a meaningful Gaussian variance, which is accomplished by taking into account not only biased data of each individual client, but also underlying feature statistics characterized by all participating clients. We offer both theoretical and empirical justifications to verify the effectiveness of FedFA. Our code is available at https://github.com/tfzhou/FedFA.
	

Published as a conference paper at ICLR 2023
FEDFA: FEDERATED FEATURE AUGMENTATION
Tianfei Zhou & Ender Konukoglu Computer Vision Lab, ETH Zurich
ABSTRACT
Federated learning is distributed paradigm that allows multiple parties to collab- oratively train deep models without exchanging the raw data. However, the data distribution among clients is naturally non-i..d.. which leads to severe degrada- tion of the learnt model. The primary goal of this paper is to develop a robust federated learning algorithm to address feature shift clients' samples, which can be caused by various factors, e.g., acquisition differences in medical imaging. To reach this goal, we propose FEDFA to tackle federated learning from a dis- tinct perspective of federated feature augmentation. FEDFA is based insight that each client's 's data distribution can be characterized by statisties (i.e., mean and standard deviation) of latent features; and it is likely to manipulate these local statistics globally, i.e., based on information in the entire federation, to let clients have better sense of the underlying distribution and therefore alleviate local data bias. Based on this insight, we propose to augment each local feature statistic probabilistically based on normal distribution, whose mean is the origi- nal statistic and variance quantifies the augmentation scope. Key to our approach is the determination of a meaningful Gaussian variance, which is accomplished by taking into account not only biased data of each individual client, but also un- derlying feature statisties characterized by all participating clients. We offer both theoretical and empirical justifications to verify the effectiveness of FEDFA. Our code is available at https: / /ttpt///.///bo/////////e////tb thub. fahou/
major
INTRODUCTION
Federated learning (FI.) (Konečný et al., 2016) is an emerging collaborative training framework that enables training on decentralized data residing devices like mobile phones. comes with the promise of training centralized models using local data points such that the privacy of participating devices is preserved, and has attracted significant attention critical fields like healthcare or finance. Since data come from different users, it is inevitable that the data of each user have different underlying distribution, incurring large heterogeneity (non-iid-ness) among users' data. In this work, focus on feature shift (Li et al., 2020b), which is common in many real-world cases, like medical data acquired from different medical devices or natural image collected in diverse environments.
While the problem of feature shift has been studied in classical centralized learning tasks like domain generalization, little is understood how to tackle it in federated learning. et 2020b; Reisizadeh et al., 2020; Jiang et al., 2022; Liu et al., 2020a) are rare exceptions. FEDROBUST (Reisizadeh al., 2020) and FEDBN (Li et al., 2020b) solve the problem through client-dependent learning by either fitting the shift with a client-specific affine distribution or learning unique BN parameters for each client. However, these algorithms may still suffer significant local dataset bias. Other works (Qu al., 2022; Jiang et al., 2022; Caldarola et al., 2022) learn robust models by adopting Sharpness Aware Minimization (SAM) (Foret et al., 2021) as the local optimizer, which, however, doubles the computational cost compared to SGD or Adam. In addition to model optimization, FEDHARMO (Jiang et al., 2022) has investigated specialized image normalization techniques to mitigate feature shift in medical domains. Despite the progress, there leaves an alternative space - data angmentation largely unexplored in federated learning, even though it has been extensively studied in centralized setting to impose regularization and improve generalizibility (Zhou et al., 2021; Zhang et al., 201 8).
While seemingly straightforward, it is non-trive non-trivial to perform effective data augmentation in feder- ated learning because users have no direct access to external data of other users. Simply applying conventional augmentation techniques each client is sub-optimal since without injecting global

---

Published as a conference paper at ICLR 2023
information, augmented samples will most likely still suffer local dataset bias. To address this, FED- MIx (Yoon et al., 2021) generalizes MIXUP (Zhang et al., 2018) into federated learning, by mixing averaged data across clients. The method performs augmentation in the imput level, which is natu- rally weak to create complicated and meaningful semantic transformations, e-g., maake-bsppectacee. Moreover, allowing exchange of averaged data will suffer certain levels of privacy issues.
In this work, we introduce a novel federation aware augmentation technique, called FedFA, into federated learning. FEDFA is based on the insight that statistics of latent features can capture essen- tial domain-aware characteristics (Huang & Belongie, 2017; Zhou et al., 2021; Li et al., 2022a;b; 2021a), thus can be treated as "features of participating client". Accordingly, we argue that the problem of feature shift in FL. no matter the shift of each local data distribution from the underlying distribution, or local distribution differences among clients, even test-time distribution shift, can be interpreted as the shift of feature statistics. This motivates us to directly addressing local feature statistic shift by incorporating universal statistic characterized by all participants in the federation.
FEDFA instantiates the idea by online augmenting feature statistics of each sample during local model training, so as to make the model robust to certain changes of "features of participating client". Concretely, we model the augmentation procedure in probabilistic manner via multi- variate Gaussian distribution. The Gaussian mean is fixed to the original statistic, and variance reflects the potential local distribution shift. In this manner, novel statisties can be effortlessly synthesized by drawing samples from the Gaussian distribution. For effective augmentation, we determine a reasonable variance based not only variances of feature statistics within each client, but also universal variances characterized by all participating clients. The augmentation in FEDFA allows each local model to be trained over samples drawn from more diverse feature distributions, facilitating local distribution shift alleviation and client-invariant representation learning, eventually contributing to a better global model.
FEDFA is conceptually simple but surprisingly effective method. It is non-parametric, requires negligible additional computation and communication costs, and can be seamlessly incorporated into arbitrary CNN architectures. We propose both theoretical and empirical insights. Theoretically, we show that FEDFA implicitly introduces regularization to local model learning by regularizing the gradients of latent representations, weighted by variances of feature statisties estimated from the entire federation. Empirically, we demonstrate that FEDFA (1) works favorably with extremely small local datasets; (2) shows remarkable generalization performance to unseen test clients outside of the federation; (3) outperforms traditional data augmentation techniques by solid margins, and can complement them quite well in the federated learning setup.
2
OUR APPROACH
2.1
PRELIMINARY: FEDERATED LEARNING
We assume standard federated learning setup with server that can transmit and receive messages from M client devices. Each client m E [M] has access Nm training instances {<I;, y4) Nm the form of image x; EX and corresponding labels y E that are drawn i.i.d. from device-indexed joint distribution, i.e., (It, y:)) Pp, Pm (z. y). The goal of standard federated learning is to learn a deep neural network: f(w,, wa) A g(wrg) a h((w)), where h X -> 2 is feature extractor with K convolutional stages: h k k o h!, and g : Z + y is a classifier To learn network parameters w- [Wy, Jr the empirical risk minimization (ERM) is widely used:
;=1 in
CERM (w)
(w) = [( (g(g h(x.). w:: w)] Σ CERM (w), where C mE[M]
(1)
Here the global objective LERM is decomposable as sum of device-level empirical loss objectives (i.e., (CERM) m.)) m. Each CLRM is computed based on a per-data loss function &, Due to the separation of clients' data, _ (w))(w) cannot be solved directly. FEDAVG (McMahan et al., 2017) algorithm to address this. It starts with client training of all the clients in parallel, with each client optimizing CERM independently. After local client training, FEDAVG performs model aggregation to average all client models into updated global model, which will be distributed back to the clients for the next round of client training. Here the client training objective in FEDAVG is equivalent to empirically approximating the local distribution Im,. by finite Nm, number of examples, i.e., Pe (x. v) = I/N... Iim 6(x== = 3i, y = =y)), where 8(x=11; y =yy) ys) is a Dirac mass centered at (æ44, y:)-
leading
2

---

Published as a conference paper at ICLR 2023
2.2 MOTIVATION
While the ERM-based formulation has achieved great success, it is straightforward to see that the solution would strongly depend how each approximated local distribution Pe mimics the un- derlying universal distribution IP. In real-world federated learning setup however, in all but trivial cases each p exhibits unique distribution shift from P. which causes not only inconsistency between local and global empirical losses (Acar et al., 2021; Wang et al., 2020), but also gener- alization issues (Yuan al., 2022). In this work, we circumvent this issue by fitting each local dataset a richer distribution (instead of the delta distribution) in the vicinal region of each sample (r,, 34) so as to estimate a more informed risk. This is precisely the principle behind vicinal risk minimization (VRM) (Chapelle et al., 2000). Particularly, for data point (zi, 11), vicinity distribu- tion V' (#,, D: (2, yk) is defined, from which novel virtual samples can be generated to enlarge the support of the local data distribution. In this way, we obtain improved approximation of Pm as Pm 1//N Di=1 Vm (f 0x/24, In centralized learning scenarios, various successful instances of Vm. e.g., MIXUP (Zhang et al., 2018), CUTMIX (Yun et al., 2019), have been developed. Simply applying them to local clients, though allowing for performance improvements (see Table 6), is sub- optimal since, without injecting any global information, P., only provides a better approximation to the local distribution Pm. rather than the true distribution We solve this by introducing a dedicated method FEDFA to estimate more reasonable V,, in federated learning.
2.3 FEDFA: FEDERATED FEATURE AUGMENTATION
FEDFA belongs to the family of label-preserving feature augmentation (Xie et al., 2020). During training. it estimates vicinity distribution Vh. at each layer ht augment hidden features in client m. Considering X E BBxCXPWW as the intermediate feature representation of B mini- batch im- ages, with spatial size (H x W) and channel number (C), and Y as corresponding label. Ve is label-preserving the sense that y/ (X m: Ym X m. Ym) A wk (X X )8(2 Ym), i.e., it only transforms the latent feature X* m.
to X ma
but preserves the original label V m
2.3.1 FEDERATED FEATURE AUGMENTATION FROM A PROBABILISTIC VIEW
Instead of explicitly modeling wh (X+ x our method performs implicit feature augmentation by manipulating channel-wist feature statistics. Specifically, for X its channel-wise statistics, ie., mean i and standard deviation tthh are given as follows:
W
Mm
HW
x (h,,)))
BxC
o
HW
Σ
(X k.(h.w) =hh)²² BxC
(2)
where X (k,,,,)) BRBx represents features at spatial location (h, w). As the abstract of latent fea- tures, these statistics carry domain -specff- information (e-g-, style). They are instrumental image generation (Huang & Belongie, 2017), and have been recently used for data augmentation in image recognition (Li et al., 2021a). In heterogeneous federated learning scenarios, the feature statistics among local clients will be inconsistent, and exhibit uncertain feature statistic shifts from the statis- ties of the true distribution. Our method explicitly captures such shift via probabilistic modeling. Concretely, instead of representing each feature X with deterministic statistics {n², ath ). hy- pothesize that the feature is conditioned on probabilistic statistics (Ah, ah.), which are sampled around the original statistics based on a and -t ~ N Sgz ). where each Gaussian's center corresponds to the original statistic, and the variance is expected to capture the potential feature statistic shift from the true distribution. Our core goal is thus to estimate proper variances 2pt /E², for reasonable and informative augmentation.
Gaussian distribution, i.e., p ~((p E3+,
Client-specific Statistic Variances. In client-side, compute client-specific variances of feature statistics based on the information within each mini-hatch:
B
B Eo =(o e E
5 ( L( C...
(3)
B
where Σ3 and Σ3, denote the variance of feature mean ph and standard deviation of that specific to each client. Each value in Σ2, or Σ², is the variance of feature statistics in particular channel, and its magnitude manifests how the chännel will change potentially in the feature space.
3

---

Published as a conference paper at ICLR 2023
Client-sharing Statistic Variances. The client-specific variances solely computed based on the data in each individual client, and thus likely biased due to local dataset bias. To solve this, further estimate client-sharing feature statistic variances taking information of all clients into account. Particularly, we maintain a momentum version of feature statistics for each client, which online estimated during training:
R
Pmm t- affim + (1 a)
1 B
A
E ERC
o² t ao + (1 a)
Σ o' E R
(4)
B
where /h and Tt are the momentum updated feature statistics of layer h in client m, and they are initialized as C-dimensional all-zero and all-one vectors, respectively. a momentum coefficient. We set same for both updating, and found no benefit to set it differently. In each communication, these accumulated local feature statistics are sent to the server along with model parameters. Let āº [pf ........ [ E RRAXC and o* = [øf..... ........ feature statistics of all clients, the client sharing statistic variances are determined in server-side by:
Jir .ff E RMxC denote collections of accumulated
M Elat1)? ERC, Eg
M E(O"1)² ER.
E m=1
(1th
(o M
Σ m=1
(S)
M
In addition, it is intuitive that some channels are more potentially to change than others, and will be favorable to highlight these channels to enable sufficient and reasonable exploration of the space of feature statistics. To this end, we further modulate client sharing estimations with a Student's I-distribution (Student, 1908; Van der Maaten & Hinton, 2008) with one degree of freedom to convert the variances to prohabilities. The i-distribution has heavier tails than other alternatives such Gaussian distribution, allowing highlight the channels with larger statistic variance, the same time, avoiding overly penalizing the others. Formally, denote x and x²1() as the shared variances of the jth channel in Σ/ and Eg (Eq. 5), respectively. They modulated by the I-distribution as follows:
(0 C(1+1/520) Ae
U C(1 ((22))=)))))-- 7 Ee (1 1/20000)-
E ER,
E R,
(6)
DC (1 1/2-0011
where 7/ and (. refer to the modulated variances the j-th channel. By applying Eq. 6 to each channel separately, obtain 7/+ = b .......... 7 RC and 7 = hyt)...., ,..., (?) RC as modulated statistic variances of all feature channels at layer h². In this way, the channels with large values = (or ) will be assigned with much higher importance in Yar 7 than other channels, allowing for more extensive augmentation along those directions.
Adaptive Variance Fusion. The modulated client sharing estimations {7 Yor) provide quan- tification of distribution difference among clients, and larger values imply potentials of more signifi- cant changes of corresponding channels in the true feature statistic space. Therefore, for each client, we weight the client specific statistic variances (E Σ2+ } by (Dut,T>> ), so that each client has sense of such difference. To avoid overly modification of client specific statistic variances, we add a residual connection for fusion, yielding an estimation of Gaussian ranges as:
Prp (7, 1) @ L E C..
bg (70* + 1) C D
ERC.
where @ denotes the Hadamard product.
Implementation of Feature Augmentation. After establishing the Gaussian distribution, we syn- thesize novel feature X k in the vicinity of X as follows:
x - 11th + pim where pt N Sge. )r 8th Ni S.
8
(8)
ott
Here X- is first normalized with its original statistics by (x), -o and further scaled with novel statistics (AE, g* } that are randomly sampled from corresponding Gaussian distribution. To make the sampling differentiable, use the re-parameterizztion trick (Kingma & Welling, 2013):
P p + E E dh tt E
4

---

Published as a conference paper at ICLR 2023
where C₁ ~N(,, 1) and 6, (O, follow the normal Gaussian distribution.
The proposed federated feature augmentation (FFA) operation in Eq. is blug-and-play layer, i.e., it can be inserted at arbitrary layers in the feature extractor h. In our implementation, we add FFA layer after each convolutional stage of the networks. During training. we follow the stochastic learning strategy (Verma al., 2019; Zhou et al., 2021; Li et al., 2022b) to activate each FFA layer with a probability of p. This allows for more diverse augmentation from iteration to iteration (based on the activated FFA layers). At test time, augmentation is applied. In Appendix A, we provide detailed descriptions of FEDFA in Algorithm and FFA in Algorithm 2.
THEORETICAL INSIGHTS
In this section, we provide mathematical analysis to gain deeper insights into FEDFA. To begin with, we show that FEDFA is noise injection process (Bishop, 1995; Camuto et al., 2020; Lim et al., 2022) that injects federation-awar noises to latent features.
Lemma 1. Consider client m [M]. for batch-wise latent X layer k. its augmentation in FEDFA (cf. Eq. 8) follows a noising process X Xx +e with the noise e taking the form:
em co Eot X montoe E tt
(10)
where Eu N,(,0 ~N(0,) 1), X = (x -xmn-wim/oo m//
Based on Lemma 1, we can identify the federation-awar implicit regularization effects of FEDFA.
Theorem 1. In FEDFA, the loss function LFEDEA of client can be expressed as:
CELDFA CERM + REG
(11)
where CERM is the standard ERM loss, and DREG is the regularization term:
CERM E(Xm. Eggh (Xm)). Ym):
(12)
CREG Yw.. (X.. (Xm)) (mm) E.Ez J' J'(Xm)e
(13)
where J denotes the Jacobian of layer (see Proposition 1 in Appendix for its explicit expression).
Theorem implies that, FEDFA implicitly introduces regularization local client learning by reg- ularizing the gradients of latent representations (i.e., VAL (X..) (x.))( ((g():: (Xm)), Ym))) weighted
by federation-aware noises in Lemma 1, i.e., Exez J (XXm)ee
4 EMPIRICAL RESULT
4.1 SETUP
Datasets. We conduct extensive experiments five datasets: Office-Calted 10 (Gong al., 2012), DPPn et al., 2019) and ProstateMRI(Liu et al., 2020b) for validation of FEDFA in terms of feature-shift non-IID, well as larger-scale datasets CIFAR-IO (Krizhevsky & Hinton, 2009) and EMNIST (Cohen et al., 2017) for cases of label distribution and data size heterogeneity, respectively.
Baselines. For comprehensive evaluation, we compare FEDFA against several state-of-the-at fed- erated learning techniques, including FEDAVG (McMahan et al., 2017). FEDAVGM (Hsu et al., 2019), FEDPROX (Li al., 2020a), FEDSAM (Qu al., 2022), FEDBN (Li et al., 2020b), FE- DROBUST (Reisizadeh et al., 2020), and FEDMIX (Yoon et al., 2021). Moreover, we compare with FEDHARMO (Jiang al., 2022) in ProstateMRI, that is specialized designed for medical imaging.
To gain more insights into FEDFA, we develop two baselines: SEDFA-R(ANDON) and FEDFA- C(I.FENT). FEDFA-R randomly perturbs feature statistics based on Gaussian distribution with a same standard deviation for all channels, i.e., Enet Eax = X, where À = 0.5. FEDFA-C performs augmentation based only on client specific variances, i.e., Eq. 7 turns into E2, Σ2,
$2.= o.. E3
Metrics. As conventions, we use top-1 accuracy for image classification and Dice coefficient for medical image segmentation, respectively. We report the performance only for the global model.
Implementation Details. We use PyTorch to implement FEDFA and other baselines. Following FEDBN (Li et al., 2020b), adopt AlexNet (Krizhevsky al., 2017) on Office-Caltech 10 and
5

---

Published as a conference paper at ICLR 2023
Table 1: Image classification performance Office-Caltech 10 and DomainNet test. Top-1 accuracy (%) is reported. Office-Caltes 10 has four clients: A(mazon), C(altech), D(SLR), Wiebcam), while DomainNet has six: C(lipart,, M(nfograph), P(ainting), Q(uickdraw), R(eal), and S(ketch). See $4.2 for details.
Office-Callen 10 (Gong et al., 2012)
DomainNet (Peng ct al., 2019)
Algorithm
C
D
W
Average
C
P
R
S
Average
FEDAVG
84.4
66.7
75.0
88.1
78.5
71.5
33.2
57.8
76.5
72.9
63.2
62.8
FEDPROX
84,9
64.0
78.1
88.1
78.8
70,9
32.9
61.2
74.1
71.1
67.9
63.0
FEDSAM
81.7
63.1
50.0
81.4
69.1
60.1
30.1
53.0
64.8
61.9
47.3
52,9
FEDAVGM
85.9
64.0
71.9
94.9
79.2
79.8
33.3
58.8
72.6
72.8
66.1
62.5
FEDROBUST
82.3
64.0
81.3
93.2
80.2
70.9
32.9
60.7
75.7
72.6
68.5
63.6
FEDBN
82.3
63.6
81.2
94.9
80.5
72.4
32.7
64.3
74.0
69.9
70.8
64.0
FEDMIX
81.7
63.1
81.3
93.2
79.8
75.9
34.1
61.7
73.8
69.4
70.6
64.3
FEDFA
88.0
65.8
90.6
88.1
83.1
77.4
34.9
61.2
78.8
73.2
73.5
66.5
DomainNet, using the SGD optimizer with learning rate 0.01 and batch size 32. Following FED- HARMO (Jiang al., 2022), employ U-Net (Ronneberger et al., 2015) on ProstateMRI using Adam as the optimizer with learning rate le-4 and batch size 16. The communication rounds are 400 for Office-Caltech 10 and NemainNet, and 500 for PROstateMRI, with the number of local up- date epoch setting to in all cases. For EMNIST, we strictly follow FEDMIX (Yoon et al., 2021) introduce data size heterogeneity by partitioning data w.r.t. writers, and train LeNet-5 (LeCun et al., 1998) using SGD with batch size 10. The total number of clients is 200 and only 10 clients sampled per communication round. We run 200 rounds in total. For CIFAR-IO we sample local data based on Dirichlet distribution Dir(a) to simulate label distribution heterogeneity. As (Qu et al., 2022: Kim et al., 2022). we set a to 0.3 or 0.6, and train a ResNet- 18 (He et al., 2016). The number of clients is 100 with participation rate 0.1, while the number of communication round is set to 100.
4.2 MAIN RESULTS
We first present the overall results on the five benchmarks, Le., Office-Caltech 10 and DomainNet in Table 1 and Fig. 1, ProstateMRI Table and Fig. 2, EMNIST and
80
75
CIFAR-10 in Table 3.
Teee Accoume (8)
Results on Office-Caltech 10 and DomainNet. FEDFA yields solid improvements over competing methods for image classification. As presented in Table 1, FEDFA leads to consistent performance gains the competi- tors across the henchmarks. The improvements over FE- DAvG can be large 4.6% and 3.7% on Office- Caltech 10 and DomainNet, respectively. Moreover, in comparison to prior data regmentation-basee algorithm FEDMIX, FEDFA also brings solid gains, i.e., 3.3% on Office-Caltecl 10 and 2.2% on DomainNet. This is en- couraging since our approach in nature better preserves privacy than FEDMIX, which requires exchanging aver- aged data across clients. Moreover, Fig. 1 depicts the convergence curves of comparative methods on Office-Caltech 10. At the early training stage (0 ~ 200 rounds), FEDFA shows similar training efficiency other baselines like FEDPROX and FEDMIX. But as the training goes, FEDFA is able to converge to a more optimal solution.
FasAugM
FedProx
60
FadAug
FedMAxx
250
Communicatin
Figure 1: Test accuracy versus communi-
cation rounds on Cfiic-Caltochh 10.
Results on ProstateMRI. FEDFA shows leading performance with extremely small local datasets. In some practical scenarios like healthcare, the size of local dataset can be very small, which poses challenge for federated learning. To examine the performance of federated learning algorithms in this scenario, we build mini-ProstateMRI by randomly sampling only 1/6 of all training samples in each client for training. Results are summarized in Table 2. FEDFA outperforms FEDAVG by significant margins (i.e., 3.0%) and it even performs better than FEDHARMO, which is specifically designed for medical scenarios. In addition, Fig. 2 shows how the performance of methods varies with respect to the size of local dataset. We train methods with different fractions (i.e., 1/6, 2/6, 2/6, 4/6, 5/6, 1) of training samples. FEDFA shows promising performance in all cases.

---

Published as a conference paper at ICLR 2023
Table 2:
Medical image segmentation accuracy mini- ProstateMRI test (Liu al., 2020b) with small-size local datasets. Dice score (%) reported. The dataset consists data from six medical institutions: B(IDMC), H(K), I(2CVB), (B)M(C). R(UNMC) and U(CL). The number in the bracket denotes the num- ber of training samples in each client. See $4.2 for more details.
94
Average (%)
13
80
Fedtwg
FedPrx
Algorithm
(32)
H (32)
I (46)
M (38)
(41)
U (32)
Average
88
FedBAM
FEDAVG
81.2
90.8
86.1
84.0
91.0
86.2
86.5
FedMix
FEDPROX
82.8
89.1
89.8
79.5
89.8
85.66
86.1
FadHarme
FEDAVGM
80.3
91.6
88.2
82.2
91.2
86.5
86.7
88
FEDSAM
82.7
92.5
91.8
83.6
92.6
88.1
88.5
2/6
3/6
FEDROBUST
81.7
91.3
91.5
88.5
89.4
84.2
87.7
Fraction of the training set in each client
FEDBN
88.9
92.3
90.6
88.1
87.6
85.4
88.8
FEDMIX
86.3
91.6
89.6
88.1
89.8
85,2
88.4
Figure 2:
Segmentation performance w.r.t local data size (i.e. fraction of train-
FEDHARMO
86.7
91.6
92.7
84.2
92.5
84.6
88.7
FEDFA 85.7
92.6
91.0
85.4
92.9
89.2
89.5
ing samples the whole training set).
Results on CIFAR-10 and EMNIST. In addi- tion to feature-shift non-i.i.d., FEDFA shows con- sistent improvements in label distribution het- erogeneity (CIFAR-1O and data size heterogene- ity (EMNIST). As shown in Table 3, in CIFAR- 10, FEDFA surpasses the second best method, FEDMIX, by 0.8% and 1.2 % with respect to two non-i.id levels Dir(0.6) and Dir(O.3), respec- tively. Notably, as the non-ii.d. level increasing from Dir(0.6) to Dir(0.3),, FEDFA tends to yield a larger gap of performance gain, showing a strong capability in handling severe non-i.i.d. scenarios. In addition, in EMNIST, FEDFA outperforms FEDROBUST by 1.1% and FEDMIX by 1.2%, respec- tively. These results reveal that though designed for feature shift non-i.i.d., FEDFA's very nature of data augmentation makes it fundamental technique to various non-ii.d. challenges in FL.
Table 3: Performance on CIEAR-I0 and EMNIST.
CIFAR-10
Algorithm
EMNIST
Dir (0.6)
Dir (0.3)
FEDAVG
73.3
69.2
84.9
FEDAVGM
73.4
69.1
85.5
FEDPROX
74.0
69.5
84.9
FEDBN
73.7
69.8
85.3
FEDSAM
74.3
70.0
86.5
FEDROBUST
74.9
70.5
86.7
FEDMIX
75.5
70.7
86.6
FEDFA
76.3
71.9
87.8
Table 4: Comparison of generalization performance to unseen test clients on the three benchmarks ($4.2).
Office-Caltech 10
Net
ProstateMRI M
Algorithm
A C
W Avg
C
Q R
S
Avg
Avg
FEDAVG
04,649.37199
55.9 60.4
63.1 27.5
49.6
44.7
51.7
48.2
47.5
60.7 85.3 78,4 67.2 83 59,0 72.3
FEDPROX
63.0 50.7 68.7 62.7 61.3
62.2 26.9 49.6
42.4
50.5 48.9 46.8
86.2
79.3 68.6 84.5 62.4 73.8
FEDROBUST 64.9 53.0 73.2 58.1 62.3 63.5 28.5 49.8 44.6 53,5 56.7 62.4
87.2 78.0 77.1 88.0 65.3 76.3
FEDMIX
65.1 52.6 73.8 58.9 62.6
28.0
50.1 45.9 53.3 56.8
62.
86.7 78.1 76.8 87.7 65.6 76.2
FEDFA
65.6 54.2 78.1 59.3 64.3
64.1 28.8 49.4 47.5 56.6 61.0 51.2 64.0 88.3 75.9 79.0 89.1 77.5
4.3
FEDERATED DOMAIN GENERALIZZATON PERFORMANCE
Federated learning dynamic systems, in which novel clients may enter the system after model training, most possibly with test-time distribution shift. However, most prior federated learning algorithms focus only improving model performance the participating clients, while neglecting model generalizability to unseen non-participating clients. Distribution shift often occurs during deployment, thus it essential evaluate the generalizability of federated learning algorithms. With core of data augmentation, FEDFA is supposed to enforce regularization to neural network learning, which could improve generalization capability.
To verify this, we perform experiments for federated domain generalization based on the leave- one-client-out strategy, i.e., training on M 1 distributed clients and testing on the held-out un- participating client. The results are presented in Table 4. As seen, FEDFA achieves leading gen- eralization performance on most unseen clients. For example, it yields consistent improvements as compared to FEDMIX, i.e., 1.7% on Office-Caltech 10, 1.6 % on DomainNet, and 1.3% on ProstateMRI, terms of average performance. Despite the improved performance, we find by com- paring to the results reported in Table and Table that, current federated learning algorithms still encounter significant participation gap (Yuan et al., 2022), i.e., the performance difference between participating and non- participating clients, which is critical issue that should be tackled in future.

---

Published as a conference paper at ICLR 2023
4.4 DIAGNOSTIC EXPERIMENT
We conduct a set of ablative experiments to enable a deep understanding of FEDFA.
FEDFA vs. FEDFA-C and FEDFA-R. We first verify FEDFA against the two baseline variants mentioned in $4.1. Both variants involve only device-dependent augmenta- tion of feature statistics, without explicitly considering any global information. As shown in Table 5, by randomly perturbing feature statistics, FEDFA-R shows no improvements or even suffers performance degradation on DomainNet and ProstateMRI against FEDAVG; FEDFA-C yields promising performance gains by taking into account client-specific feature statistic variances; by comparing FEDFA and FRDFA-C, confirm the significance of universal feature statistic information in federated augmentation.
Table 5: Efficacy of FEDFA over FEDFA- C and FEDFA R.
Variant
Office
DomainNet
ProstateMRI
FEDAVG
78,5
62.8
86.5
FEDFA-R
78.6
61.0
86.1
FEDFA-C
79.5
63.7
87.8
FEDFA 83.1
66.5
89.5
FEDFA vs. traditional augmentation methods. We compare FEDFA with four conventional data/feature augmen- tation techniques, i.e.. MIXUP (Zhang
Table 6: Efficacy of FEDFA against augmentation techniques.
Algorithm
Office
DomainNet
ProttateMRI
FEDAVG
78.5
62.8
86.5
MIXUP
79.2
63.4
87.0
et al.. 2018), MANIFOLD MIXUP (Verma et al., 2019), MIXSTYLE (Zhou et al., 2021) and MoEx (Li et al., 2021a). The results are presented in Table 6. We show that i) all the four techniques yield non-trivial improve- ments over FEDAVG, and some of them (e.g.. MoEx) even outperform well- designed federated learning algorithms (as compared to Tables 1-2); by accounting for global feature
M-MIXUP
79.6
63.5
87.6
MIXSTYLE
79.9
64.1
88.5
MoEx
80.2
64.6
88.3
FEDFA
83.1
66.5
89.5
FEDEA+MMIXUP
83.7
67.0
89.9
FEDFA + -M-MIXUP
83.6
66.9
90.2
FEDFA + +MXXSTYLE
84.0
67.2
90.2
FEDFA+MOEX
83.9
67.0
90.1
statistics, FEDFA surpasses all of them, yielding 2.9% 119% /1.01.0% % improvements over the second- best results on WeaeeeDDamaiNNN/Pstttteeeeee
respectively; iii) combining FEDFA with these tech- niques allows further performance uplifting, verifying the complementary roles of FEDFA to them.
Adaptive Variance Fusion. Next, we examine the effect of adaptive variance fusion in Eqs. 6-7. We design baseline "Direct Fusion" that directly com- bines the client-specific and client-sharing statistic variances as: Σ3 =
Table 7: Effectiveness of adaptive variance fusion.
Variant
Office DomainNet Pro
Direct Fusion 80.6
64.1
86.9
Adaptive Fusion
83.1
66.5
89.5
pm -E
We find from Table 7 that the baseline en-
counters severe performance degradation across all three benchmarks. A possible reason is that the two types of variances are mostly mis-matched, and the simple fusion strategy may cause significant changes of client-specific statistic variances, which would be harmful for local model learning.
Hyper-parameter analysis. FEDFA includes only two hyper- parameters, i.e., momentum coefficient a in Eq. 4 and stochastic learning probability p to apply feature statistic augmentation dur- ing training. As shown Fig. 3, (1) the model overall robust a. Notably, it yields promising performance at a a= 0, in which the model only uses the feature statistics of the last mini-batch in each local epoch to compute shienn-shrring statistic variances. This result reveals that FEDFA is insensitive to errors client-sharing statistic variances. (2) For the probability p, we see that FEDFA signifi- cantly improves the baseline (i.e., p= 0), even with a small proba- bility (e.g.. 0.11). The best performance is reached at p= 0.5.
Test Accuracy (%)
82
81
80
79
Prokuallsy
78
0.0
0.2
0.4
0.6
0.8
1.0
Value
Figure 3: analysis for and p.
Hyper-parameter
4.5 COMPLEXITY ANALYSIS
Computation and memory costs. FEDFA involves only several basic matrix operations, thus in- curring negligible extra computation cost. Compared to FEDAVG, it requires 4 4 C& more GPU memory allocation to store four statistic values (ii Jh Yyk ) at each of the K FFA layers. Here C2 is the number of feature channel at each layer k. The costs are in practice very minor, e.g., 18 KB/15.5 KB for AlexNet/U-Net. For comparison, FEDMIX requires 2x more GPU memory than
FEDAVG. The low computation/memory costs make FEDFA favorable for edge devices.
8

---

Published as a conference paper at ICLR 2023
Communication cost. In each round, FEDFA incurs additional communication costs since it re- quires the sending 1) from client to server the momentum feature statistics a and oh as well as 2) from server client the client sharing feature statistic variances 7/2h and Y at each layer k.
Thus, for K layers in total, the extra communication cost for each client is G. = - Cs. where the factor of 4 is for server receiving and sending two statistic values. We further denote Cm as the 18KB vs. Cm 99MB for AlexNet), hence, the extra communication bruden in FEDFA is almost negligible.
cost for exchanging model parameters in FedAvg. In general we have & Cm (e.g.,
RELATED WORK
Federated Learning. Recent years have witnessed tremendous progress in federated learning (Konečný al., 2016), which opens the door for privacy-preserving deep learning (Shokri & Shmatikov, 2015), i.e., train a global model on distributed datasets without disclosing private data information. FEDAVG (McMahan et al., 2017) is a milestone; it trains local models independently in multiple clients and then averages the resulting model updates via a central server once in a while. However, FEDAVG designed for i.i.d. data and suffers in statistical accuracy or even diverge deployed over non-i.i.d. client samples. To address this issue, numerous efforts have been devoted to handling heterogeneous federated environments, by, for example, adding a dynamic regularizer to local objectives in FEDPROX (Li et al., 2020a) and FEDDYN (Acar et al., 2021), conrecting client drift through variance reduction in SCAFFOLD (Karimireddy et al., 2020), adaptive server opti- mization in FEDOPT (Reddi et al., 2021), local batch normalization in FEDBN (Li et al., 2020b), or training a perturbed loss in FEDSAM (Qu et al., 2022) and FEDROBUST (Reisizadeh et al., 2020).
FEDMIX (Yoon et al., 2021), as far as know, is the only existing method that solves federated learning based on data augmentation. It adapts the well-known MIXUP algorithm (Zhang et al., 2018) from centralized learning into the federated learning scenario. Nevertheless, FEDMIX re- quires exchanging local data (or averaged version) across clients for data interpolation, thereby suffering privacy issues. In addition, FEDMIX operates on the input level, while our approach fo- cuses on latent feature statistic augmentation. Since deeper representations tend to disentangle the underlying factors of variation better (Bengio et al., 2013), traversing along latent space will poten- tially make our method encounter more realistic samples. This is supported by the fact that FEDFA achieves consistent performance improvements over FEDMIX in diverse scenarios.
Data Augmentation. Data augmentation has long and rich history in machine learning. Early studies (Schölkopf et al., 1996; Kukačka et al., 2017) focus on label-preserving transformations to employ regularization via data, alleviating overfitting and improving generalization. For image data, some techniques, like random horizontal flipping and cropping are commonly used for training of advanced neural networks (He et al., 2016). In addition, there is recent trend for label-perturbing augmentation, e.g., MIXUP (Zhang et al., 2018) or CUTMIX (Yun et al., 2019). Separate from these input-level augmentation techniques are feature augmentation methods (Verma et al., 2019; Li et al., 2021a; 2022b; Zhou et al., 2021) that make augmentation in latent feature space. These various data augmentation techniques have shown great successes to learn domain-invariant models in the centralized setup. Our method is an instance of tabel-preserving fcature augmentation, designed for federated learning. is inspired by recent efforts on implicit feature augmentation (Li et al., 202 a;; 2022b; Zhou et al., 2021) that synthesize samples of novel domains by manipulating instance-level feature statistics. In these works, feature statistics are treated as 'features', which capture essential domain-specific characteristics. In FEDFA, we estimate appropriate variances of feature statistics from a federated perspective, and draw novel statistics probablistically from a distribution centered on old statistics, while spanning with the variances. FEDFA avoids statistics mixing of instances from different clients, as done in FEDMIX (Yoon et al., 2021), thus can better preserve data privacy.
CONCLUSION
This work solves federated learning from a unique perspective of feature augmentation, yielding a new algorithm FEDFA that shows strong performance across various federated learning scenarios. FEDFA is based on a Gaussian modeling of feature statistic augmentation, where Gaussian vari- ances are estimated in a federated manner, based on both local feature statistic distribution within each client, well as universal feature statistic distribution across clients. We identify the im- plicit federation- aware regularization effects of FEDFA through theoretical analysis, and confirm its empirical superiority across suite of benchmarks in federated learning.

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY
Throughout the paper we have provided details facilitating reproduction of our empirical results. All our experiments are ran with single GPU (we used NVIDIA GeForce RTX 2080 Ti with 1IG memory), thus can be reproduced by researchers with computational constraints as well. The source code has been made publicly available in https:/gittubbcco/////oo//ffff wow/t/ohoooFeeAAA For the theoretical results, all assumptions, proof' and relevant discussions are provided in the Appendix.
ACKNOWLEDGMENTS
We would like thank the anonymous referees for their valuable comments for improving the paper. This study was partly supported by Varian Research, Switzerland Grant.
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In ICLR, 2021.
Yoshua Bengio, Grégoire Mesnil, Yann Dauphin, and Salah Rifai. Better mixing via deep represen- tations. In /CML, 2013.
Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108-118 116, 1995.
Debora Caldarola, Barbara Caputo, and Marco Ciccone. Improving generalization in federated learning by seeking flat minima. In ECCV, 2022.
Alexander Camuto, Matthew Willetts, Umut Simsekli, Stephen J Roberts, and Chris C Holmes.
Explicit regularisation in gaussian noise injections. In NeurlPS, 2020.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. NewrIPS, 2000.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In IJCNN, 2017.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In ICML, 2019.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. In ICLR, 2021.
Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kemee for unsupervised
domain adaptation. In CVPR, 2012.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Mcasuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:/909.06333. 2019.
Xun Huang and Serge Belongie. Arbitrary style transfer real-time with adaptive instance normal- ization. In ICCV, 2017.
Meirui Jiang, Zirui Wang. and Qi Dou. Harmofl: Harmonizing local and global drifts in federated learning on heterogeneous medical images. In AAAI, 2022.
Sai Pranceth Karimireddy, Satyen Kale, Mchryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020.
Jinkyu Kim, Gecho Kim, and Bohyung Han. Multi-level branched regularization for federated learning. In ICMI., 2022.
10

---

Published as a conference paper at ICLR 2023
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:13/2.611,, 2013.
arXiv preprint
Jakub Konečný, Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:/610.05492 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, University of Toronto, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90) 90, 2017.
Jan Kukačka, Vladimir Golkov, and Daniel Cremers. Regularization for deep learning: A taxonomy. arXiv preprint 1710v
2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25) 2324, 1998.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In IEEE Symposium on Security and Privacy (SP), pp. 656-672, 2019.
Boyi Li, Felix Wu, Ser-Nam Lim, Serge Belongie, and Kilian Weinberger. On feature normaliza- tion and data augmentation. In CVPR, 2021a.
Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. ICCV, 2021b.
Qilei Li, Jiabo Huang, Jian Hu, and Shaogang Gong. Feature-distribution perturbation and calibra- tion for generalized person reid. arXiv preprint 2rX.v:2205.11197, 2022a.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-66 60, 2020a.
Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and LINGYU DUAN. Uncertainty modeling for -iitt--distriuution generalization. In ICLR, 2022b.
Xiaoxiao Li, Meirui JIANG, Xiaofei Zhang, Michael Kamp, and Qi Dou. Fedbn: Federated learning on non iid features via local batch normalization. In ICLR, 2020b.
Soon Hoe Lim, N Benjamin Erichson, Francisco Utrera, Winnie Xu, and Michael W Mahoney. Noisy feature mixup. ICLR, 2022.
Quande Liu, Qi Dou, and Pheng Ann Heng. Shape-aware meta-learning for generalizing prostate mri segmentation to unseen domains. In MICCAI, 2020a.
Quande Liu, Qi Dou, Lequan Yu, and Pheng Ann Heng. Ms-net: Multi-site network for improving prostate segmentation with heterogeneous mri data. IEEE TMI, 2020b.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efffcieent learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282, 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang. Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. ICCV, 2019.
Zhe Qu, Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In ICML, 2022.
Sashank J Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In ICLR, 2021.
Amirhossein Reisizadeh, Farzan Farnia, Ramtin Pedarsani, and Ali Jadbabaie. Robust federated learning: The case of affine distribution shifts. In NeurIPS, 2020.
11

---

Published as a conference paper at ICLR 2023
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, 2010.
Bernhard Schölkopf, Chris Burges, and Vladimir Vapnik. Incorporating invariances in support vec- tor learning machines. In International conference on artificial neural networks, 1996.
Reza Shokri and Vitaly Shmatikov. Privacy-preserving deep learning. ACM SIGSAC, 2015.
Student. The probable error of mean. Biometrika, pp. 1-25, 1908.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9(11), 2008.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. NewrIPS, 2020.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. NeurIPS, 2020.
Tehrim Yoon, Sumin Shin, Sung Ju Hwang, and Eunho Yang. Fedmix: Approximation of mixup under mean augmented federated learning. In ICLR, 2021.
Honglin Yuan, Warren Richard Morningstar, Lin Ning, and Karan Singhal. What do mean by generalization in federated learning? ICLR, 2022.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In ICCV, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICI.R, 2018.
Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021.
12

---

Published as a conference paper at ICLR 2023
This appendix provides theoretical proofs, additional results and experimental details for the main paper. is organized five sections:
A summarizes the algorithms of FEDFA in Algorithm I and FFA in Algorithm 2;
B presents detailed theoretical analysis and proof' of our approach;
SC shows additional ablatíve experiments;
SD provides more detailed analysis of extra communication cost required by FEDFA; ŞE describes experimental details and more results.
A DETAILED ALGORITHM
In Algorithm 1, illustrate the detailed training procedure of our FEDFA. It is consistent with algorithms such as FEDAVG (McMahan et al., 2017). each communication round, the client performs local model training of the feature extractor h((w;)) and classifier g(w,,) We append FFA layer (Algorithm after each convolutional stage h Each client additionally maintains pair of momentum feature statistics (Am, Jm which is updated in momentum manner during training. which (,), }), with the momentum feature statistics are sent to server for model aggregation and computation of client-sharing statistic variances, which will be distributed back clients for the next round of local training.
The parameters from local training (i.e., {un, w,)),
omitted Algorithm 1, along
Algorithm FEDFA: federated training phase. (We omit the parameter updating procedure, which exactly same to FEDAVG.)
Input: Number of clients M; number communication rounds T; neural network f = g h; each
X m represents the collection of training images in corresponding clients;
Output: Txh, T
1: for = 1, 1,2,..., T do
2:
for each client m [M] do
3:
A. = 0, 5. =1
b Initialize averaged feature statistics for the client
4:
for each layer & E [K] do
5:
x = 1h ->((*(=1)
Run the k-th layer of the feature extractor > FFA layer in Algorithm
6:
X
m p Th. FFA(X in p ohh)
7:
Y g(X(k)
P Run classifer to get predictions
8:
Run loss computation and backward optimization
9:
2
1 MM M
(oth 1
Compute client sharing statistic variance (Eq. 5)
10:
D3 M Im=1(at (ot 1
11:
for each channel j E [C] do
Compute adaptive fusion coefficients (Eq. 6)
0(111/2(0))11
Σ (141//201
(11 De (14/////0)))
(
13:
Nath
14:
return Nuk, York
B THEORETICAL INSIGHTS
In this section, we provide mathematical analysis to understand FEDFA. We begin with interpreting FEDFA noise injection process (Bishop, 1995; Camuto et al., 2020; Lim et al., 2022), which is case of VRM ($2.1), and show that FEDFA injects federation-aware noises to latent representations ()B.1). Next, demonstrate that, induced by federation-awee aware noise injection, FEDFA exhibits natural form of federation aware implicit regularization to local client training ($B.2). Without loss of generality, conduct all analysis for an arbitrary client E [M].
13

---

Published as a conference paper at ICLR 2023
Algorithm 2 Algorithm description of FFA for the kth layer in client
Input: Original feature X mn E RBxxXxwWW momentum a= 0.99; probability p=0.5;
Client-sharing fusion coefficients T,ut E and "/ ERC downloaded from the server;
Accumulated feature statistics fi EIRC and a ERC;
Output: Augmented feature X k
m. Am om
1: if np.random. random() then
um 1 Σn Σw m- X (.(,ww)
2:
HW
Compute channel-wise feature statistics (Eq. 2)
3:
on
1 Ex=1 W
(Xmi
(h,,,)))
HW
4:
1 Σ (u 1)
D Compute client specific statistic variances (Eq. 3)
B
5:
Σ2
= (o=
(o - Elo
m
6:
$2
= (ppt + 1)
> Adaptive variance fusion (Eq. 7)
Σ2 (%a + 1) pm
Km
7:
8:
p pm - c4 kh
Sampling novel feature statistics (Eq. 9)
9:
2 = 5th + Co Lot
X m
g x -M
+ f
D Transform original feature based on novel statistics (Eq. 8)
ot
11:
p ajt
(1) Σ 1
Momentum updating feature statistics (Eq. 5)
12:
t aō m (1 a) Lb=1 mm
13:
return X 1 o
m.
B. UNDERSTANDING FEDFA FEDERATION AWARE NOISE INJECTION
Noise Injection in Neural Networks. Let x he a training sample and x its latent representation at the k-th layer, with no noise injections. The x² can be noised under a process a =z e* where e* is an addition noise drawn from a probability distribution, and x is the noised representation.
A popular choice of e is isotropic Gaussian noise (Camuto et al., 2020), i.e., e e*~NN(0, where is an identity matrix and a is scalar, controlling the amplitude of e To avoid over-perturbation that may cause model collapse, o is typically set a small value. Despite its simplicity. the strategy is confirmed as a highly effective regularized for tackling domain generalization (Li et al., 2021b) and adversarial samples (Lecuyer et al., 2019; Cohen et al., 2019). However, as shown in Table 5, its performance (see FEDFA-RJ is only marginally better sometimes worse than FEDAVG in FL.
Federation-Aware Noise Injection. From Eq. 9, we can clearly see that the feature statistic aug- mentation in our approach follows the noise injection process above. Next we show that this even- tually results in features perturbed under a federation-awart noising process.
Lemma 1. Consider client m, for batch-wise latent feature X at the k-th layer, its augmentation in FEDFA follows noising process
X X + e with the noise ex: taking the form:
e co Ep X E
(14)
where Eq ~ ~N 1), eo ~N0, 1), X (x
Proof of Lemma We can easily prove this by substituting Eq. into Eq.
8 = 8h X thth at:
+ pt::
X - pin (wi + en Su
(o + eo S.g
oh
(15)
x k + (colino 8 + cx(inp))
where (,,N(0,11,, 1), e ~~N(0,11 1), X = 1x oou
14

---

Published as a conference paper at ICLR 2023
As compared to the Gaussian noise injections (Camuto et al., 2020; Cohen et al., 2019; Lecuyer et al., 2019), the noise term e in FEDFA shows several desirable properties: is 1) data-dependent, adaptively determined based on the normalized input feature ; 2) channel-independent allowing for more extensive exploration along different directions in the feature space; 3) most importantly federation-aware, i.e., its strength is controlled by statistic variances Ene and Eo (cf. Eq. 7), which known carrying universal statistic information of all participating clients.
B.2 FEDERATION- -WARR IMPLICIT REGULARIZATION FEDFA
Next we show that with noise injections, FEDFA imposes federation-awann implicit regularization to local client training. By this, we mean regularization imposed implicitly by the stochastic learning strategy, without explicit modification of the loss, and the regularization effect is affected by the Sederation-awart noise (Lemma 1).
o Recall the deep neural network defined in $2.1: & go h, where h h- h- .... h! a K-layer CNN feature extractor and is a classifier. Given a batch of samples Xm with labels Yn, its latent representation at the k-th layer is computed as X = h h Note that we only add noises to layers in h, but not to g. Concretely, each mini-batch training, FEDFA follows stochastic optimization strategy to randomly select subset of layers from {h* Jk=1 and add noises to them. For simplicity, we denote [= }} as the index of all layers h, 2 C as the subset of layer indexes that are selected, E = {ei Vzez as the corresponding set of noises. Then, the loss function C.FDFA of client m in FRDFA can be equivalently written as CFDDF n
h (X or
we write in a simpler form, X m =X"*
F Zz>xL m2 where L a standard loss function I'm (ef. Eq. 1) imposed by adding noises to layers in 2. In the remainder, we relate the loss function CFEDEA to the original ERM loss CERM as well as a regularization term conditioned on E.
ERM
Theorem 1. In FEDFA, the loss function CEEDEA of client m can be expressed as:
FRDFA FRMM
REG
(16)
where CERM is the standard ERM loss, and CREG is the regularization term:
ERRM = F
Aggh 1: (Xm)), Ym),
(17)
CEEG EZ-xE(XX Yw) (Xm) Elg(h 2 (Xm)), Xm)
E JX
(18)
ZEZ
where J denotes the Jacobian of layer z (defined in Proposition 1).
Theorem implies that, FEDFA implicitly introduces regularization local client learning by reg- (Xm)), Yn)) (X weighted
ularizing the gradients of latent representations (iee. VAL (X..) (x.)) ((g(k
by federation-aware noises Σ:ez J² (X (Xm)ei.
In the remainder of this section, we prove Theorem 1.
For the sake of analysis, we first marginalize the effect of the noising process. We do so by defining an accumulated noise e in the final layer K, which originates from the forward propagation of all noises in E. We compute the accumulated noise based on (Camuto et al., 2020) that examines Gaussian noise injection into every latent layer in neural network. Formally, the accumulated noise on the final convolutional layer K can be expressed as follows:
Proposition 1. Consider a K Kllayer neural network, in which a random noise ei, is added to the activation of each layer z E Z. Assuming the Hessians, of the form (Xm))aam where k, are the indexes over layers, are finite. Then, the accumulation noise eK approximated as:
K
F(Xmeen +0(8)
(19)
LEEZ
where J E ROxXCs indicates the Jacobian of layer z, Le., J*(Xi = Th (Xm)s" where Cx and C2 denote the number of neurons in layer K and z, respectively. 0(3) represents higher order terms in & that lend to be zero in the limit of small noises.
0h.1 1: (Xmis
15

---

Published as a conference paper at ICLR 2023
Proof of Proposition 1. Starting with layer the first convolution layer, the accumulated noise on layer K can be approximated through recursion. If K 1, the accumulated noise is equal to e e For K =2 2, apply Taylor's theorem on h2(X,,, + te,,.) around the output feature X at
ht. . If we assume that all values in Hessian (?()))) is finite, the following approximation holds:
m)
Oh (x
1 (x + em.) 12 (x!))
m em + O(s)). 0x!
(20)
where ((K)) denotes asymptotically dominated higher order terms given the small noise. In this special case of 2, we obtain the accumulated noise as
Oh (x en + +((s)) en
K cm
(21)
0x
e + +(K1)) is the noise propagated from h while e e is the noise added to n²2 if the layer is activated; otherwise, e =. Note that Eq. 20 can be generalized to an arbitrary layer.
The noise consists of two components:
(x 8x!
Repeating this process for each layer hh (Xm) (m))aa(X..)) Ak < n are finite, we obtain the accumulated noise for layer as
2, and assuming that all Hessians of the form
Oh l (Xm) 8h1:=(Xm. en + 0(8) K
I>> I-K (Xm) en + +0(0))
ex
(22)
0h ah
SE
where ek e is the noise added to A layer K is activated; otherwise, E =0
0h 1 (X.)) as the Jacobian J= (Xm) completes the proof.
Denoting
0h (Xm)
Based on Proposition I, we provide linear approximation of the loss l for samples (X Ym)
f(g(h (Xm) e Yn) lKg(h" (Xm)), Yon) + (l.) (Xm)), T
(23)
= E(o(h (Xom)"
(Xm))), Y..) VALK lg 1: (Xm)),, Vmn
FXm)ee
in which the higher order terms in Proposition are neglected.
Based on Eq. 23, we further approximate the local training objective CLEDEA in client m and derive the regularization term as follows:
mm
Ez-xL²
L Ym [e(g(1 (Xm) ) ek)) em)).m)) Ym)]
FeNKF (Xm Y
[ (Xm)), Vm)+
VNLK(Xm) E(g(t 1: (m) Σ J(
(24)
=F(Xm.Ym)>> l L: (Xm)),
Lz En) (XEK(X E(g(h" (Xm)), Mm))
J(
16

---

Published as a conference paper at ICLR 2023
Table 9: A summary key experimental configuration for each dataset.
Hyper- parameters
Office Caltech 10
DomainNet
ProstateMRI
EMNIST
CIFAR 10
Jederation-awan configuration
Number of rounds
400
400
500
200
100
Local training epochs
1
10
10
Number of clients
100
100
Participation rate
1.0
1.0
1.0
0.1
0.1
Number total classes
62
local client training configuration
Network
AlxeNet
AlexNet
U-Net Adam
LeNet SGD
ResNet
Optimizer
SGD
SGD
SGD
Local batch size
32
32
16
64
10
Local learning rate
le-2
le-2
le-4
le-1
le-1
C ADDITIONAL ABLATION STUDY
Table 8: Performance for different sets of eligible layers to apply FFA.
In this section, we study the sensitivity of FedFA the set of eligible layers to apply FFA. For no- tation, we use {1} to represent that FFA is applied
Variant
Office
DomainNet ProstateMRI
FEDAVG
78.5
62.8
86.5
[1]
78.8
63.5
88.5
to the 1st convolutional stage; (1,2) to represent
(1,2)
80.0
63.9
88.6
that FFA is applied to both the 1st and 2nd con-
[1,2,3]
80.0
64.0
volutional stages; and so forth. The results are shown in Table 8. We observe that i) default design (using five layers) always shows the best per- formance the three datasets (Office, DomainNet and ProstateMRI). We conjecture that this due
(1,2,3,4]
80.6
64.3
88.8
(1,2.3.4.5)
83.1
66.5
89.5
(2,3,4,5)
81.6
65.2
88,6
(1,2,4,5)
82.0
65.8
(3,4.5)
78.4
63.8
87.0
(4.5)
79.4
64.7
85.9
its potential beget more comprehensive aug-
(5)
79.2
64.6
86.3
mentation; ii) applying FFA to only one particular
[1,5]
80.4
65.5
88.5
layer brings minor gains against FedAvg; but iii) by adding more layers, the performance tends to improve. This implies that our approach benefits
2,3.4]
79.5
64.3
(2,3)
78.7
64.0
88.5
[3.1]
78.3
63.2
86.5
from inherent complementarity of features in differ- network layers.
(3)
78.0
63.1
86.5
D ANALYSIS OF ADDITIONAL COMMUNICATION COST IN FEDFA
In each round, FEDFA incurs additional communication costs since it requires the sending 1) from client to server the momentum feature statistics p and at as well as 2) from server to client the client sharing feature statistic variances 7/2² and % at each layer k. Thus, for K layers in total,
the extra communication cost for each client is ce = EK Ck. where the factor of is for server receiving and sending two statistic values. As presented in Table 10 and Table 14, append one FFA layer after each convolutional stage of feature extractors AlexNet and U-Net. Hence, the total additional communication costs for AlexNet and U-Net
AlexNet: 4 (64 192 384 256 256)/1024 4 18 KB,
(25)
U-Net: 4 (32 64 + 128 256 512)
15.5 KB.
However, it should be noted that these additional costs minor in comparison with the cost quired for exchanging model parameters, which are 2 x 49.5 MB and 2 29.6 MB for AlexNet and U-Net, respectively.
17

---

Published as a conference paper at ICLR 2023
E EXPERIMENTAL DETAILS
E. DATASET
We conduct extensive experiments on five datasets:
Office-Caltech 10 (Gong et al., 2012) has four data sources, three from Office-31 (Saenko et al., 2010) and one from Caltech-256 (Griffin et al., 2007). They are collected from different camera devices or in diverse environments with different background.
DomainNet (Peng al., 2019) contains images from six domains (clipart, infograph, painting, real, and sketch), which are collected by searching category name along with a domain name in different search engines.
ProstateMMRI (Liu et al., 2020b) is multi-sit prostate segmentation dataset consisting of six data sources of T2-weighted MRI from different medical institutions. For all the three datasets, we regard each data source as client, and thus real-world feature shift exists among clients.
CTFAR-10 (Krizhevsky Hinton, 2009) a popular natural image classification dataset for feder- ated learning. It contains 50,000 training and 10,000 test images. We introduce label distribution heterogeneity for the dataset, by sampling local data based on the Dirichlet distribution Dir(a)), and consider two different concentration parameters, i.e., 0.6 or 0.3.
EMNIST (Cohen et al., 2017) is an image classification dataset with 62 classes, including all 26 capital and small letter of alphabet as well as numbers. We follow the setup in FEDMIX to simulate data size heterogeneity.
In Table 9, we summarize the configuration of our experiments for each of the datasets.
E.2 EXPERIMENTAL. DETAILS FOR IMAGE CLASSIFICATION
Network Architecture. For the image classification tasks on Office- Caltech 10 (Gong et al., 2012) and DomainNet (Peng al., 2019), use an adapted AlexNet (Krizhevsky et al., 2017), with the detailed network architecture shown in Table 10.
Training Details. For each training image in OfficeCCaltec110 and DomainNet, we reshape its size into 256. We train AlexNet with the SGD optimizier with a learning rate of 0.01, a mini-bbtch size of 32, using the standard cross-entropy loss. The total number communication round is set to 400, with one local epoch per round by default. Two basic data augmentation techniques are also ap- plied for training, i.e., random horizontal flipping and random rotation with degree in [-30, 30). The dataset splits of Office-Caltech 10 and DomainNet in experiments summarized in Table 11 and Table 12, respectively.
18

---

Published as a conference paper at ICLR 2023
Table Network architecture of AlexNet Office-Caltech10 DomainNet experiments. convolu- tional layer (Conv2D), we list parameters with sequence of input and output dimension, kernal size, stride and padding. For max pooling layer (MaxPool2D), we list kernal and stride. For fully connected layer (FC), we list input and output dimension. For Batch Normalization layer (BN), list the channel dimension. Note that FFA denotes the proposed feature augmentation layer, and list the dimension of its input feature.
Layer
Details
Conv2D(3, 64, 11, 2). BN(664) ReLU. MaxPool2D(3.2 2)
2
FFA(64)
3
Conv2D(64, 192, 5, 2), BN((92), ReLU, MaxPool2D(3, 2)
4
FFA(192)
5
Conv2D(192, 384, 3, 1. 1), BN(384), ReLU
6
(F84384)
7
Conv2D(384, 256, 3, 1, 1), BN(256), ReLU
8
FFA(256)
9
Conv2D(256, 256, 3, I, 1), BN(266), ReLU, MaxPool2D(3. 2)
10
FFA(256)
11
AdapiiveAvyPool266.. 6)
12
FC(9216, 1024), BN(1024), ReLU
13
FC(1024, 1024), BN(1024), ReLU
14
FC(1024, num_class)
Table 11: Numbers samples training, validation, and testing sets of each client Office-Caltech 10
used in our experiments.
Split
Amazon
Caltech
DSLR
Webcam
train
459
538
75
141
val
307
50
test
32
Table Numbers of samples in training, validation, and testing sets each client DomainNet used our experiments.
Split
Clipart
Infograph
Painting
Quickdraw
Real
Sketch
train
672
840
791
1280
1556
708
val
420
525
494
800
972
442
526
657
619
1000
1217
554
Table 13: Numbers of samples the training, validation, and testing sets of each client in ProstateMRI used in our experiments.
Split
BIDMC
HK
I2CVB
BMC
RUNMC
UCL
train
156
94
280
230
246
105
val
31
93
76
82
35
test
31
93
76
82
35
19

---

Published as a conference paper at ICLR 2023
E.3 EXPERIMENTAL DETAILS FOR MEDICAL IMAGE SEGMENTATION
Network Architecture. For medical image segmentation on ProstateMRI (Liu et al., 2020b), we vanilla U-Net architecture, as presented in Table 14 and Table 15.
Training Details. Following FEDHARMO, we use a combination of standard cross-entropy and Dice loss to train the network, using the Adam optimizer with learning rate 1c-4, batch size 16, and weight decay le-4. No any data augmentation techniques are applied. The dataset splits of ProstateMRR used in our experiments are summarized in Table 13
Additional Results. Table 16 provides a detailed performance statistic of different methods on ProstateMRI, w.r.t different fractions (1/6, 2/6, 9/6, 4/6, 5/6, 1) of training samples used in each client. The table corresponds to the plot in Fig. 2.
Table 14: Network architecture of U-Net for medical image segmentation experiments PRoLtateMRL The structure of 'Block' module is provided in Table 15. Note that FFA denotes the proposed feature augmentation layer, and we list the dimension of input feature.
Layer
Details
fcatures=32, name="encoderl"), MaxPoo12D(2. 2)
2
FFA(32)
3
Block(in fcatures=22 fcaturcs=64, name="oncodor2") MaxPool2D222 2)
4
FFA(64)
5
Block(in.features=6A features= 128, name="encoder3"), MaxPol12222
6
FFA(128)
7
Block(in.features// 128, featurcs=256, name="oncoder4") MaxPool2D(2, 2)
8
FFA(256)
9
Block(in_features- features=512, name="bottleneck")
10
FFA(512)
11
1aokkinneaaturs=51111 features=256, name="decodeer4)
12
Block(in features=256, features=128 128, name="decoderr")
13
flock(in.features=122 features=64, name="decoder2")
14
features=32, name="decoderl")
15
Conv2d(32, num,class, 1, 1)
Table 15: Detailed structure the 'Block' module in U-Net (Table 14)
Layer
Details
Conv2d(in features, features, 3, 1)
BatchNorr2d((feturre))
ReLU
4
Conv2dffeatures, features, 3,
5
Ratchhoom2d((eeauures)
ReLU
20

---

Published as a conference paper at ICLR 2023
Table 16: Segmentation performance ProstateMRI test (Liu et al., 2020b) in terms of Dice (%)
sample different fractions (l'/o, 2/e, 9/e, 4/a, 4/s, 1J) of training samples over original training set each client to study the effects of methods w.r. the variations of local training size. The table provides detailed statistic for Fig. 2.
Algorithm
BIDMC
HK
12CVB
BMC
RUNMC
UCL.
Average
fraction of training samples over whole training set:
FEDAYG (Mcha ct al., 2017)
81.2
90.8
86.1
84.0
91.0
86.2
86.5
FEDPROX (Li et al., 2020a)
82.8
89.1
89.8
79.4
89.8
85.6
86.
FEDAVGM (Hsu al., 2019)
80.3
91.6
88.2
82.2
91.2
86.5
86.7
FEDSAM (Qu et al., 2022)
82.7
92.5
91.8
83.6
92.6
88.1
88.5
FEDHARMO (Jiang et al., 2022)
86.7
91.6
92.7
84.2
92.5
84.6
88.7
FEDMIX (Yoon et al., 2021)
86.3
91.6
89.6
88.1
89.8
85.2
88.4
FEDFA-R
78.9
91.3
87.0
81.9
90.6
86.6
86.1
FEDFA-C
80.5
92.4
89.2
83.7
92.1
89.4
87.9
FEDFA
85.7
92.6
91.0
85.4
92.9
89.2
89.5
fraction of training samples over the whole training
FEDAVG (McMahan et al., 2017)
83.5
92.4
90.1
86.5
93.7
89.4
89.3
FEDPROX (Li et al., 2020a)
83.9
92.7
93.7
86.2
94.0
88.6
89.8
FEDAvGM (Hsu et al., 2019)
80.8
91.8
91.7
83.5
93.5
86.1
87.9
FEDSAM (Qu et al., 2022)
93.7
94.3
94.7
90.2
FEDHARMO (Jiang et al., 2022)
85.6
90.8
92.5
88.5
94.3
87.6
89.9
FEDMIX (Yoon et al., 2021)
88.6
92.7
93.3
90.0
91.7
87.3
90.6
FEDFA-R
81.5
92.1
90.9
84,2
93.1
89.4
88.5
FEDFA-C
81.1
92.0
92.9
94.0
86.9
88.7
FEDFA
87.8
92.3
92.9
86.6
95.1
89.8
90.8
fraction of training samples the whole training
3/6
FEDAVG (McMahan et al., 2017)
84,2
90.9
93.9
87.0
93.8
89.1
89.8
FEDPROX (Li et al., 2020a)
82.0
92.5
92.0
84.7
92.3
88.1
88.6
FEDAvGM (Hsu et al., 2019)
81.8
92.0
94.4
85.7
93.5
88.7
89.3
FEDSAM (Qu al., 2022)
85.1
92.9
95.0
88.2
95.4
90.3
91.2
ARMO (Jiang al., 2022)
90.9
96.3
95.0
86.7
91.8
FEDMIX (Yoon et al., 2021)
87.2
92.8
94.9
90.1
92.5
90.0
91.3
FEDFA-R
82.7
92.8
94.4
87.0
94.7
88.2
90.0
FEDFA-C
87.0
91.1
93.5
90.3
93.9
90.9
FEDFA
87.4
93.5
95.6
90.6
95.6
91.4
92.4
fraction of training samples over the whole training
4/6
FEDAyG (McMahan et al., 2017)
92.3
95.7
87.7
95.2
89.3
91.1
FEDPROX (Li et al., 2020a)
87.2
92.9
94.0
85.8
94.5
89.7
90.7
FEDAvGM (Hsu et al., 2019)
85.6
93.0
95.0
84,9
94.8
87.3
90.1
FEDSAM (Qu al., 2022)
91.3
93.8
95.8
91.1
96.0
92.5
93.4
FEDHARMO (Jiang al., 2022)
94.2
90.3
95.8
92.2
93.0
FEDMIX (Yoon et al., 2021)
90.0
94.2
95.5
91.7
93.2
91.5
92.7
FEDFA-R
84.9
91.1
95.2
86.0
95.4
91.2
90.6
FEDFA-C
87.4
93.6
95.2
88.8
95.4
91.4
92.0
FEDFA
91.1
93.6
95.1
89.5
95.3
91.2
92.6
fraction of truining sampley over the whole training set: 5/6
FEDAYG (McMahan al., 2017)
89.2
94.2
94.8
89.0
95.1
91.3
92.3
FEDPROX (Li et al., 2020a)
88.7
94.7
94.8
88.9
95.6
91.1
92.3
FEDAvGM (Hsu et al., 2019)
87.7
94.6
94.3
89.3
95.2
90.4
91.9
FEDSAM (Qu al., 2022)
91.9
95.2
96.1
90.7
96.3
91.2
93.5
FEDHARMO (Jiang al., 2022)
89.8
94.5
95.6
91.0
96.0
92.1
93.2
FEDMIX (Yoon et al., 2021)
91.6
92.9
94.4
92.8
94.0
92.0
93.0
FEDFA-R
89.9
94.6
95.6
88.8
95.7
91.4
92.7
FEDFA-C
90.3
94.7
95.5
88.7
95.8
91.5
92.7
FEDFA
95.1
96.4
96.3
93.9
fraction of training samples over the whole training set:
FEDAvG (McMahan al., 2017)
89.5
94.0
95.6
87.5
95.1
91.1
92.1
FEDPROX (Li et al., 2020a)
94.1
95.6
95.9
92.4
FEDAVGM (Hsu et al., 2019)
92.1
94.2
96.0
87.4
95.0
90.6
92.6
FEDSAM (Qu et al., 2022)
92.0
95.4
96.6
90.1
96.3
92.2
93.8
FEDHARMO (Jiang al., 2022)
87.4
95.0
95.5
91.2
96.1
92.2
92.9
FEDMIX (Yoon et al., 2021)
93.7
95.2
95.7
91.9
94.8
92.8
94.0
FEDFA-R
87.1
92.5
95.0
86.9
95.7
90.8
91.3
C
91.5
93.4
96.5
88.9
95.4
91.6
92.9
FEDFA
93.7
95.4
96.0
92.2
96.2
92.5
94.3
21


Reviews:
[{"review_id": "cuar0BZL8X7", "reviewer": "Reviewer_q8Hi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Well known problem by now is the non-iidness of the client data causes degradations in FL trained models. The paper proposes FedFA, federated feature augmentation to address feature shift in clients data. FedFA exploits instance based statistics to augment local features to curb the introduction of novel domains because of augmentations. The proposed approach sounds promising to advance FL.", "summary_of_the_review": "Please refer to the strengths and weaknesses ", "strength_and_weaknesses": "# Pros:\n\n- The paper is well-written, easy to understand and follow along.\n\n- The idea of treating multiple channels as multiple variates and measuring the statistical variances is really interesting since they will always remain normally distributed as they are measured over mean and variances already.\n\n- A very simple technique with no or negligible communication overheads.\n\n- Literature is covered adequately.\n\n# Cons:\n\n- The FL based hyper params are missing in the paper, such as the number of rounds, the number of clients participating in training, client/device selection if any, etc.\n\n- The scale of the benchmarks is small compared to CIFAR-10 or EMNIST, can you show the method works on one/both of these relatively large dataset.\n\n- Also, the number of samples per client seems to be uniform and the proposed statistical measures seem to work well for something like that. Can you experiment with an unequal number of samples per client? \n\n- The main advantage that I see with FEDFA are the communication overheads compared to mixup kind of augmentation approaches (of course privacy is there, this on top), why not asymptotically quantify the costs?The cost analysis that is currently presented is talking about some raw numbers, they will go away if it is \\BigO notation and the costs might be almost same as FedAvg.\n\n- It looks like there is an ablation on \\alpha and \\p done separately, but what is the best combination, can we call it (\\alpha, p) = (1, 0.5) or (1.,0, 1.0) or (0.5,0.5)?\n\n\n### Minor comments: \n\n- “Since data coming different users … ” should be corrected\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "bEELce5DzKs", "reviewer": "Reviewer_uZsm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper develops a robust federated learning algorithm to address feature shift in clients’ samples, which can be caused by various factors, e.g., acquisition differences in medical imaging. The paper proposes FEDFA to tackle federated learning from a distinct perspective of federated feature augmentation. The paper models each feature statistic probabilistically via a Gaussian distribution, with the mean corresponding to the original statistic and the variance quantifying the augmentation scope, from which novel feature statistics can be drawn to fulfill augmentation.", "summary_of_the_review": "The paper proposes a federated learning algorithm from a novel perspective of feature augmentation. Conceptually, it can bring additional advantages, compared to the SOTA work in this area (FEDMIX). However, the evaluations may be further improved by using the same datasets and model architecture in the original paper of FEDMIX. A key assumption of this paper is that the each feature statistic follows a multi-variate Gaussian distribution. The paper may discuss more details of how/when this assumption is satisfied in tasks relying on deep neural networks. ", "strength_and_weaknesses": "Strength:\n(1) The paper propose a federated learning algorithm from a novel perspective of feature augmentation.\n(2) The proposed approach is conceptually simple but seems to be effective.\n\n\nWeakness:\n(1) The most related baseline seems to be FEDMIX. However, the evaluated datasets and the adopted model architecture in this paper are different from the datasets in the original paper of FEDMIX. There is limited discussion of the rationale of how the datasets are selected and why they are different from the original paper of FEDMIX.\n(2) A key assumption of this paper is that the each feature statistic follows a multi-variate Gaussian distribution. I am not sure whether the assumption is valid, for the neural embeddings in general. The paper may discuss more details of how/when this assumption is satisfied in tasks relying on deep neural networks.\n(3) Previous works (e.g., FEDMIX) provide theoretical discussions. This paper may be further improved by providing theoretical discussions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Kq2dJmNlJ4b", "reviewer": "Reviewer_7jRY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to address feature shift by modelling feature statistics among clients.  Specifically, the authors model feature statistic via a Gaussian distribution.  The mean of the Gaussian distribution denote the original statistic while the variance denote the augmentation scope. The authors designed a solution to determine an appropriate variance for the Gaussian distribution. In the solution, the Gaussian variance is estimated by not only the local data but also the feature statistics from all participating clients.  In the variance estimation, the authors designed an adaptive variance fusion strategy to assign different weights to different channels in the feature statistic space.\nAfter establishing the Gaussian distribution, the novel features can be synthesised by sampling from the Gaussian distribution.", "summary_of_the_review": "This work propose to model feature statistic via a Gaussian distribution. The parameters of the Gaussian distribution is estimated from the data from active clients. For the parameter estimation, the authors designed client sharing statistic variances and adaptive variance fusion strategies.  Once the parameters are estimated, new features are generated by sampling from the Gaussian distribution. Although the idea seems interesting, the writing of this work needs significant improvement. I feel hard to fully understand the technical details.", "strength_and_weaknesses": "Strength:\n-- modeling the feature statistic via a Gaussian distribution and generating new features by sampling from the Gaussian distribution seems interesting.\n-- the client sharing statistic variances and adaptive variance fusion strategy are technically solid.\n\nWeakness:\nWhat concerns me is the writing of this work. I feel it is not clear enough and I feel a little hard to fully understand the technical details.\n-- I feel difficult to fully understand the technical details. For example, in Eq 6, there is j  while in Eq 7 there isn't. How to use the result calculated in Eq 6 for Eq 7? \n-- How to take advantage of the shared feature statistics are not clear to me. Specifically,  the relation between Eq. 5/6/7 is not clear to me.\n-- the formulation terms in this draft need refined. For example, in page 4, \\mu_m^k \\sim \\mathcal{N} (\\mu_m^k, ##).  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KatJXYY1om", "reviewer": "Reviewer_JqCN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes federated feature augmentation to solve the data heterogeneity in federated learning. Each local model models its feature with Gaussian distribution, where the variance is the key to fulfill augmentation.\nThe experiments on image classification and medical image segmentation show effectiveness of the proposed work.  ", "summary_of_the_review": "The paper proposes a novel feature augmentation method for non-i.i.d. federated leaning. \nMy main concern is the method only shows effectiveness with local data split by data origin. There should be more experiments for other kinds of local data heterogeneity (e.g. variant local data size). A typical FL setting is to split data with Dirichlet distribution. ", "strength_and_weaknesses": "Strength:\n1. The proposed method targets at the non-i.i.d. problem of FL, and the idea of feature level augmentation is interesting. \n2. The FedFA method is simple but effective. Empirical comparisons with classical FL methods demonstrate the effectiveness of the proposed feature augmentation method.\n\nWeakness:\n1. FedRobust and FedBN also targets at feature shift problem in FL, it is not clear why \"these algorithms may still suffer significant local dataset bias\". Could the authors provide in-depth theoretical analysis how the proposed method is different/superior compared with these two counterparts? Besides, the experiments should show comparisons with FedRobust and FedBN.\n2. I wonder if the method utilizes the statistics of features from all layers or specific layers. The paper lacks discussions and ablations regarding this. \n3. The experiments only focus on splitting local data with different data origin. I would expect results with other kinds of local data heterogeneity, such as large variances among local data size or different modalities among locals. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster