On the Importance and Applicability of Pre-Training for Federated Learning
Authors: Hong-You Chen
Cheng-Hao Tu
Ziwei Li
Han Wei Shen
Wei-Lun Chao
Venue: ICLR 2023 poster (2023)
Keywords: federated learning
pre-training

Abstract: Pre-training is prevalent in nowadays deep learning to improve the learned model's performance. However, in the literature on federated learning (FL), neural networks are mostly initialized with random weights. These attract our interest in conducting a systematic study to explore pre-training for FL. Across multiple visual recognition benchmarks, we found that pre-training can not only improve FL, but also close its accuracy gap to the counterpart centralized learning, especially in the challenging cases of non-IID clients' data. To make our findings applicable to situations where pre-trained models are not directly available, we explore pre-training with synthetic data or even with clients' data in a decentralized manner, and found that they can already improve FL notably. Interestingly, many of the techniques we explore are complementary to each other to further boost the performance, and we view this as a critical result toward scaling up deep FL for real-world applications. We conclude our paper with an attempt to understand the effect of pre-training on FL. We found that pre-training enables the learned global models under different clients' data conditions to converge to the same loss basin, and makes global aggregation in FL more stable. Nevertheless, pre-training seems to not alleviate local model drifting, a fundamental problem in FL under non-IID data.

Published as a conference paper at ICLR 2023
ON THE IMPORTANCE AND APPLICABILITY OF PRE- TRAINIIGG FOR FEDERATED LEARNING
Hong- You Chen, Cheng-Hao Tu, Ziwei Li, Han-Wei Shen, Wei-Lun Chao Department of Computer Science and Engineering, The Ohio State University
ABSTRACT
Pre-training is prevalent in nowadays deep learning to improve the learned model's s performance. However, in the literature on federated learning (FL), neural networks are mostly initialized with random weights. These attract our interest in conducting a systematic study to explore pre-training for FI., Across multiple visual recognition benchmarks, we found that pre-training can not only improve FL., but also close its accuracy gap to the counterpart centralized learning. especially in the challenging cases of non-IID clients' data. To make our findings applicable to situations where pre-trained models are not directly available, we explore pre-training with synthetic data even with clients' data in decentralized manner, and found that they already improve FL notably. Interestingly, many of the techniques we explore are complementary each other to further boost the performance, and view this a critical result toward scaling up deep FL. for real-world applications. We conclude our paper with an attempt understand the effect of pre-training on FL. We found that pre-training enables the Iearned global models under different clients' data conditions converge to the same loss basin, and makes global aggregation in FL more stable. Nevertheless, pre- pre-training seems to not alleviate local model drifting, a fundamental problem in FL under non-IID data.
INTRODUCTION
The increasing attention to data privacy and protection has attracted significant research interests in federated learning (FL) (Li et al. 2020a Kairouz et al. 2019). In FL, data are kept separate by individual clients. The goal is thus to learn a "global" model in decentralized way. Specifically, one would hope to obtain a model whose accuracy is as good as if it were trained using centralized data.
FEDAVG McMMhhn al., 2017 is arguably the most widely used FL algorithm, which assumes that every client is connected to a server. FEDAVG trains the global model in an iterative manner, between parallel local model training at the clients and global model aggregation at the server. FEDAVG is
easy to implement and enjoys theoretical guarantees of convergence (Zhou & Cong 2017 Stich
2019: Haddadpour & Mahdavi 2019 Li al. 2020c; Zhao al. 2018). Its performance, however, clients' data are often collected individually
can degrade drastically when clients' data are not IID
and doomed be non-IID. That is, the accuracy of the federally learned global model can be much
lower than its counterpart trained with centralized data. To alleviate this issue, existing literature has
explored better approaches for local training (Li et al. 20206] Karimireddy a [020bb Acar et al..
1 and global aggregation (Wang al.) 2020a Hsu et al. 2019:Chhnn & Chao 2021
In this paper, we explore a different and rarely studied dimension FL - model initialization. In
the literature on FL, neural networks are mostly initialized with random weights. Yet in centralized learning, model initialization using weights pre-trained on large-scale datasets (Hendrycks et al. 2019; Devlin et al.) 2018) has become prevalent, as it has been shown to improve accuracy, generalizability. robustness, etc. We are thus interested 1) whether model pre-training is applicable in the context of FL and 2) whether it can likewise improve FEDAVG, especially in alleviating the non-Ill) issue.
We conduct the very first systematic study in these aspects, using visual recognition as the running example. We consider multiple application scenarios, with the aim to make study comprehensive.
First, assuming pre-trd weights (c.g., on ImageNet (Deng et al. 2009) are available, we systemat- ically compare FEDAVG initialized with random and pre-trained weights, under different FL. settings

---

Published as a conference paper at ICLR 2023
and across multiple visual recognition tasks. These include four image classification datasets, CIFAR- 10/100 (Krizhevsky et al.] 2009). Tiny-Imane Tiny-ImaeNet (Le & Yang 2013). and iNaturalist (Van Hom et al. 2018), and one semantic segmentation dataset, Cityscapes (Cordts et al. 2016). We have the following major observations. We found that pre-training consistently improves FEDAVG; the relative gain is more pronounced in more challenging FL. settings (e.g., severer non-III conditions across clients). Moreover, pre-training largely closes the accuracy gap between FEDAVG and centralized learning Figure D. suggesting that pre-training brings additional benefits to FEDAVG than to centralized learning. We further consider more advanced FL methods (e.g., (Li et al. 2020b Acar et al. 2021 Li al. 202Ibj). We found that pre-training improves their accuracy but diminishes their gain against FEDAVG, suggesting that FEDAVG is still a strong FL approach if pre-trained weights are available.
Second, assuming pre-trained models are not avail- able and there are no real data at the server for pre- training, we explore the use of synthetic data. We investigate several simple yet effective synthetic
Acc./ mloU (%)
Avg. gains of pro-training
90
Centralized
FedAvg
Using ImageNett
+6.7
+21.9
Using Synthetic
-3.9
+11.8
image generators (Baradad et al. 202I). includ-
60
ing fractals which shown to capture geometric patterns found in nature (Mandelbrot & Mandel brot (982). We propose new pre-training scheme called Fractal Pair Similarity (FPS) inspired by the inner workings of fractals, which can consistently improve FEDAVG for the downstream FL tasks. This suggests the wide applicability of pre-training to FL. even without real data for pre-training.
50
30
20
CIFAR-10
INaturalist
Citscappes
Centralizdd random
syntheto
Cemrrllzed imaeeeet
Fecikug nendom
Feddfg synthedi
FaedAn insggiive
Figure I: Pre-training improves FEDAVG more than it improves centralized learning. We consider three initialization weights: random, pre-trained on ImageNet, and pre-trained on synthetic images. Pre- training helps both FEDAVG and centralized learning, but has larger impact on FEDAVG. Even without real data, our proposed pre- pre-training with synthetic data is sufficient to improve FEDAVG notably.
Third, we explore the possibility to directly pre- train with clients' data. Specifically, we investigate the two-stage training procedure- - -elf-supervised pre-training, followed by supervised learning - in a federated setting. Such a procedure has been shown to outperform pure supervised learning in centralized learning (Chen et al., 2020c), but has not been explored in FL. Using the offte--f-the-arr
federated self- supervised approach (Lubana et al.] (022). ). we not only demonstrate its effectiveness in FL., but also show its compatibility with available pre-trained weights to further boost the performance.
Intrigued by the improvement brought by pre-training, we make an attempt to understand its underly- ing effect FL.. We first analyze the training dynamics of FEDAVG. We found that pre-training seems to not alleviate local model drifting (Li et al., [2000]] Karimireddy et al. 2020b). well-known issue under non-IID data. Nevertheless, it makes global aggregation more stable. Concretely, FEDAVG combines the local models' weights simply by coefficients proportional to local data sizes. Due to model drifting local training, these coefficients can be far from optimal (Chen Chao 2021).
Interestingly, with pre-training, FEDAVG is less sensitive to the coefficients, resulting in stronger global model in terms of accuracy. Through visualizations of the loss landscapes (Li et al. 2018 Hao et al. 2019), we further found that pre-training enables the learned global models under different data conditions (i.c., IID or various non-lID degrees) to converge to the same loss basin. Such a phenomenon can hardly be achieved without pre- pre-training, even if we initialize FEDAVG with the same random weights. This offers another explanation of why pre-training improves FL.
Contributions and scopes. We conduct the very first systematic study on pre-training for FL., including novel synthetic data generator. We believe that such study is timely and significant to the FL community. We focus on visual recognition using five image datasets. We go beyond them by further studying semantic segmentation problems on the Cityscape dataset, not merely classification problems. Our extended analyses reveal new insights into FL, opening up future research directions.
2 RELATED WORK
Federated learning (FL). FEDAVG (MeMahan et al. 2017) is the fundamental FL algorithm. Many works were proposed to improve it, especially to alleviate its accuracy drop under non-IID data. In global aggregation, Wang et al. (2020a): Yurochkin al. (2019) matched local model weights before
2

---

Published as a conference paper at ICLR 2023
averaging. Lin et al. 2020); He et al. (2020a); Zhou et al. (2020); Chen & Chao (2021) replaced
weight average by model ensemble and distillation. Hsu et al. (2019) 19;; Reddi et al. (2021 1) applied
server momentum and adaptive optimization. In local training.
reduce local model drifting
problem commonly believed to cause the accuracy drop /hao et al. (2018) mixed client and server data; FEDPROX (Li al.) (020b), FEDDANE (Li et al. 2019), and FEDDyN (Acar et al. 2 employed regularization toward the global model; SCAFFOLD (Karimireddy et al.] (2020a) and MIME (Karimireddy al. 2020b) used control varieties or server statistics correct local gradients; Wang et al. (2020b); Yao et al. (2019) modified local update rules.
We investigate a rarely studied aspect improve FEDAVG, initialization. To our knowledge, very few works in FL have studied this (Lin et al. [2022] Stremmel & Singh 2021| Weller et al. (2022); none are as systematic and comprehensive as our (Qu et al. 2021 Hsu al., 2020 Cheng et al., 2021) used pre-trained models in their experiments but did not or only briefly ana aallyzed their impacts.
Pre-training. Pre-training has been widely applied in computer vision, natural language processing, and many other application domains to speed up convergence and boost accuracy for downstream
tasks (Kolesnikov et al.) 2020; Goyal et al.) 2021 Radford et al. 2021 Sun et al.
2017: Devlin
et al. 2018; Yang et al. 2019] Brown et al. (202)). Many works have attempted to analyze its
2019 Djolonga et al. impacts (Hendrycks et al. 2020 Erhan et al. He et al.
202 He et al.
Kornblith et al. 2019). For example, Hendrycks et al. (2019) and Chen et al. (2020a) found
that pre-training improves robustness against adversarial examples; Neyshabur et al.
(020) studied
the loss landscape when fine-tuning target tasks; Mehta al. (2021) empirically showed that
pre-training reduces forgetting in continual learning. Despite the ample research on pre-training, its
impacts on FL remain largely unexplored. We aim fill in this missing piece.
3 BACKGROUND: FEDERATED LEARNING
In federated learning (FL), the training data are collected by M clients. Each client m E [M]] has training set Dm {<,, y1)) !2r! where the input (e.g., images) and y the true label. The goal is to solve the following optimization problem
Danl Σ
Dmml
1
min (
Σ
(m (0), [D]
where (0) 1Dml
fla Vs: 0)-
Here, 0 is the model parameter; D = Um Dm is the aggregated training set from all clients; L is the empirical risk on D: 2. is the empirical risk of client m; E is the loss function on a data instance.
Federated averaging (F (FEDAVG). As clients' data separated. [Equation caanot be solved directly; otherwise, it is centralized learning. A standard way to relax it is FEDAVG (McMahan et al. (017). which iterates between steps - parallel local training at the clients and global aggregation the server - for multiple rounds of communication
M 1Dm D
Local: 0(() = arg ming (0), initialized by 0((-1); Global: 0(0)
(2)
TTD
The superscript / indicates the models after round l; (0)) denotes the initial model. That is, local training aims to minimize each client's empirical risk, often by several epochs stochastic gradient descent (SGD). Global aggregation takes element-wise average over local model parameters.
Problem. When clients' data are non-llI, g(t) would drift away from each other and from Ø(t-1),
making (() deviate from the solution of Equation and resulting in a drastic accuracy drop.
PRE-TRAINING IS APPLICABLE TO AND IMPORTANT FOR FL
In most of the FL literature that learns neural networks, (0) is initialized with random weights. We thus aim to provide a detailed and systematic study on pre-training for FL. Specifically, we are interested in whether pre- pre-training helps FEDAVG alleviate the accuracy drop in non-IID conditions.
A concurrnt work by Nguyen et al. (022)) presents an empirical study as well. They focus more on how pre-training affects federated optimization algorithms. We study both the impact and applicahility of pre-training for Fl., provide further analyses and insights to understand them, and evaluate on large-scale datasets.
3

---

Published as a conference paper at ICLR 2023
4.1 PRE-TRAINING SCENARIOS THE CONTEXT OF FL
However, to begin with, we must consider if pre-training is feasible for FL. Namely, can we obtain pre-trained weights in FL applications? We consider two scenarios. First, the server has pre- trained weights or data for pre-training: second, the server has none of them. For instance, in areas like computer vision, many pre-trained models and large-scale datasets are publicly accessible. However, for data-scarce or -costly domains, or privacy-critica domains like medicine, these resources are often not publicly accessible. For the second scenario, we explore the use of synthetic data for pre-training.
Running example. Throughout the paper, we use visual recognition as the running example to study both scenarios, even though in reality we usually have pre-trained models for it. The rationales are two-folded. First, it makes our study coherent, i.c., reporting the results on the same tasks. Second, it allows us assess the gap between collecting real or creating synthetic data for pre- pre-training
Scenario one. We mainly use weights pre-trained on ImageNet (IK) (Russakovsky et al. [015 Deng al. 2009). For downstream FL tasks derived from ImageNet (e.g., Liny-ImageNet (Le & Yang. 2015 we use weights pre-trained on Places365 (Zhou et al. (017) to prevent data leakage. For both cases, pre-training is done by standard supervised training.
Scenario two. We use synthetic data for pre-training. We consider image generators like random generative models (Baradad et al.) (022)) and fractals (Kataoka et al. 2020) that are not tied to specific tasks. These generators, while producing non-realistic images, have been shown effective for pre-training. We briefly introduce fractals, as they achieve the best performance in our study.
A fractal image can be generated via affine Iterative Function System (IFS) (Barsleey 2014). An IFS generates a fractal by "drawing" points iteratively on a canvas. The point transition is governed by small set of 2 x 2 affine transformations (each with probability), denoted by S. Concretely, given the current point v, E IR?, the IFS randomly samples one affine transformation with replacement from the set S, and uses it transform e,, into the next point #n+1- This process continues until sufficient number of iterations is reached. The collection of points can then be used to draw a fractal image: by rendering each point as a binary or continuous value on black canvas. Since S controls the generation process and hence the geometry of the fractal, it can essentially be seen as the fractal's s ID, code, or class label; different codes would generate different fractals. Due to the randomness in the IFS, the same code S can create different but www.eeerriaallycossstteeeee fractals.
Using these properties of fractals, Kataoka et al. (2020) proposed sample different codes, create for each code number of images, and pre-train a model in supervised way. Anderson & Farrell (2022) proposed to create more complex images by painting fractals on one canvas. The resulting image thus has I out of J classes, and can be used to pre-train a model via multi-label loss.
In our study, we however found none of them effective for FL, perhaps due to their limitation on J. We thus propose a novel way for fractal pre-training, inspired by one critical insight infinitely many IFS codes and create fractals with highly diverse geometry. In the extreme, every image can be synthesized with an entirely different code.
Gode
Code:
Code
can in theory sample
(a) Fractal + Multi-label supervision
Medel
Preshuter
We propose to pre-train with such fractal images using contrastive or similarity learning (Chen et al.) 2020d Chen & He. 202T). The idea is treat every image from a different class and learn to repulse different im- ages away (i.e., negative pairs) or draw an image and its augmented version closer (i.c., positive pairs). We propose unique way to create positive pairs, which is to use IFS to create a pair of images based on the same codes. We argue that this leads to stronger and argumentation: not only do dif- ferent fractals from the same codes capture intra-class variation, but we also create a pair of images with different placements of the same fractals (more like object-level augmentation). We name our approach fractal pair similarity (FPS) (see Figure 2).
Model
Siinuartty
(b) Fractal SimSiam (Chen He 202T
Model
Predettr
IFS Coder
Natarad Warghts
IFS Code: S.
Vodil
Sianiority
Ganeration
(c) Fractal Pair Similarity (ours) SimSiam
Figure 2: Pre-training with fractals. (a) multi- label training: (b) similarity learning: (c) similar- ity learning with our fractal pair similarity (FPS).
4

---

Published as a conference paper at ICLR 2023
4.2 EXPERIMENTAL SETUP AND IMPLEMENTATION DETAILS
Data. We conduct the study using five
Table 1: Summary of datasets and setups.
visual recognition datasets: 100,
Dataset
|dCClass # Traiiinn Tess
#/ ClClients M Resolution
CIFAR- 100 (Krizhevsky et al. 2009). Tiny-
ImageNet (Le & Yang 2015). iNaturalist
CIFAR-
10/100
50K
10K
10
100
32
Tiny-ImageNN
200
100K
10K
10 - 1000
643
(Van Horn et al. 2018), and Cityscapes
1203
120K
36K
39/136
2242
(Cordts et al.) 2015). The first four are for
Cityscapes
19
3K
0.5K
18
768
image classification; the last for semantic
segmentation. Table I summarizes their statistics. (We also include NLP tasks such as sentiment analysis (Caldas et al. 2018) and language modeling (McMahan et al. 2017) in the Suppl.)
Non-IID splits. To simulate non-IlID conditions across clients, follow (Hsu et al. 2019) to partition the training set of CIFAR-10, CIFAR-100, and Tiny-ImageNet into M clients. To split the data of class c, we draw an M M-dimensional vector qe from Dirichlet(a). and assign data of class to client m proportionally to [o|m] The resulting clients have different numbers of total images and class distributions. The small the a is, the more severer the non-IID condition is.
For iNaturalist, a dataset for species recognition, we use the data proposed by (Hsu et al. 2020). which are split by Geo locations. There are two versions. GEO-108 10K (39 clients) and GEO-3K 3K (136 clients). For Cityscapes. a dataset of street scenes, we use the official training/validationn sets that contain 18/3 cities in Germany. We split the training data by cities to simulate a realistic scenario.
Metries. We report the averaged accuracy (%) for classification and the mIoU (%) for segmentation on the global test set. For brevity, we denote the test accuracy or mloU of the model @ by (0).
Models. We use ResNer20 (He et al., 2016a) for CIFAR-10/100, which suitable for a 32 x 32 resolution. We use ResNet18 8 for DeepLabV3+ (Chen et al. 2018) with a MobileNet-V V2 (Sandler et al., 2018) backbone.
and iNaturalist. For semantic segmentation, we use
Initialization. We compare random and pre-trained weights. For scenario one (subsection 4.1, we obtain pre-trained weights for R8 and MobileNet-V2 from PyTorch's 's and Places365's 's official sites. For ResNet20, we pre-train the weights using down-sampled, 32 x 32 ImageNet images.
For scenario two using synthetic images, we mainly consider fractals and apply our FPS approach, but will investigate other approaches in subsecction 4.5 We apply the scheme proposed by (Anderson & ( to sample IFS codes (each with 2 ~ transformations). For efficiency, we pre-sample a total of 100K IFS codes, and uniformly sample codes from them to generate each fractal image. We then follow [nderson & Farrell (2022) to color, resize, and flip the fractals. We then apply a similarity learning approach SimSiam (Chen & He 202T) for pre-training, due to its efficiency and effectiveness. We pre-train the weights for 100 epochs; each epoch has IM image pairs (or equivalently 2M images). See Fiiuree 2 for an illustration and see the Suppl. for details.
More specifically, for ResNet20, we set I = 2 and run the IFS for 1K iterations to render one 32 32 image. For ResNet18 and DeepLabV3+, we sample I uniformly from {2, 3, 4, 5} for each image to increase diversity, and run the IFS for 100K iterations render one 224 x 224 image.
Federated learning. We perform FEDAVG for 100 iterative rounds. Each round of local training takes 5 epoch We use the SGD optimizer with weight decay 1e-4 and a 0.9 momentum, except that on Deepl Vb+3+ we use an Adam (Kingma & Ba. 2015) optimizer. We apply the standard image pre- pre-processing and data augmentation (He et al. (2016a). We reserve 2% data of the training set as the validation set for hyperparameter tuning (c.g., for the learning rate"). We follow the literature (He jet al., (016b) to decay the learning rate by 0.1 every 30 rounds. We leave more details and the selected hyperparameters in the Suppl.
4.3 EMPIRICAL STUDY ON CIFAR- 10 AND TINY-IMAGENET
We first focus on CI and Timy-ImageNet, on which we create artificial splits of clients' data to simulate various non-lID conditions. Besides applying FEDAVG to train model 6μ in an FL way, we also aggregate clients' data and train the corresponding model dCL in a centralized learning way. This enables us to assess their accuracy gap Ac. A (fcr.) - A
We found this number quite stable for all experiments and freeze it throughout the paper.
We use the same learning rate within each dataset, which selected by FEDAVG without pre-training.
5

---

Published as a conference paper at ICLR 2023
Table 2: Test accuracy on CIFAR-10 (a= 0.3).
Table 3: Test accuracy on Tiny-ImageNet (a 0.3).
Initialization Federated
Centralized ACLFL.
Initialization Federated
|Centralized
ÁCL-FI
Random
82.7
91.7
9.0
Random
42.4
47.5
5.1
FPS
84.9 (+2.2)
92.0 (+0.3)
7.1
FPS
45.7 (+3.3)
49.1 (+1.6)
3,4
ImageNet
90.8 (+8.1)
93.3 (+1.6)
2.5
Places365
50.3 (+7.5)
52.9 (+2.8)
2.6
100
384
120
CIFAR-10
Real
70-
70
Random
50-
0.9
04
0.2
50-
0.2
0.4
0.5
1.8
1.0
14
Tiny-ImagoNet
Real
13
10
Random
10
0.6
(a) #Clients
(b) Din(aa)-oo--II
(c) Participation (%)
(d) Local epochs/round
Figure 3: Comparison of model initialization across different federated settings. The default setting is 10
clients, 0.3, 100% participation, and 5 local epochs, and we change one variable at time.
Pre-training improves FEDAVG and closes its gap to centralized learning. We first set 0.3 (a mild non-lID degree) and M = 10 (i.e., 10 clients), and study the case that all clients participate in every round. Table 2] and Table 3 summarize the results; each row is for different initialization. As shown in the column "Federated," pre-training consistently improves the accuracy of FEDAVG; the gain is highlighted by the magenta digits. This is not only by pre-training using large-scale real datasets like ImageNet and Places365, but also by pre-training using synthetic images.
We further investigate whether pre-training helps bridge the gap between FEDAVG and centralized learning. As shown in the column "Centralized," pre-training also improves centralized learning. However, by looking at the column "ACL-FL." which corresponds to the difference between "Federated" and "Centralized," we found that pre-training closes their gap. The gap on CIFAR-10 10 was 9.0 without pre-traim tre-training, and reduces to 2.5 with weights pre-trained on ImageNet. In other words, pre-training seems to bring more benefits to FL than to centralized learning.
Pre-training brings larger gains more challenging settings. We now consider different federated settings. This includes different num- bers of clients M, different non-IID degrees a, and different percentages of participating clients per round. We also consider different numbers of local training epochs per round, but keep the total local epochs in FEDAvG as 500. For each configuration, we conduct three times of experiments and report the mean and standard deviation.
CIFAR10
95
FedAAg
FedProx
FedDyn m
90
MOON
85
Figure 3 summarizes the results, in which we change one variable time upon the default setting: M = 10, a = 0.3, 100% participation, and 5 local epochs. Across all the settings, we see robust gains by pre- training, either using real or synthetic images. Importantly, when the setting becomes more challenging, e.g., smaller a for larger non-IID degrees or larger M for fewer data per client, the gain gets larger. This shows the value of pre-training addressing the challenge in FL.
80
55
Tiny-lmageNet
50
45
Pre-training is compatible with other FL algorithms. We now con- sider advanced FL methods like FEDPROX (Li et al. 2020b), FEDDyN (Acar et al.] 2021), and MOON (Li et al. 2021b). They were proposed to improve FEDAVG, and we want to investigate if pre-training could still improve them. Figure shows the results, using the default feder- ated setting. We found that both pre-training with real (right bars) and synthetic (middle bars) images consistently boost their accuracy. Interestingly, when pre- trainngn with real data is considered, all these FL methods, including FEDAVG, perform quite similarly. This suggests that FEDAVG is still strong FL approach if pre-trained models are available.
40
Random
FPS
Real
Figure 4: Pre-training for
different FL methods.

---

Published as a conference paper at ICLR 2023
Pre-training makes network sizes scale better. One challenge in FL is the difficulty to train deeper networks (Chen & Chao 2021). We investigate if pre-training mitigates this issue. On CIFAR-10 (a = 0.3), we study different network depths and widths based on ResNet20. As shown in Figure 5| with random initialization, deeper models have quite limited gains; wider models improve more. With pre-training (either FPS or real), both going deeper and wider have notable gains, suggesting that pre-training makes training larger models easier in FL.
ImageNet
92
FPS
90-
Random
Test Acc
Deeper:
86
Depahe(200 32, 44, 56)
x Wider (Panhher2tty Withhepd, x2, x3)
0.5
Network Size (M)
Figure 5: Deeper/wider ResNet.
4.4 EMPIRICAL STUDY ON LARGE-SCALE AND REAL DATA SPLITS
We now study pre-training for FL on large-scale datasets, iNaturalist 2017 (Van Horn & Perona 2017) and Cityscapes (Cordts et al. 2016). Both datasets provide geo-location information, and can be split in a realistic way to simulate location-specine clients. (Please see sebseccttiionn 4.2 for details.)
Table 4 summarizes the results on iNat-
Table 4: Test accuracy on iNaturalist-GEO (Hsu ct al. 2020
uralist 2017 (Van Horn & Perona (0177)
Init.
GEO-10K
GEO-3K
|Centralized Avg. ACI.F.
using the GEO-IOK (39 clients and 50%
clients per round) and GEO-3K (136
Random
20.9
12.2
48.5
32.0
FPS
27.8 (+6.9) 17.7 (45.5)
50.2 (+1.7)
27.5
clients and 20% clients per round) splits proposed by (Hsu et al. 2020).
ImageNet 46.6 (+25.7) 45.6 (433.4)
58.4 (49.9)
12.3
summarizes the semantic segmentation re- sults (mloU) on Cityscapes (Cordts et al., 2016). The training data split into 18 clients by cities. We consider full client participation at every round.
Table 3
Table 5: mloU (%) of segmentation on Cityscapes.
Init. Random 41.0
Federated
Centralized AAL.
57.2
16.2
FPS
64.9 (+23.9) 67.2 (+10.0)
2.3
In both tables, we see clear gaps between central- ized and federated learning (i.e., ÁCL-FL) on realis- tic non-IID splits, and pre-training with either synthetic (i.e., FPS) or real data notably reduces the gaps. Specifically, compared to random initialization, FPS brings encouraging gains to FEDAVG (> 5% on iNaturalist; > 20% on Cityscapes). The gain larger than that of centralized learning.
ImageNet 67.6 (+26.6)
71.1 (+13.9)
3.5
4.5 ON PRE-TRAINING WITH SYNTHETIC DATA
Table 6: Comparison on synthetic pre- training. Means of 3 runs are reported. Init.
In previous subsections, we show that even without available pre-trained models or real data for pre-training, we can resort to pre- pre-training with synthetic data and still achieve a notable gain for FL. Here, we provide more analyses and discussions.
|C10 Cio0/Tiny
Random
|74.4 51.4 |42.4
Fractal + Multi-label
|73.0 51.0 |40.9
StyleGAN + SimSiam
|79.2 53.0
|44.6
Comparison of synthetic pre-training methods. For frac- tals, compare the three methods in Figure All of them use the same setup in subsectionn 4.2] except for multi-label learning we choose J =1 1K IFS codes it gives the best validation accuracy. We also consider synthetic images generated by (the best version of) random StyleGAN (Baradad et al., 2021 Karras (et al. 2019). We use these methods initialize FEDAVG on CIFAR-IO/I00 (a 0.1, large non-llD) and Imay-ImageNet (a 0.3), with M = 10 clients and full participation. Results are in Table 6 Our approach FPS outperforms all the baselines. By taking deeper look, we found that pre-training with multi-label supervision does not outperform random initialization. We attribute this to both the limited diversity of fractals and the learning mechanism: we tried to increase but cannot improve due to poor convergence. Self-supervised learning, the contrary, can better learn from diverse fractals (i.e., each fractal a class). By taking the inner working of fractals into account to create grometrically-mmaningge positive pairs, our FPS unleashes the power of synthetic pre- pre-training.
Fractal + SimSiam
77.4 51.7
44.2
FPS (ours) + SimSiam 80.5 54.7
80.5
45.7
Privacy-critical domain. We conduct study on chest X-ray diagnosis (Kermany et al. 2018). Such medical image domain is privacy-critical, and has a large discrepancy from ImageNet images. Under the same setup as in subsection 4.3 (a = 0.3, M = 10, full participation). FEDAVG initialized with Handom/FPS/ImageNer achieves 68.9 /...9/74./// 779.8% test accuracy. Interestingly, synthetic pre- pre-training outperforms ImageNet pre- pre-training by 5.9%. This showcases when synthetic data can be practically useful in FL: when the problem domain is far from where accessible pre-trained models are trained.
7

---

Published as a conference paper at ICLR 2023
Randorn
Pre-trained
Random
Pre-trained
Tiny- Sey--lmaeeee
CIFAR10
30
Centralied
Contruestoeeeg acc. 47.5/42.8
Deetrile $3.0/503
20
20 20
20 20 30
35
20 40
50 Rounds
Figure Training dynamics of FEDAVG. Please see the text in /scctionn 6 for details.
5
FEDERATED SELF-PRE-TRAINING CAN IMPROVE FL
Table 7: Self-pre-
training (SP) for FL.
Seeing the benefit of pre-training, we investigate another way to obtain good representation initialize FL tasks, which to perform self-supervised learn- ing directly on clients' decentralized data. Self-supervised learning (Liu et al. 202Th has been shown powerful to obtain good representations, which some- times even outperform those obtained by supervised learning (Ericsson et al.
Init.
SP Ci0
x 74,4142.4
Random
/ 79.8|43.9
X 79.9145.7
FPS
/ 82.2
2021). Chen et al. (2020c) further showed that a two-stage self self-pre-training (SP) procedure self-supervised pre-training, followed by supervised learn-
X 87.9150.3
Real
88.0
ing, on the same data - can outperform pure supervised learning.
We explore such an idea in FL. Starting from the random weights (or even pre-trained weights), first ignore the labels of clients' data and apply a state-of-the-ar federated self-supervised learning algorithm (Lubana et al.) 2022) (for 50 rounds), followed by standard FEDAVG. We use the same setup as in subsection 4.3 for CIFAR- 10 (a 0.1) and Tiny-ImageNer (a = 0.3), and summarize the results Table 7| The SP procedure can significantly improve FEDAVG initialized with random weights, without external data. (We confirmed the gain is not merely from that in total we perform more rounds Interestingly, the SP procedure can also improve upon pre-trained weights. These results further demonstrate the importance of initialization for FEDAYG. (Sce the Suppl. for more discussions.)
of FL. We extended FEDAVG without SP for 50 rounds but did not
improvement.)
AN ATTEMPT TO UNDERSTAND THE EFFECTS, AND CONCLUSION
We take a deeper look at pre-training (using real data) for Fl. and provide further analyses.
Preparation and notation. We first identify factors that may affect FEDAVG's accuracy along its training process. Let us denote by Dust {(x, 31 m
Nue the global test data, by Lesa the test loss,
and by Aresa (0) the test accuracy. Following Equation 2 we decompose Avea heea(f (0()) after round by
M
M
D ml
D m
A (0
Ares (019)
Aust (o Ause (0())
(
(0t)) m
(3)
[D]
The first term is the initial test accuracy round t; the second (4 the average gain by Local training: the third (A
(1) is the gain by Global aggregation. A negative A indicates that local models after local training (at round have somewhat "forgotten" " what the global model (t--1) has learned (Kirkpatrick et al. 2017). Namely, the local models drift away from the global model.
Analysis the training dynamics. We use the same setup in musubeection 4.3 on CIFAR- 10 and Iiny-ImaeNNet: M = 10, 0.3. Figure 6)sumaaiiess the values defined in Equation 3 For each combination (dataset pre-training or not), we show the test accuracy using the global model g(t)
(x). We also show the averaged test accuracy using each local model (% The red and green arrows indicate the gain by local training (A (()) and global aggregation (A respectively. For brevity, we only draw the first 50 rounds but the final accuracy is by 100 rounds.
We have the following observations. First, pre- pre-training seems to not alleviate local model drifting (Li et al. 2020b Karimireddy et al.) 2020b). Both FEDAVG with and without pre-training have notable negative A which can be seen from the slanting line segments: segments with negative slopes (i.e., of round is lower than x of round t 1) indicate drifting. Second, pre-training seems to have a larger global aggregation gain (() (vertical segments from
x), especially in early rounds.
8

---

Published as a conference paper at ICLR 2023
Round
CIFAR10
50
Round 10
Tiny-ImageNet
Round 50
Random
0
Random
Pre-trained
LaG
Rounds
Figure 8: We show the lest loss by FEDAVG's global model (curve) and the 95% confidence interval the test losses by the sampled convex combinations (shaded area). We also show the histograms of losses at = 10, 50.
Analysis global aggregation. We conduct a further analysis on global aggregation. In FEDAVG (Equation 2p. global aggregation is by a simple weight average, using local data sizes as coefficients. According (Karimireddy al. 2020b), this simple average may gradually deviate from the solution of Equation I juder non-lID conditions, and ultimately lead to a drastic accuracy drop. Here, we analyze if pre-training can alleviate this issue. As is unlikely to find a unique minimum ofuEqutioon I to calculate the deviation, we propose an alternative way to quantify the robustness of aggregation.
Our idea is inspired by Chen & Chao (202T), who showed that the coefficients used by FEDAVG may not optimally combine the local models. This motivates us to search for the optimal convex aggregation and calculate its accuracy gap against the simple average. The smaller the gap is, the more robust the global aggregation is. We define the optimal convex aggregation follows:
M
M
(0(1)=
Σ m=
((0)
where {Am = arg max (Am 20:E Am Am= Ateat
Am
(4)
m
m
That is, we search for À [A AM] in
CIFAR10
ImnyeemageNet
the (M- - -)-simplex that maximizes Aust. We apply SGD to find {Am, }. (See the Suppl.)
100
55
50
Figure 7 shows the curves of Aist (0()) and A (0+))) For ( we replace the simple weight average by the optimal convex aggregation throughout the entire FEDAVG: at the beginning of each round, we send the opti- mal convex aggregation back to clients for their local training. It can be seen that A outperforms (O(t)) The gap larger for FEDAVG without pre-training than with pre- training. Namely, for FEDAVG with pre-training. the simple weight average has much closer accuracy the optimal convex aggregation.
Test Acc
Test Acc
45
40
70
Random Pre-trained
35
60
30
40
Rounds
Rounds
Figure 7: We show for each dataset (and pre-training not) the test accuracy using the global model (()) (solid) and the optimal convex aggregation (x(() (dashed).
Analysis on the loss surface. To see why pre- pre- tr-tariinig leads to robust aggregation, we investigate
the variation of Lrest ( mm Am ( across different on the simplex. We sample À for 300 times, construct the global models and calculate the test losses, and compute the 95% confidence interval. As shown Figure FEDAVG with pre- pre-training has smaller interval: i.e., a lower- vower-variance loss surface in aggregation. This helps explain why it has smaller gap between Areal (0(1) and Area (0)))).
We further visualize the loss landscapes (Li et al. 2018). We use the same random (or pre-trained) weights to initialize FEDAVG on CIFAR-I0 (M 10), for different a and for an IID condition. We then gather the final global models and project them onto the loss landscape of L, i.e., the global em- pirical risk. We found that pre-training enables the global models under different data conditions (in- cluding the IID one) to converge to the same loss basin (with lower loss). In contrast, without pre- pre-training, the global models converge to isolated, and often poor loss basins. This offers another explanation of why pre-training improves FL.
(a) Random
(b) Pre-trained
Figure 9: Visualizations of the loss landscape.
We conduct the very first systematic study on pre- me--rraiing for federated learning (FL) to explore rarely studied aspect: initialization. We show that pre-training largely bridges the gap between FL and centralized learning. We make an attempt to understand the underlying effects and reveal several new insights into Fl., potentially opening up future research directions.
Conclusion.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research is supported in part by grants from the National Science Foundation (II7-2107077, OAC-21 18240, and OAC-21 1260)), the OSU GI Development funds, and Cisco Systems, Inc. We are thankful for the generous support of the computational resources by the Ohio Supercomputer Center and AWS Cloud Credits for Research. We thank all the feedback from the review committee and have incorporated them.
REFERENCES
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas, Matthew Mattina, Paul Whatmough, and Venkatesh Saligrama. Federated learning based on dynamic regularization. In ICLR, 2021.
Connor Anderson and Ryan Farrell. Improving fractal pre-training. In WACV, pp. 1300-1309, 2022.
Manel Baradad, Jonas Wulff, Tongzhou Wang, Phillip Isola, and Antonio Torralba. Learning to see by looking noise. NeurIPS, 2021.
Michael F Barnsley. Fractals everywhere. Academic press, 2014.
Monik Raj Behera, Sudhir Upadhyay, Suresh Shetry, Sudha Priyadarshini, Palka Patel, and Ker Farn Lee. Fedsyn: Synthetic data generation using federated learning. arXiv preprint arXiv:2203.05331, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, volume 33, pp. 1877-1901, 2020.
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:1812.0100 2018.
Hong-You You Chen and Wei-Lun Chao. Fedbe: Making bayesian model ensemble applicable federated learning. In ICLR, 2021.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training fine-tuning. In CVPR, pp. 699-708, 2020a.
Ting Chen, Simon Korbllihh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In JCML, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. In NeurIPS, 2020c.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0429)) 2020d.
Gary Cheng, Karan Chadha, and John Duchi, Fine-tuning is fine in federated learning. arXiv preprint arXiv:2 000:2/08.000 073 13, 2021.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset. In CVPR Workshop The Future of Datasets in Vision, 2015.
10

---

Published as a conference paper at ICLR 2023
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pp. 3213-3223, 2016.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint wrXiv:/810.04805, 2018.
Josip Djolonga, Jessica Yung. Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. CVPR, pp. 16458-16468 2021.
Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 201-208. JMLR Workshop and Conference Proceedings, 2010.
Linus Ericsson, Henry Gouk, and Timothy M Hospedales. How well do self-supervised models transfer? CVPR, pp- 5414-5423, 2021.
Amin Fazel, Wei Yang. Yulan Liu, Roberto Barra-Chicote, Yixiong Meng, Roland Maas, and Jasha Droppo. Synthasr: Unlocking synthetic data for speech recognition. In Interspeech 2021, 2021.
Jhair Gallardo, Tyler L Hayes, and Christopher Kanan. Self-supervised training enhances online continual learning. In BMVC, 2021.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.01988 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.
Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint /910, 14425, 2019.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of bert. arXiv preprint (eXvv:190.005200 2019.
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. In NeurIPS, 2020a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016b.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918-4927, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020b.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In ICML., pp. 2712-2721. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2023
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. Pretrained transformers improve -ittof--istrbbution robustness. arXiv preprint arXiv:2004.06100, 2020.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06:3: 2019.
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Federated visual classification with real- world data distribution, In ECCV, 2020.
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. OmniTab: Pretraining with natural and synthetic data for few-shot table-based question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, July 2022.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019.
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri. Sashank Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.0300 2020a.
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020b.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 4401-4410, 2019.
Hirokatsu Kataoka, Kazushige Okayasu, Asato Matsumoto, Eisuke Yamagata, Ryosuke Yamada, Nakamasa Inoue, Akio Nakamura, and Yutaka Satoh. Pre-training without natural images. In ACCV, 2020.
Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell, 122 1131, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska-Barwinske Barwinskw, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. ECCV, pp. 491-507. Springer, 2020.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In CVPR, pp. 2661-2671, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computation and communication-effiieee personalized federated learning via heterogeneous masking. In Proceedings of the 19th ACM Conference Embedded Networked Sensor Systems, pp. 42-55, 2021a.
Hao Li. Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances neural information processing systems, 31, 2018.
12

---

Published as a conference paper at ICLR 2023
Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10722, 2021b.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smithy, Feddane: A federated newton-type method. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, 2019.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3):50-00) 60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In MLSys, 2020b.
Xiang Li, Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg non-iid data. In ICLR, 2020c.
Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. Fednlp: Benchmarking federated learning methods for natural language processing tasks. In NAACL, 2022.
Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. NeurIPS, 2020.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021.
Ekdeep Singh Lubana, Chi Ian Tang, Fahim Kawsar, Robert Dick, and Akhil Mathur. Orchestra: Unsupervised federated learning via globally consistent clustering, In ICML, 2022.
Benoit B Mandelbrot and Benoit B Mandelbrot. The fractal geometry of nature, volume 1. WH freeman New York, 1982.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communicationn-fficceenn learning of deep networks from decentralized data. In AISTATS, 2017.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. An empirical investigation of the role of pre-training in lifelong learning. arXiv preprint arXiv:2112.09153, 2021.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer lcarning? In NeurIPS, 2020.
John Nguyen, Kshitiz Malik, Maziar Sanjabi, and Michael Rabbat. Where to begin? exploring the impact of pre-training and initialization in federated learning. arXiv preprint arXiv:2206. 15387, 2022.
Liangqiong Qu, Yuyin Zhou, Paul Pu Liang, Yingda Xia, Feifei Wang, Li Fei-Fei, Ehsan Adeli, and Daniel Rubin. Rethinking architecture design for tackling data heterogeneity in federated learning. arXiv preprint arXiv:2106.060// 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. In ICLR, 2021.
Colorado J Reed. Sean Metzger, Aravind Srinivas, Trevor Darrell, and Kurt Keutzer. Selfaugment: Automatic augmentation policies for self sulf-super learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2674-2683 2021.
Colorado J Reed, Xiangyu Yue, Ani Nrusimha, Sayna Ebrahimi, Vivek Vijaykumar, Richard Mao, Bo Li, Shanghang Zhang, Devin Guillory, Sean Metzger, et al. Self-supervised pretraining improves self-supervised pretraining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2584 2594, 2022.
13

---

Published as a conference paper at ICLR 2023
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-25) 252, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chigh-hhhhhhh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Sebastian U Stich. Local sgd converges fast and communicates little. In ICLR, 2019.
Joel Stremmel and Arjun Singh. Pretraining federated text models for next word prediction. In Future of Information and Communication Conference, pp. 477-488, 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. In ICCV. pp. 843-852, 2017.
Yue Tan, Guodong Long, Jie Ma, Lu Liu, Tianyi Zhou, and Jing Jiang. Federated learning from pre-trained models: A contrastive learning approach. arXiv preprint arXiv:2209. 10083, 2022.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? NeurIPS, 2020.
Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint arXiv: /tti::100000 1709.01450 2017.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. ICLR, 2020a.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In NeurIPS, 2020b.
Orion Weller, Mare Marone, Vladimir Braverman, Dawn Lawrie, and Benjamin Van Durme. Pre- trained models for multilingual federated learning. arXiv preprint arXiv:2206.02291, 2022.
Yuhuai Wu, Felix Li, and Percy Liang. Insights into pre-training via simpler synthetic tasks. arXiv preprint ar0iv:2206. 10139, 2022.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In CVPR, pp. 3733-3742, 2018
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. Advances in neural information processing systems, 33:19290-19999 2020.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xinet: Generalized autoregressive pretraining for language understanding. NeurlPS, 32, 2019.
Xin Yao, Tianchi Huang, Rui-Xiao Zhang, Ruiyu Li, and Lifeng Sun. Federated learning with unbiased gradient aggregation and controllable meta updating. arXiv preprint arXiv: 2019.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Trong Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In ICMI., 2019.
Yue Zhao, Meng Li. Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint wwXiv:/80600056. 2018.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralb. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
14

---

Published as a conference paper at ICLR 2023
Fan Zhou and Guojing Cong. On the convergence properties of k-step averaging stochastic gradient descent algorithm for nonconvex optimization. arXiv preprint 1781v:1708.0102,, 2017.
Yanlin Zhou, George Pu, Xiyao Ma. Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning, arXiv preprint arXiv:2009.0799 0.9.9999,, 2020.
Zirui Zhu and Lifeng Sun. Initialize with mask: For more efficient federated learning. In MultiMedia Modeling: 27th International Conference, MMM 2021, Prague, Czech Republic, June 22-24, 2021, Proceedings, Part 27, pp. 111-120. Springer, 2021.
15

---

Published as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIAL
We provide details omitted in the main paper.
Appendix A
additional details of fractals pre-training (cf. |section 4 of the main paper).
Appendix B
details of experimental setups (cf. section 6 and Section 4 the main paper).
Appendix additional experimental results and analysis (cf. seccionn 6f the main paper).
Appendix
additional discussions.
A ADDITIONAL DETAILS OF FRACTAL PRE-TRAINING
(a) 2 codes
(b) 3 codes
(c) 5 codes
Figure 10: Examples of image pairs of FPS. We generate three image pairs with FPS using /
2, 3, and IFS codes. The resulting pairs are shown in (a), (b), and (c), which contain 2, 3, and fractals in each image, respectively. In each pair, the two fractals rendered from the same IFS code reflect intra-class variations and different placements.
A.1 A COMPLETE INTRODUCTION OF OUR FRACTAL PAIR SIMILARITY (FPS)
In section 4 of the main paper, only provide condensed introduction of our FPS algorithm due to the page limits. Now, we give a complete version that elaborates more details.
A.1.1 BACKGROUND: SUPERVISED FRACTAL TRAINIGG
Fractal generation. A fractal image can be rendered via an affine Iterative Function System (IFS)
Kataoka al. (2020); Barnsley (2014). The IFS rendering process can be thought of as "drawing" points iteratively on a canvas. The point transition is governed by set of 2 x 2 affine transformations, which we call an IFS code S:
S= ( E 2x2 by E E PP E {R) kkm1
(5)
Here, an (A,, bx) pair specifies an affine transformation and ps: is the corresponding probability, ie.. Σx Pk = 1 and pę 0, vk = [K]. Concretely, given S, an IFS generates image follows. Starting from an initial point E R it repeatedly samples one transformation & with replacement according to the probability Pa. and performs Va+1 = A + by to arrive at the next point. This stochastic process continues until a sufficient number of iterations is reached. The traveled points (vo. Un,," ... } are then used to synthesize fractal: by rendering each point as a binary or continuous value black canvas. Due the randomness in the IFS, one code can create different but geometrically similar fractals.
Supervised pre-training. While fractal images do look real, they are diverse and capture the geometric properties of elements found in nature (Mandelbrot & Mandelbrot 1982), thus suitable for pre-training and facilitating downstream tasks (Kataoka et al.) 2020). According Baradad et al.) (2021), diversity and naturalism (i.e., capturing certain structural properties of real data) two key properties that make for good synthetic data for training vision systems. Pre-training with fractal images thus allows the model to capture features, e.g., structured properties and the diversity of patterns, that can be useful for recognizing real data.
Since the IFS code controls the generation process and hence the geometry of the fractal, it can essentially be seen as the fractal's ID or class label. (Kataoka et al. 2020). the authors proposed to sample J different codes (S, ) and create for each code number of images construct J-class
16

---

Published as a conference paper at ICLR 2023
classification dataset. This synthetic labeled dataset is then used to pre-train neural network via multi-class loss (e.g., cross-entropy). The following-up up work by Aynderson & Farrell (2022) proposed to create more complex images by painting multiple (denoted by 1) fractals resulting image thus has I out of J labels, on which a multi-label loss is more suitable for supervised pre-training.
canvas. The
A.1.2 OUR APPROACH: FRACTAL PAIR SIMILARITY (FPS)
We propose a novel way for fractal pre-training, inspired by one critical insight - we can in theory sample infinitely many IFS codes and create fractals with highly diverse geometric properties. That is, instead of creating a J-class dataset that limits the diversity of codes (e.g., J =KK) but focuses more on intra-class variation, we propose to trade the latter for the former.
We propose to sample new set of codes and create an image that contains multiple fractals on the fly. Namely, for each image, we sample a small set of codes [S generate corresponding fractals, and composite them into one image. The resulting dataset will have all its images from different classes (i.e., different sets of (Si) in theory.
Analogy to self-supervised learning. How can we pre-train a neural network using a dataset whose images are all of the different class labels? Here, we propose to view the dataset as essentially "unlabeled" and draw an analogy to self-supervised learning Liu et al. (202T), especially those based
on contrastive learning (Wu et al. 22118; He et al.) [020bb Chen et al. 2020d bjc Tian et al. 2020) or similarity learning Grill et al. 2020); Chen & He (2021). The core idea of these approaches is to treat every image from different class and learn to either repulse different images away (i.e., negative pairs) or draw an image and an augmented version of it closer (i.e., positive pairs). This line of approaches is effective that they can even outperform supervised pre-training in many tasks Ericsson et al. (202T). We note that while Barrddad el al. (2027) has applied self-supervised learning to fractals, the molivation is very different from ours: the authors directly used the supervised dataset created by Kataoka et al.) (2020) but ignored the labels.
Fractal Pair Similarity (FPS). Conventionally, to employ contrastive learning or similarity learning, one must perform data augmentation such that single image becomes positive pair. Common methods are image-level scale/color jittering, flips, crops, RandAugment Cubuk et al. (2020), Self- ReeeeentteReeeeeeeeeeeeeee
Augment Reed et al.) (,021), etc. Here, we propose to exploit one unique way for fractals (or broadly, synthetic data). which is to use the IFS to create a pair of images based on the same set of codes {S.)=z. We argue that this method can create stronger and more physicalyy-meaninghh lation: not only do different fractals from the same codes capture intra-class variation, but we also can create a pair of images with different placements of the same I fractals (more like object-level augmentation). Moreover, this method can easily be compatible with commonly used augmentation.
argumen-
Implementation. We detail the generation of a positive image pair in FPS as follows. First, we randomly sample/ I distinct IFS codes. Then, each IFS code is used to produce two fractal shapes applied with random coloring, resizing, and flipping. Finally, we obtain two sets of fractals; each set contains/ T distinct fractal shapes that can be pasted on a black canvas to generate one fractal image. The resulting two fractal images are further applied with image augmentations, such as random resized cropping, color jittering, flipping, etc. We provide more examples of image pairs generated by our FPS in Figure 10 In each pair, the two fractals generated from the same IFS code show intra-class variations, in terms of shapes and colors, and are placed at random locations.
We then apply self-supervised learning algorithms for pre- training. We use the IFS code sampling scheme proposed in Anderson & Farrell (2022) (with K E {2, 3, 43) and similarly apply scale jittering on each IFS code betore we use it to render fractals. For efficiency, we pre-sample a total of 100K IFS codes in advance and uniformly sample codes from them to generate a pair of images.
Our PyTorch code implementation is provided at hhtpps://ge http:///gituuu..cooooo//.tattttt Pre training
A.2 SELF-SUPERVISED LEARNING APPROACHESS
After generating fractal image pairs with FPS, we pre-train models by applying two self-supervised learning approaches, SimSiam Chen & He (22TI) (similarity- based) and MoCo- V2 Chen et al. (2020d) (contrastive-based), We briefly review them in the following.
17

---

Published as a conference paper at ICLR 2023
Table 8: Summary of datasets and setups.
Dataset
Task
#Class# Training # Test/Valid
Split
#Clients Reso.
Networks
ResNet-
CIFAR-10/100
Classification 10/100
50K
10K
Dirichlet
10
32
{20, 32, 44, 56)
Tiny-ImageNet Classification
200
100K
10K
Dirichlet
10/100
64
ResNet-18
iNaturalist-200 Classification
1203
120K
36K
GEO-IOK/3K
39/136
224
ResNet-18
DeepLab + MobileNet-V2
Cityscapes
Segmentation
19
3K
0.5K
Cities
18
768
SimSiam. Given positive image pair (21, Ø2). we process them by an encoder network f to extract their features. A prediction MLP head h is then applied to transform the features of one image to match the features of the other. Let p1 = (z(((z)) )) and 22 = / (x2). The objective (to learn f and
h) is minimize the negative cosine similarity between them:
P1
22
D(p x2) =
(6)
1|
where || ||2 is the 1o-nomm. Since the relation between æ1 and 22 symmetric, the final loss can be written as follows:
1 C -= - D(p.. 22) + D(p2, 21).
MoCo-V2. Similar to SimSiam Chen & He (2021) MoCo- V2 Chen et al. (2020d) also aims to maximize the similarity between features extracted from a positive image pair (æ1, x2). The main difference is that MoCo-V2 adopts contrastive loss, which also takes negative pairs into account. Following the naming in MoCo-V2 Chen et al. (202dd). let x = x1 be the query and œ = 22 be the positive key. We also have negative images/keys (#f, ar...... ) in the mini-batch. We define (2²), k4 = (x and k; = f& ) where fg is the encoder for query images and fx is the encoder for keys. The objective function for MoCo-V2 (to learn fa and f&) is written follows:
exp(q k//T)
L = - log
(8)
Σ
ii exp(q -k/T)"
where is temperature hyper-parament parameter. Besides the contrastive loss, MoCo- V2 maintains dictionary to store and reuse features keys from previous mini-batches, thereby making the negative keys not limited the current mini-batch. Finally, to enforce stability during training, momentum update is applied on the key encoder fk-
In subsection 4.5) and Table 6 of the main paper, we pre-train ResNet-18 and ResNet-20 for 100 epochs using SimSiam Chen & He (2021) and MoCo-V2 CCen et al. (2020d). Specifically, these neural network architectures are used for SimSiam and f, and fn in MoCo-V2. Each epoch consists of IM image pairs, which are generated on-the-fly, for FPS. For comparison to StyleGAN in Table 6. we use the pre-gen pre-generated StyleGAN dataset provided in Baradad al. (2021). which has 1.3M images. After pre- vre-training we keep f, (discard fi)) from MoCo-Va V2, following Chen et al. (202dd), and keep (discard h) from SimSiam, following Chen He (202T).
For SimSiam, we use the SGD optimizer with learning rate 0.05, momentum 0.9, weight decay 1e-4, and batch size 256. For MoCo- V2, we use the SGD optimizer with learning rate 0.03, momentum 0.9, weight decay le-4, and batch size 256. The dictionary size is set to 65, 536. For both SimSiam and MoCo-V2, the learning rate decay follows the cosine schedule (Loshchilov al., 2017).
For other experiments besides Table 6 we mainly use SimSiam for FPS. We use the same training setup as mentioned above (c.g.. g.. 100 epochs).
B FL EXPERIMENT DETAILS
B. DATASETS, FL SETTINGS, AND HYPERPARAMETERS
We train FEDAVG for 100 rounds, with 5 local epochs and weight decay 1e-4. Learning rates are decayed by 0.1 every 30 rounds. Besides that, summarize the training hyperparameters for each
18

---

Published as a conference paper at ICLR 2023
Table 9: Default FL. settings and training hyperparameters in the main paper.
Dataset
Non-IID
Sampling
Optimizer
Learning rate Batch size
CIFAR-10/100
DDrichlett 0.3)) 100%
SGD + 0.9 momentum
0.01
32
Tiny-ImageNa
Dirichlet(0.33)
100%/10%% SGD + 0.9 momentum
0.01
32
NNtuuralit-22017
GEO-10K, GEO-3K
50%
SGD 0.9 momentum
0.1
128
Cityscapes
Cities
100%
Adam
0.001
16
of the federated experiments included in the main paper in Table We always reserve 2% data of the training set as the validation set for hyperparameter searching for finalizing the setups.
For pre-processing, we generally follow the standard practice which normalizes the images and applies some augmentations. CIFAR-10/100 images are padded 2 pixels on each side, randomly flipped horizontally, and then randomly cropped back to 32 x 32. For the other datasets with larger resolutions, we simply randomly cropped to the desired sizes and flipped horizontally following the official Py Torch ImageNet training script.
For the Cityscapes dataset, 768 x 768 and resized to 2048 x 1024 in testing.
output stride 16. In training, the images are randomly cropped to
To further understand the effects of hyperparameters, provide more analysis Tiny-ImageNet in subsection C.4
B.2 OPTIMAL CONVEX AGGREGATION
We provide more details for learning the optimal convex experiment in /section 6 To find the optimal combinations for averaging clients' weights, we optimize À using the SGD optimizer, with learning rate le 5 for 20 epochs (batch size 32) on the global test set. The vector is ε1 normalized and each entry is constrained to be non non-negen neg-tieae (which can be done with softmax function in PyTorch) ensure the combinations are convex. Since the optimization problem with several different vectors, including uniform initialization, and return the best (in terms of the test accuracy) as {Ma,) for Equation
convex, we initialize À
B.3 FEDERATED SELF- PRE- TRAINING
In section 5 consider self-pre- pre- traintnainigg by rumning federated self-supervised learning algorithm on the same decentralized training data before starting supervised FEDAVG. We adopt the state- of-the-art algorithm [uubana al. (2022), which is based clustering. We follow their stateless configurations and run it with local epochs each round, a 0.003 learning rate, and 32 batch size, for 50 rounds. We set 32/8 local/global clusters for CIFAR-IO 10 and 256/32 for local/giobal clusters for Tiny-ImageNet.
We attribute the improvement by SP to several findings from different but related contexts. Gallardo let al. (2021); Yang & Xu 2020): Liu et al. (202I) showed that sulf-super learning is more robust learning representations from class-imbl insaliinblnneed and distribution-diicrepan data; Reed
(2022)
showed that the SP procedure helps adapt the representations to downstream tasks. With that being said, federated self-supervised learning is itself an open problem and deserves more attention. (In
preliminary trials, the algorithm by Lubana et al. (2022) can hardly scale up to iNaturalist and Cityscapes. The validation accuracy by k-nearest neighbors hardly improves.)
C ADDITIONAL EXPERIMENTS AND ANALYSES
C.1 SCOPE AND MORE EXPERIMENTS IN NLP
In this paper, we focus on computer vision (CV). We respectfully think this should not degrade our contributions. First. CV is a big area. Second, while many federated learning (FL) works focus on CV, most of them merely studied classification using simple datasets like CIFAR. We go beyond them by experimenting with iNaturalist and Cityscapes (for segmentation), which are realistic and remain challenging even in centralized learning (CL.). We believe focused study in CV and
19

---

Published as a conference paper at ICLR 2023
Table 10: Sent140 accuracy (100 rounds; local epoch= 10% clients per round).
Init. Setting FL
CL
ACLeL
Random
75.4
80.5
5.1
Pre-trained
83.4 (+8.0)
85.0 (+4.5)
1.6
Table 11: Shakespear next-character prediction accuracy (30 rounds; local epoch= 10, 10% clients per round, 606 clients in total).
Init. / Settting
FL
CL
|ACIF
Random
46.4 52.6 (+6.2)
59.1
12.7
Pre-trained
59.6 (+0.6)
7.0
Random
Pre-trained
80
Sent1 40
60
40
20
10
20
30
10
20
30
60
60
Shakespeare
50
Centralized
40
40
30
20
20:
10:
10
10
20
30
0
15 Rounds
10
20
30
Rounds
Figure 11: Dynamics FL on Sentl 40 and Shakespeare. We show the test accuracy of the global model (())
(x), and the averaged test accuracy using each local model ( (+).
encouraging results these two datasets using either real synthetic pre-training valuable the FL. community
That being said, here provide experiments natural language processing (NLP) verify that our observations are consistent with that on CV tasks, First, we conduct sentiment analysis experiment on large-scale federated Sent140 dataset (Caldas al., 2019), which has 660K clients and 1.6M samples. We use a pre-trained DistilBERT (Sanh et al., 2019). Second, we experiment with another popular FL. NLP dataset Shakespeare next-character prediction task proposed in McMahan al. (017). ). We use version provided by (Caldas et al., 2019) which contains 606 clients, and use Penn Treebank dataset (Marcinkiewiezz et al., 1994) for pre-training an LSTM of two layers of 256 units each.
Table 10 and Table 11 show the results for the Sent and Shakespeare datasets, respectively. We also similar trend in Figure 1I pre-training helps more in FL than in CL and largely closes their gap, even if the local model drifts are not alleviated.
C.2 MORE ANALYSIS GLOBAL AGGREGATION
In section 6
discuss how pre-training leads to more robust aggregation where the loss variance smaller when we sample the convex combinations of local models. In /Figure 8) we provide CIFAR-10 and Image due the space limit. In Figure 12 and Figure 13] we also include the
20

---

Published as a conference paper at ICLR 2023
Random
Pre-trained
Random
Pre-trained
100
Cityscapes INaturalist. Gellst-GEO---
CIFAR10 10
Fentrn 43.5/122
Centra leskht S8A/45.
20
40
Tiny Imageeettt
12 2. 13
Centralzed
Cantrabhec//eeee Aes: 71.1675
76
Centrnlans/eedaa Acc: 52.97503
Rounds
Rounds
Rounds
Rounds
Figure 12: Training dynamics of FEDAYG. For each combination (dataset pre-training or not), we show
the test accuracy/miou using the global model g (x), and the averaged test accuracy/mloU using each local model 0h (+). The red and green arrows indicate the gain local training (A (A,)) Vglobal aggregation (A (
For brevity, we only draw the first 50 rounds but the final accuracy/mloU are after 100 rounds.
-Naturalist-GE--3K
Cityscapes
1.0
Test Loss
0.5
Random
Test
2
0.0
1.0
4-
Test Loss
Pre-trained
0.5
Test
2
10
20
30
40
10
20
30
40
Figure 13: Pre-training leads to a lower-variance loss surface (cf. Figure 8). We show the test loss by the
FEDAVG's global model g(o) (curve) and the 95% confidence interval the test losses by the sampled convex combinations (shaded area) on INaturalist-GEO and Cityscapes.
Naturalist-GEOO33 and Cityscapes. We have consistent finding using pre-training does not reduce local model drift but does lead to a lower-variance loss surface.
C.3 MORE COMPARISONS ON SYNTHETIC PRE-TRAINING
In (susecction 4.5, we compare synthetic pre-training methods. We also consider the images generated by (the best version of) randomly-inittalized StyleGAN Baradad et al. (2021): Karras al. (2019). Due to the space limit, we only list some of the comparisons. Here we provide the complete results in Table 12] Across datasets and superviee algorithms, our FPS consistently outperforms the baselines. Since overall our FPS + SimSiam provides the strongest performance, we focus on for the main paper. Here we also include the FPS MoCo- V2 algorithm and observe similar conclusion.
C.4 DISCUSSIONS ON DIFFERENT FEDERATED SETTINGS
To understand how the FL settings affect the observations of pre-training, further conduct studies on CIFAR-I0 10 and Tmny-ImageNet Figure of the main text. We focus on the setting in the main text (i.e., 10 clients, Dir(O.3), 100% participation, local epoch= 5) but change variable of settings such as learning rate scheduling (in Figure and Figure 15). the number clients, non-IID degree, participation rate, and the number of local epochs.
For different learning rate scheduling, we observe similar trends: pre-training does not alleviate client shift much but the test accuracy of the global model after aggregating the local models is higher.
21

---

Published as a conference paper at ICLR 2023
Table 12: Comparison synthetic pre-training methods in section 5
Means of 3 runs reported.
Init. / Datasct
C10
C100
Tiny-ImgNet
Random
74.4=0.42
51.4+0.31
42.4++.20
Fractal + Multi-label
73.0=0.43 51.0.10441
40.9:10.22
StyleGAN + SimSiam
79.2=0.26 53.0.0..5
14.6:10.16
Fractal + SimSiam
77.4=0.30 51.7++0.8
44.2++0.20
FPS (ours) SimSiam
80.5:0.25
54.7+00.1
45.7++0.14
StyleGAN + Mo
75.3=0.11 53.5++0.12 12
45.8++0.4
Fractal + McCo-V2
73.5=0.29
53.3+0.24
44.5+0.17
FPS (ours) MoCo-V2
79.9=0.15
53.9+0.16
46.1+0.18
ImageNet/ Places365
87.9=0.11 02.2++009)
50.3++0.88
Decay by 0.95
every round
80
Test Acc
0.2
20
20
30
40
50
10
20
30
40
50
0.8
80
No decay
Test Acc
Ver
0.5
40
0.4
10
20
40
50
10
20
30
40
50
Rounds
Rounds
Figure 14: CIFAR-10 (10 clients, Dir(O.3), 100% participation, local epoch= 5) with different learning rate decay strategies. Left: we show Global model (x) accuracy and the averaged accuracy using each local model (+). Right: the test loss variance of 100 sampled coefficients (the width of the shaded area in Figure 8) to combine the local models in each round.
In the main paper, we focus on the schedule that decays the learning rate by 0.1 every 30 rounds given its better performance compared to the two (decay by 0.95 every round and decays) here. In Figure 14 and Figure 15] we also monitor the variance of the test loss of the global models that different coefficients (I00 random samples) aggregate the local models, i.e., the shaded area in Figuure 8 We found the pre-trained ones have lower loss variance quite consistent across different learning rate scheduling and our analyses in section
For client numbers, more clients will make the performance drop because each client has fewer data and the resulting local training prone over-fitting, However, using pre-training on both real data and our FPS still improves significantly.
For the non-IID degree, we manipulate it with the Dirichlet distributions by the a parameter. We observe that more non-III) (smaller a) settings degrade the performance of using random initialization sharply while using pre-training more robust (smaller accuracy drop).
For participation rates and the number of local epochs, we observe that they are not very sensitive variables, long as they are large enough (e.g., participation rate > 30% and #local epochs > 5). Interestingly, using either fewer more local epochs does not close the gap between using pre-training and random initialization.
22

---

Published as a conference paper at ICLR 2023
50
Decay by 0.99 .5
every round
40
Pre-traines Random
0.15
10
0-
0.10
20 Rounds
50
Rounds
50
0.4
decay
0.3
Loss Ver
30
20
0.2
0.1
10
20
30
40
50
0
10
20
30
40
50
Rounds
Rounds
Figure 15: ImygeeeeeNet clients, Dir(O.3), 100% participation, local epoch= 5) with different learning
rate decay strategies. Left: we show Global model (x) accuracy and averaged accuracy using each local model (+). Right: the test loss variance 100 sampled coefficients (the width of the shaded area in Figure 8 to combine the local models in each round.
23

---

Published as a conference paper at ICLR 2023
C.5 ANALYSIS WITH FPS
For brevity, in section of the main paper mainly focus using random initialization and using real data for pre-traie training.... Here we also provide the analyses on global aggregation using our FPS, following section 6 We observe consistent effects FL with FPS: it does not alleviate local model drift (Figure 16) but makes global aggregation more stable evidenced by smaller gains with optimal and lower loss variance Figure 18).
convex combinations Figure 17
Random
Pre-trained
FPS
100
100
CIFAR10
93
50
40
8
Centralaad/iddeg
Centrallues/Fed
Contralaad/Rideyg 02.0/84.
10
20
30
40
30
20
30
40
50
30
Tiny- liny-ImaggeNe
5 8
Centralzed
00-
50
8 8
30
20-
8
16
Cortralized/Fedaee Acc: 52.9/50.3
Fortrriooo/FFod/ 49.1/45.7
30
10
20
30
Rounds
Rounds
Rounds
Figure 16: Training dynamics of FEDAVG. For each combination (dataset pre- wve-training or not), we show the accuracy/mloU using the global model (()) (x), and the averaged test accuracy using each local model ( (+). For brevity, only draw the first 50 rounds but the final accuracy/mloU after 100 rounds. CIFAR-10 and Imageema (10 clients, Dir(O.3), 100% participation, local epoch= 5). The experiments follow (section
D ADDITIONAL DISCUSSIONS
D.1 POTENTIAL NEGATIVE SOCIETAL IMPACTS
Our work discusses how using pre-training large dataset can improve FL. While not specific to our discussions, collecting the real dataset could potentially inject data bias or undermine data privacy if the collection process is not carefully designed. However, we believe the concerns are mainly from data collection but not from the learning algorithms.
To remedy this, in section we propose an alternative of using synthetic data that does not require any real data and can still improve FL significantly.
D.2 COMPUTATION RESOURCES
We implement all the codes in PyTorch and train the models with GPUs. For experiments with 2080 Ti GPU. Pre-training takes about day and FL takes about 8
32 32 images, we trained hours. For experiments with 224 224 images, we trained on an A6000 GPU. Pre-training takes about - 3 days. and FL takes about day. For the Cityscape dataset, we trained with an A6000 GPU for about 2 days.
D.3 MORE DISCUSSION ON INITIALIZATION AND PRE-TRAINED MODELS IN FL
In subsection 4.1 we consider pre-training a model in the server and use it as the initialization for further federated training.
Another very different and complementary problem on model initialization compared to ours about selecting part of the parameters of the global model to initialize each round of local training each Zhu client improve communication and computation efficiency, which is orthogonal problem to ours.
client. Several works Li et al. (2021a); Zhu & Sun (202I) proposed to find sparse mask for each
24

---

Published as a conference paper at ICLR 2023
CIFAR10
Tiny-lmageNet
100
55
50
Test Acc
Test Acc
45
40
70
Random
35
Real
60
30
0
20
40
0
20
40
Rounds
Rounds
Figure 17: Optimal convex aggregation experiments CIFAR-IU 10 and Tiny-ImageNor (10 clients, Dir(O.3), 100% participation, local epoch= 5). The experiments follow Figure 7 We show for each combination (dataset pre-training or not) the accuracy using the global model 0 (solid). We also show O=(1) (dashed), which applics the optimal convex aggregation throughout the entire FEDAVG process. FEDAVG with pre-training smaller gap.
CIFAR10
Tiny-ImageNet
0.6
0.4
0.2
0.2
01
0.0
0.0
Figure 18: The loss variance of 100 sampled coefficients on CIFAR-10 and Tiny- Ina-emmeeNet (10 clients, Dir(O.3), 100% participation, local epoch= 5). Slightly different from Figure 8, here plot the variance, which corresponds the width of the shaded area in Figure 8 for better visualization.
recent work (Tan et al. 2022) instead uses several frozen pre-trained models in FL. and only learns the class prototypes for classification.
D.4 SYNTHETIC DATA FOR FL
Pre-training on synthetic data is a relatively new and fast-growing area in centralized learning. In our paper, we focus on generating synthetic data for images and using them for pre-training for initializing FL. We have shown the superiority of our proposed FPS against other synthetic data generation methods for pre-training, including one based on random StyleGANs Baradad al. (2021): Karras et al.] (2019). Besides vision tasks, we found many recent works showing the benefits of tre-training on carefully-designel synthetic data for downstream tasks on various modalities such as text (Wu al. 2022). SQL tables (Jiang et al. 2022), speech (Fazel et al. (021). etc. We believe it is promising to bring these methods into the federated setting and our study can be valuable reference.
Another work FedSyn (Behera al. 2022 considers training generator (e.g., GAN) in a federated setting using clients' data, which is fairly different from our problem of generating synthetic data (without looking at clients' data) at the server for pre-training the global model.
25


Reviews:
[{"review_id": "rFMuAm7U6TP", "reviewer": "Reviewer_N8td", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper discusses the applicability of pre-training in a federated learning setup. The authors did extensive experiments to demonstrate that pre-training can help close the gap in the performance difference between centralised and federated learning. The authors have shown that even pre-training with synthetic data improves performance in federated settings. The authors also show that pre-training helps the FedAvg algorithm tackle non-iid distribution. Moreover, the Authors also compare the FedAvg algorithms with different FL algorithms.\n", "summary_of_the_review": "The reviewer is fairly confident that the evaluation is correct.", "strength_and_weaknesses": "Strengths:\nThe paper discusses the impact of pre-training in federated settings\nThe experiments show that the effectiveness of pre-training increases in challenging situations when there are higher non-IID degrees.\nThe authors also showed that pre-training with synthetic data could help in privacy-critical domains.\n\nWeaknesses:\nThere is no literature on the impact of initialization in federated settings. Authors should compare the pre-training initialization with initialization techniques like FedMask.\nAuthors should compare pre-training performance using FedAVG with SGD and FedAvg with GD. This would help in understanding the impact of pre-training on communication.\nThe authors should compare the existing federated algorithms with the FedAdam algorithm. \nThere is no experiment on the impact of pre-training on system heterogeneity.\nAuthors should experiment with the impact of pre-training on convergence speed.\nThe authors showed an analysis of loss curves only for IID data. Is the initial loss for the model initialized with pre-trained weights less than the model initialized with random weights? \nThe major focus throughout the paper is on FedAvg, and analysis is reported on that only. No comparative analysis is reported for IID, and Non-IID cases with other existing FL approach.\nData at each client is generated differently to update the global model. Is there any impact of the pretraining on the convergence of each model concerning the data size (data heterogeneity)?\nIn the existing work for studying the effect of the pretraining on FL, authors have experimented with a higher value of M. However, authors in this paper have restricted to the 10.\nThe authors have proposed a method to generate synthetic data. However, in FL settings, multiple methods are introduced in existing works which are missing the paper, such as [1]. This follows no comparison of the pretraining with synthetic data generation methods.\n[1] Behera, Monik Raj, et al. \"FedSyn: Synthetic Data Generation using Federated Learning.\" arXiv preprint arXiv:2203.05931 (2022).\n*Note: arxiv version is available at https://arxiv.org/pdf/2206.11488.pdf\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "brek5n1w0Q", "reviewer": "Reviewer_n7Y1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an empirical study to investigate the effectiveness of pretraining for federated learning. On the image recognition domain, models pre-trained with ImageNet, Place365, or synthetic images are tested. Synthetic images are used in pretraining with multi-label supervision or contrastive learning. An extensive experimental evaluation has revealed that the use of pretrained models brought significant performance boosts compared to when not doing so, on multiple datasets and data splits.\n", "summary_of_the_review": "Overall, my major concern is the clarity of the paper regarding what is the main contribution of the proposed work. As the pretraining for FL has already been found effective in NLP, the current introduction seems to be a bit overclaiming in some points, such as \"a different and rarely studied dimension in FL — model initialization\", \"We conduct the very first systematic study in these aspects, using visual recognition as the running example.\" Nevertheless, later the authors found that just how to enable pretraining for visual recognition on FL is not trivial, and motivated to develop a new method (FPS). I believe that that should be a potential contribution to be studied and emphasized more. If the proposed FPS is combined with multiple FL algorithms (e.g., FedAvg, FedProx, SCAFFOLD, etc.) and validated at the current level of experimentation and analysis for multiple tasks (e.g., classification, segmentation), I would be willing to increase the score.", "strength_and_weaknesses": "## Strong points\n\n- The paper is well written and easy to read. \n- The proposed work is well motivated.\n- Experiments are systematic and extensive, demonstrating the effect of pretrained models from various perspectives. \n\n## Weak points\nAlthough the paper is largely well written and the motivation of the work is understandable, the main contribution is still unclear to me. I believe that this paper could have become stronger if it focused on how to effectively perform pretraining for federated learning on visual recognition tasks, rather than claiming to be \"the very first systematic study on pre-training for FL\". \n\n- The use of pretraining for federated learning is not a new idea. At least in NLP, the use of pretrained models is well studied [a, b]. Especially [b] already showed that fine-tuning pretrained models with federated learning improved performances in both iid and non-iid settings. Given those existing works, the fact that pretraining worked well also for visual recognition tasks does not seem to be a very strong finding.\n  - [a] Tian et al., \"FedBERT: When Federated Learning Meets Pre-training\", ACM Transactions on Intelligent Systems and Technology, 2022 https://dl.acm.org/doi/full/10.1145/3510033\n  - [b] Weller et al., \"Pretrained Models for Multilingual Federated Learning\", NAACL, 2022 https://aclanthology.org/2022.naacl-main.101.pdf\n- As the pretraining is already confirmed effective for FL in NLP, the next key question, at least to me, should be whether existing pretraining approaches are directly applicable or they should be extended in some way to be adapted to FL in CV. Indeed, the work proposed a new self-supervised pretraining algorithm called fractal pair similarity (FPS), very briefly in Section 4, as the authors found existing methods not effective. I believe that this paper could have become much more informative if this point was studied in more depth. Currently, Section 4.5 and Table 6 just showed that FPS combined with SimSiam performed the best on CIFAR10/100 and TinyImageNet. Further details and results are provided in Section A.1, C.3, and Table 12, which I believe should be more emphasized in the main body. Additionally, it also remains unclear if the same approach could be effective for various FL methods such as FedProx (Li et al., 2020b) and SCAFFOLD (Karimireddy et al., 2020a) presented in Section 2.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LLFuaLV9u6Y", "reviewer": "Reviewer_acWZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nPre-training is prevalent in nowadays deep learning to improve the learned model’s performance. But, in federated learning (FL), neural networks are mostly initialized with random weights without pre-training weights. \nThe systematic study on pre-training for FL on five image datasets including Cityscapes. \nAccording to abundant experiments, the authors find that pre-training largely bridges the gap between FL and centralized learning. \n", "summary_of_the_review": "This  paper is good and can be accepted.", "strength_and_weaknesses": "Strength:\n1. The first systematic study on pre-training for FL, using five image datasets including Cityscapes. \n2. The authors further reveal new insights into FL, opening up future research directions.\nAccording to abundant experiments, the authors find that pre-training largely bridges the gap between FL and centralized learning. \n3. The experiment and analysis of the pre-training method on FL are sufficient.\n\nWeakness:\n1. When pre-training on synthetic data, the fractal image is quite different from the real sample image (such as ImageNet). Why is it also effective in testing datasets after pre-training?\n2. The model adopted by the authors is mainly used for the classification task, and the network layer is not too deep, such as ResNet18 and ResNet20. If you pre-train on a deeper network model such as the ResNet50 or GoogleNet network, how does it work on FL?\n3. The amount of data in the test set is small, such as 10K,0.5K,36K, etc. It is recommended to validate the advantage of pre-train in FL on a larger test set.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IHPtjUoUueN", "reviewer": "Reviewer_f2aE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper deals with the problem of federated learning and demonstrates the significance of pre-training in this problem. More specifically, this paper chooses DEFAVG as the baseline for federated learning and considers using fractal-based image generators and self-supervised representation learning. The experimental results demonstrate that (1) pre-training improves the baseline, (2) performance gaps between federated and centralized learnings are getting smaller by pre-training, (3) those gaps become much smaller for more challenging settings, (4) pre-training is more effective for larger network models, and (5) pre-training by self-supervised learning with the same training examples as the primary network training can improve the performance of federated learning.", "summary_of_the_review": "This paper can be accepted as-is.", "strength_and_weaknesses": "[Strength]\n\nS1. Federated learning is one of the significant topics in machine learning and artificial intelligence. Knowledge and insights for this topic will draw attention from a broad range of researchers and engineers.\n\nS2. The problem dealt with in this paper is interesting. Applying pre-training to federated learning is thought to be difficult, mainly due to privacy concerns. This paper tackles this problem by introducing fractal-based image generators whose performances have already been proven in standard image recognition tasks.\n\nS3. The significance of pre-training in federated learning has been examined and justified well by various kinds of empirical experimentation.\n\nS4. The current manuscript is well-written and easy to follow.\n\n[Weakness]\n\nW1. I think that the current manuscript does not have any critical drawbacks. However, I am not sure whether we can provide synthetic data useful for federated learning in other (for example, audio, speech, and texts) domains.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster