Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameters
Authors: Hesham Mostafa
Venue:  (2020)
Keywords: federated learning
hyper-parameter tuning
regularization

Abstract:   Federated learning is a distributed, privacy-aware learning scenario which trains a single model on data belonging to several clients. Each client trains a local model on its data and the local models are then aggregated by a central party. Current federated learning methods struggle in cases with heterogeneous client-side data distributions which can quickly lead to divergent local models and a collapse in performance. Careful hyper-parameter tuning is particularly important in these cases but traditional automated hyper-parameter tuning methods  would require several training trials which is often impractical in a federated learning setting. We describe a two-pronged solution to the issues of robustness and hyper-parameter tuning in federated learning settings. We propose a novel representation matching scheme that reduces the divergence of local models by ensuring the feature representations in the global (aggregate) model can be derived from the locally learned representations. We also propose an online hyper-parameter tuning scheme which uses an online version of the REINFORCE algorithm to find a hyper-parameter distribution that maximizes the expected improvements in training loss. We show on several benchmarks that our two-part scheme of local representation matching and global adaptive hyper-parameters significantly improves performance and training robustness.

Under review as a conference paper at ICLR 2020
ROBUST FEDERATED LEARNING THROUGH REP- RESENTATION MATCHING AND ADAPTIVE HYPER-
PARAMETERS
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning is distributed, privacy-aware learning scenario which trains a single model on data belonging several clients. Each client trains local model on its data and the local models are then aggregated by central party. Current federated learning methods struggle in cases with heterogeneous client-side data distributions which can quickly lead to divergent local models and a collapse in performance. Careful hyper-parameter tuning particularly important in these but traditional automated hyper-paranee ppprrpprreetter tuning methods would require several training trials which is often impractical a federated learning setting. We describe wroppregeed solution to the issues of robustness and hyper- parameter tuning in federated learning settings. We propose a novel representation matching scheme that reduces the divergence of local models by ensuring the feature repre- sentations in the global (aggregate) model can be derived from the locally learned representations. We also propose online hyper-parameter tuning scheme which uses an online version of the REINFORCE algorithm to find a pereeeerrrmeter distribution that maximizes the expected improvements in training loss. We show several benchmarks that our two-part scheme of local representation match- ing and glohal adaptive hyper-parameters significantly improves performance and training robustness!
INTRODUCTION
The size of the data used to train machine learning models is steadily increasing, and the privacy concerns associated with storing and managing this data are becoming more pressing. Offloading model training to the data owners is attractive solution to address both scalability and privacy concerns. This gives rise to the Federated Learning (FL) setting (McMahan et al. 2016) where sev- eral clients collaboratively train a model without disclosing their data. In synchronous FL, training proceeds in rounds where at the beginning each round, a central party sends the latest version of the model to the clients. The clients (or a subset of them) train the received model their local datasets and then communicate the resulting models to the central party at the end of the round. The central party aggregates the client models (typically by averaging them) obtain the new version of the model which it then communicates to the clients in the next round.
The FL setting poses a unique set of challenges compared to standard stochastic gradient descent (SGD) learning on a monolithic dataset. In real-world settings, data from each client may be drawn from different distributions, and this heterogeneity can lead the local learning processes diverge from each other, harming the convergence rate and final performance of the aggregate model. We address this challenge two ways: (1) adaptive on-line tuning hyper-parameters, and (2) adding regularization term that punishes divergent representation learning across clients.
tuning schemes, such as random search (Bergstra & Bengio) 2012) or Bayesian methods (Bergstra et al. 2011 Snoek et al.) 2012). require several training runs to evaluate the fitness of different hyper- papper--rrmmters... This is impractical in the FL setting, which strives to minimize unnecessary communication. To address this issue, we formulate the
Traditional
Code used run all experiments is available under the anonymous repository httptt::////// // weaiiitabb....m anon iel ill r2020 hwwprroossssseeeeeeeeeeeee federated 1earnii___nnnn

---

Under review as a conference paper at ICLR 2020
selection problem as an online reinforcement learning (RL) problem: in each round, the learners perform an action by selecting particular reward which is the relative reduction in training loss. We then update the policy online to maximize the rewards.
values, and at the end of the round get a
selection
Unlike centralized SGD, where there is a single trajectory of parameter updates, the FL setting has many local parameter trajectories, one per each active client. Even though they share the same initial point, these trajectories could significantly diverge. Averaging the endpoints of these divergent tra- jectories at the end of a training round would then result in a poor model (Zhao et al. 2018). Clients with heterogeneous data distributions exacerbate this effect as each client could quickly learn repre- sentations specific to its local dataset. We introduce scheme that mitigates the problem of divergent representations: each client uses the representations it learns to reconstruct the representations initial model it receives. A client is thus discouraged from learning representations that are too spe- cific and that discard information about the globally learned representations. We show that this representation matching scheme significantly improves robustness and accuracy in the presence of heterogeneous client-side data distributions.
the
We evaluate the performance of our two-part scheme, representation matching and online hyper- parameter adjustments, on image classification tasks: MNIST, CIFARIO; and on a keyword spotting (KWS) task. We show that for homogeneous client-side data distributions, scheme consistently improves accuracy. For heterogeneous clients, in addition to improving accuracy, our scheme im- proves training robustness and stops catastrophic training failures without having to manually tune hyper-parameters for each task.
RELATED WORK
The simplest automated hyper-parameter tuning methods execute random searches in hyper- parameter space to find the best-performing hyper-parameters (Bergstra & Bengio (012).. Random search is outperformed by Bayesian methods (Bergstra al. 2011 Snock al.] [0122) that use the performance of previously selected hyper- to inform the choice of new hyper-parametee to try. In these methods, the fitness of byper-paramenn choice is the post-training validation accu- racy. Running the training process to completion in order to evaluate the arper-prrameter fitness is computationally intensive. This motivates running partial training runs instead (Klein et al., 2016] Li al. 2016). However, in FL settings with tight communication budgets, running several (partial) training runs to select the right hyper-parameters could still be impractical.
Hyper-parameter can be optimized using gradient descent to minimize the final validation loss. To avoid storing the entire training trajectory in order to evaluate the hyper-gradients, Maclaurin et al. (2015) uses reversible learning dynamics to allow the entire training trajectory to be recon- structed from the final model. Alternatively, the prpeerggaddents can be calculated in a forward manner, albeit at an increased memory overhead (Franceschi et al. [017). Using the hyper-gradient to optimize hyper-parameter, is an iterative process that requires several standard training runs (hyper-iterations), again making this approach impractical in communication-ccostraiie FL. set- ting. Hyper- parameters can be optimized using SGD in tandem with model parameters through the Hyper-parameters use of heper-networks (Ha et al. 2016: Lorraine & Duvenaud 2018) which map to optimal model parameters. It is unclear how hyper-networks can be learned in a FL setting as we are interested in choosing hyper-pane saper-parameters that result the best post-agg cos-aagregation model and not the
chosen by any particular client to optimize its own model on its own dataset.
The hyper- paremmteree tuning approaches most related to ours are those based on RL methods. Daniel et al. (2016); Jomaa et al. (2019) learn policies and/or state-action values that are then used to select good hyper-parameter
However, several training runs are needed to learn the policies and the action values. Perhaps the closest approach to ours the method in Xu et al. (0117) which trains an actor-critic network in tandem with the main model and uses it to select actions (hyper-pance parameter choices) that maximize the expected reduction in training loss. It is unclear how this approach can be applied in a FL setting as the training process on each client has no access to the loss of interest which is the loss of the agsr-gggre model.
FL typically struggles when the client-side data distributions are significantly different (McMahan et al. 2016 Zhao et al. 2018) this causes the parameter trajectories to significantly diverge
2

---

Under review as a conference paper at ICLR 2020
server hy
W P.O
server
We 9.0 @ A0LL
server
W+4
Weea 4-
Agyn W
9.0 Lan
P( (v))
Losalweal
Ovline Senn
We. he
wi
[w] 0
We
w
w
W f- We
8
O o Vixed client
Lixed
SG
Fiied client
client
(a)
(b)
(c)
Figure 1: The three steps in federated learning round. The server communicates the latest model parameters W; and the training hyper-parameters he to each client. (b) client trains the model parameters w; and the representation matching parameters Of to minimize the classification loss and the matching loss. (c) Each client sends back the updated model parameters to the server. The server aggregates the received parameters, evaluates the loss of the aggregate model, and updates the parameters the hhper--pranne parameter distribution, vb based on the history the aggregate losses.
across different clients. This severely degrades the performance of the averaged model (Zhao et al., 2018). One straightforward solution reduce the SGD steps each client takes between the syn- chronization (model averaging) points. More frequent model synchronization, however, increases communication volume. Various compression methods (Sattler et al., 2019 Konečný et al., 2016) are able to compress the model updates, which makes it possible to synchronize more frequently and mitigate the effect of heterogeneous data distributions. (Sttler et al. (2019) takes the extreme case of synchronizing after every SGD step, which reduces the FL setting to standard SGD, An alternative solution is to mix the datasets the different clients to obtain more homogeneous client-side data distributions (Zhao et al. 2018). This, however, compromises the privacy of the clients' data.
METHODS
We build upon the federated averaging (FedAvg) algorithm (McMahan et al., 2016) and introduce two novel aspects to it: global adaptive hyper-paramette and local (per-client) representation match- ing. We first give an informal description of the complete algorithm. We consider synchronous FedAvg setting with K clients, where s is the dataset local to client Training proceeds in rounds. Figure I iilustrates the steps involved in training round t. The server maintains distri- bution p) over the space of hyper-paramen parameters H. At the beginning of round (Fig. [a],, the selected clients receive the most recent model parameters from the server, w,, together with train- ing hyper-parameters h, sampled from P(H v)y) Client i uses two copies of w, to initialize two models: fixed model .M and trainable model Mi. The parameters w of M are trained using SGD (Fig. [bb.. Client maintains a set of local parameters 0 that it uses to map the activations in Mi the activations M Of and w are simultaneously trained to minimize two-component loss: the first component is the standard training loss of M; on D; (for example, the cross-entropy loss); the second component is the mean squared difference between the activations in MF and the activations reconstructed from Mi using fj. We denote this second component as the representation matching loss. the final step in the round (Fig. [c], each participating client sends its final model parameters, w to the central server. The server aggregates the model parameters, and uses the loss of the resulting model to update the parameters z of the pappe-prrrmtter distribution, P(H())
3.1 REPRESENTATION MATCHING
We now describe the representation matching scheme. For an example model, Fig. illlssrates how the local parameters 0² in client are used to map the activations of the model being trained, M too the activations of the fixed model M M is parameterized by w' while MM is parameterized by w which was received from the server at the beginning of training. To simplify notation, we dropped the index of the round, t. Given data point the layer activations Jay (z; w') .......ayy (z; w') and [a| (z; w)... a (2; w)], respectively. These not all the model activations, but only the M activations
D,. is fed to M and A= to obtain
interested in matching. The
3

---

Under review as a conference paper at ICLR 2020
fixed model
MM
matching layers
trainable model FC
M
file M
Fo Ruc
mutx, w
ReLU
ReLU
Fole Ant
FC
FC
ReLU
UnPooo
ReLU
whj
mnax_pool
peatood
max_pool
const o
conv
conv
input:
input
Illustration of representation matching using a model with one convolutional layer and two fully connected layers. Layers whose activations are used in the matching loss are shown in red. The matching layers map the activations M to the activations in M
Figure 2:
activations in the two models are matched by set of matching layers parameterized by 0 (0' [0;, 0g 0%] in Fig. 2) Instead of deriving the activations of one layer in M from the activations of the corresponding layer in MM found performance significantly improves if the activations of one layer are instead derived from the activations the next layer of interest above it. As shown in the representative example in Fig. 2 the activations/layers of interest are the input layer, the point-wise non-linearity layers and the top layer. We use convolutional matching layers if the input has spatial information (such as within a convolutional stack), and fully-connected matching layers otherwise. To reverse a pooling operation, we follow the scheme in Zhao et al. (2015) and unpool pooled activations using the pooling positions used during the pooling operation shown in Fig.
Given a training point and label (x, v) ~ D,, we define the matching loss at client as:
j=M-1
(x
Σ
ay (2 w)ll
(1)
where fy denotes the jth matching layer. The per-point training loss at client is:
L f' w. 0²) = Lo(ass (x; w )))) + Cir w', w,O')
(2)
where Le is the cross-entropy loss and a. (x; w*) ) is the top layer activation in the trainable model.
Our representation matching architecture is similar in respects to auto-encoding architectures, particularly ladder networks (Rasmus et al. 2015). Unlike ladder networks, however, we not reconstruct the activations of non-noisy model from noisy model but rather, reconstruct the activations of the most recent aggregate model from the activations of the model while it is being trained on local dataset. This has regularizing influence it stops the clients from learning representations that are too specific to their local datasets. This should also ameliorate the effect of model divergence in the clients as all client models have learn representations that are mappable to common set of representations defined by the common aggregate model. Additional regularization comes about because a layer's activations are used reconstruct the activations of layer below it. A similar technique was used in what-where autoencoders (Zhao et al.] 2015). and was shown to significantly improve accuracy. Our approach is fundamentally distinct, however, as we use the activations in one one model, M to reconstruct the activations in different model, ....
Our representation matching scheme is similar in spirit to the techniques used to learn invariant feature representations for domain adaptation where the goal is to minimize the discrepancy be- tween the distributions features extracted from different domains. However, current techniques for learning nomain-invarian features et al.) 2016 Shen al. (217)) are incompatible with the FL setting as they require the learner to have access to data from different domains (clients in case) in order to match the feature distributions across these domains.
4

---

Under review as a conference paper at ICLR 2020
3.2 ADAPTIVE PAPER-PRRAMETERS
At each round, the server provides the clients with training hyper-parameters. The hyper-parameters he provided in round are a sample from P(H) (x). Let Le be the loss of the aggregate model the beginning of round I. on a representative set of data points (we describe in the next subsection practical schemes for evaluating L.). We define the reward received for choosing hyper-parametee h, as Tt (L, / Lt The use of relative loss reduction is motivated by the desire to have the scale of rewards unchanged throughout training. At round f, the goal is to maximize:
Jj ( (v)) [m+]
Taking the derivative of Jr and approximating it with one-sample Monte Carlo estimate, we obtain
V v Jt BBrhh (o,.o(((((,)
re V ((h, (v:)) where h, P(h, (1))
(5)
The score-function REINFORCE gradient estimator (Williams) 1992) in Eq. can be readily evaluated and can be used to update thy However, the variance of this gradient estimator can be quite high (Rezende et al. 2014), especially since we are using a single-sample estimate. To reduce variance, we introduce a reward baseline (Greensmith et al., 2004) which is the weighted average reward in an interval [t Z, t + Z] centered around t. The update equation for the is:
T=1+2
1441 t the ma - Fe) V log ((h((()))))))) where Ft = YZ
Σ
11+1-15-
t|rr= -
(6)
T
T
where 77 is the hyper learning rate and 72 a normalizing constant. We weigh nearby rewards more heavily than distant rewards when calculating the baseline in round t. A causal version of Eq. that only depends on past rewards is:
V4+1 + 12-
Σ vo, -=
where f:
Σ rT
(7)
Z
+11
7=
which is the update equation for vi that we use. Even though we are using RL terminology, there are several fundamental differences between our problem and traditional RL problems. First, the setting is non-stationary as the same action in different rounds could lead to different reward distribution. For example, large learning rates are appropriate towards the beginning of training but would increase the loss towards the end of training. In our scheme, this reflected the design of the baseline which weighs nearby rewards more heavily as these provide a more accurate baseline of the rewards at the current state of the learning problem. This non-stationary behavior can be modeled as a murtially-observabe techniques can be used to learn policies in non-stationary settings (Padakandla et al. 2019 AAbdllahh & Kaisers 2016), these methods are not online and require several trajectories (training runs in our case) to optimize the policy. The second difference from traditional RL methods is that we do not seek to maximize the (discounted) sum of rewards, but rather, each round we seek to maximize the rewards in a small interval [ Z, 1] which is what allows us to formulate an online algorithm.
Markov decision process (Jaakkola et al. (995).. While several
3.3 FULL ALGORITHM AND PRACTICAL CONSIDERATIONS
Algorithm [ describes the FedAvg algorithm with our contributions: representation matching and adaptive hyper-parameters. There are couple of practical considerations that we address here:
Evaluating the loss of the aggregate model: After each round, the server needs evaluate the loss of the aggregate model in order to obtain the reward signal (Line 15 in algorithm []) This could be done in two ways: The server maintains a small validation set on which it evaluates the loss. This validation set has to be representative of the clients" data. Unlike the scheme in Zhhao et al.) (0155), this validation set not shared with the clients and can be collected from them in a secure way to avoid revealing the origin of each data point. 2) At the start of a round, the clients themselves evaluate the loss of the aggregate model on a small part of their training data and then send the scalar loss to the server. The server averages these losses to obtain is good if the fraction of participating clients in the round is high, which would make the evaluated loss more representative of the loss across all clients. In our experiments, use the first approach.
estimate of the loss. This estimate
5

---

Under review as a conference paper at ICLR 2020
Algorithm Federated averaging with representation matching and online nlype-paraaette tuning.
Setting with T training rounds, K clients, ne datapoints per client, total datapoints, and a fraction C' of clients participating in each round
1: initialize W1 and .i.....0% 0 and 1/1
2: L1 f Loss(w1)
3: for t=1 do
4: h, P()
Sample set of hyper-parametent
5: St <-aandomm selection of CK clients)
6:
for i S, do
Run in parallel in each client in S, b Receive model parameters from server Number of SGD iterations as defined in h, >> Sample a training mini-bate w fj] - [wi, fi] ()( V Vi (X, Y; wi, W,, 0)) L, as defined Eq.
7:
w f- Wt
8:
for n=1 to n_iiteraiionnssss h:) do
9:
XX Y & sample D2)
end for
12:
0 t- 8
end for
14:
W++1 <
Σ
nk [w: wt)
N
iES:
15:
L411 t- Loss( Loss(Wr+)) (w++1)
16:
Ti f (Lr (/-1)/Lr
Z' minl 2,,
18:
41+1 c 4/- 78
(r- Vog where A 2'-1
Σ rT
r=t-2
T=t-2'
19:
end for
Choosing the form of P(H For D hyper-parameters we use discrete D-dimensional hyper- parameter space. For the pth hyper-parameter we have finite set of allowable values Hp. H D-dimensional grid containing all possible combinations of the allowed values of the D hyper- parameter. [7]] 11 HHpl- For P(H[w)
D-dimensional discrete Gaussian. Let W( (,,A) be standard(comtinuuur D-dimensional Gaussian with mean μ and precision A. Let hy = (h), ......... ...))) be point on the grid H. P(hşlv) = (h (l, A) where & = {1r A} and (h/u, Z Enen:N (hh/,,A) A). Since different pyper-parameters can have different scales, we shift and normalize the allowed values for each
so that they have zero mean and are in the range (-0.5, 0.5] before constructing the grid. The grid of hyper-parameters H thus has zero-mean and the same scale in all dimensions. This increase the stability of the hyper-parameter tuning algorithm as the optimization space is uniform in all directions.
We choose a discrete hyper-parameter space H to force different hyper-paramette choices to be significantly different, which would provide more distinct reward signals to the online REINFORCE algorithm. The choice of a modal distribution, such as the discrete Gaussian, encourages hyper- parameter exploration to focus areas around the mode. This embodies the inductive bias that the optimal hyper-parameter in one round is close to the optimal hyper- parameter in the previous round.
EXPERIMENTAL RESULTS
We evaluate the performance of our two-part scheme on the MNIST and CIFARIO image classifi- cation datasets, and on a KWS task using the speech commands dataset (Warden 2018 We restrict the KWS task to only ten keywords. In all experiments, we use ten clients and use one of two data distributions: iid where the dataset split randomly across the clients, and non-iid where each client only has data points belonging to one of the ten classes. We use our online hyper-parameter tuning scheme to tune the learning rate and the number of SGD iterations (see algorithm [T). all ex- periments, we use the same hyper-hyper pereehyypee--aaatteer controlling the hyper-parameter tuning process which are the hyper-learning interval Z. One exception is the KWS task where modify the grid H. to reduce the number of allowed SGD iterations per round to reflect the smaller size of the dataset. A batch size of 64 is used throughout. In all experiments, we add entropy regularization (ER) term (Pereyra et al.) 2017 to
rate Tи the param-err grid H. and the REINFORCE baseline

---

Under review as a conference paper at ICLR 2020
the client losses during training. For client in round f with imput x, the ER loss term has the form:
(8) where H)) is the entropy operator and aM the top layer activity. LEN penalizes highly confident (low entropy) output distributions when their entropy falls below H mmnn- This loss term significantly improves the performance of the FedAvg algorithm in the presence of non-iid client data.
LER w() = mar (0, Hmin - H
( (sootmar(aM (x: w()))
As comparison baseline, we run experiments with a fixed decay schedule for both the learning rate and the number of SGD iterations per round. As a baseline for our representation matching scheme, we run experiments where each client's training loss augmented with a term that penalizes weight divergence between the client model parameters and the initial parameters received from the server. For client i in round t, this weight divergence (WD) loss term has the form [ww w; 12. This
penalty function was introduced into the FL setting by the FedProx algorithm (Li et al. 2018 to mitigate the effect of model divergence.
We report results for the standard FedAvg algorithm (FA), FedAvg with weight divergence loss term in the clients (FA+WD), and FedAvg augmented with adaptive hyper-parametees(AN and/or representation matching (RM). We consider two values for C (the fraction of clients participating in each round): C' = 0.5 and C' = 1.0. For the baselines, FA and FA+WD, we manually tuned the hyper-parameters decay schedule to obtain best accuracy. This tuning was done separately for the iid and non-iid cases All experiments were repeated 3 times.
We use colly-connected network with two hidden layers with 100 neurons each. As shown in table [ the best performing algorithm is FA+AH. The use of representation matching causes slight degradation
MNIST
performance (FA+RM+AH and FA+RM). the trajectory of the mean μ of the discrete Gaussian hyper-parameter distribution P(H μr, A). This is shown in the first column of Fig. for the iid and non-iid cases. In the non-iid case, the online REINFORCE algorithm pushes the mean learning rate down as it detects that smaller learning rate yields greater loss reductions, while in the iid case, the mean learning rate is pushed up instead. This aligns with what practitioner would do to ensure convergence in the more challenging non-iid case. The REINFORCE algorithm chooses to keep the mean number of SGD iterations per round relatively high for most of the training run. As shown in table D these choices yield slightly better accuracy (FA+AH) than the fixed training schedule used in the standard FA algorithm.
interesting to look at
Table 1: MNIST accuracy figures
Data
FA
FA+WD
FA+RM+AH
FA+RM
FA+AH
Distribution
iid
97.9 0.1
97.9 + 0.005
97.9 + 0.06
97.8 +
98.0 + 0.04
C=1.0
non-iid
94.7 + 0.07
94.8 0.2
94.7 + 0.08
94.5 0.3
95.5 + 0.04
iid non-iid
97.4 0.06
97.3 0.03
98.1 0.06
97.4
98.2 + 0.01
C=0.5
92.1 + 0.8
92.0
02.5
91.6 + 0.3
93.2 + 0.8
CIFARIO We use network with two convolutional layers (with 32 and 64 feature maps and 5x5
kernels) followed by two fully connected layers (with 1024 and 10 neurons). 2x2 max pooling was used after each convolutional layer. As shown in table the use of representation matching yields a significant improvement in accuracy for the non-iid case while slightly improving performance in the jid case. The benefit of adaptive hyper-parameters over fixed hyper-parameter schedule are more equivocal. The evolution of adaptive hyper-pane parameters is shown in the second column of Fig. 3 Unlike the MNIST case, the REINFORCE algorithm chooses to push down the learning rate for the iid case well which actually leads to better performance in the iid case compared to the fixed schedule (FA vs. FA+AH and FA+RM+AH vs. FA+RM in table 2 In the non-iid case, the fixed schedule performs better, though.
Keyword spotting task We calculate the mel spectrogram for each speech command to obtain 32x32 input to our network. The network we use has four convolutional layers with 3x3 kerelss and 64 feature maps each (with 2x2 max pooling after each pair), followed by two fully connected layers of 1024 and 10 neurons. In this deeper network, representation matching is essential in
7

---

Under review as a conference paper at ICLR 2020
Table 2: CIFARIO accuracy figures
Data
FA
FA+WD
FA+RM+AH
FA+RM
FA+AH
Distribution
iid
81.9 - 0.3
0.2
8..3
83.5
8.2 44.8 5.8
C=1.0
non-iid
44.2 = 0.9
4.8
2.2
52.9
iid non-iid
76.8 0.3
0.2
1.9
0.2
82.0 - 1.04
C=0.5
33.4 1.0
0.9
3
43.3 0.8
19.8 + 7.0
MNIST
CIFAR10
KWS
[aariin rate
0.04-
0.04
0.04
0.03
noniid
loarning
nanii
p.03
noniid
lid
schedule
fixed schedule
fixed schedule
0.01
0.01
0.01
0.00
300
300
300
460
300
200
400
500
5GD stops per round
60-
stops per round
50-
per
3.5
20-
SGD
100
200
300
400
500
100
250
300
400
500
100
200
300
400
500
round
round
round
Figure 3: Evolution of the mean of the hyper-parameter distribution p( for the iid and non-iid cases. Results taken from the FA+RM+AH algorithm when C = 1.0. The evolution of the means are shown separately for the learning rate (first row) and for the number of SGD steps per round (second row), with column each for the MNIST, CIFAR10 10, and keyword spotting (KWS) tasks. During training, the means are forced to stay within the ranges defined by the hyper-parameter grid.
the non-iid case as shown in table 3 since training consistently fails in its absence. might be training would be possible in the non-iid case.
possible that with fine tuning of However, representation matching obviates the need for such fine tuning. FA+RM+AH consistently outperforms all other methods indicating the hyper-parameters schedule found by the REINFORCE algorithm (third column in Fig. 3) is better than the fixed schedule.
Table 3: Keyword spotting task accuracy figures
Data Distribution
FA
FA+WD
FA+RM+AH
FA+RM
FA+AH
iid
93.0 + 0.2
93.5 0.2
94.4+0.3.
92.9
94.0
C=1.0
non-iid
28.4 + 7.2
29.9 7.3
0.5
7.6
9.8
iid non-iid
01.4
93.0
9..
91.0
0.4
C=0.5
11.45 1..5
11.8
2.59
60.7
+
Communication and computational overhead Our representation matching scheme does not introduce any communication overhead between the clients and the central server. The adaptive hyper-parameter scheme introduces a negligible communication overhead for sending two scalar hyper-parameterd to the clients each round. As for computational overhead, we quantify the wall- clock run-time] for training a client for 30 SGD iterations or mini-batches (mini-hatch size of 64). Where adaptive hyper-pance paramtparaeeters are used, we also include the time needed to execute the adaptive
EExperiments were run machine with single TitanXP GPU, 32G of RAM, and an Intel Xeon ES- A processor.
8

---

Under review as a conference paper at ICLR 2020
hyper- apper-paraeeter tuning procedure. The results are shown in table 4 We note that the overhead of our syper-parameter tuning procedure is negligible (FA vs. FA+AH) and is typically less than 2%. This negligible computational overhead is primarily the cost of single inference pass to evaluate the loss L on a very small subset of the training set. The representation matching scheme is more demanding as it requires optimizing more complicated loss during each SGD iteration the client.
Table 4: Wall-clock run-time seconds. Mean and std. from 200 trials.
Task
FA
FA+WD
FA+RM+AH
FA+RM
FA+AH
MNIST CIFARIO
0.39 + 0.007
0.41 + 0.008
0.43 + 0.007
0.42 + 0.007
0.39 0.007
0.50 + 0.02
0.54 0.01
0.67 + 0.02
0.66 + 0.02
0.51 + 0.02
KWS
1.83 = 0.05
1.96
2.03 - 0.04
2.01 0.04
1.85 + 0.05
DISCUSSION
We described two additions to the FedAvg algorithm: representation matching and adaptive hyper- parameters. We compared representation matching (FA+RMM) against scheme based penalizing weight divergence (FA+WD) and FA+RM came out on top by a significant margin in the more difficult tasks we tried: CIFARIO and KWS. We compared our adaptive hyper-parameter scheme against a fixed hyper-parameter schedule chosen based on experience. In most cases, adaptive hyper-parameters outperform the fixed-schedule scheme. The combination of our new additions (FA+RM+AH) slightly improves performance compared to standard FA in the small MNIST net- work and significantly improves performance on the deeper networks used CIFARIO and KWS.
The hyper- ppper-prramttr schedule chosen by our REINFORCE algorithm exhibits some easily inter- pretable behavior such as reducing the learning rate in the more difficult non-iid cases. However, we see some behavior that does not have an immediately obvious interpretation. For example, in order maximize rewards in the non-iid case, reducing the learning rate more important than reducing the number of SGD iterations per round can be seen in Fig. 3 It is important to note that at each round, our online REINFORCE algorithm only seeks to maximize the loss improvement in the last few rounds. Moreover, mean to determine the direction of highest reward. These noisy problematic in settings that depend on highly tuned schedules and which have room for online exploration of papre-pprameters.. We thus expect our scheme would be outperformed by traditional automated syper-paramett tuning methods that tune hyper-parameters to maximize the final valida- tion loss. However, our scheme is significantly more efficient as it does not require repeated training and introduces only small computational overhead in each round.
algorithm continuously samples hyper-parameterss parameters around the parameter choices could prove
REFERENCES
Sherief Abdallah and Michael Kaisers. Addressing environment non-stationarity by repeating q- learning updates. The Journal of Machine Learning Research, 17(1)11822-612, 2016.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-300 1-305, 2012.
J.S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pp. 2546-2554, 2011.
Christian Daniel, Jonathan Taylor, and Sebastian Nowozin. Learning step size controllers for robust neural network training. In Thirtieth AAAI Conference on Artificial Intelligence, 2016. 2016.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient- based hyperparameter optimization. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1173-113.. JMLR. org, 2017.

---

Under review as a conference paper at ICLR 2020
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Nomain-adversaee training of neural net- works. The Journal of Machine Learning Research, 17(1):2096 2030, 2016.
Evan Greensmith, P.L. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(Nov):1471-1550, 2004.
David Ha, Andrew Dai, and Q. Le. Hypernetworss. arXiv preprint vrxiv:/609.091000 2016.
Tommi Jaakkola, S.P. Singh, and M.I. Jordan. Reinforcement learning algorithm for partially ob- servable markov decision problems. Advances in neural information processing systems, pp. 345-352, 1995.
H.S. Jomaa, Josif Grabocka, and Lars Schmidt-Thieme. Hyp-rl: Hyperparameter optimization by reinforcement learning. arXiv preprint 192,,,,90 2019.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters large datasets. arXiv preprint 1r5i071605.07779. 2016.
Jakub Konečný, H.B. McMahan, F.X. Yu, Peter Richtárik, A.T. Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. arXiv preprint arXiv: 1603.06600, 2016.
Tian Li, A.K. Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. arXiv preprint arXiv: 18/200/// 2018.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypemet-
works. arXiv preprint arXiv:/802.094 /802.09419, 2018.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimiza- tion through reversible learning. In International Conference Machine Learning, pp. 2113- 2122, 2015.
H.B. McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-efficie learn- ing of deep networks from decentralized data. arXiv preprint
05629, 2016.
Sindhu Padakandla, Shalabh Bhatnagar, et al. Reinforcement learning in non-stationary environ- ments. arXiv preprint arXiv: 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint 17X1v:1701.06548, 2017.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. Semi- supervised learning with ladder networks. In Advances in Neural Information Processing Systems, pp. 3546-3554, 2015.
D.J. Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, pp. 1278-1286. 2014.
Felix Sattler, Simon Wiedemann, Klaus-Robert Müller, and Wojeiech Samek. Robust and communication-ffiiientt federated learning from non-iid data. arXiv preprint 11X1v:1903.02891, 2019.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. arXiv preprint 1rx 2017.
Jasper Snoek, Hugo Larochelle, and R.P. Adams. Practical bayesian optimization of machine learn- ing algorithms. In Advances in neural information processing systems, pp. 2951-22122 1951-2955, 2012.
10

---

Under review as a conference paper at ICLR 2020
Pete Warden. Speech commands: A dataset for limite--vocabulary speech recognition. arXiv preprint arXiv:1804.032099 2018.
R.J. Williams. Simple statistical gradient-folowinng algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):22--256, 1992.
Chang Xu, Tao Qin, Gang Wang. and Tie- Yan Liu. Reinforcement learning for learning rate control. arXiv preprint arXiv: 159, 2017.
J.J. Zhao, Michael Mathieu, Ross Goroshin, and Yann LeCun. Stacked what-where auto-encoders. arXiv preprint arXiv:1506.02355/ 2015.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data, arXiv preprint arXiv:/806.0055 2018.


Reviews:
[{"review_id": "S1lsVxBjFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Good paper. However, the theoretical novelty is quite limited. There is no guarantee whatsoever whether the good empirical results achieved on the three experimented would persist with other datasets. Similarly, there is no analysis of the conditions required so that such empirical superiority would hold. \n\n- The flow of the ideas in the paper is clear and unequivocal. However, writing has a lot of room for improvement in terms of typos and grammatical mistakes. \nExamples\n -- p1: \"are become\"\n -- p5: \"several fundamental difference\"\n\n- I think an analysis and comparisons based on computational run-time would be necessary to check how the RL-based methodology would fare w.r.t. FA and the other FL frameworks in comparison. \n\n- What about comparing to other federated learning frameworks, i.e. other than versions and extensions of FA? \n\n- In the Introduction, it was promised that the proposed method would stop catastrophic training failures; has that been actually described in the experiments?\n\n- Similarly so for the robustness issue (which is also the first word in the paper title), where is the empirical demonstration of the robustness of the proposed method?\n\n- Regarding the last paragraph in Section 2.2 and the first fundamental difference between the proposed modelling choice and RL, is the former still different from non-stationary RL? An example of recent works on non-stationary RL is \"Reinforcement learning in non-stationary environments\" by Padakandla et al. 2019. \n\n- This is not strictly necessary, but might be a done in a future work or so: Did you consider comparing to methods which are based on learning invariant representations (which is quite similar to the methodology pursued in the paper) adopted in different ML paradigms like domain adaptation?\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1lYvJP6tH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a novel representation matching scheme to reduce the divergence of local models in federated learning. In addition, the authors propose an online hyper-parameter tuning scheme. The paper is well-written. The empirical results show good performance. In overall, I think the authors propose an interesting alternative of weight regularization (also called weight divergence loss in this paper).\n\nDetailed comments:\n\n1. (Very minor, does not affect the score) The baseline FA+WD is actually the same as FedProx proposed in [1].\n\n2. (Major concern) In most experiments, there is a huge gap in the performance between FA+WD and FA+RM. However, it is unclear such improvement is caused by the better robustness of RM, or simply simply caused by bad hyperparameters of FA+WD. Since FA, FA+WD, and FA+RM have different loss functions, it is unreasonable and unfair to use the same hyperparameters for them. The authors should report results with fine-tuned hyperparameters, so that we can confirm that RM really works. Otherwise, the results of the experiments are questionable. \n\n3. It seems that AH is irrelevant to federated learning. Even if we use fully synchronous SGD to train the model, we can still use AH to tune the hyperparameter on the server side. Ah does have some contribution, but seemingly it doesn't really contribute to the federated learning algorithm.\n\n\n\n----------------\nReference\n\n[1]  Li, Tian et al. “Federated Optimization for Heterogeneous Networks.” (2018).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkePCjw6KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript proposes two strategies to improve both the robustness and accuracy of local agents under the setting of federated learning. Specifically, online reinforcement learning is used to perform adaptive hyperparameter search in order to maximize the utility of local models. This is quite an interesting idea since traditional hyperparameter tuning techniques, including random search or Bayesian optimization, usually requires access to monolithic dataset, which is clearly impractical under federated learning. The second contribution is an idea on using local distribution matching in order to synchronize the learning trajectories of different local models. This again is a novel and interesting idea. Overall the paper is well-written and clear to follow, which is a plus. Detailed comments and questions follow:\n\n-   I understand that change of hyperparameter in local model affects the global model during in model aggregation stage. However, under the federated learning setting, if the number of local models is huge, then the influence of a single local model should be small. Hence instead of using online reinforcement learning for hyperparameter search, which is notoriously data-inefficient, why not framing the problem as a pure online learning problem? This helps to increase the data efficiency and also increases the stability of learning. \n\n-   In the design of matching network, is there any intuition why performance significantly improves if the activations of\none layer are instead derived from the activations of the next layer of interest above it? Furthermore, why we need to have additional model with parameters $\\theta_i$ for alignment? Intuitively if we want the new model $M$ to be close to the original one $M^F$, shouldn't we just use some distance measure, e.g., $\\ell_2$ norm, to measure the distance of feature activations in the corresponding layers directly? \n\n-   The experiments are illustrative, but might be too toyish under the federated learning setting. I appreciate the ablation studies the authors performed to show the relative impact of different strategies, which makes the relative contributions more clear. However, even in the case of MNIST and CIFAR, the improvement over baseline FA is not very significant. In this case, it would be better if the authors could also report the computational overhead in terms of running time.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript proposes strategies to improve both the robustness and accuracy of federated learning. Two proposals are online reinforcement learning for adaptive hyperparameter search, and local distribution matching to synchronize the learning trajectories of different local models. 

The reviewers and AC agree that the problem studied is timely and interesting, as it addresses known issues with federated learning. However, this manuscript also received quite divergent reviews, resulting from differences in opinion about the novelty and clarity of the conceptual and empirical results. Taken together, the AC's opinion is that the paper may not be ready for publication.
Decision: Reject