Does Federated Learning Really Need Backpropagation?
Authors: Haozhe Feng
Tianyu Pang
Chao Du
Wei Chen
Shuicheng YAN
Min Lin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Federated Learning
Backpropagation-Free Training

Abstract: Federated learning (FL) provides general principles for decentralized clients to train a server model collectively without sharing local data. FL is a promising framework with practical applications, but its standard training paradigm requires the clients to backpropagate through the model to compute gradients. Since these clients are typically edge devices and not fully trusted, executing backpropagation on them incurs computational and storage overhead as well as white-box vulnerability. In light of this, we develop backpropagation-free federated learning, dubbed BAFFLE, in which backpropagation is replaced by multiple forward processes to estimate gradients. BAFFLE is 1) memory-efficient and easily fits uploading bandwidth; 2) compatible with inference-only hardware optimization and model quantization or pruning; and 3) well-suited to trusted execution environments, because the clients in BAFFLE only execute forward propagation and return a set of scalars to the server. In experiments, we demonstrate that BAFFLE-trained models can achieve empirically comparable performance to conventional FL models.

Under review as a conference paper at ICLR 2023
DOES FEDERATED LEARNING REALLY NEED BACKPROPAGATION?
Anonymous authors Paper under double-blind review
ABSTRACT
Federated learning (FL.) provides general principles for decentralized clients to train a server model collectively without sharing local data. FL is a promising framework with practical applications, but its standard training paradigm requires the clients to backpropagate through the model to compute gradients. Since these clients are typically edge devices and not fully trusted, executing backpropagation on them incurs computational and storage overhead as well as white-box vulnerability. In light of this, we develop backpropagation-fiee federated learning, dubbed BAFFLE, in which backpropagation is replaced by multiple forward processes to estimate gradients. BAFFLE is 1) memory-efficien and easily fits uploading bandwidth; 2) compatible with inference-only hardware optimization and model quantization pruning; and 3) well-suited to trusted execution environments, because the clients in BAFFLE only execute forward propagation and return set of scalars the server. In experiments, we use BAFFLE to train models from scratch or to finetune pretrained models, achieving empirically acceptable results.
INTRODUCTION
Federated learning (FL) allows decentralized clients to collaboratively train server model (Konečný al., 2016; McMahan et al., 2017). each training round, the selected clients compute model gradients or updates on their local private datasets, without explicitly exchanging sample points to the server. While FL describes a promising blueprint and has several applications (Yang et al., 2018; Hard et al., 2018: 8; Li et al., 2020b), the mainstream training paradigm of FL is still gradient- that requires the clients to locally execute backpropagation, which leads to two practical limitations:
(i) Overhead for edge devices. The clients in FL are usually edge devices, such as mobile phones and loT sensors, whose hardware is primarily optimized for inference-only purposes (Sharma et al., 2018; Umuroglu et al., 2018), rather than for backpropagation. Due to the limited resources, computationally affordable models running on edge devices are typically quantized and pruned (Wang et al., 201 9a), making exact backpropagation difficult. In addition, standard implementations of backpropagation rely on either forward- mode or reverse-mode auto auto-differentiation in contemporary machine learning packages (Bradbury et al., 2018; Paszke et al., 201 2019b), which increases storage requirements.
(ii) White-box vulnerability. To facilitate gradient computing, the server regularly distributes its model status the clients, but this white-box exposure of the model renders the server vulnerable to, e.g., poisoning or inversion attacks from malicious clients (Shokri et al., 2017: 7; Xie et al., 2020; Zhang al., 2020; Geiping et al., 2020). With that, recent attempts are made to exploit trusted execution environments (TEEs) in FL, which can isolate the model status within a black-bor secure area and significantly reduce the success rate of malicious evasion (Chen et al., 2020; Mo al., 2021; Zhang et al., 2021; Mondal et al., 2021). However, TEEs are highly memory-constraine (Truong et al., 2021), while backpropagation is memory-consuming to restore intermediate states.
While numerous solutions have been proposed to alleviate these limitations (discussed in Appendix B), in this paper, we raise an essential question: does FL really need backpropagation? Inspired by the literature on zero-order optimization (Stein, 1981), we intend to substitute backpropagation with multiple forward or inference processes to estimate the gradients. Technically speaking, we propose the framework of BAckpropagation- Free Federated LEarning (BAFFLE). As illustrated in Figure 1, BAFFLE consists of three conceptual steps: (1) each client locally perturbs the model parameters 2K times W + dy (the server sends the random seed to clients for generating (5x ) (2) each client

---

Under review as a conference paper at ICLR 2023
Clients
Server
Inference
(& B1)
Upload AL
1
AC(W.8,)
AC(W.
Stein's identity
Inference
ACCWW 6,; B2)
Upload AL
WZ(W. &2)
2
W
AC(W.
AW
A4(W. 8g)
Inference
(AL 65 Bc)
Upload A.C
D
AC(W, 8yt Dz)
A.(WW 8,)
Download random seeds for 85, and the global update AW
AL 8; L(W + 8; D) L(W 8; ID)
Trusted Execution Environments (TEEs)
Secure Aggregation
Figure 1: A sketch map of BAFFLE. In addition the global parameters update A AW, each client downloads random seeds to locally generate perturbations = and perform 2K times of forward propagation (i.e., inference) compute loss differences. The server recover these perturbations using the same random seeds and obtain AC(W, 6x) by secure aggregation. Each loss difference AC (W dR: (D.) is floating-polt point number, so K can be easily adjusted fit the uploading bandwidth.
executes forward processes on the perturbed models using its private dataset DB and obtains K loss differences (AC(W) 8x; D.) D (3) the server aggregates loss differences to estimate gradients.
BAFFLE's defining characteristic is that it only utilizes forward propagation, which is memory- efficient and does not require auto-differemtiation. It is well-adapted to model quantization and pruning well inference-only hardware optimization computation graph of forward propagation BAFFLE may be more casily optimized, such as by slicing into per-layer calculation (Kim et al., 2020). Since each loss difference AL(W, 5g; DD2) is scalar, BAFFLE can easily accommodate the uploading bandwidth clients by adjusting the value of K opposed using, e.g., gradient compression (Suresh et al., 2017). BAFFLE also compatible with recent advances in inference approaches for TEE (Tramer & Boneh, 2019; Truong et al., 2021), providing an efficient solution for combining TEE into FL and preventing white-box evasion.
edge devices. Compared to backpropagation, the
Base on our convergence analyses, we adapt secure aggregation (Bonawitz et al., 2017a) to zero- order optimization and investigate ways improve gradient estimation in BAFFLE. In our ex- periments, BAFFLE is used to train models from scratch on MNIST (LeCun et al., 1998) and CIFAR-10/100 (Krizhevsky Hinton, 2009), and finetune ImageNettpretrained models transfer to OfficeHome (Venkateswara et al., 2017). Compared to conventional FL, BAFFLE achieves sub- optimal but acceptable performance. These results shed light on the potential of BAFFLE and the effectiveness of backpropagation-fice methods in FL.
2
PRELIMINARIES
In this section, we introduce the basic concepts of federated learning (FL) (Kairouz et al., 2021) and the finite difference formulas that will serve as the foundation for our methods.
2.1 FEDERATED LEARNING
Suppose we have C' clients, and the c-th client's private dataset is defined D = {(X,, ( y!) 1 N with N, input-label pairs. Let L(W; D.)) represent the loss function calculated on the dataset D..- where W E RM denotes the server model's global parameters. The training objective of FL is to find W that minimize the total loss function as
C
C
((W) = Σ N: DD); where N
N
(1)
N
c=1
In the conventional FL. framework, clients locally compute gradients ( L(W; D.)C model updates through backpropagation and then upload them to the server. Federated average (McMahan et al., 2017) performs global aggregation using AW AW = C No update obtained via executing W. W. 1) Vw L W De) multiple times and 1) learning rate.
N A where A AW, is the local
2

---

Under review as a conference paper at ICLR 2023
2.2
FINITE DIFFERENCE
Gradient-based optimization techniques (either first-order or higher-order) are the most frequently used tools to train deep networks (Goodfellow al., 2016). Nevertheless, recent progress demon- strates promising applications of zero-order optimization methods for training, partieularly when exact derivatives cannot be obtained (Flaxman et al., 2004; Nesterov & Spokoiny, 2017; Liu et al.. 2020a) or backward processes are computationally prohibitive (Pang et al., 2020; He et al., 2022). Zero-order approaches require only multiple forward processes that may be executed in parallel. Along this routine, finite difference stems from the definition of derivatives and can be generalized to higher-order and multivariate cases by Taylor's s expansion. For any differentiable loss function C(W; D) and small perturbation d E R", finite difference employs the forward difference scheme
L(W + o; D) L(W; = 5 W + o(1)))l),
(2)
where VW C(W: D) is a scaled directional derivative along Furthermore, we can use the central difference scheme to obtain higher-order residuals as
L(W + D) L(W - 8; D) = 28 VwC(W; (D) + o(1)))2)
(3)
Finite difference formulas typically used to estimate quantities such as gradient norm or Hessian trace, where 5 is sampled from random projection vectors (Pang et al., 2020).
BAFFLE: BACKPROPAGATION- FEDERATED LEARNING
In this section, we introduce zero-order optimization techniques into FL and develop BAFFLE, backpropagation-Fi -free federated learning framework that uses multiple forward processes in place of backpropagation. An initial attempt is apply finite difference as the gradient estimator. To estimate the full gradients, we need to perturb each parameter w E W once to approximate the partial derivative ac(W:D) causing the forward computations to grow with n (recall that W R") and making it difficult to scale to large models. In light of this, we resort to Stein's 's identity (Stein, 1981) to obtain unbiased estimation of gradients from loss differences calculated on various perturbations. As depicted in Figure 1, BAFFLE clients need only download random seeds and global parameters update, generate perturbations locally, exccute multiple forward propagations and upload loss differences back to the server. Furthermore, we also present convergence analyses of BAFFLE, which provides guidelines for model design and acceleration of training progress.
Pwr
3.1
UNBIASED GRADIENT ESTIMATION WITH STEIN'S IDENTITY
Previous work on iign-baeed optimization (Moulay et al., 2019) demonstrates that deep networks can be effectively trained if the majority of gradients have proper signs. Thus, we propose performing forward propagation multiple times on perturbed parameters, in order to obtain a stochastic estimation of gradients without backpropagation. Specifically, assuming that the loss function L(W; (D) is continuously differentiable w.r.t. W given any dataset D. which is true (almost everywhere) for deep networks using non-linear activation functions, we define a smoothed loss function
Lo (W; DD) == 1B) + D;
(4)
where the perturbation ō follows Gaussian distribution with zero mean and covariance I Given this, Stein (1981) proves the Stein's identity (we recap the proof in Appendix A.1), formulated as
WwC. W: D) (0,
AL(W, 6: 6:
20
where AC(W, D; = L(W + D) L(W-((WWW- 6; DD) is the loss difference, Note that computing a loss difference only requires the execution of two forward processes L(W+ D) and L(W- 6: D) without backpropagation. It is straightforward to show that L. D) is continuously differen- tiable for any a 2 and WwC, ID) converges uniformly as a - 0; hence, follows that V WWW: ID) = lim Vw (W; D). Therefore, we can obtain a stochastic estimation of gradi- ents using Monte Carlo approximation by 1) selecting a small value of a; 2) randomly sampling K perturbations from. (/(0, I as (ōx J and 3) utilizing the Stein's 's identity in Eq. (5) to calculate
VWL(W:D)
1 K
Σ
K
8% AL(W, 8c: D)
(6)
20
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1 BBackpopaagation-iie federated learning (BAFFLE)
1: Notations: Se denotes the operations done on servers; CI denotes the operations done on clients;
TEE for the TEE module; and = denotes the communication process.
2: Inputs: C clients with local dataset {De] containing N₂ input-label pairs, N Ees No:
learning rate 7) training iterations T. perturbation number K, noise scale
3: Se: initializing model parameters W +- Wo:
4: Se: encoding the computing paradigm into TEE TEE A 6;ID);
# optional
5: for t = 0 to T do
6: Se > all Cl: downloading model parameters W, and the computing paradigm:
7: Se = all Cl: downloading the random seed s;;
# 4 Bytes
8: Se: sampling
perturbations (6 { from from.W(0 I²I) using the random seed 85;
9:
all Cl: negotiating a group of zero-sum noises { for secure aggregation;
10:
for c to C' do
11:
Cl: sampling perturbations (&x L from (0, I²I) using the random seed 8t:
12:
Cl: computing TEE AL(W,. e, du; De)) via forward propagation for each k;;
K
13:
Cl +
Se: uploading outputs
TEE AC(W, De) +
N No
EC
#4*K Bytes
k=1
end for
Se: aggregating AC(W,, 66) Eo
TEE o AL(W, 8c: De) + No co
for each k;;
N
16: Se: computing Vw, ((W.) R
E 6e 20
AL(W,, 8%);
17: Se: W < W. VW ((W.):
18: end for
19: Return: final model parameters WT.
3.2 OPERATING FLOW OF BAFFLE
Based on the forward-only gradient estimator V L(W:D) derived in Eq. (6), we outline the basic operating flow of BAFFLE system as Algorithm 1, which consists of the following:
Model initialization. (Lines 3-4, done by server) The server initializes the model parameters to Wn and optionally encodes the computing paradigm of loss differences AL(W, 8; TD) into the TEE module (see Appendix C for more information on TEE);
Downloading paradigms. (Lines 6~7, server + all clients) round the server distributes the most recent model parameters W, (or the model update AW, W, W,-1) and the computing paradigm all the clients. In addition, in BAFFLE, the server sends a random seed st (rather than directly sending the perturbations to reduce communication burden);
Local computation. (Lines 11~12, done by clients) Each client generates K perturbations { locally from N (0, I²)) using random seed and executes the computing paradigm to obtain loss differences. K is chosen adaptively based on clients' upload bandwidth and computation capability;
Uploading loss differences. (Line 13, all clients = server) Each client uploads K noisy outputs (AC(W,, ,. 5u: ID.)) + N ) noise e, negotiated by all clients be zero-sum. The Bytes uploaded for K noisy outputs 4 K;
N. k=1 the server, where each output a floating prating-pointt point number and the
Secure aggregation. (Lines 15~16, done by server) In order to prevent the server from recovering the exact loss differences and causing privacy leakage (Geiping et al., 2020), we adopt the secure aggregation method (Bonawitz et al., 2017a) that was originally proposed for conventional FL and apply to BAFFLE. Specifically, all clients negotiate a group of noises {Eo]e=1 satisfying D =
0. Then we can reorganize our gradient estimator as
VwiL(W,) K
N
dz AL(W, 8c: De) 20
1 K
Σ
8, AC(W,, c, 82),
2
where AC(W,, 5z) = E N. N AC(W,, 5k) = DE N. AL(W, 5n D.) and Eq. (7) holds. Thus, the server can correctly aggregate AL(W,, 0x) and protect client privacy without recovering individual AL(W,, e, 8n; ID..)
AA(WW 6n ID.) + N En). Since { are zero-sum, there N.
4

---

Under review as a conference paper at ICLR 2023
Remark. After calculating the gradient estimation Vw, ((W.), the server updates the parameters to W++1 using techniques such as gradient descent with learning rate Similar to the discussion in McMahan et al. (2017), the BAFFLE form presented in Algorithm corresponds where Lines 11~12 execute once for each round t. We can generalize BAFFLE to an analog of FedAvg. in which each client updates its local parameters multiple steps using the gradient estimator 2, Vw, ((W,, D.) derived from A,,W,, dx; De) via Eq. (6), and upload model updates to the server.
FedSGD
3.3
CONVERGENCE ANALYSES
Now we analyze the convergence rate of our gradient estimation method. For continuously differ- entiable loss functions, we have (wL(W;D) = lim Co W;D), so we choose a relatively small value for a. The convergence guarantee can be derived as follows:
Theorem 1. (Proof in Appendix A.2) For perturbations {5x K
lid o the empirical
k=
~
covariance matrix Σ == Ko Ex=1 5gō% and mean is 8 = K
Ex
K
8z. Then for any W e R",
k=1
the relation between VWL(W: IB) and the true gradient V L LWW D3 can be written as
VWL(W:D) D) WV o(0); s.t. EJΣ| I, E(ō) 0,
(8)
where a is a small value and the central difference scheme in Eq. (3) holds.
When expectation is applied to both sides of Eq. (8), we obtain E w ((; (w D), which degrades to Stein's 's identity. To determine the convergence rate w.r.t. the value of K, we have
Theorem 2. (Adamczak et al. (2011) 1)) With overwhelming probability, the empirical covariance matrix satisfies the inequality [ Il2 S CoV k.. where || denotes the operator 2-norm for matrix and Cu is an absolute positive constant.
Note that in the finetuning setting, represents the number of trainable parameters, excluding frozen parameters. As concluded, V V (WW; D) provides an unbiased estimation for the true gradients with convergence rate of O ( (v K Empirically, CwC(W; D) is used as a noisy gradient to train models, the generalization of which has been analyzed in previous work (Zhu et al., 2019; Li et al., 2020a).
4 EXPERIMENTS
We evaluate the performance of BAFFLE on four benchmark datasets: MNIST (LeCun et al., 1998), LIFAR-10/100 (Krizhevsky & Hinton, 2009) and OfficeHome (Venkateswara et al., 2017). We consider three models: 1) LeNet (LeCun al., 1998) with two convolutional layers as the shallow model (2.7 x 10a parameters); 2) WideResNet (Zagoruyko & Komodakis, 2016) with depth = 10 and width = 2 (WRN-10-2) as the light weight deep model (3.0 10y parameters) and 3) MobileNet (Howard et al., 2017) as the deep neural networks (1.3 x 10? parameters) that works on ImageNet. To perform a comprehensive evaluation of BAFFLE, we simulate three popular FL scenarios (Caldas et al., 2018b) with the participation tools from FedLab (Zeng et al., 2021): iid participations, label non-iid participations and feature non-iid participations. For iid participations, we set the client number C = 10 and use uniform distribution to build local datasets. Then we evaluate our BAFFLE on MNIST and CIFAR- 10/100 under both batch-level (FedSGD) and epoch- level (FedAvg) communication settings. For label non-iid participations, we set client number C' == 100, use Dirichlet distribution to build clients. For feature non-iid participations, we build clients from the prevailing domain adaptation dataset OfficeHome, which contains 65 categories from 4 different domains, i.e. Art, Clipart, Product and Real-world. We set the total client number to
C = 40 and generate 10 clients from each domain. As results, report Top-1 accuracy for MNIST, CIFAR-10 and OfficeHome and Top-5 accuracy for OfficeHome and CIFAR-100.
4.1 EXPERIMENTAL SETTINGS
Following the settings in Section 2. we use FedAVG to aggregate gradients from multiple clients and use SGD-based optimizer to update global parameters. Specifically, we use Adam (Kingma & Ba, 2015) to train random initialized model with 3 = (0.9, 0.99), learning rate 0.01 and epochs 20/4 40 for MNIST and CIFAR-10/100, For OfficeHome, we adapt the transfer learning strategy (Huh et al., 2016 by loading the pretrained model ImageNet and finetuning the final layers with Adam, but setting learning rate 0.005 and epochs 40. In BAFFLE, the perturbation scale o and number K the most important hyperparameters. As shown in Theorem with less noise and more samples,
5

---

Under review as a conference paper at ICLR 2023
Table 1: The classification accuracy (%) of BAFFLE in iid scenarios (C 10) and epoch-level communication settings with different values (K1/ K₂ annotations mean using K for MNIST and K2 for CIFAR-10/100), In this configuration, each client updates its local model based BAFFLE estimated gradients and uploads model updates to the server after an entire epoch on the local dataset. The four guidelines work well under epoch-level communication settings.
LeNet
WRN-10-2
Settings
MNIST
CIFAR-10 CIFAR-100
MNIST CIFAR-I0 CIFAR-100
100/200
87.27
48.78
41.54
88.35
52.27
46.61
K
200/500
89.48
51.82
45.68
89.57
55.59
51.65
500/1000
92.18
53.62
48.72
95.17
58.63
53.15
w/o EMA
85.06
47.97
36.81
85.89
50.01
45.86
Ablation
ReLU
81.55
44.99
39.49
79.08
49.76
44,44
Study
SELU
86.18
48.65
37.34
76.44
43.37
41.79
(100/200)
Central
76.02
45.97
36.53
77.45
42.89
39.62
Ground truth
94.31
58.75
54.67
97.11
62.29
60.08
MNIST
Accatary
Accaraty
Esact Gradiente 500
Excect Gradients
0.8
0.8
200
X 200
0.7
0.7
Number of Eparia
Number Epocha
0.0
CIFAR-10
0.5
Accuracy
0.5
Esacs Gondicuts
Goadicuts
0.4
6000
0.4
K - 6000
K 2000
0.3
1000
0.3
NC 1000
0.2
500
0.2
K
200
K 200
Number of Epurhs
Number of Egpnciis
LeNet
WRN-10-2
Figure 2: The classification accuracy (%) BAFFLE in iid scenarios (C 10) and batch-level communication settings with various K values. We treat the models trained by exact gradients on conventional FL systems as the ground truth. On different datasets and architectures, our BAFFLE achieves comparable performance to the exact gradient results with reasonable K.
the BAFFLE will obtain accurate gradients, leading to improved performance. However, there exists a trade-off between accuracy and computational efficiency: an extremely small will cause the underflow problem (Goodfellow et al., 2016) and large K will increase computational cost. In practice, we empirically set 10-1 because it is the smallest value that does not cause numerical problems in all experiments, and works well on edge devices with half-precision floating-pon -pint numbers. We also evaluate the impact of K across a broad range from 100 to 5000.
For general family of continuously differentiable models, we analyze their convergence rate of BAFFLE in Section 3.3. Since deep networks are usually stacked with multiple linear layers and non-linear activation, this layer linearity can be utilized improve the accuracy-efficieng, trade-off. Combining the linearity property and the unique conditions in edge devices (e.g., small data size and half-precision format), we present four guidelines for model design and training that can increase accuracy without introducing extra computation (for the details of linearity analysis, see Appendix D):
Using twice forward difference (twice-FDD scheme rather than central scheme. Combining difference scheme Eq. (2) and Eq. (3), we find that by executing twice many forward inferences (i.e. W = 5), the central scheme achieves lower residuals than twice-Fi FD, despite the fact that twice-FD can benefit from additional sample times. With the same forward times (e.g.. 2K),, determining which scheme performs better is a practical issue. As shown in Appendix D. we find that twice-FD performs better in all experiments, in part because the linearity reduces the benefit from second-order residuals.

---

Under review as a conference paper at ICLR 2023
Table 2: The classification accuracy (%) of BAFFLE in label non-iid scenarios (C 100) and epoch-level communication settings with different K values. We employ Dirichlet distribution to ensure that each client has unique lahel distribution.
LeNet
WRN-10-2
Settings
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR- 100
200
35.21
28.12
39.53
30.44
K
500
38.
30.92
41.69
32.89
1000
39.71
33.35
43.42
34.08
Ground truth
44.41
38.43
51.18
40.85
1.0
1.0
0.9
0.9
MNIST
Aocuraer
BAFFLE w/oo EMA
BAFFLE
w/o EMA
0.8
ReLU
0.8
ReLU SELU
0.5
0.5
Central Stheme
Central Stheme
0.4
0.4
0
10
15
20
10
20
Nomber of Epocts
Number Egocio
0.50
0.90
CIFAR-10
Accurac
BAFFLE w/o EMA
BAFFLE
0.35
0.45
w/a EMA
0.30
ReLU
0.60
ReLU
SPLU
0.25
Central Scheme
0.35
Central Schemm
0.20
0.50
30
Number Fgoda
Number of Pqorta
LeNet
WRN-10-2
Figure 3: The ablation study of BAFFLE guidelines. with K 100 on MNIST and K 500 on CIFAR-10. As seen, twice-FD, Hardswish, and EMA all improve performance with the same computational cost. EMA reduces oscillations in training by lessening the effect of white noise.
Using Hardswish in BAFFLE. ReLU is effective when the middle features (h(-) denotes the feature mapping) have the same sign before and after perturbations, i.e. h(W + 5) h( ((W)& > 0. Since ReLU not differentiable at zero, the value jump occurs when the sign of features changes after perturbations, i.e. h(W + 5) h(W) < O. We use Hardswish (Howard et al., 2019) to overcome this problem as is continuously differentiable zero and easy to implement on edge devices.
Using exponential moving average (EMA) to reduce oscillations. As shown in Theorem 1, there exists zero-mean white-noise 8 between the true gradient and our estimation. To smooth out the oscillations caused by white noise, we apply EMA strategies from BYLL (Grill et al., 2020) to the global parameters, with smoothing coefficient of 0.995.
Using GroupNorm as opposed to BatchNorm. On edge devices, the dataset size is typically small, which leads to inaccurate batch statisties estimation and degrades performance when using BatchNorm. Thus we employ GroupNorm (Wu & He, 2020) to solve this issue.
4.2 PERFORMANCE ON IID CLIENTS
Following the settings in Section 4.1, we evaluate the performance of BAFFLE in the iid scenarios. We reproduce experiments on the backpropagation-be based FL. systems with the same settings and use them as the ground truth. We refer to the ground truth results as exact gradients and report the training process BAFFLE in Figure 2. The value of (e.g.. g.. 200 for LeNet and 500 for WRN-10-2) significantly less than the dimensions of parameter space (e.g., 2.7 x 10a for LeNet and 3 x 1 for WRN-10-2). Since the convergence rate to the exact gradient is O (V increasing K decreases. For instance. increasing K from 2000 to 5000 CIFAR-10 with WRN-10-2 barely improves accuracy by 2%. Given that the convergence rate normal distribution is O R the sampling efficiency may be improved by choosing alternative distribution for perturbations.
TE K the marginal benefit of
7

---

Under review as a conference paper at ICLR 2023
Table 3: The Top-1 Top-5 classification accuracy (%) of BAFFLE on OfficeHome with feature non-iid participations (C = 40) and epoch-level communication settings. We utilize the pretrained MobileNet, freeze the backbone parameters, and retrain the final classification layers.
Domains
Settings
Avg.
Art
Clipart
Product
Real World
20
44.75
69.46
52.48
66.63 777
63.78 87.83
56.91
80.24
50
47.87
71.32
53.43 76.83
71.28
91.74
67.02 89.95
59.90
82, 46
K
100
50.32 74.42
57.43 80.73
74.19
93.02
69.53
90.43
62.87
84.65
200
51.42
76.64
60.98
86.41
76.05
94.42
71.51
[93.14
64.98 87.65
500
53.33
77.85
62.58
86.64
78.84
95.23
73.17
(9.885
66.98
88.4 40
Ground truth
55.71 88..43
65.13
88.65
82.44(96.05
77.19|95.04
70. 12 90.04
Hasdam Nove
Randen Noms
Rancom
LeNet
2..08
0.01
0.01
0.01
-0.010
0,385
0:00
6:055
6100
-4
-.000
-4 166
0.006
0.518
0.07
Readoms Note
Rei, Dat
0.67
Raddon Noice
Red Thes
6.07
Reel Deto
Readiess Noae
WRN-10-2
rebaniry
Putesiity
0,000
a.
6:33
-1.00
11.005
0.000
0.335
6000
0.00
+0/335
0.030
00005
BM0
1.000
-3.500
0.0M0
0.006
0.000
(a) MINIST
(b) C1FAR-10
(c)
CLEAR-100
Figure 4: The robustness of BAFFLE to inference attacks. For real data, we randomly sample some input-label pairs from the validation dataset. For random noise, we generate input-label pairs from standard normal distribution. We sample 500 perturbations ō from N/0, o I)), collect the values of AC(W, ó; D) for real data and random noise separately, and compare their distributions.
Ablation studies. As depicted in Figure 3, we conduct ablation studies for BAFFLE to evaluate the aforementioned guidelines. In general, twice-FD, Hardswish and EMA can all improve the accuracy. For two difference schemes, we compare the twice-FD to central scheme with the same computation cost and show that the former outperforms the later, demonstrating that linearity reduces the gain from second-order residuals. As to activation functions, Hardswish is superior to ReLU and SELU because it is differentiable at zero and vanishes to zero in the negative part. Moreover, EMA enhances the performance of training strategies by reducing the effect of white noise.
Communication efficiency. Each client uploads a K dimeensiional vector the server and downloads the updated global parameters during the communication round of BAFFLE. Since K is significantly less than the parameter amounts (e.g.. 500 versus 0.3 million), BAFFLE reduces data transfer by approximately half when compared to the batch-level communication settings (FedSGD) in backpropagation-bue based FL system. In order to reduce communication costs, the prevalent FL system requires each client to perform model optimization on the local training dataset and upload the model updates to the server after a specified number of local epochs. BAFFLE can also communicate at the epoch level by employing an O(n) additional memory store the perturbation in each forward process and estimate the local gradient using Eq. (6). Each client optimizes the local model with SGD and uploads local updates after a number of epochs. As shown in Table 1, we evaluate the performance of BAFFLE under one-epoch communication settings. As epoch- level communication more prevalent in the real- roal-world FL, all the following experiments will be conducted in this context.
4.3 PERFORMANCE ON NID-IID CLIENTS
Following the settings in Section 4. we evaluate the performance of BAFFLE in both label non-iid and feature non-iid scenarios. For label non-iid scenarios, we use the CIFAR- datasets and employ Dirichlet distribution to ensure that each client has a unique label distribution. We evaluate the performance of BAFFLE with 100 clients and various K values. As seen in Table 2. the model suffers a significant drop in accuracy (e.g., 14% in CIFAR-IO and 16% in CIFAR-100) due to the label non-iid effect. For feature non-iid scenarios, we construct clients using the OfficeHome dataset and use MobileNet strategy to train MobileNet, i.c., we load the parameters pretrained on ImageNet, freeze the backbone parameters, and retrain the classification layers. The accuracy decrease is approximately 3% ~ 5%.
the deep model. As seen in Table 3, we use the transfer learning
8

---

Under review as a conference paper at ICLR 2023
Table The GPU memory cost (MB) of vanilla backpropagation and BAFFLE, respectively. Here 'min~-mxx' denotes the minimum and maximum dynamic memory requirements for BAFFLE. We
also report the ratio of vanilla backpropagation to BAFFLE's maximal memory cost.
CIFAR-10/100
OfficeHlome/lmageNeet
Architectures
BP
BAFFLE
Ratio
BP
BAFFLE
Ratio
LeNet
1680
67-174
10.35%
2527
86~201
7.95%
WRN-10- 10-2
1878
75~1196
10.43%
3425
94~251
7.32%
MobileNet
2041
102-2177
10.63%
5271
121~2899
5.48%
4.4 COMPUTATION EFFICIENCY, MEMORY EFFICIENCY, AND ROBUSTNESS
BAFFLE uses K times forward passes instead of backward. Since the backward pass is about as expensive as Iwo normal forward passes (Hinton & Srivastava, 2010) and five single-precision accelerated forward passes Nakandala et al. (2020), BAFFLE results in approximately K 5 times the
computation expense of BP-based FL. Although BAFFLE results in 4 - 1 times extra computation cost, we show the cost can be reduced with proper training strategies, e.g., the transfer learning in Table 3 can reduce K to 20 on the MobileNet and the 224 x 224 sized OfficeHome dataset.
Morcover, BAFFLE can reduce huge memory cost on edge devices with the efficiency in static mem- ory and dynamic memory. The auto-differential framework is used to run BP on deep networks, which requires extra static memory (e.g.. 200MB for Caffe (Jia et al., 2014) and 1GB for Pytorch (Paszke et al., 2019a)) and imposes considerable burden edge devices such IoT sensors. Due to the necessity of restoring intermediate states, BP also requires enormous amounts of dynamic memory 5GB for MobileNet (Gao et al., 2020)). Since BAFFLE only requires inference, we can slice the computation graph and execute the forward calculations per layer (Kim et al., 2020). As shown in Table 4, BAFFLE reduces the memory cost 5% by executing inference-only computations layer-by-layer By applying kernel-wise computations, we can further reduce the memory cost to approximately 1% (e.g., 64MB for MobileNet (Truong et al., 2021)) 1)) which is suitable for scenarios with extremely limited storage resources, such as TEE.
Recent works exploit TEE to protect models from white-box attacks (Kim et al., 2020), which can defend against white-box attacks by preventing model exposure. However, due to the security guarantee, the usable memory of TEE is usually small (Truong et al., 2021) (e.g., 90MB on Intel SGX for Skylake CPU (McKeen et al., 2016)), which is typically far less than what backpropagation based FL system requires. In contrast, BAFFLE can execute in TEE due to its little memory cost (more details are in Appendix C). Membership inference attack and model inversion attack need to repeatedly perform model inference and obtain confidence values or classification scores (Shokri et al., 2017; Zhang et al., 2020). Given that BAFFLE provides stochastic loss differences AL(W, ö; D) associated with the random perturbation 8, the off-the-shel inference attacks may not perform on BAFFLE directly (while adaptively designed attacking strategies are possible to evade BAFFLE). We further select random samples from the validation dataset and generate random input pairs as (X. 9). As shown in Figure 4, it is difficult to distinguish between real data and random noise, indicating that it is difficult for attackers to obtain useful information from BAFFLE's outputs.
CONCLUSION AND DISCUSSION
Backpropagation is the gold standard for training deep networks, and it is also utilized by traditional FL systems. However, backpropagation is unsuited for edge devices due to their limited resources and possible lack of reliability. Using zero-order optimization techniques, we explore the possibility of backpropagation-fi ffree FL in this paper. We need to specify that there are scenarios in which clients are fully trusted and have sufficient computing and storage resources. In these situations, traditional FL with backpropagation is preferred over BAFFLE.
While our preliminary studies on BAFFLE have generated encouraging results, there are still a number of tough topics to investigate: (i) Compared to the models trained using exact gradients, the accuracy of models trained using BAFFLE is inferior. One reason is that we select small values of K (e.g., 500) relative to the number of model parameters (e.g., 3.0 x 10*); another reason is that gradient descent is designed for exact gradients, whereas our noisy gradient estimation may require advanced learning algorithms. (ii) The empirical variance of zero-order gradient estimators affects training convergence in BAFFLE. It is crucial to research variance reduction approaches, such as control variates and non- Gaussian sampling distributions. (iii) Stein's 's identity is proposed for loss functions with Gaussian noises imposed on model parameters. Intuitively, this smoothness is related to differential privacy in FL., but determining their relationship requires theoretical derivations.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jayadev Acharya, Clément I. Canonne, and Himanshu Tyagi. Inference under information constraints Lower bounds from chi-square contraction. IEEE Transactions on Information Theory, 2020.
Radoslaw Adamczak, Alexander E Litvak, Alain Pajor, and Nicole Tomczak-Jaegermann Sharp bounds on the rate of convergence of the empirical covariance matrix. Comples Rendus Mathema- tique, (49(3-4):195-200 2011.
Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communicationn efficient sgd via gradient quantization and encoding. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
Ravikumar Balakrishnan, Tian Li, Tianyi Zhou, Nageen Himayat, Virginia Smith, and Jeff Bilmes. Diverse client selection for federated learning via submodular maximization. In International Conference on Learning Representations (ICLR), 2022.
Leighton Pate Barnes, Yanjun Han, and Ayfer Özgür. Lower bounds learning distributions under communication constraints via fisher information. Journal of Machine Learning Research (JMLR), 2020.
Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi. Qsparse-local-sgd: Distributed sgd with quantization, sparsification and local computations. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through adversarial lens. In International Conference Machine Learning (ICML), 2019.
Kallista A. Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H. Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy-preserving machine learning. In ACM Conference on Computer and Communications Security (CCS), 2017a
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. arXiv preprint 1rXiv:1611.044500 2016.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. ACM SIGSAC Conference on Computer and Communications Securiry, 2017b. 20177.
Eitan Borgnia, Jonas Geiping, Valeriia Cherepanova, Liam Fowl, Arjun Gupta, Amin Ghiasi, Furong Huang, Micah Goldblum, and Tom Goldstein. Dp-instahide: Provably defusing poisoning and backdoor attacks with differentially private data augmentations. arXiv preprint arXiv:2103.02079, 2021.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miin and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http: petp://giin /ttp:///ithhb com/ com/ google/ jax.
Mark Braverman, Ankit Garg, Tengyu Ma, Huy L Nguyen, and David P Woodruff. Communication lower bounds for statistical estimation problems via a distributed data processing inequality. In ACM Symposium on Theory of Computing, 2016.
Sebastian Caldas, Jakub Konečny, H Brendan McMahan, and Ameet Talwalkar. Expanding the reach of federated learning by reducing client resource requirements. arXiv preprint arXiv:1812.072.10 201 8a.
10

---

Under review as a conference paper at ICLR 2023
Sebastian Caldas, Peter Wu, Tian Li, Jakub Konečný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. LEAF: A benchmark for federated settings. CoRR, 2018b.
Mingzhe Chen, Nir Shlezinger, H Vincent Poor, Yonina C Eldar, and Shuguang Cui. Communication efficient federated learning. Pruceedings of the National Academy of Sciences, 2021.
Yu Chen, Fang Luo, Tong Li, Tao Xiang, Zheli Liu, and Jin Li. A training-integrity privacy-preserving federated learning scheme with trusted execution environment. Information Sciences, 2020.
Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black-box detection of backdoor attacks with limited information and data. In IEEE International Conference on Computer Vision (ICCV), 2021.
Hubert Eichner, Tomer Koren, Brendan McMahan, Nathan Srebro, and Kunal Talwar. Semi-cyclic stochastic gradient descent. International Conference on Machine Learning (ICML), 2019.
Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without gradient. arXiv preprint cs/0408007, 2004.
Yanjic Gao, Yu Liu, Hongyu Zhang, Zhengxian Li, Yonghao Zhu, Haoxiang Lin, and Mao Yang. Estimating gpu memory consumption of deep learning models. In ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020.
Jonas Geiping. Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients-hov easy is it to break privacy in federated learning? In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Badih Ghazi, Ravi Kumar, Pasin Manurangsi, and Rasmus Pagh. Private counting from anonymous messages: Near-optimal accuracy with vanishing communication overhead. International Conference on Machine Learning (ICML), 2020.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach self-supervised learning. In NeurlPS 2020, 2020.
Jenny Hamer, Mehryar Mohri, and Ananda Theertha Suresh. Fedboost: communication-efficine algorithm for federated learning. In International Conference on Machine Learning (ICML), 2020.
Yanjun Han, Ayfer Özgür, and Tsachy Weissman. Geometric lower bounds for distributed parameter estimation under communication constraints. In Conference On Learning Theory (COLT), 2018.
Meng Hao, Hongwei Li, Xizhao Luo, Guowen Xu, Haomiao Yang, and Sen Liu. Efficient and privacy-enhanced federated learning for industrial artificial intelligence. IEEE Transactions on Industrial Informatics, 2019.
Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein. Hubert Eichner, Chloé Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint 10X.::8111006600 2018
Chaoyang He, Murali Annavaram, and Salman Avestimehr. Group knowledge transfer: Federated learning of large cnns at the edge. In Advances Neural Information Processing Systems (NeurIPS), 2020.
Di He, Wenlei Shi, Shanda Li, Xiaotian Gao, Jia Zhang, Jiang Bian, Liwei Wang, and Tie-Yan Liu. Learning physics-informed neural networks without stacked back- bock-propagation. arXiv preprint 09440 2022.
Geoffrey Hinton and Nitish Srivastava. Csc321: Introduction to neural networks and machine learning. Lecture, 10, 2010.
11

---

Under review as a conference paper at ICLR 2023
Samuel Horváth, Chen- Yu Ho, Ludovit Horvath, Atal Narayan Sahu, Marco Canini, and Peter Richtárik. Natural compression for distributed deep learning, arXiv preprint 1905. 10988, 2019.
Andrew Howard, Ruoming Pang, Hartwig Adam, Quoe Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu, Searching for mobilenetv3. In IEEE International Conference on Computer Vision (ICCV), 2019.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, 2017.
Yangsibo Huang, Samyak Gupta, Zhao Song, Kai Li, and Sanjeev Arora. Evaluating gradient inversion attacks and defenses in federated learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Mi-Your Young Huh, Pulkit Agrawal, and Alexei A. Efros. What makes imagenet good for transfer learning? CoRR, 2016.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross B. Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Kien A. Hua, Yong Rui, Ralf Steinmetz, Alan Hanjalic, Apostol Natsev, and Wenwu Zhu (eds.), Proceedings of the ACM International Conference on Multimedia, 2014.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems federated learning. Foundations and Trends@ in Machine Learning. 14(1-2):1-21 2021.
Jiawen Kang, Zehui Xiong, Dusit Niyato, Yuze Zou, Yang Zhang, and Mohsen Guizani. Reliable federated learning for mobile networks. IEEE Wireless Communications, 2020.
Kyungtae Kim, Chung Hwan Kim, Junghwan" John" Rhee, Xiao Yu, Haifeng Chen, Dave Tian, and Byoungyoung Lee. Vessels: Efficient and scalable deep learning prediction on trusted processors. In ACM Symposium Cloud Computing, 2020.
Yejin Kim, Jimeng Sun, Hwanjo Yu, and Xiaoqian Jiang. Federated tensor factorization for computa- tional phenotyping. In ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD), 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 2016
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basee learning applied to document recognition. Proceedings of IEEE, 1998.
Jian Li, Xuanyuan Luo, and Mingda Qiao. On generalization enror bounds of noisy gradient methods for non-convex learning. In International Conference on Learning Representations (ICLR), 2020a.
Li Li, Yuxi Fan, Mike Tse, and Kuo-Yi Lin. A review of applications in federated learning. Computers & Industrial Engineering, 2020b.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In International Conference on Machine Learning (ICML), 2021.
Xiang Li. Kaixuan Huang, Wenhao Yang. Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations (ICI.), 2020c.
12

---

Under review as a conference paper at ICLR 2023
Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod K Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 2020a.
Yuhan Liu, Ananda Theertha Suresh, Felix Xinnan X Yu, Sanjiv Kumar, and Michael Riley. Learning discrete distributions: user vs item-level privacy. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Xinjian Luo, Yuncheng Wu, Xiaokui Xiao, and Beng Chin Ooi. Feature inference attack on model predictions in vertical federated learning. In IEEE International Conference on Data Engineering (ICDE), 2021.
Lingjuan Lyu, Han Yu, Xingjun Ma, Lichao Sun, Jun Zhao, Qiang Yang, and Philip S Yu. Privacy and robustness in federated learning: Attacks and defenses. arXiv preprint arXiv:2012.06337, 2020.
Jing Ma, Qiuchen Zhang, Jian Lou, Joyce C Ho, Li Xiong, and Xiaoqian Jiang. Privacy-preserving tensor factorization for collaborative health data analysis. In ACM International Conference on Information and Knowledge Management (CIKM), 2019.
Othmane Marfoq, Giovanni Neglia, Aurélien Bellet, Laetitia Kameni, and Richard Vidal. Federated multi-task learning under a mixture of distributions. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Frank McKeen, Ilya Alexandrovich, Ittai Anati, Dror Caspi, Simon Johnson, Rebekah Leslie-Hurd, and Carlos Rozas. Intel@ software guard extensions (intel@ sgx) support for dynamic memory management inside an enclave. Proceedings of the Hardware and Architectural Support for Security and Privacy, 2016.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüeea y Arcas. Communication -fffccctt learning of deep networks from decentralized data. In International Conference on Artificial Intelligence and Statisties (AISTATS), 2017.
Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations (ICLR), 2018.
Fan Mo. Hamed Haddadi, Kleomenis Katevas, Eduard Marin, Diego Perino, and Nicolas Kourtellis. Ppfl: privacy-preserving federated learning with trusted execution environments. In Annual International Conference Mobile Systems, Applications, and Services, 2021.
Arup Mondal, Yash More, Ruthu Hulikal Rooparaghunath, and Debayan Gupta. Flatee: Federated learning across trusted execution environments. arXiv preprint (rXiv:2/11.06800 2021.
Emmanuel Moulay, Vincent Léchappé, and Franck Plestan. Properties of the sign gradient descent algorithms. Inf. Sci., 492:29-39 -99999999999 2019.
Supun Nakandala, Kabir Nagrecha, Arun Kumar, and Yannis Papakonstantinou. Incremental and approximate computations for accelerating deep CNN inference. ACM Trans. Database Syst., 45, 2020.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In IEEE symposium on security and privacy (S&P). IEEE, 2019.
Yurii Nesterov and Vladimir Spokoiny. Random gradient-fiee free minimization convex functions. Foundations of Computational Mathematics, 2017.
Tianyu Pang, Kun Xu. Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu. Efficient learning of generative models via kiite-differenne score matching. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Accumulative poisoning attacks on real-time data. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
13

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, 2019a.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style. high-peeffrmance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019b.
Daniel Rothchild, Ashwinee Panda, Enayat Ullah, Nikita Ivkin, Ion Stoica, Vladimir Braverman, Joseph Gonzalez, and Raman Arora. Fetchsgd: Commanication-efficien federated learning with sketching. In International Conference on Machine Learning (ICML), 2020.
Mohamed Sabt, Mohammed Achemlal, and Abdelmadjid Bouabdallah. Trusted execution environ- ment: what it is, and what it is not. In IEEE Trustcom/BiggDaaas///SPA 2015.
Felix Sattler, Simon Wiedemann, communication--ffiicii federated learning from non-iid data. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 2019.
Müller, and Wojeiech Samek. Robust and
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 2013.
Sanjay Seetharaman, Shubham Malaviya, Rosni KV, Manish Shukla, and Sachin Lodha. Influence based defense against data poisoning attacks in online learning, arXiv preprint arXiv:2104.1323., 2021.
Hardik Sharma, Jongse Park, Naveen Suda. Liangzhen Lai, Benson Chau, Vikas Chandra, and Hadi Esmaeilzadeh. Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. In IEEE Annual International Symposium Computer Architecture (ISCA), 2018.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. IEEE symposium on security and privacy (S&P). IEEE, 2017.
Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Charles M Stein, Estimation of the mean of a multivariate normal distribution. The annals of Statistics, 1981.
Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can you really backdoor federated learning? arXiv preprint 191iv./911.000 1.07663, 2019.
Ananda Theertha Suresh, Yu Felix, Sanjiv Kumar, and H Brendan McMahan. Distributed mean estimation with limited communication. In International Conference on Machine Learning (ICML), 2017.
Florian Tramer and Dan Bonch. Slalom: Fast, verifiable and private execution of neural networks in trusted hardware. In International Conference on Learning Representations (ICLR), 2019.
Stacey Truex, Nathalie Baracaldo, Ali Anwar, Thomas Steinke, Heiko Ludwig, Rui Zhang, and Yi Zhou. hybrid approach to privacy-preserving federated learning. In ACM workshop on Artificial Intelligence and Security, 2019.
Jean- Baptiste Truong, William Gallagher, Tian Guo, and Robert J Walls. Memory-efficient deep learning inference in trusted execution environments. In IEEE International Conference on Cloud Engineering (IC2E), 2021.
Yaman Umuroglu. Lahiru Rasnayake, and Magnus Själander. Bismo: A scalable bit-serial ma- trix multiplication overlay for reconfigurable computing. In International Conference on Field Programmable Logic and Applications (FP1.), 2018.
14

---

Under review as a conference paper at ICLR 2023
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020a.
Hui-Po Wang. Sebastian Stich, Yang He, and Mario Fritz. Progfed: effective, communication, and computation efficient federated learning by progressive training. In International Conference on Machine Learning (ICML), 2022.
Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020b.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quanti- zation with mixed precision. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019a.
Luping Wang, Wei Wang, and Bo Li. Cmfl: Mitigating communication overhead for federated learning. IEEE International Conference on Distributed Computing Systems (ICDCS), 2019h
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard Yang. Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE Transactions on Information Forensics and Security, 2020.
Yuxin Wu and Kaiming He. Group normalization. Int. J. Comput. Vis., 128(3):742-755, 2020.
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. In International Conference on Learning Representations (ICLR), 2020.
Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Françoise Beaufays. Applied federated learning: Improving google keyboard query suggestions. arXiv preprint arXiv: 001.11222....0 2018.
Xue Yang, Yan Feng, Weijun Fang, Jun Shao, Xiaohu Tang, Shu-Tao Xia, and Rongxing Lu. An accuracy-lossless perturbation method for defending privacy attacks in federated learning. ACM Web Conference (WWW), 2022.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, 2016.
Dun Zeng, Siqi Liang, Xiangjing Hu, Hui Wang, and Zenglin Xu. Fedlab: A flexible federated learning framework. arXiv preprint arXiv:2107.1162) 2021.
Yuchen Zhang, John Duchi, Michael Jordan, and Martin J Wainwright. Information-theoreii lower bounds for distributed statistical estimation with communication constraints. Advances Neural Information Processing Systems (NeurIPS), 2013.
Yuheng Zhang, Ruoxi Jia, Hengzhi Pei, Wenxiao Wang, Bo Li, and Dawn Song. The secret revealer: Generative model-inversion attacks against deep neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Yuhui Zhang, Zhiwei Wang, Jiangfeng Cao, Rui Hou, and Dan Meng. Shufflefl: gradient- preseervig federated learning using trusted execution environment. In ACM International Conference on Computing Frontiers, 2021.
Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.0055: 2018.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior escaping from sharp minima and regularization effects. In International Conference Machine Learning (ICMI.), 2019.
15

---

Under review as a conference paper at ICLR 2023
A PROOFS
A.1 PROOF OF STEIN'S IDENTITY
We recap the proof of Stein's identity following He al. (2022), where
VwC (W; Vw B W( C(W + 6:
(2x) -. Vw / C(W + D; exp
||011
dô
2
(2x) 7. / ((W:)) Vw exp
IW
W
dW
20²
(2x)- 7. | ((W:D) W W
W
W
(9)
exp
dW
20
||
(2x) 7 [[W++ + D)
exp
d6
r²
2a²
8 C(W 6:D)
By symmetry, we change ö ôô and obtain
8
W D) - -E8-N(0,00))
L(W [)
(10)
o2
and further prove that
Vw EE-N((0.01)
6
L(W d:D))
- -B8-N(0,001)
[ 5: 1:)1)
8
02
8
AC(W 8; DD)
20
A.2 PROOF OF THEOREM 1
We rewrite the format of V (WWWW:D) as follows:
1
K
8x
WWC(W;D)
=
K
20
A 6z; D)]
(28) V LDW:
(using central scheme in Eq. (3))
K
20
k=1
(11)
K
1 K
1
T V
WWW:W +
Σ
o11 20
KØ²
K
k=1
K
8%
(V +
K
202
Then we prove
K k= 20 01()))111) = o(6). Suppose ò, (6x.1. r. 8k,n), ,6x......,,,.1. then we have
16
12
-
Σi
) Since Vi, den (0, 1), we have | (²) and EE [6ا11 n. So
with high probability, of 1 o(n). Substituting it into Eq. (11), we have with high probability,
K 5% K
o( 5, (2 8- o(n) = o(
202
16

---

Under review as a conference paper at ICLR 2023
where we regard constant for given model architecture. Finally, we prove E[6] 0 and E|Σ] I. It is trivial that 18(6] = since 8 NO, Koz I). For E
I, we can observe by
examining each of its entries
K K
1
K O
(12)
where we have used subscripts [Kg]] and [i] to denote the usual indexing of matrices and vectors.
Specifically, for diagonal entries (i.e., i = j). we observe K Ena >
distributes
as x (K),, which means E| pa) = -1 = [[] and Var 2 [1a]]] = k for non- con-diagonal entries (i.c.,
i * j), we have E| L Σx E deg Bejs
R Σx E/64ul Ed = 0 = Tas) due to the
independence between different dimensions in og.
B RELATED WORK
Along the research routine of FL., many efforts have been devoted to, e.g., dealing with non-III
distributions (Zhao et al., 2018; Sattler al., 2019; Eichner al., 2019; Wang et al., 2020b; Li et al., 2020c), multi-task learning (Smith et al., 2017; Marfoq et al., 2021), and preserving privacy of clients (Bonawitz al., 2016; 2017b; McMahan al., 2018; Truex et al., 2019; Hao et al., 2019; Lyu et al., 2020; Ghazi et al., 2020; Liu et al., 2020b). Below we introduce the work on efficiency and vulnerability in FL following the survey of Kairouz et al. (2021), which is more related to this paper.
Efficiency in FL. It is widely understood that the communication and computational efficiency is primary bottleneck for deploying FL in practice (Wang et al., 2019b; Rothchild et al., 2020; Chen et al., 2021; Balakrishnan et al., 2022; Wang et al., 2022). Specifically, communicating between the server and clients could be potentially expensive and unreliable. The seminal work of Konečný et al. (2016) introduces sparsification and quantization to reduce the communication cost, where several theoretical works investigate the optimal trade-off between the communication cost and model accuracy (Zhang et al., 2013; Braverman et al., 2016; Han et al., 2018; Acharya et al., 2020; Barnes et al., 2020). Since practical clients usually have slower upload than download bandwidth, much research interest focuses on gradient compression (Suresh al., 2017; Alistarh et al., 2017; Horváth et al., 2019; Basu et al., 2019). On the other hand, different methods have been proposed to reduce the computational burden of local clients (Caldas et al., 201 8a; Hamer et al., 2020; He et al., 2020), since these clients are usually edge devices with limited resources. Training paradigms exploiting tensor factorization in FL can also achieve promising performance (Kim al., 2017: 7; Ma et al., 2019).
Vulnerability in FL. The characteristic of decentralization in FL is beneficial to protecting data privacy of clients, but in the meanwhile, providing white-box accessibility of model status leaves flexibility for malicious clients to perform poisoning/backdoor attacks (Bhagoji et al., 2019; Bag- dasaryan et al., 2020; Wang et al., 2020a; Xie et al., 2020; Pang et al., 2021), model/gradient inversion attacks (Zhang et al., 2020; Geiping et al., 2020; Huang et al., 2021), and membership inference attacks (Shokri et al., 2017; Nast et al., 2019; Luo et al., 20 To alleviate the vulnerability in FL, several defense strategies have been proposed via selecting reliable clients (Kang et al., 2020), data augmentation (Borgnia et al., 2021), 1), update clipping (Sun et al., 2019), robust training (Li et al., 2021), model perturbation (Yang et al., 2022), detection methods (Seetharaman et al., 2021; Dong et al., 2021), 1), and methods based on differential privacy (Wei et al., 2020), just to name a few.
C TRUSTED EXECUTION ENVIRONMENTS
A trusted execution environment (TEE) (Sabt et al., 2015) is regarded as the ultimate solution for defending against all white-box attacks by preventing any model exposure. TEE protects both data and model security with three components: physical secure storage to ensure the confidentiality, integrity, and tesper-re of stored data; a root of trust to load trusted code; and a separate kernel to execute code in an isolated environment, system is able to train deep models without revealing any model specifics. However, due to the security guarantee, the usable memory of TEE is typically small (Truong al., 2021) (e.g., 90MB
illustrated in Figure 5. Using TEE, the FL
17

---

Under review as a conference paper at ICLR 2023
Trusted Execution Environment (TEE)
Security Storage
Root of Trust
D
Separation Kernel
Input
Output
AC(W, 8;
-(W 6;
Data
Model
Inference
Noise
-L(W - D)
Figure 5: A sketch map of trusted execution environments.
on Intel SGX for Skylake CPU (McKeen et al., 2016)), which considerably less than what deep models require for backpropagation (e.g.. 5GB for VGG-16 (Gao et al., 2020)).
D CONVERGENCE ANALYSES OF DEEP LINEAR NETWORKS IN BAFFLE
We analyze the convergence of BAFFLE in Section 3.3 using a general technique applicable to any continuously differentiable models corresponding to the loss function L(W; Since deep networks are the most prevalent models in FL, which has strong linearity, it is simpler to investigate the convergence of deep linear networks (Saxe et al., 2013).
Consider two-layer deep linear network in classification task with L categories. We denote the model parameters as { 11, W2). where the first layer W1 E I2 R in the second layer W2 RLxn consists of L vectors related to the L categories the input data X E mx with label y. we train the deep linear network by maximizing the classification score on the y-th class. Since there is no non-linear activation in deep linear networks, the forward inference can be represented as h = w W X, and the loss is hh. It is easy to show (Xw?)T We sample 51. 82 from noise generator W0, d²I), where 81 E R and 82 E R X Let (601. 82) == (w² + ( + 8, ) we discover that the BAFFLE estimation in Eq. (6) follows the same pattern for both forward (2) and central schemes (3):
(w/) and w E RRxx. For
that ah 2w (W and ZW,
8h
Aich(6 52) h(ō1, 52) h(0, 0);
A 82) 1(,,, 82) h(--1,
(13)
Amdh( 62) Aen 52)
(w 6 WX).
20
This equivalent form in deep linear networks illustrates that the residual benefit from the central scheme is reduced by the linearity. hence the performance of the two finite difference schemes described above is same deep linear networks. We refer this characteristic as FD scheme independence. We also find the property of a independence, that is, the choice of does not effect the results of finite difference, due to the fact that é1 and & follow the standard normal distribution.
Based the findings from Eq. (13), we propose the following useful guideline that improves accuracy under the same computation cost: Using twice forward difference (twice-FD) scheme rather than central scheme. Combining the forward scheme (2) and central scheme (3), we find that the central scheme produces smaller residuals than the forward scheme by executing twice many forward inferences, i.e. W 8. With the same forward inference times (e.g.. 2K), one practical difficulty is to identify which scheme performs better. We find that the forward scheme performs better in all experiments, in part because the linearity reduces the benefit from -ccond-order residuals, as demonstrated by Eq. (13).
18


Reviews:
[{"review_id": "xUaV8UryOl5", "reviewer": "Reviewer_7JBh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose BAFFLE, backpropagation-free federated learning (FL) framework that uses multiple forward processes instead\nof backpropagation. The main idea is to use Monte-Carlo approximation for stein's identity to approximate the analytical gradient. BAFFLE is mainly designed for two specific setups: low-resource edge devices, and FL systems with malicious clients. The authors prove convergence analysis for their estimator and show the effectiveness of their method in several FL setups (standard classification, robustness to attacks, and communication efficiency).", "summary_of_the_review": "In my opinion this is a nice paper that deals with an interesting learning setup. The proposed solution is elegant and seems to be relatively efficient.\n", "strength_and_weaknesses": "Strength:\n* First, it is written clearly and relatively easy to follow. \n* Second, the goal of the paper (addressing cases of low-resource end devices and robustness to malicious clients) is important and refreshing. \n* Third, the method, which is relatively simple, is constructed in a thoughtful way to adjust for FL setups. It seems to work in a descent way compared to the standard FedAvg. \n* Forth, although not mandatory in my opinion, Thm 1 & 2 are a nice addition. \n\nWeaknesses/Questions:\n* As the authors stated, there is a noticeable drop in the model accuracy compared to FedAvg, but in my opinion it is fine for a first paper of this kind.\n* Some information is missing, how did you perform the data split between the clients? There are multiple strategies for that. How did you optimized the hyper-parameters of the model? Was there a validation set? The authors should present the Top-1 results for CIFAR-100 for completeness (even though they are probably not so good). \n* Did you consider Softplus as a continuously differentiable alternative to Relu?\n* Did you test how will the model perform in Personalized FL? It is especially interesting to show plots similar to those in Fig. 4 for real datasets that come from a different class distribution (as in PFL).\n* How will the model perform under a more realistic FL setup such as having 100 or 500 clients?\n* Can you quantify the computation gain (in terms of FLOPS or wall-clock time) compared to standard backpropagation?\n* A comparison to more recent baselines other than FedAvg is missing in my opinion.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "f4UcFPaewV", "reviewer": "Reviewer_EJwi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a zeroth order Federated Learning method BAFFLE using forward only computations to replace back-propagation. The method intends to reduce memory storage and strengthen privacy against white-box vulnerability. The soundness of the finite difference method is justified by theory, experiments show that the method achieve comparable performance to conventional FL methods.    ", "summary_of_the_review": "The paper proposes a zeroth order Federated Learning algorithm which is back propagation free. The main idea is to replace the gradient evaluation by zeroth order finite difference. This is very well known in zeroth order optimization literature. The more challenging part is to provide guidance on how to set the discretization stepsize and the number of samplings where the paper fail to provide any theoretical guidance. In particular, the more samples we draw, the more accurate the gradient estimation is, but the computational expenses will increase. The balance between the precision on the gradient estimation and the computational cost is a key for zeroth order method, which is a major omission of the paper. \n\nBesides the lack of convergence analysis, the experiment is also insufficient. Only shallow network are considered (two layers), it is expected that when the network goes deeper the gradient is more difficult to be estimated by zeroth order method (noise level increase), hence it will not be as competitive as gradient base method. ", "strength_and_weaknesses": "Strength: \n- clear presentation\n\nWeakness: \n- Lack of novelty \n- Lack of convergence analysis \n- Lack of guidance on parameter choices and computational comparison \n- Limited experiments: shallow networks, iid environment ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HhMxw5DYjA", "reviewer": "Reviewer_wGpR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to train federated learning using zero-order optimization. They use a gradient-free optimization approach and evaluate it on several benchmarks.", "summary_of_the_review": "The authors do now show empirical evidence to support their claims and as such I do not think this paper should be accepted.", "strength_and_weaknesses": "Strengths:\n- The problem the paper addresses is important. A common use case for FL is with clients with limited computing. As such, the reduction in memory by avoiding backdrop can be significant.\n- The paper is clear and easy to follow\n\nWeaknesses:\n- Limited novelty: Mostly applies an existing zero-order optimization algorithm to FL\n- Limited experimentation: Only uses LeNet, which has very bad performance on cifar10/100 even for the ground truth model and WRN only on MNIST. Not enough to be convincing.\n- Inexact/misleading theoretical results: The authors ignore the dimension factor in both theorem 1&2 which is a critical factor especially as it is much larger than $\\sqrt{k}$. Also, in theorem 2 what we care about is the operator 2 norm of $\\hat{\\Sigma}$, so just the fact that each element has $1/\\sqrt{K}$ std around the desired value isn't enough (again will depend on the dimension). Ignoring the dimension is misleading, as there is no theoretical justification to use this approach with the given K values. The approach may empirically work well, but the theory doesn't support it.\n- Exaggerated claims: \n   The authors in several cases raise claims that, in my opinion, are not supported by the empirical data. For example, they claim that \n   \"BAFFLE achieves comparable performance to conventional FL using a relatively small value of K\". In CIFAR10/100 we see about 7% \n   drop in accuracy even when using K=500, i.e. 1000 forward passes. Running 1000 forward passes on low resource compute is a \n   considerable amount and 7% drop is a considerable degradation in performance. \n\n   They also \"assert that it is difficult to perform inference attack on BAFFLE\". The support for that is very limited at best. They show the \n   distribution of $\\Delta \\mathcal{L}$ is the same between val data and random noise, but I do not agree this shows robustness to \n   inference attacks. The model is trained during many update steps and has to acquire information regarding the task to succeed. The \n   authors need to show that reasonable attacks fail to claim such robustness. \n- Unclear use of TEE. TEE from my understanding is part of the infrastructure to run FL and is agnostic to the exact algorithm. As such, it is not clear why is it important to include as part of the work. If the TEE details are relevant, it should be explained better.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject