Differentially Private Learning with Adaptive Clipping
Authors: Galen Andrew
Om Thakkar
Hugh Brendan McMahan
Swaroop Ramaswamy
Venue: NeurIPS 2021 Poster (2021)
Keywords: Differential privacy
federated learning
privacy
security
federated

Abstract: Existing approaches for training neural networks with user-level differential privacy (e.g., DP Federated Averaging) in federated learning (FL) settings involve bounding the contribution of each user's model update by {\em clipping} it to some constant value. However there is no good {\em a priori} setting of the clipping norm across tasks and learning settings: the update norm distribution depends on the model architecture and loss, the amount of data on each device, the client learning rate, and possibly various other parameters. We propose a method wherein instead of a fixed clipping norm, one clips to a value at a specified quantile of the update norm distribution, where the value at the quantile is itself estimated online, with differential privacy. The method tracks the quantile closely, uses a negligible amount of privacy budget, is compatible with other federated learning technologies such as compression and secure aggregation, and has a straightforward joint DP analysis with DP-FedAvg. Experiments demonstrate that adaptive clipping to the median update norm works well across a range of federated learning tasks, eliminating the need to tune any clipping hyperparameter.

Differentially Private Learning with Adaptive Clipping
Galen Andrew gal gallennddre@@googe.oo com
Om Thakkar omthikkr@google.com
Brendan McMahan mcmahan@google.co com
Swaroop Ramaswamy swaroopram@google.co com
Abstract
Existing approaches for training neural networks with user-level differential privacy (e.g., DP Federated Averaging) in federated learning (FI.) settings involve bounding the contribution of each user's model update by clipping However there is no good priori setting of the clipping norm across tasks and learning settings: the update norm distribution depends the model architecture and loss, the amount of data on each device, the client learning rate, and possibly various other parameters. We propose method wherein instead of a fixed clipping norm, one clips to value at specified quantile of the update norm distribution, where the value at the quantile itself estimated online, with differential privacy. The method tracks the quantile closely, uses negligible amount of privacy budget, is compatible with other federated learning technologies such as compression and secure aggregation, and has straightforward joint DP analysis with Avg..dAvg. Experiments demonstrate that adaptive clipping to the median update norm works well across range of realistic federated learning tasks, sometimes outperforming even the best fixed clip chosen in hindsight, and without the need to tune any clipping hyperparameter.
some constant value.
Introduction
Deep learning has become ubiquitous, with applications diverse as image processing, natural language translation, and music generation [27, 13, 28, 7]. Deep models are able to perform well in part due to their ability to utilize vast amounts of data for training. However, recent work has shown that it is possible to extract information about individual training examples using only the parameters of trained model [12, 30, 26, 8, 19]. When the training data potentially contains privacy-sensitive user information, becomes imperative to use learning techniques that limit such memorization.
Differential privacy (DP) [10, 11] is widely considered gold standard for bounding and quantifying the privacy leakage of sensitive data when performing learning tasks. Intuitively, DP prevents an adversary from confidently making any conclusions about whether some user's data was used in training model, even given access arbitrary side information. The formal definition of DP depends on the notion of neighboring datasets: we will refer to a pair of datasets D, D' E 'D as neighbors if D' can be obtained from by adding or removing one element.
Definition 1.1 (Differential Privacy). A (randomized) algorithm M : D + R with input domain D and output range R (s, (5 -'diffeernt private for all pairs of neighboring datasets D, D' D. and every measurable S C R, we have Pr (M() E S) < e Pr (M E S) + 8, where probabilities are with respect to the coin flips of M.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Following McMahan et al. [18], we define two common settings of privacy corresponding to two different definitions of neighboring datasets. In example-level DP, datasets are considered neighbors when they differ by the addition or removal of single example [9, 4, 2, 22, 23, 15]. In user-level DP, neighhoring datasets differ by the addition or removal of all of the data of one user [18]. User- level DP is the stronger form, and is preferred when one user may contribute many training examples to the learning task, as privacy is protected even if the same privacy-sensitie information occurs in all the examples from one user. In this paper, we will describe the technique and perform experiments in terms of the stronger user-level form, but we note that example-level DP can be achieved by simply giving each user a single example.
To achieve user-level DP, we employ the Federated Averaging algorithm [17], introduced as a decentralized approach to model training in which the training data is left distributed on user devices, and each training round aggregates updates that are computed locally. On each round, a sample of devices are selected for training, and then each selected device performs potentially many steps of local SGD over minibatches of its own data, sending back the model delta as update.
Bounding the influence of any user in Federated Averaging is both necessary for privacy and often desirable for stability. One popular way do this cap the L2 norm of its model update by projecting larger updates back to the ball of norm C. Since such clipping also effectively bounds the L2 sensitivity of the aggregate with respect to the addition or removal of any user's data, adding Gaussian noise to the aggregate is sufficient to obtain a central differential privacy guarantee for the update [9. 4, 2]. Standard composition techniques can then be used extend the per-update guarantee to the final model [20].
Setting appropriate value for the clipping threshold C' is crucial for the utility of the private training mechanism. Setting it too low can result in high bias since we discard information contained in the magnitude of the gradient. However setting it too high entails the addition of more noise, because the amount of Gaussian noise necessary for a given level of privacy must be proportional to the norm bound (the L2 sensitivity), and this will eventually destroy model utility. The clipping bias-variance trade-off was observed empirically by McMahan et al. [18], and was theoretically analyzed and shown to be an inherent property of differentially private learning by Amin et al. [3].
Learning large models using the Federated Averaging/SGI algorithm [17, 18] can take thousands of rounds of interaction between the central server and the clients. The norms of the updates can vary as the rounds progress. Prior work [18] has shown that decreasing the value of the clipping threshold after training language model for some initial number of rounds can improve model accuracy. However, the behavior of the norms can be difficult to predict without prior knowledge about the system, and if it is difficult to choose fixed clipping norm for a given learning task, it is even more difficult to choose a parameterized clipping norm schedule.
While there has been substantial work DP techniques for learning, almost every technique has hyperparameters which need to be set appropriately for obtaining good utility. Besides the clipping norm, learning techniques have other hyperparameters which might interact with privacy hyperparameters. For example, the server learning rate in DP SGD might need to be set to a high value if the clipping threshold is very low, and vice-versa. Such tuning for large networks can have an exorbitant cost in computation and efficiency, which can be a bottleneck for real-world systems that involve communicating with millions of samples for training a single network. Tuning may also incur an additional cost for privacy, which needs to be accounted for when providing a privacy guarantee for the released model with tuned hyperparameters (though in some cases, hyperparameters can be tuned using the same model and algorithm on a cefficienntl-simiille public proxy dataset).
Related Work The heuristic of clipping to the median update norm was suggested by Abadi et al. [1], but no method for doing so privately was proposed, nor was the heuristic empirically tested. One possibility would be to estimate the median unclipped update norm at each round privately by adding noise calibrated the smooth sensitivity as defined by Nissim et al. [21]. However this approach has several drawbacks compared to the algorithm we will present. It would require the clients to send their unclipped updates to the server each round,' which is incompatible with the foundational
1 AAteennativly, clients could first report their update norms, after which the server could reply with the clipping norm for the round, and finally the clients would send the clipped updates. However, this would still give the server strictly more information than our approach, and would require an extra round of communication. both of which are undesirable.
2

---

40
40
y-
30
y-0.5
20
20
20
y=7.5
y=
20
15
28
45
loss
15
10
20 20
20
25
40 Per-element loss
48
15
25 28
40 45 48
Average per-element
Figure I: Loss functions to estimate the 0.01-, 0.5-, 0.75-, and 0.99-quantiles for random variable X that uniformly takes values in {15, 25, 28, 40, 45, 483. The loss function is the average of convex piecewise-linear functions, one for each value. For instance, for the median (? 0.5), this is just (C; X) = X C₁2 where X is the random value, and C' is the estimate. When we average these functions, we arrive at the yellow function in the plot showing the average loss, which indeed is minimized by any value between the central two elements, i.c., in the interval [28, 40J. The function for 7 0.75 is minimized
45 because Pr(X C) < 0.75 for values C. in [40, 45), while Pr(X < C) > 0.75 for values C' in (45, 48).
principle of federated learning transmit only focused, minimal updates, and precluding the use of secure aggregation [5] and certain forms of compression [16, 6]. Also, by incorporating information across multiple rounds, our method is able track the underlying quantile closely, with less jitter associated with sampling at each round, and using only a negligible fraction of the privacy budget.
Contributions In this paper, we describe a method for adaptively and privately tuning the clipping threshold to track given quantile of the update norm distribution during training. The method uses a negligible amount of privacy budget, and is compatible with other FL technologies such as compression and secure aggregation [5. 6]. We perform careful empirical comparison of our adaptive clipping method to highly optimized fixed-clip baseline on a suite of realistic and publicly exceeding
available FL tasks to demonstrate that high-utility and high-privace any fixed clipping norm in utility can be trained using our method without the need to tune any clipping hyperparameter.
2 Private adaptive quantile clipping
In this section, we will describe the adaptive strategy that can be used for adjusting the clipping threshold so that comes to approximate the value at a specified quantile.
Let X E R be a random variable, let y E [0, 1] be quantile to be matched. For any C, define
((C:x) (1->)(C i (x(X- ((X-)) CC)
otherwise,
SO
((C; = (1-7) I
-- otherwise.
Hence, (CC (IC,(C";;) = 0, we have Pr(X < (²) Thus, C* is the 7thh quantile of X. Because the loss is convex and has gradients bounded by 1, we can get an online estimate of C that converges to the 7hth quantile of X using online gradient descent (see, e.g., Shalev-Shwartz. [25]). See Figure for plot of the loss for random variable that takes six values with equal probability.
< C]= Pr|X S SC] 7. For C* such that
3

---

Suppose at some round we have m samples of X, with values (m)) The average derivative of the loss for that round is
1 (1
m
7)
xf C. otherwise
Ex(C:X)=
m
1
7)
Σ xE(m)
7 Σ
Io,sc) = %
m
'E [m)
where 5 _ Zie(m] Le, is the empirical fraction of samples with value at most C. For given learning rate nJc, we can perform the update: C <- C -C-mcbb 7).
Geometric updates. Since b and ry take values in [0. 1], the linear update rule above changes C' by maximum of πo at each step. This can be slow if C is on the wrong order of magnitude. At the other extreme, if the optimal value of C' is orders of magnitude smaller than l7c. the update can be very coarse, and may overshoot to become negative. To remedy such issues, we propose the following geometric update rule: C t C' exp( nc 7)). This update rule converges quickly to the true quantile even if the initial estimate off by orders of magnitude. It also has the attractive property that the variance of the estimate around the true quantile at convergence is proportional to the value at that quantile. In our experiments, we use the geometric update rule with Ijc = 0.2.
2.1 DP-FedAvg with adaptive quantile clipping
Let m be the number of users in a round and let 7 E [0, 1] denote the target quantile of the norm distribution at which we want to clip. For iteration E [T). let C2 be the clipping threshold, and ne be the learning rate. Let Q' be set of users sampled in round t. Each user i E Q' will send the bit bf along with the usual model delta update A% where bf = I|asllasc* Defining b' m Eie b/. 7)). However, we can't 't use b' directly,
we would like to apply the update C' < C' since it may reveal private information about the magnitude of users' updates. To remedy this, we add Gaussian noise to the sum: 5' m with adaptive clipping is shown in Algorithm 1. We augment basic federated averaging with server momentum, which improves convergence [14, 24j.2
(Eregr 6f + ++(O of))..)) The DPFedAvg algorithm
Algorithm 1 DPFedAvg-M M with adaptive clipping
function Train(m, 7. 7e Ne. TJC, 2, Tb.
Ct+1 + - (-no(0- - -7)
Initialize model 60, clipping bound (r)
FA + (=-22- (2a,)-2)-11
function FedAvg(i, 0, 1/. C)
for each round f 1, do
x- 1
O' +- (sample m users uniformly) for each user E Q' in parallel do + FedAvg(i, ne, Ct)
G + (user i's local data split into batches) for batch g E g do
(A b/)
0 - 0 (((0:))
aA
Ct
A + 0 - 8
A = 1
m (EiEo)
A N0, Io)))
b +- 1 AllC
A = 3A--1 + -))
A' A min (1, C
0t+1 + Of + ''
return (A),b)
8' 1 EEEE 0 +N((0))))
"Note that since the momentum update computed using privatized estimates of average client delta, privacy properties are unchanged when momentum rates, but found that they were less effective when noise is added for DP, perhaps because the noise causes the preconditioner become large prematurely.
is added. We also experimented with adaptive learning

---

125
100
Figure 2: Evolution of the quantile estimate data drawn from log-normal distributions. The three plots use data drawn from the exponential of NVO00, 1.0), (0.000.1), and M(log 10, 1.0), respectively. Curves are shown for each of five quantiles: (0.1, 0.3, 0.5, 0.7, 0.9), and the dashed lines show the true value at each quantile. Hyperparameters are as discussed in the text and used in the experiments of Section 3: fcc 0.2, C 0.1, exponential growth, the true quantile is fairly closely tracked. A smaller value of T/C: would allow more accurate tracking the cost of slower convergence, but since the quantile value is only used as a heuristic for clipping, small amount of noise is tolerable. The entire sequence of values estimated for each target quantile satisfy (0.034, dddfferrential privacy using RDP composition across the 200 rounds assuming fixed-size samples of m 100 out of total population of = 10t [29].
100, σ. m/20. After initial phase of
Theorem 1. One step of DP- De-FedAvg with adaptive clipping using noise standard deviation on the clipped counts Σ bf and 2Λ noise multiplier on the vector sums Σ A equivalent (so far as privacy accounting is concerned) to one step of non-adaptive DP-FedAve with noise multiplier z we set Zι = (=-2 (2o,)
Proof. We make conceptual change to the algorithm that does not change the behavior privacy properties but allows us to analyze each step as if it were a single private Gaussian sum. Instead of sending (Af, bf), each user sends (Á., B{) & (AI/aa, (b) 1/2)/ou). The server adds noise with covariance and averages, then reverses the transformation so Á / (Eieor A1+,((0,T) and
8' = do Eieor bj +NO, 1)) + 1/2.. Noting that [|(Á), b/) < ((C%)a)) (1/20)) )² is clear that the two Gaussian sum queries of Algorithm 1 are equivalent to pre- and post- post-processing of single query with sensitivity S and covariance or noise multiplier 1/s 22 (20b) Rearranging yields the result.
-//
In practice, we recommend using value of a m/20. Since the noise Gaussian, this implies that the error 16 8-1 will be less than 0.1 with 95.4% probability, and will be no more than 0.15 with 99.7% probability. Even in this unlikely case, assuming a geometric update and a learning rate of nJc- = 0.2, the error on the update would So this default gives high privacy for an acceptable amount of noise in the quantile estimation process. Using Thm. we can compute that to achieve an effective combined noise multiplier of 1, with m 100 clients per round, the noise multiplier =A is approximately 1.005. So we are paying only factor of 0.5% more noise on the updates for adaptive clipping with the same privacy guarantee (a quantity which only gets smaller with increasing m.) These constants (a, = m/20 and Tjc 0.2) are what we use in the experiments of Section
factor of cxp(0.2 x 0.15) 1.03, a small deviation.
The clipping norm can be initialized to any value C*9 that is safely on the low end of the expected norm distribution. If too high and needs to adapt downward, lot of noise may be added the beginning of model training. which may swamp the model. However there is little danger in setting quite low, since the geometric update will make grow exponentially until it matches the true quantile. In our experiments we use an initial clip of 0.1 for all tasks. It is easy to compute that with learning rate of rıc: 0.2 and a target quantile of 7 = 0.5, if every update clipped, the quantile estimate will increase by factor of ten every 23 iterations. In order to show the effectiveness of the algorithm at tracking known quantile, we ran on simulated data for which we can compute the true quantile exactly. Figure 2 shows the result of this experiment.
5

---

Task
model
N
T
11a
Cmin
Cmax
CIFAR-ION 100
ResNet
11M
500
4000
0.669
2231
0.1
0.32
0.75
2.2
EMNIST-CR
CNN
1.2M
3400
1500
0.513
513
0.032
1.0
0.28
0.85
EMNIST-AE
Deep AE
2.8M
3400
3000
0.659
2197
3.2
1.78
0.22
0.95
SHAKESPEARE
C-LSTM
820k
715
1200
0.510
510
1.0
0.32
0.25
3.6
SO-NWP
W-LSTM
4.1M
342k
1,500
1.396
13958
0.18
1.78
0.30
1.6
SO-LR
Multi-LR
5M
342k
1500
1.396
13958
320.0
1.78
16.0
135.0
Table I: Dataset statistics and chosen hyperparameters, Left: model type, number of trainable
parameters N, number of training clients n, and the number of training rounds T used (following Reddi et al. [24]). Middle: the noise multiplier and number of clients per round m necessary to
achieve (5, n 1.1). J-D with less than 5% model performance loss if each task had a population of
10 [29]. Right: the optimal unclipped baseline client and server learning rates (Sec. 3.1) for
each task and chosen values of minimum and maximum fixed clips (Sec. 3.2).
3 Experiments
To empirically validate the approach, we examine the behavior of our algorithm on six of the public benchmark federated learning tasks defined by Reddi et al. [24], which are to our knowledge the most realistic and representative publicly available federated learning tasks that exist date. All six tasks are non-i.i.d. with respect to user partitioning: indeed with the exception of CIFAR- 100, the data is partitioned according to the actual human user who generated the data, for example the writer of the EMNIST characters or the Stack Overflow user who asked or answered a question. Table (left) lists the characteristics of the datasets.
Two of the tasks derived from Stack Overflow data (SO-NWP and SO-LR) are ideal for DP research due to the very high number of users (342k) making it possible train models with good leeel privacy without sacrificing accuracy. The other four tasks (CIFAR- 100, EMNIST-AE, EMNIST- CR. SHAKESPEARE) are representative learning tasks, but not representative population sizes for real world cross-device FL. applications. Therefore we focus on establishing that adaptive clipping works well with 100 clients per round on these tasks the regime where the noise is at a level such that utility is just beginning to degrade. Under the assumption that a larger population were available, one could increase m, aA, and σ proportionally to achieve comparable utility with high privacy. This should not significantly affect convergence (indeed, it might be beneficial) since the only effect is to increase the number of users in the average Á reducing the variance. Table 1 (middle) shows the number of clients per round with which our experiments indicate we could achieve (5, n --D for each dataset with acceptable model performance loss (less than 5% relative to non-private training, as discussed later) if each dataset had n = 10 clients, using RDP composition with fixed-size subsampling [29].
3.1 Baseline client and server learning rates
Reddi et al. [24] provide optimized client and server learning rates for federated averaging with momentum that serve starting point for experimental setup. For almost all hyperparameters (model configuration, evaluation metrics, client batch size, total rounds, etc.) we replicate their experiments, but with two changes. First, increase the number of clients per round to 100 for all tasks. This reduces the variance the updates to a level where we can reasonably assume that adding more clients is unlikely significantly change convergence properties [18]. Second, as shown in Algorithm 1 we use unweighted federated averaging, thus eliminating the need to set yet another difficulttto-fit hyperparameter: the expected total weight of clients round. Since these changes might require different settings, we reoptimize the client and server learning rates for our baseline with no clipping or noise. We ran a small grid of 25 configurations for each task jointly exploring client and server learning rates whose logarithm (base- 10) differs from the values in Table 10 of Reddi et [24] by { -V/, -V/, V4, 1/2 ). The optimal baseline client and server learning rates for our experimental setup are shown in Table (right).
'The code used for all of our experiments is publicly available at https: com googge wttpppaaccruuufooteetttttteeeeee

---

CIFAR-100
EMNIST CR
EMNIST AE
057
GAB
85
055
060
y0R
Y47
Shakespeare
StackOverlow NWP
SlacKOventow LR
224
6:21 1053 11.0
P:31
P12
1.80
P.3
V108
P130
P00
Figure 3: Impact of clipping without noise. Performance the unclipped baseline compared to five settings of from = 0.1 (aggressive clipping) to 7 = 0.9 (mild clipping). The values shown are the evaluation metrics on the validation set averaged over the last 100 rounds. Note that the y-axes have been compressed to show small differences, and that for EMNIST-AE lower values are better.
Because clipping (whether fixed or adaptive) reduces the average norm of the client updates, it may be necessary to use a higher server learning rate to compensate, Therefore, for all approaches with search over a small grid of five server learning rates, scaling the values in Table by {1, 10 10 // 10"/ 10). For all configurations, we report the best performing model whose server learning rate was chosen from this small grid on the validation set.*
clipping-fixee or adaptive-
We first examine the impact of adaptive clipping without noise to see how it affects model performance. Figure 3 compares baseline performance without clipping to adaptive clipping with five different quantiles. For each quantile, we show the best model after tuning over the five server learning rates mentioned above the validation set. On three tasks (CIFAR-100, EMNIST-AE, SO-NWP), clipping improves performance relative to the unclipped baseline. On SHAKESPEARE and SO-LR performance is slightly worse, but we can conclude that adaptive clipping to the median generally fares well compared to not using clipping across tasks. Note that for our primary goal of training with DP, it essential limit the sensitivity one way or another, so the modest decrease in performance observed from clipping on some tasks may be part of the inevitable tension between privacy and utility.
3.2 Fixed-clip baselines
We would like compare our adaptive clipping approach fixed clipping baseline, but comparing to just one fixed-clip baseline may not be enough to demonstrate that adaptive clipping consistently performs well. Instead, our strategy will to show that quantile-based adaptive clipping performs as well or nearly as well as any fixed clip chosen in hindsight. If we can first identify clipping norms that span the range of normal values during training on each problem/conffguration we can compare adaptive clipping to fixed clipping with those norms.
To that end, first adaptive clipping without noise discover the value of the update norm distribution at the following five quantiles: 1, 0.3, 0.5, 0.9). Then we choose as the minimum of our fixed clipping range the smallest value the 0.1 quantile over the course of training, and as the maximum the largest value at the 0.9 quantile. Plots of the update norms during training on each of the tasks are shown in Figure
On each task there a ramp up period where the clipping norm, initialized to 0.1 for all tasks, catches up to the correct norm distribution. Thus disregard norm values collected until the actual fraction of clipped counts on some round within 0.05 of the target quantile yy The chosen values for the
*Note that this modest retuning of server learning rate only necessary because we are starting from configuration that was optimized without clipping. In practice as we will discuss in Section 4, we recommend that all hyperparameter optimization should be done with adaptive clipping enabled from the start, eliminating the need for this extra tuning.

---

34
3.20
1.50
1e00l
StackOvertorw
33
reand
1eand
Figure 4: Evolution of the adaptive clipping norm five different quantiles (0.1, 0.3, 0.5, 0.7, 0.9) on each task with no noise. The norms are estimated using geometric updates with TJC 0.2 and an initial value C 0.1. With the possible exception of SO-LR, the estimated quantiles appear to closely track an evolving update norm distribution. Note that each task has unique shape to its update norm evolution, which further motivates an adaptive approach.
minimum and maximum fixed clips for each task are shown in Table I (right). Our fixed-clipping baseline uses five fixed clipping norms logarithmically spaced in that range. Here we are taking advantage of having already run adaptive clipping to minimize the number of fixed clip settings need to explore for each task. we had to explore over the entire range knowing only the endpoints across all tasks (0.22, 135.0) the same resolution, would need nearly four times many clip values per task.
For each value of noise multiplier {0, 0.01, 0.03, 0.1} we trained using the five fixed clipping norms and compare to adaptive clipping with the five quantiles (0.1, 0.3, 0.5, 0.7, 0.9). Note that for the fixed clipping runs za that is, for fixed clip C, the noise applied to the the updates has standard deviation zC. As discussed section 2.1, on adaptive clipping runs na slightly higher due to the need to account for privacy when estimating the clipped counts.
3.3 Comparison of fixed and adaptive clipping
Validation set results with adaptive clipping are shown in Figure 5 and with fixed clipping in Figure 6. These charts show that we have identified the noise regime in which performance is beginning to degrade. There is always tension between privacy and utility: as the amount of noise increases, eventually performance will go down. For the purpose of this study we consider more than 5% relative reduction in evaluation metric to be unacceptable. Therefore for each task, we look at the level of noise =* at which the evaluation metric on the validation set is still within 5% the value with no noise, but adding more noise would degrade performance beyond 5%. Given z" for each task, we then choose C* to be the fixed clip value that gives best performance on the validation set. The values of 2* and C* are shown in Figure 7 (left).
For our final C* The results are in Figure (right). We show the average test set performance and bootstrapped 95% confidence interval over 20 runs varying the random seed used for client selection and DP noise. On three of the tasks (CIFAR- 100 EMNIST-AE, SHAKESPEARE), clipping to the median actually outperforms fixed clipping to the best fixed clip chosen in hindsight, and on two more (EMNIST-CR, SO-NWP), the performance is comparable. Only on SO-LR the best fixed clip does perform somewhat better. This task seems to unusual in that best performance comes from aggressive clipping, small fixed clip fares better than adaptive clipping to the median. However,
evaluation, compare adaptive clipping the median fixed clipping at

---

CIFAR-100
EMNIST CR
EMNIST AE
MCS
1427
1400
H40
MCC
1427
M43
MC5
1437
M40
6.55
0.18
52
C:18
75
90
0.00
200.0
0M921
24010
240.10
24010
Shakespeare
SlackOverflow NWP
SlackOverflow LR
y4?
(42
1403
y427
voa
MCS
y42
04
240.00
992.03
240
24010
20010
21000
Figure 5: Evaluation metric performance of adaptive clipping with five settings of for each of four effective noise multipliers x. Note that the y-axes have been compressed to show small differences, and that for EMNIST-AR AE lower values are better.
CIFAR-100
EMNIST CR
EMNIST AE
CHOUT
CH1
CH126
CH146
0-22
C-220 CH0ST
B CHE4R
CHE
CHOSS
C-4222
C-0.8
CHC46
C-1 MI CH346
6:8
C118
C.12
000
EH9.21
040.1
EX0.11
24000
64010
+4001
24010
Shakesppeae
SlackOverflow NWP
SlackOverflow LR
C:695
C1046
01166
C.000
B CCE
01166
0:923
0136
200.0
210
04010
2M2.00
24010
Figure 6: Evaluation metric performance of fixed clipping with five settings of C' for each of four noise multipliers z. Note that the y-axes have been compressed to show small differences, and that for AM lower values are better.
looking at Figure 6 (and noting the scale of the y axis), on this task more than the others, getting the exact right fixed clip is important. The development set recall @5 5 value of 55.1 corresponds to the optimal fixed clip of 16.0. The next larger fixed clip of 27.3 gave a recall of only 51.8, and larger clips fared even worse. So an expensive hyperparameter search may be necessary even get close to this high-performing fixed clip value.
4 Conclusions and implications for practice
In our experiments, we started with a nigh-perffomiing non-pri con-private baseline with optimized client and server learning rates. We then searched over small grid of larger server learning rates for our experiments with clipping (adaptive or fixed). This is one way to proceed in practice, if such non-private baseline results are available. More often, such baseline learning rates are not available, which will necessitate a search over client learning rates as well. In that case, it would be beneficial to enable adaptive clipping to the median during that hyperparameter search. The advantage of clipping relative to the unclipped baseline observed on some tasks could only increase if the other hyperparameters such as client learning rate were also chosen conditioned on the presence of clipping.
Although the experiments indicate that adaptive clipping to the median yields generally good results, on some problems (like SO-LR in our study) there may be gains from tuning the target quantile. It
9

---

Task
z²
C
CIFAR- 100
0.03
0.98
25.4
$5.00
EMNIST-CR
0.10
0.37
EMNIST-AE
0.03
0.32
SHAKESPEARE
0.10
0.95
SO-NWP
0.01
0.30
SO-LR
0.01
16.0
54.9
5226
11.0
Figure 7: Left: for each task, the maximum noise possible before performance begins to significantly degrade (z"), and the best fixed clip (C*) chosen on the development set. Right: average test set performance and bootstrapped 95% confidence interval over 20 runs. In practice, finding the best fixed clipping norm would require substantial additional hyperparameter tuning.
would require adding another dimension to the hyperparameter grid, exponentially increasing the tuning effort, but even this would be preferable to tuning the fixed clipping norm from scratch, since the grid can be smaller: we obtained good results on all problems by exploring only five quantiles, but the update norms in the experiments range over four orders of magnitude, from 0.22 to 135.
Combining our results with the lessons taken from [18] and [24], the following strategy emerges for training a high-performing model with user-level differential privacy. We assume some non-private proxy data is available that may have comparatively few users n', well that the true private data has enough users n that the desired level of privacy is achievable.
1. With adaptive clipping to the median enabled, using relatively small number of clients per round (m x 100), and a small amount of noise (z = 0.01), search over client and server learning rates on non-private proxy data.
2. Fix the client and server learning rates. Using the non-private data and low value of m, train several models increasing the level of noise until model performance at convergence begins degrade.
3. To train the final model on private data, if (m, is too small for the desired level of privacy even given n, set (m, 2) + a (m, z) for some a > such that the privacy target (e, 8) is achieved. Finally, train the private model using that value of m and
By eliminating the need to tune the fixed clipping norm hyperparameter which interacts signifi- cantly with the client and server learning rates, the adaptive clipping method proposed in this work exponentially reduces the work necessary to perform the expensive first step of this procedure.
'Hree employ assumption that sufficiently large, since cannot exceed
10

---

References
[1] Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang, Deep learning with differential privacy. 23rd ACM Conference on Computer and Communications Security (ACM CCS), 2016.
[2] Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan MeMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '16, pages 308-318, New York,
NY, USA, 2016. ACM. ISBN 1188--4503-4419-44 doi: 10.1 10.11455977749929998
[3] Kareem Amin, Alex Kulesza, Andres Munoz, and Sergei Vassilvtiskii. Bounding user contribu- tions: A bias- variance trade-off in differential privacy. In International Conference on Machine Learning, pages 263-271. PMLR, 2019.
[4] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. Foundations of Computer Science (FOCS), 2014 IEEE 55th Annual Symposium on, pages 464 4744473. IEFF, 2014.
[5] Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for federated learning on user-held data. arXiv preprint arXiv:1611.04452, 2016.
[6] Keith Bonawitz, Fariborz Salehi, Jakub Konečný. Brendan McMahan, and Marco Gruteser. Federated learning with autotuned communication- secure aggregation. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers, pages 1222-1226 IEEE, 2019.
[7] Jean-Pierre Briot, Gaëtan Hadjeres, and François- Pachet. Deep Learning Techniques for Music Generation A Survey. arXiv e-prints, art. arXiv: 1nXiv:1709.0164 700.016600 Sep 2017.
[8] Nicholas Carlini, Chang Liu, Jemej Kos, Úlfar Erlingsson, and Dawn Song. The secret sharer: Measuring unintended neural network memorization & extracting secrets. CoRR, abs/1 abs/1802. 2018. URL http://aaxivvorr/aa//1110002223
[9] Kamalika Chaudhuri, Claire Monteleoni, and Anand Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(Mar): 1069 1109, 2011.
[10] Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In EUROCRYPT, 2006.
[11] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference, pages 265-284. Springer, 2006.
[12] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22Nd ACM SIGSAC Conference on Computer and Communications Security, CCS 15, pages 1322-1333, New York,
NY, USA, 2015. ACM. ISBN 177-1145503-383-55 doi: 10. 10.144/28101003.8833777
[13] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, 1034, Dec 2015. doi: 10. 11.110//.C..2015.22..
[14] Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification, 2019.
[15] Roger Iyengar, Joseph P. Near, Dawn Song, Om Thakkar, Abhradeep Thakurta, and Lun Wang. Towards practical differentially private convex optimization. S&P 2019, 2019.
[16] Jakub Konečný, H Brendan McMahan, Felix X Yu, Peter Richtárik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint
2016.
11

---

[17] Brendan McMahan, Eider Moore. Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efice learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTAT'S 2017, 20-22 April 2017, Fort Lauderdale, FI., USA, pages 1273-1282, 2017. URL http://prooceeedngg.mmm press/v54/ccmhhnn 7a.html.
[18] H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In ICLR 2018, 2018.
[19] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting Unintended Feature Leakage Collaborative Learning. arXiv e-prints, art. 1.Xix:8805.04444 May 2018.
[20] Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 275. IEEE, 2017.
[21] Kobbi Nissim, Sofya Raskhodnikova, and Adam Smith. Smooth sensitivity and sampling in private data analysis. In Proceedings of the thirty-ninth annual ACM symposium Theory of computing, pages 75-84, 2007.
[22] Nicolas Papernot, Martin Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. stat, 1050, 2017.
[23] Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ülfar Erlingsson. Scalable private learning with pate. arXiv preprint arXiv: 2018.
[24] Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H. Brendan McMahan. Adaptive federated optimization, 2020.
[25] Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends in Machine Learning, 2012.
[26] R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18, May 2017. doi: 10. 109/SP.2017.41.
[27] C. Szegedy, .. P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1-9, June 2015. doi: (0.109/CVVR.201557999999
[28] Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, llya Sutskever, and Geoffrey Hinton. Grammar as foreign language. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume NIPS' 15, pages 2773-2781, Cambridge, MA, USA, 2969999550
2015. MIT Press. URL
[29] Yu- Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. The 22nd International Conference on Artificial Intelligence and Statistics, pages 1226-1235. PMLR, 2019.
[30] X. Wu, M. Fredrikson, S. Jha, and J. F. Naughton. A methodology for formalizing model- inversion attacks. In 2016 IEEE 29th Computer Security Foundations Symposium (CSF), pages 355-370, June 2016. doi: 10. 201.252200.....
[31] Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton. Bolt- on differential privacy for scalable stochastic gradient descent-based analytics. In Proceedings of the 2017 ACM International Conference Management of Data, SIGMOD '17, pages 1307-1322, New York, NY, USA, 2017. ACM. ISBN 978-1-4503-4197-4, 0035918.30044047
doi: 10. 1455/
12


Reviews:
[{"review_id": "v6pOLUNI2Z7", "reviewer": "Reviewer_BgsA", "summary": "This paper studies the problem of ERM under user-level differential privacy. They propose a method based on quantile clipping to bound the contribution of each user (L_2 sensitivity of the gradient estimation), eliminating the need to specify a clipping threshold in advance.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The problem of ERM under user-level differential privacy is important and has real-world applications.\n- To bound the contribution of each user, this paper chooses the clipping threshold by an adaptive method based on quantile, instead of requiring a pre-specified clipping threshold.\n\nWeaknesses:\n- The novelty and technical depth is low: the quantile computation method directly follows from a standard method based on gradient descent; there is no theoretical analysis for the accuracy guarantee (e.g. for the quantile computation), and the heuristic of clipping to median is proposed in DP-SGD.\n- For quantile computation under differential privacy, the comparison with smooth sensitivity is not accurate and complete. At lines 72-73, the authors say that smooth sensitivity requires each client send unclipped updates to the server, however, only their norms are needed to estimate the median. In fact, since the privacy guarantee is in the centralized model, this paper also requires the server knowns the exact answers (i.e., the information of unclipped updates to estimate the median, and the aggregation of all client updates) to be perturbed by Gaussian noise.\n- Lack of clear statements (e.g., theorems and proofs) for the privacy and accuracy guarantee. It is not easy to follow and understand the details of the proposed quantile computation method.\n\n[1] Answering Range Queries Under Local Differential Privacy\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "aO4rCm1Kvlp", "reviewer": "Reviewer_nrc3", "summary": "This works studies the problem of training neural networks with user-level differential privacy using federated learning. To bound user contribution, the authors propose to clip the gradient norms to the private estimate of the median of gradient norms, as opposed to some fixed clipping threshold. This method reduces the need to tune the clip threshold and achieves competitive performance across a range of tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work proposes to clip to the median of the gradient norms to bound user contributions in federated learning under user-level DP. The authors run extensive experiments to show that clipping to the median yields desired performance comparable to clipping at the best threshold chosen in hindsight. The experiment setup is thorough, and the results support the claims of the work.\n\nThe algorithm to estimate the median (and arbitrary quantiles) is interesting. It does not require prior knowledge of the upper bound on the gradient norm. However, it is unclear the advantage of this method over existing private quantile estimators, for example, \n\nDwork and Lei, Differential Privacy and Robust Statistics \n\nTzamos et.al., Optimal Private Median Estimation under Minimal Distributional Assumptions. \n\nThe authors may want briefly discuss why the current algorithm is preferred over existing algorithms for quantile estimation.\n\nAnother issue is the privacy guarantees. It seems that in the experiments the authors choose $\\sigma_b, \\sigma_\\Delta$ that guarantees $(5, n^{-1.1})$-DP, which falls into the low privacy regime. The authors provided a procedure to tune the parameters to satisfy arbitrary privacy guarantees, but it is unclear to me when we should stop increasing the noise levels. The authors may wish to provide a more formal privacy guarantee in terms of the parameters (e.g., clip threshold, noise magnitudes). \n\nOverall the paper is well-organized.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "VJOeo4-bJCJ", "reviewer": "Reviewer_Z3R2", "summary": "This paper tackles the problem of training accurate neural networks with user-level privacy in the federated learning settings. Classical implementations usually ensure user-level differential privacy of the learning scheme by applying the following procedure:  \n\n1) before sending their gradient update to the server, every user clip (or rescale) its gradient update so that its norm is smaller or equal to a constant clipping parameter $C$ \n2) after aggregating the clipped gradients, the server adds Gaussian noise proportional to $C$ before applying a gradient update rule.\n\nIn this paper, instead of applying step 1) with a fixed constant $C$, the authors present a method to adaptively select the clipping constant during the training. This method is based on a quantile regression problem under differential privacy constraints. The authors demonstrate through numerical experiments that this new adaptive clipping method obtain similar result as classical implementations while eliminating the need for hyper parameter search on $C$.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Relevance: Federated Learning is currently a hot topic within the ML community. Furthermore, improving the convergence/accuracy of privacy preserving techniques is also very important and relevant to the NeurIPS community.\n\nClarity: The paper is overall well written and statements are clearly presented. \n\nQuality: Both the technical and experimental part of the paper look sound to me. I just have some minor comments/questions. \n- In the proof of Theorem 1, why are you building a query that sends $(\\frac{\\Delta_i^t}{\\sigma_\\Delta},\\frac{(b_i^t -1/2)}{\\sigma_b})$ instead of $(\\frac{\\Delta_i^t}{\\sigma_\\Delta},\\frac{b_i^t }{\\sigma_b})$ ? Also why is $\\frac{1}{2\\sigma_b}$ the sensitivity of $(b_i^t-1/2)/\\sigma_b$ and not $1/\\sigma_b$? \n- Figure 2 is interesting as it presents a sanity check of the quantile regression problem for a fixed gradient distribution. However it is unclear to me how informative it is in the context of the gradient descent, as the gradient distribution will be changing at every step.  \n- I am not aware of the classical privacy level the literature on Private FL considers as acceptable, but, from my point of view, $\\epsilon=5$ is not a very meaningful privacy budget. Maybe it would have been interesting to compare the privacy/accuracy tradeoff for several privacy budgets.\n\nOriginality and Significance: I think originality is not the main strength of this paper. The idea of using the median to adapt the gradient clipping during the learning was first presented in [1], even though it was not properly studied. Similarly, the problem of quantile regression is a plain old machine learning problem and adapting it with differential privacy only asked for Gaussian noise injection during SGD. Nevertheless, I think the idea of combining these techniques in a sound way to build efficient adaptive clipping is new. Overall, while the idea is not very surprising, I think this is an interesting paper that could benefit to the community by helping automatize the study of differentially private implementations of SGD. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see any negative social impact rising from this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6-7 hours"}, "extra_text": {}}, {"review_id": "8OpaZ3xJ-ui", "reviewer": "Reviewer_qHtK", "summary": "This paper studies the problem of how to choose a good clipping norm in differentially private federated learning. The authors propose a new method to select the clipping threshold by privately approximate the quantile of the update norm distribution. Experiments validate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of the proposed method seems to be interesting, but the analysis of the current method needs to be further justified. Here are my main concerns of the current paper:\n1. The motivation of using the quantile of the update distribution as the clipping threshold is not well justified.\n2. The privacy guarantee of the current method is not very clear. For example, how the number of sampled clients will affect the privacy guarantee is not clear. In addition, it is not clear how the proposed method tracks the privacy loss in the experiments.\n3. It is unclear how the proposed method will affect the utility guarantee of the algorithm. The contribution of the current paper will be stronger if the authors could add more analysis about the utility using the proposed clipping method.\n4. One important metric to evaluate the performance of federated learning is the communication round. How the proposed method will affect the communication round is also unclear.\n5. Experimental results show that the improvements of the proposed methods are not very significant compared with the fixed clipping method.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for adaptively setting the gradient clipping threshold for DP federated learning. The proposed method is mostly a combination of known techniques, but it provides a neat and well-presented solution to an important practical problem.

The reviews are mixed: some reviewers are more positive because of more emphasis on the practical relevance while others are more negative because of emphasis on limited technical novelty.

The AC is in favour of accepting the paper because of its practical usefulness and potential impact. In my opinion, the authors have been able to answer the criticisms of the negative reviews in their rebuttal. Unfortunately these reviewers have been unresponsive in the discussion and thus the reviews have not been updated in light of the author responses, potentially leaving the reviews unnecessarily negative.

While not reflected in the reviews, this paper was discussed extensively by the AC (who read it in full and championed the work) and the SAC, leading to the decision to accept.
Decision: Accept (Poster)