CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learning
Authors: Samuel Maddock
Alexandre Sablayrolles
Pierre Stock
Venue: ICLR 2023 poster (2023)
Keywords: Federated Learning
Differential Privacy
Empirical Privacy
Model Auditing
Membership Inference Attack

Abstract: Federated Learning (FL) is a setting for training machine learning models in distributed environments where the clients do not share their raw data but instead send model updates to a server. However, model updates can be subject to attacks and leak private information. Differential Privacy (DP) is a leading mitigation strategy which involves adding noise to clipped model updates, trading off performance for strong theoretical privacy guarantees. Previous work has shown that the threat model of DP is conservative and that the obtained guarantees may be vacuous or may overestimate information leakage in practice. In this paper, we aim to achieve a tighter measurement of the model exposure by considering a realistic threat model. We propose a novel method, CANIFE, that uses canaries - carefully crafted samples by a strong adversary to evaluate the empirical privacy of a training round. We apply this attack to vision models trained on CIFAR-10 and CelebA and to language models trained on Sent140 and Shakespeare. In particular, in realistic FL scenarios, we demonstrate that the empirical per-round epsilon obtained with CANIFE is 4 -- 5$\times$ lower than the theoretical bound.

Published as a conference paper at ICLR 2023
CANIFE: CRAFTING CANARIES FOR EMPIRICAL PRI- VACY MEASUREMENT IN FEDERATED LEARNING
Samuel Maddock
Alexandre Sablayrolles Meta Al
Pierre Stock Meta AI
University of Warwick
ABSTRACT
Federated Learning (FL) is a setting for training machine learning models in dis- tributed environments where the clients do not share their raw data but instead send model updates to a server. However, model updates can be subject to attacks and leak private information. Differential Privacy (DP) is a leading mitigation strategy which involves adding noise to clipped model updates, trading off perfor- mance for strong theoretical privacy guarantees. Previous work has shown that the threat model of DP is conservative and that the obtained guarantees may be vac- uous or may overestimate information leakage practice. this paper, we aim to achieve tighter measurement of the model exposure by considering realistic threat model. We propose novel method, CANIFE, that uses ceaaaris-ccrefinn crafted samples by strong adversary to evaluate the empirical privacy of train- ing round. We apply this attack to vision models trained on CIFAR- 10 and CelebA and to language models trained on 1ent140 and Shakespeare. In particular, in re- alistic Fl. scenarios, we demonstrate that the empirical per-round epsilon obtained with CANIFE is 4 5x lower than the theoretical bound.
INTRODUCTION
Federated Learning (FL) has recently become a popular paradigm for training machine learning models across large number of clients, each holding local data samples (McMahan et al. 2017a). The primary driver of FL's adoption by the industry is its compatibility with the "privacy by design" principle, since the clients' raw data are not communicated training procedure (Kairouz et al.] 2019 Huba et al. 2022, Xu et al.) (022). Instead, clients train the global model locally before sending back updates, which are aggregated by a central server However, model updates, in their individual or aggregate form, leak information about the client
other parties during the
local samples (Geiping et al. 2020 Gupta et al., 2022).
Differential Privacy (DP) (Dwork al. 2006 Abadi et al. 2016) is standard mitigation to such privacy leakage. Its adaptation to the FL setting, DP-FEDAVG (McMahan et al. 2017b), provides user-level guarantees by adding Gaussian noise to the aggregated clipped model updates received by the server. In practice, training with strong privacy guarantees comes at the expense of model utility (Bassily et al.) 2014 Kairouz et al., 2019), notwithstanding efforts to close this gap, either with public pre-training and partial model updates (Xu et al. 2022). accountants with better compo- sitionality properties (Mironov) 2017) DP variants such
(Kairouz et al. 2021).
Hence, is common in practical deployments of DP-FL to train with a high privacy budget E re- sulting in loose privacy guarantees (Ramaswamy al. 2020). Such large privacy budgets often provide vacuous guarantees on the information leakage, for instance, against membership inference attacks (Mahloujifar al. (022). Encouragingly, recent work has shown that the information recov- ered in practice using of attacks is less than what theoretical bounds may allow (Nasr jet al. 202T). This suggests that DP is conservative and that a tighter measurement of the model exposure may be achieved by considering more realistic threat models.
In this paper, we propose to complement DP-FL training with novel attack method, CANaries In Federated Environments (CANIFE), to measure empirical privacy under realistic threat model. We assume that a rogue client wants to reconstruct data samples from the model updates. To make its job
Work done during an intersship Meta.

---

Published as a conference paper at ICLR 2023
50
50
Privany Budget (E)
40
40
Privecy Budge:
Budget
30
30
20
2.0
20
10
10
0
2000
4000
6000
8000
500
1000
1500
2000
2500
100
200
300
400
500
Global Round (r)
Global Round (r)
Globel Round
(a) Sent140 (2-layer LSTM)
(b) CelebA (ResNet18)
(c) Shakespeare (2-layer LSTM)
Figure 1: Empirical privacy measurements over the course FL training for LEAF benchmarks Sent140, CelebA and Shakespeare with E {10, 30, 50). We observe notable gap between the theoretical obtained with DP-FEDSGD and the empirical obtained with CANIFE.
easier, this adversary is allowed to craft outlier training sample, the canary. The training round proceeds normally, after which the rogue client performs a statistical test to detect the canary in the global noisy model update provided the server by any secure aggregation protocol (see Figure 2 Finally, we translate the attack results into per round measure of empirical privacy (Jagielski et al., 2020 Nast et al., 202T) and propose method using amplification by subsampling to compute the empirical privacy incurred during training as depicted in Figure [ for standard FL benchmarks.
Critically, our privacy attack is designed to approximate the worst-case data sample, not the worst- case update vector. The rogue client seeks to undermine the privacy guarantee by manipulating its input, which consistent with FL. environments using secure sandboxing to protect the integrity of the training process (Frey 2021). We additionally model the server the honest party, not allowing to poison the global model in order to reconstruct training samples, contrast with a recent line
of work (Fowl et al. 2021 Boenisch al. 2021 Wen et al. 2022 Fowl al. 2022).
In summary, our contributions are as follows:
We propose CANIFR (Section 3) novel and practical privacy attack FL that injects crafted canary samples. It augments the standard DP-FL training with tight measure of the model's privacy exposure given a realistic yet conservative threat model.
CANIFE is compatible with natural language and image modalities, lightweight and quires little representative data and computation to be effective. As a sanity check, we demonstrate that CANIFE tightly matches DP guarantees in toy setup (Section 4.1) before exploring how it behaves in the federated setting (Section 4.2p.
Our work highlights the gap between the practical privacy leakage and the DP guarantees in various scenarios. For instance, the CelebA benchmark, we obtain empirical measure ê zu for model trained with a formal privacy guarantee of 50.
BACKGROUND
DIFFERENTIAL PRIVACY
Differential Privacy (Dwork al. 2006 Dwork & Roth 2014) defines standard notion of privacy that guarantees the output of an al algorithm does not depend significantly on a single sample or user. Definition (Differential Privacy). randomised algorithu M D R satisfies (e. d)ffiferreti privacy if for any two adjucent datasets D' E D and any subset of outputs S C R.
P(M(D) E S) S EP(((D') e S) 6
The privacy parameter is called the privacy budget and it determines an upper bound on the infor- mation an adversary can obtain from the output of an (e. 8DP algorithm. The parameter defines the probability of failing guarantee the differential privacy bound for any two adjacent datasets. In this work, we are interested in user-level differential privacy which takes and D' to be adjacent if D' can be formed by adding removing all samples associated with single user from D.
2

---

Published as a conference paper at ICLR 2023
Standard DP results state that the privacy budget (c, accumulates (roughly) in proportion to the
square root of the number of iterations. Advanced privacy accountants leverage the uncertainty due to random sampling (Mironov 2017: Wang al. 2019b Gopi et al. 202T). We use the Rényi Differential Privacy (RDP) accounting implemented in the Opacus library (Yousefpour et al. 2021
2.2 PRIVATE FEDERATED LEARNING
A standard FL protocol, such as FEDAVG (MeMahan et al. 2017a) computes a weighted average of the model updates from clients before performing gradient descent step on the global model. Other variants exist to deal with common optimization problems in the federated setting such as convergence speed (Wang al. 2019a), heterogeneity (Karimireddy et al.] 2020a b). reducing com- munication bandwidth (Alistarh et al. 2017) and adding momentum (Reddi et al. (020). In this
work, we focus on DP-FEDSGD, a private extension of FEDAvG (MeMahan et 201 At each round, the selected clients compute their clipped model update u, and transmit it to the server, which aggregates the model updates and adds Gaussian noise:
i Σ U1 +N
In practice, federated algorithms rely on Secure Aggregation (SegAgg) protocols to aggregate each client update without revealing any individual u, to the server or to other participants (Bonawitz eet al., 2017 Bell et al. 2020). In TEE-based SecAgg (Huba al. 2022), trusted execution environment (TEE) aggregates the individual model updates and calibrated DP noise before handing over the noisy model update ū to the server. These specific models are orthogonal to our work and we assume from now on that the server and clients participate in a TEE-based SecAgg protocol. We discuss our threat model with regards our attack in Section 3.1
2.3 ATTACKS & EMPIRICAL PRIVACY
Centralized and FL Attacks. There is vast body of literature on attacking models trained in the centralised setting. For example, membership inference attacks (MIA) attempt to distinguish whether sample was present in the training given only the trained model (Shokri et al., 2017 Sablayrolles et al. (2019). Others attacks consider the more difficult problem of reconstructing entire training samples from a trained model, often using (batch) gradient information (Yin et al. 202T
Jeon et al. 2021 Balle et 2022). Since model updates u, are essentially just aggregated gradients, is natural that FL. updates may leak private information as well. Nasr et al.) (2019) show that attacks in the federated setting. Other works such as that of Fowl et al. (202T) and Wen et al. (2022) have designed attacks on model updates which allow for the reconstruction of training samples used in federated training. However, they assume that the server can poison the global model, whereas we assume an honest server.
possible to perform both passive and active membership-infereen
The notion of canary samples usually refers to natural data samples used to measure memorization in large language models (Carlini et al., 2019;) 2021 Thakkar et al., 2020 Shi et al.) 2022). For instance, Parikh et al.] (2022) propose to reconstruct canaries inserted training data and (Sook et al. (202)) insert canaries to track and quantify the information leakage when training a causal language model. In all prior work, the canary is either sample from the training set or handcrafted instance such "My SSN is 123-45-6789". In contrast, CANIFE provides an explicit method for crafting canary samples that are as adversarial as possible within the given threat model (see Appendix E for canary samples) to obtain tight measurement of the model's 's exposure. Moreover, the proposed method applies to any training modality allowing to backpropagate in the sample space (in particular, pixels for images and tokens for natural language).
Canaries.
Empirical Privacy. The proposed approach departs from existing attacks in the FL setup. For instance, Jayaraman & Evans (2019) and Jagielski et al. (2020) have derived empirical measures of privacy through attacks and often shown gaps between the empirically measured and theoretical privacy. More recently, Nasr et al. (202l) study a range of membership-infere -inferrenee attacks by varying adversary's 's powers. We argue that the threat model of many of these attacks is often too permissive relative to what a realistic adversary can achieve in the federated setting. For example, attacks in Nasr al. 2222] assume knowledge of the other samples in the dataset.
3

---

Published as a conference paper at ICLR 2023
Client Dataset D2.
SecAgg
Server
Dataset
Client
24
u = Σ 44 N10,0
Back to server
Adversary canary Dataset D. (4)
Ue x Vne(()
Adversary tries to detect canary
(Z, Me) CANIFE &
Figure 2: Illustration of the proposed CANIFE method for one training round. The adversary is rogue client that crafts a training sample resulting an extremely out-of-disrribtioon model update Me and performs a membership inference attack on the (public) aggregated noisy model update i provided to the server by SecAgg. Attack results are converted to empirical privacy guarantees é for this round. We then compound these guarantees over the course over the training as in Section 3.4
METHODOLOGY
The adversary wants design canary sample measure the empirical privacy of given FL training round under a realistic threat model defined in Section[.1 We view this problem as mem- bership inference game where a rogue client crafts canary specific to the global model at round r (Section 3.3) and uses this canary to produce an extremely out-of-distribution model update. After this, the server proceeds as usual, aggregating model updates via any secure aggregation primitive and adding noise. At end of the round, the adversary attempts to detect the presence of the canary in the aggregated noisy model update (Section (.2) and computes per-round empirical privacy measure. These per-round empirical privacy guarantees are compounded over the whole course of the training to give a final empirical privacy guarantee as explained in Section 3.4.
3.1 THREAT MODEL.
We assume honest- ---bbbuut-cctt threat model where the clients and the server do not deviate from the training protocol (Bonawitz et al.] 2017 Kairouz et al.] 2019: Huba et al.] (022). In particular, the clients cannot directly manipulate gradients nor the final model updates from the local training and the server cannot modify the model maliciously to eavesdrop on the clients.
To craft the canary for given training round, assume that the adversary rogue client has access to (1) the current server-side public model and to (2) emulated mock clients. Mock clients are created using a public set of samples called the design pool whose distribution similar that of the clients. We show in Section 4 that the adversary is able to design a strong and robust canary sample even with small amount of public data and under mild assumptions about the true data distribution. We also demonstrate that the computational resources required to design the canary are small, which means that even adversaries under resource constraints are able to launch such attacks.
We argue such a threat model is realistic and discuss practical limitations that influence the adver- sary's success in Section (3.4] our goal to design an attack that as adversarial as possible (in order to derive worst-case measure of empirical privacy) under a set of reasonable assumptions.
3.2 CANARY DETECTION
The adversary's 's objective is to craft a canary z such that the resulting model update is extremely out- of-distribution. More precisely, require that the canary gradient Vol(z) is orthogonal to all indi- vidual clipped model updates u, in the current round: (u,, Vol(z)) = 0. We demonstrate that this allows the adversary to detect the presence of the canary by separating two Gaussian distributions.
The server trains model parameterized by f E Rd with a certain level of DP noise a across a population of clients each holding local dataset D,. Since the rogue client holds dataset that
4

---

Published as a conference paper at ICLR 2023
only contains the canary D. } its clipped model update will be proportional to the canary gradient: Ue x Vol(z). Recall that the aggregated private model update à is formed by the noisy sum of individual clipped model updates:
E, u, + +O o² Ia). Then, coutterffactual if the rogue client does not participate in the training round:
(ū, Uc) x Σ (us, vol(=)) + [(0..2110
=0 by design
Hence, (ū, ue) follows a one-dimensional zero-mean Gaussian with variance o². where de accounts for the proportionality factor. Similarly, if the rogue client participates in the training round, (ū, uz) follows dinesiimnsional Gaussian with the same variance of centered at ||uell? 3 Thus, the mem- bership inference game is reduced to separating two Gaussian distributions centered at O and ||u/l respectively. Note that the result is unchanged up fixed scaling factor if the client updates u, weighted by their local dataset size as in (McMahan et al. 2017b). The testing approach is described in full in Algorithm | and involves computing the attack score s, (ü, ue). We derive a connection with the likelihood ratio test in Appendix A
3.3 CANARY DESIGN
At a given round r, the rogue client is given the current server-side model parameterized by Given set of heldout clipped model updates {uz}. it creates the canary by minimizing:
((=)= = Σ (ui, Vol(=))² m (\)o(())))
-
Recall that Vol(=) denotes the gradient of the network's parameters with respect to its training loss l when forwarding the canary z through the network. The first loss term designed to make the canary gradient Vo((z) orthogonal to the set of heldout model updates while the second term enforces the canary norm not smaller than some constant, that we set to the clipping constant C of DP-FL. This ensures ||u=||2 C2 and for simplicity we fix C = 1. In Appendix D we provide experiments that show that choosing the gradient norm constant too large (i.e., much larger than C?) has detrimental effect on optimization.
Using an automatic differentiation framework such as PyTorch (Paszke et al. 2019). we com- pute (=)(2) and perform stochastic gradient descent directly in the sample space as described in Algorithm]] [ (Recall that the model parameters 9 are fixed during the optimization procedure.) Computing V V=((2) is straightforward for continuous data, such images,
can simply back- propagate in the pixel space. For language models, the problem is more complex as the input is a sequence of discrete tokens. Hence, we leverage the work of al. (202T), who use the Gumbel- Softmax distribution (Jang et al.] 2016) to forge adversarial language examples. This allows them to use gradient-based methods by optimising probability distribution over each token in sequence.
We investigate various methods to initialize the canary, including starting from a random training sample or random pixels or tokens. Depending on the task hand, we might need to fix a target for the canary. For instance, for image classification, we need to assign the canary to a class in order to be able to compute Vol(z). We investigate various canary initialization and target choice strategies experimentally in Section 4 We also investigate other optimization considerations for designing the canary such as slightly modifying the loss in Appendix D
Adversarial Examples. We can view the canary as 'aavveraaral" in the sense that it should be extremely out-of-distribution to get a worst-case measure of privacy. This is different from the unrelated notion of adversarial examples which typically constrain the changes of a sample to be imperceptible to humans (Biggio et al. 2013).. our setup, we do not impose this constraint as we wish encompass the realistic worst-case of an extreme out-of-distribution sample.
3.4 MEASURING EMPIRICAL PRIVACY
The CANIFE attack is carried out over certain number (n) of fake rounds where the server does not update its global model but where the pool of selected (regular) clients differs every time. Hence,
We assume that the client's local optimizer is SGD with momentum McMahan al. 201
5

---

Published as a conference paper at ICLR 2023
Algorithm 1 CANIFE attack by rogue client
Input: Design pool pont Design iterations T, Canary learning rate B, Global model 0
1: Form mock clients from the design pool pool
2: For each mock client compute the clipped model update u4
3: Initialise the canary 20
D See Section
3.3
4: for .,..., T do
(ui, 5: [(z+) - (u,,, C. Vol(z))) + max(C ([Vol(
D Canary optimization loss D Gradient of the canary loss w.r.t 2 > For NLP, see Section 3.3
6: Compute V [(2)
7: Zr+11 f 2t 3. 22, (zx)
8: end for
9:
Compute ue
> Model update with D. = { Be)} ū is deemed public after the round has finished
return S,- c (ū, uc)
CANIFE has no influence on the model's final accuracy. The adversary crafts single canary and inserts it into every fake round with probability 1/2. Once the n attack scores are computed, the adversary deduces the performance of the attack at calibrated threshold 7. Building on previous work (Jagielski et al.) 2020; Nasr et al. (021), we compute the empirical privacy guarantee based on the False Positive Rate (FPR) and False Negative Rate (FNR) of the attack at the threshold y as
1 1
FPR 1 - 5 - FNR
= max
log
,log
FNR
FPR
Our measure differs slightly from that of (Nasr et al. 2021) as our attack measures privacy at a single round of training. Thus, is a per-round privacy measure which denote as é, and compare this to per-round theoretical Ey. In order to deduce an overall empirical epsilon é we convert the per-round É, into a noise estimate o, under amplification by subsampling. We then compose for steps with an RDP accountant, until we perform the next attack and update our o,. To provide a worst-case measure we choose threshold that maximises é under our attack. We note that n determines bounds on é, and its Cls. With n 100, é, is at most 3.89. Similarly, when computing ε,, we set 1/n. For further accounting details and the algorithm to compute é see Appendix C
Relationship to DP. Our threat model and canary design procedure have been chosen to ensure that the adversary is as strong as possible but still constrained by realistic assumptions. At least three factors of the design process restrict the adversary in practice:
1. The adversary only has partial knowledge of the model updates in the form of an aggregated noisy sum u. Furthermore, when designing canary, the adversary uses design pool to mock clients. In our design, we have been conservative by assuming the adversary has ac- cess heldout data that matches the federated distribution by constructing the design pool from the test set of our datasets. In practice, adversaries can form design pools from public datasets that match the task of the target model. See Appendix For further discussion.
2. The optimization process induces two possible sources of error: the convergence itself and the generalization ability of the canaries unseen model updates.
3. We calculate the maximum accuracy that can be derived from the attack and use the thresh- old that maximises é,. This is conservative, as in practice an adversary would have to calibrate threshold y on public heldout dataset. Furthermore, they would not be able to perform n fake rounds with the same canary.
If these practical constraints were not imposed on adversary, then the privacy measure we derive from our attack would essentially be tight with the DP upper bound. For example, if we could always design perfect canary that has gradient orthogonal to all possible model updates u, then the DP noise a completely determines whether we can distinguish if is present in á or not. However, in will see in Section 4.1
practice, the assumptions listed above make the attack more difficult
4 EXPERIMENTS
In this section, we begin by highlighting the power of our attack on the simplest federated setting where participants only have single sample (Section 4.1). We then show how our attack allows

---

Published as a conference paper at ICLR 2023
20
Without j...-000002. 0032, 600.006
Without (e-00.035 6.0.512
Deviation
0.025
Number of dients
15
With z, (-000081. dw0,007 007
With jrr 8955, $-0.845
0.020
16
120
32
Erequency
Proquency
0.015
10
0.010
2
0.000
0.D
0.2
0.4
0.6
0.8
1.0
500
1000
1500
2000
Attack score (5,)
Attack score (5,)
Design Pool Size (m)
(a)
(b) 0.423
Figure 3: Experiments CIFAR-10 on a toy setup where each client holds a single data sample. (a) Attack histograms without DP noise, showing that our attack completely succeeds. (b) Attack his- tograms with DP noise: as the DP noise increases, it becomes harder separate both histograms. (c) Varying the design pool size and the number of participating clients. The y-axis shows the mean standard deviation of the histograms formed from the attack.
us to monitor privacy empirically over a federated training run (Section 4.2 before concluding with
an ablation study that shows our canary is robust to various design choices (Section 4.3) source the code for CANIFE design and test to reproduce our results
We open-
Setup. We evaluate our attack on both image and language models. We utilise LEAF (Caldas et al., 2018) which provides benchmark federated datasets for simulating clients with non-IID data and varying number of local samples. We study image classification on CIFARIO (IID) (Krizhessky) al. 2009) and CelebA (non-IID) (Liu et al. 2015). We train a simple Convolutional Neural Net- work (CNN) and a ResNet18 model. For our language tasks, we train an LSTM model on non-lII splits of Sent140 (Go et al.] 2009) and Shakespeare (McMahan et al.) 2017a). For more infor- mation on datasets, model architectures and training hyperparameters, see Appendix B We train, design, and evaluate our attacks using the FL.im framework All experiments were run on a single A100 40GB GPU with model training taking at most few hours. We discuss CPU benchmarks for canary design in Section 4.3 For canary design, we use the Adam optimizer (Kingma & Ba 2014) with learning rate 3 = I and fix C' 1. We form the CANIFE design pool from LEAF test split, resulting in canaries designed on non-IID mock clients which approximates the training distribution. We have clients perform single local epoch in all experiments, but see Appendix F for possible extensions to multiple local epochs. From now, we refer to "epoch" as one pass of the federated dataset (in expectation). For privacy accounting, we utilise the RDP accountant with subsampling (Mironov et al.) 2019) implemented via the Opacus library (Yousefpour et al.) [021], sampling clients uniformly at each round (i.e., Poisson sampling). see Appendix C for more details.
4.1 EXAMPLE ATTACK: CIFARIO
We first investigate the simplest federated setting where each client holds single sample. It follows that the clipped model update u, is simply a scaled gradient of the client's single sample. This corresponds to the central setting where the number of clients per round is equivalent to a (central) batch size. In Figure ] we train a ResNet18 model on CIFAR 10 to 60% test accuracy and perform attack. We design a single canary and have the server perform
100 mock rounds where the model is frozen, with half having the canary client inserted and half without. We compute attack scores from these 100 trials and form histograms.
In Figure 3a we present one histogram of the attack scores on model that has 64 clients partic- ipating per round with no DP We observe that our attack is quite tight in this non- non-private setting. Since o O, we hope that if the canary was designed well, the standard deviation of the histograms would also be close to O. It turns out that the average standard deviation of the histograms is 0.006. Recall in Section 3.4 we discussed there is inherent error from both the optimization procedure and the fact the server
oc). We use a design pool of m = 512 design the canary.

---

Published as a conference paper at ICLR 2023
12.5
10
Privany Rodgett
15
Privacy Budpet
10.0
10
7.5
5.0
MM
0.0
2000
4000
8000
500
1500
2000
2500
CelebA
Sent140 Dataset
Shalesseare
Global Round
Global Round
(a) S4nt140 (2-layer LSTM)
(b) CelebA (ResNet18)
(c) CANIFE empirical epsilon
Figure 4: (a) and (b) Monitoring per-round empirical privacy on LEAF benchmarks. Models are trained to e = 50 with 100 clients per round. Cls for é, are compared with the theoretical per-round epsilon S,, rounds with an upper CT of do do not have CIs displayed. (c) CANIFE empirical privacy incurred during the whole training, averaged over 5 independent runs (see Figure I for details).
designs the canary on heldout data but in this case the error is small. In Figure 3b we display an-
other example attack, this time for model trained under privacy with a final = =25 corresponding to a 0.423 with 64 clients per round. We observe the attack is still fairly tight as the standard deviation of the histograms (average 0.478) is close to that of the privacy noise a.
Finally, we explore how both the design pool size and the number of clients affect the standard deviation of the histograms (Figure 3c). We vary both the number of clients and the design pool size. We train model without privacy for each combination of parameters until it reaches 60% train accuracy and then attack the model, plotting the average standard deviation the histograms. We conclude with two observations. First, the attack is much tighter when there is smaller number of clients participating per round. Second, the size the design pool has a diminishing effect on reducing the standard deviation. We further explore these in Section 4.3
4.2 MONITORING EMPIRICAL PRIVACY
One main goal of our attack is to provide lightweight method for monitoring empirical privacy during federated training. We explore this in Figure 4 training ResNet18 on CelebA and 2-layer LSTT model on Sent140 and Shakespeare. We train to final accuracy on Sent140, 89.9% on CelebA and 44.8% on Shakespeare. We carry out the CANIFR attack at a regular interval, freezing the model, designing a single canary, and performing n = 100 attack trials with the designed canary before continuing training. This generates scores which are used to compute an empirical per-round privacy measure é, as outlined in Section 3.4 (see also Appendix C.3 for further details). The canary sample 2 is initialised randomly and we explore how this affects optimization in Section 4.3 We note in practice that participants (and the server) would not want to waste rounds with a frozen model, see Appendix for possible extensions.
50 and achieve 63.9% test
We monitor the optimization of the canary loss by computing the canary health which measures the percentage improvement of the final canary loss compared to the initial loss. perfect health of 1 means the canary achieved minimal design loss. As an example, for Sent140, we find that the average canary health across 5 separate training runs is 0.970 + 0.098, Excluding the first 1000 rounds from each run, the average canary health becomes 0.990 + 0.01. Thus, after two model epochs, optimization stabilises and the initial canary loss is reduced by 99%.
In Figure 4al we display per-round privacy estimates é, and their 95% confidence intervals (CIs) for Sent trained with = 50 and compare to the (constant) theoretical per-round privacy E,. We observe é, is initially small and grows to an almost constant level within single epoch and stays there during training. This results in a 4.5x gap between the theoretical c, and & measured by our attack. We obtain similar 4 - 5x gaps for CelebA (ResNet18) in Figure 44b and Shakespeare in Appendix C.5 We compound these per-round estimates E, to provide cumulative measure of privacy and display this in Figure for models trained with = E {10,30 50). Again, there is significant gap between the final theoretical privacy and the measure derived under CANIFE. The final é averaged over separate training runs for each tasks are shown in Figure 4c) We note significant gaps between tasks, most notably with Sent140. This is likely due to the small sample
8

---

Published as a conference paper at ICLR 2023
Moen Standard Declation
1.0
1.0
Canary Initialisation random Image
Accuracy
0.9
Accuracy
0.9
0.6
0.8
0.8
0.4
0.7
0.7
0.2
0.8
5000 10000 15000 20000 25000
1000
2000
3000
Design Sample
Design Bteratioss
Test Accuracy
(a) Canary Initialisation
(b) Design Iterations: Varying T
(c) Model test vs attack accuracy
Figure 5: Ablations studies for CANIFE without addition of DP noise. (a) Canary initialization when varying the design pool size (CelebA, 4-layer CNN). (b) Number of design iterations required to produce robust and well-designed canaries (CelebA, 4-layer CNN). (c) Model vs attack accuracy across 5 runs (Sent140, 2-layer LSTM). Attack accuracy increases model test accuracy increases.
rate which provides significant privacy amplification. We also observe CANIFE estimates are stable across runs with relatively small standard deviation.
4.3 ABLATION STUDY
Canary Initialisation. In Figure Sal we explore how initialising the canary affects the optimiza- tion. We consider two initialisation strategies: initialising the canary randomly initialising the canary as a sample from the design pool (which is then excluded from the design). We observe no significant difference on the average standard deviation of the attack histograms.
Design Pool Size. In Figure 5a we vary the design pool size. We observe there is significant effect on the average standard deviation of the attack histograms. This confirms what we observed in Figure 3e the design pool size has diminishing impact reducing the standard deviation.
Design Iterations. In Figure 5b we explore how the number of design iterations impacts the
quality the canary measured through the calibrated attack accuracy. We observe that just 1000 iterations are needed to obtain 95% accuracy and that with i = 3000 the attack improves to almost 100% accuracy, staying close to constant as increases further. We additionally benchmarked the average CPU time takes an average of 0.06s per iteration and 0.23s for CelebA (ResNet18). For t = 2500, the design procedure took average 165s and 591s respectively. Hence, the canary design process takes at most 10 minutes on CPU and only a few minutes with a GPU. Thus our attack is lightweight, requiring only few thousand design iterations to achieve near-optimal canaries.
MI MacBook Air (2020) for single design iteration. For Sent it
Model Accuracy. We conclude by noting the accuracy of the model and that of our attack are highly correlated. Figure 5c we plot both model test accuracy and calibrated attack accuracy across Sent140 trained without DP. We observe early in training, when model accuracy is low, that the attack accuracy is similarly low. Once the model converges to a sufficiently high test accuracy the attack accuracy close to 100%. In experiments that require comparison across different models (e.g., Figure 3cb we checkpoint to fixed test accuracy to avoid this confounding effect.
5 CONCLUSION
Motivated by the fact that DP is conservative, we consider more realistic threat model to extract private information about client data when training a model with FL. We introduce CANIFE, a novel method to measure empirical privacy where a rogue client crafts canary sample that results in an outlier model update. We argue that the difficulty of tracing this malicious model update in the aggregated noisy model update provides tighter measurement of the model's privacy exposure. We hope this work can benefit practical deployments DP-FL pipelines by complementing the theoretical bound with arguably more realistic measure of the privacy leakage.

---

Published as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, lan Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Dan Alistarh, Demjan Grubic, Jerry Li. Ryota Tomioka, and Milan Vojnovie. QSGD:
Communication--fficieee SGD via gradient quantization and encoding. Advances in neural formation processing systems, 30, 2017.
Borja Balle, Gilles Barthe, Marco Gaboardi, Justin Hsu, and Tetsuya Sato. Hypothesis testing interpretations and renyi differential privacy. International Conference on Artificial Intelligence and Statistics, pp. 2496-2506. PMLR, 2020.
Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data with informed adversaries. arXiv preprint arXiv:2201. 044445,, 2022.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Differentially private empirical risk minimiza- tion: Efficient algorithms and tight error bounds, 2014.
James Henry Bell, Kallista A Bonawitz, Adrià Gascón, Tancrède Lepoint, and Mariana Raykova. Secure single-server aggregation with (poly) logarithmic overhead. In Proceedings of the 2020 ACM SIGSAC Conference Computer and Communications Security, pp. 1253-1269, 2020.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šındić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Franziska Boenisch, Adam Dziedzic, Roei Schuster, Ali Shahin Shamsabadi, Ilia Shumailov, and Nicolas Papernot. When the curious abandon honesty: Federated learning is not private. arXiv preprint arXiv:2 /2.02291 2021.
Keith Bonawitz, Vladimir Ivanov, Ben Kreuter, Antonio Marcedone, H Brendan McMahan, Sarvar Patel, Daniel Ramage, Aaron Segal, and Karn Seth. Practical secure aggregation for privacy- preserving machine learning. proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175-1191, 2017.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. LEAF: A benchmark for federated settings. arXiv preprint arXiv: 18/2.000/ 2018
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284. 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voos Katherine Lee. Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. 30th USENIX Security Symposium (USENIX Security 21), pp- 2633-2650, 2021.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 2014.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pp. 265-284. Springer, 2006.
Liam Fowl, Jonas Geiping, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Robbing the fed: Directly obtaining private data in federated learning with modified models. arXiv preprint 1rXiv:2110.1308: 2021.
Liam Fowl, Jonas Geiping, Steven Reich, Yuxin Wen, Wojtek Czaja, Micah Goldblum, and Tom Goldstein. Decepticons: Corrupted transformers breach privacy in federated learning for language models. arXiv preprint 0at.:2201.22775 2022.
10

---

Published as a conference paper at ICLR 2023
Suzanne Frey. Introducing Android's private compute services. 2021.
URL https://
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients how easy is it to break privacy in federated learning? Advances Neural Information Processing Systems, 33:16937-16947, 2020.
Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision. CS224N project report, Stanford. 1(12):2009, 2009.
Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy.
Advances in Neural Information Processing Systems, 34: 14:11631-11642, 2021.
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. Gradient-based adversarial attacks against text transformers. arXiv preprint 100. 2021.
Samyak Gupta, Yangsibo Huang. Zexuan Zhong, Tianyu Gao, Kai Li, and Danqi Chen. Recovering private text in federated learning of language models, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pailern recognition, pp- 770-778, 2016.
Dzmitry Huba, John Nguyen, Kshitiz Malik, Ruiyu Zhu, Mike Rabbat, Ashkan Yousefpour, Carole- Jean Wu, Hongyuan Zhan, Pavel Ustinov, Harish Srinivas, et al. Papaya: Practical, private and scalable federated learning. Proceedings of Machine Learning and Systems, 4:8 4:814-833 832, 2022.
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? Advances in Neural Information Processing Systems, 33: 02205-22216, 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizzttoo with Gumbel-Softma arXiv preprint arXiv: 16tv:6611..0 1.01 144, 2016.
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice.
In 28th USENIX Security Symposium (USENIX Security 19), pp. 1895-1912, 2019.
Jinwoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, et al. Gradient inversion with generative image prior. Advances in Neural Information Processing Systems,
-999898-99988, 2021.
Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning, pp. 1376- 1385. PMLR, 2015.
Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D'Oliveira, Hubert Eichner, Salim El Rouayheb. David Evans, Josh Gardner, Zachary Garrett, Adrià Gascón, Badih Ghazi, Phillip B. Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo, Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak, Jakub Konečný, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancrède Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Özgür,, Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang. Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated learning. 2019.
Peter Kairouz, Brendan McMahan, Shuang Song, Om Thakkar, Abhradeep Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling shuffling. In International Conference on Machine Learning, pp. 5213-5225 PMLR, 2021.
Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebas- tian Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.036000 2020a.

---

Published as a conference paper at ICLR 2023
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference Machine Learning, pp. 5132-5143, PMLR, 2020b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:14/2.6980 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Saeed Mahloujifar, Alexandre Sablayrolles, Graham Cormode, and Somesh Jha. Optimal member- ship inference bounds for adaptive composition of sampled gaussian mechanisms, 2022.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-eficient learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1273-1282. PMLR, 2017a.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.06663, 2017b.
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer security foundations sympo- sium (CSF), pp. 263-275. IEEE, 2017.
Iya Mironov, Kunal Talwar, and Li Zhang. RR'en enyi differential privacy of the sampled gaussian mechanism. arXiv preprint arXiv: /908. 10530, 2019.
Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pp. 739-753. IEEE, 2019.
Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papemoti, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 866-882. IEEE, 2021.
Rahil Parikh, Christophe Dupuy, and Rahul Gupta. Canary extraction in natural language under- standing models. arXiv preprint arXiv:2203./3922 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, perf-peeformance deep learning library. In Advances in Neural Information Processing Systems 32. 2019.
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. Training production language models without memorizing user data. arXiv preprint ar arXiv:2009. 2020.
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and Brendan McMahan. Adaptive federated optimization. arXiv preprint arXiv:2003. 00002 2020.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White- box vs black-box: Bayes optimal strategies for membership inference. In International Confer- ence on Machine Learning, pp. 5558 5567.5567.11 PMLR, 2019.
Weiyan Shi, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. Just fine-tune twice: Selective differential privacy for large language models. arXiv preprint arXiv:2004.06667, 2022.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
12

---

Published as a conference paper at ICLR 2023
Pierre Stock, Igor Shilov, Ilya Mironov, and Alexandre Sablayrolles. Defending against reconstruc- tion attacks with Rényi differential privacy. arXiv preprint arXiv:2202.076623, 2022.
Om Thakkar, Swaroop Ramaswamy, Rajiv Mathews, and Françoise Beaufays. Understanding unin- tended memorization in federated learning. arXiv preprint urXiv:2006.074900 2020.
Jianyu Wang, Anit Kumar Sahu, Zhouyi Yang, Gauri Joshi, and Soummya Kar. Matcha: Speed- ing up decentralized SGD via matching decomposition sampling. In 2019 Sixth Indian Control Conference (ICC), pp. 299-300. IEEE, 2019a.
Yu-Xiang Wang, Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled Rényi differential privacy and analytical moments accountant. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 1226-1235. PMLR, 2019b.
Yuxin Wen, Jonas Geiping, Liam Fowl, Micah Goldblum, and Tom Goldstein. Fishing for user data in large-batch federated learning via gradient magnification. arXiv preprint arXiv:2202.0055 005800 2022.
Mingbin Xu, Congzheng Song, Ye Tian, Neha Agrawal, Filip Granqvist, Rogier van Dalen, Xiao Zhang, Arturo Argueta, Shiyi Han, Yaqiao Deng, Training large-vocabulary neural lan- guage models by private federated learning for resource-constrained devices. arXiv preprint arXiv:2207.08988, 2022.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 16337-16346, 2021.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica 7hao, al. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprini arXiv:2 12298, 2021.
13

---

Published as a conference paper at ICLR 2023
A CONNECTIONS TO LIKELIHOOD RATIO TEST
Let assume that each model update follows Gaussian distribution (u, E). The sum of & model updates then either Wollowss(kkpp kλ) (without the canary) or NK + V((2), (with the canary), recalling that ue Vε(z). Then we have
Po(u)= /et((2kk))
exp (- k1) (1:E)-
(K=)) (u kp)/22)
1
Pi(u)
(dct(2 exp (-(u (kμ V((2))) (ky
+k V((=)))/22
[det(27k2) exp (-((u k4) v()))
ky) ((x))))))))))
We can write the (log) likelihood ratio
Pi (i(
1
log po(u)
((u
kp) - ((u kp) ( kp) VE(=)))
2
1
2 ((u kp)) (kE) -u kp) - ((u kp) ((E)) ((u- ky) V((2)))
1 2 (12)))
k())
In particular for the centers of the Gaussian with and without the canary, E (kp, V((2)).
Pi Pi(u) (o(u))
log
- (A2) VV(())
Maximizing this term will thus help separate the two Gaussians. However, doing this directly is infeasible as it requires to form and invert the full covariance matrix in very high dimensions. it is tractable and can be done with
Instead, propose to minimize SGD. Note that for sample model updates {u,) we can estimate the (uncentered) covariance matrix as n Σ u and thus
(V(()))))(V((()
1 _ (un 1 - E(4,, (u,, v
(V(+))))()))) ~
Which, ignoring constants, the first term L(2) defined in Equation I One could alternatively minimise ()) with j E, uz. We explore the empirical differences in Appendix D
B DATASETS & MODEL ARCHITECTURES
Here we detail the training setup and model architectures for our experiments. In all experiments we train with DP-FEDDGD and without momentum. We use both client (nc) and server (ms) learning rates. We train without dropout in all model architectures. In more detail:
CIFARIO is an image classification tasks with 10 classes (Krizhevsky et al. 2009). We train ResNet18 model (He al. 2016) CIFAR10. We form III split of 50, 000 train users and 10, 000 test users where each user holds a single sample and thus has a local batch size of 1. We use a client learning rate of nic 0.01 and server learning rate n/ 1.
CelebA is a binary image classification task (Liu et al. 2015). We train a ResNet18 model with 11, 177, 538 parameters and simple Convolutional Neural Network (CNN) with four convolutional layers following that used by Caldas et al. (2018) with 29, 282 parameters. We the standard non-IID -ID LEAF split resulting in 8408 train
and 935 test users and a local batch size 32. We train with a client learning rate of nc = 0.899 and a server learning rate of is 0.0797.
14

---

Published as a conference paper at ICLR 2023
Algorithm 2 Measuring 6
Input: Number of rounds R. Privacy parameter 6, Sampling rate g, Number of attack scores n, Attack frequency
1: for ....., R do
2: i mod = 0 then
3:
Freeze the model 0, and use Algorithm 1 compute attack scores {8i) Calculate FPR, and FNR, from {&i) at each threshold E {81,...,8Hi 8n]
4:
Compute & + max, Ioog
1-8 ,..,,,,
FNR. .log FNF..
5:
FPR,
6:
Compute 8, f- GetNoisee(r,
Estimate one-step noise multiplier
7:
0,44 o, for i E (1,8)
6, is the estimate for rounds (r, 8)
8:
end
9:
end for
10:
return t- GetPrrvvacy
& Compose each noise estimate under an RDP accountant with amplification by subsampling using sample rate q, see (Mironov et al. 2019)
10
20
Privacy Budget
15
10
100
200
300
400
500
100
200
300
400
500
Global Round (n)
Global Round (r)
(a) Shakespeare per-round privacy (e = 50)
(b) Shakespeare per-round privacy (s = oc)
Figure 6: Measuring privacy during Shakespeare training
Sent140 is a sentiment analysis (binary classification) task (Go al., 2009). We train a 2- layer LSTM with 272, 102 parameters Sent140 following the architecture of Caldas al. (2018). We use standard non-lID LEAF splits resulting in 59, 214 train users and 39, 477 test users and local batch size of 32. We train for 15 epochs, and for 0 achieve an average test accuracy of 64.8%. We train with a client learning rate of mJC = 5.75 and a server learning of 7J5 0.244.
Shakespeare is a next character prediction task with 47 classes (McMahan et al. 2017a). We train similar LSTM model to Sentl 40, based on the architecture used by Caldas et al. (2018) with 819, 920 parameters. We use standard non-IID LEAF splits with 1016 train and 113 test users and a local batch size of 128. We train models for 15 epochs resulting in an average final test accuracy of 44.4% for = O. We use a client learning rate of '|c and server learning rate of iss 0.524.
C MEASURING EMPIRICAL PRIVACY
C.1 PRIVACY MEASURES AND ACCOUNTING
Here we provide further details about privacy accounting and the different privacy measures that we analyse. In all experiments use Rényi Differential Privacy (RDP) accounting with subsampling (Poisson sampling over the set of clients) to guarantee user-level DP. This is based on the DP- FEDAVG algorithm (McMahan et al., 2017b) with accounting implemented via the Opacus library (Yousefpour et al.] (021). More specifically. the accounting uses the RDP subsampling analysis derived from Mironov et al. (2019) and the RDP to (e, 0DP conversion from Balle al. (2020).
15

---

Published as a conference paper at ICLR 2023
0.80
0.9
Arcurric
Atfack Ancurary
WAD
Attack Accussy
0.75
0.8
0.70
0.7
0.63
0.6
1.5
2.0
2.5
3.0
3.5
4.0
1.5
2.0
2.5
3.0
2.0
2.5
3.0
3.5
4
(a) Sent140 (2-layer LSTM)
(b) Shakespeare (2-layer LSTM)
(c) CelebA (ResNet18)
Figure 7: Per-round ε, estimates against attack accuracy. Estimates are computed over training runs with a final theoretical & = 50.
Acmrray
0.9
0.8
0.82
Loss
0.80
2
03
Loss 2
100
0.78
0
5000
10000 15000 20000 25000
40
60
100
120
2000
4000
6000
Design Sample Size
Clients Per Round
Global Round (O)
Loss modifications
(b) Varying number of clients per round: Attack accuracy
(c) Varying number of clients per round: Empirical
Figure 8: Further Ablations on CelebA; (a) CNN (b), (c); ResNet18
In this work, we have exactly four different privacy measures. For the theoretical quantities we have the per-round guarantee E, and final privacy guarantee e which are computed as follows:
E,- -his is the per-round theoretical epsilon that is derived from the RDP accountant when the subsampling rate is set to 1 and number of steps R = 1. This is a constant value (dependent on the noise multiplier a and d) and is the privacy guarantee of performing single step of DP-FEDAVE
E This is the theoretical (c, 6)-DP guarantee of the model trained under (user-level) DP. We calculate this via an RDP accountant with subsampling where the sampling rate is chosen to be the number of participating clients over the total number of clients in the population. Since are using Gaussian noise this corresponds to the RDP analysis of the Subsampled Gaussian Mechanism (SGM), see (Mironov et al.) 2019) for technical details.
Similarly, we have analogous empirical measures & and which are computed follows:
é Since the attack used by CANIFE infers membership of the canary update particular round, the privacy measure derived from a set of CANIFE attack scores is a per-round measure. This is computed via the formula derived in Kairouz et al. (2055 i.e.,
8 FPR, FNR,
.log 1- - FNR,
6, max log
FPR,
where FPR, FNR are computed from the attack scores at round r. In our experiments we maximise over the threshold y provide worst-case measure. The quantity é, is directly comparable to the theoretical é,. We also compute 95% confidence intervals (CIs) for é, from the attack scores via the Clopper-Pearee Pearson method as Nasr al. (2021).
غ This is the empirical privacy measure of the model derived from CANIFE. One could apply basic composition to ?, over R rounds to obtain the empirical measure of RE, but this results in suboptimal composition. Instead we compute é under the tighter composition
16

---

Published as a conference paper at ICLR 2023
Canary Sample after optimization
AA.... V2,E > &3 Yet confess that often thisMday,x mnPlmitw77DNAA) :XXII Pmr 6Fa th, nothing bu; aZ. empty box, sir, wQich in my lord's beealf cote to e? rtat
m 'd.......d.......
Table I: Canary Samples on Shakespeare.
of RDP with amplification by subsampling. To do so, convert each é, into equivalent noise multiplier o, and compound the noise over a number of rounds with the accountant. The quantity é directly comparable See Section
C.2 for more information.
C.2 ALGORITHM FOR E
In Section 3.4
we explained how we obtain a per-round privacy measurement é, from CANIFE and compound this to form global privacy estimate over a training run. We detail this method in Algorithm 2 In order to compound our per-round estimates é, from CANIFE, since we only attack the model every s rounds we assume that the noise estimate o, remains constant between rounds and + 8 before we attack the model again and re-estimate the noise o,.
C.3 DETAILS FOR MONITORING PRIVACY
In Section 4.2 we present experiments using CANIFE to measure empirical privacy during the train- ing run of federated models. We ran each training run five times and examples of these runs are displayed in Figure Here we provide extra details of the training setup:
CelebA. We train a ResNet18 model for 30 epochs and have 100 clients participate per training round. This results in 85 rounds per epoch. We freeze and attack the model every
== 40 rounds resulting in 64 empirical privacy estimates (É,) across training.
Sent140. We train for 15 epochs and have 100 clients participate per training round. This results in 593 rounds per epoch and 8895 rounds in total. We freeze and attack the model every s = 100 resulting in 90 empirical privacy measurements across training.
Shakespeare. We train for 30 epochs and have 60 clients participate per training round. This results in 17 rounds per epoch and 510 training rounds in total. We freeze and attack the model every s 8 rounds resulting 64 empirical privacy estimates across training.
C.4 RELATIONSHIP BETWEEN Ê, AND ATTACK ACCURACY
We display the relationship between the per-round measurement é, and the accuracy the attack for models with final privacy of 50 in Figure We also found a consistent relationship for e E {10,30) 30).
C.5 FURTHER EXPERIMENTS: SHAKESPEARE
In Figure 4 we displayed example per-round estimates é, across training runs for CelebA and Sent140. In Figure 6a we display similar plot but for an example training run on Shakespeare. We also note that one can use CANIFE to measure é for models that are trained without privacy. In Figure 6b) we show an example training run on Shakespeare without DP (E oc). As mentioned in Section the number of attack trials determines an upper bound on ε,. Here we can see that the empirical privacy measure is essentially constant throughout (non (oon-private) training and consistently reaches these bounds. We note that é, = oo only once (when the maximum attack accuracy reaches 100%).
D FURTHER ABLATION STUDIES
Loss modification. As discussed in Appendix A we can minimise the dot-product of the canary with the average model update ("Loss 1"") the covariance loss ("Loss 2") which choose to
17

---

Published as a conference paper at ICLR 2023
Attack Accuracy
0.9
Canary Health
0.8
0.7
0.6
0.6
200
400
600
1000
400
600
1000
Canary Cradient Norm Constant
Canary Gradient Norm Constant
(a) Attack accuracy
(b) Canary health
Figure 9: Varying the canary gradient norm constant CelebA (CNN)
use in all experiments. In Figure Sa we vary the design sample size and carry out 5 attacks with checkpointed CNN model on CelebA while varying these losses. We observe that there is clear difference in accuracy between the two losses, with Loss 2 having consistently high attack accuracy which is almost constant we increase the sample size. We note that Loss seems more sensitive the total number of design samples, with increasing attack accuracy as the design sample size increases.
Number of clients per round. In Figure 8b we vary the number of clients per round and plot the average attack accuracy over 5 training runs. We note that the attack accuracy decreases slowly the number of clients increases which is consistent with the observations made in Figure Be In Figure 8c we plot the empirical estimate é during training. We find that although the average accuracy decreases, this does not have a significant effect the final estimate which decreases as the number of clients per round decreases.
Canary gradient norm. To conclude, we investigate the effect of the canary gradient norm con- stant the second term of the loss C(2). We fix the privacy clipping constant to be the gradient norm constant on CelebA. We use checkpointed CNN model trained without DP to 70% test accuracy. We attack the model 10 times and average the results. In Figure Dal we display the average accuracy of the attack and in Figure 9b the average canary health as we vary the gradient norm constant. We observe that choosing the constant too large can significantly decrease the effi cacy of the attack and that choosing the constant around C' is enough to guarantee high accuracy and well-behaved optimization. We note that the canary health does not significantly decrease until the norm constant is chosen be very large (>> 600). yet the attack suffers large drop in accuracy for constants > 50. This implies that as you increase the constant, it is first possible to design a canary that has large gradient norm but at the expense of orthogonality to the other model updates, and as the constant gets very large, it becomes too difficult to optimise either term of L(z).
and vary
E CANARY SAMPLES
We display in Table I some examples of designed canaries with CANIFE when initializing the craft- ing procedure with training sentences on Shakespeare.
F LIMITATIONS AND EXTENSIONS
In our experiments we have made assumptions for CANIFE that may be limiting in specific practical FL scenarios. In this section, highlight such limitations with various extensions for CANIFE.
have restricted clients to performing single local epoch for simplicity. But in practice clients often perform multiple local epochs and this has shown to help improve the convergence of models trained in FL. CANIFE is not limited methods
Multiple Local Epochs. Throughout Section 4
18

---

Published as a conference paper at ICLR 2023
with single local epoch and in practice to audit methods like DP-FEDAVG with multiple local epochs there are two main solutions:
1. Use CANIFE as is: There is nothing in the current formulation of CANIFE that does not apply to multiple local updates. Since in practice, the server will mock the adversary, they are free to choose how many local epochs the canary client performs. Thus, they can design a canary with a single local epoch to be orthogonal to clients who do multiple local epochs. We also believe this is most practical for an adversary since it is the easiest from the attackers viewpoint to design and optimize for.
2. Modify CANIFE loss: An alternative approach is to design canary sample that has model update (formed from multiple local epochs) that is orthogonal to all other model updates (which are also formed from multiple local epochs). To carry out such a design we can modify the CANIFE loss in equation [| to include the canary model update ue instead of the canary gradient Vol(z). In order to calculate the gradient of such a loss it requires backpropagating through the multiple local updates and essentially "unrolling" the local SGD steps. This may be computationally burdensome, and so while it is possible to design such canary sample, it may not be practically viable for any adversary (and/or server) depending on the model size and number of local epochs. Thus in practice, may be simpler to audit the model via (1).
Multiple Privacy Measurements. One current limitation of CANIFE is that it only produces a single privacy measurement (ε,) at a specific round. In practice, model auditors may want a more comprehensive empirical analysis of the model's 's privacy via multiple measurements, such attacks that vary the threat model like that of Nasr et al. (202I). We believe CANIFE can be extended to support a "multiple measurement" approach with ease since there is some degree of freedom in the canary design. For example, by using different design pools, each one strengthening the adversary further (e.g. with more data and/or design pools that better approximate the federated distribution based on prior knowledge) and thus obtain a more holistic measure of the model's I's privacy. One can similarly vary the number of design epochs to simulate clients with limited computation. Designing multiple canaries per round under different constraints will generate a set of empirical epsilons that allow for more fine-grained statistics (for example, taking the maximum of the per-round empirical epsilons for a worst-case measure).
Preventing Wasted FL Rounds. Another limitation of CANIFE is that in our experimental setup we freeze the model for set number of rounds to compute attack scores of a designed canary. In practice, the server and clients would be unwilling to waste federated rounds to compute CANIFE scores. We believe CANIFE can be extended to support more practical attack without wasting training rounds as follows:
Multiple Canaries: One alternative that the server could design multiple canaries at a single round, and use these to obtain (multiple) attack scores (subject to compute limita- tions). This can help reduce the number of mock rounds being run.
Running measurements: The previous approach may be computationally prohibitive de- pending on server resources and still requires frozen rounds. An alternative could be to maintain "rumning" attack measurements where we allow CANIFE to run alongside normal model training, letting the model change at each round. In this setup, the server can design a canary at each round, calculate attack scores and then proceed with updating the global model (without the canary inserted). This has minimal overhead to the server (who just needs to design the canary) and no additional overhead to clients (who just believe they are participating in standard FL training round). The set of attack scores can be used to calculate empirical epsilons over various periods of training and one can change this period however they like. We emphasize that since our results (specifically Figure 4 show the per-round empirical measure is fairly stable after the first epoch or so of training, then the approach described here should still give stable results (even though the model would be changing at each round).
Design Pool Assumptions.
The design pool is an important component of our attack, and in order to present a conservative (worst-case) privacy measure in this work, we assume that the adversary
19

---

Published as a conference paper at ICLR 2023
has access to held-out data that approximates the federated training data well. To do this, form the design pool from the test set our datasets but in many practical scenarios this is not possible since the server may not know much about the (private) federated data. While this is a limitation of CANIFE, we believe that for many tasks it would be possible to form design pool from public data. For example, if the adversary knows the exact task of the model (e.g. sentiment analysis) then the adversary can form a design pool from public datasets (e.g. Sent140) or even craft their own language data for canary design. We expect that this would be reasonable proxy for the true federated dataset and would not significantly affect privacy measurement. In scenarios where the server has no prior knowledge, they could utilise private synthetic data generation methods or Federated Analytics (FA) to privately compute statistics about client data to guide the choice of design pool.
20


Reviews:
[{"review_id": "rt8S12lBcr", "reviewer": "Reviewer_FybW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces CANIFE, a method for auditing the empirical privacy guarantees of federated learning algorithms. In short, CANIFE does the following: It considers a canary client with a single data point that contributes in k \"frozen\" rounds, and doesn't contribute in k other  \"frozen\" rounds (frozen means the global model update is computed but not applied). The data point is chosen to be as out-of-distribution as possible, using a small number of held out examples from the data distribution; in particular, the authors propose a loss function over the data domain whose optimizers should be data points whose gradient is orthogonal to the gradients of the held out examples, and as large in norm as possible. One can then compute an \"attack score\", equal to the dot product of the aggregated round update and the canary client's computed update. Comparing the attack scores for each set of k rounds, one can look at a hypothesis test for detecting the canary based on thresholding the attack score, and compute a per-round empirical (epsilon, delta)-DP for all delta using the false positive/negative rates of this test, which can be used to estimate the privacy guarantee of the whole algorithm. The authors argue this is an effective empirical auditing scheme since the adversary is much less power than an information theoretic one (e.g., no knowledge of the other data samples participating in a round, can only generate adversarial data points rather than adversarial model updates), but still has more power than an adversary in practice (e.g., has access to held out examples, gets to participate in multiple rounds).\n\nThe authors perform experiments auditing federated model training using CANIFE. The authors show that for a simple CIFAR10 training setup where every client has one example, the noise multiplier corresponding to the difference between the distributions of attack scores is a very good estimate of the actual noise multiplier used in training. For more realistic setups, the authors consider the CelebA, Sent140, and Shakespeare tasks and compute empirical epsilon values using CANIFE. The authors compute empirical epsilons varying from 4-7 for theoretical epsilons of 10, 30, 50.", "summary_of_the_review": "Overall, I lean towards rejecting the paper due to the aforementioned concerns. I think the paper carries a very nice message, that by using constrained adversaries we can get more reasonable practical epsilon estimates, and the problem of optimizing over data points in a canary is a very interesting one. But the fact that the paper seems tailored to single-client-epoch algorithms and uses a suboptimal accountant for its theoretical bounds makes it hard to discern if CANIFE is broadly applicable to a variety of federated learning algorithms, and how much of the gap in empirical and theoretical epsilons is due to a suboptimal accountant. So while I really like the ideas in the paper, it's hard to be confident in their level of impact. Furthermore, I think since the paper is breaching into a somewhat new space (so e.g., it is unclear if the problem setup will still be interesting in a few years, it's unclear how the empirical results would compare to other approaches), it needs to be held to a somewhat higher standard. ", "strength_and_weaknesses": "The main strength of the paper is the approach of designing the CANIFE algorithm. I felt the authors did a good job motivating the assumptions. The assumptions on the adversary do lie in a nice space where they are slightly more powerful than what is possible in practice, but still far-removed from an information theoretic adversary, and I haven't seen such a set of assumptions before. The assumption that one can only generate model updates from data points within the data domain rather than arbitrary model updates is especially a natural one. Furthermore, the method for designing canary data points rather than canary model updates is intuitive and elegant. Overall the method is lightweight and I think would be easy to apply in practice on a variety of federated model training algorithms. In addition, pending the caveats in the subsequent bullet points, I think the experiment results convey an interesting message, that a slightly powerful adversary still exhibits a reasonably large separation between its empirical privacy guarantees and the theoretical ones.\n\nI think a main weakness of the paper is that it seems to assume both in the algorithm's motivation, design, and in the experiments that the federated model update computed by a client with a single data point would be a rescaling of the gradient of the model sent to the client on that data point. However, this isn't necessarily true if e.g. clients do multiple epochs of training over their local data (and if the canary is acting honestly given the data, which seems to be an important assumption in the paper). The ability for clients to do multiple passes over their dataset is a key feature of federated learning (e.g. if one looks at McMahan et al. 2016, Figure 3, one sees that multiple passes actually can improve the training process if communication costs are the primary bottleneck). Since the loss CANIFE optimizes to choose its canary and the discussion in the paper (including e.g. the experiments) are based on this assumption, while CANIFE could still be applied to auditing central training methods or simple federated algorithms like FedSGD or FedAvg restricted to clients doing 1 epoch (for which this assumption does hold), it's unclear to me that CANIFE is an appropriate method for auditing even FedAvg in general. One thing that confuses this point for me is that in the experiments section, \"epoch\" seems to refer to a pass over the clients rather than a pass by the clients over their dataset. So it's unclear if clients are using one or multiple epochs in the experiment. One could perhaps show that the update generated by multiple client epochs has a large dot product with the initial gradient, in which case perhaps having the initial gradient be orthogonal is a good heuristic choice.\n\nAlso, it seems the RDP accountant was used for the theoretical epsilons, which as the authors mention is not tight if subsampling is used (which I believe is the case in the experiments), so the gap between the empirical and theoretical epsilons could be tightened by using a tighter accounting scheme to lower the theoretical epsilon (One might argue that clients in practice do not appear uniformly at random, but in that case one should prefer an algorithm like DP-FTRL anyway, I think). I think it is unlikely that e.g., entire gap would be closed, but I think part of the message of the paper (empirical epsilons are much smaller than theoretical epsilons, even for an adversary that is stronger than is practical) is weakened since this message is vacuously true if the theoretical epsilon is far from tight.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "aja6anHBT7k", "reviewer": "Reviewer_hrNj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated by the insight that DP budgets are conservative (i.e. a given budget overestimates the privacy/membership inference information that can be extracted realistically), this work designs a canary-based method to produce a more realistic estimate of the privacy budget in FL settings. The threat model is\n- safe but cooperative server\n- one rogue client\n- access to public model weights once trained\n- access to a public dataset (\"design pool\")\n- access to public noisy model updates\n\nThe proposed method injects (or not) a canary data point in a given training round, computes scores using public noisy model updates, then aggregates per-round privacy estimates to global privacy estimates. The canary is obtained by a computationally cheap local SGD relying on the derivative of the loss wrt the canary value, holding model parameters fixed. ", "summary_of_the_review": "The paper is very good and has very few flaws, if any. It addresses a problem of practical importance, comes up with a relatively simple, general solution which solves the problem in many cases. Therefore its impact can be expected to be strong. ", "strength_and_weaknesses": "#Strengths\n\nThe privacy estimate provided by this method is practically useful as complement to a theoretical guarantee. The method is computationally accessible. The threat model is applicable in practice (though it looks to me as though client samples are wasted during training rounds ran for the purpose of the method). \n\nThe paper is remarkably well written, well structured, self-contained but with useful extra detail in the appendices. Experimental details, including other software libraries, datasets, other methods are detailed. Plots are clear, legible, to the point. Notation, nomenclature, external references are all unambiguous and helpful. \n\n#Weaknesses\n\nThe proposed method seems to be limited intrinsically, probably like other methods with the same purpose: it requires a very specific protocol to be applied, and will return the specific privacy measurement corresponding to its threat model. However, complete practical privacy measurements presumably require multiple such methods, under varied threat models eg Nasr+ 2021, to form a complete picture as a counterpoint to the theoretical DP guarantee. \n\nThe threat model might be demanding, especially the availability of public data (design pool) and cooperative clients and servers. \n\nExperiments are limited to classification tasks.\n\nLooking hard for weaknesses, one could wish for the paper to be more overt about the method's shortcomings and limitations. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "K8zlXQMtig", "reviewer": "Reviewer_rAJ2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an empirical evaluation of differential privacy parameter epsilon in federated learning setting. It does so by inserting canaries and seeing if it can determine the presence of these canaries in the model after each round update. It then uses this to estimate per round epsilon.", "summary_of_the_review": "The paper considers an important problem but it is unclear how it is different from previous work on estimating epsilon in centralised setting and where those methods would fail. ", "strength_and_weaknesses": "Strengths:\n- it seems to be one of the first works to look into empirical epsilon estimate in FL\n- the method is intuitive and based on examples that would cause worst case clipping\n\nWeaknesses:\n- DP guarantees are not explained well (i.e., is epsilon per user or overall epsilon, how doe shuffling come into play)\n- it is not clear how the work is different from that of empirical estimation of epsilon in central setting (i.e., previous work in this space)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1R4Drry_Oh", "reviewer": "Reviewer_K314", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed CANIFE for measuring the empirical privacy loss under looser assumptions about the adversary’s capability compared to differential privacy (DP). CANIFE generates worst-case data samples in federated learning (FL) such that the gradients of such samples are orthogonal to most of the model updates from other participants, and from which the adversary can tell whether such outliers participated in training or not in each round. The authors evaluated the effectiveness of CANIFE on benchmark image and text datasets and showed that DP overestimated the privacy by 2 to 7 times. \n", "summary_of_the_review": "Overall I think it is a good paper despite the weakness mentioned above. \n", "strength_and_weaknesses": "Strength \n\n1. The paper is well-motivated as currently training large models with DP has non-negligible impact on utility and it is less understood whether such strong guarantees are necessary under more realistic settings. This paper could be useful for developing more practical private learning algorithms.\n2. The paper is well-written and easy to follow, and the proposed algorithms are straightforward and simple to implement.\nThe evaluation is done on multiple image and text datasets demonstrating the effectiveness of the proposed approach. \n\nWeakness\n\n1. Although the author mentioned a few existing empirical privacy loss estimation approaches, there is no empirical comparison between this work and prior works. For example, would any of the settings in Milad et al [1] be comparable with CANIFE, in terms of privacy loss estimation and attack efficiency? \n2. The main goal of the paper is to monitor empirical privacy during FL training while one limitation is that one has to have a good Design Pool (i.e. held-out data in the experiments) to simulate the other clients for a more accurate measurement. If the design pool is very different from the real distribution, then the measurement could be overly optimistic. However, in many real FL applications, there is no held-out data available since these are also sensitive data on device, and having an accurate design pool might be hard to achieve. The paper would be stronger if the authors also provided methods to generate a design pool when no held-out data are available yet the privacy measurement is tight. Relatedly, the distribution assumption about the design pool is also not rigorously defined.\n\nReferences:\n\n[1] Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papemoti, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 866–882. IEEE, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster