DPVIm: Differentially Private Variational Inference Improved
Authors: Joonas Jälkö
Lukas Prediger
Antti Honkela
Samuel Kaski
Venue: Accepted by TMLR ()

Abstract: Differentially private (DP) release of multidimensional statistics typically considers an aggregate sensitivity, e.g. the vector norm of a high-dimensional vector. However, different dimensions of that vector might have widely different magnitudes and therefore DP perturbation disproportionately affects the signal across dimensions. We observe this problem in the gradient release of the DP-SGD algorithm when using it for variational inference (VI), where it manifests in poor convergence as well as high variance in outputs for certain variational parameters, and make the following contributions: (i) We mathematically isolate the cause for the difference in magnitudes between gradient parts corresponding to different variational parameters. Using this as prior knowledge we establish a link between the gradients of the variational parameters, and propose an efficient while simple fix for the problem to obtain a less noisy gradient estimator, which we call \emph{aligned} gradients. This approach allows us to obtain the updates for the covariance parameter of a Gaussian posterior approximation without a privacy cost. We compare this to alternative approaches for scaling the gradients using analytically derived preconditioning, e.g. natural gradients. (ii) We suggest using iterate averaging over the DP parameter traces recovered during the training, to reduce the DP-induced noise in parameter estimates at no additional cost in privacy. Finally, (iii) to accurately capture the additional uncertainty DP introduces to the model parameters, we infer the DP-induced noise from the parameter traces and include that in the learned posteriors to make them \emph{noise aware}. We demonstrate the efficacy of our proposed improvements through various experiments on real data.

Published in Transactions on Machine Learning Research (09/2023)
DPVIm: Differentially Private Variational Inference Improved
Joonas Jälkö"
joonassjackoShccssikk.fp
Deportment of Computer Science, University of Helsinki
Lukas Prediger
lukas. mm.reedggerBoatto.f
Department of Computer Science, Aalto University
Antti Honkela
antti.honkela@helsiikk
Deportment of Computer Science, University of Helsinki
Samuel Kaski
sammel kaskkullkkkkiiinltto
Department of Computer Science, Aalto University
Deportment of Computer Science, University of Monchester
Reviewed on OpenReview: https: // opt.revieeerevveeeeommmm net/ ftt/m? id= GUAM6XX1uv
Abstract
Differentially private (DP) release of multidimensional statistics typically considers an ag- gregate sensitivity, e.g. the vector norm a higl-dimensional vector. However, different dimensions of that vector might have widely different magnitudes and therefore DP perturba- tion disproportionately affects the signal across dimensions. We observe this problem in the gradient release of the DP-SGD algorithm when using it for variational inference (VT), where it manifests in poor convergence as well as high variance in outputs for certain variational parameters, and make the following contributions: (i) We mathematically isolate the cause for the difference in magnitudes between gradient parts corresponding to different variational parameters. Using this as prior knowledge we establish a link between the gradients of the variational parameters, and propose an efficient while simple for the problem to obtain less noisy gradient estimator, which we call aligned gradients. This approach allows us to obtain the updates for the covariance parameter of Gaussian posterior approximation without a privacy cost. We compare this to alternative approaches for scaling the gradients using analytically derived preconditioning, e.g. natural gradients. (ii) We suggest using iterate averaging over the parameter iterates recovered during the training, to reduce the DP-induced noise in parameter estimates at no additional cost privacy. Finally, (iii) to accurately capture the additional uncertainty DP introduces to the model parameters, we infer the DP-induced noise from the parameter iterates and include that in the learned poste- riors to make them noise aware. We demonstrate the efficacy of our proposed improvements through various experiments real data.
1 Introduction
Differential privacy (DP) (Dwork al.) 2006) protects privacy of data subjects by limiting how much about the input data can be learned from the oulput of algorithm. Additive noise mechanisms achieve DP by adding noise calibrated to the maximum change in function output due to single individual, known sensitivity. When releasing high-dimensional data through such mechanisms, different variables may have widely different sensitivities. However, this issue of varying sensitivities is often neglected or overlooked.
Work done while Aalto University

---

Published in Transactions on Machine Learning Research (09/2023)
Instead, the sensitivity of the release is computed an aggregate over all the dimensions, which we call total sensitivily, in contrast to mariable-specific sensilivily. As the DP noise is subsequently scaled with this total sensitivity, it affects dimensions with lower sensitivities more. A prominent example where this occurs is the gradient release in DP stochastic gradient descent (DP-SGD) (Song et al.) 2013 Bassily et al. 2014 Abadi jet al. 2016). where it can affect the convergence rale of the corresponding parameters. Furthermore, the final parameters releasod from DP-SGD are noisy estimators of the optimal parameters and the resulting error is usually treated as an unavoidable trade-off of providing privacy (Abadi al. 2016). As a result, final parameter estimates with small crriable-speciffe sensitivity may exhibit larger errors due to DP randomness compared to other dimensions. The combination of these two issues means that parameters with relatively small sensitivity are at a double disadvantage.
We discover these issues in the perturbed gradients used in the DP-SGD based DP variational inference (DPVI) algorithm (Jälkö et al., 2017), which widely applicable state-of-the-art method for vrivac-preserviin (approximate) Bayesian inference. We find that gradient magnitudes for different paramcters in DPVI often differ significantly, resulting in severe errors in capturing the posterior. This results e.g. in poor predictive uncertainty estimation, making the predictions of the learned model less accurate.
We mathematically isolate the cause for these problems in DPVI and propose and evaluate two ways of alleviating the problem of gradient scales in DPVT: one scales gradients with a preconditioning matrix before applying the DP mechanism, the other is based on insights into the mathematical structure of the gradients, which reveals that their components are mathematically linked and can be derived from each other in a post-processing step.
Additionally, we theorctically and experimentally evaluate the method of iterate averaging way to further improve the parameter estimate well as approximate the additional variance induced by DP perturbations to DPVI to make the posterior approximation noise aware at no additional cost in privacy.
1.1 Related work
In the context of DP-SGD, the following previous works acknowledge the different sensitivities of different. parts of the full gradient: McMahan et al. (2018) suggested clipping the gradients of a neural network scparately for each layer to avoid the happing-inducces bias (Chen et al. 2020). Other lines of work (Andrew
et al. 2021 Wang et al. 2022) suggest adaptive clipping, where the total sensitivity is re-evaluated throughout. the optimisation process to avoid adding excessive amounts of noise to the gradients. However, since in all these the perturbation is still scaled with the total sensitivity aggregated over the dimensions, this approach does not improve the disparate effect that the Gaussian noise will have on the dimensions with smaller gradients, so we see these approaches as orthogonal to our work. Besides the aforementioned works that study the tuning of clipping threshold, there are some recent works that study the use of clipping in terms of obtaining optimal rates in DP convex optimization Kamath et al. (0022 : Lowy & Razaviyayn 2023).
For noise aware DP Bayesian inference, the most related work is by Bernstein and Sheldon (Bernstein & Sheldon 2018 2019) and Kulkarni et al. (202I). These works include the DP perturbation mechanism into a probabilistic model using perturbed sufficient. statistics as the inputs. This allows capturing the DP-induced additional uncertainty in the posterior distribution of model parameters.
2
Preliminaries
2.1 Differential privacy
Definition 2.1 (Differential Privacy (Dwork et al. (006]). For e 2 0 and ò E [0, 1], a randomised mechanism M: D + R satisfies (c, 5)-differential privacy for any two data sets different in only one clement, D, E D, and for all outputs C im(M), the following constraint holds:
Pr(M(D) E S) < e Pr(M(D') S))+
(1)
Property 2.1 mechanism and f : R 2 any function that does not access the sensitive data. Then fo M is (e.
immunity (cf. (woor et al. 2014 )). Let M : D + R be a (c. 8CP
2

---

Published in Transactions on Machine Learning Research (09/2023)
2.2 Variational inference
Variational inference is commonly applied technique in probabilistic inference, where the aim is to learn an approximation for a (typically intractable) posterior distribution of the parameters of a probabilistic model (Jordan ct al. 1999). The goal is to minimise the Kullback-Leibler (KL) divergence of this approximation to the true posterior. However, computing the KL divergence directly is intractable as well, so instead we maximise a quantity called evidence lower bound (ELBO) over the parameters of the variational approximation. For probabilistic model p(D, 0), where D denotes the observed variables and the model parameters, and for variational approximation (()) of the posterior, the ELBO L(g) is derived follows:
(()
kI(g(0) D)) = Ey)
log
(2)
P(D.0)
logp(D) By( log P(D,0) 0)
(3)
(())
:=C(g)
Now, the KL divergence is positive we have
log 2 L(g),
(4)
hence the quantity L(g) is called the evidence lower bound. We can easily see that if the KL divergence between the and the posterior is O, the ELBO matches the evidence. Since the evidence is independent of the variational parameters, minimizing the KL divergence w.r.t the variational parameters equivalent to maximizing the ELBO.
In the remainder of this paper the following equivalent formulation of the ELBO:
((g) Ey( [logp(D, + (1),,
(5)
where H(q) denotes the (differential) entropy of
In the following we first restrict ourselves to the commonly used mean-field variational inference, i.e., using a Gaussian with diagonal covariance as the posterior approximation. We will later generalise this to a full-rank coveriance approximation. For d-dimensional data the diagonal approximation is parametrised by the means m, Rd and the dimension-wise standard deviations og IRd. We further reparametrise the model with 8, (a,)),,, where T : RR - R2, monotonic, in order facilitate optimisation in an unconstrained domain. Both T and T-1 are applied element wise for each of the parameters. Common choices for T are the exponential function (s)) exp(s) and the softplus function T(s) = log(exp(s) + 1) (used e.g. in the Pyro probabilistic programming package (Bingham et al. 2019 We use & = (m,, 8,) to refer to the complete set. of variational parameters.
A draw from this posterior distribution can then be written as (Kingma Welling 2014
0 = 0 (7; mag sy) my + T(s)).
(6)
where 77 ~ N(O, n E Rd and Ia is a d-dimensional identity matrix. Kucukelbir et al. (017) use this reparametrisation trick together with single-sample MC integration give the ELBO a differentiable form with gradients:
Im V [(g Vm, logr(D, O(1 may 8q)) 9, :- f(g) = Va, logp(D, 0 mq: m V H()::
(7)
(8)
where n N(0, 7). Throughout this work assume that the likelihood factorises as: pl p(D (0) = T p(æ
0). Using N to denote the size of D, we can now further decompose the gradients in 7 and (5) as
9m
Σ
V m. logp(æ 0( m.: sy)) + N
m, log logp((((n: my: 8q))
(9)
BED
9,
Σ
V., log 1 (11 my 8q)) )} + N (V logg(p((); mZy 8q)) + V., H(g))
(10)
RED
3

---

Published in Transactions on Machine Learning Research (09/2023)
We denote the per-example gradient components (i.e., those for each individual x)) that appear in the above sums with Im, and s respectively.
A common approach to performing variational inference in practice is to initialise Bn to small values, which allows the algorithm to move my quickly close to their optimal values due to large error in the KL term of the ELBO induced by the narrow approximation.
Assumptions for the probabilistic model and variational posterior For the rest of the paper, we make the following assumptions:
Exchangeability: We assumee that there is no ordering in observing the elements of data set D: p(D (0) = IILED p(x ()
Gaussian posterior approximation: (0) = Na(m.. Eg): where Na denotes the pdf of d-
dimensional Gaussian. When working with isotropic Gaussian posterior approximation (i.e. diagonal covariance), we denote the dimension-wise standard deviations with dg.
Optimising the ogi For the isotropic Gaussian, we use a mapping function T R + R+ to optimize the variational standard deviations. We apply this function element-wist the parameter vector 8y. Commonly used examples for this are the softplus function T(s) log(1 + exp(s)) as well as the exponential function T(s) cxp(s).
3
Differentially private variational inference
The first algorithm for differentially private variational inference for non-conjugate models (Jälkö et al. 2017 optimises the ELBO using gradients 7 and (8) with differentially privale stochastic gradient descent badi 2016) to provide privacy. This involves concatenating each of the per-example gradients to obtain Ir 9 "m..."""."""" y² clipping In so that it has f2 norm no larger than threshold C' to limit the sensitivity, and finally adding Gaussian noise to the sum of these clipped per-example gradients to obtain 9, which is used for the parameter update. We refer this algorithm in the following as wonilla DPVT.
let al.
This formulation induces a problem which, while seemingly minor at first glance, severely affects accuracy of solutions. We next isolate this problem and then propose solution through detailed analysis the gradients of the variational parameters.
3.1 Disparate perturbation of variational parameter gradients
While the clipping of the gradients allows us to bound the global sensitivity of the gradient vector, it completely ignores any differences in gradient magnitudes across the dimensions. As DPVI (and more generally DP-SGD) proceeds to add Gaussian noise with standard deviation proportional to the clipping threshold to all of the dimensions, the signal-to-noise-ratio can vary greatly across the parameter dimensions. Parameter dimensions that experience low nooise ratio will converge much slower than others (ef. Domke 2019 and references therein) Next, we will show that such a magnitude difference arises between the gradients of variational parameters my and Sq.
Note that the gradient of Equation 6 w.r.t. my is Vm, ((: my mg.8y) 1, which leads to the following proposition (a more detailed derivation can be found in Appendix Bh:
Proposition 3.1. Assume y to be diagonal Gaussian, then the gradient 9, in Equation g
becomes
9, = (s.)g, + +V +V, (g),
(11)
where T" denotes the derivative of T.
As the entropy term independent of the data, our update equation for 8, depends on the data only through gm In order to show that this term gets affected by the noise more than Im itself, it suffices to inspect
We also provide high-level argument why this the in Appendix
4

---

Published in Transactions on Machine Learning Research (09/2023)
the magnitudes of 77 and (sy). We have Pr[In; <2] > 97.5% and (s,) S 0g for common choices of T
discussed above (a proof for softplus and exp can be found in Appendix C For each dimension of the data dependent part of it then follows that
ny ( Imm.l S an 7 < 20 q. Imm with probability > 97.5%.
(12)
Therefore, is easy to see that as og becomes small, the data-dependent part of the gradient 9, becomes small compared to Im in the overwhelming majority of cases. The entropy part of 9, is typically small as well, especially when we are far from the posterior mode, i.c., carly in training. As a result, 9, as a whole becomes small in comparison to 9m Note that this especially problematic combined with the practice of initialising 8q to small values to speed up the convergence of myy discussed in Sec. 2.2
3.1.1 Addressing the differing gradient scales
Above we have identified a magnitude difference between the gradient components, which leads to variational standard devialion parameters being disproportionately affected by DP noise. Next we use this structural knowledge to propose a method for scaling the gradients to more closely matching magnitudes, after discussing two alternatives based on standard techniques.
Natural Gradients As first solution, we consider natural gradients (Amari 1998). This is common approach for improving convergence for VI (cf. e.g. Honkela et al. 2010 Khan & Nielsen 2018 Salimbeni
ct al. 2018), which relies on scaling the gradients using the information geometry of the optimisation problem.
The natural gradients nna are computed using the inverse of the Fisher information malrix I as
I= Eoja,.m, [Vo log ((O) log {())
(13)
nnat= = =I-
(14)
For our setting this leads to
nat = T(8g)) 22
m. f(g)
(15)
ghat 1 (sg
"ngaat + T(sg)² Ve, H(1)
(16)
T'(ss)
We observe that in the natural gradients the scaling by (s,) in the gradients of 8g is reversed, meaning that for small (s,, the gradients of 8, will tend to dominate over those of my. Therefore we expect natural gradients to result in a different instance of the problem of disproportionate DP noise instead of resolving it.
Preconditioning of Gradients The simplest way to fix the disproportionate DP noise is preconditioning of the gradients to undo the downscaling the dala-dependent part in Eq. 11), by multiplying with (T') obtain
9, = 79 + V., H(g) T'(sg)
precon
(17)
T'(8g)
We can see that the data dependent part of grecoon (the first term) is of the same magnitude as Im. and thus the noise alfects the gradient components equally 2 Note that while this approach addresses the issue of different magnitude in the gradients, does so at the cost of increasing the overall f2-norm of the full gradient (by increasing that of 9, while keeping 9m, fixed). This in turn requires a higher clipping threshold in order avoid additional bias due to clipping, which increases DP noise variance.
Note that this scaling also affects the data-Independent entropy term in the gradient for While the scaling term (s,))) -2 does increase the entropy part small By, the data-dependent term is still typically much larger and will dominate the gradient.
5

---

Published in Transactions on Machine Learning Research (09/2023)
Aligned Gradients We will now discuss new alternative method for resolving the disproportionate DP noise problem that addresses the gradient magnitude problem while avoiding the issues of the preconditioning approach. Equation [11] shows that we can write 9, in terms of 9m and an additional entropy term. Since neither the scaling factor (sg) nor the entropy gradient Ve, (() depend on the data D, it suffices to release the gradients Im under DP as Im from which we obtain the 9, via Eq. (11). As this simply post-processing. does not incur additional DP cost. Because 9, now computed directly as a transformation of 9m. the noise term in both gradients is aligned in proportion to the gradient signals. We refer to this approach aligned DPVI for the rest of the paper. The procedure for computing the aligned DPVI gradients is summarized in Algorithm 1
Algorithm The aligned gradient procedure (single step)
0 my + r),) where 1 10,
Draw sample from the variational posterior Compute the per-example gradients for my
9m. t m,, km,2i ((g) for x E D
Ta f min(1, (/1l9m.-ll) for E D
Compute the clipping multiplier Get DP relcase for I
4: Im t- ExED m DDDC,, where ab ~ I0,
g galsgned (8,)) mm + Vo, Hq))
Get DP aligned 9, via post-processing
Corrolary 3.1. Let o(;; T) be any privacy accounting oracle. Let >> 0, aDp > 0. Then aligned DPVI consisting of T iterations of Alg. 1 is (c.
Proof. The 9m relcased in step 4 of Algorithm | satisfies the privacy guarantce provided by the Gaussian mechanism. As the 77, (8,, and the entropy contribution Vx, (q) are independent of the data, the g, galigned satisfies the same privacy guarantee through post post-processing immunity (Property 2.Ip. Hence the privacy guarantocs of the entire algorithm follow from the composed privacy cost of the released gradicnts for the variational mcans (g( 1t=1
The following thcorem (proved in Appendix D guarantces that the variance in the gradients of sy is reducod in aligned DPVI:
Theorem 3.1. Assume that Ceonilla Calligned If we obtain og through transformation T such that (s) < 1, then for any fixed batch,
Var Igoligned
<
Var [granilla]
(18)
where 17 is the random variable of the MC approximation to the ELBO and sp that of the DP perturbation.
Note that the softplus transformation used in our experiments satisfics the assumption (*) s 1.
Aligned Natural Gradients Finally we also consider a combination of natural gradients and aligning, to
enable the benefits of natural gradients for convergence while simultaneously removing the need to consider the gradient of Sy for DP clipping and perturbation. The full procedure is given in Algorithm 2 We use Im, I, to refer to the blocks of I corresponding to the gradient components.
Algorithm The aligned natural gradient procedure (single step)
0 my + ()g) where n NO
Draw sample from the variational posterior b Compute per-example natural gradients for m. Compute the clipping multiplier D Get DP release for g > Get DP aligned g. via post- post-processing
nat I m,
(n, L(q) for D
D 4: gnat &- -EOED I m. + DDb where 1b ~ () geat. aliggnd I, TT (sq Imm nat + Vo,H(()).
Nie min(1, ///l)= for
3.1.2 Extending to full-rank covariance matrices
So far we have only considered diagonal Gaussian as the variational posterior. Due to the low dimensionality of the variational parameters, this approach is computationally effective and often applied in practice, but it

---

Published in Transactions on Machine Learning Research (09/2023)
has limitations: It cannot capture correlations among different model parameters and, more importantly, it will inderestimate the marginal variances of the parameters when the true covariance structure is non-diagonal. For those reasons, a full-rank covariance approximation would be favored to correctly capture the uncertainty of the parameters.
However, learning the full-rank covariance approximation results quadratic (in the number of dimensions d) expansion of the number of learnable parameters. This not only increases computational costs but also implies less accurate learning of the parameters under DP, the available privacy budget has to be spread over more parameters. Fortunately, the aligning procedure can be extended to full-rank Gaussian approximations well, which allows us to alleviate the issuc of increased sensitivity. The proof is very similar to the diagonal case. Instead of the parameters 8n corresponding to marginal standard deviations, we now consider a parameter vector Og E I Ad+11 and a transformation function T : d64+1 + Rdxd such that T(a;) corresponds to the Cholesky factor of the posterior covariance. That is, T must guarantee that T(ag) is a lower triangular with positive entries along its diagonal, which will require similar transformations as in the purcly diagonal covariance case discussed previously. Now, the reparamctrisation step in (6) becomes
0 == my + T(a)).
(19)
and the gradient w.rt ag can be written as
9. = (T + H()),
(20)
where Ja denotes the Jacobian of T w.r.t ag. Therefore, the gradient 9, again be written a data- independent transformation of Im. Thus under the post-processing immunity of DP, we can get the DP gradients for ag from DP versions of Im without suffering the quadratic increasc of the size of the input to underlying the Gaussian mechanism present in vanilla DPVI.
3.2 Leveraging DP parameter iterates to reduce error and for uncertainty estimation
As other applications of DP-SGD, vanilla DPVT does not take the uncertainty that the DP mechanism introduces into account. Instead, after finite number of iterations, the values found in the last iteration are usually treated as the true variational parameters. We argue that treating them this way can lcad to severe errors in accuracy because these values are merely a noisy estimate of the optimal values. Annealing the learning rate to reduce fluctuations around the optimum does not help: Without knowledge of the convergence point the distance to an optimum can still be large due to the random walk prior to annealing. Instead in the following we suggest making use of some fraction of the DP parameter iterates output by the algorithm to 1) average out and 2) estimate the additional variance introduced by DP noise additions, making the learned model approximately noise-aware. While 1) is known technique known as iterate averaging (Polyak & Juditsky 1992), 2) has not been previously applied in this context to the best of our knowledge. Nole that privacy guarantees of DP-SGD algorithms extend to all parameter iterates, so this comes at no additional privacy cost. We first briefly review some theory about the random walk behavior around the optimum and then discuss the details of our proposed approach.
Mandt et al. (2017) investigated the random walk behaviour around the optimum for regular (non-DP) SGD arising from subsampling. They assume that near the optimum ex the loss function is well approximated by quadratic approxination L(E)= R 1 - - A(Ş - {"), and show that the stochastic process around the (OU) process
optimum can be characterised as an
dE(t) = - ((((() - -tt + VS
BdW(())
(21)
where W(t) is a Wiener process, a is the step size of the SGD (assumed constant), S is the size of the subsampled data and B the Cholesky decomposition of the covariance matrix of the noise due to the subsampling. Directly adapting this analysis, we suggest that under the same regularity assumptions, DP-SGD still is an OU process. The principle of the proof is straightforward: DP-SGD adds an additional Gaussian noise component, allowing us to add the (diagonal) covariance matrix of the DP noise to Z and obtain B
7

---

Published in Transactions on Machine Learning Research (09/2023)
such that
Z + So = BB"
(22)
The more detailed proof can be found in Appendix E This insight allows us to make the following two suggestions to improve the parameter estimates of DPVI.
Iterate averaging to reduce noise in parameter estimate In order to reduce noise in our learned variational parameters, we apply iterate averaging and average the parameter traces, ie., the sequence of parameter iterates during optimisation, over the last Turn-out iterates for which we assume the trace has converged. As the OU process in Equation (21) is symmetric around the optimum, the mean of the trace is an unbiased cstimator of g" Compared to using the final iterate of the chain (also an unbiased estimator of f"); the averaged trace reduces the variance of the estimator by up to a factor of Torr Iterate averaging has been previously been used to reduce the noise of DP-SGD for example by Bassily al. (2014) by scaling the learning rate w.r.t total number of iterations and by Bassily et al. (2019) and Lowy & Razaviyayn [2021) by taking average over all the iterates obtained during training. However, the key difference in our method is that we are actively estimating the point of convergence for the iterates, and only averaging after that. The convergence is required for estimating the DP induced noise from the OU process.
Estimating the increased variance due DP Finally, since our posterior approximation Gaussian and the stationary distribution of the OU is Gaussian as well, we can add the variance of the averaged traces to the variances of our posterior to absorb the remaining uncertainty due to the inference process, and recover noise-aware posterior approximation.
Now the remaining problem is to determine Tuurn-out, the length of the trace where the parameters have converged. For this we suggest a simple convergence check based on linear regression: For each of the traces, we fit linear regression models over different candidate Tourn-out- The regressor X Xinrrgg is set to interval [0, 1] split to burn-out points in an ascending order. The responses y are set to the corresponding parameter If the linear regression model has a sufficiently small slope coefficient, consider the trace as converged and pick the longest Toura-out for which this is the case.
values in the trace, e.g. y = {m, (2) JI-r -Trre
Thrna out'
4 Experiments
We experimentally test our methods for two different lasks using mean-field approximation with real data: learning a probabilistic generative model for private data sharing and learning a logistic regression model. We also experimentally explore aligned DPVI with full-rank Gaussian approximation using simulated data.
4.1
Implementation details
We implemented the different variants for DPVI introduced in Soc. 3.1.1 using the d3p package (Prediger 2022) for the NumPyro probabilistic programming framework (Phan et al. 2019 Bingham al., use the Fourier accountant method (Koskela 2021b. The hyperparameters used in our experiments are discussed in the Appendix F We use the
et al.
2019 To compute the privacy cost in our experiments,
let al. softplus function as our transformation T in all experiments. The code for reproducing the experiments can be found at httts:///githeei cot/:/Bay////eeeee /dpvim eewpii--exprimments
In order to asses learning over multiple parameters which converge to different values, and over repeated rums with different initial values, we define mean proportional absolute error (MPAE): Let g( e R be the parameter vector at iteration i and gx be the parameter vector at the optimum We measure the MPAF at iteration
D
MP
-
D
1
(23)
E
An MPAE value of 0 indicates perfect recovery of the optimum, value of 1 suggests that the parameters average did not move away from their initialisation.
Sinne the optimal value e typically unknown, we instead the results of classical non- non-DP variational inference in its place in practice.
8

---

Published in Transactions on Machine Learning Research (09/2023)
Evolution of error learned parameters
Accuracy trade-off between the variational parameters
g 1.0, after 1000 epochs
2.3 10
moan
vanilla
10a
2.2 10
aligned
MPAE in mean
2.1 10
precon
x
1.9x
109
vanilla
10-1
al aligned
MPKE
11-1
10-1
10
10
MPAE std.
200
1000
current epoch
(a)
(b)
Figure 1: UKB experiment: (a) Aligned DPVI makes the smallest error in learning the variational parameters across all different initial values for Tg. implying it is the most robust. (b) Aligned DPVI converges faster than the other variants while also having less deviation across the repeats (all initialised at
0g = 1). Both subfigures show averaged MPAE for vanilla, aligned and preconditioned DPVI with error bars in (b) indicating standard error over repeats.
4.2 Using DPVI to learn a generative model
Recently, Jälkö et al. (2021) suggested using DPVI to learn a probabilistic generative model for differentially private data sharing, Note that in this application it is especially crucial to learn the posterior variances well to faithfully reproduce the uncertainty in the original data in the synthetic data set.
A recent study by Niedzwiedz et al. (2020) on personal health data from the United Kingdom Biobank (UKB) (Sudlow et al.) (2155) studied how socio-economic factors affect an individual's risk of catching the SARS-CoV. virus. We aim to produce synthetic data, using DPVI to learn the generative model, from which we can draw similar discoveries.
Following Niedzwiedz et al. (2020), we consider a subset of UKB data which comprises of 58 261 individuals with d discrete (categorical) features. We split the features into set of explanatory variables and response variable indicating whether the individual was infected by SARS-CoV-2.
We place a mixture model for the explanatory variables X, and a Poisson regression model mapping the explanatory variables to the responses y, using 0x O, and π to designate the model parameters:
p(X 1x, TT) Σ T II Categorical(X; (%)) j=1
(24)
pΦy X, x,0,) = Poisson (ex((X0x)) XOy)).
(25)
In our experiments, set the number of mixture components K 16 which was chosen based internal tests. Priors for the model parameters are specified in Appendix G.1
Aligned DPVI is more robust to initialisation We first demonstrate that aligned DPVI improves robustness to initialisation over vanilla and preconditioned DPVI. To do so we fix privacy budget of = and the number of passes over the inlire data set, i.e., epochs, to 1000 and vary the initial value of 8, such that or is one 0.01, 0.032, 0.1, 0.316 or 1. We perform 10 repetitions with different random seeds over which we keep the initialisation s, fixed but initialisc m, randomly. We compute the MPAE over the paramcters of the Poisson regression part in the model, which corresponds directly to the downstream prediction task we are ultimately interested in.

---

Published in Transactions on Machine Learning Research (09/2023)
Error in parameters after different training lengths
12 orror of downstream caefficionts on synthotic data
Effect of iterate averaging on downstream cocfficients
==
1.8
Methed
MPAE in mean
6 10
vanilla
Last
4 10-1
aligned
1.6
Averaged
3
precon
1.4
10
1.2
MPAE in std
10-1
0.6
0.4
0.2
1.0
2.0 4.0 10.0
2000
4000
6000
8000
Total number of training epochs
Figure 3: UKB experiment: RMSE of parameters found in downstream analysis when doing iterate aver- aging with different Toun-ouut- Iterative averaging can reduce the error and significantly reduce variance of error compared to using only the last iterate. Initial og 1, 4000 epochs of aligned DPVI.
The aligned variant remains the most accurate method, even if we run the algorithm for longer. Initial 0g- =
Figure 2: UKB experiment:
Figure la shows the trade-off different variants of DPVT make between the MPAE in variational means (m,) and stds (8a) averaged over the 10 repetitions for the different initial values of 8g. We observe that the aligned variant able to achieve small errors my and 8y simultancously while the alternatives cannot. To see how the MPAFs for my and &2 behave individually w.r.t. the initial value for 8 refer to Appendix The natural gradient as well as the aligned natural gradient method performed slightly worse than the aligned method in this experiment and we report results for them in the appendix as well.
Longer runs do not help vanilla DPVI Figure 15 suggests that vanilla DPVI has not converged in terms of MPAE, in the allotted number of iterations for an initial 0g = 1. An obvious solution then seems to be to run the inference for longer. We now fix the initialisation of 0, to 1, which designates the least relative scaling of gradients at the beginning of training and thus a best-case scenario for vanilla DPVI. We vary the number of training epochs from 200 to 8000 while always keeping the privacy budget fixed at E = 1. Since longer runs require more accesses to the data, the DP perturbation scale increases with the number of iterations. As before, we repeat 10 inference runs for each parameter choice.
Figure shows the final MPAE over all 10 repetitions and all parameters in the Poisson regression part of the model for the different numbers of total epochs P The upper panel shows that with an increasing number of epochs, the difference in MPAE of variational means between vanilla and aligned DPVI vanishes. However, the lower panel shows clearly that even in a long training regime, vanilla DPVI still does not converge in variational variances and is consistently beaten by our aligned variant.
Iterate averaging increases robustness of downstream task Next, we test the iterate averaging of noisy parameters traces for the generative model. We use the linear regression technique discussed Sec. 3.1.2 individually for each parameter, to determine the length of the trace average. We then use synthetic data from the generative model to learn the Poisson regression model used by Niedzwiedz et al. (2020) and compare the regression coefficients against those obtained from the original data. Further details on the downstream analysis setup are given in Appendix I Figure 3 shows that the results from iterate averaged model are less noisy compared to just using the last iterate as the true parameters. However, the approach appears to be somewhat unstable: Changing the initial values of og to 0.1 causes the variance of error for & = to increase over the non-averaged case. This is likely due to the simple linear regression heuristic we used in this experiment not detecting convergence correctly in this case.
Note that it is showing the evolution of the error over single training 8000 epochs.
10

---

Published in Transactions on Machine Learning Research (09/2023)
4.2.1 Experiments the US Census data set
Evelution of in learned parameters
To ensure the results are not specific to only a sin- gle data sel, we next. applied DPVT on a large set of US Census 1990 data (from UCI Dua & Graff (2017)), in the same data sharing setting the UK Biobank experiment. We locused on individuals having military background, and using the similar mixture model/Poisson regression combination as with the UK Biobank experiment, we tried to pre- dict the poverty indicator of the individual given military service related and demographic features. After preprocessing the data comprised of 320 754 samples with 13 features. The hyperparameters for the different DPVI variants were identical to the ones used in UK Biobank experiment.
4x
Noil for menn
30
10a
20c
NPRE tor std
aligned
6x
10
3xx 10-1
200
400
600
800
1000
cUrTeTE
Figure 4: US Census experiment: Aligned DPVI method learns the variational standard deviations faster than the alternatives. The plot shows the average MPAE and the standard deviation across 10 indepen- dent repeats of the variational inference with og ini- tialized to 1.0 applied on the US Census data sharing experiment.
Figure shows again that the aligned variant learns the variational scales better in terms of the mean proportional absolute error (MPAE) of DPVI param- the case
eters to the non- private optimum, with UK Biobank data.
further demonstrate methods the publicly available 4.3 Logistic regression with the Adult data
As the UK Biobank data is access-restrccted Adult data set from the UCI machine learning repository (Dua & Grall, 2017), which contains 30 162 training records. We learn a logistic regression model, classifying whether the income feature of the data exceeds $50k based on all other features.
Aligned natural gradient outperforms the other variants We compare our private logistic regression coefficients the ones obtained using privacy-agnostic VI. We also test the aligned natural gradient and the natural gradient variants for this data. Figure 5a shows the f--nrr between variational parameters learned
12-error means
12 12errorr in stds
4
0.20
9s
4m
q. bas-baslinne
3
0.15
MSE
q, gasbaseline
0.10
0.05
2.5
5.0
7.5
10.0
2.5
5.0
7.5
10.0
E
vanilla
+ natural grad.
aligned
aligned NG
precon
(b)
Figure 5: Adult logistic regression experiment: (a) The aligned natural gradient method is closest to the non-private variational parameters. Error is compuled as mean fz-norm against non-private baseline over 20 repeats. Error bars show the standard error the mean. (b) The standard deviation inferred from the converged traces (determined by the linear regression method) close that computod over last iterates across different repeats. Lines show the average MSE, error bars show the standard deviation across repeats. The baselines show the mean squared norm of the standard deviation estimated from repeated runs, corresponding to the MSE for not estimating DP-induced noise.
11

---

Published in Transactions on Machine Learning Research (09/2023)
with and without DP. From this figure we see that the aligned natural gradient method clearly outperforms all the other variants in this setting. Additionally, we clearly see again that vanilla. DPVT learns the stds poorly, and also that the natural gradient variant reverses the problem compared to vanilla and struggles in learning the variational means as suggested in Scction 3.1.1
DP noise can be inferred from the (converged) traces We test how well we can recover the DP noise effect from the parameter traces. We limit the test to the coefficients that have converged according to the linear regression Lest described in Section 3.1.2 Based on our internal lests, we chose a slope of 0.05 as the threshold for convergence. We compare the standard deviation of the converged parameter trace to an estimate of the DP-induced noise estimated by the standard deviation of the last itcrates over 50 repeats in terms of mean squared error over different parameter sites.
Figure 5b shows that the noise std estimated from the converged traces is close to the noise std have across the last itcrates of multiple independent repeats.
4.4 Experiments with full-rank covariance
As final experiment, we investigate aligned gradients for full-rank Gaussian posterior approximation, We perform Bayesian linear regression over simulated data set where can control the number of feature dimensions as well as the strength of correlations. We control the latter by setting the rate of nonzero off-diagonal entries in the covariance matrix for simulated data points. Further details of the experimental setup can be found in Appendix[] []]
Figure 0 confirms that aligned gradients improve the average predietive log-likelihood of the posterior approximation over held-out test set. This is true even when the data is not strongly correlated (pancls in the right columm), as the large increase in parame- ters over which vanilla DPVT has to split the privacy budget negatively impacts the learning.
Predictive log- likellhood for full-rank approximation
density 0.2
density 0.8
average predictive lag-llkellhood
-1.5
-2.0
-2.5
5 Discussion
-3.0
In this paper we introduced the aligned gradient so- lution for the specific task of learning a Gaussian variational posterior. The technique should be appli- cable also in other tasks where gradients with respect different parameters depend on data through a common term. Detection of such cases could be even automated by inspecting how the data enters the computational graph of the task. This would be an interesting future direction.
vanilla
100
200
100
200
aligned
nandp
number dimensions
Figure 6: Full-rank covariance experiment: DPVI with aligned gradients achieves better predictive log- likelihood for full-rank approximation than vanilla DPVI. Higher density means more nonzero entries in data covariance. E == 50 repetitions.
A limitation of DP in general is that it guarantees indistingushabbiity among the individuals in the data set by aiming to preserve more common characteristics of the data. Therefore the utility of a DP algorithm might be worse for individuals from less common groups.
The somewhat unexpected performance of aligned natural gradients in the UKB example might be due to bad hyperparameter choices. While we performed some hyperparameter tuning with limited success, more comprehensive search would be needed to fully assess the performance of these methods.
The literature on MCMC holds many existing diagnostics for the converge of chains, such as the (split) R-hat estimator (Gelman & Rubin 1992 Vehtari al.) (202)), that could be used to test the convergence of the parameter traces as well. Tuming these methods to work well diagnosing the converge of the parameter trace would require extensive testing which we leave for future work.
12

---

Published in Transactions on Machine Learning Research (09/2023)
Acknowledgements
This work was supported by the Research Council of Finland (Flagship programme: Finnish Center for Artificial Intelligence, FCAI; and grants 325572, 325573), the Strategic Research Council (SRC) established within the Research Council of Finland (grant 336032), UKRI Turing AI World-Leading Researcher Fellowship, EP/ PP//00993//1 as well as the European Union (Project 101070617). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them. The authors also acknowledge the computational resources provided by the Aalto Science-IT project.
This research has been conducted using the UK Biobank Resource under Application Number 65101. This work used data provided by patients and collected by the NHS as part of their care and support (Copyright O 2021, NHS England. Re-used with the permission of the NHS England and UK Biobank. All rights reserved). This work used data assets made available by National Safe Haven as part of the Data and Connectivity National Core Study, led by Health Data Research UK in partnership with the Office for National Statistics and funded by UK Research and Innovation (grant MC PC 20058).
References
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Nya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference Computer and Communications Security, CCS '16, pp. 308-318, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450341394. doi: 00.1155/97767492998811 URL htttss///di..ooo
10.1145/2976749 2978318
Shun-ichi Amari. Natural gradient works efficiently in learning. 10(2):251-275 02 1998. ISSN 0899-7667. doi: 10.1122/089977933001177
Galen Andrew, Om Thakkar, Hugh Brendan McMahan, and Swaroop Ramaswamy. Differentially private lcarning with adaptive clipping. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (cds.), Advances in Neural Information Processing Systems, volume 34, 2021. URL https://ppenreview..ntt
forum?id= Rorum/id=RUQ1222RR
Raef Bassily, Adam D. Smith, and Abhradeep Thakurta. Private empirical risk minimization, revisited. CoRR, abs/1405.7086 2014. URL http://arxiv. httpaaawwwvv.oorr/////aaaae/e/ee org/abs/1405. 7085
Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett (ods.). Advances in Neural Information Processing Systems 32: Ansual Conference on Newral Information Processing Systems 2019, NeurIPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pp. 11279 11288, 2019. URL https::/pproceeddngs. http://roeedingg..uuuur......o heurips.cc/papppee
Garrett Bernstein and Daniel R. Sheldon. Differentially private Bayesian inference for exponential families. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, (NeurIPS) 2018., 2018.
Garrett Bernstein and Daniel R. Sheldon. Differentially private Bayesian linear regression. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS) 2019, 2019.
Eli Bingham, Jonathan Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah D. Goodman. Pyro: Deep universal probabilistic programming. J. Mach. Learn. Res., 20:28:1 28:6, 2019. URL httt::/jjnlr.oorr/pppees//////// html
Xiangyi Chen, Zhiwei Steven Wu, and Mingyi Hong. Understanding gradient clipping in private SGD: A geometric perspective. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Floring Balcan, and
13

---

Published in Transactions on Machine Learning Research (09/2023)
Hsuan- Tien Lin (eds.), Advances in Newral Information Processing Systems 33: Annual Conference on Neu- rol Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https :
Justin Domke. Provable gradient variance guarantees for black-box variational inference. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'AlchéBBuc, E. Fox, and R. Garnett (eds.), Advances in Newral
Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httpp://archive..tee www.iuieee.............o m1
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam D. Smith. Calibrating noise to scnsitivity in private data analysis. In Theory of Cryplogruphy, Third Theory of Cryplography Conference, TCC 2006, Proceedings, 2006.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and TrendsS in Theoretical Computer Science, 9(3 (13-4):211-407, 2014.
Andrew Gelman and Donald B. Rubin. Inference from Iterative Simulation Using Multiple Sequences. Statistical Science, 7(4):457 - 472, 1992. doi: 1111214///11770111115
Antti Honkela, Tapani Raiko, Mikael Kuusela, Matti Tornio, and Juha Karhunen. Approximate Riemannian conjugate gradient learning for fixed-form variational Bayes Journal of Machine Learning Research, 11 (106):3235-) 3268, 2010. URL http:// http:///ggggptt////////ooelaalla html
Joonas Jälkö, Onur Dikmen, and Antti Honkela. Differentially private variational inference for non-conjugate models. In Uncertainty in Artificial Intelligence 2017, Proceedings of the 33rd Conference (UAT), 2017.
Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to
variational methods for graphical models. Mach. Learn., 37(2):183-25) 233, 1999. doi: 10.1033/A11000065507777
URL https:///oi.coo org/ 1023/A: 1007665907178
Joonas Jälkö, Eemil Lagerspetz, Jari Haukka, Sasu Tarkoma, Antti Honkela, and Samuel Kaski. Privacy- preserving data sharing via probabilistic modeling. Patterns, 277)110027, 2021. ISSN 2666-3899. doi: https:
//dbi.orgg/0..011/// $2666389920000970
wwtp:///.................. patter. 2022.100077.... URL
Gautam Kamath, Xingtu Liu, and Huanyu Zhang. Improved rates for differentially private stochastic convex optimization with heavy-tailed data. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Ballimore, Maryland, USA, volune 162 of Proceedings of Machine Learning Research, pp- mlr. PMLR, 2022. URL htttt:::/pppceeeeeeeeee http:/iiinnnll..rrss//eee prress/v162/kamathh/e httm
Mohammad Emtiyaz Khan and Didrik Nielsen. Fast yet simple natural-gradient descent for variational inference in complex models. In 2018 International Symposium on Information Theory and Its Applications (ISITA), pp. 31 35, 2018. doi: 10.23919 0.23919/ISITA..
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL httt:httt///xi..../
abs/1412.6980
Diederik P. Kingma and Max Welling, Anto-encoding variational Bayes In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Leurning Representations, ICLR 2014, Bansf, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:://rxiv.. org/ abs/ wtb///11/ //////112..1111
Antti Koskela, Joonas Jälkö, Lukas Prediger, and Antti Honkela. Tight differential privacy for discrete-valued mechanisms and for the subsampled Gaussian mechanism using FFT. In Arindam Banerjee and Kenji Fukumizu (cds.), The 24th International Conference on Artificial Intelligence and Statisties, AISTATS 2021, April 13-15, 2021, Virtual Event, volume 130 of Proceedings of Machine Learning Research, pp. 3358 3366. PMLR, 2021. URL http:://proceeennggs.... mlr.
14

---

Published in Transactions on Machine Learning Research (09/2023)
Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, and David M. Blei. Automatic differentia- tion variational inference. J. Mach. Learn. Res., 18:14:1-14:45, 2017. URL http:://jnll.orr//ppprrs/////
v18/16-107 10m.
Tejas Kulkarni, Joonas Jälkö, Antti Koskela, Samuel Kaski, and Antti Honkela. Differentially private Bayesian inference for generalized linear models. Tn Proceedings of the 38th International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2021.
Andrew Lowy and Mcisam Razaviyayn. Locally differentially private federated learning: Efficient algorithms with tight risk bounds. CoRR, fas/22060099799 2021. URL http::///arrii.orgaa////////////
Andrew Lowy and Meisam Razaviyayn. Private stochastic optimization with large worst-case Lipschitz parameter: Optimal rates for (non-smooth) convex losses and extension to non-convex losses. In Shipra Agrawal and Francesco Orabona (eds.), International Conference Algorillmic Learning Theory, February 20-23, 2023, Singapore, volume 201 of Proceedings of Machine Learning Research, pp. 986 1054. PMLR, 2023. URL
mlr. wwwpss/eee/oooioeooe
html
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic gradient descent as approximate Bayesian inference. 18(1):4873 4907, 2017. ISSN 1532-4435.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In 6th International Conference on Learning Represenlalions, ICLR 2018, Vancoweer, Canada, Conference Track Proceedings, 2018.
Shubhankar Mohapatra, Sajin Sasy, Xi He, Gautam Kamath, and Om Thakkar. The role of adaptive optimizers for honest private hyperparameter selection. 11 2021.
Claire L. Niedzwiedz, Catherine A O'Donnell, Bhautesh Dinesh Jani, Evangelia Demou, Frederick K Ho, Carlos Celis-Morales Barbara I Nicholl, Frances S Mair, Paul Welsh, Naveed Sattar, et al. Ethnic and socioeconomic differences in SARS-CoV-2 infection: prospective cohort study using uk biobank. BMC medicine, (5(1):1-4, 2020.
Du Phan, Neeraj Pradhan, and Martin Jankowiak. Composable effects for flexible and accelerated probabilistic programming in NumPyro. arXie preprint urXiv: 10012.1100 11554, 2019.
B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. STAM Journal Control and Optimization, 30(4):838 855, 1992. doi: 10.1137/0330044
Lukas Prediger, Niki Loppi, Samuel Kaski, and Antti Honkela, d3p a python package for differentially-pin private probabilistic programming. Proceedings on Privacy Enhancing Technologies, 2022(2):407-422) 2022. doi: doi:10.2478 2 poppts- 2022-0052.
Jerome P Reiter and Trivellore E Raghunathan. The multiple adaptations of multiple imputation. Journal of the American Statisticnl Association, 102(480):1462 1471, 2007.
Donald B Rubin. Multiple imputation for nonresponse in surveys, volume 81. John Wiley & Sons, 2004.
Hugh Salimbeni, Stefanos Eleftheriadis, and James Hensman. Natural gradients in practice: Non-conjugate variational inference in Gaussian process models. In Amos Storkey and Fernando Perez- Cruz (eds.), Proceedings of the Twenty-Fire International Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pp. 689-697. PMLR, Apr 2018.
Shuang Song, Kamalika Chaudhuri, and Anand D. Sarwate. Stochastic gradient descent with differentially private updates. In JEEE Global Conference on Signal and Information Processing, GlobolSTP 2019, Austin, TX, USA, December 3-5, 2013, pp. 245 248. IEEE, 2013. doi: 10.1109/ https: http::://bbi.orr//.111////////// http://t0.1199//ooala11/ 2013 6736861
10011.10//GGGobalIPP /ot.aaalIP22013.67555000 URL
15

---

Published in Transactions on Machine Learning Research (09/2023)
Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, Bette Liu, Paul Matthews, Giok Ong, Jill Pell, Alan Silman, Alan Young, Tim Sprosen, Tim Peakman, and Rory Collins. Uk biobank: An open access resource for identifying the causes wide range of complex discases of middle and old age. PLOS Medicine, 12(3))1-10, 03 2015. doi: 10.1371/journall 1001d.0007779. 1001779.
Aki Vehtari, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul-Christian Bürkner. Rank- normalization, folding, and localization: An improved R for assessing convergence of MCMC (with discussion). Bayessan analysis, 16(2):667-718, 2021.
Ning Wang, Yang Xiao, Yimin Chen, Ning Zhang, Wenjing Lou, and Y. Thomas Hou. Squeezing more utility via adaptive clipping differentially private gradients in federated meta-learning. In Proceedings of the 38th Annual Computer Security Applications Conference, ACSAC '22, pp. 647-657, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450397509. doi: 10.1145/ 10.1145/3504622
URL
https: /tops://doi.. org.l.rg///// 10.g/ 11w./1111//1655.1111 65666
16

---

Published in Transactions on Machine Learning Research (09/2023)
Appendices
A Intuitive reasoning why larger noise would slow convergence
We consider only a single optimisation stop in a single dimension for simplicity. Assume we are at Q(0 and have noisy gradient
VC((()) + n. - N00,
(A.1)
for some perturbation scale o. We update the parameter as
g(++1) 0(0) ag
(A.2)
with learning rate a.
In order to get closer to the optimuum, we want (g(()) ) sign (V Assume wlog that V2((()) > 0,
then
Pr ()g sign sign(VL(((()))
Pr VC(((0) + > 0]
(A.3)
Pr 1/ -VE(0(0)]
(A.4)
Pr 1/ -VC(O(1)
(A.5)
D ((-V2((())
(A.6)
=
((V[(((())
(A.7)
1 1+ erf (
(A.8)
2
erf(-) is monotonically increasing function, so see from the above that the probability of progressing towards the optimum decreases with decreasing Vc((()) Le., for fixed gradient, larger variance o² will decrease the probability of progressing towards the optimum in each step.
B Proof of Proposition 3.1
We begin by restating Proposition 3.
Proposition B.1 (Proposition 3.
Assume q to be diagonal Gaussian, then the gradient 9, in Equation
becomes
9. (s,)), + V., (q),
where T" denotes the derivative of T.
Proof. We first recall the reparametrisation for the diagonal Gaussian approximation from Eq. 6 my m
m. T(s,) and observe that V, 1 and Va, rs,) (where abbreviate
((; my 8,) to simply 0). With this obtain the gradient of the ELBO with respect to my by applying the chain rule in Eq. (7) as:
9m = V m, C(q) = Vo log
(B.1)
Similarly applying the chain rule in Eq. and inserting the above yields
9. Va, log D, 0) - V., H())
(B.2)
Volog D, 0) 0 + V., H())
(B.3)
-oloog(DD 0 (T' + Ve, H(4)
(B.4)
= (s,)), + Vo,H(()
(B.5)
17

---

Published in Transactions on Machine Learning Research (09/2023)
C Proof of (s,) < T(s,) for softplus and exponential function
T(8.)) - softplus(s,)
Consider we transform 8n in positive real numbers using the softplus function:
T(sg) = log(1 + exp(s,)).
(C.1)
First, we make the following observation which connects the softplus to the sigmoid function
T(sg) log
1 + exp(s,))
1
log
1 + (xp(-58)
- log (1 o(8n)),
(C.2)
where a denotes the sigmoid function. We then get ('(s)) = =ss)). It is casy to see that log(z) S x and
hence
T(8)) = log (1 o(8))
21 - (1 ((s))) = o(sg) = (sg)))
(C.3)
We have therefore shown that T(sy) 2 T' (sgs) R.
T(s,)= exp(s,) For T(s,) exp(s,) the proof follows immediately from the fact that (s,) = exp(s,) ((sg)-
Proof of Theorem 3.1: Variance in aligned scale gradients smaller
We begin by restating the theorem:
Theorem D.1 (Theorem 3.1 Assume that Creanilla T such that (*(s) < 1, then for any fixed batch,
Caligned If we obtain og through transformation
Var [g gglligeed
<
Var
9
granilla]
where 7 E (O, 1) the random variable of the MC approximation to the ELBO in the reparametrisation approach and ab E (0,, that of the DP perturbation.
Proof. We want show, that the variance of the perturbed std parameter gradient for the aligned method is less or equal than the one for the vanilla approach in all the dimensions of the gradient. We start by setting the clipping threshold for the vanilla approach C and the one for the aligned C.. Denote the atigned for the aligned. For jth dimension of g, obt from the vanilla approach as gagt and similarly Is.j the vanilla approach, we have
① H(g) + + bb
Var [āng] = Var n, T
(84.))
C(g) +
(D.1)
img
OsqJ
i
-Vann
n, ('
C(y)
+ p. Varw [4b,]
(D.2)
Jm
Varn
n, ()T'(
L(g)
+ 2op.22
(D.3)
0m
18

---

Published in Transactions on Machine Learning Research (09/2023)
and for the aligned
Varn, n.ub
gos
galigned
Varn
n, T' 3m.g
[(g) + dsq.
(44) + 7, (8))))
(D.4)
Varn
Eygin n (8 gmqa
[(4) + 0gq.
H (q) + 11 T 1 OppC
(D.5)
a
0 (1 + n, (8 1 ,
+ En Var nj T
[(g) + ng
(D.6)
@m
Φ ((a)
Varn
11 TT imgi
+ En, T (8..) POp C
(D.7)
= Varn
n; TT 0mgi
((1)
+ ( 20
(D.8)
We can easily see that the two variances differ only in the noise term. Now, it is easy to see, that we set Ca = C' and have a transformation T s.t. (s)s) s 1, we have
aaligned] Varn, 9as
S Varm [Gn.] Vj.
(D.9)
Setting C Ca already shows that the aligned method's variance cannot exceed the vanilla method. However, consider now that C. is a clipping threshold that satisfies Pr(Ilg,.lI > Ca) < a for some a E [0, i.e. only an a fraction of the gradients get clipped the aligned approach. Now, since the vanilla mothod clips basod on the norm ||9|1 ll 2 + ||g, Ilgmll?, we need to set the C' at least as large the same probability of clipping. Furthermore, we have the equality ||]| |l only in the case when ||9.11 = 0 which would only happen if the std parameter has converged. Therefore, the C' should be chosen larger than C. to avoid clipping prior to the convergence of Sg. Note that the magnitude of the g, does not affect the aligned approach, and therefore the clipping bound C.- Now, as we need to choose C > C. and we have assumed (s) S 1, we have
C. to facilitate
Var
aligned 9es Varm [9n.] Vj.
(D.10)
Furthermore, from the above variance expressions, it is easy to see that the variance in the aligned approach gets significantly smaller than the vanilla one when (s) << 1. This would be the case for example if we initialize the variational posterior with small 8g, or similarly towards the end of the convergence for models with small posterior variance.
E Variance of DP from OU process on convergence
We follow closely the analysis performed by Mandt et al. (2017), Sec. 3.2., which makes the following assumptions for the loss function L around its optimum e":
1. Minc-batch gradients of the loss functions are well approximated by a zero-mean Gaussian distribution with covariance matrix 3 Z, where S denotes the size of mini-batch,
is locally well approximated by a quadratic function.
We make the additional assumption that the clipping threshold C is chosen such that no clipping occurs for gradients close to the optimum in order to avoid clipping-indun induced bias.
We begin by stating the SGD parameter update equation and the resulting update step AE(t) for update steps close to the optimum.
19

---

Published in Transactions on Machine Learning Research (09/2023)
1(++1) = E(t) Ve((E(t)) + By tDplib) ~~N(0, + ~~N(0,11)
VS
=E(t) (VeC(E())) + (v/ VS
B 16) ~''' ~V0.11
(E.1)
VsS" B (potpp)))) & AE(t) E(t 1) E(t) -VVe((((t)
(E.2)
We denote with B the triangular matrix resulting from Cholesky decomposition of Z = BB"" Since we are performing DP-SGD, we have an additional independent noise term with scale dp. However, as both sources of stochasticity are independent zero-mean Gaussians, we can easily reformulate using a single Gaussian source of noise with the total variance.
We now restate E.2 by the following stochastic differential equation:
dÉ(t) = -=Va - a( VS
B (oDpl)dW(()
(E.3)
From our second assumption, we know that
((E) = (5 - --- A(E -5),
(F.4)
where
88 L(E").
[og)]
Inserting
E.3 in
E.4
get
dE(t) = -aAE - dd + al
Vs 1 B Boo
(E.5)
with describes an Ornstein-Uhlenbeck. (OU) process with Gaussian stationary distribution
({E) x exp { - 1
- (5- 5
(E.6)
2
where E satisfies the Lyapunov equation
EA+ AE a ( oppT)
(E.7)
Even without determining A already make important discovery from this: Since A is fixed around the optimum, see that the noise covariance of our OU process scales linearly with respect to obp-
Note that the above analysis holds for a constant learning rate. We have used the Adam optimization method (Kingma & Ba 2015) in our experiments throughout the paper. While Adam does adapt the learning rate, recently Mohapatra et al. (2021) showed that the learning rate of Adam will converge to a static value, which means that the analysis above still holds as it only concerned with the learning rale at convergence.
F Hyperparameters
We Adam Kingmaa & Ba 2015) the optimiser for all the experiments with starting learning rate of 10-3, In all of our experiments, the 8 privacy parameter was set to 1/N where N denotes the size of the training data.
For the UKB experiment In the experiments, we used various different training lengths (depicted e.g. in Figure D. For all of our runs, we set the subsampling rate as 0.01. The clipping threshold was set C 2.0 for the aligned and vanilla, 4.0 for the preconditioned variant and to 0.1 for the natural gradient based variants.
20

---

Published in Transactions on Machine Learning Research (09/2023)
For the Adult experiment The training run for 4.000 epochs with subsampling ratio of 0.01, corresponding to total of 400000 gradient steps.
We chose the clipping thresholds for the gradient perturbation algorithm as the 97.5% upper quantile of the training data gradient norms at the non-private optima. This was done to avoid clipping-indine induced bias, thus making the models comparable to the non-private baseline. This lead to clipping thresholds C presented Table F.
Table F.1: Clipping thresholds for the Adult data logistic regression model
Variant
Aligned
3.0
Aligned Natural Grad.
0.1
Natural Grad.
0.1
Vanilla
3.0
Preconditioned
4.0
G Model priors
G.1 For the UKB and the US Census experiments
Recall the probabilistic model used in the experiments:
K
p(X 10 =
Σ n II
Categorical(X, 1
(G.1)
1=1
j=1
ply x, x, Poisson (exp(X0,))
(G.2)
The categorical probabilities (*k) for each of the categorical features X,, were given a uniform Dirichlet (1) prior. Similarly the mixture weights æ we assigned a uniform Dirichlet prior. The regression coefficients 0y were given a std. normal (0, prior.
G.2 For the Adult experiment
We use the following model and prior
y (Xw),
(G.3)
w ~ 1(,,
(G.4)
where a(-) denotes the logistic regression function, alx) 1/1+exp(=x).
H More results for robustness
Figure H.1 shows how the MPAE for the different DPVI variants behave for the different initial values for sn for both, variational mean (upper panel) and standard deviation (lower panel) after 1 000 epochs.
Figure H.2 shows the parameters traces for 8g initialised such that ag = 0.01 (left) and og 0.1 (right) with the same split in upper and lower panels, similar to Figure 15 for o = 1.0 in the main body of the paper.
We observe that with decreasing initial values for 8, (/o,) it becomes increasingly difficult for vanilla DPVI learn variational standard deviation but learning of means slightly improved. Preconditioned DPVI performs better overall in terms of standard deviation but learns means worse. Aligned DPVI consistently outperforms both competing variants.
21

---

Published in Transactions on Machine Learning Research (09/2023)
Error in parameters over initialization
= 1.0, after1000epochs
MPAE in mean
x.6 10
2.4x
202x10
2x10
x 10
10-1
1.4 10
MPAE std.
vanilla
10
aligned
precon
10-1 -1
10-2
10-1
100
initial values for 9s
Figure H.1: The aligned variant has consistently low error across different initial values of qa. Figure shows the error for ==1 for 1000 epochs of training.
Evolution of error in learned parameters
Evolution of error in learned pa parameters
1.0
xx10
2 10
MPAE for mean
MHAE for mean
10"
10a
6xx10
60-101
1x01
10
3x10
10-1
MPAE for std
MPALE for std
vanilla aligned
x
vanilla aligned
25
75
100
125
150
175
200
50
75
100
125
150
175
200
epoch
epoch
(a)
(b)
Figure H.2: Aligned DPVI consistently converges faster than the other methods for different initialisation of 8g. On left, the is initialised such that og = 0.01 and on right such that og = 0.1.
H.1 Natural gradients and the aligned natural gradients in the UKB experiment
Besides the vanilla and aligned variant, we also fitted the UKB model using the natural gradient and aligned natural gradient variants. From Figure H.3 we can see the trade-off natural gradients make; the means are learned worse than standard deviations, which what we expect based on the analysis of Section 3.1.1 Somewhat surprisingly, the aligned natural grad. variant performs slightly worse than the aligned variant this experiment. This might be due poor choice of hyperparameter, for example the learning rate for the Adam optimiser used in the experiments set to 10-3 for all the variants, while know that the natural grad. variants tend to have smaller gradients than the others although Adam should in theory be able to adapt to that.
To evaluate whether the difference between aligned and aligned natural grad. results due to clipping threshold being too large too small for the aligned natural grad. approach, thus perturbing the gradients excessively introducing excessive clipping bias, we repeat the experiment with higher and lower clipping thresholds. Figure H.4a shows that increasing the clipping threshold to C 0.1 harms the natural gradient.
22

---

Published in Transactions on Machine Learning Research (09/2023)
Evolution of error in learned parameters
Vanils
Aligned grad.
g = 1.0, naturalgradient clipping threshold: 0
Natural
Aligned natural
MPAE for mean
10a
MPAE for std
200
400
600
800
1000
current
The natural gradient based variants variant achieve comparable performance in terms of variational parameters for std but perform worse for variational means. This is strongly observable for the plain natural gradients while aligned natural gradients improve error in variational means in comparison. However, both are outperformed by (non-natural) aligned gradients this experiment. Lines show the mean MPAE over independent repcats as well as std. of mean as error bars. Clipping threshold for both natural gradient basod variants is set to C 0.02, oy was initialised to The natural gradient variant appears diverge from the true variational std. which might be caused by clipping- induced bias, while still struggling to learn the correct variational mcan.
Figure H.3:
based variants, most likely due to introducing too much privacy noise and thus preventing convergence of the variational mean parameters. Conversely, Figure H.4b shows that setting clipping threshold C = 0.01 natural gradient variants is too small, and the aligned natural gradients start to suffer from clipping-induced bias.
These observations highlight the importance of choosing the clipping threshold appropriately to the method, which is not a trivial (ask. For this experiment, this seems to have direct influence how well aligned natural gradients can reign in the inversal of relative scaling of components present in plain natural gradients. For future work, it would be interesting to perform more extensive hyperparameter tuning including for example the learning rate and number of iterations as the tunable hyperparameters, to see if the natural gradient methods make some non-trivial trade-offs in the learning that differ fundamentally from the aligned and vanilla methods.
Further details on downstream analysis for UKB data
In the UKB experiment, we use the learned variational posterior to sample a synthetic data set from the posterior predictive distribution (PPD) as suggested by Jālkö al. (202T). We test the method by comparing the synthetic data downstream analysis to the original data. As the downstream task, we fitted Poisson regression model that aims predict whether individual catches SARS-CoV-2 based on the predictors in the data. Note that this downstream perfectly overlaps with our generative model.
In order to properly reflect the uncertainty rising from the data generating process to the final results compuled from the synthetic data, we will employ so called Rubin's rules [Rubin] 2004). In this procedure, we first sample multiple synthetic data sets from the PPD and compute the downstream analysis on each of the sampled synthetic data. Next, the results are aggregated according to set of rules and we recover finally more robust estimator for our downstream analysis. Further discussion about the Rubin's rules can be found for example in (Reiter & Raghunathan 2007).
23

---

Published in Transactions on Machine Learning Research (09/2023)
Evolution of error in learned parameters naturalgradient clipping threshold:
Evolution of ererr learned parameters dayne
1.0, gradient clipping threshold:
rature
300
MIKE
10"
for
NPWE
10-4
200
400 600
800
1000
200
400
600
800
1000
current epoch
current spoch
(a) The natural gradient based variants struggle converge in the UKB experiment with clipping thresh- old set to 0.1. Figure shows the evolution MPAE for both of the variational parameters over 1000 epochs.
(b) Both natural gradient variants start diverge if clipping is set too low C = 0.01.
Figure H.4: Tests with smaller clipping threshold for the natural gradient variants. The clipping threshold for the vanilla and aligned is still set to 2.0.
In our experiments, we sampled 100 data sets from the PPD learned using the aligned variant, and applied the Rubin's rules compute a mean and std. estimate for the Poisson regression coefficients. Finally, the obtained means were compared to the Poisson regression coefficients learned using the original data.
J Experimental setup for full-rank Gaussian approximation
In this experiment create simulated data where control the amount of correlations between data dimensions the ratio p of non-zero off-diagonal entries in the correlation matrix. To generate data with d dimensions and correlation density P,
1. generate correlation matrix C using Algorithm J.1 with inputs d, p, a 8, 10,
sample a diagonal matrix D of marginal variances, where (D)u expp(N(00000))
3. obtain the covariance matrix E = DCD
sample 10000 data points æ,, NVO
sample random regression weight vector w (0,
6. sample y ~ ((Xw, o with σy 1.
We perform the above for all combinations of d 100, 200 and p 0.2, 0.8. We then vanilla DPVT and DPVI with aligned gradients to learn the full-rank Gaussian posterior approximation to the Bayesian linear regression model with priors
~ N(ww, of),
(J.1)
(J.2)
3g ~ (011,.00.100.))
(J.3)
We run the inference for 1 100 epochs, gradient clipping threshold 0.2 and subsampling ratio 0.01.
For the same d, p we then generate another 10000 data points and compute the log-likclihood using the obtained posterior approximation. We repeat the inference and evaluation 50 times for each method and combination of d and p, keeping the generated training and testing set fixed.
24

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm J.1 Routine to generate d-dimensional correlation matrix with given density p and strength of correlations controlled by a. B.
Require: d. E [0, 11, > 0,
Ensure: correlation matrix C
K d(d-1)
b number of non-zero off-diagonal entries
U ++0
k +
C+ Id
for = l,..., K
sample (i E T \ random sample Beta B)
sample correlation strength, controlled by and B i sample sign of correlation
sample f E {-1,1} at random
Ci + fe
C₁ - Cc U+ (-.,)()
end for
Table K.1: Estimated runtimes for UKB experiment
#epochs
single repeat runtime
repeats
#
epsilon values
# initial values
200
4-8min
10
400
3-16min
10
600
12- 18min
10
4
800
16-32min
10
4
1000
20-40min
10
4
2000
40-80min
10
4
4000
80-160min
10
4
8000
160-320min
10
4
K
Runtimes
K.1 UKB experiments
In this experiment, we ran all the variants separately for 10 seeds and levels of privacy. Additionally, we experimented with different runtimes and initialisations. For a training of 1000 epochs, single repeat takes between 20 to 40 minutes. The runtime scales linearly with the number of epochs.
Further, we computed the downstream task for the aligned variants, which includes generating the 100 synthetic data sets and fitting the downstream Poisson regression model on those 100 synthetic data. This procedure takes between 30 to 60 mimutes to complete.
A rough estimate of the runtimes for the UKB experiment given in Table K.1 single CPU core with 8gb of memory used for all the runs.
K.2 Adult experiments
A single training repeat of learning the logistic regression model for all the different variants of DPVI, took between 10 and 30 minutes to finish on single CPU core with Sgb of memory assigned. In total, the Adult experiment was repeated 50 times for four different levels of privacy. Therefore the total runtime of all the experiments is between 2000 and 6 000 minutes.
The variance in running times likely due differences in computation nodes in the clustered assigned by a automatic scheduler.
25

---

Published in Transactions on Machine Learning Research (09/2023)
K.3 Full-rank experiments
A single run for this experiment consisted of the inference using both vanilla and aligned DPVI with full-rank approximations. All runs were executed on computing cluster utilising Nvidia K80, A100, P100, V100 GPU hardware, to which the runs were allocated automatically to balance overall load. As a result, runtimes varied slightly: Runs for and 100 dimensional data took 6-8 minutes to finish, runs for 200 dimensions took 8-10 minutes. With a total of data set configurations and 50 repeats for each, the total runtime is 400 800 minutes.
L Gradient distributions for different variants
Distribution gradlent norms
Distribttonn of gradient norms
Distibution gradient norms
2700
2002
1300
1250
1300
1003
1030
752
500
530
-6
-22
-10
Instietal
(a) Vanilla gradients
(b) Natural gradients
(c) Preconditioned gradients
Figure L.1
Figure L.1 shows the distributions of gradient norms for variational means and scales for different variants of DPVI discussed in Section 3.1.1 when Sy is set to 0.1. Figure L.la clearly shows the different magnitudes for variational standard deviation in vanilla DPVT. Figure L.1b demonstrates that natural gradients simply reverse the problem. Figure Lice shows that the scaling approach achieves matching magnitudes quite well. However, comes the cost of increasing the norm of the full (combined) gradient and therefore incrcasod sensitivity.
26


Reviews:
[{"review_id": "rFTKRlBGWC", "reviewer": "Reviewer_bK6Z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and easy to follow. The experimental results are thorough, and the authors explore a variety of improvements over the vanilla algorithms (including iterate averaging and natural gradients). The paper offers an easy to implement improvement over the vanilla Gaussian mechanism for the problem of interest.\n\nThe key observation of this paper seems to be that that \n\t1. if you want to privately estimate 2 statistics, and one is a function of the other, then you shouldn’t spend privacy budget learning both, and \n\t2. for the case of variational inference with a Gaussian posterior, the derivative wrt the standard deviation is a function of the derivative wrt the mean. \nThe first fact is not very surprising. The proof of the second is short, but (given that the authors do not cite prior work for this result) does seem to be new to this work. The experimental results (excluding Figure H.3) behave as one would expect in that the “aligned” version of the algorithm uniformly outperforms the other methods in estimating both the mean and standard deviation. \n\nI was surprised by the swapping of the order of the aligned and vanilla versions of natural grad in estimating the standard deviation is Figure H.3. The authors mention this is possibly due to hyper-parameter tuning? I would have liked to hear more about this since other than this graph, it seems like the aligned version is uniformly better than the vanilla making the decision of whether to use it in practice very simple.\n\nMinor: On page 1 the authors refer to global sensitivity and local sensitivity as (sensitivity over all variables) and (sensitivity wrt a single variable). This is not the standard definition of these terms in differential privacy so the authors may want to consider alternate terminology. \n\nMinor: Is the second last paragraph in the discussion referring to Figure H.3? If so, a pointer would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the problem of learning a Gaussian variational posterior with differential privacy. They propose a version of DP SGD to learn the mean and variance of the Gaussian posterior which they call \"aligned DPVI\". Rather than use the Gaussian mechanism to add spherical noise to the gradient wrt both the mean and standard deviation, they notice that the gradient wrt the standard deviation can be written as a function of the gradient wrt the mean. They propose using the Gaussian mechanism to release the gradient wrt the mean, then post-process this into an estimate of the gradient wrt the standard deviation. They also compare to a version that scales the gradient wrt standard deviation so its scale matches the gradient wrt the mean, then adds Gaussian noise to both. They perform a series of experiments to show that these three algorithms behave as one would expect (aligned outperforms both other mechanisms).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Discussion of Figure H.3 discussing under what circumstances the aligned algorithm can be worse than the vanilla algorithm.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5SpysvEzzD", "reviewer": "Reviewer_E9ZR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n-The Algorithms 1 and 2 make sense and are intuitive ways to address the misaligned gradients problem. \n\n-Empirically, the algorithms show improvements over vanilla and preconditioned DPVI \n\n**Weaknesses:**\n\n-The writing is unclear in many places, with long sentences and/or insufficient explanation of the core ideas and/or excessive detail too early. Specifically: \n\n--The Abstract is too detailed and not accessible. I am a DP expert, and was confused after reading the abstract, as there was a lot of non-standard terminology (e.g. \"parameter traces'') used before it was defined. Now imagine someone reading the abstract who is only somewhat familiar with DP and/or VI. I would suggest that you re-write the abstract to focus on: explaining the problem of DP VI clearly; then explain what the previous approaches to tackling DP VI are; then explain the shortcomings of these prior approaches; then explain (at a high level, not in great detail, and without using undefined terminology) how you address these shortcomings. \n\n--Intro:  \"which is typically ignored or unknown'' is unclear. I would suggest deleting this and replacing with a new sentence: ``However, the issue of varying sensitivities is often neglected or overlooked.'' \n\n--last sentence of paragraph 1 is unclear. Delete or re-word. \n\n--The problem of DP VI should be defined early in the introduction, *before* the particular problem of perturbed gradients in DP VI (paragraph 2) is discussed. \n\n--``DPVI is a widely applicable...based on DP-SGD'' sentence is out of place. I suggest moving it to the second sentence of paragraph 2. \n\n(More writing issues may be brought up later.) \n\n-Missed references: \n\n--While not directly relevant, it would be good to cite (e.g. in a sentence at the end of the first paragraph of the Related Work section) works that study the use of gradient clipping for obtaining optimal (at least in reference 2) below) rates in DP heavy-tailed stochastic optimization: \n\n1) Kamath, G., Liu, X. and Zhang, H., 2022, June. Improved rates for differentially private stochastic convex optimization with heavy-tailed data. In International Conference on Machine Learning (pp. 10633-10660). PMLR.\n\n2) Lowy, A. and Razaviyayn, M., 2023, February. Private Stochastic Optimization with Large Worst-Case Lipschitz Parameter: Optimal Rates for (Non-Smooth) Convex Losses and Extension to Non-Convex Losses. In International Conference on Algorithmic Learning Theory (pp. 986-1054). PMLR.\n\n--When discussing iterate averaging to reduce noise in parameter estimation, it should be noted that this general idea is not novel and has been used to obtain optimal rates for various problems in DP optimization, e.g.: \n\n1) Bassily, R., Smith, A. and Thakurta, A., 2014, October. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science (pp. 464-473). IEEE.\n\n2) Bassily, R., Feldman, V., Talwar, K. and Guha Thakurta, A., 2019. Private stochastic convex optimization with optimal rates. Advances in neural information processing systems, 32.\n\n3) Lowy, A. and Razaviyayn, M., 2021. Private federated learning without a trusted server: Optimal algorithms for convex losses. arXiv preprint arXiv:2106.09779.  \n\n-Section 2.2: explanation/motivation/intuition of ELBO (and why it is called ELBOW) would be great \n\n-$\\mathbf{\\sigma}_q$ is actually a scalar in the diagonal case \n\n-Section 3.1: $\\eta$ is random. Thus, the statement ``$\\eta_j = O(1)$ does not make sense without some probabilistic condition. \n\n-In Equation 9, you take absolute value of vector-valued quantities. This does not make sense. Also, since these are random variables, you need some probabilistic condition. \n \n-The claim that as $\\sigma_q$ shrinks, $g_s$ becomes small relative to $g_m$ is not precisely stated. Please state this precisely and explain clearly why it is true. Also, you don't address the gradient of the entropy term in your current discussion around Equation 9. \n\n-Aligned gradients: please explain why ``it comes at the cost of increased $\\ell_2$-norm of the full gradient.'' This is an important point that needs to be clearly justified. \n\n-DP of Algorithm 1: Although the post-processing argument seems correct, a rigorous proof that Algorithm 1 is DP should be provided somewhere in the main body or appendix. \n\n-Theorem 3.1: The first sentence is not clear/precise. Also, the claim before Theorem 3.1 that the variance is ``reduced'' is not quite established by Theorem 3.1 since the inequality in Equation 15 may not be strict. Either strengthen (15) to a strict inequality if possible, or weaken the claim before Theorem 3.1. \n\n-Section 3.2 needs major improvements. The purpose of the section is unclear at the outset. Even the title of the section is not very informative. Clarify what the section aims to show or accomplish at the beginning of the section. Quantification of uncertainty (e.g. by DP confidence intervals) would be interesting, but it doesn't seem that you do that. Instead, you seem to aim to reduce the variance of your estimator via iterate averaging. \n\n-Experiments: can you add plots showing error vs. epsilon for DPVI? \n\n-experiments: In Figure 1a, where is the initial value of $\\sigma_q$? I only see a plot comparing error in estimating mean vs. error in estimating std (which, by the way, is not necessarily the most natural or informative plot to include as your first plot). \n\n-Section 4.3: logistic regression is usually thought of as a machine learning problem, not a VI problem. Please clarify why this is a VI problem. Is he ``Vanilla'' baseline just DP-SGD? Please clarify", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers differentially private (DP) variational inference (VI). The paper identifies a problem of \"misaligned'' gradients that occurs when the standard (DP-SGD-based) solution is used: the gradients of different components of the loss functions may have different magnitudes and sensitivities, requiring different amounts of privacy noise. To address this issue, the paper proposes an \"aligned gradient procedure\" which reduces the variance of the estimates of the gradients of the VI loss. Moreover, the paper proposes using iterate averaging to reduce the variance of the parameter estimate. The efficacy of their methods is evaluated empirically. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see \"**Weaknesses**'' above. ", "broader_impact_concerns": "None. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GWYmuh9PXd", "reviewer": "Reviewer_n9qi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The message is clear and simple. \n2. Experiments show that the proposed method is effective and efficiency. \n\nWeakness:\n1. The idea of Alg 1 is natural, and there is no theoretical innovation.\n2. My main concern is the assumption ‘Assume C is chosen such that the bias induced by clipping is the same in vanilla and aligned DPVI’ in Thm 3.1. Is this assumption fair/reasonable? Are there any references? In addition, in the experiments, how to choose C and C’ for two methods?\n3. Some statements are not rigorous. For example, In the paragraph above Eq(9), the author stated that “As $\\eta\\sim N(0,I_d)$, we have $\\eta_j=O(1)$”. It is not correct? We could only say that with high probability, $\\eta_j\\le M$ where $M$ is a constant. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studied differentially private variational inference problems solved by gradient-based methods. A new method called the aligned gradient procedure is proposed, which provides a less noisy gradient estimator. The method can be extended to learn the full-rank covariance approximation. Experiment results in simulated data and real data are provided. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The related statements about $\\eta$ mentioned in Weakness 3 should be revised if I am correct.\n\n2. More discussion on the assumption on $C$ in Thm 3.1\n\n3. More details for experiments, i.e., how to choose $C$.   ", "broader_impact_concerns": "no", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that the main concerns have been addressed, and the paper is technically solid. The paper is a great contribution to the differentially private variational inference problem. Thus I recommend acceptance.
Decision: Accept (poster)