Title: Dual control variate for faster black-box variational inference
Venue: NeurIPS 2023
Accepted: no
Keywords: approximate inference

variational inference
Abstract: Black-box variational inference is a widely-used framework for Bayesian posterior inference, but in some cases suffers from high variance in gradient estimates, harming accuracy and efficiency. This variance comes from two sources of randomness: Data subsampling and Monte Carlo sampling. Whereas existing control variates only address Monte Carlo noise and incremental gradient methods typically only address data subsampling, we propose a new "dual" control variate capable of jointly reducing variance from both sources of noise. We confirm that this leads to reduced variance and improved optimization in several real-world applications.

Abstract

Black-box variational inference is a widely-used framework for Bayesian posterior inference, but in some cases suffers from high variance in gradient estimates, harming accuracy and efficiency. This variance comes from two sources of randomness: Data subsampling and Monte Carlo sampling. Whereas existing control variates only address Monte Carlo noise and incremental gradient methods typically only address data subsampling, we propose a new "dual" control variate capable of jointly reducing variance from both sources of noise. We confirm that this leads to reduced variance and improved optimization in several real-world applications.

1 Introduction 9

Black-box variational inference (BBVI) [12, 22, 16, 2] has become a popular alternative to Markov 10 Chain Monte Carlo (MCMC) methods. The idea is to posit a variational family and optimize it to be 11 close to the posterior, using only "black-box" access to the target model (evaluations of the density 12 or gradient). This is done by estimating a stochastic gradient of the KL-divergence and deploying 13 it in stochastic optimization. A key advantage of this procedure is that it allows the use of data 14 subsampling in each iteration, which can greatly speed-up optimization with large datasets. 15

16

17

18

19

20

The optimization of BBVI is often described as a doubly-stochastic optimization problem [30, 27] in that BBVI's gradient estimation involves two sources of randomness: Monte Carlo sampling from the variational posterior and data subsampling from the full dataset. Because of the doubly-stochastic nature, one common challenge for BBVI is the variance of the gradient estimates: If this is very high, it forces very small stepsizes, leading to slow optimization convergence [20, 3].

21

22

23

24

25

26

27

28

Numerous works have been devoted to reducing the "Monte Carlo" noise that results from drawing samples from the current variational distribution [19, 25, 8, 9, 4]. These methods can typically be seen as creating an approximation of the objective for which the Monte Carlo noise can be integrated exactly, and using this to define a zero-mean random variable, i.e. a control variate, that is negatively correlated with the original gradient estimator. These methods can be used with subsampling by creating approximations for each datum. However, they are only able to reduce Monte Carlo noise for each datum-they do not reduce subsampling noise. This is critical, as subsampling noise is often the dominant source of gradient variance (Sec. 3).

At the same time, for (non-BBVI) optimization problems with only subsampling noise, the opti29

mization community has developed incremental gradient methods that "recycle" previous gradient 30 evaluations [26, 28, 13, 6, 7], leading to faster convergence. These methods do not address Monte 31 Carlo noise. In fact, due to the way these methods rely on efficiently maintaining running averages, 32

they cannot typically be applied to doubly-stochastic problems at all. 33

In this paper, we present a method that jointly controls Monte Carlo and subsampling noise in BBVI. 34

The idea is to create approximations of the target for each datum where the Monte Carlo noise 35

can be integrated exactly. Then, we maintain running averages of the approximate gradients, with 36 noise integrated, overcoming the issue of applying incremental gradient ideas to doubly-stochastic 37 problems. The resulting method not only addresses both forms of noise but interactions between 38 them as well. We demonstrate through a series of experiments with diagonal Gaussian variational 39 inference on a range of probabilistic models that the method leads to lower variance and significantly 40 faster convergence than existing methods. 41

42

2 Background: Black-box variational inference

Given a probabilistic model p ( x, z ) = ∏ N n =1 p ( x n | z ) p ( z ) and observed data { x 1 , . . . , x N } , 43 variational inference's goal is to find a tractable distribution q w ( z ) with parameters w to approximate 44 the (often intractable) posterior p ( z | x ) over the latent variable z ∈ R d . BBVI achieves this by 45 finding the set of parameters w that minimize the KL-divergence from q w ( z ) to p ( z | x ) , which is 46 equivalent to minimizing the negative Evidence Lower Bound (ELBO) 47

where H ( w ) denotes the entropy of q w .

48

Since the inner expectation with respect to z is typically intractable, BBVI methods rely on stochastic 49 optimization with unbiased gradient estimates. These gradient estimates are typically obtained using 50 the score function method [33] or the reparameterization trick [15, 23, 30]. The latter is often the 51 method of choice, as it usually seems to yield estimators with lower variance. The idea is to define a 52 fixed base distribution s ( /epsilon1 ) and a deterministic transformation T w ( /epsilon1 ) such that for /epsilon1 ∼ s , T w ( /epsilon1 ) is 53 equal in distribution to q w . Then, the objective from Equation (1) can be re-written as 54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

and its gradient can be estimated "naively" by drawing a random n and /epsilon1 , and evaluating

BBVI has two advantages. First, since it only evaluates log p (and its gradient) at various points, it can be applied to a diverse range of models, including those with complex and non-conjugate likelihoods. Second, by subsampling data it can be applied to large datasets that might be impractical for traditional methods like MCMC [12, 16].

3 Sources of gradient variance in BBVI

Let V n , /epsilon1 [ ∇ f ( w ; n , /epsilon1 )] denote the variance 1 of the naive estimator from Eq. 3. The two sources for this variance correspond to data subsampling ( n ) and Monte Carlo noise ( /epsilon1 ). It is natural to ask how much variance each of these sources contributes. We study this by (numerically) integrating out each of these random variables individually and comparing the variances of the resulting estimators.

Let f ( w ; n ) = E /epsilon1 f ( w ; n, /epsilon1 ) be the objective for a single datum n with Monte Carlo noise integrated out. This can be thought of as an estimator for datum n with a "perfect" control variate. Similarly, let f ( w ; /epsilon1 ) = E n f ( w ; n , /epsilon1 ) be the objective for a fixed /epsilon1 evaluated on the full dataset. In Fig. 1 we generate a single optimization trace using our gradient estimator (described below). Then, for each iteration, we estimate the variance of ∇ f ( w ; n , /epsilon1 ) , ∇ f ( w ; /epsilon1 ) , and ∇ f ( w ; n ) 2 using a large number of samples. In Table 1 we show the variance at the final iterate on a variety of datasets. (For large datasets, it is too expensive to compute the variance this way at each iteration.)

Our empirical findings suggest that, despite the exact mix of the two sources being task dependent, 72 subsampling noise is usually larger than MC noise. They also show the limits of reducing a single 73 source of noise: No control variate applied to each datum could do better than ∇ f ( w ; n ) , while no 74 incremental-gradient-type method could do better than ∇ f ( w ; /epsilon1 ) . 75

4 Dual Control Variate 76

We now introduce the dual control variate , a new approach for controlling the variance of gradient 77 estimators for BBVI. Control variates [24] can reduce the variance of a gradient estimator by adding 78 a zero-mean random variable that is negatively correlated with the gradient estimator. Considering 79 that the objective of BBVI is a function of both n and /epsilon1 , an ideal control variate should also be a 80 function of these variables. We take two steps to construct such a control variate. 81

82

83

84

85

86

Inspired by existing control variates for BBVI [19, 9], we create an approximation ˜ f ( w ; n, /epsilon1 ) of the true objective f ( w ; n, /epsilon1 ) , designed so that the expectation E /epsilon1 ∇ ˜ f ( w ; n, /epsilon1 ) can easily be computed for any datum n . A common strategy for this is a Taylor-appproximation-to replace f with a low-order polynomial. Then, if the base distribution s ( /epsilon1 ) is simple, the expectation E /epsilon1 [ ∇ ˜ f ( w ; n, /epsilon1 )] is often available in closed-form.

87

88

89

90

91

Inspired by SAGA [6], we maintain a table W = { w 1 , . . . , w N } that stores the variational parameters at the last iteration each of the data points x 1 , · · · , x N were accessed. We also maintain a running average of gradient estimates evaluated at the stored parameters, denoted by M . Unlike SAGA, however, this running average is for the gradients of the approximation ˜ f , with the Monte Carlo noise /epsilon1 integrated out, i.e. M = E n E /epsilon1 ∇ ˜ f ( w n ; n , /epsilon1 ) .

Intuitively, as optimization nears the solution, the weights w tend to change slowly. This means 92 that the entries w n in W will tend to become close to the current iterate w . Thus, if ˜ f is a good 93 approximation of the true objective, we can expect ∇ f ( w ; n, /epsilon1 ) to be close to ∇ ˜ f ( w n ; n, /epsilon1 ) , meaning 94 the two will be strongly correlated. However, thanks to the running average M , the full expectation 95 of ∇ ˜ f ( w n ; n, /epsilon1 ) is available in closed-form. This leads to our proposed gradient estimator 96

Algorithm 1 Black-box variational inference with the dual control variate.

Require: Learning rate λ , variational family q w ( z ) , target p ( z, x )

Require: Estimator f ( w ; n, /epsilon1 ) whose expectation over n and /epsilon1 is the negative ELBO from q w and p (Eq. 2)

Require: Approximate estimator ˜ f ( w ; n, /epsilon1 ) that has an expectation over /epsilon1 in closed form

Initialize the parameter w 0 , the parameter table W = { w 1 , . . . , w N }

Initialize running mean M = E m E η ∇ ˜ f ( w 0 ; m , η ) /triangleright Closed-form expectation over η , explicit sum over m for k = 1 , 2 , · · · do

Sample n and /epsilon1

Extract the value of w n from the table W

Compute the base gradient g ←∇ f ( w k ; n, /epsilon1 )

E

Compute the control variate

c

E

η

m

∇

-∇

m

E

η

∇

˜

f

(

w

m

;

m

,

η

)

˜

f

(

w

n

;

n,

/epsilon1

)

using

E

˜

f

(

w

m

;

m

,

η

) =

M

Update the running mean M ← M + 1 N ( E η ∇ ˜ f ( w k ; n, η ) -E η ∇ ˜ f ( w n ; n, η ) ) /triangleright Closed-form over η

Update the parameter w k +1 ← w k -λ ( g + c ) . /triangleright Or use g + c in any stochastic optimization algorithm end for

Update the table w n ← w k

The running average M = E n E /epsilon1 ∇ ˜ f ( w n ; n , /epsilon1 ) can be cheaply maintained through optimization, 97 since a single value w n changes per iteration and E /epsilon1 ∇ ˜ f ( w ; n, /epsilon1 ) is known in closed form. The 98 variance of the proposed gradient estimator is given by 99

Critically, this expression illustrates that the variance of g dual can be arbitrarily small, only limited by 100 how close ˜ f is to f and how close the stored values w n are to the current parameters w . 101

We illustrate how this gradient estimator can be used for black-box variational inference in Alg. 1. 102 The same idea could be applied more generally to doubly-stochastic objectives in other domains, 103 using the more generic version of the algorithm given in Appendix. D. 104

105

106

107

108

109

110

111

112

113

114

115

116

117

5 Variance reduction for stochastic optimization

This section considers existing variance reduction techniques and how they compare to the proposed dual estimator.

5.1 Monte Carlo sampling and approximation-based control variates

Consider the variational objective from Eq. 2 where we sum over the full dataset in each iteration to define the objective f ( w ) = E /epsilon1 f ( w ; /epsilon1 ) . The gradient estimator obtained by sampling /epsilon1 has been observed to sometimes have problematic variance. Previous work [21, 31, 11, 4] proposed to reduce this variance by constructing a (zero-mean) control variate c ( w ; /epsilon1 ) and defining the new estimator

The hope is that c ( w ; /epsilon1 ) ≈ ∇ f ( w ) - ∇ f ( w ; /epsilon1 ) approximates the noise of the original estimator, which can lead to large reductions in variance and thus more efficient and reliable inference.

A general way to construct control variates involves using an approximation function ˜ f ≈ f for which the expectation E /epsilon1 ˜ f ( w, /epsilon1 ) is available in closed-form [19, 9]. Then, the control variate is defined as c ( w ; /epsilon1 ) = E η ∇ ˜ f ( w ; η ) -∇ ˜ f ( w ; /epsilon1 ) , and the estimator from Eq. (6) becomes

Intuitively, the better ˜ f approximates f , the lower the variance of this estimator tends to be (for a 118 perfect approximation, the variance is fully removed). A popular choice for ˜ f involves a quadratic 119 function, either learned [9] or obtained through a second order Taylor expansion [19], since their 120 expectation under general Gaussian variational distributions is tractable. 121

In doubly-stochastic problems with objectives of the form f ( w ; n, /epsilon1 ) , data n is subsampled as well as 122 /epsilon1 . While the above control variate has most commonly been used without subsampling, it can also be 123 used with subsampling, by developing an approximation ˜ f ( w ; n, /epsilon1 ) to f ( w ; n, /epsilon1 ) for each datum n . 124 This leads to the control variate E η ∇ ˜ f ( w ; n, η ) -∇ ˜ f ( w ; n, /epsilon1 ) and gradient estimator 125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

It is important to note that this control variate is unable to reduce variance coming from data subsampling. Even if ˜ f ( w ; n, /epsilon1 ) were a perfect approximation there would still be gradient variance due to n being sampled randomly. This can be shown by noting that the variance of this estimator is given by (see Appendix. B.1 for a full derivation using the law of total variance)

While the first term of the expression above can be made arbitrarily small in the ideal case of a perfect approximation ˜ f ≈ f , the second term is irreducible, regardless of the quality of the approximation used. Therefore, this approach cannot reduce subsampling variance. As shown in Fig. 2 and Table 1, subsampling variance is typically substantial, and often several orders of magnitude larger than Monte-Carlo variance. When this is true, this control variate, which is only able to reduce variance coming from Monte Carlo sampling, will have minimal effect on the overall gradient variance.

5.2 Data subsampling and incremental gradient methods

We now consider a stochastic optimization problem with objective f ( w ) = E n f ( w ; n ) , where n is uniformly distributed on { 1 , . . . , N } , representing data indices, but no other stochasticity (i.e. no Monte Carlo sampling). While one could compute f or its gradient exactly, this is expensive when N is large. A popular alternative involves drawing a random n and using the estimator ∇ f ( w ; n ) with a stochastic optimization method, such as stochastic gradient descent. Alternatively, for such problems, incremental gradient methods [26, 28, 13, 7, 10] often lead to faster convergence.

While details vary by algorithm, the basic idea of incremental gradient methods is to "recycle" previous gradient evaluations to reduce randomness. For example, SAGA [6] stores the parameters w n of the most recent iteration where f ( w ; n ) was evaluated and takes a step as

where λ is a step size and the expectation over m is tracked efficiently using a running average, meaning the cost per iteration is independent of N . The update rule above can be interpreted as using a control variate to reduce the variance of the naive estimator ∇ f ( w ; n ) as

When w m ≈ w , the first and last terms in Eq. (11) will approximately cancel, leading to a gradient 149 estimator with significantly lower variance. 150

Wenow consider a doubly-stochastic objective f ( w ; n, /epsilon1 ) . In principle, one might consider computing 151 the estimator from Eq. (11) for each value of /epsilon1 , i.e. using the gradient estimator 152

153

154

This has two issues. First, the resulting method does not address Monte Carlo noise due to sampling /epsilon1 . This can be shown by noting that the variance of this estimator is given by (see Appendix B.2)

Since the second term in the variance expression above is irreducible, the variance cannot be expected 155 to go to zero, no matter how close all the stored vectors n are to the current parameters. Intuitively, 156

w this approach cannot do better than simply evaluating the objective on the full dataset for a random /epsilon1 . 157

The second issue is more critical: g inc cannot be implemented efficiently . The value of ∇ f ( w n ; n, /epsilon1 ) 158 is dependent on /epsilon1 , which is resampled at each iteration. Therefore, it is not possible to efficiently 159 maintain E m ∇ f ( w m ; m , /epsilon1 ) needed by Eq. (12) as a running average. In general, this can only 160 be computed by looping over the full dataset in each iteration. While possible, this destroys the 161 computational advantage of subsampling. For some models with special structure [32, 34] it is 162 possible to efficiently maintain the needed running gradient. However, this can only be done in 163 special cases with model-specific derivations, breaking the universality of BBVI. 164

165

166

167

168

It may seem odd that g inc has these computational issues, while g dual -an estimator intended to reduce variance even further-does not. The fundamental reason is that the dual estimator only stores (approximate) gradients after integrating over the Monte Carlo variable /epsilon1 , so the needed running average is independent of /epsilon1 .

169

170

171

172

173

174

175

176

5.3 Ensembles of control variate

It is possible to combine multiple control variates. For example, [8] combined control variates that reduced Monte Carlo noise [19] with one that reduced subsampling noise [32] (for a special case where g inc is tractable). While this approach can be better than either control variate alone, it still does not reduce joint variance. To see this, consider a gradient estimator that uses a convex combination of the two above control variates. For any β ∈ (0 , 1) write

It can be shown (Appendix B.3) that if both c cv and c inc are "perfect", that is, if ˜ f ( w ; n, /epsilon1 ) = f ( w ; n, /epsilon1 ) and w n = w for all n , then

Even in this idealized scenario, such an estimator cannot reduce variance to zero, because each of 177 the individual control variates leaves one source of noise uncontrolled. The dual control variate 178 overcomes this because it models interactions between /epsilon1 and n . 179

180

181

182

183

184

185

6 Related work

Recent work proposed to approximate the optimal batch-dependent control variate for BBVI using a recognition network [4]. Similar to our work, they take into account the usage of subsampling when designing their variance reduction techniques for BBVI. However, like g cv , their control variate reduces the conditional variance of MC noise (conditioned on n ) but is unable to reduce subsampling noise (like g cv ).

It is also worth discussing a special incremental gradient method called SMISO [1], designed 186 for doubly-stochastic problems. Intuitively, SMISO uses exponential averaging to approximately 187 marginalize out /epsilon1 , and then runs MISO/Finito [7, 18] (an incremental gradient method similar to 188 SAGA) to control the subsampling noise. While the method is similar to running SGD with an 189 incremental control variate, it is not obvious how to separate the control variate from the algorithm, 190 meaning we cannot use the SMISO idea as a control variate to get a gradient estimator that can be 191 used with other optimizers like Adam, we include a detailed discussion on this issue in Appendix. A. 192 Nevertheless, we still include SMISO as one of our baselines. 193

194

195

196

197

7 Experiments

This section empirically demonstrates the effectiveness of the dual control variate for BBVI. We focus on mean-field Gaussian BBVI, where the variational posterior follows a multivariate Gaussian with diagonal covariance q w ( z ) = N ( µ , diag( σ 2 )) , with parameters w = ( µ , log( σ )) .

The gradient estimators g cv ( w ; n, /epsilon1 ) and g dual ( w ; n, /epsilon1 ) require an approximation function with 198 expectation over /epsilon1 available in closed form. Inspired by previous work [19], we get an approximation 199 for f ( w ; n, /epsilon1 ) using a second order Taylor expansion for the negative total likelihood k n ( z ) = 200

N log p ( x n | z ) + log p ( z ) around z 0 = T w (0) 3 , which yields 201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

where we assume the entropy can be computed in closed-form. For a mean-field Gaussian variational distribution, the expected gradient of the approximation Eq. (16) can only be computed efficiently (via Hessian-vector products) with respect to the mean parameter µ but not for the scale parameter σ , which means g cv ( w ; n, /epsilon1 ) and g dual ( w ; n, /epsilon1 ) can only be used as the gradient estimator for µ . Fortunately, controlling only the gradient variance on µ often means controlling most of the variance, as, with mean-field Gaussians, the total gradient variance is often dominated by variance from µ [9].

7.1 Experiment setup

We evaluate our methods by performing BBVI on a range of tasks: binary Bayesian logistic regression on two datasets, Sonar (number of samples N = 208 , dimensionality D = 60 ) and Australian ( N = 690 , D = 14 ); multi-class Bayesian logistic regression on MNIST [17] ( N = 60000 , D = 7840 ); probabilistic principal component analysis [29] (PPCA, N = 60000 , D = 12544 ); and BradleyTerry model [5] for tennis player ranking (Tennis, N = 169405 , D = 5525 ). We give full model descriptions in Sec. 7.3.

Baselines. We compare g dual (Eq. (4)) with g naive (Eq. (3)) and g cv (Eq. (8)). For Sonar and Australian (small datasets) we include g inc (Eq. (12)) as well, which requires a full pass through the full dataset at each iteration. For larger-scale problems, g inc becomes intractable, so we use SMISO instead.

Optimization details. We optimize using Adam [14] for the larger-scale MNIST, PPCA, and Tennis datasets and SGD without momentum for the small-scale Sonar and Australian dataset for transparency. The optimizer for SMISO is pre-determined by its algorithmic structure and cannot be changed. For all estimators, we perform a step-size search (see Appendix C) to ensure a fair comparison and use a single shared /epsilon1 for all samples in the batch.

Mini-batching. In practice, for efficient implementation on GPUs, we draw a mini-batch B of data at each iteration (reshuffling for each epoch). For inc , dual , and SMISO, we update multiple entities in the parameter table per iteration and adjust the running mean accordingly. For the Sonar and Australian datasets, due to their small sizes, we use | B | = 5 . For other datasets we use | B | = 100 .

Evaluation metrics. We track the ELBO on the full dataset, explicitly computing E n (summing over the full dataset) and approximating E /epsilon1 with 5000 Monte Carlo samples. We present ELBO vs. iterations plots for a single example learning rate as well as ELBO values for the best learning rate chosen retrospectively for each iteration. In addition, we present the final ELBO after training vs. step size at different iterations. For the Sonar and Australian datasets, given the small size, we include a detailed trace of gradient variance on µ across different estimators. This enables empirical validation of the lower bounds derived in Eq. (9) and Eq. (13).

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

GLYPH<c=7,font=/AAAABN+DejaVuSans>

GLYPH<c=9,font=/AAAABN+DejaVuSans>GLYPH<c=12,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>

GLYPH<c=12,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>

GLYPH<c=18,font=/AAAABN+DejaVuSans>!GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>!GLYPH<c=27,font=/AAAABN+DejaVuSans>GLYPH<c=30,font=/AAAABN+DejaVuSans>GLYPH<c=29,font=/AAAABN+DejaVuSans>

$GLYPH<c=8,font=/AAAABN+DejaVuSans>GLYPH<c=14,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>

$GLYPH<c=8,font=/AAAABN+DejaVuSans>GLYPH<c=13,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>

GLYPH<c=17,font=/AAAABN+DejaVuSans>GLYPH<c=19,font=/AAAABN+DejaVuSans>GLYPH<c=16,font=/AAAABN+DejaVuSans> GLYPH<c=20,font=/AAAABN+DejaVuSans>GLYPH<c=3,font=/AAAABN+DejaVuSans>GLYPH<c=4,font=/AAAABN+DejaVuSans>GLYPH<c=21,font=/AAAABN+DejaVuSans>GLYPH<c=30,font=/AAAABN+DejaVuSans>GLYPH<c=29,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>GLYPH<c=5,font=/AAAABN+DejaVuSans>

GLYPH<c=28,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>GLYPH<c=15,font=/AAAABN+DejaVuSans>GLYPH<c=8,font=/AAAABN+DejaVuSans>GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=6,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=10,font=/AAAABN+DejaVuSans>

GLYPH<c=7,font=/AAAABN+DejaVuSans>

GLYPH<c=9,font=/AAAABN+DejaVuSans>GLYPH<c=12,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>

GLYPH<c=12,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>

GLYPH<c=18,font=/AAAABN+DejaVuSans>!GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>!GLYPH<c=27,font=/AAAABN+DejaVuSans>GLYPH<c=30,font=/AAAABN+DejaVuSans>GLYPH<c=29,font=/AAAABN+DejaVuSans>

GLYPH<c=16,font=/AAAABN+DejaVuSans>GLYPH<c=25,font=/AAAABN+DejaVuSans> !GLYPH<c=3,font=/AAAABN+DejaVuSans>GLYPH<c=28,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>

GLYPH<c=8,font=/AAAABN+DejaVuSans>GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=6,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=11,font=/AAAABN+DejaVuSans>

GLYPH<c=8,font=/AAAABN+DejaVuSans>GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=6,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=10,font=/AAAABN+DejaVuSans>

GLYPH<c=19,font=/AAAABN+DejaVuSans>

GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>

GLYPH<c=29,font=/AAAABN+DejaVuSans>GLYPH<c=27,font=/AAAABN+DejaVuSans>GLYPH<c=29,font=/AAAABN+DejaVuSans>GLYPH<c=26,font=/AAAABN+DejaVuSans>GLYPH<c=3,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>!GLYPH<c=25,font=/AAAABN+DejaVuSans>

GLYPH<c=8,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=3,font=/AAAABN+DejaVuSans>GLYPH<c=27,font=/AAAABN+DejaVuSans>!GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>!GLYPH<c=27,font=/AAAABN+DejaVuSans>GLYPH<c=30,font=/AAAABN+DejaVuSans>GLYPH<c=29,font=/AAAABN+DejaVuSans>

GLYPH<c=8,font=/AAAABN+DejaVuSans>GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=6,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=11,font=/AAAABN+DejaVuSans>

GLYPH<c=8,font=/AAAABN+DejaVuSans>GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=6,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=10,font=/AAAABN+DejaVuSans>

GLYPH<c=19,font=/AAAABN+DejaVuSans>

GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>

GLYPH<c=29,font=/AAAABN+DejaVuSans>GLYPH<c=27,font=/AAAABN+DejaVuSans>GLYPH<c=29,font=/AAAABN+DejaVuSans>GLYPH<c=26,font=/AAAABN+DejaVuSans>GLYPH<c=3,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>!GLYPH<c=25,font=/AAAABN+DejaVuSans>

GLYPH<c=12,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=7,font=/AAAABN+DejaVuSans>GLYPH<c=3,font=/AAAABN+DejaVuSans>GLYPH<c=27,font=/AAAABN+DejaVuSans>!GLYPH<c=25,font=/AAAABN+DejaVuSans>GLYPH<c=31,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>!GLYPH<c=27,font=/AAAABN+DejaVuSans>GLYPH<c=30,font=/AAAABN+DejaVuSans>GLYPH<c=29,font=/AAAABN+DejaVuSans>

GLYPH<c=29,font=/AAAABN+DejaVuSans>GLYPH<c=22,font=/AAAABN+DejaVuSans>GLYPH<c=27,font=/AAAABN+DejaVuSans>#GLYPH<c=25,font=/AAAABN+DejaVuSans>

GLYPH<c=23,font=/AAAABN+DejaVuSans>#

GLYPH<c=27,font=/AAAABN+DejaVuSans>GLYPH<c=29,font=/AAAABN+DejaVuSans>GLYPH<c=23,font=/AAAABN+DejaVuSans>

GLYPH<c=24,font=/AAAABN+DejaVuSans>"GLYPH<c=22,font=/AAAABN+DejaVuSans>GLYPH<c=28,font=/AAAABN+DejaVuSans>

GLYPH<c=7,font=/AAAABO+DejaVuSans>

GLYPH<c=9,font=/AAAABO+DejaVuSans>GLYPH<c=12,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>

GLYPH<c=12,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>

GLYPH<c=19,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=25,font=/AAAABO+DejaVuSans>GLYPH<c=28,font=/AAAABO+DejaVuSans>GLYPH<c=27,font=/AAAABO+DejaVuSans>GLYPH<c=30,font=/AAAABO+DejaVuSans>

!GLYPH<c=9,font=/AAAABO+DejaVuSans>GLYPH<c=14,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>

!GLYPH<c=9,font=/AAAABO+DejaVuSans>GLYPH<c=13,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>

GLYPH<c=18,font=/AAAABO+DejaVuSans>GLYPH<c=20,font=/AAAABO+DejaVuSans>GLYPH<c=17,font=/AAAABO+DejaVuSans> GLYPH<c=21,font=/AAAABO+DejaVuSans>GLYPH<c=3,font=/AAAABO+DejaVuSans>

GLYPH<c=3,font=/AAAABO+DejaVuSans>GLYPH<c=4,font=/AAAABO+DejaVuSans>GLYPH<c=16,font=/AAAABO+DejaVuSans> GLYPH<c=30,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=26,font=/AAAABO+DejaVuSans>GLYPH<c=25,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=27,font=/AAAABO+DejaVuSans>GLYPH<c=5,font=/AAAABO+DejaVuSans>

GLYPH<c=26,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>GLYPH<c=15,font=/AAAABO+DejaVuSans>GLYPH<c=8,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=6,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=10,font=/AAAABO+DejaVuSans>

GLYPH<c=7,font=/AAAABO+DejaVuSans>

GLYPH<c=9,font=/AAAABO+DejaVuSans>GLYPH<c=12,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>

GLYPH<c=12,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>

GLYPH<c=19,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=25,font=/AAAABO+DejaVuSans>GLYPH<c=28,font=/AAAABO+DejaVuSans>GLYPH<c=27,font=/AAAABO+DejaVuSans>GLYPH<c=30,font=/AAAABO+DejaVuSans>

GLYPH<c=17,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=30,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=3,font=/AAAABO+DejaVuSans>GLYPH<c=26,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>

GLYPH<c=8,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=6,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=11,font=/AAAABO+DejaVuSans>

GLYPH<c=8,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=6,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=10,font=/AAAABO+DejaVuSans>

GLYPH<c=20,font=/AAAABO+DejaVuSans>

GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>

GLYPH<c=27,font=/AAAABO+DejaVuSans>GLYPH<c=25,font=/AAAABO+DejaVuSans>GLYPH<c=27,font=/AAAABO+DejaVuSans>GLYPH<c=24,font=/AAAABO+DejaVuSans>GLYPH<c=3,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>

GLYPH<c=8,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=3,font=/AAAABO+DejaVuSans>GLYPH<c=25,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=25,font=/AAAABO+DejaVuSans>GLYPH<c=28,font=/AAAABO+DejaVuSans>GLYPH<c=27,font=/AAAABO+DejaVuSans>GLYPH<c=30,font=/AAAABO+DejaVuSans>

GLYPH<c=8,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=6,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=11,font=/AAAABO+DejaVuSans>

GLYPH<c=8,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=6,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=10,font=/AAAABO+DejaVuSans>

GLYPH<c=20,font=/AAAABO+DejaVuSans>

GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>

GLYPH<c=27,font=/AAAABO+DejaVuSans>GLYPH<c=25,font=/AAAABO+DejaVuSans>GLYPH<c=27,font=/AAAABO+DejaVuSans>GLYPH<c=24,font=/AAAABO+DejaVuSans>GLYPH<c=3,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>

GLYPH<c=12,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=7,font=/AAAABO+DejaVuSans>GLYPH<c=3,font=/AAAABO+DejaVuSans>GLYPH<c=25,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=23,font=/AAAABO+DejaVuSans>GLYPH<c=29,font=/AAAABO+DejaVuSans>GLYPH<c=22,font=/AAAABO+DejaVuSans>GLYPH<c=31,font=/AAAABO+DejaVuSans>GLYPH<c=25,font=/AAAABO+DejaVuSans>GLYPH<c=28,font=/AAAABO+DejaVuSans>GLYPH<c=27,font=/AAAABO+DejaVuSans>GLYPH<c=30,font=/AAAABO+DejaVuSans>

Initialization. The variational parameters are randomly initialized using a standard Gaussian and all results reported are averages over multiple independent trials: We run 10 trials for Sonar and Australian, and 5 for the larger scale problems due to resource constraint.

7.2 Results

The experiment results for Sonar and Australian are presented in Fig. 2 and Fig. 3. Both the inc and cv estimators have lower variance than the naive estimator, but the improvement varies by the dataset. The excellent performance of the (impractical) inc estimator on Australian shows the importance of reducing subsampling noise. Overall, the dual estimator has the lowest variance, which enables larger learning rates and thus faster optimization.

Similar results can be observed on MNIST, PPCA, and Tennis in Fig. 4 (for these datasets inc is intractable, so we include SMISO as a baseline instead). Again, dual yields faster and better convergence than naive and cv . Whereas SMISO, which does not adopt momentum nor adaptive step size, suffers from slow convergence speed in that it has to utilize a small step size to prevent diverging during optimization. We provide comparisons of different estimators using SGD in Appendix. E.

7.3 Model descriptions

Binary/Multi-class Bayesian logistic regression. Astandard logistic regression model with standard Gaussian prior.

Probabilistic principal component analysis (PPCA). Given a centered dataset x 1 , . . . , x N ∈ R D , PPCA [29] seeks to extract its principal axes W ∈ R D × K by assuming x n ∼ N ( 0 , WW /latticetop + diag( λ 2 )) . In our experiments, we employ a standard Gaussian prior on W and use BBVI to approximate the posterior over W . We then test PPCA on the standardized training set of MNIST with K = 16 and λ = 1 .

Bradley Terry model (Tennis). This is a model used to rank players from pair-wise matches. Each player is represented by a score θ i , and each score is assigned a standard Gaussian prior. The result of a match between two players is modeled by the inverse logit of their score difference y n ∼ Bernoulli(logit -1 ( θ i -θ j )) where y n = 1 denotes a win by player n . We subsample over matches and perform inference over the score of each player. We evaluate the model on men's tennis matches log starting from 1960, which contains the results of 169405 matches among 5525 players.

263

264

265

266

267

268

269

270

271

272

273

274

7.4 Efficiency analysis

We now study the computational cost of different estimators. In terms of the number of "oracle" evaluations (i.e. evaluations of f ( w ; n, /epsilon1 ) or its gradient), the naive estimator is the most efficient, requiring a single oracle evaluation per iteration. The cv estimator requires one gradient and also one Hessian-vector product, while the dual estimator needs one gradient and two Hessian-vector products, one for the control variate and one for updating the running mean M .

Additionally, Table 2 shows measured runtimes based on a JAX implementation on an Nvidia 2080ti GPU. All numbers are for a single optimization step, averaged over 200 steps. Overall, each iteration with the dual estimator is between 1.5 to 2.5 times slower than naive , and around 1.2 times slower than cv . Lastly, given that dual achieves a given performance in an order of magnitude fewer iterations (Figs. 3 and 4), it is the fastest in terms of wall-clock time. The exact wall-clock time v.s. ELBO results are presented in Appendix. F.

275

References

Alberto Bietti and Julien Mairal. Stochastic optimization with variance reduction for infinite 276 datasets with finite sum structure. Advances in Neural Information Processing Systems , 30:1623277 1633, 2017. 278

David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for 279 statisticians. Journal of the American statistical Association , 112(518):859-877, 2017. 280

Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine 281 learning. Siam Review , 60(2):223-311, 2018. 282

Ayman Boustati, Sattar Vakili, James Hensman, and ST John. Amortized variance reduction 283 for doubly stochastic objective. In Conference on Uncertainty in Artificial Intelligence , pages 284 61-70. PMLR, 2020. 285

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the 286 method of paired comparisons. Biometrika , 39(3/4):324-345, 1952. 287

288

289

290

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in neural information processing systems , pages 1646-1654, 2014.

291

292

293

294

295

296

297

298

299

300

301

302

Aaron Defazio, Justin Domke, et al. Finito: A faster, permutable incremental gradient method for big data problems. In International Conference on Machine Learning , pages 1125-1133. PMLR, 2014.

Tomas Geffner and Justin Domke. Using large ensembles of control variates for variational inference. In Advances in Neural Information Processing Systems , pages 9982-9992, 2018.

Tomas Geffner and Justin Domke. Approximation based variance reduction for reparameterization gradients. Advances in Neural Information Processing Systems , 33, 2020.

Robert M Gower, Mark Schmidt, Francis Bach, and Peter Richtárik. Variance-reduced methods for machine learning. Proceedings of the IEEE , 108(11):1968-1983, 2020.

Will Grathwohl, Dami Choi, Yuhuai Wu, Geoff Roeder, and David Duvenaud. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. In International Conference on Learning Representations , 2018.

Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational 303 inference. Journal of Machine Learning Research , 2013. 304

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance 305 reduction. Advances in neural information processing systems , 26:315-323, 2013. 306

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 307 arXiv:1412.6980 , 2014. 308

Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In International 309 Conference on Learning Representations , 2014. 310

Alp Kucukelbir, Dustin Tran, Rajesh Ranganath, Andrew Gelman, and David M Blei. Automatic 311 differentiation variational inference. The Journal of Machine Learning Research , 18(1):430-474, 312 2017. 313

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning 314 applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. 315

Julien Mairal. Incremental majorization-minimization optimization with application to large316 scale machine learning. SIAM Journal on Optimization , 25(2):829-855, 2015. 317

Andrew C Miller, Nicholas J Foti, Alexander D'Amour, and Ryan P Adams. Reducing 318 reparameterization gradient variance. Advances in Neural Information Processing Systems , 319 2017:3709-3719, 2017. 320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on optimization , 19(4):15741609, 2009.

John Paisley, David M Blei, and Michael I Jordan. Variational bayesian inference with stochastic search. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1363-1370, 2012.

Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Artificial intelligence and statistics , pages 814-822. PMLR, 2014.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , pages 1278-1286. PMLR, 2014.

Christian P Robert, George Casella, and George Casella. Monte Carlo statistical methods , volume 2. Springer, 1999.

Geoffrey Roeder, Yuhuai Wu, and David K Duvenaud. Sticking the landing: Simple, lowervariance gradient estimators for variational inference. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Nicolas Roux, Mark Schmidt, and Francis Bach. A stochastic gradient method with an exponential convergence _rate for finite training sets. Advances in neural information processing systems , 25, 2012.

Hugh Salimbeni and Marc Deisenroth. Doubly stochastic variational inference for deep gaussian processes. Advances in neural information processing systems , 30, 2017.

Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research , 14(2), 2013.

Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 61(3):611-622, 1999.

Michalis Titsias and Miguel Lázaro-Gredilla. Doubly stochastic variational bayes for nonconjugate inference. In International conference on machine learning , pages 1971-1979. PMLR, 2014.

George Tucker, Andriy Mnih, Chris J Maddison, Dieterich Lawson, and Jascha Sohl-Dickstein. Rebar: low-variance, unbiased gradient estimates for discrete latent variable models. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 2624-2633, 2017.

Chong Wang, Xi Chen, Alexander J Smola, and Eric P Xing. Variance reduction for stochastic gradient optimization. Advances in neural information processing systems , 26, 2013.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement learning , pages 5-32, 1992.

Shuai Zheng and James Tin-Yau Kwok. Lightweight stochastic optimization for minimizing 359 finite sums with infinite data. In International Conference on Machine Learning , pages 5932360 5940. PMLR, 2018. 361

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a \"dual\" control variate for reducing gradient variance in black-box variational inference in the context of models that admit data subsampling (i.e. exhibit the required conditional independence). The dual control variant is joint in that it simultaneously addresses the two sources of Monte Carlo variance in ELBO approximations: that due to latent variable sampling and that due to data subsampling. The basic idea, in essence, is to leverage linear or quadratic ELBO approximations (which admit closed form evaluation for e.g. gaussian mean field variational families) in conjunction with a running average of gradient estimates for each data point using the variational parameters from the last iteration in which each data point was accessed. Experiments demonstrate that the proposed method can substantially reduce gradient variance (in particular that due to data subsampling, which is often dominant), thus yielding better ELBO optimization (both w.r.t. wallclock time and final ELBOs obtained).\n\nSTRENGTHS:\nThe main strengths of this submission include the following:\n- it addresses a general problem of relatively wide interest in the NeurIPS community (namely, how best to do black-box variational inference) \n- it addresses a particular component of that problem that is often somewhat overlooked compared to other aspects (namely how best to do optimization)\n- the experiments are pretty convincing in establishing the efficacy of the method\n- the suggested method is technically sound and would appear to be pretty simple to implement\n- the discussion in Sec 5 and the variance analysis (appendix B) help the reader conceptually place the proposed control variate alongside other alternatives\n\nWEAKNESSES:\nThe main weaknesses of this submission, as I see them, are the following:\n- the notation is a bit confusing in some places\n- some of the limitations of the method and/or extensions to more general problem settings are either not discussed or are insufficiently discussed\n\nLet me expand on these points:\n\nWhile the notation for this kind of paper will necessarily be somewhat clunky given the various sources of sampling variability that have to be carefully tracked, I think some improvements are possible. In particular I find the choice of M for the running gradient particularly suboptimal. Since capital N is a positive integer and little n and little m are used to index integers, one might expect that M is also a positive integer. I suggest that M be replaced by something like G(bold $w$) to avoid this confusion and to emphasize that G depends on the \"table\" of $w$s.\n\nThe authors consider a generic but still somewhat limited problem specification, in particular they do not consider local latent variables, model learning, or amortization. More discussion of these points would probably be of considerable interest to the reader (for more discussion see below). In addition one of the weaknesses of this method is the potentially large memory requirements, which are O(Nd) where d is the latent dimension. This needs to be *very clearly emphasized*.\n\nQUESTIONS:\n- Do you expect your method would work more or less unchanged for hierarchical models that feature both global and local latent variables? Or would the algorithm need to be adapted to that setting?\n- Arguably besides the possibility of data subsampling for appropriate model classes, some of the biggest selling points of variational inference are amortization and model learning (i.e. learning point estimates for model parameters). \n  - I guess you could support model learning directly without any change to your algorithm, although I suspect the performance of your control variate would tend to decrease since now in addition to \"$w$ drift\" you'd also have \"$\\theta$ drift\" (where theta is the model parameter). Can you please comment?\n  - I guess you could also support amortization (e.g. of local latent variables in hierarchical models) without any changes to your algorithm, although since amortization typically involves a neural network with a largeish number of parameters, this would make the memory requirements needed to cheaply compute the running average M intractably large. Can you please comment?\n- typo: appproximation\n- It would be great to see some ablation studies comparing first order to second order taylor approximations. How important is going to second order in practice?\n- What kind of runtime performance would you expect if used a multivariatiate gaussian distribution for the variational distribution? Would the dual estimator still be ~1.5 to ~2.5 times slower than computing control variant free gradients? Or would the gap increase?\n- It should be possible to do a more fine-grained theoretical analysis for a model that admits more analytic control, e.g. bayesian linear regression. Such an analysis could be particularly valuable in further delineating the regimes in which the dual variate is expected to perform best.\n- You state that \"the total gradient variance is often dominated by variance from μ\" [line 207]. Do you have any intuition for this claim?\n- Comment: For a large dataset initializing the running mean M at the beginning seems like it could be a waste of compute time. It might make more sense to not use the control variate during the first epoch and use the first epoch to collect M.\n\nLIMITATIONS:\nAs discussed in the weaknesses section, I believe some of the limitations of the method (e.g. w.r.t.~memory requirements and likely trouble with amortization) should be discussed and/or better emphasized.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new control variate for black-box variational inference. In particular, the proposed \"dual\" control variate attempts to reduce the subsampling noise and Monte Carlo noise at the same time. For this, the paper utilizes an incremental gradient-like scheme. The performance of the new control variate is empirically verified on Bayesian inference tasks with large datasets.\n\nSTRENGTHS:\n* While control variates have been an active area of research for BBVI, reducing the subsampling noise has certainly been a problem that hasn't been addressed. In fact, the paper shows that conventional control variate solutions do not solve this problem *at all*, despite the fact that subsampling is a major source of variance. \n* The paper motivates the latter point by empirically computing the subsampling noise. Overall, the paper conveys the motivations for the proposed control variate very clearly.\n* The proposed control variate based on incremental gradients seems fairly simple to implement, but not trivial. Thus, the proposed strategy has clear technical contribution.\n* Empirical evaluations are thorough and adequate to show the superiority of the proposed control variate.\n\nWEAKNESSES:\n* Given that the paper builds on top of incremental gradient methods, which, as typical of the optimization community, comes with a heap of theoretical tools for rigorous analysis. Given this fact, it would have been amazing if the paper also provided some rigorous analysis of the proposed strategy.\n* There seems to be some room for improvement in terms of paper organization and notation. The notation can be quite confusing at times, for example, lots of things happen behind innocent-looking superscripts. More on this below.\n\nQUESTIONS:\n### Major Comments\n* What was the motivation behind calling the control variate \"dual\"? At least to people who have some experience in optimization, I believe the word dual would automatically trigger the notion of Lagrangian duality. How about \"doubly\" in accordance with the term \"doubly stochastic?\"\n* The section organization seems to have room for improvement. For example, a general explanation of \"general\" control variates appears way back in Section 5.1. Considering readers that are not familiar with control variates, it would be better to start with a high-level explanation of how a control variate is supposed to work. A good place would be after Section 3, where the variance of BBVI is motivated to be a problem, and before Section 4 where a control variate is first introduced.\n* Similarly, I felt that the whole discussion in Section 5.1 and 5.2 would have been a very good way to inspire the readers before actually unrevealing the dual control variate. While reading Section 5.1 and 5.2 I had to constantly go back to Section 4 to remind how the dual control variate attempts to solve the problem.\n* Algorithm 1 is not very helpful to precisely understand the implementation. For example, in \"Compute the control variate\" what \"using ... $=M$ \" mean? Does this mean we simply plug $M$ in place of $ E_m E_{\\eta} \\nabla \\tilde{f} \\left(w^m; m, \\eta\\right) $ ? Then why denote  $ E_m E_{\\eta} \\nabla \\tilde{f} \\left(w^m; m, \\eta\\right) $ ? I think here the authors need to clearly distinguish between values and computed quantities. Similarly, the notation for the running mean $M$ is too uninformative. I think something like $\\tilde{g}$ or $\\hat{g}$ would be more appropriate. This is also useful to understand the dimensionality/domain of each variable.\n* Line 207: Doesn't this contradict the conclusions of [2,3]? From my understanding of [2], the $\\mathcal{O}\\left(d\\right)$ dimensional dependence comes from the gradients with respect to the scale.\n* Figure 4 Tennis: Do you have any insight as why the ELBO for the dual control variate peaks and then decreases? This seems quite weird given that the control variate is unbiased.\n* Caption of Figure 4: This is perhaps a minor point, but could you elaborate why gradient noise of Tennis is correlated? Is this because reshuffling was used here instead of iid subsampling?\n* Section 6: Please include a discussion of other non-dual control variates since this works, in the broader scheme of things, is a control variate paper. Also, why not include Section 5.3 in Section 6?\n\n### Minor Comments\n* Eq. (1) I was a little bit confused at first because I am so used to see the likelihood adjustment ratio to be N/B, where B is the batch size. I think it would be useful to either use N/B overall so that the notion that we're dealing with batches is more clear, or to mention in the text that the batch size is assumed to be 1.\n* Line 59: The classic paper that first proposed doubly stochstic BBVI was [1]. I recommend adding it here.\n* Table 1: How about using a bar plot instead of a table? I think it would be more appropriate since the relative magnitude is key here.\n* Line 118: I think it would be better to explicitly show that the performance of a control variate is entirely driven by the covariance between the estimate and the control variate.\n\n### References\n* [1] Titsias, Michalis, and Miguel Lázaro-Gredilla. \"Doubly stochastic variational Bayes for non-conjugate inference.\" In International conference on machine learning, pp. 1971-1979. PMLR, 2014.\n* [2] Domke, Justin. \"Provable gradient variance guarantees for black-box variational inference.\" Advances in Neural Information Processing Systems 32 (2019).\n* [3] Bhatia, Kush, Nikki Lijing Kuang, Yi-An Ma, and Yixin Wang. \"Statistical and computational trade-offs in variational inference: A case study in inferential model selection.\" arXiv preprint arXiv:2207.11208 (2022).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for variance reduction in stochastic gradient estimation in doubly stochastic variational inference \nwhere there exists two sources of variance: (i) Monte Carlo noise when sampling from the variational distribution and (ii) gradient \nvariance due to the minibatch sampling. The authors consider reparametrizable Gaussian variational inference and introduce a control variate that tries to reduce simultaneously both sources of variance. This control variate combines previous ideas, such as a second order Taylor expansion of the function as in [19], and it seems to reduce the variance in the presented experiments.\n\nSTRENGTHS:\nThe paper is very clearly written and all derivations appear to be correct. It contains also intuitive discussions why the proposed \"coupled\" control variate can be useful as opposed to other control variates that deal separately with each source of variance. \n\nCertainly variance reduction is a very important topic for stochastic gradient estimation in variational inference, and the paper proposes a potentially useful method.   \n\nThe experiments provide many details including running times.\n\nWEAKNESSES:\nI was not so impressed by the experimental results for two reasons. Firstly the models used are quite small and it would be useful to include e.g. a big neural network. Secondly, I am not sure if the comparison is done in a fair way for methods such the \"naive\" method.  This is because the \"dual method is more expensive and requires more gradient evaluations, such as the ones for the numerical approximation of the Hessian-vector products. Given that computations are dominated by the number of gradient evaluations, a fair comparison should try to match this number across different estimators. For example,  for the \"naive\" estimator someone could increase the minibatch size so that the number of gradient evaluations matches the one of the \"dual\" method.\n\nQUESTIONS:\nWhy the control variate in equation (16) controls only the variance for the $\\mu$?  Is this discussed in [9]?\n\nLIMITATIONS:\nThe limitations are explained above regarding the current experimental comparison.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nExisting stochastic methods for black-box variational inference only attempt to reduce the noise either due to data subsampling or Monte-Carlo sampling of the expectation. This paper proposed a new \"dual control variate\", which addresses both types of noise at the same time. In experiments, the proposed control variate is shown to perform favorably to existing baselines.\n\nSTRENGTHS:\n1) The paper disentangles the effects of noise through the data and noise from Monte-Carlo estimation of the expectation in the design of the new dual control variate, which addresses both at the same time. \n2) The proposed control variate is shown to greatly improve performance on the considered examples, at seemingly minimal overhead. If the x-axis in the plots would have been wall clock time, this would be clearer to see.\n\nWEAKNESSES:\nThe main weakness of the paper is in my opinion that the experimental evaluation is a bit lacking: \na) larger experiments would be desirable, see questions;  \nb) the plots were hard to read (difficult to distinguish the red and green curves), and could be wrt. wall clock time and not iterations. \n\nThe algorithm looks promising, and I like that it can be used in a \"black-box\" fashion with any optimizer (e.g. Adam); but I believe it is not ready yet in its current state and could require a more thorough experimental evaluation to warrant a clear acceptance.\n\nQUESTIONS:\nWould the proposed dual control variate approach also work for VI in neural networks? Experiments on modern neural networks (e.g. ResNets, transformers) could be a way to convincingly demonstrate the utility of the proposed method. \n\nHow much additional overhead (memory & runtime) does it have when compared to, say, Bayes-by-Backprop when actually implemented in practice? \n\nFrom Table 2 it seems that one would require three backpropagation passes rather than a single one for the naive method -- it would be interesting to see whether this additional effort has a real practical benefit (e.g. higher accuracy, faster convergence wrt wall clock time).\n\nLIMITATIONS:\nAll limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the drawback of the black-box variational inference framework for Bayesian posterior inference by proposing dual control variate that is capable of jointly reducing the variances from both data subsampling and Monte Carlo sampling. The experimental evaluations on various datasets demonstrates reduced variance and improved optimization.\n\nSTRENGTHS:\nThe paper is fairly written well. The background of the black-box variational inference (BBVI) is nicely described. The doubly-stochastic optimization problem in BBVI's gradient estimation is clearly explained involving two sources of randomness - Monte Carlo sampling from the variational posterior and data subsampling from the full dataset. \n\nThe proposed dual control variate that jointly controls Monte Carlo and subsampling noise in BBVI to create approximations of the target for each datum where the Monte Carlo noise can be integrated exactly, addressing both forms of noise and interactions between them.\n\nExperimental evaluation and visualization of dual estimator.\n\nWEAKNESSES:\n- Although the idea is novel but its usefulness/impact could have explained well.\n\n- The approximation function for the gradient estimators g_cv and g_dual could have been clarified.\n\n- It is unclear how the noise in Monte Carlo sampling influences the noise in data sampling. Is there a way to measure it?\n\n- What is the role of Beta in eqn(14 - 15). Is it experimentally evaluated?\n\nQUESTIONS:\nI am not fully familiar with this subject. However, I did get idea about the problem and how the propose idea could address it. Therefore, I  will consider the reviews of other reviewers. \n\nThe main problem is the writing/explanation of the technical terms to make readers to understand the problem. The paper started well and somehow there is a lost connection between experimental evaluation to demonstrate the impact of the dual control variate real-world applications\n\nLIMITATIONS:\nThere is no issue with potential negative societal impact. However, the article does not address limitations (if any).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}