Fourier Stochastic Backpropagation
Authors: Amine Echraibi
Joachim Flocon Cholet
Stéphane Gosselin
Sandrine Vaton
Venue:  (2021)
Keywords: Stochastic Backpropagation
Variational Inference
Probabilistic Graphical Models
Deep Learning

Abstract: Backpropagating gradients through random variables is at the heart of numerous machine learning applications. In this paper, we present a general framework for deriving stochastic backpropagation rules for any distribution, discrete or continuous. Our approach exploits the link between the characteristic function and the Fourier transform, to transport the derivatives from the parameters of the distribution to the random variable. Our method generalizes previously known estimators, and results in new estimators for the gamma, beta, Dirichlet and Laplace distributions. Furthermore, we show that the classical deterministic backproapagation rule and the discrete random variable case, can also be interpreted through stochastic backpropagation.

Under review as a conference paper at ICLR 2021
FOURIER STOCHASTIC BACKPROPAGATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Backpropagating gradients through random variables is at the heart of numerous machine learning applications. In this paper, we present a general framework for deriving stochastic backpropagation rules for any distribution, discrete or contin- uous. Our approach exploits the link between the characteristic function and the Fourier transform, transport the derivatives from the parameters of the distribu- tion to the random variable. Our method generalizes previously known estimators, and results in new estimators for the gamma, beta, Dirichlet and Laplace distribu- tions. Furthermore, we show that the classical deterministic backproapagation rule and the discrete random variable case, can also be interpreted through stochastic backpropagation.
INTRODUCTION
Deep neural networks with stochastic hidden layers have become crucial in multiple domains, such
as generative modeling (Kingma & Welling [013 Rezende et al., 2014: Mnih & Gregor 2014). deep reinforcement learning (Suttonn et al.) 2000). and attention mechanisms (Mnih et al.] 2014). The
difficulty encountered in training such models arises in the computation of gradients for functions of the form [())= := Ex-po [f)] with respect to the parameters 0, thus needing to backpropagate the gradient through the random variable z. One of the first and most used methods is the score function or reinforce method (Glynn 1989; Williams 1992), that requires the computation and estimation of the derivative of the log probability function. For high dimensional applications however, it has been noted that reinforce gradients have high variance, making the training process unstable (Rezendel et al. 2014).
Recently, significant progress has been made in tackling the variance problem. The first class of approaches dealing with continuous random variables are reparameterization tricks. In that case standardization function is introduced, that separates the stochasticity from the dependency on the parameters A. Thus being able to transport the derivative inside the expectation and sample from fixed distribution, resulting in low variance gradient (Kingma & Welling 2013] Rezende et al. 2014]
Titsias & Lázaro-Gredilla 2014 Ruiz et al. [016]] Naesseth et al.) 2017 Figurnov al.) 2018). The second class of approaches concerns discrete random variables, for which a direct reparameterizatioo is not known. The first solution uses the score function gradient with control variate methods to reduce its variance (Mnih & Gregor) 2014, Gu et al. 2016). The second consists in introducing continuous relaxation admitting reparameterization trick of the discrete random variable, thus being able to backpropagate low-variance reparameterized gradients by sampling from the concrete distribution (Jang et al., 2016) Maddison et al. 2016 Tucker et al. 2017 Grathwohl et al.; (0188).
Although recent developments have advanced the state-of-the-au in terms of variance reduction and performance, stochastic backpropagation (i.e computing gradients through random variables) still lacks theoretical foundation. In particular, the following questions remain open: How to develop stochastic backpropagation rules, where the derivative transferred explicitly the function f for a broader range of distributions? And can the discrete and deterministic cases be interpreted in the sense of stochastic backpropagation? In this paper, we provide a new method to address these questions, and our main contributions are the following:
We present theoretical framework based on the link between the multivariate Fourier transform and the characteristic function, that provides a standard method for deriving stochastic backpropagation rules, for any distribution discrete or continuous.

---

Under review as a conference paper at ICLR 2021
We show that deterministic backpropagation be interpreted as special case of stochastic backpropagation, where the probability distribution pe is a Dirac delta distribution, and that also be interpreted as backpropagating discrete derivative.
the discrete
We generalize previously known estimators, and provide rules for the special cases of the Laplace, gamma, beta, and Dirichlet distributions.
stochastic backpropagation
We demonstrate experimentally that the resulting new estimators are competitive with state-of-the art methods on simple tasks.
BACKGROUND & PRELIMINARIES
Let (E, be dd-iienniioall measure space equipped with the standard inner product, and f be square summable positive real valued function on E, that is, f: - R+, with J (d)) < co. Let pa be an arbitrary parameterized probability density on the space E. We denote by ypo its characteristic function, defined as: (p((w) = Eempole" J.
We denote by f the Fourier transform of
the function defined
(ww) (ww:=FU(() [(2)e [(=)e= =1(2))
(1)
The inverse Fourier transform is given in this case by:
=())=)))))))= I f(w)ee
& su(dw),
(2)
where ((dw) represents the measure in the Fourier domain. this paper we treat the cases where dw and the case where E is a discrete set, for which the measure E Rd for which p(dw) = (2x)a.
Ju defined as: p(dw) = 1| E [-x, T]
(2m) Throughout the paper, we reserve the letter to denote the imaginary unit: ,2 1 To denote higher order derivatives of the function f, the multi-index notation (Saint Raymond 2018). For multi-index n = (71, nad) E Nd. we define:
du
glml
0
Oz ...O2
where
|
n
and wn
II w
To clarify the multi-index notation, let us consider the example where d = 3, and = (1,0, in this case:
g3
8
02,023 and, w" =
:p::w w/w3.
The objective is to derive stochastic backpropagation rules, similar to that of (Rezende et al. 2014). ) for functions of the form: L())= Ex-po [z)]. for any arbitrary distribution pe. discrete or continuous.
FOURIER STOCHASTIC BACKPROPAGATION
Stochastic backpropagation rules similar to that (Rezende et al. 2014) can fact be derived for any continuous distribution, under certain conditions the characteristic function. In the following theorem we present the main result of our paper concerning the derivation Fourier stochastic backpropagation rules.
Theorem 1. (Continuous Stochastic Backpropagation) Let C (Rd R4), under the condition that V log Pe is a holomorphic function of iw, then there exists a unique {an (f)) E R such that:
VoL
Σ (0)BEa-p 10 fz)).
(3)
In|20
Where (1)) are the Taylor expansion coefficients of Vo log soa (w::
Vo log (olg) = Σ
an (nww)
(4)
ml/20
2

---

Under review as a conference paper at ICLR 2021
Proof. Let rewrite L in terms of
((0)= = / (o{=){(=)A(da)
po(3)=-
(5)
S f(w) / po(2)e ((dz)u(dw) Fubini's theorem J flw)po(w)ylad).
By introducing the derivative under the integral sign, and using the reinforce trick (Williams applied to Pa. where (w) (w)(w) V log (o(w)) Equationn]becca becomes:
1992)
VeL / f(w))o(w)) Va log po (w)) 2
(6)
Under analyticity conditions of the gradient of the log characteristic function, we can expand the gradient term V log (ol(w) (w), in terms of Taylor series around zero
Ve log (o(()) Σ an (O)
1n/20
Putting everything together, and replacing the characteristic function by its expression, the gradient of L becomes:
VoL f ((w) / po po(z)el>>> ((
ml
By rearranging the sums using Fubini's theorem second time, we obtain the following expression for the gradient:
VeL Ezups
F
W
(z) [ {wi+ (iw) ((w) (z)
Σ an (ww f(w) f(W)
1m/20
Σ
On (0)
(9)
m/20
Σ
an (0) [0
m2/0
Q.E.D
Identically, we can follow the same procedure for discrete random variables. We suppose that pe factorizes over disjoint cliques of the dependency graph, where each dimension z, takes values in discrete space Vzl(.)). In theorem 2 we derive the result concerning the discrete case.
Theorem 2. (Discrete Stochastic Backpropagation) Let E be discrete space: E 11 Val(z;). and C the set of disjoint cliques of the dependency graph over that is,
(2(
II po(ze)
EEC
then,
VoL. Σ Σ
VoPo(ze)BB cupe [D((2co2 tz
(10)
CEC
Where:
z represents the normalizing assignment po(2%) 1-E1- Eznfte Po(Ze).
3

---

Under review as a conference paper at ICLR 2021
2f( == f(2-f((a2ee0 f(z-c1
(1I)
Proof. The characteristic function for the factored distribution is given by:
(p(ww) II so (wo), (wi)) Σ Zat1aE
po(zo)e"0
Σ
po(ze)
educto.
(12)
CEC
Thus the gradient of the log characteristic function becomes:
cuuf eiwb 20
(0) (wi)) Vg log Po
Σ Zofaz
Vepe(ze)
(13)
48 (wec)
By plugging this expression equation [ we obtain:
Vepo(ze) / II p (we) <EC 26
VoL Σ
- f(w)
c'
(14)
Vopo Ex ~pe ~ (f)
CEC 2nylag
Q.E.D
The estimator of equation 10 has been derived in the literature through Rao-Blackwellization of the score function gradient, and it has been known under different names (Titsias &
2015 Asadi al., 2017; Cong al.) 2019). Theorem 2 shows that the discrete case can also be seen backpropagating derivative of the function J, in this case discrete derivative given by
equation 11
4 APPLICATIONS OF FOURIER STOCHASTIC BACKPROPAGATION
Following from the previous section, we derive the stochastic backpropagation estimators for certain commonly used distributions.
The multivariate Gaussian distribution: In this case (=(z) N(2;; 148 Eg). The log characteristic
function is given by: log (w) inu w + Tr Egri Thus by applying theorem 1, we recover the stochastic backpropagation rule of (Rezende al. 2014):
{(OH))
T vef(z)
(DE0 00
VoL = Ez~po
2 Tr
(15)
00
where, V. and 2 represent the gradient and hessian operators.
The multivariate Dirac distribution: pg(2) = 5a, (2), the log characteristic function of the Dirac distribution is given by: log (oo(w) iw"" ag. Thus the stochastic backpropagation rule of the Dirac given by:
iaa 00
T
(
(aas 00
T
VeL
Ex-dap [V.f(z)] =
V.f(ag),
(16)
resulting in the classical backpropagation rule. In other words, the deterministic backpropagation rule is a special case of stochastic backpropagation where the distribution is Dirac delta distribution. This result provides a link between probabilistic graphical models and classical neural networks. We investigate this link further in Appendix A.
The multivariate Bernoulli: (z(2) = I B(2z; To (i)), where - = Plzz = 1]. By applying theorem 2
we obtain the local expectation gradient of (Titsias & Lázaro-Gredilla 2015):
d
Im (f)
VoL
Σ
E2 p pm 1) 1
(17)
30
4

---

Under review as a conference paper at ICLR 2021
The multivariate categorical: p=(=) I cat(zz; =))))=))))) where the dimensions are independent and take values in the set {L..., K.. Similarly to the Bernoulli case, we obtain the following stochastic backpropagation rule:
d
K
OTok ()
VOL
En_ [D((z,),))]
(18)
00
The Laplace distribution: po(2) = (2; pr. bo). in this case the log characteristic function is the following: log (w) kuuh log(1 + 2 using the Taylor series expansion for the function get the following stochastic backpropagation rule for the Laplace distribution:
0pe
df
ab bg
2m
VoL
Eg
(z) dz
+
2
E
(z)
(19)
00
30
d2
The gamma distribution: po(2) = T(2; ko (O), the log characteristic function of the Gamma
distribution is given by: log (w(w) - kig log(1 (. By expanding it using Taylor series of the logarithm function, we obtain the following stochastic backpropagation rule:
aka 0e
ka θpe
dd
VoL =
00 14 Eu-po
den (z)
(20)
The estimator of equation 20 gives stochastic backpropagation rule for the gamma distribution and, hence also applies by extension to the special cases of the exponential, Erlang, and chi-squared distributions.
The beta distribution: po(=) Beta(z; 2; Cia, Bo), in this case the characteristic function is the confluent hypergeometric function: Ye (w( = Fi (arg (ig + :g; A series expansion of gradient of the log of this function is not trivial to derive. However, linking the gamma and beta distributions to derive a stochastic backpropagation rule. Indeed,
the parameterization
~ Beta(as, Bo). By substituting in Ş1 ~ [(as, 1) and Ç2 ~ T(3,, 1). then z = g(L1, = ci+C2
S1
the gamma stochastic backpropagation rule, we obtain:
dag 00
f 21 (4// )
0
0
C1 C1 + C2 )
VoL = Σ n=1
62
1CT C1 C
00
Bg
(21)
The Dirichlet distribution: po())
Dir(z; K, ag), following the same procedure, as for the beta
distribution and using the following parameterization: Zn
Σ C
St with, So I[ag (k)
1),
obtain:
K
(k) Eg,vj
S1
SK
VoL
1 EK "
(22)
ae
ack
EEj=
5 TRACTABLE CASES & APPROXIMATIONS OF FOURIER STOCHASTIC BACKPROPAGATION
The Fourier stochastic backpropagation gradient presented in previous sections presents two major computational bottlenecks for non-trivial distributions. The first is the computation of infinite series, and the second is evaluating higher order derivatives of the function f. Depending on the application, the function f could be chosen in order to bypass the computational bottlenecks. A trivial example, if the higher order derivatives of the function f vanish at a certain order: f 0. Another example, is the exponential function f(z) = exp(cT=)) 22). From the fact that it obeys the following partial differential equation rule reduces in this case to:
of d2, (z) {z), one can deduce that the stochastic backpropagation
VoL V log yo
En-ns [f(z)]
Ex-go
(33)
In most real world applications however, the infinite sum will not often reduce to a tractable expression such that of the exponential. An example of this case is the evidence lower bound of generative
5

---

Under review as a conference paper at ICLR 2021
d
10
d 100
2.4
0.2
0.2
0.2
2.0
6.0
30000
20000
30009
40000
50000
10000
20000
50000
40300
50000
10000
20000
39000
40000
30000
30
Log the
10000
23000
30000
45000
52000
10000
20000
10000
40000
50000
10020
20026
10030
40030
50800
Stops
Stops
Steps
Figure 1: Training loss and log variance of the gradients for the different estimators for f(z) = d
Ej= (zj e)² for d 100)
model with Bernoulli observations. In this case, the natural solution to truncate the sum up to a finite order. The assumption (although might be wrong), is that the components associated to higher frequencies of the spectrum of the gradient of the log characteristic function, do not contribute as much. And by analogy the signal processing field, apply Low-pass filter eliminate them. In this case the gradient of the log characteristic function of equation becomes:
Vo log (ww) = Σ (ni()) )((ww)))
(24)
ASN
6 EXPERIMENTS
In our experimental evaluations, we test the stochastic backpropagation estimators of equations and 20 for the gamma and Laplace distributions. In the case of the gamma estimator, we use toy examples where we can derive exact stochastic backpropagation rules without truncating the infinite sum. As for the Laplace stochastic backpropagation rule, test the estimator in the case of Bayesian logistic regression with Laplacian priors and variational posteriors on the weights. We compare our estimators with the pathwise (Jankowiak & Karaletsos] 2019: Jankowiak & Obermeyer] 2018), and score function estimators, in addition to the weak reparameterization estimator in the gamma case (Ruiz al., 2016). We do not control variates in setup, the goal is verify the exactness of the proposed infinite series estimators and how they compare to current state-of-the-an methods in simple settings. all our experiments, use the Adam optimizer update the weights (Kingma Ba 2014), with a standard learning rate of 10- In all the curves, we report the mean and standard deviation for all the metries considered over iterations.
6.1
TOY PROBLEMS
In the toy problem setting, we test the gamma stochastic backpropagation rule following the same procedure as [Mohamed et al. 2019). we consider the following cases:
Toy problem 1: C(0) = Ex-po Zz <ll?], where pe(z) = II 1'(zyık, M4), 0 = [k, μJ. and
c= -4. In this case, we only need to compute the first and second order derivatives of the function f.
cz,) in this case, the infinite sum transfers to which results in the following estimator: VoL = Va log yoo (+) Ex-pp [f(z)].
Toy problem 2: L()) Ex~pa E
In figures D and 2 we report the training loss and log variance of the gradient across iterations of gra-
dient descent for different values of the dimension d {1, 10, 100). The stochastic backpropagation
estimator converges to the minimal value all cases faster than the other estimators and the variance
of the gradient is competitive with the pathwise gradient.

---

Under review as a conference paper at ICLR 2021
= 100
1.00
Tranting ic)
0.75
9.75
0.5
0.25
0.0
0.00
0.00
19000
20030
30000
40000
30000
10000
20000
S0000
40090
50000
10000
20030
30000
40000
50000
kiddin
20
20
20
-29
10000
20000
30000
40000
50000
10000
20000
30000
40500
50000
10000
20000
30000
40000
50000
Steps
Steps
Steps
Figure 2: Training loss and log variance of the gradients for the different estimators for f(z)
=
j==1
d
(EEz) for d E {1, 10, 100}.
-0.3
0.9
Training (c)
log variance
Accuracy
-0.5
0.8
-0.6
0.7
Stochartic badiomppageen (W-11
-0.7
-10
Stxchas9c backarspgg it (Hee)) Stochards backorspegiite.oo (ii-0)
20000
40000
20000
40000
20000
40000
Steps
Steps
Steps
Figure 3: Bayesian Logistic Regression with Laplacian priors
6.2 BAYESIAN LOGISTIC REGRESSION WITH LAPLACIAN PRIORS
We evaluate the Laplace stochastic backpropagation estimator using Bayesian logistic regression model (Jaakkola Jordan, 1997). similarly to (Mohamed al. 2019). In our case, we substitute the normal prior and posterior on the weights with Laplace priors and posteriors. We adopt the same notations (Murphy/ 2012), where the data, target and weight variables are respectively: 2n E gd. V/ E 1,, and w. The probabilistic model in case is the following:
p(w) II Lww,, 1) ply/x, w) (yx w).
(25)
where a represents the sigmoid function. We consider Laplacian variational posteriors of the form (ww) = 1 Lww, My. bj), with = b). The evidence lower bound of single sample is given by:
[(zn, Bna 0) Ew-po [logo (vn -T w) - -DuLpelll))
(26)
where the Kullback-Leibler divergence between the two Laplace distributions is the following:
leg - -log by Insltbbe
DeL(Polle)
(27)
We the model on the UCI women's breast cancer dataset (Dua Graff 2017), with a batch size of 64 and 50 samples from the posterior to evaluate the expectation. In the case of the stochastic backpropagation estimator we truncate the infinite series for the scale parameter b of equation 19 to N = 4 and N = 8. In figure we report the training evidence lower bound, the log variance of the gradient, and the accuracy computed on the entire dataset for the different estimators. The stochastic backpropagation estimator converges faster than the considered estimators and the variance significantly lower. We also notice that the truncation level of the infinite series for the scale parameter has little effect the outcome. In figure/4 at different values of the truncation level, for a fixed parameter value during the training phase (0poch=100). The bias and variance do not vary much, with the truncation level in this case, this
report the bias and variance of the estimator
7

---

Under review as a conference paper at ICLR 2021
3.044
0.0000
3,002
5:10
3.00
2.05
Parkol on lead
DiAraain lowal
Figure 4: (Left) Bias and variance of the gradient for different values of the truncation level at fixed parameter value. (Right-top) Mean square error between the Laplace gradient estimator and the score function and reparameterization estimators across iterations. (Right-bottom) Norm of the gradient estimators.
result confirms the intuition of neglecting higher frequencies presented in section 5 In addition, we compare the mean squared error between the Laplace stochastic backpropagation estimator and the score function and reparameterization estimators. As shown in figure 4 the mean squared error is small, thus the values of the gradients across iterations are close. However, the reparameterization gradient is closer our estimator than the score function gradient, probably due to the fact that the reparameterization gradient is more stable and has lower variance.
RELATED WORK & DISCUSSION
Computing gradients through stochastic computation graphs has received considerable attention from the community, due to its application in many fields. The first general approach that provides a closed form solution for any probability distribution is the score function method (Glynn 1989 Williams) 1992, Sutton et al.] 22000) Schulman et al.] (2155). The main inconvenience of this approach, is that results in high variance gradients when the dimension of the random variable becomes high. In order to bypass this issue, the second approach consisted of designing control variates to reduce the
variance of the score function estimator (Paisley al. 2012 Weaver & Tao 2013 Mnih & Gregor
2014: Ranganath et al.
2014 Tokui & Sato 2017). In addition to the score function gradient, it
was proposed to use an importance weighted estimator instead of the classical score function with multi-sample objective (Mnih & Rezende 2016 Burda et al., 2015).
The second class of approaches that concerning reparameterization tricks (Kingma Welling 2013 Rezende et al., 2014). Through the decoupling of the computation of the gradient from the expectancy, reparameterization tricks have shown that they provide low-variance gradients using often a single sample. The issue for these methods is the necessity to find reparamcterization for each probability distribution. Certain distributions such as the Gaussian are easy to reparametrize but others like the gamma not. In addition, discrete random variables do not admit an easy reparameterization as well. Recently, these issues has been partially solved through implicit reparameterization. the generalized reparameterization gradient, and the pathwise gradient (Ruiz al. 2016 FFiuurov cet al. [2018] Jankowiak & Obermeyer] 2018). For the discrete case, continuous relaxations that are reparameterizable have been proposed and combined with control variate methods (Gu al. 2016
Maddison et al. 2016 Jang et al. 2016 Tucker et al. 2017: Grathwohl et al. 2018).
Our approach, in contrast provides a new broad family of stochastic backpropagation rules derived using the Fourier transform. One interesting aspect of our approach is the fact that the weighting a, separated from the expectation of the higher order derivatives of the function f. Thus the sampled variable does not intervene in the weighting in contrast to other methods such as reparameterization and pathwise gradients. In addition, the function f contains weakly correlated terms, by applying the derivative, some random variables would be eliminated. Thus the estimators of the higher order derivatives would be sampled with respect to the derivation variable (and other correlated variables), which would result in lower variance.
is worth noting that deriving stochastic backpropagation rules using the Fourier transform has been proposed for the Gaussian case (Fellows et al.) 2018). In work, we extend it non Gaussian distributions by way of the characteristic function, and exploiting the invariance of the functional inner product under Fourier transformation (Parseval's theorem).
8

---

Under review as a conference paper at ICLR 2021
8 CONCLUSION
In conclusion, in this paper we presented new method to compute gradients through random variables for any probability distribution, by explicitly transferring the derivative to the random variable using the Fourier transform. Our approach, gives framework to be applied for any distribution, where the gradient of the log characteristic function is analytic, resulting in new broad family of stochastic backpropagation rules, that are unique for each distribution.
REFERENCES
Kavosh Asadi, Cameron Allen, Melrose Roderick, Abdel-rahman Mohamed, George Konidaris, Michael Littman, and Brown University Amazon. Mean actor critic. stat, 1050:1, 2017. 2017.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv: 1509.0050 2015.
Yulai Cong, Miaoyun Zhao, Ke Bai, and Lawrence Carin. Go gradient for expectation-based objectives. arXiv preprint arXiv:1901.06022. 2019.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive
ics. uci. ledumml
Matthew Fellows, Kamil Ciosek, and Shimon Whiteson. Fourier policy gradients. arXiv preprint arXiv:1802.06881, 2018.
Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterization gradients. In Advances in Neural Information Processing Systems, pp. 441-452 2018.
Peter W Glynn. Optimization of stochastic systems via simulation. Proceedings of the 21st conference on Winter simulation, pp. 90-105, 1989.
Will Grathwohl, Dami Choi, Yuhuai Wu, Geoff Roeder, and David Duvenaud. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. 2018.
Shixiang Gu, Sergey Levine, Ilya Sutskever, and Andriy Mnih. Muprop: Unbiased backpropagation for stochastic neural networks. 2016.
Tommi Jaakkola and Michael Jordan. A variational approach to bayesian logistic regression models and their extensions. In Sixth International Workshop on Artificial Intelligence and Statistics, volume 82, 1997.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.00 0114011411.11 2016.
Martin Jankowiak and Theofanis Karaletsos. Pathwise derivatives for multivariate distributions. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 333-342, 2019.
Martin Jankowiak and Fritz Obermeyer. Pathwise derivatives beyond the reparameterizzatio trick. In International Conference Machine Learning, pp. 2235-2244, 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: arXiv:/412.600 141266980, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.6114, 2013.
Brenden M Lake, Russ R Salakhutdinov, and Josh Tenenbaum. One-shot learning by inverting compositional causal process. In Advances in neural information processing systems, pp. 2526- 2534, 2013.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: :/yt::///nnn lecun. cwm /e////xdbmmmmmm st.

---

Under review as a conference paper at ICLR 2021
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. 2016.
Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on International Conference on Machine Learning-Volen Volume 32. pp. 1I-1791, 2014.
Andriy Mnih and Danilo J Rezende. Variational inference for monte carlo objectives. arXiv preprint 18K.006602.0625.. 2016.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In Advances in neural information processing systems, pp. 2204-2212, 2014.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. arXiv preprini arXiv: /906. 10652, 2019.
Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.
CA Naesseth, FJR Ruiz, SW Linderman, and DM Blei. Reparameterization gradients through acceptance-rejeccioon sampling algorithms. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, 2017.
Radford M Neal. Connectionist learning of belief networks. Artificial intelligence, 56(1):71-113, 1992.
John Paisley, David Blei, and Michael Jordan. Variational bayesian inference with stochastic search. arXiv preprint 1206.6630 2012.
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Artificial Intelligence and Statistics, pp. 814-822. PMLR, 2014.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wienstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pp. 1278-1286, 2014.
Francisco R Ruiz, Michalis Titsias RC AUEB, and David Blei. The generalized reparameterizatiio gradient. In Advances in neural information processing systems, pp. 460-468, 2016.
Xavier Saint Raymond. Elementary introduction to the theory of pseudodiffereetial operators, chapter 1, pp. 2-3. Routledge, 2018
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems, pp. 3528
3536, 2015.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in neural information processing systems, pp. 1057-1063, 2000.
Michalis Titsias and Miguel -reeilll.. Doubly stochastic variational bayes for non-conjugate
inference. In International conference on machine learning, pp. 1971-1979, 2014.
Michalis Titsias and Miguel Greaill Local expectation gradients for black box variational inference. In Advances in neural information processing systems, pp. 2638-2646, 2015.
Seiya Tokui and Issei Sato. Evaluating the variance of likelihood-ratio gradient estimators. In International Conference Machine Learning, pp. 3414-3423, 2017.
George Tucker, Andriy Mnih, Chris Maddison, John Lawson, and Jascha Sohl-Dickstein. Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models. In Advances in Neural Information Processing Systems, pp. 2627-2636, 2017.
Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-based reinforcement learning. arXiv preprini arXiv: 1301.2315, 2013.
Ronald J Williams. Simple statistical follient algorithms for connectionist reinforcement learning. Machine learning, 8(3-4)222-255 1992.
10

---

Under review as a conference paper at ICLR 2021
Dataset
Model
REBAR
RELAX
BSB (S=1)
BSB (S=5)
BSB (S=10)
one layer SBN
-114.14 + 0.44
-114.55 + 0.48
-10.87 + 0.2
-11.770 + 0.11
-110.59 0.08
MNIST
two layer SBN
-101.33 = 0.04
-10109 = 0.07
-99.74 = 0.3
-100.44 + 0.28
-100.66 0.21
Bern. VAE
-127.76 1 0.84
-128.06
1 2.66
-107.4 1: 1.47
-108.46 0.37
-109.19 1.31
one layer SBN
-123.66 + 0.05
-123.82 + 0.17
-113.53 + 0.21
-114,34 + 0.16
-114.37 + 0.19
Omniglot
two layer SBN
-117.81 = 0.17
-117.89
= 0.04
-102.05 + 0.19
-102.16 + 0.09
100.229 = 0.14
Bern. VAE
-136.83 1 0.31
-13.553 1 0.32
-126.94 1 0.81
-128.69 1 0.34
-129.48 1 0.38
Table 1: Test likelihood for the Bernoulli stochastic backpropagation (BSB) estimator compared to the REBAR and RELAX estimators. We report the mean and standard deviation over runs.
A THE DIRAC DISTRIBUTION: THE LINK BETWEEN NEURAL NETWORKS AND PROBABILISTIC GRAPHICAL MODELS
x
h (1)
h (L)
y
Figure 5:
A hidden variable probabilistic model, where the observed variables are the data x and target y, with L hidden stochastic layers h (1:L..
In this section, we explore the connection between neural networks and probabilistic graphical models following from the stochastic backpropagation rule of the Dirac delta distribution. To this end, let us consider the probabilistic graphical model of figure [ The observed random variables in this model denoted x and y representing the data and target variables. We place the analysis in supervised learning context, but the argument is valid for unsupervised models as well. As usual the goal is maximize the log likelihood for the data samples (Ir, y). which is intractable, given that need to integrate over the hidden variables. However using variational inference, we can maximize an evidence lower bound of the form:
C(O: x, y) = Enttit log h (1:E) m)] + H[gg( (-x]
(28)
As suggested in the Dirac stochastic backpropagation rule, let us assume that the variational posteriors and priors are Dirac delta distribution of the form:
qo h (1) p(h (++1) (1) 62 () 4800) (h (h(+)))
AO SOSIS- (29)
where, the (1), (!),, and ((4) represent respectively the activation functions, the weights and biases for layer 1, with the convention x (0) Under these assumptions, the Kullback-Leibler divergence term is equal to zero, and the evidence lower bound reduces to the the logeliielihood of a classical neural network:
L(0; y) log (x))) with, go(x) a (W + b())...
Thus, when using neural networks we are indirectly using probabilistic graphical model and making the strong assumption that the hidden layers follow a parameterized Dirac distribution knowing the previous layer.
B EXPERIMENTS USING DISCRETE STOCHASTIC BACKPROPAGATION
We evaluate the Bernoulli and Categorical Stochastic Backpropagation estimators (BSB and CSB) of equations 17 and 18 on standard generative modeling benchmark tasks, using the MNIST and Omniglot datasets (LeCun & Cortes) 2010: Lake et al. 2013). We use the REBAR, RELAX. and Gumbell-offmant (or Concrete) estimators as baselines for our comparison (Jng et al. 2016
Maddison et al., 2016; Tucker et al. 2017 Grathwohl et al. 2018). The Bernoulli stochastic backpropagation is compared to the REBAR and RELAX estimators for three models: the sigmoid belief network of one and two stochastic hidden layers (Neal) 1992) and the variational autoencoder.
11

---

Under review as a conference paper at ICLR 2021
One Layer
Two Layer
-100
-120
-117
-120
-130
-139
The Hig
340
-150
1140
B58 (5=1)
558 (5=10)
B50 (5--)
140
B5B (5-1)
5:8 (5+10)
658 [5+1]
-100
555 (5m10)
353 (5-51
5S1 [S-11
-160
RELAX
R:84
NESA
130500
123000
Stegs
Strps
Ctpt
Iogrutieeee pooken:
(5+1)
RGR [5-11
-55
-10
ese 15-10) REBAR
RELAX
15-5)
-3
535 (5wa)
555 15mi0)
SCBAR
SCLAX
-60
6.5
253 15m11
860 (5-55)
851 (5:13]
Resat
RELAX
20002
40003
50020
130300
320300
20:00
40300
500C0 100030
20020 40020 90:00
202000
$0023
Stapo
Figure 6: The training evidence lower bound on the MNIST training set (top) and the log variance of the gradient (bottom) 5 runs. Comparison with the REBAR and RELAX estimators.
Dataset
Model
Gumbel-softmax
CSB (S=1)
CSB (S=5)
CSB (S=10)
one layer
-113.46 + 0.59
-107.48 + 0.37
-107.24 0.31
-107.33 + 0.13
MNIST
Cat. VAE
-122.97 + 5.68
-103.49 + 0.73
-102.68 + 0.63
-101.78 + 0.88
one layer
-125.76 + 0.24
-122.49 + 0.80
-122.98 + 0.30
-122.98 + 0.21
Omniglot
Cat. VAE
-140.25 1.99
-130.20 + 0.74
-131.66 0.84
-131.63 + 1.05
Table Test likelihood for the categorical stochastic backpropagation (CSB) estimator, compared to the Gumbel-softmax estimator. We report the mean and standard deviation over 5 runs.
In this case, we adopt the same architectures as (Grathwohl et al. 2018). The categorical stochastic backpropagation estimator is compared to the Gumbel-softmax estimator (Maddison et al. 2016) Jang et al. 2016) using two models: a variational autoencoder and single layer belief network with categorical priors. In this case, we set the dimension of the hidden layer to d 20 and the number of modalities for each dimension to K 10.
All models are trained using the ADAM optimizer (Kingma & Ba 2014 using a standard learning rate a == 10 and batch size 100. We train the models for 500 epochs on the MNIST dataset and 100 epochs the Omniglot dataset, longer learning epochs leads to overfitting and lower performance on the test sets for all estimators and models. We perform iterations of training in all experiments and we report the mean and standard deviation of each performance metric considered.
For all models and estimators, we report the mean marginal test likelihood in tables I and 2 for both datasets. The test likelihood is estimated via importance sampling using 200 samples from the variational posterior. In all cases the stochastic backpropagation estimator, control variate free method outperforms the baselines. In the case of the one layer sigmoid belief network the BSB estimator exhibits an increase of performance of about 4 nats in the case of the MNIST dataset and 10 nats in the case of the Omniglot dataset. We also vary the number of samples used to estimate the expectation in the stochastic backpropagation rule S E {1, 5, 10). We notice that using a single sample estimate does not hurt performance and leads to faster training process.
We estimate the mean variance of the gradients w.r.t the parameters of the models using exponential moving averages of the first and second moments computed by the ADAM optimizer. The BSB estimator significantly outperforms the REBAR. RELAX estimators in terms of variance reduction with a difference of about 2 nats in the case of sigmoid belief networks, and nat in the case of the categorical variational autoencoder the mnist dataset, leading more stable training process as shown in figures 6 and 7
Finally, evaluate the computational overhead of the categorical stochastic backpropagation estimator compared the Gumbel-softmax estimator. We compare the two estimators in terms of execution time of one epoch of training. The comparison is done using GPU implementations
12

---

Under review as a conference paper at ICLR 2021
Tranting (43
-120
vertance uffth
Gumbel Softmax
Gumbel Softmax
Model
(S-1)
CSB
(S+5)
CSB
CSB
CSB (5=1]
CSB (S=1)
(5+5)
(5-5)
one Linear
layer
4.11 (6)
6.32 (6)
10.93
(6.83
CaL.
10000 20000 30000 40000
50000 60000
20000
40000
60000
21.91
stops
Figure 7: Training evidence lower bound and the log vari- ance of the gradient for the categorical VAE on the MNIST dataset.
Table 3: Execution time of one epoch of training on the mnist dataset per es- timator, per model.
NVIDIA GeForce RTX 2080 Ti GPU, where the stochastic backpropagation rule of equation equation 18] is vectorized, thus leveraging the parallel batch treatment of the GPU. As shown in table 3 the Gumbel-softmax method is faster than stochastic backpropagation (S 1) by difference of about 3 seconds per training epoch. This due to the forward passes performed to compute each of the terms in equation equation] [8 The variance reduction and the increase in performance outweigh
however the computational cost.
13


Reviews:
[{"review_id": "tMdzpnLfcot", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Update**\nThe errors in the paper were fixed, and the discussion was improved.\nIt is very rare to see a novel result as fundamental as the one presented in this paper, and I believe this puts it in the top 5% of accepted papers, so I have updated my score accordingly. \n\nI think the discussion and experimentation still has room for improvement, but I am not too bothered, as there do not appear to be any major errors remaining in the paper.\n\n**Summary**\nThe paper considers the task of estimating the gradient $\\frac{d}{d\\theta}\\mathbb{E}_{p_\\theta(z)}\\left[f(z)\\right]$.\nThis is a fundamental task relevant in all fields of machine learning, e.g. policy gradients, variational inference, or any other situation with stochastic computations.\n\nThe authors come up with a constructive method for deriving an estimator for this gradient for any distribution $p(z)$ based on Fourier analysis. The derived gradients include some known estimators, e.g. the Gaussian gradient identities, but also include some new estimators based on infinite series of higher order gradients of $f(z)$ for distributions such as Gamma or Laplace.\n\nThe analysis also works for Dirac delta distributions or discrete distributions.\n\nThey perform toy experiments to evaluate the new gradient estimators, and show that they work and seem to give better accuracy than previous pathwise gradients (but they use higher order gradients). For the infinite series based estimators, they truncated the length of the series at a certain depth (they tested up to 4th order and up to 8th order).\n\n**Strengths**\nGradient estimation is fundamental, and deriving new methods is a great contribution.\n\nThe paper didn't discuss this well, but what is particular about the method derived is the form of the estimator based on separating out the computation of the expected gradients of $f(z)$, and the weighting applied to it, i.e. all of the gradient estimators derived with the method have the form $\\sum_k a_k(\\theta)\\mathbb{E}_{p(z)}\\left[\\frac{d^k f(z)}{dz^k}\\right]$. What is neat about this, is that one only needs estimators for the expected gradients $\\frac{d^k f(z)}{dz^k}$, it is not necessary to know what the sampled $z$ values were. If you contrast this to some other gradient estimators, for example, the reparameterization gradient for the standard deviation parameter of a Gaussian distribution has the form $\\epsilon \\nabla_z f(z)$, where it is necessary to weight the separate values of $\\nabla_z f(z)$ with different multipliers, but in the current paper, they derive a method where the multiplier is the same for all samples $z$. Moreover, from the derivation it is clear that there is only one way to perform this decomposition for each distribution, and the method to perform the decomposition for any $p(z)$ is provided in this paper. This is a general fundamental result.\n\n**Weaknesses**\n\nThe discussion could be improved, e.g. the above points I mentioned were not explained.\n\nI believe Lemma 3 and Corollary 3.1 are incorrect, because they ignore the singularity at the \"kink\" of the relu. This issue does not affect the experiments though, because relus were not used in the experiments. (This Lemma was about making the expressions based on infinite series of higher order derivatives tractable in cases where the higher order derivatives are 0.) I explain this in more detail later.\n\nThe discussion around the \"discrete derivative\" for categorical distributions was unconvincing to me. It appears that the derivation does not require selecting a special $z^{\\*}$ choice, and the derivation goes through by just summing all $z$ without computing any difference at all.\n\nFrom a practical point of view, probably the method will not be immediately used.\n\n**Recommendation**\n\nI recommend accepting the paper, because the theoretical result is fundamental. They performed experiments showing that the new methods work on toy problems, which I think is sufficient. Any issues I found were minor, and could be improved with a bit of revision. Probably, I will further increase my score if they adequately revise the paper.\n\n**Questions**\n\nIn the experiments, what is the performance for truncation depth 1, 2, 3?\nI am interested in at what point the performance starts degrading. Currently\nthe experiments only show 4 and 8, and both give similar performance, so it\nis not clear whether, for example, 1 may also work or not.\n\nIn equation 6, $\\alpha$ is a multi index, not an integer. What does\n$(i\\omega)^{\\alpha}$ mean?\n\n**Additional comments, suggestions, clarifications**\nKink in the relu: For example, consider $f(z) = max(0, z)$ is a relu. Then consider \n$\\mathbb{E}_{p(z)}\\left[\\frac{d^2 f}{dz^2}\\right]$. Due to the effectively infinite second derivative at 0, I believe this expectation should be considered as $p(0)(\\frac{df(0+\\epsilon)}{dz} - \\frac{df(0-\\epsilon)}{dz}) = p(0)$, and I believe the values at the singularities will matter for the gradient estimator. As long as I have not misunderstood something, I would suggest to just remove the discussion about relus, and say that if the higher order derivatives disappear, it becomes tractable.\n\nIn general, I think the discussion would be greatly improved if you emphasize the structure of the gradient estimator based on separating out the weighting for the expected gradients $a$, and computing the expected gradient \n$\\mathbb{E}_{p(z)}\\left[\\frac{d^n f}{dz^n}\\right]$. And contrast this to the other existing gradient estimators, e.g. reparameterization or pathwise derivatives, which often apply a different weighting for derivatives at different $z$ values. It would be good to emphasize that what you have derived is not a general form for gradient estimators, it is a particular (fairly broad) family of gradient estimators among other techniques that are not described by your derivation. I also did not find the discussion about the Dirac delta or the discrete distributions particularly insightful, so I would suggest to spend less time emphasizing these points.\n\n\"Furthermore, we show that the classical deterministic\nbackproapagation rule is a special case of stochastic backpropagation\nwhere the distribution is a Dirac delta, bridging the domains of neural\nnetworks and probabilistic graphical models.\"\nI would not emphasize this point, as it is probably obvious to many\nresearchers. The Dirac delta can be acquired by letting $\\sigma \\to 0$\nfor a Gaussian distribution, which directly gives the result.\n\nIn the introduction, there are some spaces before the '?' signs, which should\nbe removed.\n\nI disagree with some of the discussion in the introduction. In particular\nthe questions posed are already answered to some extent:\n\n\"How to develop stochastic backpropagation rules similar to those of\n(Rezende et al., 2014) for a broader range of distributions?\"\nImplicit reparameterization gradients work for a broad range of\ndistributions. Moreover, the reparameterization based on the cdf of\na distribution always works as long as the cdf can be inverted (this is\njust a computational issue, rather than a theoretical one).\n\n\"What is the link between the discrete random variable case and the\ncontinuous case? And finally, what is the relation between stochastic\nbackpropagation and classical deterministic backpropagation?\"\nI wouldn't say the paper gives a definite answer to these\nquestions. It just provides another interpretation based on Fourier\ntransforms/characteristic functions, which is good, but I don't\nbelieve the new interpretation is better than previous ones; it is\njust different. I would tone down the discussion and just say that\nyou provide interpretations based on Fourier analysis.\n\nsommable -> summable\n\nPerhaps the multi-index notation could be clarified by 1-2 examples, and/or\nemphasized by putting in a definition block. I am not sure whether it will\nimprove it though.\n\nThe weighting function for the derivatives is outside the expectation,\ni.e. the weights do not depend on the sampled z position, and the z\nonly comes into play for computing the expectation of some gradients\nof f(z). I think this structure of the gradient estimator should be\nemphasized more.\n\nEquation 3 claims that the weights are unique, so there is only one\nway to construct the estimators that they have constructed. I think\nthis should be emphasized more. Also, instead of $\\exists!$ it may be\nbetter to just write \"there exists a unique\".\n\n$a_\\alpha$ is a bit difficult to read. Consider using different notation\nfor either $a$ or $\\alpha$.\n\n\" Putting everything together\": you have an extra space \" \" in the beginning\nof the sentence.\n\nIn the Theorem 1 statement , can you write out that $a$ are the\nTaylor weights of $\\nabla_\\theta\\log \\varphi_\\theta(\\omega)$?\n\n\"Plugging this expression to equation equation 5\" -> there's a duplicate\n\"equation\"\n\nThe \"(AUEB & Lazaro-Gredilla,2015)\" citation author names should be fixed\n(the default on google scholar is not good).\n\nThe discrete derivative explanation in equations 10 and 13 is not\nconvincing to me. Instead of pulling out the $\\varphi(z\\*)$ term in eq\n11, you could have just kept the summation across all $z$, and it\nwould have lead to the gradient estimator also summing all $f(z)$. It\nis not clear why the difference between $f(z)$ and $f(z\\*)$ is necessary\nor why it should be interpreted as a discrete derivative as the $z$ vector\nis not being perturbed.\n\nLaplace is reparameterizable. What is meant by the pathwise derivative\nin the experiments? The work from Jankowiak and Obermeyer defines a\nfamily of gradient estimators not a single one.\n\nFor the toy problems. Rather than showing only the learning curves,\nit would be good to also test that the expected gradients for the\nnew methods and old methods are the same at some particular parameter\nvalues (up to estimation accuracy, but you should be able to get the estimates accurate by repeating the computation many times and averaging). Also, for the experiments with truncation, probably what is more important than the variance is the gradient accuracy. So, it would be good to show both the bias and variance at different truncation depths. (Ideally, I envision a graph showing bias and variance of the gradient at a particular parameter value plotted against different truncation depth values.)\n\nRezende et al (2014) also mention the reparameterization of the variable\nunder stochastic backpropagation, and this gives a different gradient\nfor the covariance parameters of a Gaussian compared to the stochastic\nbackpropagation rule derived in the present paper. Hence, it does not\ngeneralize stochastic backpropagation as envisioned by Rezende. Instead,\nit is a separate method for deriving one particular type of stochastic\nbackpropagation rule for any distribution. I would suggest something\nlike Fourier stochastic backpropagation, characteristic stochastic\nbackpropagation, Fourier expectation gradients, etc.\n\nThe log characteristic function for Laplace should be\n$i\\omega\\mu - \\log(1 + b^2\\omega^2)$, you're missing the $\\mu$ in\nyour equation.\n\n\"Our approach, in contrast generalizes stochastic backpropagation as\npresented by (Rezende et al.,2014), where the derivative is explicitly\ntransported to the random variable\"\nNo, it doesn't generalize it as presented by Rezende. In Rezende's work,\nreparameterization is a subset of stochastic backpropagation (it is\nlisted under section 3 titled stochastic backpropagation). And\nreparameterization contains gradient estimators not derived by your\nmethod (e.g. the $\\frac{d}{d\\sigma}(\\cdot)$ gradient for a Gaussian).\nHence, it is wrong to say that your method generalizes\nstochastic backpropagation.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "m1EA9JHYyjt", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the present work, the authors present a general method for deriving stochastic back-propagation rules, using the link between Fourier transforms and the characteristic function associated to the random variable probability distribution and transferring the derivative directly to the random variable. They are thus able to encompass many well known back-propagation rules from the literature and derive new ones for different special cases. \nThe presented numerical experiments show how this method can be used to match state-of-art performance in simple models.\nThe authors also briefly discuss the bottlenecks associated to this method and propose some workarounds and rules of thumb (e.g., truncation of the series expansion entailed in the method) that seem to allow robust and efficient optimization.\nFinally, the authors highlight the fact that deterministic neural networks and usual back-propagation can also be framed by the same method, by simply considering a Dirac's delta probability distribution for the parameters.\nI think the paper is well written and that the presented framework nicely connects many ideas and methods developed in the literature in the past decades. It seems clear that methods based on the reparametrization trick will always be more viable for deep models, but they are based on ad hoc rules. The presented method is instead general and might be more effective in special cases where an early truncation of the series is justified. Personally, I would only put less accent on the link with deterministic back-propagation, which looks more like a sanity check than an important results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "F7mt02-eCg", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a general framework for deriving stochastic back-propagation rules for both discrete and continuous distributions. With the help of characteristic function and Fourier transform, this paper derives a general formulation to calculate the gradients of model parameters with respect to the random variables.\n\nThe main contribution of this paper is to derive two general formula of gradient computation with respect to continuous and discrete random variables. The technique used in this paper is standard, and the applications on different distributions can conclude similar results as previous research works. It also verifies the correctness of the proposed generalized stochastic back-propagation. In general, this is a good finding or connection with the help of characteristic function and Fourier transformation. However, the content of this paper seems not enough to be a conference paper, and it is more like a workshop paper. In addition, using Fourier transformation (https://arxiv.org/pdf/1808.03953.pdf) in such case is not new. Another weakness I want to mention is the experiment conducted in this paper is not solid. Bayesian Logistic Regression is a quite toy setting and used in a bunch of variational inference papers. This paper should compare with some baselines. Similarly, the MNIST and Omniglot are two popular and widely used datasets. It should be easy to find reproducible baselines.\n\nSome missing references:\n[1] https://papers.nips.cc/paper/5670-fast-second-order-stochastic-backpropagation-for-variational-inference.pdf\n[2] http://papers.nips.cc/paper/8325-provable-gradient-variance-guarantees-for-black-box-variational-inference.pdf", "extra_scores": {}, "extra_text": {}}, {"review_id": "DMEVKMp8Azt", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper derives a unified view for stochastic back-propagation for both continuous and discrete distributions. It uses the Taylor expansion of the characteristic function and then inverse Fourier transform to convert them infinite series of high order derivatives of the original function.\n\nIt is nice that the paper exposes several interesting connections between stochastic back-propagation in both continuous and discrete cases as well as deterministic back-propagation.\n\nHowever, it seems the approach is quite limited in its usage and it is not so clear what the advantage is. Even though the paper addresses some issues with higher order derivatives and approximating the infinite Taylor series with truncation, it seems it is only applicable to a few distributions.\n\nAlthough synthetic experiments show advantage in the proposed approach in some toy examples, it is not clear why the proposed approach is better than alternatives, in terms of lower variance or faster convergence.\n\nAlso it would be useful to fully specify one or two discrete examples and experiment with them in applications such as sparse feature selection etc.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The focus of the paper is stochastic backpropagation for both continuous and discrete random variables. By using standard results from Fourier analysis the authors rewrite the corresponding gradients in an infinite weighted sum form ((3) and (9)), extending the results of (Rezende et al. 2014) and (Fellows. et al., 2018). The efficiency of the approach is illustrated in 2 toy examples.

As summarized by the reviewers, the problem tackled is interesting. However, they also pointed out that the novelty of the approach is quite limited and its practical usefulness is not clear (it should by demonstrated against state-of-the-art baselines, on realistic benchmarks).
Decision: Reject