Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper derives a unified view for stochastic back-propagation for both continuous and discrete distributions. It uses the Taylor expansion of the characteristic function and then inverse Fourier transform to convert them infinite series of high order derivatives of the original function.

It is nice that the paper exposes several interesting connections between stochastic back-propagation in both continuous and discrete cases as well as deterministic back-propagation.

However, it seems the approach is quite limited in its usage and it is not so clear what the advantage is. Even though the paper addresses some issues with higher order derivatives and approximating the infinite Taylor series with truncation, it seems it is only applicable to a few distributions.

Although synthetic experiments show advantage in the proposed approach in some toy examples, it is not clear why the proposed approach is better than alternatives, in terms of lower variance or faster convergence.

Also it would be useful to fully specify one or two discrete examples and experiment with them in applications such as sparse feature selection etc.
review_title: Interesting approach with limitations

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a general framework for deriving stochastic back-propagation rules for both discrete and continuous distributions. With the help of characteristic function and Fourier transform, this paper derives a general formulation to calculate the gradients of model parameters with respect to the random variables.

The main contribution of this paper is to derive two general formula of gradient computation with respect to continuous and discrete random variables. The technique used in this paper is standard, and the applications on different distributions can conclude similar results as previous research works. It also verifies the correctness of the proposed generalized stochastic back-propagation. In general, this is a good finding or connection with the help of characteristic function and Fourier transformation. However, the content of this paper seems not enough to be a conference paper, and it is more like a workshop paper. In addition, using Fourier transformation (https://arxiv.org/pdf/1808.03953.pdf) in such case is not new. Another weakness I want to mention is the experiment conducted in this paper is not solid. Bayesian Logistic Regression is a quite toy setting and used in a bunch of variational inference papers. This paper should compare with some baselines. Similarly, the MNIST and Omniglot are two popular and widely used datasets. It should be easy to find reproducible baselines.

Some missing references:
[1] https://papers.nips.cc/paper/5670-fast-second-order-stochastic-backpropagation-for-variational-inference.pdf
[2] http://papers.nips.cc/paper/8325-provable-gradient-variance-guarantees-for-black-box-variational-inference.pdf
review_title: A good finding but not enough for publishing as a conference paper

final_score: rating: 6: Marginally above acceptance threshold
review_content: In the present work, the authors present a general method for deriving stochastic back-propagation rules, using the link between Fourier transforms and the characteristic function associated to the random variable probability distribution and transferring the derivative directly to the random variable. They are thus able to encompass many well known back-propagation rules from the literature and derive new ones for different special cases. 
The presented numerical experiments show how this method can be used to match state-of-art performance in simple models.
The authors also briefly discuss the bottlenecks associated to this method and propose some workarounds and rules of thumb (e.g., truncation of the series expansion entailed in the method) that seem to allow robust and efficient optimization.
Finally, the authors highlight the fact that deterministic neural networks and usual back-propagation can also be framed by the same method, by simply considering a Dirac's delta probability distribution for the parameters.
I think the paper is well written and that the presented framework nicely connects many ideas and methods developed in the literature in the past decades. It seems clear that methods based on the reparametrization trick will always be more viable for deep models, but they are based on ad hoc rules. The presented method is instead general and might be more effective in special cases where an early truncation of the series is justified. Personally, I would only put less accent on the link with deterministic back-propagation, which looks more like a sanity check than an important results.
review_title: The authors propose a general framework for deriving stochastic back-propagation rules, reconnecting with known rules and proposing a general method for deriving new ones. 

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: **Update**
The errors in the paper were fixed, and the discussion was improved.
It is very rare to see a novel result as fundamental as the one presented in this paper, and I believe this puts it in the top 5% of accepted papers, so I have updated my score accordingly. 

I think the discussion and experimentation still has room for improvement, but I am not too bothered, as there do not appear to be any major errors remaining in the paper.

**Summary**
The paper considers the task of estimating the gradient $\frac{d}{d\theta}\mathbb{E}_{p_\theta(z)}\left[f(z)\right]$.
This is a fundamental task relevant in all fields of machine learning, e.g. policy gradients, variational inference, or any other situation with stochastic computations.

The authors come up with a constructive method for deriving an estimator for this gradient for any distribution $p(z)$ based on Fourier analysis. The derived gradients include some known estimators, e.g. the Gaussian gradient identities, but also include some new estimators based on infinite series of higher order gradients of $f(z)$ for distributions such as Gamma or Laplace.

The analysis also works for Dirac delta distributions or discrete distributions.

They perform toy experiments to evaluate the new gradient estimators, and show that they work and seem to give better accuracy than previous pathwise gradients (but they use higher order gradients). For the infinite series based estimators, they truncated the length of the series at a certain depth (they tested up to 4th order and up to 8th order).

**Strengths**
Gradient estimation is fundamental, and deriving new methods is a great contribution.

The paper didn't discuss this well, but what is particular about the method derived is the form of the estimator based on separating out the computation of the expected gradients of $f(z)$, and the weighting applied to it, i.e. all of the gradient estimators derived with the method have the form $\sum_k a_k(\theta)\mathbb{E}_{p(z)}\left[\frac{d^k f(z)}{dz^k}\right]$. What is neat about this, is that one only needs estimators for the expected gradients $\frac{d^k f(z)}{dz^k}$, it is not necessary to know what the sampled $z$ values were. If you contrast this to some other gradient estimators, for example, the reparameterization gradient for the standard deviation parameter of a Gaussian distribution has the form $\epsilon \nabla_z f(z)$, where it is necessary to weight the separate values of $\nabla_z f(z)$ with different multipliers, but in the current paper, they derive a method where the multiplier is the same for all samples $z$. Moreover, from the derivation it is clear that there is only one way to perform this decomposition for each distribution, and the method to perform the decomposition for any $p(z)$ is provided in this paper. This is a general fundamental result.

**Weaknesses**

The discussion could be improved, e.g. the above points I mentioned were not explained.

I believe Lemma 3 and Corollary 3.1 are incorrect, because they ignore the singularity at the "kink" of the relu. This issue does not affect the experiments though, because relus were not used in the experiments. (This Lemma was about making the expressions based on infinite series of higher order derivatives tractable in cases where the higher order derivatives are 0.) I explain this in more detail later.

The discussion around the "discrete derivative" for categorical distributions was unconvincing to me. It appears that the derivation does not require selecting a special $z^{\*}$ choice, and the derivation goes through by just summing all $z$ without computing any difference at all.

From a practical point of view, probably the method will not be immediately used.

**Recommendation**

I recommend accepting the paper, because the theoretical result is fundamental. They performed experiments showing that the new methods work on toy problems, which I think is sufficient. Any issues I found were minor, and could be improved with a bit of revision. Probably, I will further increase my score if they adequately revise the paper.

**Questions**

In the experiments, what is the performance for truncation depth 1, 2, 3?
I am interested in at what point the performance starts degrading. Currently
the experiments only show 4 and 8, and both give similar performance, so it
is not clear whether, for example, 1 may also work or not.

In equation 6, $\alpha$ is a multi index, not an integer. What does
$(i\omega)^{\alpha}$ mean?

**Additional comments, suggestions, clarifications**
Kink in the relu: For example, consider $f(z) = max(0, z)$ is a relu. Then consider 
$\mathbb{E}_{p(z)}\left[\frac{d^2 f}{dz^2}\right]$. Due to the effectively infinite second derivative at 0, I believe this expectation should be considered as $p(0)(\frac{df(0+\epsilon)}{dz} - \frac{df(0-\epsilon)}{dz}) = p(0)$, and I believe the values at the singularities will matter for the gradient estimator. As long as I have not misunderstood something, I would suggest to just remove the discussion about relus, and say that if the higher order derivatives disappear, it becomes tractable.

In general, I think the discussion would be greatly improved if you emphasize the structure of the gradient estimator based on separating out the weighting for the expected gradients $a$, and computing the expected gradient 
$\mathbb{E}_{p(z)}\left[\frac{d^n f}{dz^n}\right]$. And contrast this to the other existing gradient estimators, e.g. reparameterization or pathwise derivatives, which often apply a different weighting for derivatives at different $z$ values. It would be good to emphasize that what you have derived is not a general form for gradient estimators, it is a particular (fairly broad) family of gradient estimators among other techniques that are not described by your derivation. I also did not find the discussion about the Dirac delta or the discrete distributions particularly insightful, so I would suggest to spend less time emphasizing these points.

"Furthermore, we show that the classical deterministic
backproapagation rule is a special case of stochastic backpropagation
where the distribution is a Dirac delta, bridging the domains of neural
networks and probabilistic graphical models."
I would not emphasize this point, as it is probably obvious to many
researchers. The Dirac delta can be acquired by letting $\sigma \to 0$
for a Gaussian distribution, which directly gives the result.

In the introduction, there are some spaces before the '?' signs, which should
be removed.

I disagree with some of the discussion in the introduction. In particular
the questions posed are already answered to some extent:

"How to develop stochastic backpropagation rules similar to those of
(Rezende et al., 2014) for a broader range of distributions?"
Implicit reparameterization gradients work for a broad range of
distributions. Moreover, the reparameterization based on the cdf of
a distribution always works as long as the cdf can be inverted (this is
just a computational issue, rather than a theoretical one).

"What is the link between the discrete random variable case and the
continuous case? And finally, what is the relation between stochastic
backpropagation and classical deterministic backpropagation?"
I wouldn't say the paper gives a definite answer to these
questions. It just provides another interpretation based on Fourier
transforms/characteristic functions, which is good, but I don't
believe the new interpretation is better than previous ones; it is
just different. I would tone down the discussion and just say that
you provide interpretations based on Fourier analysis.

sommable -> summable

Perhaps the multi-index notation could be clarified by 1-2 examples, and/or
emphasized by putting in a definition block. I am not sure whether it will
improve it though.

The weighting function for the derivatives is outside the expectation,
i.e. the weights do not depend on the sampled z position, and the z
only comes into play for computing the expectation of some gradients
of f(z). I think this structure of the gradient estimator should be
emphasized more.

Equation 3 claims that the weights are unique, so there is only one
way to construct the estimators that they have constructed. I think
this should be emphasized more. Also, instead of $\exists!$ it may be
better to just write "there exists a unique".

$a_\alpha$ is a bit difficult to read. Consider using different notation
for either $a$ or $\alpha$.

" Putting everything together": you have an extra space " " in the beginning
of the sentence.

In the Theorem 1 statement , can you write out that $a$ are the
Taylor weights of $\nabla_\theta\log \varphi_\theta(\omega)$?

"Plugging this expression to equation equation 5" -> there's a duplicate
"equation"

The "(AUEB & Lazaro-Gredilla,2015)" citation author names should be fixed
(the default on google scholar is not good).

The discrete derivative explanation in equations 10 and 13 is not
convincing to me. Instead of pulling out the $\varphi(z\*)$ term in eq
11, you could have just kept the summation across all $z$, and it
would have lead to the gradient estimator also summing all $f(z)$. It
is not clear why the difference between $f(z)$ and $f(z\*)$ is necessary
or why it should be interpreted as a discrete derivative as the $z$ vector
is not being perturbed.

Laplace is reparameterizable. What is meant by the pathwise derivative
in the experiments? The work from Jankowiak and Obermeyer defines a
family of gradient estimators not a single one.

For the toy problems. Rather than showing only the learning curves,
it would be good to also test that the expected gradients for the
new methods and old methods are the same at some particular parameter
values (up to estimation accuracy, but you should be able to get the estimates accurate by repeating the computation many times and averaging). Also, for the experiments with truncation, probably what is more important than the variance is the gradient accuracy. So, it would be good to show both the bias and variance at different truncation depths. (Ideally, I envision a graph showing bias and variance of the gradient at a particular parameter value plotted against different truncation depth values.)

Rezende et al (2014) also mention the reparameterization of the variable
under stochastic backpropagation, and this gives a different gradient
for the covariance parameters of a Gaussian compared to the stochastic
backpropagation rule derived in the present paper. Hence, it does not
generalize stochastic backpropagation as envisioned by Rezende. Instead,
it is a separate method for deriving one particular type of stochastic
backpropagation rule for any distribution. I would suggest something
like Fourier stochastic backpropagation, characteristic stochastic
backpropagation, Fourier expectation gradients, etc.

The log characteristic function for Laplace should be
$i\omega\mu - \log(1 + b^2\omega^2)$, you're missing the $\mu$ in
your equation.

"Our approach, in contrast generalizes stochastic backpropagation as
presented by (Rezende et al.,2014), where the derivative is explicitly
transported to the random variable"
No, it doesn't generalize it as presented by Rezende. In Rezende's work,
reparameterization is a subset of stochastic backpropagation (it is
listed under section 3 titled stochastic backpropagation). And
reparameterization contains gradient estimators not derived by your
method (e.g. the $\frac{d}{d\sigma}(\cdot)$ gradient for a Gaussian).
Hence, it is wrong to say that your method generalizes
stochastic backpropagation.

review_title: Major theoretical result in gradient estimation research. Possibly small errors, and discussion could be improved. Update: small errors were fixed, discussion was improved

Meta-review: The focus of the paper is stochastic backpropagation for both continuous and discrete random variables. By using standard results from Fourier analysis the authors rewrite the corresponding gradients in an infinite weighted sum form ((3) and (9)), extending the results of (Rezende et al. 2014) and (Fellows. et al., 2018). The efficiency of the approach is illustrated in 2 toy examples.

As summarized by the reviewers, the problem tackled is interesting. However, they also pointed out that the novelty of the approach is quite limited and its practical usefulness is not clear (it should by demonstrated against state-of-the-art baselines, on realistic benchmarks).
Decision: Reject