Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper considers the problem of choosing the parameters of distribution to maximize an expectation over that distribution. This setting has attracted a huge interest in ML communities (that is related to learning policy in RL as well as variational inference with hidden variables). The paper provides a framework for such optimization, by interestingly combining three standard ways. 

Given Tucker et al, its contribution is somehow incremental, but I think it is an interesting idea to use neural networks for control variate to handle the case where f is unknown.  

The main issue of this paper seems to be in limited experimental results; they only showed three quite simple experiments (I guess they need to focus one setting; RL or VAE). Moreover, it would be good to actually show if the variation of g_hat is much smaller than other standard methods.
   
There is missing reference at page 8.


review_title: I think the paper provides interesting idea for optimizing black-box functions of random variables. 

final_score: rating: 7: Good paper, accept
review_content: This paper suggests a new approach to performing gradient descent for blackbox optimization or training discrete latent variable models. The paper gives a very clear account of existing gradient estimators and finds a way to combine them so as to construct and optimize a differentiable surrogate function. The resulting new gradient estimator is then studied both theoretically and empirically. The empirical study shows the benefits of the new estimator for training discrete variational autoencoders and for performing deep reinforcement learning.

To me, the main strengths of the paper is the very clear account of existing gradient estimators (among other things it helped me understand obscurities of the Q-prop paper) and a nice conceptual idea. The empirical study itself is more limited and the paper suffers from a few mistakes and missing information, but to me the good points are enough to warrant publication of the paper in a good conference like ICLR.

Below are my comments for the authors.

---------------------------------
General, conceptual comments:

When reading (6), it is clear that the framework performs regression of $c_\phi$ towards the unknown $f$ simultaneously with optimization over $c_\phi$.
Taking this perspective, I would be glad to see how the regression part performs with respect to standard least square regression,
i.e. just using $||f(b)-c_\phi(b)||^2$ as loss function. You may compare the speed of convergence of $c_\phi$ towards $f$ using (6) and the least squared error.
You may also investigate the role of this regression part into the global g_LAX optimization by studying the evolution of the components of (6).

Related to the above comment, in Algo. 1, you mention "f(.)" as given to the algo. Actually, the algo does not know f itself, otherwise it would not be blackbox optimization. So you may mean different things. In a batch setting, you may give a batch of [x,f(x) (,cost(x)?)] points to the algo. You more probably mean here that you have an "oracle" that, given some x, tells you f(x) on demand. But the way you are sampling x is not specified clearly.

This becomes more striking when you move to reinforcement learning problems, which is my main interest. The RL algorithm itself is not much specified. Does it use a replay buffer (probably not)? Is it on-policy or off-policy (probably on-policy)? What about the exploration policy? I want to know more... Probably you just replace (10) with (11) in A2C, but this is not clearly specified.

In Section 4, can you explain why, in the RL case, you must introduce stochasticity to the inputs? Is this related to the exploration issue (see above)?

Last sentence of conclusion: you are too allusive about the relationship between your learned control variate and the Q-function. I don't get it, and I want to know more...

-----------------------------------
Local comments:

Backpropagation through the void: I don't understand why this title. I'm not a native english speaker, I'm probably missing a reference to something, I would be glad to get it.

Figure 1 right. Caption states variance, but it is log variance. Why does it oscillate so much with RELAX?

Beginning of 3.1: you may state more clearly that optimizing $c_\phi$ the way you do it will also "minimize" the variance, and explain better why ("we require the gradient of the variance of our gradient estimator"...). It took me a while to get it.

In 3.1.1 a weighting based on $\d/\d\theta log p(b)$ => shouldn't you write $... log p(b|\theta)$ as before?

Figure 2 is mentioned in p.3, it should appear much sooner than p6.

In Figure 2, there is nothing about the REINFORCE PART. Why?

In 3.4 you alternate sums over an infinite horizon and sums over T time steps. You should stick to the T horizon case, as you mention the case T=1 later.

p6 Related work

The link to the work of Salimans 2017 is far from obvious, I would be glad to know more...

Q-prop (Haarnoja et al.,2017): this is not the adequate reference to Q-prop, it should be (Gu et al. 2016), you have it correct later ;)

Figure 3: why do you stop after so few epochs? I wondered how expensive is the computation of your estimator, but since in the RL case you go up to 50 millions (or 4 millions?), it's probably not the issue. I would be glad to see another horizontal lowest validation error for your RELAX estimator (so you need to run more epochs).
"ELBO" should be explained here (it is only explained in the appendices).

6.2, Table 1: Best obtained training objective: what does this mean? Should it be small or large? You need to explain better. How much is the modest improvement (rather give relative improvement in the text?)? To me, you should not defer Table 3 to an appendix (nor Table 4).

Figure 4: Any idea why A2C oscillates so much on inverted pendulum? Any idea why variance starts to decrease after 500 episodes using RELAX? Isn't related to the combination of regression and optimization, as suggested above?

About Double Inverted Pendulum, Appendix E3 mentions 50 million frames, but the figure shows 4 millions steps. Where is the truth?

Why do you give steps for the reward, and episodes for log-variance? The caption mentions "variance (log-scale)", but saying "log-variance" would be more adequate.

p9: the optimal control variate: what is this exactly? How do you compare a control variate over another? This may be explained in Section 2.

GAE (Kimura, 2000). I'm glad you refer to former work (there is a very annoying tendency those days to refer only to very recent papers from a small set of people who do not correctly refer themselves to previous work), but you may nevertheless refer to John Schulman's paper about GAEs anyways... ;)

Appendix E.1 could be reorganized, with a common hat and then E.1.1 for one layer model(s?) and E.1.2 for the two layer model(s?)

A sensitivity analysis wrt to your hyper-parameters would be welcome, this is true for all empirical studies.

In E2, is the output layer linear? You just say it is not ReLU...

The networks used in E2 are very small (a standard would be 300 and 400 neurons in hidden layers). Do you have a constraint on this?

"As our control variate does not have the same interpretation as the value function of A2C, it was not directly clear how to add reward bootstrapping and other variance reduction techniques common in RL into our model. We leave the task of incorporating these and other variance reduction techniques to future work."
First, this is important, so if this is true I would move this to the main text (not in appendix).
But also, it seems to me that the first sentence of E3 contradicts this, so where is the truth?

{0.01,0.003,0.001} I don't believe you just tried these values. Most probably, you played with other values before deciding to perform grid search on these, right?
The same for 25 in E3.

Globally, you experimental part is rather weak, we would expect a stronger methodology, more experiments also with more difficult benchmarks (half-cheetah and the whole gym zoo ;)), more detailed analyses of the results, but to me the value of your paper is more didactical and conceptual than experimental, which I really appreciate, so I will support your paper despite these weaknesses.

Good luck! :)

---------------------------------------
Typos:

p5
monte-carlo => Monte(-)Carlo (no - later...)
taylor => Taylor
you should always capitalize Section, equation, table, figure, appendix, ...

gradient decent => descent (twice)

p11: probabalistic

p15 ELU(Djork-... => missing space


review_title: A good conceptual contribution with a clear background, a not so good empirical study, but I like it much

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces LAX/RELAX, a method to reduce the variance of the REINFORCE gradient estimator. The method builds on and is directly inspired by REBAR. Similarly to REBAR, RELAX is an unbiased estimator, and the idea is to introduce a control variate that leverages the reparameterization gradient. In contrast to REBAR, RELAX learns a free-from control variate, which allows for low-variance gradient estimates for both discrete and continuous random variables. The method is evaluated on a toy experiment, as well as the discrete VAE and reinforcement learning. It effectively reduces the variance of state-of-the-art methods (namely, REBAR and actor-critic).

Overall, I enjoyed reading the paper. I think it is a neat idea that can be of interest for researchers in the field. The paper is clearly explained, and I found the experiments convincing. I have minor comments only.

+ Is there a good way to initialize c_phi prior to optimization? Given that c_phi must be a proxy for f(), maybe you can take advantage of this observation to find a good initialization for phi?

+ I was confused with the Bernoulli example in Appendix B. Consider the case theta=0.5. Then, b=H(z) takes value 1 if z>0, and 0 otherwise. Thus, p(z|b,theta) should assign mass zero to values z>0 when b=0, which does not seem to be the case with the proposed sampling scheme in page 11, since v*theta=0.5*v, which gives values in [0,0.5]. And similarly for the case b=1.

+ Why is the method called LAX? What does it stand for?

+ In Section 3.3, it is unclear to me why rho!=phi. Given that c_phi(z)=f(sigma_lambda(z))+r_rho(z), with lambda being a temperature parameter, why isn't rho renamed as phi? (the first term doesn't seem to have any parameters). In general, this section was a little bit unclear if you are not familiar with the REBAR method; consider adding more details.

+ Consider adding a brief review of the REBAR estimator in the Background section for those readers who are less familiar with this approach.

+ In the abstract, consider adding two of the main ideas that the estimator relies on: control variates and reparameterization gradients. This would probably be more clear than "based on gradients of a learned function."

+ In the first paragraph of Section 3, the sentence "f is not differentiable or not computable" may be misleading, because it is unclear what "not computable" means (one may think that it cannot be evaluated). Consider replacing with "not analytically computable."

+ In Section 3.3, it reads "differentiable function of discrete random variables," which does not make sense.

+ Before Eq. 11, it reads "where epsilon_t does not depend on theta". I think it should be the distribution over epsilon_t what doesn't depend on theta.

+ In Section 6.1, it was unclear to me why t=.499 is a more challenging setting.

+ The header of Section 6.3.1 should be removed, as Section 6.3 is short.

+ In Section 6.3.1, there is a broken reference to a figure.

+ Please avoid contractions (doesn't, we'll, it's, etc.)

+ There were some other typos; please read carefully the paper and double-check the writing. In particular, I found some missing commas, some proper nouns that are not capitalized in Section 5, and others (e.g., "an learned," "gradient decent").
review_title: Good paper, convincing experiments, writing is clear but there are quite a few typos

Meta-review: This is an interesting and well-written paper introducing two unbiased gradient estimators for optimizing expectations of black box functions. LAX can handle functions of both continuous and discrete random variables, while RELAX is specialized to functions of discrete variables and can be seen as a version of the recently introduced REBAR with its concrete-relaxation-based control variate replaced by (or augmented with) a free-form function. The experimental section of the paper is adequate but not particularly strong. If Q-prop is the most similar existing RL approach, as is state in the paper, why not include it as a baseline? It would also be good to see how RELAX performs at optimizing discrete VAEs using just the free-form control variate (instead of combining it with the REBAR control variate).
Decision: Accept (Poster)