Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposed a class of control variate methods based on Stein's identity. Stein's identity has been widely used in classical statistics and recently in statistical machine learning literature. Nevertheless, applying Stein's identity to estimating policy gradient is a novel approach in reinforcement learning community. To me, this approach is the right way of constructing control variates for estimating policy gradient. The authors also did a good job in connecting with existing works and gave concrete examples for Gaussian policies. The experimental results also look promising.

It would be nice to include some theoretical analyses like under what conditions, the proposed method can achieve smaller sample complexity than existing works.    

Overall this is a strong paper and I recommend to accept.
 



review_title: Nice Application of Stein's identity

final_score: rating: 7: Good paper, accept
review_content: In this work, the authors suggest the use of control variate schemes for estimating gradient values, within a reinforcement learning  framework. The authors also introduce a specific control variate technique based on the so-called Stein’s identity. The paper is interesting and well-written.

I have some question and some consideration that can be useful for improving the appealing of the paper.

- I believe that different Monte Carlo (or Quasi-Monte Carlo) strategies can be applied in order to estimate the integral (expected value) in Eq. (1), as also suggested in this work. Are there other alternatives in the literature? Please, please discuss and cite some papers if required.  

- I suggest to divide Section 3.1 in two subsections. The first one introducing Stein’s identity and the related comments that you need, and a second one, starting after Theorem 3.1, with title “Stein Control Variate”.

-  Please also discuss the relationships, connections, and possible applications of your technique to other algorithms used in Bayesian optimization, active learning and/or sequential learning, for instance as

M. U. Gutmann and J. Corander, “Bayesian optimization for likelihood-free inference of simulator-based statistical mod- els,” Journal of Machine Learning Research, vol. 16, pp. 4256– 4302, 2015. 

G. da Silva Ferreira and D. Gamerman, “Optimal design in geostatistics under preferential sampling,” Bayesian Analysis, vol. 10, no. 3, pp. 711–735, 2015. 

L. Martino, J. Vicent, G. Camps-Valls, "Automatic Emulator and Optimized Look-up Table Generation for Radiative Transfer Models", IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2017.

-  Please also discuss the dependence of your algorithm with respect to the starting baseline function \phi_0.
review_title: Nice work; some suggestions in order to improve the paper are provided.

final_score: rating: 7: Good paper, accept
review_content: The paper proposes action-dependent baselines for reducing variance in policy gradient, through the derivation based on Stein’s identity and control functionals. The method relates closely to prior work on action-dependent baselines, but explores in particular on-policy fitting and a few other design choices that empirically improve the performance. 

A criticism of the paper is that it does not require Stein’s identity/control functionals literature to derive Eq. 8, since it can be derived similarly to linear control variate and it has also previously been discussed in IPG [Gu et. al., 2017] as reparameterizable control variate. The derivation through Stein’s identity does not seem to provide additional insights/algorithm designs beyond direct derivation through reparameterization trick.

The empirical results appear promising, and in particular in comparison with Q-Prop, which fits Q-function using off-policy TD learning. However, the discussion on the causes of the difference should be elaborated much more, as it appears there are substantial differences besides on-policy/off-policy fitting of the Q, such as:

-FitLinear fits linear Q (through parameterization based on linearization of Q) using on-policy learning, rather than fitting nonlinear Q and then at application time linearize around the mean action. A closer comparison would be to use same locally linear Q function for off-policy learning in Q-Prop.

-The use of on-policy fitted value baseline within Q-function parameterization during on-policy fitting is nice. Similar comparison should be done with off-policy fitting in Q-Prop.

I wonder if on-policy fitting of Q can be elaborated more. Specifically, on-policy fitting of V seems to require a few design details to have best performance [GAE, Schulman et. al., 2016]: fitting on previous batch instead of current batch to avoid overfitting  (this is expected for your method as well, since by fitting to current batch the control variate then depends nontrivially on samples that are being applied), and possible use of trust-region regularization to prevent V from changing too much across iterations. 

The paper presents promising results with direct on-policy fitting of action-dependent baseline, which is promising since it does not require long training iterations as in off-policy fitting in Q-Prop. As discussed above, it is encouraged to elaborate other potential causes that led to performance differences. The experimental results are presented well for a range of Mujoco tasks. 

Pros:

-Simple, effective method that appears readily available to be incorporated to any on-policy PG methods without significantly increase in computational time

-Good empirical evaluation

Cons:

-The name Stein control variate seems misleading since the algorithm/method does not rely on derivation through Stein’s identity etc. and does not inherit novel insights due to this derivation.

review_title: Good empirical study of modifications to action-dependent baselines

Meta-review: Thank you for submitting you paper to ICLR. The reviewers agree that the paper’s development of action-dependent baselines for reducing variance in policy gradient is a strong contribution and that the use of Stein's identity to provide a principled way to think about control variates is sensible. The revision clarified an number of the reviewers’ questions and the resulting paper is suitable for publication in ICLR.
Decision: Accept (Poster)