Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper considers the problem of model-based policy search. The authors 
consider the use of Bayesian Neural Networks to learn a model of the environment
and advocate for the $\alpha$-divergence minimization rather than the more usual 
variational Bayes. 

The ability of alpha-divergence to capture bi-modality however 
comes at a price and most of the paper is devoted to finding tractable approximations. 
The authors therefore use the approach of Hernandez-Lobato
et al. (2016) as proxy to the alpha-divergence . 

The environment/system dynamics is clearly defined as a well as the policy parametrization 
(section 3) and would constitute a useful reference point for other researchers. 
Simulated roll-outs, using the learned model, then provide samples of the expected 
return. Since a model of the environment is available, stochastic gradient descent 
can be performed in the usual way, without policy gradient estimators, via automatic 
differentiation tools. 

The experiments demonstrate that alpha-divergence is capable of capturing multi-model 
structure which competing methods (variational Bayes and GP) would otherwise
struggle with. The proposed approach also compares favorably in a real-world
batch setting.

The paper is well-written, technically rich and combines many recent tools 
into a coherent algorithm. However, the repeated use of approximations to original 
quantities seems to somehow defeat the benefits of the original problem formulation. 
The scalability and computational effectiveness of this approach is also questionable 
and I am uncertain if many problem would warrant such complexity in their solution. 
As with other Bayesian methods, the proposed approach would probably shine in low-samples 
regime and in this case might be preferable to other methods in the same class (VB, GP).


review_title: Clear problem formulation, scalability questionable 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a novel way of using Bayesian NNs for policy search in stochastic dynamical systems. Specifically, the authors minimize alpha-divergence with alpha=0.5 as opposed to standard VB. The authors claim that their method is the first model-based system to solve a 20 year old benchmark problem; I'm not very familiar with this literature, so it's difficult for me to assess this claim.

The paper seems technically sound. I feel the writing could be improved. The notation in sections 2-3 feels a bit dense and there are a lot of terminology / approximations introduced, which makes it hard to follow. The writing could be better structured to distinguish between novel contributions vs review of prior work. If I understand section 2.3 correctly, it's mostly a review of black box alpha divergence minimization. If so, it would probably make sense to move this to the appendix. 

There was a paper at NIPS 2016 showing promising results using SGHMC for Bayesian optimization: "Bayesian optimization with robust Bayesian neural networks" by Springenberg et al. Could you comment on applicability of stochastic gradient MCMC (SGLD / SGHMC) for your setup?

Can you comment on the computational complexity of the different approaches?

Section 4.2.1: why can't you use the original data? in what sense is it fair to simulate data using another neural network? can you evaluate PSO-P on this problem?
review_title: Policy search using Bayesian NNs

final_score: rating: 7: Good paper, accept
review_content: This paper introduces an approach for model-based control of stochastic dynamical systems with policy search, based on (1) learning the stochastic dynamics of the underlying system with a Bayesian deep neural network (BNN) that allows some of its inputs to be stochastic, and (2) a policy optimization method based on simulated rollouts from the learned dynamics. BNN training is carried out using \alpha-divergence minimization, the specific form of which was introduced in previous work by the authors. Validation and comparison of the approach is undertaken on a simulated domain, as well as real-world scenarios. 

The paper is tightly written, and easy to follow. Its approach to fitting Bayesian neural networks with \alpha divergence is interesting and appears novel in this context. The resulting application to model-based control appears to have significant practical impact, particularly in light of the explainability that a system model can bring to specific decisions made by the policy. As such, I think that the paper brings a valuable contribution to the literature.

That said, I have a few questions and suggestions:

1) In section 2.2, it should be explained how the random z_n input is used by the neural network: is it just concatenated to the other inputs and used as-is, or is there a special treatment?

2) Moreover, much case is made for the need to have stochastic inputs, but only a scalar input seems to be provided throughout. Is this enough? How computationally difficult would providing stochastic inputs of higher dimensionality be?

3) How important is the normality assumption in z_n? How is the variance \gamma established?

4) It is mentioned that the hidden layers of the neural network are made of rectifiers, but no further utilization of this fact is made in the paper. Is this assumption somehow important in the optimization of the alpha-divergence (beyond what we know about rectifiers to mitigate the vanishing gradient problem) ?

5) Equation (3), denominator \mathbf{y} should be \mathbf{Y} ?

6) Section 2.3: it would be helpful to have an overview or discussion of the computational complexity of training BNNs, to understand whether and when they can practicably be used.

7) Between eq (12) and (13), a citation to the statement of the time embedding theorem would be helpful, as well as an indication of how the embedding dimension should be chosen.

8) Figure 1: the subplots should have the letters by which they are referenced in the text on p. 7.

9) In section 4.2.1, it is not clear if the gas turbine data is publicly available, and if so where. In addition more details should be provided, such as the dimensionality of the variables E_t, N_t and A_t.

10) Perhaps the comparisons with Gaussian processes should include variants that support stochastic inputs, such as Girard et al. (2003), to provide some of the same modelling capabilities as what’s made use of here. At least, this strand of work should be mentioned in Section 5.


References:

Girard, A., Rasmussen, C. E., Quiñonero Candela, J., & Murray Smith, R. (2003). Gaussian process priors with uncertain inputs-application to multiple-step ahead time series forecasting. Advances in Neural Information Processing Systems, 545-552.

review_title: Useful contribution to model-based policy search with Bayesian neural networks

final_score: rating: null
review_content: I didn't have any pre-review question. Have posted my review titled "Policy search using Bayesian NNs" now.
review_title: no pre-review question

final_score: rating: null
review_content: 1. [Related work] "[...] popular modes for transition functions [...]". It might be relevant to mention the work on Predictive State Representation which played an important role in RL in the early 2000s and subsequently applied to real world problems with Boots' TPSRs (2011). 

2. The claim that the proposed method allows for "previously impossible" problems to be learned might have to backed up with more extensive experimental comparaisons. I'm thinking for example of Guided Policy Search, which also builds a model, and seems to be rather sample efficient. If the goal the paper is to improve within the class of Bayesian methods (which on its own, is already good !), then it should be stated more clearly. To make a fully convincing point, I would also like to see wall clock times and number of samples in table 3. 

3. I've been re-reading your explanation on why $\epsilon_n$ is needed and I'm still not 100% sure that I fully understand. From what I understand:  $z_n$ is really part of your  problem/system/environment while $\epsilon_n$ has to do with your choice of function approximator (BNN), on the side of the agent.  Since you already estimate $z_n$, why can't you model the entire stochastic effects through it ? A lack of representational power ? Couldn't you reparametrize your model and push everything in $z_n$ ?

4. Line 14 of Algorithm 1: [UNFOLD(s_n)], do you assume that you can restart your system in the same state ?




review_title: Related work, model, assumptions

Meta-review: Despite it's initial emphasis on policy search, this paper is really about a learning method for Bayesian neural networks, which it then uses in a policy search setting. Specifically, the authors advocate modeling a stochastic system using a BNN trained to minimize alpha-divergence with alpha=0.5 (this involves a great deal of approximation to make computationally tractable). They then use this in the policy search setting.
 
 The paper is quite clear, and proposes a nice approach to learning BNNs. The algorithmic impact honestly seems fairly minor (the idea of using different alpha-divergences instead of KL divergence has been considered many times in the content of general variational approximations), but combining this with the policy search setting, and reasonable examples of industrial control, together these all make this a fairly strong paper.
 
 Pros:
 + Nice derivation of alternative variational formulation (I'll still call it variational even though it uses alpha-divergence with alpha=0.5)
 + Good integration into policy search setting
 + Nice application to industrial control systems
 
 Cons:
 - Advance from the algorithmic standpoint seems fairly straightforward, if complicated to make tractable (variational inference using alpha-divergence is not a new idea)
 - The RL components here aren't particularly novel, really the novelty is in the learning
Decision: Accept (Poster)