Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This is an interesting policy-as-inference approach, presented in a reasonably clear and well-motivated way. I have a couple questions which somewhat echo questions of other commenters here. Unfortunately, I am not sufficiently familiar with the relevant recent policy learning literature to judge novelty. However, as best I am aware the empirical results presented here seem quite impressive for off-policy learning.

- When is it possible to normalize the non-parametric q(a|s) in equation (6)? It seems to me this will be challenging in most any situation where the action space is continuous. Is this guaranteed to be Gaussian? If so, I don’t understand why.

– In equations (5) and (10), a KL divergence regularizer is replaced by a “hard” constraint. However, for optimization purposes, in C.3 the hard constraint is then replaced by a soft constraint (with Lagrange multipliers), which depend on values of epsilon. Are these values of epsilon easy to pick in practice? If so, why are they easier to pick than e.g. the lambda value in eq (10)?


review_title: Interesting off-policy algorithms with nice results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies new off-policy policy optimization algorithm using relative entropy objective and use EM algorithm to solve it. The general idea is not new, aka, formulating the MDP problem as a probabilistic inference problem. 

There are some technical questions:
1. For parametric EM case, there is asymptotic convergence guarantee to local optima case; However, for nonparametric EM case, there is no guarantee for that. This is the biggest concern I have for the theoretical justification of the paper.

2. In section 4, it is said that Retrace algorithm from Munos et al. (2016) is used for policy evaluation. This is not true. The Retrace algorithm, is per se, a value iteration algorithm. I think the author could say using the policy evaluation version of Retrace, or use the truncated importance weights technique as used in Retrace algorithm, which is more accurate.

Besides, a minor point: Retrace algorithm is not off-policy stable with function approximation, as shown in several recent papers, such as 
“Convergent Tree-Backup and Retrace with Function Approximation”. But this is a minor point if the author doesn’t emphasize too much about off-policy stability.

3. The shifting between the unconstrained multiplier formulation in Eq.9 to the constrained optimization formulation in Eq.10 should be clarified. Usually, an in-depth analysis between the choice of \lambda in multiplier formulation and the \epsilon in the constraint should be discussed, which is necessary for further theoretical analysis. 

4. The experimental conclusions are conducted without sound evidence. For example, the author claims the method to be 'highly data efficient' compared with existing approaches, however, there is no strong evidence supporting this claim. 


Overall, although the motivation of this paper is interesting, I think there is still a lot of details to improve. 
review_title: some details to discuss

final_score: rating: 7: Good paper, accept
review_content: The paper presents a new algorithm for inference-based reinforcement learning for deep RL. The algorithm decomposes the policy update in two steps, an E and an M-step. In the E-step, the algorithm estimates a variational distribution q which is subsequentially used for the M-step to obtain a new policy. Two versions of the algorithm are presented, using a parametric or a non-parametric (sample-based) distribution for q. The algorithm is used in combination with the retrace algorithm to estimate the q-function, which is also needed in the policy update.

This is a well written paper presenting an interesting algorithm. The algorithm is similar to other inference-based RL algorithm, but is the first application of inference based RL to deep reinforcement learning. The results look very promising and define a new state of the art or deep reinforcement learning in continuous control, which is a very active topic right now. Hence, I think the paper should be accepted. 


I do have a few comments / corrections / questions about the paper:

- There are several approaches that already use the a combination of the KL-constraint with reverse KL on a non-parametric distribution and subsequently an M-projection to obtain again a parametric distribution, see HiREPS, non-parametric REPS [Hoof2017, JMLR] or AC-REPS [Wirth2016, AAAI]. These algorithms do not use the inference-based view but the trust region justification. As in the non-parametric case, the asymptotic performance guarantees from the EM framework are gone, why is it beneficial to formulate it with EM instead of directly with a trust region of the expected reward?

- It is not clear to me whether the algorithm really optimizes the original maximum a posteriori objective defined in Equation 1. First, alpha changes every iteration of the algorithm while the objective assumes that alpha is constant. This means that we change the objective all the time which is theoretically a bit weird. Moreover, the presented algorithm also changes the prior all the time (in order to introduce the 2nd trust region) in the M-step. Again, this changes the objective, so it is unclear to me what exactly is maximised in the end. Would it not be cleaner to start with the average reward objective (no prior or alpha) and then introduce both trust regions just out of the motivation that we need trust regions in policy search? Then the objective is clearly defined.    

- I did not get whether the additional "one-step KL regularisation" is obtained from the lower bound or just added as additional regularisation? Could you explain?

- The algorithm has now 2 KL constraints, for E and M step. Is the epsilon for both the same or can we achieve better performance by using different epsilons?

- I think the following experiments would be very informative:

   - MPO without trust region in M-step
   
   - MPO without retrace algorithm for getting the Q-value

   - test different epsilons for E and M step



review_title: The paper presents an interesting new algorithm for deep reinforcement learning which outperforms state of the art methods. 

final_score: rating: null
review_content: 
I do have a few comments / corrections / questions about the paper:

- There are several approaches that already use the a combination of the KL-constraint with reverse KL on a non-parametric distribution and subsequently an M-projection to obtain again a parametric distribution, see HiREPS, non-parametric REPS [Hoof2017, JMLR] or AC-REPS [Wirth2016, AAAI]. These algorithms do not use the inference-based view but the trust region justification. As in the non-parametric case, the asymptotic performance guarantees from the EM framework are gone, why is it beneficial to formulate it with EM instead of directly with a trust region of the expected reward?

- It is not clear to me whether the algorithm really optimizes the original maximum a posteriori objective defined in Equation 1. First, alpha changes every iteration of the algorithm while the objective assumes that alpha is constant. This means that we change the objective all the time which is theoretically a bit weird. Moreover, the presented algorithm also changes the prior all the time (in order to introduce the 2nd trust region) in the M-step. Again, this changes the objective, so it is unclear to me what exactly is maximised in the end. Would it not be cleaner to start with the average reward objective (no prior or alpha) and then introduce both trust regions just out of the motivation that we need trust regions in policy search? Then the objective is clearly defined.    

- I did not get whether the additional "one-step KL regularisation" is obtained from the lower bound or just added as additional regularisation? Could you explain?

- The algorithm has now 2 KL constraints, for E and M step. Is the epsilon for both the same or can we achieve better performance by using different epsilons?

- I think the following experiments would be very informative:

   - MPO without trust region in M-step
   
   - MPO without retrace algorithm for getting the Q-value

   - test different epsilons for E and M step


review_title: Few comments...

Meta-review: The main idea of policy-as-inference is not new, but it seems to be the first application of this idea to deep RL, and is somewhat well motivated.  The computational details get a bit hairy, but the good experimental results and the inclusion of ablation studies pushes this above the bar.

Decision: Accept (Poster)