Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper discusses an interesting approach, which is very similar to existing research that the paper does not mention.

Eq. (2) is the minimum free energy over trajectories xi given initial state s:
V(s; beta) = min_{pi'} KL[pi'(xi|s) || pi(xi|s)] / beta - E[R(xi) | s; pi'].
(EDITED: to fix + to - in the R term.)

This has been studied extensively, for example:
Linearly Solvable Markov Decision Problems (Todorov, 2006)
Information Theory of Decisions and Actions (Tishby and Polani, 2011)
Optimal Control as a Graphical Model Inference Problem (Kappen et al., 2012)

Rewards as exponential HMM emissions have been studied in:
An Approximate Inference Approach to Temporal Optimization in Optimal Control (Rawlik et al., 2011)

The properties of positive vs. negative values of beta have been studied in:
Free Energy and the Generalized Optimality Equations for Sequential Decision Making (Ortega and Braun, 2012)

The risk-aversion properties, and in particular the cliff domain, have been studied in:
Taming the Noise in Reinforcement Learning via Soft Updates (Fox et al., 2016)


The application of the particle filtering technique in this context seems somewhat novel, however it is hard to assess this contribution in isolation from comparison to the large body of existing research.
(EDITED: to retract allusion to recent works by Todorov and Kappen)
review_title: Incremental progress over existing research

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
The paper studies a family of risk-sensitive value functions in the context of reinforcement learning. These value functions are parametrized by a single scalar \beta which governs their "risk sensitivity", that is, essentially the value assigned to small short-term rewards relative to large delayed rewards. Large \beta leads to "risk-taking" behavior, making better use of sparse rewards. The authors propose an efficient algorithm for optimizing (an approximation to) these value functions, based on a bootstrap particle filter. The approach is tested in Cliffworld with tabular policies.

The paper is written clearly, with an extensive appendix describing all details. I have not checked the derivations in the appendix. I am not aware of similar approaches in recent RL literature. However, I am not very knowledgeable about literature in some related fields, like optimal control or bandits.

Could the authors please comment on computational complexity compared to usual policy gradients? Is it correct that it is the same, perhaps up to a constant factor?

Pros:
- Addressing an important problem: exploration/exploitation, sparse rewards
- Departing from the standard expected reward objective
- Compiling techniques from different fields in a new and useful way
- Experiments in Cliffworld with tabular policies support the intuitions behind the approach. Risk-taking agent is able to solve the task unlike its more conservative versions.

Cons:
- No experiments in realistic RL scenarios with function approximators. 
- Basically no discussion of alternative approaches or comparisons to baselines (one that comes to mind is Prioritized DQN, but there may be more)

Overall, despite weak experimental evaluation, I tend to recommend acceptance, since the approach looks interesting, new and promising. The paper is outside of my main field of expertise, therefore I am only moderately certain.
review_title: Official review

Decision: Accept