Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors provide a novel approximation rule for the risk-sensitive criterion in RL. In particular they present an iterative update rule for the estimation of the free energy, which has direct application to risk-sensitive RL. They point out relevant properties of the approach and their relationship to game-theoretic and finance and evaluate these on a set of benchmark problems.
main_review: The paper is written very well.  While heavy on theory it is explained in an understandable way and the resulting methodology is simple. 

The experiments make sense given the research question. I particular like that the authors start with simple toy problems (including verifying the estimator), before using to more complex domains such as 2-armed bandits


Cons:
As the authors point out the main limitation is the Gaussian assumption for the Returns.  In this case the paper would have been stronger had the authors constructed an (toy) experiment where the Gaussian assumption does not hold, ideally with long-tail reward distributions, as these are often seen in practice. 

Disclaimer: I did not check the math in Section 2, so for the review I will assume it to be correct.

A question: The free energy as certainty-equivalent applied to RL always appeared a bit odd to me: If we have the expected return (units return) we are adding the variance (units squared return) to it, weighted by  \beta.  This does not seem right. Wouldn't it make more sense to use \mu + \beta \sigma here (as the standard deviation and the mean share the same unit)?  Could from your update rule also an analogous rule for this be derived?
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper is about computing the log of the expectation an exp-transformed random variable (free energy) by stochastic approximation, under the assumption that this random variable is drawn from a Gaussian distribution. While not stated explicitely in this way, it seems that the underlying motivation may come from computation neuroscience: that of obtaining an explainable learning rule, in line the Rescorla-Wagner rule. I'm saying this because under the Gaussian assumption, the free energy admits a closed-form solution where the unknows are the mean and standard deviation which we don't necessarily need to estimate by stochastic approximation unless with have specic reasons to do so. The asymptotics behavior of the learning rule is studied in lemma 2 via an analysis akin to the "ODE method" in stochastic approximation: by considering the continuous time limit of the expected deterministic counterpart. Rather than talking about stability in terms of the eigenvalues of the Jacobian, the proof considers the possible values taken by the deterministic learning rule and shows that it has the free energy as a root. An application in reinforcement learning is presented where the authors match the form of their stochastic approximation method to that of temporal difference learning for policy evaluation and later present results in the control setting with deep reinforcement learning. 
main_review: This paper is impeccably writen: the flow, clear notation and figures (latex-styled plots, pdf output), historical context and cited papers older than a year ago. This really sets the standard for what our community should be striving to achieve. It is really refreshing to read a paper structured like this one! It starts right from the begining with a clear and direct description of what has been found, in math. The high level motivation then follows. It's not wasting anyone's time trying to sell their work. I wouldn't be surprise to learn that the first author is a physicist who moved to computational neuroscience. 

# Issues

- The experiments sections departs quite far from the original setup where the assumptions are clear and the analysis is tractable. It feels like those experiments (the deep rl ones) were designed to make the paper feel more "neurips-y" and appeal to a more ML-focused audience. I would have preferred to see simpler but more carefully designed experiments. More on that in the comments below

- Regarding the application to TD, of course many questions are left unanswered. Note that you start with the policy evaluation setup and then go to the control setting in the experiments. Let's just start with policy evaluation: in the tabular setting, is the corresponding method stable? What could you say via the ODE method? Can you show that the corresponding deterministic operator is a contraction? If not, can you establish stability in a different way? Now one step further: in the policy evaluation setting, what happens if now throw in linear functiona approximation? Do you obtain a system of "projected Bellman equations"? And finally the control setting: take Q learning in the tabular setting, add your update rule, does it converge? What do you get when you apply the ODE method once more (see Kushner and Yin for Q learning)?

(Reflecting on the above, perhaps nothing could be said due to the iid assumption.)

# Notes

Line 37: "is defined" it'd be nice to have a citation

Equation 7: Time indices would be appropriate here and throughout the entire paper: $v_{t+1} = v_t + \alpha_t \cdot (x_t - v_t)$.

Line 153: It doesn't look like the analytical error function (21) would lend itself easily to automatic differentiation due to the presence of the dilogarithm term. Tensorflow seems to offer tf.math.special.spence, but what is it really going on under the hood? What is the corresponding vector-Jacobian rule? 

Line 183: "trained R2D2" The experimental setup and the choice of figures is appropriate. However, using R2D2 seems to be overkill for the problem being solved here. I would have preferred to see a minimalistic vanilla Q-learning + SA Free energy demonstration than a deep RL setup for a grid world problem. 

Line 208: "memory-based meta-learning" again, the experimental setup seems overkill and clashes with the more methodical approach taken earlier in the paper. 

Line 239: "This is the main contribution." I love this
limitations_and_societal_impact: I appreciate how figure 2 also shows how the algorithm performs when the Gaussian assumption is not satisfied. 

See above comments on issues regarding the TD case
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors introduced a stochastic approximation rule for estimating the free energy from
i.i.d. samples generated by a Gaussian distribution of unknown mean and variance. They apply their result then to optimize the entropic risk measure in an RL task. 
main_review: ## Originality:
At the best of my knowledge, the proposed update rule is indeed novel. The authors correctly give an overview of the state of the art related to the free energy, even if a related work section is missing.

## Quality: 
The main result of the authors is Lemma 1, which seems to be correct. However it is not possible to apply it to  MDPs with a transition kernel which does not follow a Gaussian distribution. This limitation is recognized by the authors in a specific section, nevertheless, they show its use as a heuristic for TD learning.
They set a RL experiment on a gridworld task, in which the desired behaviour is obtained.
However, in order to prove that the heuristic is valid, it would have been nice to see it applied in a set of environments, with different difficulty levels. It would be interesting to have more general results about the update rule, for general distributions, or, at least, for some other families. 

## Clarity: 
The paper is clearly written, however, the structure is unusual: there is not a clear separation between introduction, related works, problem formulation and proposed methods. Organizing better the contents would help the reader figuring out the real contribution of the paper.

## Significance: 
The result is interesting indeed, however, it is difficult to see if it will have an impact alone. There are indeed two problems that limits the application of the rule to RL:
- the Gaussian distribution assumption;
- the continuous time assumption.
The latter is necessary to prove the convergence to the fix point, however, RL is usually focused on discrete time settings.
limitations_and_societal_impact: No potential negative impact is present.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed an algorithm to estimate free energy. It provided a lemma to justify its update rule and conducted experiments to show the validity of its algorithm.
main_review: Originality. The algorithm it proposed is an improvement of former algorithm. The intuition behind this improvement require further explanation. The lemma it proved is a basic property. 

Quality. This work is complete and its result is supported. However, there might be some little mistake in it. For example, equation (5) should be double checked.

Clarity. This paper is written clearly.

Significance. I am not persuaded that it is significant by this paper. First, the author proposed using it for risk-sensitive RL. However, there are already several risk-sensitive RL algorithms. Why should we use free energy instead of other algorithm? Why is it superior? Secondly, the lemma here required Gaussian assumption and its only conclusion is that free energy is the fixed point. It did not include the rate of convergence. 
limitations_and_societal_impact: The value of this algorithm (Why should we use it when we have other mature risk-sensitive RL algorithms? ) should be emphasized more.
review_title: null

Meta-review: The paper introduces some technical ideas that may interest a significant segment of the community.  The theoretical work is carried out under quite restrictive assumptions.  But the ideas may be of broader interest.
Decision: Reject