Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this work the authors studied the entropic risk RL problem and derived a regret bound of the entropic value iteration algorithm via the risk-sensitive optimism in face of uncertainty approach. Utilizing the theoretical framework of Osband 2014, they extend the results to entropic-risk RL with linear function approximations on the transition dynamics and value function and later with general function approximation in episodic MDPs.

Unfortunately I am not very familiar with the recent theoretical exploration results with randomized value functions. On the high-level, I find the contribution of extending this to entropic-risk RL reasonable. One main question is why episodic MDPs are chosen for deriving the main theoretical results, and can these results be extended to more standard MDP frameworks such as discounting MDPs. Moreover, on top of the technical description, can the authors provide some intuitive explanations of condition 1 (which is a main RSOFU assumption that pivots most of the theoretical results) and some examples on when this condition is satisfied? 

While I understand this paper's main contribution is theoretical and appreciate the efforts for deriving that. Since the algorithm in analysis is fairly standard, it'd be great to see at least some simple, proof-of-concept numerical experiments to evaluate the tightness of the regret bound. Without that, it's rather difficult to understand the several theoretical results, compare with the references mentioned, and justify their usefulness. 
review_title: Paper presents new theoretical (regret) results on entropic RL with linear parametrization, lack experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a risk-sensitive algorithm with function approximation in reinforcement learning. To handle the uncertainty, the proposed algorithms consider an entropic risk value function controlled by a risk parameter, which provides a unified framework for both risk-sensitive and risk-averse settings. The main contribution of this paper is to provide theoretical guarantees for the proposed algorithms. 

Pros 

The idea of using entropic risk value functions in RL is a very interesting. 

The paper is well-written and the proofs are present in details. 

Cons

My major concern is the novelty in this paper. It is closely related to a recent paper, which considers the same entropic value function in RL, but focus on the tabular setting:

Fei, Y., Yang, Z., Chen, Y., Wang, Z. and Xie, Q., 2020. Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret. arXiv preprint arXiv:2006.13827.

This paper is trying to extend the results in [Fei et al. 2020] to the function approximation setting. Then the key to decide if the paper’s contribution is enough is to see how hard such extension is. 

However, although [Fei et al. 2020] focus on the tabular setting, it seems that their proof techniques can be easily generalized to the linear function approximation setting (as they write the tabular representation using canonical basis, then the values could be written as the solution of a linear regression). I check the details of the proof and it indeed shares a lot of similarities with [Fei et al. 2020]. The major difference is how to choose a proper regularization (Lemma 10). 

Minor comments: 

Two related papers that use entropic value functions in RL. 

O'Donoghue, B., 2018. Variational bayesian reinforcement learning with regret bounds. arXiv preprint arXiv:1807.09647.

O'Donoghue, B., Osband, I. and Ionescu, C., 2020. Making sense of reinforcement learning and probabilistic inference. arXiv preprint arXiv:2001.00805.

review_title: Entropic Risk-Sensitive Reinforcement Learning: A Meta Regret Framework with Function Approximation

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper considers a finite horizon episodic RL problem where the objective is risk-sensitive using the entropic risk measure. The paper proposes two algorithms based on the optimism in the face of uncertainty principle, one for linear function approximation and the other for general function approximation. Regret bounds are derived for both algorithms. 

The strong points of this paper are that the algorithms proposed seem well grounded based on the optimism principle, and that a formal regret analysis is provided for general optimistic algorithms in the context of risk-sensitive object, which is then specialized to the two algorithms proposed. 

One weak point of this paper is that the algorithms proposed do not seem very practical. What’s proposed for general function approximation is mostly a conceptual algorithm. It’s unclear how it could be applied in practice. Even with linear function approximation, it seems that the algorithm requires storing all past iterates as well as sweeping over all states and actions for each episode, which is extremely computationally costly. 

Even though there’s some nice theoretical contributions, I am not sure if this is the best venue for this paper, since it’s not really related to representation learning… (although the conference has been constantly evolving over the years). 

Given the points above, I am leaning towards rejection.

The organization of the paper is pretty clear. One minor suggestion is that the authors might want to compress some of the background on value functions to make room for a concluding section. 

The paper is easy to read in general, but I find the naming and terminology like “meta RSVI” and “meta regret analysis” a bit confusing. It seems to me that the proposed algorithms fall straight under the OFU principle, and it might be more straightforward to just say that instead of coming up with new names like MetaRSVI. 
review_title: not related to theme of the conference...

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################

Summary:

The paper studies risk-sensitive reinforcement learning with the entropic risk measure and function approximation. A meta algorithm based on value iteration is first proposed, then the paper proposes two concrete instantiations, one for linear function approximation and one for general function approximation. Regret bound for both algorithms depend sub-linearly in the number of episodes, and the linear one depends polynomially on the ambient dimension, whereas the general one depends polynomially on the Eluder dimension.

##########################################################################

Reasons for score: 
 
Overall, I vote for accepting. I think this paper considers an important topic: risk-sensitive reinforcement learning, and provides a first step in establishing theoretical foundation under that setting with function approximation. 

##########################################################################

Pros: 

1) The paper tackles an important issue: risk-sensitive reinforcement learning with function approximation, and shows regret bounds that are near optimal under some assumptions.   

2) The value iteration based meta algorithm framework and the meta regret bound provides a nice abstraction beyond concrete implementation under different assumptions. It might benefit future studies on function approximation under risk-sensitive reinforcement learning. 
 

##########################################################################

Cons: 
 
1. The technical tools used in this paper seem to be based on a combination of ideas from Fei et al. (2020), Cai et al (2019), Russo and Van Roy (2014) and other papers. The analysis itself does not bring new insights.
 

##########################################################################

Minor comments: 

It might be worthwhile to discuss other definition of “risks” beyond entropic risk measure under the risk-sensitive reinforcement learning settings, and mention any relevant theoretical results under those definitions.

review_title: Decent theoretical paper 

Meta-review: 
The paper considers the risk sensitive RL by exploiting entropic risk. The major contribution of this paper is providing the theoretical guarantees for the proposed risk-senstive value iteration with function approximation. 

The major concern of this paper is the similarity to the existing work in (Fei et al., 2020). I encourage the authors to reorganize the paper and emphasize the differences to highlight the major contribution. 
Decision: Reject