Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers safety in transfer RL, and proposes a method, called Risk-aware successor features, that exploits the task structure to avoid risks and achieve task generalization between tasks with shared dynamics and different goals. The entropic utility is introduced to represent risk-awareness and specialized to entropic Q-value with bellman update. Risk-aware GPI and GPE are extended from GPI, GPE, and proved its convergence. Experiments on two domains show the method outperforms previous works.
main_review: This paper presents Risk-aware Successor Features (RaSFs) for realizing policy transfer between tasks with shared dynamics and different goals. The objective is to optimize the trade-off between expected return and risk, which is measured by the variance of return. 

Strengths:

1. This paper extends risk-aware GPI, provides monotone guarantees and optimality of GPI, extends GPE to the mean-variance objective, inheriting the superior task generalization abilities of successor features. 
2. The analysis on both discrete and continuous domains has showed.
3. The paper is clear to read, easy to follow.

Weaknesses:

Why PRQL(2005) is a challenging baseline? Why not consider more recent policy transfer methods?

Typos:
Line269: The w are also learned --> is
limitations_and_societal_impact: The authors only discuss some limitations in the conclusion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper extends the successor feature framework including the GPE/GPI methods for generalisation across tasks with risk awareness. The paper includes experiments for the associated methods on toy domains.
main_review: Overall the paper has good clarity, however the theoretical results do not seem super useful and the the paper simultaneously lacks on strong empirical evaluation.

The authors should try to address the following:
- How is sample inefficiency a cause for not addressing safety, lines 21-24 are bit unclear.
- A little derivation for taylor expansion in line 98 should be provided in main text
- How useful are Theorem 1 and 2? Ideally one would want to have little uncertainty in the initial Q estimates for transfer as the policy objective is to maximise expected value under initial state distribution, however the theorems gives a very weak bound for h=0
- The method not only assumes access to SFs, which while being problematic to compute themselves also requires estimates for additional quantities like their covariance matrix, I feel this pushes most of the burden of solving hard tasks under the carpet. It is not very convincing to me that tasks which are interesting to solve and pose real challenges with respect to risk uncertainties would simultaneously admit low dimensional SF descriptors (note this is really important for the method given computability of O(d^2) structures like covariance matrix, d being SF dimensionality). Further the lack of thorough experimentation on this front compounds my doubts.
- Are there guarantees which extend for general \beta for the GPI case? Currently it seems that the source and the target tasks assume same risk awareness coefficient.

Updates post discussion:
- I think the authors have made convincing points to address some of the issues raised in my review, also since they will be adding more experiments as stated, I have therefore decided to increase my rating for acceptance.


limitations_and_societal_impact: As mentioned before the main limitation of the work is lack of evidence to support existence and learnability of succinct SFs to describe the space of practical risk sensitive problem domains. Additionally the theoretical results presented do not appear very informative for generalization.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The article extends the Successor Feature framework to the risk-averse setting in which the Entropic Risk Measure is optimized in place of the standard risk-neutral objective. The proposed approach is tested on a gridworld domain and on a Mujoco one, this time integrated with a distributional RL algorithm.
main_review: ## Originality:
The approach is a direct extension of successor features, based on the properties of the Bellman equation of the entropic risk-measure and on its Taylor expansion. Related works are clearly cited in a dedicated section.


## Quality: 
The submission is technically sound from both the theoretical and the empirical point of view. The author determine the theoretical properties of the proposed approach an then tested them in valid environments.
From the experimental side, I only have doubt about the choice of the performance measure. The author chose to show the expected return, together with the failures. However, it would have been fairer to show the optimized objective (the entropic risk measure). The latter may also suffer of instability issues, hence, it would have been interesting to see if this aspects affect the proposed technique. 

## Clarity: 
The paper is well written and organized in a adequate way. Both the experimental analysis and the theory are sufficiently clear to understand the illustrated results. I would suggest anyway minor improvements on some specific parts of the article:
- the example in 3.1, can be exposed in a clearer way, in particular, it is not clear what happens when \beta is equal to 0: which are the optimal policies in the different tasks, in this case?
- In Theorem 1, it is not clear, without looking at the proof, what is the difference between Q and \Tilde{Q}.
- in Figure 3, it is not immediate to understand the performance of the different techniques, maybe a table would do a better job.
- In Figure 4, I would avoid to use faded curves, they are difficult to read. Also, in the second figure, the x-label should be "Test task Instance", from what I understand from the caption.
- In the "Reacher" paragraph of the experimental section, it is not clear what "without interactions effect" means.
- Showing the variance instead of the total failures is also an interesting alternative, justified by the fact that ERM approximate mean-variance, and the studied setting is not a CMDP.

## Significance: 
The proposed approach outperforms the baselines in the first environment, but not its risk-neutral version, even if it allows for a lower number of failures. A similar behavior is observed in the second environment. It is hence difficult to assess whether the proposed approach is better then its risk-neutral version, if the final perfomance w.r.t. the optimization goal is not shown. Nevertheless, the approach offers an interesting an theoretically justified way to transfer learning in risk-averse tasks, representing a clear advance in the state of the art.
limitations_and_societal_impact: No potential negative impact are present.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, a (variance-based) risk-aware transfer method for multi-task RL is proposed. 
In the theory part, GPI and GPE (and the notion of successor features) are extended to entropic utility (for the risk-aware transfer). 
Based on the results in the theory part (specifically, the GPE/successor features parts), the successor feature for entropic utility is proposed to use for risk-aware setting. 
In the experiment part, the proposed method is evaluated in the Four-room and Reacher environments. 

main_review: Key strength:  
Solid theoretical results are provided.  

Theories of (vanilla) GPI and GPE (Theories 1 and 2 in [2]) are extended to entropic utility (i.e., Theories 1 and 2 in the paper under review). Although the derivation (proof) flow for the theories is basically similar to those for the vanilla GPI and GPE in [2], the theoretical results are meaningful because the results extend the GPI and GPE into a risk-aware setting.  
In addition, Theorem 3 (convergence of covariance) is novel.  

Key weakness:  
Evaluation of the proposed method (RaSF) is conducted only in a simple environment. 
In the evaluation, Four-Room (grid-world) and Reacher (simple control) are used for benchmark environments.  
It is unclear if it works well in more complex environments (e.g., Ant-maze with traps). 


I will improve my score if a convincing rebuttal to my comments on "Key weakness" is provided (e.g., the additional experiment result in more complex environments). 


Minor comments:  
Line 76 Other previous work on safe options: Takuya Hiraoka, Takahisa Imagawa, Tatsuya Mori, Takashi Onishi, Yoshimasa Tsuruoka. Learning Robust Options by Conditional Value at Risk Optimization, In NeurIPS, 2019  
Line 431 Deep rl -> Deep RL  
Line 442 NeurIPS -> In NeurIPS  

limitations_and_societal_impact: Yes
review_title: null

Meta-review: After reading each other's reviews and the authors' feedback, the reviewers discussed the merits and flaws of the paper. 
The authors' feedback was instrumental in the reviewers' change of mind.
The paper is still borderline, but I trust the authors' promises to revise their paper according to the reviewers' suggestions. and I propose to accept this paper.
I will check the final version of this paper hoping to find all the additions that have been requested.
Decision: Accept (Poster)