Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a risk-sensitive reinforcement learning algorithm based on DDPG for the problem of risk-averse option hedging and equal risk pricing based on dynamic expectiles. The proposed algorithm is empirically tested on a simulated option hedging environment.
main_review: Overall, the paper is well-written but certain parts, especially finance-related concepts could be further elaborated for the general AI/ML audience. 

The paper addresses the important issue of risk-sensitive RL. However, the focus here is very narrow, on applications in option hedging and equal-risk pricing. Usually, when it comes to methods targeting a specific application, the paper must at least achieve one of the following two:
1) Can the proposed approach be easily applied or generalized to other problems or application domains ? 
This is not obvious. The dynamic risk measures have very good theoretical properties -- everything is Markov, and therefore very convenient for dynamic-programming-based solutions. The proposed DDPG extensions are therefore quite straightforward compared to the case of static risk measures. Furthermore, as pointed out by the authors, the state-transition dynamic depends only on external uncertainties thereby making the problem even easier than solving general MDPs.

2) If the above condition is not satisfied, then the target application must itself be significant (e.g. solve chess).
Here, the problem is option hedging and equal-risk pricing. I can accept these as significant but the evidence presented in the paper is insufficient. 
 - Firstly, all experiments are only done in simulated environments based on the fitted Brownian-motion model (for both training and testing). This means that RL can be trained using unlimited data and there is no train-test mismatch. This does not test the RL ability to generalize beyond the training examples. We have many years of stock prices available, so why not test on real data?
 - The comparison between policies obtained via optimizing static/dynamic risks seems confusing. It seems that using methods that optimize the static risk directly, we would obtain a better test-set static risk. Similarly, using methods optimizing the dynamic risk directly, we would obtain better test-set dynamic risk. What is wrong here?


summary_of_the_review: Based on my concerns above, I cannot recommend acceptance at this stage.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper targets a fundamentally important issue: risk-aversion in pricing and hedging. The authors claims the first algorithm to identify optimal risk averse for option hedging strategies, which are time-consistent with respect to a dynamic convex riskmeasure, etc., those are impressive.
main_review: Strengths:
1. The target problem is interesting and maybe potentially useful.
2. The proposed algorithm seems to be rigorous, the theoretical guarantees are appreciated.

Weaknesses:
1. A first question I would hope the authors to clarify is that: from the organization and presentation of this paper, this paper is more like an application paper of DRL to pricing and hedging tasks. But for an AI conference, the readers are expecting to see some general form behind, so that this solution can also benefit related studies. If it focuses on one particular application, then an ML&finance conference or journal would be a better choice for this work.
2. From Alg. 1, one key change from DDPG to the proposed ACRL is taking an average of N, which would effectively reduce variance.  Then, I believe it would be more valuable to discuss the variance.
3. As for the experiment results in Section 4.3 and Section 4.4, the major issue is that the rationale of performance metrics are not clearly presented, so it is not easy to evaluate the results.

After rebuttal:
1. The first concern is addressed.
2. The second one still remains, so there is a question about novelty.
3. The third one still remained, which is still not very easy to evaluate its values to practioners.
summary_of_the_review: Interesting application, but need the authors to justify some ambiguity.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper offers a solution to the problem of Equal Risk Pricing (ERP) by framing it as a risk-sensitive MDP formulation. In particular, the authors considers a dynamic risk-measures using expectiles measures as conditional risk-measure for each step. An extension of DDPG is derived by exploiting the properties of the chosen risk-measure and it is tested on two experiments. The first one involves a vanilla option, while the second one feature a more complex basket option.
main_review: The work is novel since, as highlighted by the authors, they provide the first DDPG-like algorithm optimizing a dynamic risk-measure. Differently from previous approaches, their work allow to produce time-consistent hedging policies.
One problem with this article consists with its accessibility to a wide audience: the financial introduction may result difficult to understand to a reader who is unfamiliar to option hedging. The paper has also some other passages which may be difficult to follow, for instance in the derivation of the DDPG-like algorithm and in the experimental analysis. 
In particular I would like to ask the authors for two specific issues.
1) A stochastic gradient step is used to improve expectile estimators taken from (Shen et al., 2014): can the authors elaborate more on why the suggested one is a good estimate?
2) In section 4.3 you say that "First, unlike with DRM, the static risk of SRM policies for the seller (resp. buyer) can increase (resp. decrease) when hedging an option with shorter maturity. The possibility that a seller’s policy may actually increase risk when applied to an option with shorter maturity is clearly problematic here", however, it is not clear to me how this is apparent from the results of Table 1.

Moreover, it is not clear to me wheter the algorithm is applicable only to option hedging or it always applicable in MDP where the objective is the dynamic risk-measure in exam.
The article also lacks a conclusion, which the author should surely add to their final version in order to summarize their work and to indicate some directions for future research.
summary_of_the_review: The paper offer a novel perspective on Equal Risk Pricing for option hedging using reinforcement learnin and dynamic risk-measures. The contribution is mainly an algorithmical and experimental one, but the proposed approach seems to be sound and effective. I would like to propose a borderline accept to this paper.
review_title: null

Meta-review: This paper proposes a risk-sensitive actor critic reinforcement learning (RL) method that optimizes the policy with respect to a dynamic (iterated) expectile risk measure.  The expectile risk measure has the elicitability property and can be expressed as the minimizer of an expected scoring function, which is exploited in critic update.  The proposed approach is applied to option pricing and hedging.

A main point of discussion was the applicability and effectiveness of the proposed method beyond particular financial problems.  The original submission was indeed specialized to particular financial tasks.  The authors have rewritten the paper in a way that it claims to propose a risk-sensitive RL method for the general MDP with finite horizon.  This however leaves the question regarding the advantages of the proposed approach over existing methods of risk-sensitive RL, including those that work with non-coherent (dynamic) risk measures (since coherence is often not needed in domains outside finance).
Decision: Reject