Model Risk-sensitive Offline Reinforcement Learning

Authors: Gwangpyo Yoo, Honguk Woo
Primary area: reinforcement learning
Keywords: risk-sensitive offline reinforcement learning, reinforcement learning, offline reinforcement learning, risk, model risk

Abstract:
Offline reinforcement learning (RL) is becoming critical in risk-sensitive areas such as finance and autonomous driving, where incorrect decisions can lead to substantial financial loss or compromised safety. However, traditional risk-sensitive offline RL methods often struggle with accurately assessing risk, with minor errors in the estimated return potentially causing significant inaccuracies of risk estimation. These challenges are intensified by distribution shifts inherent in offline RL. To mitigate these issues, we propose a model risk-sensitive offline RL framework designed to minimize the worst-case of risks across a set of plausible alternative scenarios rather than solely focusing on minimizing estimated risk. We present a critic-ensemble criterion method that identifies the plausible alternative scenarios without introducing additional hyperparameters. We also incorporate the learned Fourier feature framework and the IQN framework to address spectral bias in neural networks, which can otherwise lead to severe errors in calculating model risk. Our experiments in finance and self-driving scenarios demonstrate that the proposed framework significantly reduces risk, by $11.2\%$ to $18.5\%$, compared to the most outperforming risk-sensitive offline RL baseline, particularly in highly uncertain environments.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729874442823

thread_type:
official_review

<thread_object>
creator: reviewer_3isb

<content>
summary:
This work considers the risk-sensitive setting in offline RL, which is important for real-world applications. This work proposes to minimize the worst-case risks across a set of plausible alternative scenarios rather than solely focusing on minimizing estimated risk. The proposed method, MR-IQN, introduces a critic-ensemble criterion method, the learned Fourier feature framework, and the IQN framework to address spectral bias in neural networks. Extensive experiments show that MR-IQN can significantly reduce the different kinds of risk, like CVaR and Wang risk.

strengths:
- Considering spectral risk measure is interesting in risk-sensitive RL as it seems a general form of several risk metrics like CVaR, worst case, and Wang risk.

- Extensive ablation studies show the importance of each component in MR-IQN.

- Fig. 4 explains the advantage of Fourier feature clearly.

weaknesses:
- It seems that the algorithm is a direct application of previous theoretical results (Bernard et al., 2023) as the major theoretical results (Thm 1, Cor 1) are from (Bernard et al., 2023). What is the major challenge in applying these theoretical analyses to your methods?

- There seems to be a gap between the objective 10 (can be simplified by Cor. 1) and the proposed algorithm. Why can MR-IQN handle objective 10? As objective 10 is a constrained optimization problem and hard to solve directly, do you use bi-level optimization, or use the Lagrange multiplier method, or some other methods? It will make the work more solid by providing a more detailed explanation of how MR-IQN addresses the constrained optimization problem in objective 10, including any specific optimization techniques they use.

- Lack of some related work on several risk-sensitive RL methods including different risk metrics which are closely related to this work, like CVaR [1-4], EaR [5], worst case [6], and so on.

Reference:

[1] TRC: Trust region conditional value at risk for safe reinforcement learning

[2] Towards safe reinforcement learning via constraining conditional value-at-risk

[3] Distributional reinforcement learning for risk-sensitive policies

[4] Efficient risk-averse reinforcement learning

[5] Risk-sensitive reinforcement learning via Entropic-VaR optimization

[6] Provably efficient risk-sensitive reinforcement learning: Iterated cvar and worst path

questions:
- Why can we translate the objective (8) to the objective (10)?

- In step 1, why does MR-IQN choose the **smallest** expectation and the **largest** deviation?

- Why do Fourier features benefit a lot, are there any explanations?

- Are there ablation studies about the number of ensembled critics?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730356843681

thread_type:
official_review

<thread_object>
creator: reviewer_mn9s

<content>
summary:
This paper introduces MR-IQN, a model-risk-sensitive offline RL algorithm, which aims to minimize risk in worst-case scenarios instead of conventional risk. Building on TD3+BC, it incorporates a critic ensemble to estimate model risk and employs a Fourier feature quantile regression network to mitigate spectral bias, allowing for precise distributional statistics estimation. MR-IQN outperforms baseline algorithms CODAC and ORAAC in two finance environments, Forex and Stock, as well as in the Airsim environment.

strengths:
1. The approach of minimizing model risk in offline RL using a critic ensemble, rather than focusing on conventional risk, is an interesting and novel idea.
2. MR-IQN demonstrates strong results in the presented environments, indicating promising potential.

weaknesses:
1. The experimental section feels underdeveloped, as MR-IQN and the baselines are only tested in three environments. Expanding the evaluation to include a broader set of benchmarks, such as those in D4RL (as done in CODAC and ORAAC), would significantly strengthen the results.
2. The choice of metrics in Table 1 raises some concerns. Evaluating performance at the 50% quantile is unusual, as this metric doesn’t emphasize rare events, which seems at odds with the motivation of risk-sensitive RL. To the best of my knowledge, previous literature typically uses the 10% quantile. Could you clarify why CV@R(50%) was selected for this study?
3. The explanation of model risk is somewhat unclear, and Figure 1 is particularly challenging to interpret. A more detailed and structured presentation of model risk concepts would improve readability and understanding.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730518425762

thread_type:
official_review

<thread_object>
creator: reviewer_ahnq

<content>
summary:
This work introduces a framework and learning algorithm, MR-IQN, for offline reinforcement learning in a risk-sensitive context. The MR-IQN algorithm aims to identify a policy that minimizes model risk while remaining close to the behavior policy derived from the dataset. The authors evaluate the algorithm on both finance and self-driving datasets, demonstrating that it outperforms baseline methods in terms of model risk measures and average return.

strengths:
1. **Novelty:** To the best of my knowledge, this is a novel formulation for risk-sensitive offline RL, with no similar approach appearing in current literature. Additionally, the use of a critic ensemble and a Fourier feature network for variance and bias reduction is a thoughtful design choice.
2. **Clarity:** The paper is well-organized and self-contained, providing a clear presentation of the problem, algorithmic approach, and experimental setup.
3. **Comprehensive Experiments:** The evaluation on finance and self-driving applications aligns well with the risk-sensitive focus of the paper. Extensive experiments, including a range of baselines, lend further credibility to the results.

weaknesses:
**Major Issues:**

1. The formulation of risk-sensitive offline RL optimizes model risk regularized by the dataset's behavior policy. It is unclear how this formulation guarantees that the optimality considers both model risk and expected return, especially when the behavior policy in the dataset may not be optimal, as is often the case in offline RL.
2. There is a lack of theoretical insight into the algorithm. Specifically, it is not evident why the proposed loss function in Equation (11) adequately solves the problem defined in Equation (10). Additional theoretical justification would enhance understanding of the algorithm's effectiveness.

**Suggested Improvements:**

1. In the experimental section, an ablation study on ensemble size (i.e., number of critics) would help substantiate the value of using an ensemble for performance gains.
2. The authors choose $\mu$ as the minimum value among critics' estimations and $\sigma$ as the maximum, aiming for conservatism. It would strengthen the argument if they defined conservatism and justified the choices of $\mu$ and $\sigma$.

**Minor Issues:**

The following suggestions may improve clarity:

1. In Equation (3), the form $ F^{-1}_{\phi}(U[0, 1]) $ is not clearly defined by Definition 1.
2. Is $ Z_{\pi} $ intended to mean the same as $ Z^{\pi} $? If so, consistent notation is recommended.
3. In lines 275-276, the notation $ E_{p_j} $ appears undefined or unclear.
4. In Figure 4, the x-axis and y-axis require explicit labeling for clarity.
5. In Tables 1 and 2, the meaning of "Average" should be clarified.

questions:
1. I appreciate the illustrative explanation of the calculation for $ F_{ens.}^{-1} $; however, is there a mathematical formula for this calculation? Additionally, were gradients preserved when calculating $ F_{ens.}^{-1} $ during experiments?
2. In the experiments, how was the offline dataset generated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730629254576

thread_type:
official_review

<thread_object>
creator: reviewer_dstd

<content>
summary:
This paper addresses the problem of model risk in offline reinforcement learning, which arises from distributional shifts in the learning dataset. The authors provide a critic ensemble framework, integrating well-researched predecessor methods into a critic ensemble for risk estimation that can be applied regardless of the transition distribution or differences in distributional support. Experimental results in two domains demonstrate the superiority of the method and highlight the tradeoffs between robustness and high performance.

strengths:
The motivation is clear; the problem statement, issues with existing works, and connections to the provided solution are clean and easy to follow. I also appreciate the label brackets underneath parts of the equations. 

The contribution addresses the issue both distributional shifts in data and model inaccuracies, which are critical challenges in offline RL.

weaknesses:
Most of the formalizations are cited directly from previous works, with the exception of 4.1. Given that 4.1 is only half of the proposed methods section, this makes the contribution seem more incremental. Are there changes or improvements that can be made to any of these components to better suit them to the domain? As an example, the Bellamn equation in Eq. 4 could be extended to measures such as minimum reward, expected value (over actions), or regret.

There are a few minor spelling errors, like 'qunatile' in the Fig. 4 caption.

questions:
1) It is mentioned that when $\epsilon \leq (\mu_z-\mu)^2 + (\sigma_z - \sigma)^2$ holds, the quantile function doesn't have a solution. Under what distribution conditions does this occur, and how often is it observed? It would be useful to know if the strictly-less-than case occurs with relative infrequency, or very often.

2) Baselines ORAAC, CODAC, and 1R2R are mentioned in the introduction and related works, but only the first two are used as comparison baselines. Is there a reason the most recent method was excluded?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732947905021

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We appreciate the valuable and constructive feedback provided by all reviewers-Reviewer DSTd, Reviewer AhNq, Reviewer mn9s, and Reviewer 3isb. We have completed the revisions to our paper based on the reviewers' feedback and our own efforts, and the final version has been uploaded. We hope the reviewers will carefully evaluate these updates and find that they address any remaining concerns. A detailed list of revisions is provided below:

---

**Main Manuscript Revisions** 
* **Texts** 
    * We have improved the related work section (3isb). 
    * We corrected the typo in Eq. (3) and added an explanation using label braces (AhNq). 
    * We elaborated on the mathematical explanations for the necessity of the Fourier feature network (3isb, DSTd). 
    * We provided justification for the implementation, specifically explaining why we chose $\min \mu$ and $\max \sigma$ (AhNq, 3isb). 
    * We added text to notify readers about the additional experiments included in the Appendix (DSTd, mn9s). 
    * We slightly paraphrased and corrected typos to ensure the content fits within the margin.
* **Figure, Algorithm, and Table**
    * We revised Figure 1, adding a more intuitive explanation of model risk (mn9s).
    * We fixed the typo and added axis labels in Figure 4 (AhNq, DSTd).
    * We changed the table label from "Average" to "Mean" to avoid confusion (AhNq).
    * We added an explanation for the first line of the Algorithm (AhNq).

---

**Appendix**

* We added experimental results from an ablation study addressing the correlation between performance and ensemble size (3isb, AhNq).
* We included experimental results for *CV@R*(10%) along with a detailed analysis (mn9s).

---
We acknowledge that the discussion period is ongoing and welcome any additional feedback, questions, or responses regarding our submission. We will do our best to respond promptly and thoroughly. Once again, we sincerely thank all reviewers for their time and dedication.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers acknowledged that the paper studies an important problem of risk-sensitive offline RL and proposes a novel methodology to minimize the worst-case risk instead of the expected risk. However, the reviewers also raised several concerns and questions in their initial reviews; in particular, one concern was about how the proposed methodology differs from the prior work. We want to thank the authors for their responses and active engagement during the discussion phase. The reviewers appreciated the responses and have an overall positive assessment of the paper, though the final ratings for all the reviewers remain borderline. The reviewers have provided detailed feedback, and we strongly encourage the authors to incorporate this feedback when preparing an updated version of the paper.