Title: RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning Human Feedback

Risk-aware Reinforcement Learning
Abstract: Reinforcement Learning from Human Feedback (RLHF) has recently surged in popularity, particularly for aligning large language models and other AI systems with human intentions. At its core, RLHF can be viewed as a specialized instance of Preference-based Reinforcement Learning (PbRL), where the preferences specifically originate from human judgments rather than arbitrary evaluators. Despite this connection, most existing approaches in both RLHF and PbRL primarily focus on optimizing a mean reward objective, neglecting scenarios that necessitate risk-awareness, such as AI safety, healthcare, and autonomous driving. These scenarios often operate under a one-episode-reward setting, which makes conventional risk-sensitive objectives inapplicable. To address this, we explore and prove the applicability of two risk-aware objectives to PbRL : nested and static quantile risk objectives. We also introduce Risk-AwarePbRL (RA-PbRL), an algorithm designed to optimize both nested and static objectives. Additionally, we provide a theoretical analysis of the regret upper bounds, demonstrating that they are sublinear with respect to the number of episodes, and present empirical results to support our findings. Our code is available in https://github.com/aguilarjose11/PbRLNeurips.

RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning

Abstract

Reinforcement Learning from Human Feedback (RLHF) has recently surged in popularity, particularly for aligning large language models and other AI systems with human intentions. At its core, RLHF can be viewed as a specialized instance of Preference-based Reinforcement Learning (PbRL), where the preferences specifically originate from human judgments rather than arbitrary evaluators. Despite this connection, most existing approaches in both RLHF and PbRL primarily focus on optimizing a mean reward objective, neglecting scenarios that necessitate risk-awareness, such as AI safety, healthcare, and autonomous driving. These scenarios often operate under a one-episode-reward setting, which makes conventional risk-sensitive objectives inapplicable. To address this, we explore and prove the applicability of two risk-aware objectives to PbRL: nested and static quantile risk objectives. We also introduce Risk-AwarePbRL (RA-PbRL), an algorithm designed to optimize both nested and static objectives. Additionally, we provide a theoretical analysis of the regret upper bounds, demonstrating that they are sublinear with respect to the number of episodes, and present empirical results to support our findings. Our code is available in https://github.com/aguilarjose11/PbRLNeurips .

1 INTRODUCTION

Reinforcement Learning (RL) [Russell and Norvig, 2010] is a fundamental framework for sequential decision-making, enabling intelligent agents to interact with and learn from unknown environments. This framework utilizes a reward signal to guide the selection of policies, where optimal policies maximize this signal. RL has demonstrated state-of-the-art performance in various domains, including clinical trials [Coronato et al., 2020], gaming [Silver et al., 2017], and autonomous driving [Basu et al., 2017].

Despite its performance, a significant limitation of the standard RL paradigm is the selection of a state-action reward function. In many real-world scenarios, constructing an explicit reward function is often a complex or unfeasible task. As a compelling alternative, Preference-based Reinforcement Learning (PbRL) [Busa-Fekete et al., 2014, Wirth et al., 2016] addresses this challenge by deviating from traditional quantifiable rewards for each step. Instead, PbRL employs binary preference feedback on trajectory pairs generated by two policies, which can be provided directly by human subjects. This method is increasingly recognized as a more intuitive and direct approach in fields involving human interaction and assessment, such as autonomous driving [Basu et al., 2017], healthcare [Coronato et al., 2020], and language models [Bai et al., 2022].A specialized and increasingly popular variant of PbRL is Reinforcement Learning from Human Feedback (RLHF)Bai et al. [2022], which has garnered considerable attention for aligning AI systems with human intentions-particularly in the domain of large-scale language models.

Previous approaches to PbRL [Xu et al., 2020a, Coronato et al., 2020, Xu et al., 2020b, Chen et al., 2022, Zhan et al., 2023] mainly aim to maximize the mean reward or utility, which is risk-neutral. However, there is a growing need for risk-aware strategies in various fields where PbRL has shown empirical success. For example, in autonomous driving, PbRL reduces the computational burden by skipping the need to calculate reward signals for every state-action pair [Chen et al., 2022]. Despite this improvement, the nature of the problem makes dangerous actions costly. Thus, such risk-sensitive problem settings require risk awareness to ensure safety.

Risk-Aware PbRL also has implications for fields like generative AI [OpenAI, 2023, Chen et al., 2023], where harmful content generation remains a challenge for fine-tuning. In this scenario, a large language model (LLM) is often fine-tuned with user feedback by generating two prompt responses, A and B. Many approaches using RLHF [Ouyang et al., 2022] consider human feedback to penalize harmful content generation. Unfortunately, current approaches only minimize the average harmfulness of a response. This can be a challenge when responses are only harmful to a minority of users. Risk-aware PbRL tackles this challenge by directly aiming to decrease the harmfulness directly, rather than indirectly as with human feedback fine-tuning.

Despite substantial evidence highlighting the importance of risk awareness in PbRL, a significant gap persists in the theoretical analysis and formal substantiation of risk-aware PbRL approaches. This deficiency has spurred us to develop risk-aware measures and their theoretical analysis within PbRL. In standard RL, a variety of risk-aware measures have been explored, including a general family of risk-aware utility functions [Fei et al., 2020], iterated (nested) Conditional Value at Risk (CVaR) [Du et al., 2022], and risk-sensitive with quantile function form [Bastani et al., 2022]. In general, these measures can be categorized into two types: nested or static. Nested measures [Fei et al., 2020, Du et al., 2022] utilize MDPs to ensure risk sensitivity of the value iteration at each step under the current state, resulting in a more conservative approach. In contrast, static risk-aware measures Bastani et al., 2022 analyze the risk sensitivity of the whole trajectory's reward distribution. In developing and introducing risk-aware objectives in PbRL, we have encountered the following technical challenges in algorithm design and theoretical analysis:

Rewards are defined over trajectories preference In PbRL, the reward function depends on the preference between two trajectories generated by the agent. We refer to this difference in how the reward function is computed as the one-episode-feedback characteristic. Consequently, the risk-aware objectives of standard RL like Du et al. [2022] and Fei et al. [2020] become unmeasurable since they depend on the state-action reward.

Trajectory embedding reward assumption When computing the trajectory reward, it is assumed that an embedding mapping exists. By using the trajectory embedding along with some other vector embedding pointing towards high-rewarding trajectories, the reward is computed with a dot product. Unfortunately, the embedding mapping may not be linear. This means that the embedded trajectory vectors may not follow the Markovian assumption, making the embeddings history-dependent.

Loss of linearity of Bellman function When using a quantile function to transform a risk-neutral PbRL algorithm into a risk-aware algorithm, the Bellman equation used to solve the problem becomes non-linear. This change to the bellman equation disrupts calculations on regret, making risk-neutral PbRL inapplicable. This is primarily due to the additional parameter α , which modifies the underlying distribution.

In this paper, we address these challenges by studying the feasibility of risk-aware objectives in PbRL. We propose a provably efficient algorithm, Risk-Aware-PbRL(RA-PbRL), with theoretical and empirical results on its performance and risk-awareness. Our summary of contributions is as follows:

We analyze the feasibility of several risk-aware measures in PbRL settings and prove that in the one-episode-reward setting, nested and static quantile risk-aware objectives are applicable since they can be solved and computed uniquely in a given PbRL MDP.

We expand the state space in our formulation of a PbRL MDP and modify value iteration to address its history-dependent characteristics from the one-episode setting. These modifications enable us to use techniques like DPP to search for the optimal policy.

Wedevelop a provably efficient (both computationally and statistically) algorithm, RA-PbRL, for nested and static quantile risk-aware objectives. To the best of our knowledge, we are the

first to formulate and analyze the finite time regret guarantee for a risk-aware algorithm with non-Markovian reward models for both nested and static risk-aware objectives. Moreover, we construct a hard-to-learn instance for RA-PbRL to establish a regret lower bound.

2 Related Work

2.1 Preference-based Feedback Reinforcement Learning

The incorporation of human preferences in RL, such as Jain et al. [2013], has been a subject of study for over a decade. This approach has proved to be successful and has been widely used in various applications, including language model training [Ouyang et al., 2022], clinical trials [Coronato et al., 2020], gaming [Silver et al., 2017], and autonomous driving [Basu et al., 2017]. PbRL can be categorized into three distinct types Wirth et al. [2017]: action preference, policy preference, and trajectory preference. Among these, trajectory preference is identified as the most general and widely studied form of preference-based feedback, as evidenced by the rich literature on the topic Chen et al. [2022], Xu et al. [2020a], Wu and Sun [2023]. As noted in our introduction, previous theoretical explorations on PbRL have predominantly aimed at achieving higher average rewards, which encompasses risk-neutral PbRL. We distinguish our work by taking the novel approach of formalizing the risk-aware PbRL problem.

2.2 Risk-aware Reinforcement Learning

In recent years, research on risk-aware RL has proposed various risk measures. Works such as Fei et al. [2020], Shen et al. [2014], Eriksson and Dimitrakakis [2019] integrate RL with a general family of risk-aware utility functions or the exponential utility criterion. Accordingly, studies like Bastani et al. [2022], Wu and Xu [2023] delve into the CVaR measure for the whole trajectory's reward distribution in standard RL. Further, Du et al. [2022] propose ICVaR-RL, a nested risk-aware RL formulation that addresses both regret minimization and best policy identification. Additionally, the work of Chen et al. [2023] presents an advancement in the form of a nested CVaR measure within the framework of RLHF. The limitation of this work lies in the selection of a random reference trajectory for comparison, causing an unavoidable linear strong nested CVaR regret. Consequently, we are left with only a preference equation from which we are unable to compute the state-action reward function for each step.

Practical and relevant trajectory or state-action embeddings are described in works such as [Pacchiano et al., 2021]. Therefore, the one-episode-reward might not even be sum-decomposable (the trajectory embedding details can be seen in sec.3.1 ). Compared to previous work, we use non-Markovian reward models that do not require estimating the reward at each step and explore both nested and static risk-aware objectives, aiming to provide a more general method.

3 Problem Set-up and Preliminary Analysis

3.1 PbRL MDP

We first define a modification of the classical Markov Decision Process (MDP) to account for risk: Risk Aware Preference-Based MDP (RA-PB-MDP). The standard MDP is described as a tuple, M ( S , A , r ⋆ ξ , P ⋆ , H ) , where S and A represent finite state and action spaces, and H denotes the length of episodes. Additionally, let S := |S| and A := |A| denote the cardinalities of the state and action spaces, respectively. P ⋆ : S × A → P ( S ) is the transition kernel, where P ( X ) denotes the space of probability measures on space X . A trajectory is a sequence

Intuitively, a trajectory encapsulates the interactions between an agent and the environment M up to step h . In contrast to the standard RL setting, where the reward function r ⋆ h ( s h , a h ) specifies the reward at each step h , a significant distinction in the PbRL MDP framework is that the reward function r ⋆ is defined as r ⋆ ξ ( ξ H ) : ( S × A ) H -1 × S → [0 , 1] , denoting the reward of the entire trajectory.

Reward model for the entire trajectory. For any trajectory ξ H , we assume the existence of a trajectory embedding mapping ϕ : Z H → R dim T , and the reward of the entire trajectory is defined as the function: r ⋆ ξ ( ξ H ) := ⟨ ϕ ( ξ H ) , w ⋆ r ⟩ . Here, dim T denotes the trajectory embedding dimension. Finally, we denote ϕ ( ξ H ) = ( ϕ 1 ( ξ H ) , . . . , ϕ dim T ( ξ H )) .

Assumption 3.1. We assume that for all trajectories ξ H and for all d ∈ { 1 , . . . , dim T } , ∥ ϕ d ( ξ H ) ∥ ∈ { 0 } ∪ [ b, B ] where b, B > 0 are known. ∥ w ⋆ r ∥ ≤ ρ w and ρ w is known as well.

Remark 3.2 . We assume the map ϕ is known to the learner. The sum of the state-action reward used in Chen et al. [2023] is one case of such map , where ϕ ( ξ H ) = ∑ H h =1 I ( s h , a h ) and dim T = S × A . Therefore, for all d ∈ { 1 , . . . , dim T } , ∥ ϕ d ( ξ H ) ∥ ∈ { 0 } ∪ { 1 , . . . , H }

Remark 3.3 . Assumption 3.1 implies that there is a gap between zero and some positive number b in the absolute values of components of trajectory embeddings. This is evident for finite-step discrete action spaces, where we can enumerate all trajectory embeddings to find the smallest non-zero component, satisfying most application scenarios.

At each iteration k ∈ [ K ] , the agent selects two policies under a deterministic policy framework, π 1 ,k and π 2 ,k , which generate two (randomized) trajectories ξ 1 ,k H and ξ 2 ,k H . In PbRL, unlike standard RL where the agent receives rewards every step, the agent can only obtain the preference o k between two trajectories ( ξ 1 ,k H , ξ 2 ,k H ) . By making a query, we obtain a preference feedback o k ∈ { 0 , 1 } that is sampled from a Bernoulli distribution:

where σ : R → [0 , 1] is a monotonically increasing link function. We assume σ is known like the popular Bradley-Terry model [Hunter, 2004], wherein σ is represented by the logistic function. It is known that we can not estimate the trajectory reward without a known σ . Also, we assume σ is Lipschitz continuous and κ is its Lipschitz coefficient.

History dependent policy. Since the algorithm can only observe the reward for an entire episode until the end, it cannot make decisions based solely on the current state. The agent cannot observe the individual reward r h ( s, a ) and thus cannot compute the target value at each step. To circumvent this challenge, the algorithm should take action according to a history-dependent policy. A historydependent policy Π = { π h } h ∈ [ H ] is defined as a sequence of mappings ψ h : Z h →A , providing the agent with guidance to select an action, given a trajectory ξ h ∈ Z h at time step h . For notation convenience, let Π denote the set of all history-dependent deterministic policies.

3.2 Risk Measure

Because in PbRL we can only estimate the reward for the entire trajectory, the risk measure selected for PbRL must rely solely on the reward of the entire trajectory. That is, two trajectories with the same trajectory reward should contribute equally to the risk measure, even if their potential rewards at each step are different. Unlike Chen et al. [2023] that decomposes the reward at each step (where the solution is likely not unique) and then calculates the risk measure, this requirement ensures that the risk measure consistently and holistically reflects the underlying preference. We refer to risk measures that suitable for PbRL problems as PbRL-risk-measures . Here, we introduce two different risk-measures: nested and static quantile risk-aware measures, which are appropriate for PbRL-MDPs.

We first introduce the definition of quantile function and risk-aware objective. The quantile function of a random variable X is F † X ( τ ) = inf { x ∈ R | F X ( x ) ≥ τ } . We assume F X is strictly monotone, so it is invertible and we have F † X ( τ ) = F -1 X ( τ ) . The risk-aware objective is given by the RiemannStieljes integral:

where X is the random variable encoding the value of MDP, and G is a weighting function over the quantiles. This class captures a broad range of useful objectives, including the popular CVaR objective [Bastani et al., 2022].

Remark 3.4 . ( α -CVaR objective) Specifically, in α -CVaR,

and Φ( X ) becomes

Assumption 3.5. G is L G -Lipschitz continuous for some L G ∈ R > 0 , and G (0) = 0 , G (1) = 1 .

For example, for the α -CVaR objective, we have L G = 1 /α .

There are two prevalent approaches to Risk-aware-MDPs: nested (or iterated) (such as Iterated CVaR (ICVAR) [Du et al., 2022] and Risk-Sensitive Value Iteration (RSVI) [Fei et al., 2020]), and static (referenced in [Bastani et al., 2022, Wu and Xu, 2023]). MDPs characterized by an iterated risk-aware objective facilitate a value function and uphold a Bellman-type recursion.

Nested PbRL-risk-measures. For standard RL's MDP, the nested quantile risk-aware measure can be elucidated in Bellman equation type as follows:

Here r ⋆ h ( s, a ) denotes the decomposed state-action reward in step h .

For the PbRL-MDP setting M ( S , A , r ⋆ ξ , P ⋆ , H ) , the state-action's reward might not be calculated or the reward of the entire trajectory might not be decomposed. Therefore, the policy should be history-dependent. We rewrite the nested quantile objective's Bellman equation with the embedded trajectory reward as follows:

For any PbRL MDP M ( S , A , r ξ , P , H ) , we use ˜ V π,r ξ , P h ( ξ h ) to denote the value iteration under the policy π , where π is a history dependent policy.

Lemma 3.6. For a given tabular MDP, the reward on the entire trajectory can be decomposed as r ⋆ ξ ( ξ H ) = ∑ H h =1 r ⋆ h ( s h , a h ) , V π 1 in Eq. 3 and ˜ V π 1 in Eq. 4 are equivalent.

The proof is detailed in Appendix B.1 due to space limitations.

Static PbRL-risk-measures. Standard MDPs with a static risk aware objective [Bellemare et al., 2017, Dabney et al., 2018] can be written in the distributional Bellman equation as follows:

Where S ( ξ ) = s for ξ = ( . . . , s ) is the current state in trajectory ξ , Ξ h ( ξ H ) = ( s 1 , a 1 , s 2 , a 2 , . . . , s h -1 , a h -1 , s h ) denotes the first h steps' trajectory. Z ( π ) h ( ξ h ) denoted the reward from step t given the current history. The static reward of π is Z ( π ) 1 ( ξ ) , where ξ = ( s ) ∈ Z 1 for s ∼ D is the initial history.

Also, we modify the distributional Bellman equation for PbRL MDP M ( S , A , r ⋆ ξ , P ⋆ , H ) settings as follows:

Lemma 3.7. For a tabular MDP and a reward of the entire trajectory can be decomposed as r ⋆ ξ ( ξ H ) = ∑ H h =1 r ⋆ h ( s h , a h ) , V π 1 in Eq. 5 and ˜ V π 1 in Eq. 6 are equivalent.

The proof is detailed in Appendix B.2 due to space limitation.

Each of these risk measures possesses distinct advantages and limitations. Nested risk measures, which incorporate a Bellman-type recursion, can directly employ techniques such as the Dynamic Programming Principle (DPP) for computation. However, they are challenging to interpret and are not law-invariant [Hau et al., 2023]. On the other hand, static risk measures are straightforward to interpret, but the resulting optimal policy may not remain Markovian and becomes history-dependent. Consequently, techniques such as the DPP and the Bellman equation become inapplicable.

3.3 Objective

We define an optimal policy as:

i.e., it maximizes the given objective for M . ˜ V π 1 ( s 1 ) will be decided by the selected risk measure, where value iteration calculated using Eq. 4 and static calculated using Eq. 6.

Assumption 3.8. Regardless of nested or static CVaR objectives, we are given an algorithm for computing π ⋆ M for a known PbRL-MDP M .

A formal proof of Assumption 3.8 is given in Appendix F. When unambiguous, we drop M and simply write π ⋆ .

At the beginning of each episode k ∈ [ K ] , our algorithm A chooses two policies ( π 1 ,k , π 2 ,k ) = A ( H k ) , where H k = { ξ 1 ,k ′ ,H , ξ 2 ,k ′ ,H , o k } k k ′ =0 is the random set of episodes observed so far. Then, our goal is to design an algorithm A that minimizes regret, which is naturally defined as:

4 Risk Aware Preference based RL Algorithm

In this section, we introduce and analyze an algorithm called RA-PbRL for solving the PbRL problem with both nested and static risk aware objectives. Also, we establish a regret bound for it.

4.1 Algorithm

RA-PbRL is formally described in Algorithm 1. The development of RA-PbRL is primarily inspired by the PbOP algorithm, as delineated in Chen et al. [2022], which was originally proposed for risk-neutral PbRL environments. Building upon this foundation, one significant difference is how to choose a risk aware policy in estimated PbRL MDP, where the value iteration is different. We also use novel techniques to estimate the confidence set and explore for a policy, instead of using a bonus [Chen et al., 2022] (which is difficult to calculate in risk-aware problems) as in standard RL.

Algorithm 1 RA-PbRL

Require: episode K , step H , initial state space P , initial reward space R , risk level α , confidence parameter δ

1: Set B P 0 = P , B r 0 = R , Execute two arbitrary policies π 1 , 0 and π 2 , 0 for one episode, respectively, and then observe the trajectory τ 1 , 0 and τ 2 , 0 and the preference o 0 .

2: for k = 1 · · · K do

Calculate the probability estimation P k : ˆ P k = arg min P ∈P ∑ 2 i =1 ∑ k -1 k ′ =0 ∑ H h =1 |⟨ P ( s i,k ′ ,h , a i,k ′ ,h ) , I ( s i,k ′ ,h +1 ) ⟩| 2 .

3: ˆ

4: Update transition confidence set :

5: Calculate the reward estimation:

ˆ

r

k

(

·

) = arg min

r

∈R

∑

k

k

6: Update the confidence set:

7: Update policy confidence set:

8: Compute ( π , π ) :

9: Observe the trajectory ξ 1 ,k,H , ξ 2 ,k,H , and the preference o k

10: Calculate the state-action visiting time before episode k : n k ( s, a )

11: end for

The overview of the algorithm . Now we introduce the main part of our algorithm. In line 1, we initialize the transition kernel function and reward function confidence set, and execute two arbitrary policies at first. For every episode, we observe history samples and accordingly estimate the transition kernel function (line 3) and update its confidence set (line 4) as well as the reward function (line 5) and its confidence set (line 6). Both estimation and calculation used the standard least-squares regression. Based on the confidence sets, we maintain a policy set in which all policies are near-optimal with minor sub-optimality gap with high probability in line 7. In line 8, we execute the most exploratory policy pair in the policy set and observe the preference between the trajectories sampled using these two policies.

The key difference between nested and static objective. The estimation of the transition kernel (line 4 in Algorithm 1) and the construction of confidence set (line 6 in Algorithm 1) are similar for both nested and static objectives. The difference lies in the value iteration, which is defined in Eq. 4 for nested objective and Eq. 6 for static objective. The bounds for regrets are different since the estimation error's impact is different as we are going to show below.

4.2 Analysis

Theorem 4.1 ( Nested object regret upper bound ) . With at least probability 1 -δ , the nested quantile risk aware object regret of RA-PBRL is bounded by:

-

′

1

=0

(

σ

2

(

r

(

τ

1

,k

′

)

-

r

(

τ

2

,k

′

))

-

o

k

′

)

̸

Where w π,h ( s ) denotes the probability of visiting state-action pair at h th step with policy π and min π,d ω dim,π ( d ) denotes probability of trajectory ξ H 's d th feature Φ d ( ξ H ) = 0 with the policy π .

The proof of this theorem is provided in Appendix D.20. The first term of the regret arises from the estimation error of the transition kernel, primarily dominated by min π,h,s : w π,h ( s ) > 0 w π,h ( s ) . The second term is due to the estimation error of the trajectory reward weights, significantly impacted by min π,d ω π ( d ) . In fact, these factors are unavoidable in the lower bound in certain challenging cases. Thus, they characterize the inherent problem difficulty, i.e., in achieving the nested risk-aware objective, the agent will be highly sensitive to some state-actions or features that are difficult to observe and require substantial effort to explore. This may result in inefficiency in many scenarios.

.

Theorem 4.2 ( Static object regret upper bound ) . The static quantile risk aware object regret of RA-PBRL is bounded by:

The proof of this theorem is provided in Appendix D.21. Notice that the regret for both the nested risk objective and the static risk objective of Algorithm 1 are sublinear with respect to K , making RA-PbRL the first provably efficient algorithm with one-episode-reward for these two objectives. Additionally, compared to Chen et al. [2023], we achieve the goal of having both policies gradually approach optimality. Moreover, in comparison to the nested risk-aware objective, the static objective focuses on the risk measure of the entire distribution, primarily influenced by the Lipschitz coefficient L G of the quantile function and is less constrained by certain specific cases.

Theorem 4.3 ( Nested object regret lower bound ) . The nested quantile risk aware object regret of RA-PBRL is bounded by:

We provide our proof in E.1 by two hard-to-learn constructions. By the two instances, we show that the two factors, min π,h,s : p π,h ( s ) > 0 w π,h ( s ) , min π,d ω dim,π ( d ) , are unavoidable in some cases.

Theorem 4.4 ( Static object regret lower bound ) . The static quantile risk aware object regret of RA-PBRL is bounded by:

The proof of this theorem is similar to Theorem 4.5 in Chen et al. [2022].

5 Experiment Results

In this section, we assess the empirical performance of RA-PbRL (Algorithm 1). For a comparative analysis, we select two baseline algorithms: PbOP, as described in Chen et al. [2022], which is a PbRL algorithm utilizing general function approximation, and ICVaR-RLHF, detailed in Chen et al. [2023], which is a risk-sensitive Human Feedback RL algorithm. These baselines represent the most closely aligned algorithms with RA-PbRL, especially in terms of employing general function approximation. The evaluation of empirical performance is conducted through the lens of static regret, as defined in Eq. 8.

5.1 Experiment settings: MDP

In our experimental framework, we configure a straightforward tabular MDP characterized by finite steps H = 6 , finite actions A = 3 , state space S = 4 , and risk levels α ∈ { 0 . 05 , 0 . 10 , 0 . 20 , 0 . 40 } . For each configuration and algorithms, we perform 50 independent trials and report the mean regret across these trials, along with 95% confidence intervals. The outcomes are depicted in Figures 1 and 3, where the solid lines represent the empirical means obtained from the experiments, and the width of the shaded regions indicates the standard deviation of the experiments.

(a)

α

= 0

.

05

(b)

α

= 0

.

10

(c)

α

= 0

.

20

(d)

α

= 0

.

40

40000

30000

RA-PbRL

PbOP

ICVaR-RLHF

30000

40000

RA-PbRL

PbOP

ICVaR-RLHF

25000

35000

30000

PbOP

ICVaR-RLHF

RA-PbRL

25000

20000

30000

ICVaR-RLHF

RA-PbRL

PbOP

Regret

20000

Regret

20000

&

20000

15000

留

15000

10000

10000

10000

10000

5000

5000

0

0

0

0

0

2000 400060008000 100012000

0

2000400060008000 100012000

0

2000 4000 60008000 100012000

0

20004000 60008000100012000

Episodes

Episodes

Episodes

Episodes

(a)

α

= 0

.

05

(b)

α

= 0

.

10

(c)

α

= 0

.

20

(d)

α

= 0

.

40

40000

ICVaR-RLHF

RA-PbRL

PbOP

40000

RA-PbRL

PbOP

ICVaR-RLHF

40000

ICVaR-RLHF

RA-PbRL

PbOP

30000

35000

RA-PbRL

PbOP

ICVaR-RLHF

30000

30000

30000

25000

Regret

20000

Regret

20000

Regret

20000

20000

15000

10000

10000

10000

10000

5000

0

0

0

2000 4000 60008000 1000 12000

0

2000 40006000 80001000 12000

0

2000 40006000 80001000 12000

0

2000 4000 60008000 100012000

Episodes

Episodes

Episodes

Episodes

5.2 Experiment settings: Half Cheetah

Aditionally, we implement our algorithm to solve MuJoCo's Half-cheetah simulation. We implement our proposed algorithm alongside PbOP and TRC Kim and Oh [2022]. The objective of the algorithms is to learn to control a robot simulation of a cheetah to run forward. This problem differs from the previous setting in that it is more challenging and uses a continuous state-action space. Because the algorithm ws originally implemented for discrete state-action spaces, we assume the transition, reward, and policy functions can be parameterized by a linear function, which is optimized for using gradient descent.

Similarly to the previous experiment, we run both policies in the MuJoCo setting until 1,000 timesteps have passed. We repeat this for 100 episodes, saving the interactions and final preferences based on the cumulative reward after each episode. Finally, we use the data to perform gradient descent for a pre-defined number of repetitions. We repeat the cycle of interaction and training another 100 times. In total, the algorithm sees 10,000,000 timesteps and 10,000 preference reward signals.

The key idea behind the implementation of our algorithm lies in the initial optimization of the learned transition and reward functions using the data collected during the interaction. We iterate through tuples containing the initial state, action taken, and transitioned state. We perform stochastic gradient descent to find the best vectors that parameterize the transition and reward functions to predict the collected data.

After obtaining the best transition and reward functions, we use their parameterization to create a new parameterization of the value function in line 8. In our case, we simply concatenate the vectors parameterizing the transition and reward functions alongside a vector parameterizing the policies. The policy vector used depends on the policy being optimized (the best or exploratory policies.) We then compute the α -CVaR over the preferences obtained using the final cumulative reward. We then optimize the value function parameterization using this as the training data, and perform stochastic gradient descent. To follow the theoretical bounds we establish, we compute the distance between the initial transition and reward parameterizations used in the value function, rolling back the parameterization if the distance is larger than what is established by the theoretical bounds.

5.3 Experimental results

As depicted in Figure 1 and 3, the regret of RA-PbRL over static and nested CVaR initially exhibits a linear growth with respect to K , transitioning to sublinear growth upon reaching a certain threshold. This behavior aligns with the conclusions drawn in Section 4.2. It is important to note that increased

(a)

α

= 0

.

05

(b)

α

= 0

.

10

(c)

α

= 0

.

20

(d)

α

= 0

.

40

0.55

Experiment

CVaR-PbRL (Proposed)

TRC

0.50

PbOP

0.45

CVaR

0.40

0.35

0.30

0

200

400

600

800

1000

Timestep

0.45

Experiment

CVaR-PbRL (Proposed)

TRC

0.40

PbOP

CVaR

0.35

0.30

0.25

0

200

400

600

800

1000

Timestep

Experiment

0.40

CVaR-PbRL (Proposed)

TRC

0.35

PbOP

CVaR

0.30

0.25

0.20

0.15

0

200

400

600

800

1000

Timestep

Experiment

0.35

CVaR-PbRL (Proposed)

TRC

PbOP

0.30

CVaR

0.25

0.20

0.15

0.10

0

200

400

600

800

1000

Timestep

risk aversion ( α → 0 ) introduces greater uncertainty, as evidenced by the larger variance regions observed in the experiments. Notably, the regret of the bad-scenarios associated with RA-PbRL is significantly lower compared to those of other algorithms. It can additionally be observed that as α is increased, the regret improves, which is expected as riskier behavior can also improve the odds of finding useful behaviors.

6 Conclusion

In this paper, we investigate a novel PbRL algorithm for solving problems requiring risk awareness. We explore static and nested measures to introduce risk awareness to PbRL settings. To the best of our knowledge, our proposed RA-PbRL algorithm is the first provably efficient Preference-based Reinforcement Learning (PbRL) algorithm that incorporates both nested and static risk objectives in one algorithm. Our algorithm is built on innovative techniques for the efficient approximation of regret. A core finding in our investigation is the strong influence of the state and trajectory dimensions with respect to the nested risk objective regret. On the other hand, the static risk objective regret is mainly determined by the quantile function.

We have also identified the following four limitations to our work. (1) Our comparison feedback is limited to two trajectories. An interesting, more general approach could consider n-wise comparisons. (2) The reward functions are assumed to be linear in this work for the sake of simplicity. (3) Although this work has considered more general risk measures, we have still made certain assumptions that limit the generality of our results. (4) There is still room for future improvements to further close the gap between upper and lower bounds. We believe this work opens several avenues for future research, including establishing the concrete lower bounds of risk-aware PbRL, improving the computational complexity of the algorithm, and conducting experiments in more diverse and interesting environments/simulations.

References

Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages

463-474. PMLR, 2020.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Osbert Bastani, Jason Yecheng Ma, Estelle Shen, and Wanqiao Xu. Regret bounds for risk-sensitive reinforcement learning. Advances in Neural Information Processing Systems , 35:36259-36269, 2022.

Chandrayee Basu, Qian Yang, David Hungerman, Mukesh Singhal, and Anca D Dragan. Do you want your autonomous car to drive like you? In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction , pages 417-425, 2017.

Nicole Bäuerle and Jonathan Ott. Markov decision processes with average-value-at-risk criteria. Mathematical Methods of Operations Research , 74:361-379, 2011.

Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International conference on machine learning , pages 449-458. PMLR, 2017.

Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preferencebased reinforcement learning: evolutionary direct policy search using a preference-based racing algorithm. Machine learning , 97:327-351, 2014.

Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning , pages 3773-3793. PMLR, 2022.

Yu Chen, Yihan Du, Pihe Hu, Siwei Wang, Desheng Wu, and Longbo Huang. Provably efficient iterated cvar reinforcement learning with function approximation. arXiv preprint arXiv:2307.02842 , 2023.

Antonio Coronato, Muddasar Naeem, Giuseppe De Pietro, and Giovanni Paragliola. Reinforcement learning for intelligent healthcare applications: A survey. Artificial Intelligence in Medicine , 109: 101964, 2020.

Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems , 30, 2017.

Yihan Du, Siwei Wang, and Longbo Huang. Risk-sensitive reinforcement learning: Iterated cvar and the worst path. arXiv preprint arXiv:2206.02678 , 2022.

Hannes Eriksson and Christos Dimitrakakis. Epistemic risk-sensitive reinforcement learning. arXiv preprint arXiv:1906.06273 , 2019.

Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret. Advances in Neural Information Processing Systems , 33:22384-22395, 2020.

Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimization in markov decision processes. Artificial Intelligence , 147(1-2):163-223, 2003.

Jia Lin Hau, Marek Petrik, and Mohammad Ghavamzadeh. Entropic risk optimization in discounted mdps. In International Conference on Artificial Intelligence and Statistics , pages 47-76. PMLR, 2023.

Wassily Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding , pages 409-426, 1994.

David R Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics , 32(1): 384-406, 2004.

Ashesh Jain, Brian Wojcik, Thorsten Joachims, and Ashutosh Saxena. Learning trajectory preferences for manipulators via iterative improvement. Advances in neural information processing systems , 26, 2013.

Dohyeong Kim and Songhwai Oh. Trc: Trust region conditional value at risk for safe reinforcement learning. IEEE Robotics and Automation Letters , 7(2):2621-2628, 2022.

Daniel Lowd and Jesse Davis. Learning markov network structure with decision trees. In 2010 IEEE International Conference on Data Mining , pages 334-343. IEEE, 2010.

R OpenAI. Gpt-4 technical report. arxiv 2303.08774. View in Article , 2:13, 2023.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.

Aldo Pacchiano, Aadirupa Saha, and Jonathan Lee. Dueling rl: reinforcement learning with trajectory preferences. arXiv preprint arXiv:2111.04850 , 2021.

Stuart J Russell and Peter Norvig. Artificial intelligence a modern approach . London, 2010.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

Yun Shen, Michael J Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sensitive reinforcement learning. Neural computation , 26(7):1298-1328, 2014.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Christian Wirth, Johannes Fürnkranz, and Gerhard Neumann. Model-free preference-based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30, 2016.

Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preferencebased reinforcement learning methods. Journal of Machine Learning Research , 18(136):1-46, 2017.

Runzhe Wu and Wen Sun. Making rl with preference-based feedback efficient via randomization. arXiv preprint arXiv:2310.14554 , 2023.

Zhengqi Wu and Renyuan Xu. Risk-sensitive markov decision process and learning under general utility functions. arXiv preprint arXiv:2311.13589 , 2023.

Yichong Xu, Ruosong Wang, Lin Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems , 33:18784-18794, 2020a.

Yichong Xu, Ruosong Wang, Lin F. Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. arXiv preprint arXiv:2006.08910 , 2020b.

Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning , pages 7304-7312. PMLR, 2019.

Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D. Lee, and Wen Sun. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305.14816 , 2023.

A Notation

We clarify the notations that appear uniquely in this paper to avoid confusion.

B Risk Aware Object Computability

Unlike standard RL where each step's reward can be observed, PbRL represents a type of RL characterized by once-per-episode feedback. As a result, our observable and estimable parameters are confined to the trajectory reward r ⋆ ξ and transition probability functions P ⋆ . Consequently, the traditional risk-aware objective might be unsuitable, as the reduction in available information prevents the computation of the original risk-aware measure. The risk measure selected for PbRL must satisfy the following condition: it should remain unique across MDPs where policies, trajectory rewards and transition probability functions even when each trajectory is fixed, but different step rewards, r ⋆ ( s, a ) , vary. This requirement ensures that the risk measure consistently reflects the underlying preferences regardless of variations in specific step rewards.

B.1 Nested Object

Theorem B.1. For the tabular MDP and the reward of the entire trajectory can be decomposed as r ⋆ ξ ( ξ H ) = ∑ H h =1 r ⋆ h ( s h , a h ) , V π 1 in Eq. 5 and ˜ V π 1 in Eq.6 are equivalent.

Proof. Firtly, according to Givan et al. [2003], Lowd and Davis [2010], any tabular MDP can be reformulated as a decision tree-like MDP. Thus, considering a tree-like structure for an MDP implies the following characteristics:

1. The state transition graph of the MDP is connected and acyclic. 2. Each state in the MDP corresponds to a unique node in the tree. 3. There is a single root node from which every other node is reachable via a unique path. 4. The transition probabilities between states follow the Markov property, i.e., the probability of transitioning to any future state depends only on the current state and not on the sequence of events that preceded it.

Formally, let S be the set of states and p ij the transition probabilities between states s i and s j . The transition matrix P for an MDP with a tree-like structure is defined such that:

p ij > 0 if there is an edge between s i and s j in the tree, and p ij = 0 otherwise .

Moreover, for each non-root node s j , there exists exactly one s i such that p ij > 0 , and s i is the unique parent of s j in the tree structure.

To classify the two value iteration in Eq. 3 and Eq. 4, we denote the value given by Eq. 4 as ˜ V π h and the value given by Eq. 3 as V π h , thus, in tabular tree-like MDP with the reward of the entire trajectory

which can be decomposed as r ⋆ ξ ( ξ H ) = ∑ H h =1 r ⋆ h ( s h , a h ) , we have the following relationship:

where r 1: h denotes Reward of the 1 ∼ h steps of a trajectory. We prove this relationship by mathematical induction.

Initial case. Using the tree-like PbRL-MDP algorithm and the initial conditions of the Bellman equation, at the final step h = H , we have

Induction step. We now proved that if holds, then

also holds.

Since this tree-like MDP's policy π is fixed, it has only one path to arrive h th state ( s h ), denoted as:

Therefore, r ⋆ 1: h -1 is unique.

By applying conclusion, we observe that when h = 1

Thus, we have proven that the for the tabular MDP and the reward of the entire trajectory can be decomposed as r ⋆ ξ ( ξ H ) = ∑ H h =1 r ⋆ h ( s h , a h ) , V π 1 in Eq. 3 and Eq. 4 are equivalent.

B.2 Static Object

Lemma B.2. For the tabular MDP and the reward of the entire trajectory can be decomposed as r ⋆ ξ ( ξ H ) = ∑ H h =1 r ⋆ h ( s h , a h ) , V π 1 in Eq. 5 and ˜ V π 1 in Eq.6 are equivalent.

Proof. To classify the two value iteration in Eq. 5 and Eq. 6, we denote the value given by Eq. 5 as ˜ V π h and the value given by Eq. 6 as V π h , thus, in tabular tree-like MDP with the reward of the entire trajectory which can be decomposed as r ⋆ ξ ( ξ H ) = ∑ H h =1 r ⋆ h ( s h , a h ) , we have the following relationship:

where r 1: h denotes Reward of the 1 ∼ h steps of a trajectory.

Now, We prove this relationship.

Since this tree-like MDP's policy π is fixed, it has only one path to arrive h th state ( s h ), denoted as:

Therefore, r ⋆ 1: h -1 is unique. By definition,

By applying conclusion, we observe that when h = 1

Thus, we have proven that the for the tabular MDP and the reward of the entire trajectory can be decomposed as r ⋆ ξ ( ξ H ) = ∑ H h =1 r ⋆ h ( s h , a h ) , V π 1 in Eq. 5 and ˜ V π 1 in Eq.6 are equivalent.

C Difference between nested and static risk measure

To explain the difference between nested and static risk measure, we present a simple example that demonstrates their characters.

π

A

S

2,1

S

2,2

P

= 0.5

P

= 0.5

S

3,1

S

3,2

P

= 0.9

P

= 0.1

S

3,3

S

3,4

P

= 0.1

P

= 0.9

Action a

2

r

2

= 0.1

r

2

= 0.2

S

1

Action a

1

r

1

= 0

r

3

= 0.3

r

3

= 0.2

r

3

= 0.5

r

3

= 0.4

0.4

0.3

0.6

0.7

Accumulativ

Probability

0.45

0.

0

5

0.

0

5

0.45

e

Reward

Action a

′

3

(a) policy A

π

B

S

2,1

S

2,2

P

= 0.5

P

= 0.5

S

3,1

S

3,2

P

= 0.9

P

= 0.1

S

3,3

S

3,4

P

= 0.1

P

= 0.9

Action a

2

r

2

= 0.1

r

2

= 0.2

S

1

Action a

1

r

1

= 0

r

3

= 0.6

r

3

= 0.2

r

3

= 0.2

r

3

= 0.4

0.7

0.3

0.6

0.4

Accumulativ

Probability

0.45

0.

0

5

0.

0

5

0.45

e

Reward

Action a

′

3

(b) policy B

First, we construct a MDP instance in fig. 4 where two policies exhibit identical reward distributions and consequently demonstrate equivalent preference will be observed . However, within this instance, these policies yield different outcomes under the nested and static CVaR metrics.

The state space is S = { S 1 , S 2 , 1 , S 2 , 2 , S 3 , 1 , S 3 , 2 , S 3 , 3 , S 3 , 4 } , where S 1 is the initial state.

The policy space is Π = { π A , π B } . Both policy have have the same action in the first step ( a 1 ) and second step ( a 2 ) , but have the different action in the third step ( a 3 , a ′ 3 ) .

The reward functions are as follows. r ( S 1 , a 1 ) = 0 , r ( S 2 , 1 , a 2 ) = 0 . 1 , r ( S 2 , 2 , a 2 ) = 0 . 2 , r ( S 3 , 1 , a 3 ) = 0 . 3 , r ( S 3 , 2 , a 3 ) = 0 . 2 , r ( S 3 , 3 , a 3 ) = 0 . 4 , r ( S 3 , 2 , a 3 ) = 0 . 5 , r ( S 3 , 1 , a ′ 3 ) = 0 . 6 , r ( S 3 , 2 , a ′ 3 ) = 0 . 2 , r ( S 3 , 3 , a ′ 3 ) = 0 . 4 , r ( S 3 , 2 , a ′ 3 ) = 0 . 2 .

The transition distributions are as follows. P ( S 2 , 1 | S 1 , a 1 ) = 0 . 5 , P ( S 2 , 2 | S 1 , a 1 ) = 0 . 5 , P ( S 3 , 1 | S 2 , 1 , a 2 ) = 0 . 1 , P ( S 3 , 2 | S 2 , 1 , a 2 ) = 0 . 9 , P ( S 3 , 3 | S 2 , 2 , a 2 ) = 0 . 1 , P ( S 3 , 4 | S 2 , 2 , a 2 ) = 0 . 9 .

As depicted on the right side of the figure, the distribution of rewards is consistent. Consequently, the human feedback preferences for the two policies are identical. We list the differing risk measures for

these two policies in Table C.

D REGRET UPPER BOUND FOR ALGORITHM 1

In this section, we present the full proof of Assumption 1's regret upper bound. The proof consists of parts.

D.1 REWARD ESTIMATION ERROR

Lemma D.1. Reward confidence set construction. Fix δ ∈ (0 , 1) , with probability at least 1 -δ , for all k ∈ [ K ] ,

where

Proof. This lemma can be proved by the direct application of Lemma G.5. Let X t = ( ξ t, 1 , ξ t, 2 ) and Y t = o t , and F r,t = { f r | f r ( · , · ) = σ ( r ( · ) -r ( · )) } . Then, we have that X t is F t -1 measurable and Y t is F t measurable. According to Hoeffding's inequality (Theorem G.8) , { Y t -f r ( X t ) } is 1 2 -sub-gaussian conditioning, and E [ Y t -f r ( X t ) | F t -1 ] = 0 . By Lemma G.5 and G.4 , since the linear trajectory embedding function ϕ : Z H → R dim T , with probability at least 1 -δ , we have

LemmaD.2. Reward estimation error of trajectory embedding . For any k ∈ [0 , . . . , K ] , reward confidence set B r k , where the reward function embedding weight can be noted as w r = ( w 1 , . . . , w dim T ) , any fixed trajectory ξ H ∈ Z H , trajectory embedding ϕ ( ξ H ) = ( ϕ 1 , . . . , ϕ dim T ) , with probability at least 1 -δ , it holds that,

where n ξ,k denotes the number of times ξ H was visted up to episode k.

Proof. According to Lemma D.1 and the assumption of link function, for fixed k ∈ [0 , . . . , K ] , and d ∈ [0 , . . . , dim T ] , we have,

̸

̸

Where n dim,k ( d ) denotes the number of ϕ d ( ξ H ) = 0 among 1 ∼ k episode's trajectory.

Using Cauchy-Schwarz inequality and Lemma G.6, we have,

Lemma D.3. Reward estimation error of the whole distribution.

Proof. Let F k = { f r | f r ( x, y ) = σ ( r ( x ) -r ( y )) , r ∈ B r k } , then we define diam ( F B r k ) = b σ k ( x, y ) = max r 1 ,r 2 ∈B r k σ ( r 1 ( x ) -r 1 ( y )) -σ ( r 2 ( x ) -r 2 ( y )) . According to Lemma. D.1, δ k = max 1 ≤ k ≤ K diam ( F k | x 1: K ) . Let α = 1 /K , T = k , and d = dim E ( F R , 1 /K ) According to lemma G.6,

Lemma D.4. Transition estimation error of fixed state action pair .For any fixed k , with probability at least 1 -2 δ , for any ( s, a ) ∈ S × A .

Proof. The proof is same as Eq. (55) in Zanette and Brunskill [2019].

Lemma D.5. Transition estimation error of whole distribution .For any fixed k , with probability at least 1 -δ , for any ( s, a ) ∈ S × A , excuted policy π 1 , π 2 ∈ Π k and any transition possibility kernel P 1 , P 2 ∈ B P k

Proof. The proof is same as Lemma. A.5 in Chen et al. [2022].

Lemma D.6. For any iteration value V : S → R , any two transition possibility kernel P , ˆ P : S × S × A → , and the risk aware object form:

Proof. We firstly sort all successor states s ′ ∈ S by V ( s ′ ) in ascending order (from the left to the right) as s ′ 1 , s ′ 2 . . . s ′ S . And we assume that V ( s ′ S +1 = 1) . Thus, according to the quantile function's definition,

The second to fourth lines follow the discussion on the equivalent expression of the discontinuous distribution Φ in Lemma 5.1 of Bastani et al. [2022]. The fifth line is derived from the properties of Lipschitz functions and Assumption 3.5.

Since we use the emperical estimation ˆ P k of the transaction kernel P ⋆ ,

Lemma D.7. Concentration for V.

With probability at least 1 -2 δ , it holds that, for any k ∈ [ K ] , ( s, a ) ∈ S × A , any transition possibility kernel P 1 ∈ B P k and function V : S ↦→ [0 , H ] ,

Here n k ( s, a ) is the number of times ( s, a ) was visited up to episode k .

Proof. According to Lemma D.4 and recall the definition of the transition possibility confidence set, with probability at least 1 -2 δ ,

(49)

Plugging Lemma .D.6, we obtain that with probability at least 1 -2 δ , for any k ∈ [ K ] , ( s, a ) ∈ S ×A and function V : S ↦→ [0 , H ] ,

Recall the definition of quantile function Φ , G is the quantile CDF weight. Given any target value V : S → R , we use β G,V P ⋆ ( s ′ | s, a ) denotes the conditional probability of transitioning to s ′ from ( s, a ) , conditioning on transitioning to the quantile distribution, and it holds that

Lemma D.8. Quantile Reward Gap due to Value Function Shift. For any ( s, a ) ∈ S × A , distribution P , and functions V, V ′ : S ↦→ [0 , H ] , for any s ′ ∈ S ,

This Lemma's proof is similar to Lemma 11 in Du et al. [2022].

Lemma D.9. For any ( s, a ) ∈ S × A , distribution P , and functions V, V ′ : S ↦→ [0 , H ] , for any s ′ ∈ S ,

Proof. This Lemma comes from Lemma .D.8 and

For any k > 0 , h ∈ [ H ] and ( s, a ) ∈ S × A , let p kh ( s, a ) denote the probability of visiting ( s, a ) at step h of episode k . Then, it holds that for any k > 0 , h ∈ [ H ] and ( s, a ) ∈ S × A , p kh ( s, a ) ∈ [0 , 1] and ∑ ( s,a ) ∈S×A p kh ( s, a ) = 1

Lemma D.10. (Concentration of state-action visitation). It holds that

This Lemma is a direct application of Lemma G.9 and same as Du et al. [2022].

Lemma D.11. (Concentration of trajectory visitation). It holds that

Proof. This Lemma a direct application of G.9. For any dimension d ∈ [0 , . . . , dim T ] , it holds that

Since d ∈ [0 , . . . , dim T ] , Therefore,

Definition of sufficient state-action visitations .Following Zanette and Brunskill [2019], for any episode k > 0 , we define the set of state-action pairs which have sufficient visitations in expectation as follows.

Definition of sufficient trajetory visitations . Following Zanette and Brunskill [2019], for any episode k > 0 , we define the set of trajrctory dimension which have sufficient visitations in expectation as follows.

̸

We use n dim,k ( d ) to denote the number of ϕ d ( ξ H ) = 0 among 1 ∼ k episode's trajectory.

Lemma D.12. (Standard state action visitation ratio). For any K > 0 , we have

This proof is the same as that of Lemma 13 in Zanette and Brunskill [2019].

Lemma D.13. (Standard trajectory visitation ratio). For any K > 0 , we have

This proof is the same as that of Lemma 13 in Zanette and Brunskill [2019].

Lemma D.14. (Standard Invisitation Ratio). For any K > 0 , we have

This proof is the same as that of Lemma 10 in Du et al. [2022].

Lemma D.15. (Standard trajectory Invisitation Ratio). For any K > 0 , we have )

This proof is the same as that of Lemma 10 in Du et al. [2022].

Lemma D.16. For any functions V : S ↦→ R , k > 0 , h ∈ [ H ] and ( s, a ) ∈ S × A such that p kh ( s, a ) > 0 ,

where p G,V kh ( s, a ) denotes the conditional probability of visiting ( s, a ) at step h of episode k , conditioning on transitioning distortion by the quantion function G which works on at each step h ′ = 1 , . . . , h -1 .

Proof. Since p G,V kh ( s, a ) is the conditional probability of visiting ( s, a ) , we have p Ga,V kh ( s, a ) ∈ [0 , 1] . Since p kh ( s, a ) is the probability of visiting ( s, a ) at step h under policy π k and min π,h, ( s,a ): w π,h ( s,a ) > 0 w π,h ( s, a ) is the minimum probability of visiting any reachable ( s, a ) at any step h over all policies π , we have

Hence, we have

Thus,

Lemma D.17. For any functions V : S ↦→ R , k > 0 , h ∈ [ H ] ,

where p G,V dim,k ( d ) denotes the conditional probability of ϕ d ( ξ H ) = 0 of episode k , conditioning on transitioning distortion by the quantion function G which always works on at each step h = 1 , . . . , H .

The proof is similar to Lemma .D.16.

D.2 Proof of Algorithm 1's regret upper bound

Lemma D.18. For a probability of 1 -2 δ , it holds that π ⋆ ∈ Π k , which is calculated as follows:

Proof. It equals to for any policy π 0 ∈ Π ,

According to Lemma D.1 and Lemma D.4, with at least possibility 1 -2 δ , it holds that:

Where the last equation comes from the definition of optimal policy π ⋆ (Eq. 7).

Lemma D.19. Given a positive constant δ ∈ (0 , 1] , with probability at least 1 -4 Kδ , we have the following inequality holds:

Proof.

̸

Where (a) comes from the definition of the optimal policy confidence set (Lemma D.18) when π ⋆ ∈ Π k at each episode , (b) derives from the characters of max value.

D.3 Nested Regret

Lemma D.20. Nested regret upper bound. Given a positive constant δ ∈ (0 , 1] , with probability at least 1 -δ , we have the following inequality holds for every k ∈ [ K ] .

Proof. We use V π,h 1 ,r ⋆ ,p ( s k, 1 ) to denote that the first h steps in the trajectory ξ is sampled using policy π from the MDP with transition ̂ P , and the state-action pairs from step h +1 up until the last step is sampled using policy π from the MDP with the true transition kernal P ⋆ . Therefore,

Here (a) is due to Lemma D.7 and D.8. (b) comes from that p CVaR,α,V π k kh ( s, a ) is defined as the probability of visiting ( s, a ) at step h of episode k under the conditional transition probability β α,V π k h ′ +1 ( · | · , · ) for each step h ′ = 1 , . . . , h -1 .

Firstly, we analyze the term I 1 and I 5 . Recall that for any policy π, h ∈ [ H ] and ( s, a ) ∈ S × A , w π,h ( s, a ) and w π,h ( s ) denote th probabilities of visiting ( s, a ) and s at step h under policy π , respectively. Thus, we have:

̸

̸

̸

Here (a) is due to Cauchy-Schwartz inequality and the fact that for any k > 0 and h ∈ [ H ] , ∑ ( s,a ) ∈S×A p CVaR ,α,V π k kh ( s, a ) = 1 . (b) comes from Lemma D.16. (c) uses Lemma D.12 and the fact that for any deterministic policy π, h ∈ [ H ] and ( s, a ) ∈ S × A , we have either w π,h ( s, a ) = w π,h ( s ) or w π,h ( s, a ) = 0 , and thus min π,h, ( s,a ): w π,h ( s,a ) > 0 w π,h ( s, a ) = min π,h,s : w π,h ( s ) > 0 w π,h ( s ) .

For the term I 2 and I 6 , using Lemma D.14, we have:

For the term I 3 and I 7 , we have:

̸

Here (a) is due to Cauchy-Schwartz inequality. (b) comes from Lemma D.17. (c) uses Lemma D.15. For the term I 4 and I 8 , using Lemma D.15, we have:

Then, summing all the term, we have:

D.4 Static Regret

Lemma D.21. Static regret upper bound. Given a positive constant δ ∈ (0 , 1] , with probability at least 1 -δ , we have the following inequality holds for every k ∈ [ K ] .

Proof. We use V π,h 1 ,r ⋆ ,p ( s k, 1 ) to denote that the first h steps in the trajectory ξ is sampled using policy π from the MDP with transition ̂ P , and the state-action pairs from step h +1 up until the last step is sampled using policy π from the MDP with the true transition kernal P ⋆ . Therefore,

E Lower bound of the regret.

= 0.8

S

1

S

2

/uni22EF

S

n

-1

S

n

x

2

x

3

x

1

μ

μ

μ

μ

1 - 3

μ

1 -

μ

1 -

μ

1 -

α

α

Reward

= 1

Reward

Reward

= 0.2

μ

μ

π

⋆

(a) optimal policy

S

1

S

2

/uni22EF

S

n

-1

S

n

x

2

x

3

x

1

μ

μ

μ

μ

1 - 3

μ

1 -

μ

1 -

μ

α

-

η

1 -

α

+

η

Reward

= 1

Reward

Reward

= 0.2

μ

μ

π

∈ Π\

π

⋆

(b) sub optimal policy

E.1 Regret Lower Bound of Nested Reward

Lemma E.1. Nested Regret Lower Bound. There exists an instance of Nested CVaR RL-RM, where the regret of any algorithm is at least:

Proof. Hard to learn case 1. Consider the instance shown in Figure 5.The state space is defined as S = { s 1 , s 2 , . . . , s n , x 1 , x 2 , x 3 } , where s 1 is the initial state, and n = S -3 < S . We define the trajectory reward functions as follows: For any ξ H ∈ Z H , ϕ ( ξ H ) = ( I ( S H ( ξ H ) = x 1 ) B, I ( S H ( ξ H ) = x 2 ) B, I ( S H ( ξ H ) = x 3 ) B ) , w r = (1 ρ, 0 . 8 ρ, 0 . 2 ρ )

The transition distributions are as follows. Let µ be a parameter which satisfies that 0 < α < µ < 1 3 . For any a ∈ A , p ( s 2 | s 1 , a ) = µ, p ( x 1 | s 1 , a ) = 1 -3 µ, p ( x 2 | s 1 , a ) = µ and p ( x 3 | s 1 , a ) = µ . For any i ∈ { 2 , . . . , n -1 } and a ∈ A , p ( s i +1 | s i , a ) = µ and p ( x 1 | s i , a ) = 1 -µ.x 1 , x 2 and x 3 are absorbing states, i.e., for any a ∈ A , p ( x 1 | x 1 , a ) = 1 , p ( x 2 | x 2 , a ) = 1 and p ( x 3 | x 3 , a ) = 1 . p ( x 2 | s n , a J ) = 1 -α + η and p ( x 3 | s n , a J ) = α -η , where η is a parameter which satisfies 0 < η < α and will be chosen later. For any suboptimal action a ∈ A\{ a J } , p ( x 2 | s n , a ) = 1 -α and p ( x 3 | s n , a ) = α For any a j ∈ A , let E j [ · ] and Pr j [ · ] denote the expectation and probability operators under the instance with a J = a j . Let E unif [ · ] and Pr unif [ · ] denote the expectation and probability operators under the uniform instance.

Fix an algorithm A . Let π k denote the policy taken by algorithm A in episode k . Let N s n ,a j = ∑ K k =1 ⊮ { π k ( s n ) = a j } denote the number of episodes that the policy chooses a j in state s n . Let V s n ,a j denote the number of episodes that the algorithm A visits ( s n , a j ) . Let w ( s n ) denote the probability of visiting s n in an episode (the probability of visiting s n is the same for all policies). Then, it holds that E [ V s n , a j ] = w ( s n ) · E [ N s n ,a j ] .

According to the definition of nested-CVaR-PbRL risk objective in Eq. 4, we have:

and summing over all episodes k ∈ [ K ] , we have

= 0.8

Therefore, we have

Thus, we have :

Let η = c 2 √ αA w ( s n ) K for a small enough constant c 2 . We have

Hard to learn case 2.

The trajectory reward functions are as follows. For any ξ H ∈ Z H , ϕ ( ξ H ) = ( I ( S H ( ξ H ) = x 1 ) B, I ( S H ( ξ H ) = x 2 ) B, I ( S H ( ξ H ) = x 3 ) B, I ( S H ( ξ H ) = x 4 ) B ) , w r = (1 ρ, 0 . 8 ρ, 0 . 2 ρ, (0 . 2 -η ) ρ )

The transition distributions are as follows. Let µ be a parameter which satisfies that 0 < α < µ < 1 3 . For any a ∈ A , p ( s 2 | s 1 , a ) = µ, p ( x 1 | s 1 , a ) = 1 -3 µ, p ( x 2 | s 1 , a ) = µ and p ( x 3 | s 1 , a ) = µ . For any i ∈ { 2 , . . . , n -1 } and a ∈ A , p ( s i +1 | s i , a ) = µ and p ( x 1 | s i , a ) = 1 -µ.x 1 , x 2 and x 3 are absorbing states, i.e., for any a ∈ A , p ( x 1 | x 1 , a ) = 1 , p ( x 2 | x 2 , a ) = 1 and p ( x 3 | x 3 , a ) = 1 .

For any j ∈ { A,B } , using Pinsker's inequality and 0 < α < 1 3 , we have that KL( p unif ( s n , π j ( s n )) ∥ p j ( s n , π j ( s n ))) = KL(Ber( α ) ∥ Ber( α -η )) ≤ η 2 ( α -η )(1 -α + η ) ≤ c 1 η 2 α for some constant c 1 and small enough η . Then, using Lemma A. 1 in Du et al. [2022], we have ,

Then, Using ∑ A j =1 E unif [ N s n ,a j ] = 2 K and Cauchy-Schwarz inequality, we have:

Thus,

Therefore, we have

= 0.8

S

1

S

2

/uni22EF

S

n

-1

S

n

x

2

x

3

x

1

μ

μ

μ

μ

1 - 3

μ

1 -

μ

1 -

μ

1 -

α

α

Reward

= 1

Reward

Reward

= 0.2

μ

μ

π

⋆

(a) optimal policy

S

1

S

2

/uni22EF

S

n

-1

S

n

x

2

x

4

x

1

μ

μ

μ

μ

1 - 3

μ

1 -

μ

1 -

μ

1 -

α

α

Reward

= 1

Reward

Reward

= 0.2 -

η

μ

μ

π

∈ Π\

π

⋆

(b) sub optimal policy

p ( x 2 | s n , a J ) = 1 -α and p ( x 3 | s n , a J ) = α , where η is a parameter which satisfies 0 < η < α and will be chosen later. For any suboptimal action a ∈ A\{ a J } , p ( x 2 | s n , a ) = 1 -α and p ( x 4 | s n , a ) = α For any a j ∈ A , let E j [ · ] and Pr j [ · ] denote the expectation and probability operators under the instance with a J = a j . Let E unif [ · ] and Pr unif [ · ] denote the expectation and probability operators under the uniform instance.

According to the definition of nested-CVaR-PbRL risk objective in 4, we have:

For any j ∈ { A,B } , since the preference based on Bernoulli distribution, using Pinsker's inequality and 0 < α < 1 3 , we have that KL( p unif ( s n , π j ( s n )) ∥ p j ( s n , π j ( s n ))) = KL(Ber( α ) ∥ Ber( α -η )) ≤ η 2 (0 . 2 -η )(0 . 8+ η ) ≤ c 1 η 2 0 . 16 for some constant c 1 and small enough η . Then, we have ,

Then, Using ∑ A j =1 E unif [ N s n ,a j ] = 2 K and Cauchy-Schwarz inequality, we have:

= 0.8

Thus, we have :

Let η = c 2 √ 0 . 16 A w ( d ) K for a small enough constant c 2 . We have

F The optimal policy calculation for known PbRL MDP

In this section, we describe how to compute the optimal policy for the CVaR objective when the PbRL-MDP is known; this approach is described in detail in Bäuerle and Ott [2011], Bastani et al. [2022]. Following this work, we consider the setting where we are trying to minimize cost rather than maximize reward. In particular, consider an know PbRL MDP M ( S , A , r ⋆ ξ , P ⋆ , H ) , where S and A , and our goal is to compute a policy π that maximizes its CVaR objective.

Lemma F.1. CVaR definition. For any random variable Z, we have

where the minimum is achieved by ρ ∗ = VaR α ( Z ) .

F.1 Static CVaR object

Since the optimal policy for static CVaR object satisfies:

As a consequence of Lemma F.1, we have

Thus, the optimal policy is:

where

Value iteration. we reconstruct the MDP as enlarge the state space ˜ s h = ( ξ h , ρ ) , where ρ will work as a quantile value. Letting S 1 be the initial state of the original PbRL MDP M .

We iterate the value and calculate the policy as follows:

Then, given an initial state s 1 , we construct state ℑ 1 = ( s 1 , -ρ ∗ ) , where

and then acting optimally in M .

F.2 Nested CVaR object

According to Eq. 4, nested CVaR object could directly use the Bellman equation to iterate the value.

Value iteration. we reconstruct the MDP as enlarge the state space ˜ s h = ( ξ h , ρ ) , where ρ will work as a quantile value. Letting S 1 be the initial state of the original PbRL MDP M .

We iterate the value and calculate the policy as follows:

then acting optimally in M .

G AUXILIARY LEMMAS

Definition G.1. α -dependence in Russo and Van Roy [2013] . For α > 0 and function class Z whose elements are with domain X , an element x ∈ X is α -dependent on the set X n := { x 1 , x 2 , · · · , x n } ⊂ X with respect to Z , if any pair of functions z, z ′ ∈ Z with √ ∑ n i =1 ( z ( x i ) -z ′ ( x i )) 2 ⩽ α satisfies z ( x ) -z ′ ( x ) ⩽ α . Otherwise, x is α -independent on X n if it does not satisfy the condition.

Definition G.2. Eluder dimension in Russo and Van Roy [2013] . For α > 0 and function class Z whose elements are with domain X , the Eluder dimension dim E ( Z , α ) , is defined as the length of the longest possible sequence of elements in X such that for some α ′ ⩾ α , every element is α ′ independent of its predecessors.

Definition G.3. Covering number Given two functions l and u , the bracket [ l, u ] is the set of all functions f satisfying l ≤ f ≤ u . An α -bracket is a bracket [ l, u ] with ∥ u -l ∥ <α . The covering number N [ · ] ( F , α, ∥ · ∥ ) is the minimum number of α -brackets needed to cover F .

Lemma G.4. (Linear Preference Models Eluder dimension and Covering number). For the case of d -dimensional generalized trajectory linear feature models r ξ ( ξ H ) := ⟨ ϕ ( ξ H ) , w r ⟩ , where ϕ : Traj → R dim T is a known dim T dimension feature map satisfying ∥ ψ ( ξ H ) ∥ 2 ≤ B and θ ∈ R d is an unknown parameter with ∥ w r ∥ 2 ≤ ρ w . Then the α -Eluder dimension of r ξ ( ξ H ) is at most

Let ( X p , Y p ) p =1 , 2 ,... be a sequence of random elements, X p ∈ X for some measurable set X and Y p ∈ R . Let F be a subset of the set of real-valued measurable functions with domain X . Let F = ( F p ) p =0 , 1 , ··· be a filtration such that for all p ≥ 1 , ( X 1 , Y 1 , · · · , X p -1 , Y p -1 , X p ) is F p -1 measurable and such that there exists some function f ⋆ ∈ F such that E [ Y p | F p -1 ] = f ∗ ( X p ) holds for all p ≥ 1 . The (nonlinear) least square predictor given ( X 1 , Y 1 , · · · , X t , Y t ) is ˆ f t = argmin f ∈F ∑ t p =1 ( f ( X p ) -Y p ) 2 . Wesay that Z is conditionally ρ -subgaussion given the σ -algebra F is for all λ ∈ R , log E [exp( λZ ) | F ] ≤ 1 2 λ 2 ρ 2 . For α > 0 , let N α be the ∥ · ∥ ∞ -covering number of F at scale α . For β > 0 , define

Lemma G.5. (Theorem 5 of Ayoub et al. [2020]). Let F be the filtration defined above and assume that the functions in F are bounded by the positive constant C > 0 . Assume that for each s ≥ 1 , ( Y p -f ∗ ( X p )) is conditionally σ -subgaussian given F p -1 . Then, for any α > 0 , with probability 1 -δ , for all t ≥ 1 , f ∗ ∈ F t ( β t ( δ, α )) , where

Lemma G.6. (Lemma 5 of Russo and Van Roy [2013]). Let F ∈ B ∞ ( X,C ) be a set of functions bounded by C > 0 , ( F t ) t ≥ 1 and ( x t ) t ≥ 1 be sequences such that F t ⊂ F and x t ∈ X hold for t ≥ 1 . Let F| x 1: t = { ( f ( x 1 ) , . . . , f ( x t )) : f ∈ F} ( ⊂ R t ) and for S ⊂ R t , let diam( S ) = sup u,v ∈ S ∥ u -v ∥ 2 be the diameter of S . Then, for any T ≥ 1 and α > 0 it, holds that

where δ T = max 1 ≤ t ≤ T diam ( F t | x 1: t ) and d = dim E ( F , α ) .

Lemma G.7. If ( β t ≥ 0 | t ∈ N ) is a nondecreasing sequence and F t := { f ∈ F : ∥ ∥ ∥ f -ˆ f LS t ∥ ∥ ∥ 2 ,E t ≤ √ β t } , where ˆ f LS t ∈ arg min f ∈F L 2 ,t ( f ) and L 2 ,t ( f ) = ∑ t -1 1 ( f ( A t ) -R t ) 2 , then for all T ∈ N and ϵ > 0 ,

where w F ( a ) := sup f ∈F f ( a ) -inf f ∈F f ( a ) denotes confidence interval widths.

Theorem G.8. Hoeffding's inequality [Hoeffding, 1994]. Let X 1 , X 2 , . . . , X n be independent random variables that are sub-Gaussian with parameter σ . Define S n = ∑ n i =1 X i . Then, for any t > 0 , Hoeffding's inequality provides an upper bound on the tail probabilities of S n , which is given by:

This result emphasizes the robustness of the sum S n against deviations from its expected value, particularly useful in applications requiring high confidence in estimations from independent subGaussian observations.

Lemma G.9. (Lemma F.4. in Dann et al. [2017]) Let F i for i = 1 . . . be a filtration and X 1 , . . . X n be a sequence of Bernoulli random variables with P ( X i = 1 | F i -1 ) = P i with P i being F i -1 -measurable and X i being F i measurable. It holds that

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contribution and scope: RA-PbRL algorithm.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discussed the limitations of the work in section 3.2, and section 6. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The paper provided the full set of assumptions and a complete proof in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental results presented in the paper are reproducible. We have provided detailed pseudocode and full information for implementing the code, allowing any researcher to easily implement our algorithm and benchmark it against other algorithms.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a link to our code repository.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experimental details are provided in section 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: the results are accompanied standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computer resources needed to implement experiments are provided in the section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: the paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subject.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subject.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies and proves the applicability of two risk-aware objectives to Preference-Based Reinforcement Learning (PbRL), i.e., iterated and accumulated quantile risk objectives. The authors design an algorithm called Risk-Aware-PbRL (RA-PbRL), which can optimize both iterated and accumulated objectives. Furthermore, the authors provide a theoretical analysis of the regret bounds. The results demonstrate that the regret bounds of algorithm RA-PbRL under both the iterated and accumulated objectives are sublinear with respect to the number of episodes. Finally, the authors present empirical results to support their theoretical findings.\n\nSTRENGTHS:\n1.\tThe studied problem, i.e., applying the iterated and accumulated risk-aware objectives to PbRL, is relevant and useful for some applications such as healthcare and AI systems. The considered reward model depends on the feature of the trajectory, instead of the feature of the state-action pair, which is more general than the prior works.\n2.\tThe authors consider both the iterated and accumulated risk-aware objectives, which encompass the popular CVaR objective. In addition, the authors also design an algorithm and provide regret analysis for both objectives. The algorithm design and theoretical analysis are well executed.\n\nWEAKNESSES:\n1.\tThe proposed algorithm is very straightforward, which seems to simply combine the confidence set construction and the risk-aware objective. This algorithm is computationally inefficient and hard to implement in practice. Can the authors explain more on how to implement this algorithm?\n2.\tWhat is the intuition behind the factor $\\min_{\\pi,d} \\omega_{\\pi}(d)$ in Theorem 4.1? In particular, why the probability that the feature is not zero will influence the regret? More discussions on the regret due to reward estimation are needed.\n3.\tIt seems that $L_G$ appears in the upper bound (Theorem 4.2) but not in the lower bound (Theorem 4.4). Why the authors said that “it demonstrates the significant impact of LG” below Theorem 4.4? In addition, it seems that $dim_{T}$ appears in the lower bound, but not in the upper bound. Can the authors explain more on it?\n4.\tThis paper needs careful proof-reading. There are many typos. For example, the factor $\\min_{\\pi,d} \\omega_{\\pi}(d)$ in Theorems 4.1 and 4.3, and the $O(…)$ notation in Theorems 4.1 and 4.2. In Line 130, “use” should be “used”. In Line 197, the “and” should be moved to the front of “$V^{\\pi}_i$ in Eq.5 …”?\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nPlease see the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper incorporates risk-awareness into Preference-based Reinforcement Learning (PbRL). Specifically, it tackles the issue that under PbRL, the reward is episodic, meaning that it can only be computed on full trajectories. The authors adapt both iterative and accumulated quantile risk objectives to deal with episodic rewards.

Additionally, the paper presents an algorithm (RA-PbRL) to incorporate these objectives into PbRL.

Lastly, the authors provide regret boundaries for both iterative and accumulated quantile risk objectives with RA-PbRL.

STRENGTHS:
[Quality, Clarity] The paper is well written (particularly the introduction). As far as I could follow the mathematical development is robust, with regret upper- and lower-bounds being established for both types of risk considered.

WEAKNESSES:
*  **W1** [Quality]: As the manuscript already mentions, the experimental setting is very simple. It would have been interesting to train more complex settings, particular from actual human preferences.
 *  **W2** [Significance]: I think more could be done to stress the important of risk-awareness in the PbRL setting. After reading the paper, it was still not clear to me which applications would benefit from RA-PbRL.

QUESTIONS:
In PbRL, it is often assumed that the given preferences $o_i$ may be noisy. Do authors assume perfect preferences? If not, it would be interesting to analyse how the amount of noise in the preference affects the regret bounds.

LIMITATIONS:
The limitations have been correctly addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
This paper focuses on the theoretical analysis of risk-aware preference-based reinforcement learning and introduces Risk-Aware-PbRL (RA-PbRL) to optimize both iterated and accumulated risk-aware objectives.

STRENGTHS:
- This paper proves that both iterated and accumulated quantile risk-aware objectives can be applied to one-episode reward settings. This may provide a theoretical foundation for future episodic RL or PbRL methods focusing on risk-related objectives.
- This paper provide analysis of regret guarantee on the proposed risk-aware algorithm.

WEAKNESSES:
- I think it is necessary to clearly clarify all the strong assumptions, not only the linear reward function assumption, as these strong assumptions typically cannot be met in real-world control scenarios. This would make the theoretical results in this paper more applicable and useful for researchers using deep PbRL methods to address real-world control problems.
- Why is the regret of RA-PbRL higher than PbOP when $\alpha$ is small (Fig. 1(b) and 2(b))? Can the authors provide an intuitive explanation for this phenomenon?

QUESTIONS:
See above.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies preference-based RL (PbRL) where instead of the expected return, the agent optimizes a risk measure based on preference feedback. The authors study two settings called \"iterated\" and \"accumulated\" quantile risks, otherwise known as nested and static risks. They provide sublinear regret bounds for both approaches and instantiate a hard-to-solve MDP to establish a lower bound.\n\nSTRENGTHS:\nThe proposed research is well-motivated, as it combines the accessibility of preference-based feedback with risk-sensitive RL, which is crucial for safety-critical applications. \n\nIf correct, the contributions are meaningful, as they englobe both nested and static risk measures. \n\nThe text has a nice flow. I have to say that I reviewed a previous version of this submission and the structure is much clearer now.\n\nWEAKNESSES:\n- Some used terminologies are non-standard for the risk-sensitive RL (RS-RL) community. For example, the authors discuss \"iterated\" versus \"accumulated\" risk measures. As far as I know, most of the RS-RL works name these \"nested\" versus \"static\" risk measures [1,2,3]. This confused me, as I could not understand the abstract until the middle of the introduction. \n\n- line 62: \"the optimal policy becomes history-dependent, which is more general than assuming the trajectory reward is a linear function of the sum of per-state features\". Why? This deserves more explanation, as it justifies the novelty of this work compared to Chen et al (2023). \n\n- Previous works have studied RL under trajectory feedback, although not in the PbRL setting, see e.g., [4]. Therefore, contribution 3. seems overstated. \n\n[1] Hau, J. L., Petrik, M., & Ghavamzadeh, M. (2023, April). Entropic risk optimization in discounted MDPs. In International Conference on Artificial Intelligence and Statistics (pp. 47-76). PMLR.\n\n[2] Hau, J. L., Delage, E., Ghavamzadeh, M., & Petrik, M. (2024). On dynamic programming decompositions of static risk measures in Markov decision processes. Advances in Neural Information Processing Systems, 36.\n\n[3] Tamar, A., Chow, Y., Ghavamzadeh, M., & Mannor, S. (2015). Policy gradient for coherent risk measures. Advances in neural information processing systems, 28.\n\n[4] Efroni, Y., Merlis, N., & Mannor, S. (2021, May). Reinforcement learning with trajectory feedback. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 8, pp. 7288-7295).\n\nQUESTIONS:\n- The authors tackle nested and static CVaR with the same methodology. Classically, the static formulation encounters time consistency issues that do not appear in nested CVaR. Why isn't it the case here? Could the authors discuss time-consistent risks in the preference-based setup? \n\n- What does Assp. 3.1 formally say? How should it be interpreted and how restrictive is it? \n\n- lines 142-148: Why can we restrict policy-search to deterministic policies even in the preference-based setting? Is it an underlying assumption or is there a formal result stating that we do not lose optimality? \n\n- Are the results from Sec. 3.2 novel - especially the recursions (5) and (6)? \n\n- It is perhaps a philosophical concern but I would be glad to have the authors' feedback. If the feedback is preference-based, what is the difference between average and risk-sensitive return maximization? In other words, doesn't the agent's objective express itself directly through the preference outputs? This question is crucial to me because I currently do not see the motivation for RS-PbRL.\n\n*Minor comments and suggestions*\n\n- Please follow the formatting guidelines for section titles - Secs. 2, 5\n- line 58: \"may not be applicable\"; line 60: \"cannot\"\n- line 125: \"and the reward\"; line 126: \"the trajectory embedding dimension\"\n- Assp. 3.1: \"We\" - capital letter\n- line 133: \" At each iteration\"; line 134: \"unlike standard RL\"; line 137: \"a Bernoulli distribution\"\n- lines 139-140: \"It is aware\" --> \"It is known\" or \"We are aware that\"; line 157: \"risk-measures\"\n- line 160: \"a random variable\" (remove distribution); line 161: \"then\" --> \"so\"\n- line 163: What is $Z$? Should it be $X$?\n- line 176: \"denotes the\"\n- lines 178-179: Reformulate \"the policy should be history-dependent policies\"\n- line 185: \"The proof is\"\n- Lemma 3.5 and 3.6: \"For a tabular MDP and a reward of the entire trajectory being decomposable as ...\"\n- line 191: \"given the current history\" \n- line 199: \"The proof is\"\n- line 207: \"We define\"\n- line 211: \"regardless of iterated or accumulated\"\n- line 214: \"of the entire episode training\"\n- line 218: \"that minimizes regret\"\n- lines 221-22: \"We present the algorithm called RA-PbRL and establish and upper-bound regret for it\"\n- line 225: Remove \"The algorithm\"; \"defined\" --> \"described\" or \"presented\"; \"which main part\"\n- line 229: \"choose a policy\"; line 232: \"find an estimated optimal policy\"\n- line 236: \"we initialize\"; line 237: \"we observe history samples\"; line 239: \"use\" (present)\n- Algo 1, line 9: \"Receive trajectory ... and preference\"; line 10: italic $k$\n- lines 253, 254: \"with policy $\\pi$\" - remove \"the\"\n- line 274: \"hard-to-learn constructions\" (hyphen and \"s\")\n- lines 275-6: \"are unavoidable in some cases\"\n- line 282: \"For a comparative analysis\"; line 286: \"aligned ... with RA-PbRL\"\n\nLIMITATIONS:\nAlthough some limitations are not addressed in this work (see previous remarks), most are properly described in Sec 6.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-15"}