Beyond Expected Returns: A Policy Gradient Algorithm for Cumulative Prospect Theoretic Reinforcement Learning

Authors: Olivier Lepel, Anas Barakat
Primary area: reinforcement learning
Keywords: Cumulative prospect theory, policy gradient, policy optimization, reinforcement learning

Abstract:
The widely used expected utility theory has been shown to be empirically inconsistent with human preferences in the psychology and behavioral economy literatures. Cumulative Prospect Theory (CPT) has been developed to fill in this gap and provide a better model for human-based decision-making supported by empirical evidence. It allows to express a wide range of attitudes and perceptions towards risk, gains and losses. A few years ago, CPT has been combined with Reinforcement Learning (RL) to formulate a CPT policy optimization problem where the goal of the agent is to search for a policy generating long-term returns which are aligned with their preferences. In this work, we revisit this policy optimization problem and provide new insights on optimal policies and their nature depending on the utility function under consideration. We further derive a novel policy gradient theorem for the CPT policy optimization objective generalizing the seminal corresponding result in standard RL. This result enables us to design a model-free policy gradient algorithm to solve the CPT-RL problem. We illustrate the performance of our algorithm in simple examples motivated by traffic control and electricity management applications. We also demonstrate that our policy gradient algorithm scales better to larger state spaces compared to the existing zeroth order algorithm for solving the same problem.

Decision: reject
Reviewer scores: 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730151286470

thread_type:
official_review

<thread_object>
creator: reviewer_alsn

<content>
summary:
This paper makes three key contributions to the field of cumulative-prospect-theory (CPT)-based reinforcement learning, a framework where the agent aims to maximize not the raw cumulative reward but a transformed version of it. This transformation involves both a utility function and a probability weighting function, introduced to better reflect human value assessments of rewards (e.g., monetary amounts converted into perceived value). The objective is to find a policy that maximizes this transformed cumulative reward.

CPT-based reinforcement learning generalizes various prior approaches, including: (1) the standard RL framework, (2) risk-sensitive RL with an exponential utility function (e.g., as in Noorani et al., 2022), (3) optimization based on VaR/CVaR of the total reward, and (4) the case studied by Kahneman and Tversky (1992).

The first contribution presents new insights into the characteristics of optimal policies within CPT-based reinforcement learning. These findings include: (1) deterministic, history-dependent optimal policies may not always exist; (2) in the absence of weighting functions, a deterministic, non-stationary optimal policy exists that depends on the current state and past cumulative reward; and (3) without weighting functions, a Markovian, non-stationary optimal policy exists if and only if the utility function is either affine or exponential.

The second contribution introduces a policy gradient theorem and a novel algorithm for CPT-based reinforcement learning.

The third contribution is an empirical evaluation of the proposed algorithm. This study includes (1) a performance comparison with CPT-SPSA-G, an existing RL algorithm for CPT problems, across various grid-world environments, demonstrating the proposed algorithm's advantages, particularly in larger environments; and (2) an exploration of different behaviors under varying utility and weighting functions.

strengths:
The theoretical and algorithmic contributions are original and clear. If one believes that CPT-based RL problems are important, then these contributions are also significant. 

The paper did a very nice job relating its studied problem with other related problem settings studied previously, making it easy for readers to understand the position of this paper in the literature.

The paper's presentation is clear.

weaknesses:
While I appreciate the contributions the authors made to CPT-based RL problems, I am not convinced of the importance of CPT-based RL problems by reading this paper. The paper cites another paper "... this is particularly important in applications directly involving humans
in the loop such as e-commerce, crowdsourcing and recommendation to name a few" to argue that this class of problems has important applications. However, it is not clear from the cited paper whether other formulations that take into account human behavior can also handle these applications well. In fact, compared with CPT, I feel that a more principled way that considers human behavior is trajectory-based reward RL problems (one reward for the entire trajectory) with human preferences, like RLHF in large language models. While CPT assumes a structure of the final reward (the final metric to optimize involves a weight function, a utility function, and a summation of rewards), these weight functions are unknown and must be learned from data. In contrast, in trajectory-based reward RL problems, there is no assumption on the structure of the metric being optimized. And the metric is learned with human preference data. I wonder how authors think about the pros and cons of CPT in terms of its applications compared to trajectory-based reward RL problem settings.

Another weakness is related to the empirical study of the paper. The experiments only demonstrate the effectiveness of the proposed algorithm but do not include those justifying the necessity of the CPT problem. It would be much better if the authors could demonstrate that in some applications, the data shows that human behavior can be accurately predicted with CPT but not other theories.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730498204601

thread_type:
official_review

<thread_object>
creator: reviewer_hwfl

<content>
summary:
This paper is a follow-up work to [L.A. et al., 2016] on Cumulative Prospect Theory (CPT) value optimization in reinforcement learning problems. The original paper utilized the simultaneous perturbation stochastic approximation (SPSA) method to update the policy, while this paper provides a policy gradient method. The policy gradient is evaluated and compared with the SPSA method in several domains.

strengths:
+ Discussion on the optimal policies under CPT, showing that optimal policies are generally stochastic and non-Markovian when CPT is applied.
+ Policy gradient algorithm for CPT value optimization, compared to SPSA methods in the original paper.

weaknesses:
+ The algorithm is only validated in a few small domains, making it difficult to assess the performance of this policy gradient in more complex tasks.
+ The computation of the integral for the CPT value appears complex and time-consuming due to the use of Monte Carlo sampling.

Suggestions:

+ The authors may consider demonstrating the benefits of using CPT over standard risk-neutral RL in a concrete reinforcement learning problem at the beginning of the paper to help readers better understand CPT.

+ This paper considers the problem in a non-discounted setting, i.e., $\gamma=1$ as shown in Line 181. The claim in line 261 does not apply to the discounted case for exponential utilities. Discussion may be needed. see [Entropic Risk Optimization in Discounted MDPs, AISTATS 2023]

+ As mentioned in line 309, Proposition 7 in the paper is different from the Proposition 6 in [L.A. et.al. 2016], authors may need to provide a comparison and justification.

questions:
+ $\phi(R(\tau))$ in line 295, 303, 305 should be $\varphi(R(\tau))$ in Theorem 6? 

+ How to choose the proper utility function for different problems?

+ The policy gradient calculation depends on the property that the function $u$ is non-decreasing. Can we work with other $u$ where non-decreasing is not guaranteed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666717923

thread_type:
official_review

<thread_object>
creator: reviewer_ydtj

<content>
summary:
The paper introduces a novel approach to reinforcement learning (RL) by leveraging Cumulative Prospect Theory (CPT) to account for human decision-making biases, moving beyond the traditional expected utility framework. It focuses on the policy optimization problem, where the objective is the CPT value of the random variable recording the cumulative discounted rewards (CPT-PO). Additionally, it considers the particular case of the expected utility objective, where only returns are distorted by the utility function (EUT-PO). This work derives a policy gradient (PG) theorem for CPT-based RL, presenting a model-free policy gradient algorithm for optimizing a CPT policy.

strengths:
This paper addresses an objective that encompasses a broad class, including Conditional Value at Risk (CVaR), distortion risk measures, and expected utility.

The paper effectively explains various policy classes associated with different objectives, helping to clarify how these classes relate to CPT-RL versus standard RL approaches.

The derivation of a policy gradient theorem for CPT objectives is a significant extension of the traditional PG results in reinforcement learning, broadening its applicability in human-centered decision contexts.

weaknesses:
The experimental section provides limited insight due to its use of relatively straightforward examples (traffic control and electricity management), which may not fully illustrate the complexities that arise in more realistic, high-stakes environments, such as finance or healthcare.

In Section 5(a), the observation that different weight functions lead to different policies in the grid environment could be strengthened by assessing these policies against risk measures (such as CVaR) beyond expected return. Without this, it is unclear if the CPT-RL-PO policy outperforms standard RL-PO under any specific risk-sensitive criteria.

The grid environment results are overly simplified and provide little substantive information. It would be beneficial to evaluate whether the derived policies' performance aligns meaningfully with their respective objectives.

Limited applicability: The experiments did not demonstrate the advantages of using the proposed algorithm. It is unclear what the benefits of using CPT learning are over risk-sensitive RL or distributional RL. Although CPT is theoretically valuable, the empirical advantages of CPT over other risk-sensitive measures remain ambiguous in the presented results. The authors may consider including direct comparisons with risk-sensitive RL or distributional RL methods on the same tasks.

questions:
Minor comment: On page 4, under Problem formulation: CPT-RL, the cumulative discounted rewards variable $X$ is referenced, but the definition provided does not discount the rewards. It would be clearer to either update the variable's definition to include discounting or adjust the notation accordingly to prevent confusion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732882215471

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision and contributions

comment:
We would like to thank the reviewers again for their feedback and their valuable suggestions.  

**Revision.** In addition to our detailed responses to each one of the reviewers, we have uploaded a revised version of our manuscript containing the following main modifications (highlighted in blue) to improve our work based on the reviewers’ feedback which we are grateful for: 
- Regarding the importance of CPT-RL (highlighted by reviewers aLSN and hWfL), we have updated the introduction with l. 63-68 to highlight applications of CPT in general and we have incorporated a new extended discussion in appendix C p. 17. 
- Concerning examples to clarify the features of CPT-RL and its comparison to risk-sensitive RL, we have added another concrete example in healthcare (personalized treatment for pain management) to section 2 right after the problem formulation to illustrate it and highlight why we consider CPT-RL instead of prior existing risk-sensitive approaches. We will further update the manuscript to incorporate more details in the lines of our response to reviewer hWfL. 
- We have added appendix D p. 18-19 to compare CPT-RL and trajectory-based reward RL as preference based learning paradigms to address the questions and concerns of reviewer aLSN. 
- We deferred the related work section to appendix B due to space constraints. 
- **Experiments:** In addition to our simulations in grid worlds, simple traffic control settings as well as our electricity management application using real-world data, we have added simulations for trading in financial markets (appendix H. 7, as suggested by reviewer Ydtj) and on the Mujoco inverted pendulum environment (appendix H. 8, as suggested by reviewer hWfL) to show the applicability of our PG algorithm to other settings.   

**Our key contributions.** Our main goal in this work is to study and solve a policy optimization problem accounting for human perception and behavior via the use of the well-established cumulative prospect theory from behavioral economics. We provide further insights on the optimal policies and their nature in such problems compared to standard RL policy optimization for maximizing expected returns. We would like to highlight that one of our main contributions is our PG theorem and algorithm to solve this problem. We believe that our practical PG algorithm is widely applicable. We have tested it in several applications, **going far beyond prior work in CPT-RL which only considered solving a single traffic control application on a 2x2 grid using a zeroth order algorithm**. Testing it in other applications and even larger scale problems would certainly be interesting. Our work also opens up the way to several interesting future work opportunities towards more realistic human-based sequential decision making both from the algorithmic and methodological viewpoints (such as learning reference points, utilities and probability weight functions as we highlight in our paper and in our responses to reviewers). 

**Potential.** We are excited by the potential that CPT-RL has to offer to better model human sequential decision making and incorporate cognitive and psychological biases which are of most importance in several high-stakes applications. Only few works have been devoted to this endeavour in RL and our work contributes to this goal.  We have only scratched the surface in this regard by showcasing the potential of solving such a CPT-RL problem formulation in simple settings for healthcare decision making, energy, finance and investment. Many other meaningful applications such as legal and ethical decision making, cybersecurity and human-robot interaction are yet to be explored.   

We would be happy to address any further questions or concerns during the remainder of the discussion period.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper is on policy optimization when the objective is Cumulative Prospect Theory (CPT) value of the cumulative reward random variable. It proves a policy gradient theorem for CPT-RL and derives a policy gradient algorithm for optimization in CPT-RL. Finally, it empirically shows the efficacy of the proposed algorithm and compares it with an existing method for optimizing CPT-RL that is based on simultaneous perturbation stochastic approximation (SPSA). 

(+) The policy gradient theorem, the corresponding algorithm, and experimenting with several problems are all positive aspects of this work. 

(-) Given that CPT-RL was previously studied by L.A. Prashanth et al. (2016), the reviewers expected that this paper provides a better comparison, in terms of both theoretical and empirical results, with the existing work. 
(-) The reviewers seem to have concerns about the motivation behind the CPT formulation in RL. I believe the authors need to better justify the importance of this formulation in RL. 
(-) There are several questions about the connections between the CPT-RL formulation and the proposed policy gradient algorithm, and the variety of risk-sensitive RL formulations and algorithms (many of them being policy gradient) that exist in the literature. I believe addressing this issue can significantly improve the overall quality of the paper.