Two steps to risk sensitivity

1 Introduction . Risk is integral to decision making , arising whenever there are uncertain outcomes . It is especially critical when those outcomes are potentially calamitous , and plays an important role in psychiatric illness [ 1 ] . However , psychological investigations into choice under risk ( i ) have yet to embrace the strong formal foundations being developed in finance , AI and machine learning ; and ( ii ) have mostly focused on one-shot decision tasks , despite the ubiquity of situations in the real-world that require planning , and growing interest in multi-step tasks for elucidating richer mechanisms of choice [ 2 , 3 ] . To address these lacunæ , we consider a modern , theoretically well-founded coherent [ 4 ] risk measure called conditional value-at-risk ( CVaR ) . This exactly quantifies the lower tail of possible outcomes – those which are important for survival and also tend to drive our most persistent worries . CVaR has been applied to sequential decision problems , notably by means of distributional reinforcement learning ( RL ; [ 5–7 ] ) . In this paradigm , the agent ( or decision maker ) estimates whole distributions for potential outcomes that can arise from its actions . Risk measures , such as CVaR , can be applied to these distributions to adjust decision making to any desired level of risk sensitivity . In this paper , we combine CVaR with a distributional approach to examine risk sensitivity in the intensively-investigated two-step sequential decision task [ 2 ] . We find that the choices of many individuals reflect a large degree of risk aversion . Moreover , we show that more standard analyses ascribe this to enhanced choice perseveration and/or reduced estimates for learning rate , thus misspecifying the effects of relatively high levels of uncertainty . Incorporating risk sensitivity into 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . choice , through CVaR or other risk measures , however , raises subtle issues regarding the consistency of choices over time [ 8–11 ] . Such issues are well explored in psychology in the context of hyperbolic temporal discounting [ 12 ] , but their thorough investigation in finance and operations research has yet to permeate judgement and decision making research . We discuss how a direct incorporation of CVaR into distributional RL can lead to time inconsistency and point to two additional time-consistent approaches to CVaR for sequential decision making . Finally , we show how these three approaches can give rise to distinct patterns of behavior in a problem designed to tease them apart and which can serve as the basis for future empirical investigation . Preliminaries : Conditional value-at-risk ( CVaR ) . A risk measure ρ ( Z ) maps a probability distribution of outcomes associated with a random variable Z to a real number . In distributional RL , Z is typically a discounted sum of rewards minus costs ( i.e . the return ) . ρ ( Z ) represents the risk inherent in the uncertainty about Z , and is often interpreted as the amount one is willing to pay to avoid adopting this risk . Well known risk measures include the variance and the value-at-risk ( VaRα ) , which is defined as : VaRα ( Z ) = F−1 ( α ) : = inf { z : F ( z ) ≥ α } ( 1 ) for a cumulative distribution function F ( z ) and α-quantile ; see Figure 1a . While variance and VaR are venerable risk measures , neither satisfy the full set of axiomatically desirable properties associated with coherent risk measures [ 4 ] . For instance , the VaR is not sub-additive ( so fails to reward diversification ) , and variance is neither sub-additive nor positively homogeneous ( as it changes non-linearly with the units in which costs or rewards are measured ) . Conditional value-at-risk ( CVaR ) was introduced as a coherent risk measure [ 4 ] , and is particularly popular in finance , robotics , operations research , and recently machine learning and RL [ 6 , 13–19 ] . For the lower tail of a continuous distribution , it is defined as the average of the values lower than the VaR ( Figure 1a ) :1 CVaRα [ Z ] : = E [ Z|Z < VaRα ( Z ) ] ( 2 ) α determines risk aversion by emphasizing the lower tail of the distribution more or less ( Figure 1b ) . That CVaR concentrates on the lower tail makes it particularly attractive for capturing aspects of normal and pathological reasoning in animals , whose lives often hang on rather thin threads , and humans , who can catastrophize about diverse , unlikely , possibilities . But it is then particularly important to consider CVaR in sequential decision making , since this represents the ecological norm and is etched into the neural structure of decision making . We therefore start by examining CVaR in the simplest such problem that is widely studied in humans : the two-step task . 2 Modeling risk sensitivity in human planning : CVaR in the two-step task . The two-step task ( Figure 2 ) is a popular option for studying sequential choice in humans ( and animals ) [ 2 ] . It was originally designed to investigate model-based ( MB ) and model-free ( MF ) learning and planning , distinguishing the two by examining the consequences of progressive changes in the probabilities of rewards . However , these changes necessarily induce uncertainty – which could affect risk-sensitive subjects . We therefore used a CVaR-based form of MB and MF reasoning to fit a very substantial dataset of human behavior in this task ( out of more than 2000 participants in [ 3 ] , the 792 who responded on every trial ) . Task : On each of 200 trials , participants make decisions at two successive stages or steps . At the first stage , participants choose between two actions ( the fractals in Figure 2a ) . As shown by the arrows in the figure , depending on their choice , they then transition to one of two possible second stage states either 70 % or 30 % of the time . Each second stage state has its own pair of options ( the colored squares ) between which the participants must then choose . According to this second choice , the participant receives a binary outcome , which is drawn according to a probability that drifts randomly across the trials ( shown by color in Figure 2b ) . Before the actual experiment , participants were given 40 trials of practice , during which they learned about the general structure of the task 1For discrete random variables , alternative definitions such as CVaRα ( Z ) : = supν { ν − 1αE [ ( ν − Z ) + ] } [ 20 ] are used . including the 70 % /30 % transition probabilities . This structure was not presented to participants explicitly , but participants were tested about their knowledge of the task and excluded if they failed this test . Participants were instructed about the drifting reward probabilities . CVaR-based model : We included CVaR in a model that is conventionally applied to the two-step task . We assume that participants learn a distribution of values [ 5 ] associated with each of the four second-stage options using an approximate Kalman filter : µt+1 = µt + λ ( rt − µt ) Ψ2t+1 = ( 1− φ2 ) Ψ2t + η2 − λΨ2t ( 3 ) updating the mean µt of each distribution on each trial using the observed outcome rt via a delta-rule with a participant-specific learning rate λ , and also updating the variance Ψt . When the outcome is not observed ( i.e . when the participant chose a different option ) , the mean is updated towards 0.5 and the variance is updated without the last term in equation ( 3 ) . Thus , the dispersion parameter η2 controls the increase of the variance whether or not outcomes are observed , and the learning rate λ controls the amount it decreases when they are . The term ( 1− φ2 ) controls the asymptotic variance for unobserved outcomes ( in relation to η2 ) and was set based on the other two parameters ( see later ) . The CVaRα , t ( a ) at risk preference α ( estimated per participant ) was calculated for each option a using the mean µt ( a ) and variance Ψt ( a ) on each trial t. Note that because this variance represents uncertainty about the reward probabilities themselves , CVaR here captures both ambiguity- as well as risk-sensitivity ; however , to keep terminology consistent with the finance and machine learning communities [ 21–24 ] , we simply refer to this as to risk-sensitivity . Equipped with these CVaRα , t ( a ) values , participants ’ second stage choices were modeled using a soft-max choice rule : P ( 2nd stage choice=a ) ∝ exp ( β2ndCVaRα , t ( a ) ) ( 4 ) where the parameter β2nd controls the relative stochasticity/determinism . Decisions between the two first-stage options were modeled as involving a combination of model-free ( MF ) and model-based ( MB ) approaches to value estimation – both of which were modified to include CVaR . MF estimates were calculated using the same formulæ , learning rate λ and dispersion parameter η2 as at the second stage to learn an additional pair of means and variances based on the actual outcomes received in the second stage . For model-based estimation , the 70 % /30 % transition probabilities were used to calculate a mixture distribution for each of the top stage actions , from which the CVaR was calculated directly . CVaRα , t ( a ) for the model-free and model-based first-stage distributions again determined the choice probability through a soft-max choice rule : P ( 1ststage choice=a ) ∝ exp ( βMBCVaRMBα , t ( a ) + βMFCVaRMFα , t ( a ) + βstickyδa , at−1 ) ( 5 ) Parameter βsticky was included to capture the tendency of participants to repeat ( or to switch ) the previously chosen action regardless of its value . Parameter estimation : The 7 parameters of the CVaR-model were estimated for each participant : CVaR-based risk-sensitivity α ∈ [ 0.1 , 1 ] , learning rate λ ∈ [ 0.01 , 0.99 ] , dispersion η2 ∈ [ 0.001 , 0.09 ] , perseveration βsticky ∈ [ 0 , 20 ] and three inverse temperature parameters β2nd , βMB , βMF ∈ [ 0 , 30 ] . Parameters were estimated in Python using L-BFGS-B . Parameter recovery analyses were conducted to investigate parameter estimability . Preliminary recovery simulations suggested that estimating both η2 and ( 1 − φ2 ) was difficult , so we determined the value of ( 1 − φ2 ) that would pin the asymptotic variance to 0.1 . This was done so that a never-chosen option with an estimated mean 0.5 would have a CVaR of 0 at the lower-bound value for α ( i.e . 0.1 ) , which is appropriate since the outcomes themselves were between 0 and 1 . The learning rate lay between 0.01 to 0.99 and was additionally constrained such that λ < ( 1− φ2 ) . With these constraints , we ran further parameter recovery analyses , generating new data based on participants ’ parameter estimates and re-estimating the model ; a rank-correlation of 0.71 between the generative and recovered CVaR α parameter indicated moderately good estimability/identifiability ; Supplemental Figure 1 . Baseline and alternate models : Setting α = 1 arranges for CVaR choices that depend only on expected values ( in which case the variance estimation can also be removed , because it does not influence choice probability ) . This mean-model is used as a baseline against the risk-sensitive CVaRα < 1 model ; it is very similar to typical models used in the two-step task [ 2 , 3 ] , but is more directly comparable to its risk-sensitive counterpart . We provide a more detailed description of the differences between the current and previously used model in Appendix A.2 . We also tried two forgetful beta-binomial models ( one with and without risk-sensitivity ) since rewards are binary , and a risk-seeking version of the CVaR-model , which emphasized the upper rather than lower tail . However , these models fit participants ’ data less well , so we omit them for the sake of parsimony .

Summary: This study proposes how we can integrate distributional RL and CVaR to account for seemingly irrational human behavior. CVaR is widely accepted in finance but not yet in other fields provides a new way to measure the level of subjective risk. Recent findings suggest that the parameter associated with CVaR may well capture the individual differences in risk sensitivity.  This study shows that the model combining the distributional RL and CVaR indeed explain better human choices in two-step task. Furthermore, this study provides a potential solution of how this method can be extended in multistep decisions in a more complicated task. While this model is better predicts the actual choices of human subjects, many parts of this model have not yet fully tested, as the author pointed out, because the data was acquired from too simple a two-step task. Despite this limitation, this might be appreciated by broad research fields (e.g. from investigating psychiatric diseases to designing robots or self-driving cars). I have few comments to make this study to be appreciated more by other filed of researchers.