Two steps to risk sensitivity

1 Introduction . Risk is integral to decision making , arising whenever there are uncertain outcomes . It is especially critical when those outcomes are potentially calamitous , and plays an important role in psychiatric illness [ 1 ] . However , psychological investigations into choice under risk ( i ) have yet to embrace the strong formal foundations being developed in finance , AI and machine learning ; and ( ii ) have mostly focused on one-shot decision tasks , despite the ubiquity of situations in the real-world that require planning , and growing interest in multi-step tasks for elucidating richer mechanisms of choice [ 2 , 3 ] . To address these lacunæ , we consider a modern , theoretically well-founded coherent [ 4 ] risk measure called conditional value-at-risk ( CVaR ) . This exactly quantifies the lower tail of possible outcomes – those which are important for survival and also tend to drive our most persistent worries . CVaR has been applied to sequential decision problems , notably by means of distributional reinforcement learning ( RL ; [ 5–7 ] ) . In this paradigm , the agent ( or decision maker ) estimates whole distributions for potential outcomes that can arise from its actions . Risk measures , such as CVaR , can be applied to these distributions to adjust decision making to any desired level of risk sensitivity . In this paper , we combine CVaR with a distributional approach to examine risk sensitivity in the intensively-investigated two-step sequential decision task [ 2 ] . We find that the choices of many individuals reflect a large degree of risk aversion . Moreover , we show that more standard analyses ascribe this to enhanced choice perseveration and/or reduced estimates for learning rate , thus misspecifying the effects of relatively high levels of uncertainty . Incorporating risk sensitivity into 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . choice , through CVaR or other risk measures , however , raises subtle issues regarding the consistency of choices over time [ 8–11 ] . Such issues are well explored in psychology in the context of hyperbolic temporal discounting [ 12 ] , but their thorough investigation in finance and operations research has yet to permeate judgement and decision making research . We discuss how a direct incorporation of CVaR into distributional RL can lead to time inconsistency and point to two additional time-consistent approaches to CVaR for sequential decision making . Finally , we show how these three approaches can give rise to distinct patterns of behavior in a problem designed to tease them apart and which can serve as the basis for future empirical investigation . Preliminaries : Conditional value-at-risk ( CVaR ) . A risk measure ρ ( Z ) maps a probability distribution of outcomes associated with a random variable Z to a real number . In distributional RL , Z is typically a discounted sum of rewards minus costs ( i.e . the return ) . ρ ( Z ) represents the risk inherent in the uncertainty about Z , and is often interpreted as the amount one is willing to pay to avoid adopting this risk . Well known risk measures include the variance and the value-at-risk ( VaRα ) , which is defined as : VaRα ( Z ) = F−1 ( α ) : = inf { z : F ( z ) ≥ α } ( 1 ) for a cumulative distribution function F ( z ) and α-quantile ; see Figure 1a . While variance and VaR are venerable risk measures , neither satisfy the full set of axiomatically desirable properties associated with coherent risk measures [ 4 ] . For instance , the VaR is not sub-additive ( so fails to reward diversification ) , and variance is neither sub-additive nor positively homogeneous ( as it changes non-linearly with the units in which costs or rewards are measured ) . Conditional value-at-risk ( CVaR ) was introduced as a coherent risk measure [ 4 ] , and is particularly popular in finance , robotics , operations research , and recently machine learning and RL [ 6 , 13–19 ] . For the lower tail of a continuous distribution , it is defined as the average of the values lower than the VaR ( Figure 1a ) :1 CVaRα [ Z ] : = E [ Z|Z < VaRα ( Z ) ] ( 2 ) α determines risk aversion by emphasizing the lower tail of the distribution more or less ( Figure 1b ) . That CVaR concentrates on the lower tail makes it particularly attractive for capturing aspects of normal and pathological reasoning in animals , whose lives often hang on rather thin threads , and humans , who can catastrophize about diverse , unlikely , possibilities . But it is then particularly important to consider CVaR in sequential decision making , since this represents the ecological norm and is etched into the neural structure of decision making . We therefore start by examining CVaR in the simplest such problem that is widely studied in humans : the two-step task . 2 Modeling risk sensitivity in human planning : CVaR in the two-step task . The two-step task ( Figure 2 ) is a popular option for studying sequential choice in humans ( and animals ) [ 2 ] . It was originally designed to investigate model-based ( MB ) and model-free ( MF ) learning and planning , distinguishing the two by examining the consequences of progressive changes in the probabilities of rewards . However , these changes necessarily induce uncertainty – which could affect risk-sensitive subjects . We therefore used a CVaR-based form of MB and MF reasoning to fit a very substantial dataset of human behavior in this task ( out of more than 2000 participants in [ 3 ] , the 792 who responded on every trial ) . Task : On each of 200 trials , participants make decisions at two successive stages or steps . At the first stage , participants choose between two actions ( the fractals in Figure 2a ) . As shown by the arrows in the figure , depending on their choice , they then transition to one of two possible second stage states either 70 % or 30 % of the time . Each second stage state has its own pair of options ( the colored squares ) between which the participants must then choose . According to this second choice , the participant receives a binary outcome , which is drawn according to a probability that drifts randomly across the trials ( shown by color in Figure 2b ) . Before the actual experiment , participants were given 40 trials of practice , during which they learned about the general structure of the task 1For discrete random variables , alternative definitions such as CVaRα ( Z ) : = supν { ν − 1αE [ ( ν − Z ) + ] } [ 20 ] are used . including the 70 % /30 % transition probabilities . This structure was not presented to participants explicitly , but participants were tested about their knowledge of the task and excluded if they failed this test . Participants were instructed about the drifting reward probabilities . CVaR-based model : We included CVaR in a model that is conventionally applied to the two-step task . We assume that participants learn a distribution of values [ 5 ] associated with each of the four second-stage options using an approximate Kalman filter : µt+1 = µt + λ ( rt − µt ) Ψ2t+1 = ( 1− φ2 ) Ψ2t + η2 − λΨ2t ( 3 ) updating the mean µt of each distribution on each trial using the observed outcome rt via a delta-rule with a participant-specific learning rate λ , and also updating the variance Ψt . When the outcome is not observed ( i.e . when the participant chose a different option ) , the mean is updated towards 0.5 and the variance is updated without the last term in equation ( 3 ) . Thus , the dispersion parameter η2 controls the increase of the variance whether or not outcomes are observed , and the learning rate λ controls the amount it decreases when they are . The term ( 1− φ2 ) controls the asymptotic variance for unobserved outcomes ( in relation to η2 ) and was set based on the other two parameters ( see later ) . The CVaRα , t ( a ) at risk preference α ( estimated per participant ) was calculated for each option a using the mean µt ( a ) and variance Ψt ( a ) on each trial t. Note that because this variance represents uncertainty about the reward probabilities themselves , CVaR here captures both ambiguity- as well as risk-sensitivity ; however , to keep terminology consistent with the finance and machine learning communities [ 21–24 ] , we simply refer to this as to risk-sensitivity . Equipped with these CVaRα , t ( a ) values , participants ’ second stage choices were modeled using a soft-max choice rule : P ( 2nd stage choice=a ) ∝ exp ( β2ndCVaRα , t ( a ) ) ( 4 ) where the parameter β2nd controls the relative stochasticity/determinism . Decisions between the two first-stage options were modeled as involving a combination of model-free ( MF ) and model-based ( MB ) approaches to value estimation – both of which were modified to include CVaR . MF estimates were calculated using the same formulæ , learning rate λ and dispersion parameter η2 as at the second stage to learn an additional pair of means and variances based on the actual outcomes received in the second stage . For model-based estimation , the 70 % /30 % transition probabilities were used to calculate a mixture distribution for each of the top stage actions , from which the CVaR was calculated directly . CVaRα , t ( a ) for the model-free and model-based first-stage distributions again determined the choice probability through a soft-max choice rule : P ( 1ststage choice=a ) ∝ exp ( βMBCVaRMBα , t ( a ) + βMFCVaRMFα , t ( a ) + βstickyδa , at−1 ) ( 5 ) Parameter βsticky was included to capture the tendency of participants to repeat ( or to switch ) the previously chosen action regardless of its value . Parameter estimation : The 7 parameters of the CVaR-model were estimated for each participant : CVaR-based risk-sensitivity α ∈ [ 0.1 , 1 ] , learning rate λ ∈ [ 0.01 , 0.99 ] , dispersion η2 ∈ [ 0.001 , 0.09 ] , perseveration βsticky ∈ [ 0 , 20 ] and three inverse temperature parameters β2nd , βMB , βMF ∈ [ 0 , 30 ] . Parameters were estimated in Python using L-BFGS-B . Parameter recovery analyses were conducted to investigate parameter estimability . Preliminary recovery simulations suggested that estimating both η2 and ( 1 − φ2 ) was difficult , so we determined the value of ( 1 − φ2 ) that would pin the asymptotic variance to 0.1 . This was done so that a never-chosen option with an estimated mean 0.5 would have a CVaR of 0 at the lower-bound value for α ( i.e . 0.1 ) , which is appropriate since the outcomes themselves were between 0 and 1 . The learning rate lay between 0.01 to 0.99 and was additionally constrained such that λ < ( 1− φ2 ) . With these constraints , we ran further parameter recovery analyses , generating new data based on participants ’ parameter estimates and re-estimating the model ; a rank-correlation of 0.71 between the generative and recovered CVaR α parameter indicated moderately good estimability/identifiability ; Supplemental Figure 1 . Baseline and alternate models : Setting α = 1 arranges for CVaR choices that depend only on expected values ( in which case the variance estimation can also be removed , because it does not influence choice probability ) . This mean-model is used as a baseline against the risk-sensitive CVaRα < 1 model ; it is very similar to typical models used in the two-step task [ 2 , 3 ] , but is more directly comparable to its risk-sensitive counterpart . We provide a more detailed description of the differences between the current and previously used model in Appendix A.2 . We also tried two forgetful beta-binomial models ( one with and without risk-sensitivity ) since rewards are binary , and a risk-seeking version of the CVaR-model , which emphasized the upper rather than lower tail . However , these models fit participants ’ data less well , so we omit them for the sake of parsimony .

Summary: This paper suggests a new model of behaviour based on conditional-value at risk principle that can provide interesting alternative interpretations to the "two step" task used in many "model-free" vs. "model-based" RL studies. Although the scope of the framework is somewhat narrow (albeit with a variety of modifications to address time consistency), the paper provides a number of interesting explanations to the data that should be considered for the sake of accurate interpretation of computational model-based analyses. The analyses are also very detailed and interesting although I would have preferred to see more focus on empirical data as opposed to theoretical considerations. Overall a strong paper for NeurIPS.