Two steps to risk sensitivity
Authors: Christopher Gagne
Peter Dayan
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Risk measures
Conditional value-at-risk
Two-step task
Time-consistency
Distributional reinforcement learning
Anxiety

Abstract: Distributional reinforcement learning (RL) – in which agents learn about all the possible long-term consequences of their actions, and not just the expected value – is of great recent interest. One of the most important affordances of a distributional view is facilitating a modern, measured, approach to risk when outcomes are not completely certain. By contrast, psychological and neuroscientific investigations into decision making under risk have utilized a variety of more venerable theoretical models such as prospect theory that lack axiomatically desirable properties such as coherence. Here, we consider a particularly relevant risk measure for modeling human and animal planning, called conditional value-at-risk (CVaR), which quantifies worst-case outcomes (e.g., vehicle accidents or predation). We first adopt a conventional distributional approach to CVaR in a sequential setting and reanalyze the choices of human decision-makers in the well-known two-step task, revealing substantial risk aversion that had been lurking under stickiness and perseveration. We then consider a further critical property of risk sensitivity, namely time consistency, showing alternatives to this form of CVaR that enjoy this desirable characteristic. We use simulations to examine settings in which the various forms differ in ways that have implications for human and animal planning and behavior.

Two steps to risk sensitivity
Chris Gagne
Peter Dayan
MPI for Biological Cybernetics
MPI for Biological Cybernetics
University of Tübingen Tübingen, Germany eertttteeemp.....mmmm.... dayanEtue mpg.
Tübingen, Germany christopher gww.e@tuebingnn
Abstract
Distributional reinforcement learning (RL) - which agents learn about all the possible long-term consequences of their actions, and not just the expected value is of great recent interest, One of the most important affordances of distributional view is facilitating modern, measured, approach to risk when outcomes are not completely certain. By contrast, psychological and neuroscientific investigations into decision making under risk have utilized a variety of more venerable theoret- ical models such as prospect theory that lack axiomatically desirable properties such coherence. Here, we consider particularly relevant risk measure for modeling human and animal planning, called conditional value-at-risk (CVaR), which quantifies worst-case outcomes (e.g., vehicle accidents or predation). We first adopt conventional distributional approach to CVaR in sequential setting the well-known two-step task, revealing substantial risk aversion that had been lurking under stickiness and perseveration. We then consider a further critical property risk sensitivity, namely time consistency, showing alternatives to this form of CVaR that enjoy this desirable characteristic. We use simulations to examine settings in which the various forms differ in ways that have implications for human and animal planning and behavior.
and reanalyze the choices human socision-makers
1 Introduction
Risk is integral to decision making, arising whenever there are uncertain outcomes. especially critical when those outcomes are potentially calamitous, and plays an important role in psychiatric illness [T]. However, psychological investigations into choice under risk have yet embrace the strong formal foundations being developed in finance, Al and machine learning; and (ii) have mostly focused on one-shot decision tasks, despite the ubiquity of situations in the real- sallworld that require planning, and growing interest in multi-step tasks for elucidating richer mechanisms of choice 22
To address these lacunæ, consider modern, theoretically well-founded coherent risk measure called conditional value-at-risk (CVaR). This exactly quantifies the lower tail of possible outcomes those which are important for survival and also tend to drive our most persistent worries. CVaR has been applied to sequential decision problems, notably by means of distributional reinforcement learning (RL: (577)) In this paradigm, the agent (or decision maker) estimates whole distributions for potential outcomes that can arise from its actions. Risk measures, such as CVaR, can be applied to these distributions to adjust decision making to any desired level of risk sensitivity.
In this paper, we combine CVaR with a distributional approach to examine risk sensitivity in the aaleesiveey-iivesttiaeedd two-step sequential decision task 22]. We find that the choices of many individuals reflect a large degree of risk aversion. Moreover, we show that more standard analy- ascribe this enhanced choice perseveration and/or reduced estimates for learning rate, thus misspecifying the effects of relatively high levels of uncertainty. Incorporating risk sensitivity into
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

choice, through CVaR or other risk measures, however, raises subtle issues regarding the consisteney of choices over time X 8 Such issues are well explored in psychology in the context of hyperbolic temporal discounting [22 but their thorough investigation in finance and operations research has yet to permeate judgement and decision making research. We discuss how direct incorporation of CVaR into distributional RL. can lead to time inconsistency and point to two additional time-consistent approaches to CVaR for sequential decision making. Finally, we show how these three approaches can give rise to distinct patterns of behavior in problem designed to tease them apart and which can serve as the basis for future empirical investigation.
Preliminaries: Conditional value-at-risk (CVaR)
A risk measure p(Z) maps probability distribution of outcomes associated with a random variable Z to a real number. distributional RI., Z is typically a discounted sum of rewards minus costs (i.e. the return). pl Z)) represents the risk inherent in the uncertainty about Z, and is often interpreted as the amount one is willing to pay to avoid adopting this risk.
Well known risk measures include the variance and the value-at-risk (VaR.), which is defined as:
(1) for a cumulative distribution function F(z) and a-quantile; see Figure While variance and VaR are venerable risk measures, neither satisfy the full set of axiomatically desirable properties associated with coherent risk measures [4]. For instance, the VaR is not sub-additive (so fails to reward diversification), and variance is neither sub-additive nor positively homogeneous (as it changes non-linearly with the units in which costs or rewards are measured).
VaR. = F (a) == inf( F(2) 2())2a) 2}
Conditional value-at-risk (CVaR) was introduced as a coherent risk measure 4 and is particularly popular in finance, robotics, operations research, and recently machine learning and RI. [6] 19] For the lower tail of a continuous distribution, it is defined as the average of the values lower than the VaR (Figure 1
CVaR, [2] == ETZIz < VaR,(2))
determines risk aversion by emphasizing the lower tail of the distribution more or less (Figure[p).
That CVaR concentrates on the lower tail makes it particularly attractive for capturing aspects of normal and pathological reasoning in animals, whose lives often hang on rather thin threads, and humans, who can catastrophize about diverse, unlikely, possibilities. But it is then particularly important to consider CVaR in sequential decision making, since this represents the ecological norm and is etched into the neural structure of decision making. We therefore start by examining CVaR in the simplest such problem that is widely studied in humans: the two-step task.
2 Modeling risk sensitivity in human planning: CVaR in the two-step task
The two-step task (Figure 2 is a popular option for studying sequential choice in humans (and animals) 22] It was originally designed to investigate model-based (MB) and model-free (MF) learning and planning, distinguishing the two by examining the consequences of progressive changes in the probabilities of rewards. However, these changes necessarily induce uncertainty which could affect risk-sensitive subjects. We therefore used a CVaR- based form of MB and MF reasoning to fit a very substantial dataset of human behavior in this task (out of more than 2000 participants in [3, the 792 who responded on every trial).
Task:
On each of 200 trials, participants make decisions at two successive stages or steps. At the first stage, participants choose between two actions (the fractals in Figure 2h). As shown by the arrows in the figure, depending on their choice, they then transition to one of two possible second stage states either 70% or 30% of the time. Each second stage state has its own pair of options (the colored squares) between which the participants must then choose. According this second choice, the participant receives a binary outcome, which is drawn according to a probability that drifts randomly across the trials (shown by color in Figure 2p). Before the actual experiment, participants were given 40 trials of practice, during which they learned about the general structure of the task
For discrete random variables, alternative definitions such as (Z) == sup, EET(V 2)+1} used.
20

---

(b)
VaR
CVaRa
risk preference (a)
0.6
0.1
0.05
-3
-2
outcomes
outcomes (2)
Figure 1: Conditional value-at-risk (CVaR). (a) CVaR, the average of the values in the lower a-% --% of distribution, i.e. below the VaR,. (b) Adjusting emphasizes the lower tail of the distribution more or less and sets the level risk aversion. At a 1.0, CVaR is equal to the mean. As a decreases to 0, it approaches the minimum.
(a)
[b]
Stage
reward probaciiity
0.6
(comnos)
0.5
(nare)
0.4
Stage
0.3-
25
50
75
100 triats
125
150
175
200
0.0
0.6 0.
08
0.0
0.2
08
3.0
seARe prstanilyy
mwwaed protabilty
Figure 2: Two-step task. (a) On each trial, participants choose between two options (depicted as fractals) in stage and then transition stochastically to one of two states (purple/blue or orange/red) stage 2. They again choose between two options (the colored squares) and receive a binary reward. (b) This reward is sampled according to probability that drifts randomly across trials (separately for each option). Distributions over reward probabilities are assumed by the model to be estimated for each option (depicted at the bottom in panel a).
including the 70%/30% transition probabilities. This structure was not presented to participants explicitly, but participants were tested about their knowledge of the task and excluded if they failed this test. Participants were instructed about the drifting reward probabilities.
CVaR-based model:
We included CVaR model that conventionally applied the two-step task. We assume that participants learn a distribution of values [5] associated with each of the four second-stage options using an approximate Kalman filter:
Pt+1 = 14: + X(r= (4))
2 = (1 x + Av?
(3)
updating the mean μ4 of each distribution on each trial using the observed outcome via delta-rule with participant-specine
learning rate X, and also updating the variance V,. When the outcome not observed (i.e. when the participant chose a different option), the mean is updated towards 0.5 and the variance updated without the last term in equation (3). Thus, the dispersion parameter r/ controls the increase of the variance whether or not outcomes are observed, and the learning rate À controls the amount decreases when they are. The term for unobserved outcomes (in relation to 7?) and was set based on the other two parameters (see later).
(1 x²) controls the asymptotic variance

---

The cV) at risk preference a (estimated per participant) was calculated for each option using the mean fe (r(a) and variance (a) on each trial Note that because this variance represents uncertainty about the reward probabilities themselves, CVaR here captures both ambiguity- as well as risk-sensitivity; however, to keep terminology consistent with the finance and machine learning communities (21, we simply refer to this as to risk-sensi
Equipped with these CVaRo,r values, participants' second stage choices were modeled using soft-max choice rule:
P(2nd stage choice-a) a)
where the parameter grnd controls the relative siochaticity/ddeemminiss
Decisions between the two first-stage options were modeled as involving a combination of model-free (MF) and model-based (MB) approaches to value estimation both of which were modified to include CVaR. MF estimates were calculated using the same formulæ, learning rate À and dispersion parameter 17 at the second stage to learn an additional pair of means and variances based on the actual outcomes received in the second stage. For model- based estimation, the 70%/30% transition probabilities were used calculate a mixture distribution for each of the top stage actions, from which the CVaR was calculated directly. CVaRa,, for the model-free and model-based first-stage distributions again determined the choice probability through soft-max choice rule:
P(1 stage choice=a) x exp exp(3MMMCaal MB at (a) + GMM.CVRR C, o,t (a) + fatick 8a,ar-1)
(5)
Parameter gsticky was included to capture the tendency of participants to repeat (or to switch) the previously chosen action regardless of its value.
Parameter estimation: The 7 parameters of the CVaR-model were estimated for each participant:
CVaR-based risk-sensitivity a E [0.1, 1], learning rate À E [0.01, 0.99), dispersion 77² E [0.001, 0.09), perseveration gsicky E [0, 20] and three inverse temperature parameters z2nd, , , E [0,30]
Parameters were estimated in Python using L-BFGS-B. Parameter recovery analyses were conducted to investigate parameter estimability. Preliminary recovery simulations suggested that estimating both 17² and (1 x²) was difficult,
we determined the value of (1 o²) that would pin the asymptotic variance to 0. This was done so that a never-chosen option with an estimated mean 0.5 would have CVaR of O at the lower-bound value for (i.e. 0.1), which is appropriate since the outcomes themselves were between and 1. The learning rate lay between 0.01 to 0.99 and was additionally constrained such that À < (1 Φ²). With these constraints, we ran further parameter recovery analyses, generating new data based on participants' parameter estimates and re-estimating the model; rank-correlation of 0.71 between the generative and recovered CVaR a parameter indicated moderately good estimabilitthideeniiabbiii
Supplemental Figure 1.
Baseline and alternate models: Setting 1 arranges for CVaR choices that depend only on expected values (in which case the variance estimation can also be removed, because it does not influence choice probability). This mean-model used as baseline against the risk-sensitive CVaR. model; it is very similar to typical models used in the two-step task 3] but is more directly comparable its risk-sensitive counterpart. We provide more detailed description of the differences between the current and previously used model in Appendix A.2. We also tried two forgetful veta-binomiil models (one with and without risk-sensitivity) since rewards are binary, and risk-seeking version of the model, which emphasized the upper rather than lower tail. However, these models fit participants' data less well, so we omit them for the sake of parsimony.
Results
We first compared the the mean-model (CVaR,=1) to the risk-sensitive model (CVaRa<1). Using BIC to account for the two extra parameters, the risk-sensitive model was slightly preferred overall (avg. BIC=368.8 versus BIC=371.5; diff=-2.67, t(790)= p=0.004), albeit in a minority (40.2%) of participants. For many such subjects, however, the improved fit was substantial (red points in Figure 3)) and many of them had a 0.2 (Figure 3p), indicating substantial aversion risk.
To investigate the improved fit and for insight into the choice characteristics associated with this risk aversion, analyzed how other parameters altered from the a= case for significantly risk-averse

---

(8)
(c)
(e)
1.0
0.45
dffferetce CCss Coakmods beterr
0.5
0.40
-30
recoered
0.35
0.4
0.30
-100
0.20-
0.0
0.2
0.4
0.6
0.8
04
0.6
200
300
400
500
600
generative CVaR
(b)
BIC mesanmmddl
(d)
0
(9)
number. of jantijannt
200
CVaReroddel
6
150
recovered gatw
1.5
100
1.0
2-
50
05
0.2
0.4
CVaR
0.6
0.8
1.0
0
2 ggtto mean-model
0.0
0.4 0.6
1.0
Figure 3: Two-step task modeling results. (a) Many participants are better fit by the CVaR-model than the meen--odeel, and (b) are substantially risk averse, as estimated by the CVaR a parameter. (c-d) Including risk sensitivity increases/decreases estimates of learning rate and perseveration, respectively, in these participants. (c-f) Simulations show that fitting a risk-agnostic model to risk- sensitive behavior
lead to underestimated learning rates (e) and/or apparent perseveration (f). Shading around the regression line in panels e-f indicate 95%-cconfidenc intervals. Note that outliers located near (x=12, y=10) were removed from panel d.
subjects. There were two related changes: participants' estimated learning rates were higher in the CVaR-model (Figure 3b), and their estimated perseveration parameters were lower (Figure3))) One potential reason for these is that participants tended not to prefer less frequently chosen (and thus more uncertain) options, even when the more certain option is apparently worse.
To demonstrate this directly, we simulated choices from the CVaR-model at increasing levels of risk sensitivity (a E [1.0, 0.6, 0.3, 0.1)) in response to predetermined set of outcomes, with the other parameters held constant (Figure 4. Option A is chosen by design for the first 6 trials, but the model then allowed to switch to an option B after observing what it thinks a sufficient number of negative outcomes. For 1, this switch occurs when the mean estimate for option A crosses the mean estimate for option B at 0.5 (trial 10 in panel b). For a < (panels c-e), this switch occurs when the CVaRa<<1. estimate for the options cross. Due to the uncertainty around the mean and the fact that the model is more uncertain about option B than A, the crossing point occurs later for lower (i.e. on trials 11. 12, and 13). Thus, what looks like tendency to stick with apparently worse option, here arises from an aversion to uncertainty. For a = 1, this can only occur if the learning rate is low (so the value of the chosen option does not decrease too much upon experiencing an unfortunate outcome), and/or with a high perseveration parameter. But, as a consequence, genuine risk aversion (or uncertainty aversion) might be misattributed one of these parameters this is shown directly in Figure Jeif where generating choices from a non-perseverative CVaR-model with low a leads inferred learning rates that are too low (panel e) and inferred non-zero perseveration (panel f). Note, though, that in participants' data, some perseveration remains even with CVaR, as removing perseveration from the "VaR-modes (i.e. setting gutiky 0) led to worse BIC values.
3 Three approaches to sequential risk and time (in)consistency
One of the most important issues for sequential decision making is time consistency that the choices the decision maker at time assumes will be executed at time + are indeed carried out ? Inconsistency, famously caused by hyperbolic discounting [12], can lead to reneging on past
resolutions or require potentially sub-optimal and expensive commitment behavior to prevent that
2 An intimately related, yet differently defined, notion of time consistency involves the consistency of successive risk evaluations of' a stochastic process rather than choices per se. Of course, inconsistent evaluation lead to inconsistent choices. See [25] 27 for in-depth discussion.
5

---

[6]
(d)
o-1.0
0=0.3
1.00
cholce probabitt
1.0
value estmates
cholten and
Dutcomes and
outcomess and
noin
0.50
0.50
0.25
0.5
COMF
2000r
0.00
0.00
11
15
5
5
11
13
15
9
11
13
15
trials
trials
(c)
(e)
Um2.0
um0.6
3-33
a-1.
1.00
ouncomes and
vale ertiestss
0.75
outcomes and
0.75
0.50
0.50
0.00
0.00
Figure 4: CVaR- perseveration. (a) The CVaR-model is simulated at four values of a on a series of choices between options A and B. For low (high risk sensitivity), the model chooses option for more trials, despite receiving 0 outcomes after trial 7; binary outcomes are shown as dots in panels b-e. (b-e) The switch from A to B occurs when the model's estimate for the CVaR (or mean for a 1.0) of A (in color) goes below that of B (in gray). The crossing point occurs later for lower levels of due to uncertainty around the mean which is greater for B.
from occurring. Another, less well-known route to time inconsistency comes from mishandling risk sensitivity [8
and, indeed, different variants of CVaR have been developed deal with this issue. The two-step task lacks sufficient temporal complexity to make this issue clear. Therefore in this section examine it in greater depth, and, using simulations, suggest task that highlights the differences between these CVaR variants.
26 34
Fixed, precommitted, and nested CVaR
In the two-step task, we followed recent work on distributional RL (6] in assuming that participants applied CVaR at the same risk preference (a) to the estimated value distributions at both the first and
(a)
[ageright, #g=right] =righit #y=left]
x²" lly=left
deanght
1.0
0.1
0.9
-4
syeaft Renn
agangh
81
0.
10
-2
1
2
Figure 5: Three approaches to CVaR in a two-stage problem. (a) choice of left or right can be made in both state A and B (if state B is visited). Stochastic transitions are denoted by dashed lines (with the probabilities shown), and the possible terminal outcomes are [-4, -2, 1, 0, 2). The fixed approach CVaR (discussed in the text) would choose policy n' state A, but policy issues with time consistency. The precommitted approach would consistently choose policy T'. (b) In adjusted problem, additional state transitions (colored in gray) are appended state B. The nested approach to CVaR (with 0.1) chooses policy n", taking a sure loss of -4, despite the only extremely remote chance of getting 55.
state B, illustrating

---

second stage. We call this the fixed approach or fCVaR. To see why this approach is time-inconsisteni consider calculating the optimal fCV policy for the simplified two-stage choice scenario depicted in Figure Sa.
From the perspective of the top state A, consider the two policies л π= {a right, 02 = right), which leads to a distribution of outcomes p {0.1 : 0.9 2 and x² ='= ={a= right, 42 left), which leads a distribution of outcomes p' = [0.01: 0.09: 1, 0.9:2.. Just considering these overall distributions, has a CV of 0 and x' has an C of 0.7, s' would be preferred. However, if the agent takes action 01 right and arrives at state B, the remainder of x (i.e. action 02 = left) now looks worse than the remainder of π (i.e. 02 = right), with CVaRu.1 's of -2 and 0, respectively; as a result, the FCVaR agent will defect on its original plan x'.
One way to circumvent this issue with time consistency is to skirt dynamic evaluation altogether and precommit to evaluating risk with respect to only one stage or time-point (the start being most natural)- thus enforcing a commitment n' at state A. One way to make this contract is to change the value of after each transition in the light of the probability of its happening. For instance, the decision in T' made at state A to go left rather than right at state B was based on considering all the outcomes in state B (i.e. the decision was based considering CVaR at a = 1.0 rather than = -..1), This dynamic adjustment of a (in this case from 0.1 in state A to 1.0 in state B) prevents time inconsistency by allowing later risk evaluations and decisions all to be coordinated with respect to the single risk preference at the start stage [31]
A second way to deal with time consistency is to evaluate risk dynamically using a series of nested 35):
one-step (conditional) risk measures 11
PK.N (RR........ RN) = R2 + (R + P pw -
+N
(6)
where Rg is the reward (or cost) for time step k.
If CVaR, with the same value of a is used for each conditional risk measure Pa. this is known as nested or nCVaR, [36]. Although like fCVaR, it keeps risk preferences fixed across time, it applies the CVaR each stage to subsequent CVaR evaluations (which themselves are random scalars) rather than to the full distribution of future random costs or rewards under the policy lower in the tree.
One consequence of this nesting compounding of risk evaluations, is that nCVaR sometimes become much more conservative than the other two approaches. To see this, consider panel b in Figure Here, the same decision tree is appended with set of alternative transitions (depicted in gray) from state B. Now, each possible action is additionally associated with a 10% chance of ending up in state D. which either returns 0 with probability 0.9 or makes a transition yet another state with probability 0.1. The outcomes in this state are either- -5 (the worst possible outcome) or again O. Importantly, from the perspective of state B, the -5 is so remote (having probability of merely 0.001) that it barely impacts the distribution there (and therefore pCVaR and fCV,)). However, due to the nested structure of nCVaR (at a 0.1), the -5 is propagated backwards first to E then D, and then in fact, all the way back to the top state A. As a consequence, the nCVaR agent would then rather choose x" (a1 = left, N/A), for a certain loss of -4.
Of course at a = 1.0, all three approaches are equivalent to using the expected value, which is time consistent. As approaches 0, all three approaches again become equivalent, but to the worst-case
risk measure 37.. which is also time consistent.
The three approaches in a gridworld
As noted, the two-step task is too simple to surface these issues (evaluating the other forms of CVaR leads to equivalent results on the 792 subjects). Thus, we used the simulations in Figure highlight their differences the basis of possible future tests.
Consider an agent (or decision maker) who starts in gridworld in the top left corner and can move either right or left. Exiting the gridworld on the right hand side informally represents a goal and is associated with +3 reward, while exiting the map on the left hand side represents quitting and a loss of -2. The agent's 's actions are stochastic with the possibility of moving downwards with some error probability. If the agent falls off the bottom of the gridworld, a substantial loss of -15 incurred (schematized by the lavapit). The right action has twice the error probability of the left action, meaning that heading towards the goal is riskier than attempting to quit; this leads to interesting differences for the three approaches.
7

---

(4)
'golett"
To natt"
(b)
PCVS
nCVa
stat
Quit
Gosl
13
F
Mart
Figure 6: Three approaches to CVaR a gridworld. (a) The agent starts in the upper left corner and choose to go left right in each state. Actions result in stochastic transitions (as depicted at the top). Exiting the map on the left, right, or bottom results in an outcome of -2, +3, -15. (b) The optimal policy is shown for each of three approaches, highlighting their distinct behaviors.
Optimal polices for the PCVaR, fCVaR and nCVaR were calculated using dynamic programming equations (see Appendix B; based on [II [6] for various levels of a (for pCVaR, which requires interpolating between different values of alpha, we used 21 log spaced points, following [16] The optimal policies for each approach to CVaR at moderate-to-high level of risk (i.e. 0.18) are depicted Figure
For PCVaR (with a start state risk of a = 0. 0.18), the optimal policy in every state is to head towards the goal. This is the case even in the bottom row, where rightward actions are much more risky due to the proximity of the lava pit. In fact, if the agent started in the bottom row with the same a = 0.18, would head left an attempt quit. However, since the PCV approach coordinates risk with respect to the start state and these bottom states are part of the start state's 's lower tail, the pCVaR agent knows that it is better to make 'riskier' decisions here for the sake of its former self Indeed, doing so yields higher start state CVaR than abandoning its plans. In contrast, abandoning the pursuit of the goal is exactly what the fCVaR agent does, as it re-evaluates risk using a 0.18 at every state; i.e. it heads towards the left in rows 2 and 3 after getting knocked off course. Similarly to fCVaR, the nCVaR agent also re-evaluates risk using a 0.18 every state. However, it chooses to quit from the start due to the nesting of risk (risk evaluated on top of future risk evaluations) even from the start, the chances the distal lava pit looms larger than does for the other two approaches (at the same nominal level of alpha). At lower and higher values of alpha, the behavior of the three approaches more closely aligns, either quitting from the start or pursuing the goal from every state.
Discussion and related work
We first showed that adopting the modern risk measure CVaR 4 form based on ideas from distributional RL 66) suggests that a large minority of healthy volunteer subjects exhibit quite significantly risk-averse behavior. This was true even in simple two-step task for extremely low stakes and with only rather limited amounts of uncertainty. That this risk aversion masqueraded as enhanced perseveration and slowed learning is reminder of the complexities of building models of behavior, and an invitation to consider risk models of other common tasks. These effects of risk in behavior can be complemented by considerations of its effects in the sort of off-line planning that has recently been suggested to model rumination in anxiety disorders, as subjects struggle to find ways of mitigating potential future threats 38 That distributional RL has recently been suggested as a way of understanding facets of the activity of major neural systems involved in processing affective outcomes
7 offers highly attractive link to understanding risk aversion in animals (and
"This increase in risk sensitivity can be seen in the average values of the adjusted alphas in each state; shown in the upper right hand corner of each grid cell.

---

also tempts us to consider the role of other neuromodulators that have been implicated in representing and processing uncertainty: (3
Since pathological effects of uncertainty play a malign role in the effects of many psychiatric conditions [1, it would be of great interest to design tasks, for instance based the gridworld of section 3, that decompose various of its different facets to help determine which might be responsible. Indeed, this task can be seen structurally rich form of a popular method for assessing impulsivity - the balloon analogue risk task (BART; 40).. Many potential variants would be of interest - for instance studying forms of precommitment by allowing subjects to buy a form of 'insurance' at the outset against one or more unlucky downwards transitions. We could modify the task further with intermediate rewards and punishments to examine the observation that individuals only take on greater risk after a loss, if it is not yet realized (in our terms, before reaching one of the two sides of the grid) [41]. Then, we might expect precommitments to be abandoned and risk to be re-evaluated to the degree to which previous outcomes seem irreversible or the current situation sufficiently distinct from the one in which the committment was made. One could examine consistency between subjects' choices in the environment and their willingness to pay protect themselves ahead of time. Furthermore, richer distributions of possible outcomes and a diverse set of navigation environments, with features that differentiate the three approaches, could be used to reverse engineer uncertainty calculations in the brain given stable risk preferences.
One particular facet of uncertainty that we did not explore is that, in some circumstances, it is possible to collect more information that reduces it to acceptable levels. Indeed, modern risk measures have recently been applied applied in settings with model uncertainty (i.e. POMDPs and Bayes adaptive MDPs, 24) Information gathering. as explored in the sequential information sampling task 42 has been of particular interest in obsessive compulsive disorder, where subjects have been algorithmically modelled as exhibiting less urgency to make up their minds [43]; it would be interesting to model them computationally as being more risk averse. Similarly, if trust is seen as willingness to risk vulnerability to others 44 45), then risk sensitivity could be an important factor in social pathologies such as borderline personality disorder.
We showed that fCVaR, although intuitive, is not time consistent. The other forms of precommitted pCVaR and nested naR are, which provides them with more attractive formal properties. In fact, one can see fCVaR as living inbetween pCVaR and nCVaR (rather literally, in the problem shown in Figure 6p). There are other potential interpolants for instance, cases in which the updating of following lucky or unlucky transitions (which justifies form of gambler's fallacy; (38) is incomplete (or perhaps asymmetric). Such incompetent calculations have been suggested as underlying psychiatric conditions themselves [46).
Finally, although we focused on CVaR, there are many other risk measures that satisfy the coherence axioms, and indeed other more stringent conditions. Adding the requirements of comontonicity and law invariance lead to the class of distortion risk measures [47] (or equivalently, spectral risk measures [50)). which apply a distortion function to cumulative probabilities. This allows them to be linked directly to the dual theory of choice [51], therefore inheriting an additional set of rational choice axioms (i.e. the axioms of expected utility theory, with an alternative independence axiom) 52
CVaR itself is part of this class, and can be used basis to construct all other members 34 49
Interestingly, the probability weighting function from cumulative prospect theory [53] 54]. a popular model in psychology, can be considered distortion risk measure, even though the full prospect theory adds reference dependence and loss aversion. While prospect theory has been well validated
for single decisions, there is substantial opportunity for theoretical and empirical investigations of its realization in sequential decision -maki...
5 Broader Impact
Coherent risk measures, despite their widespread use in formal applications and favorable theory, have yet to permeate fully psychological and neuroscientific models of decision making, especially for sequential problems. We take early steps in this direction using a widely-recognized and commonly used experimental paradigm, and a psychologically relevant coherent risk measure, CVaR. In doing so, we highlight key issue that can arise, namely time inconsistency. with naive application of risk measures to sequential choice, and discuss two alternative (time-consistent) approaches. We expect that bringing awareness to this issue will lead to interesting future discussions and empirical tests

---

in psychological and neuroscientific communities. We do not anticipate our research to selectively impact some groups at the expense of others. Our study had the limitation of not examining how subjects might compute these various forms of risk sensitivity in neurobiologically credible manner, or showing that the risk aversion that we inferred from behaviour in the two-step task would generalize to other choices that the subjects might make.
Acknowledgements
The authors have no competing interests to disclose. We thank Fabian Renz for his contributions to preliminary analyses of the two-step task and helpful discussions. CG and PD are funded by the Max Planck Society. PD is also funded by the Alexander von Humboldt Foundation.
References
[1] Matthias Brand, Kirsten Labudda, and Hans Markowitsch. Neuropsychologicall correlates of decision-making in ambiguous and risky situations. Neural Networks, 1968):126-1227,, 2006.
[2] Nathaniel D Daw, Samuel J Gershman, Ben Seymour, Peter Dayan, and Raymond Dolan. Model-based influences on humans' choices and striatal prediction errors. Neuron, 69(6)):204 1215, 2011.
[3] Claire M Gillan, Michal Kosinski, Robert Whelan, Elizabeth A Phelps, and Nathaniel D Daw. Characterizing psychiatric symptom dimension related to deficits in goal-directed control. Elife, 5:e1 1305, 2016.
[4] Philippe Artzmer, Freddy Delbaen, Jean-Mare Eber, and David Heath. Coherent measures of risk. Mathematical finance, 9(3):203-222) 1999.
[5] Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforce- ment learning. In International Conference on Machine Learning, pages 449-458. PMLR, 2017.
[6] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096-1105. PMLR, 2018
[7] Will Dabney, Zeb Kurth-Nelson, Naoshige Uchida, Clara Kwon Starkweather, Demis Hassabis, Rémi Munos, and Matthew Botvinick. A distributional code for value in dopamine- based reinforcement learning. Nature, 577(7792):671 -655, 2020.
[8] Kang Boda and Jerzy A Filar. Time consistent dynamic risk measures. Mathematical Methods of Operations Research, 63(1):169-186, 2006.
[9] Philippe Artzner, Freddy Delbaen, Jean-Mare Eber, David Heath, and Hyejin Ku. Coherent multiperiod risk adjusted values and bellman's principle. Annals of Operations Research, 152(1):5-22) 22, 2007.
[10] Alexander Shapiro. On time consistency concept in risk averse multistage stochastic program- ming. Operations Research Letters, 3733):44-1147, 2009.
[11] Andrzej Ruszezyóski. Risk-averse dynamic programming for markov decision processes. Mathematical programming, 125(2):235-25) 2611, 2010.
[12] George Ainslie and Nick Haslam. Hyperbolic discounting. 1992.
[13] Nicole Bäuerle and Jonathan Ott. Markov decision processes with average- velaag--aluue-t-rrsk criteria. Mathematical Methods of Operations Research, 74(3):36-377, 201
[14] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. arXiv preprint arXiv: 2012.
10

---

[15] Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[16] Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decision- making: a evar optimization approach. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015.
[17] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 1 18(1):6070-6120 2017.
(18) Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being optimistic to be conservative: Quickly learning a evar policy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4436-4443, 2020.
[19] Núria Armengol Urpí, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. arXiv preprint urXiv:2102.0537/ 2021.
[20] Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distribu- tions. Journal of bunking & finance, 26(7):1443-1471, 2002.
[21] Jingnan Fan and Andrzej Ruszczyúski. Dynamic risk measures for finite-state partially observ- able markov decision problems. In 2015 Proceedings of the Conference on Control and its Applications, pages 153-158. SIAM, 2015.
[22] Apoorva Sharma, James Harrison, Matthew Tsao, and Marco Pavone. Robust and adaptive planning under model uncertainty. Proceedings the International Conference on Automated Planning and Scheduling, volume 29, pages 410-418, 2019.
[23] Mohamadreza Ahmadi, Masahiro Ono, Michel D Ingham, Richard M Murray, and Aaron D Ames. Risk-averse planning under uncertainty. 2020 American Control Conference (ACC), pages 3305-3312. IEEE, 2020.
[24] Marc Rigter. Bruno Lacerda, and Nick Hawes. Risk-averse bayes-adaptive reinforcement learning. arXiv preprint 1rXiv:2102.05762, 2021.
[25] Michel De Lara and Vincent Leclère. Building up time-consistency for risk measures and dynamic optimization. European Journal of Operational Research, 249(1):177-187, 2016.
[26] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszezyáski. Lectures stochastic
programming: modeling and theory. SIAM, 2014.
[27] Birgit Rudloff, Alexandre Street, and Davi M Valladão. Time consistency and risk averse dynamic decision models: Definition, interpretation and practical consequences. European Journal of Operational Research, 234(3)7743-50,, 2014.
[28] David Kreps and Evan Porteus. Temporal von Neumann-Morrensten and Induced Preferences. 1979.
[29] Larry G Epstein and Martin Schneider. Recursive multiple-priors. Journal of Economic Theory, 113(1):133,, 2003.
[30] Berend Roorda and Johannes M Schumacher. Time consistency conditions for acceptability measures, with an application to tail value at risk. Insurance: Mathematics and Economics, 40(2):209-230, 2007.
[31] Georg Ch Pfuug and Alois Pichler. Time-consiseent decisions and temporal decomposition of coherent risk functionals. Mathematics of Operations Research, 41(2):682-699, 2016.
[32] Alois Pichler and Alexander Shapiro. Risk averse stochastic programming: time consistency and optimal stopping. arXiv preprint arXiv:1808 10807, 2018.
[33] Rouven Schur, Jochen Gönsch, and Michael Hassler. Time-consistent, risk-averse dynamic pricing. European Journal of Operational Research, 277(2):587-6033 2019.
11

---

[34] Anirudha Majumdar and Marco Pavone. How should robot assess risk? towards an axiomatic theory of risk in robotics. Robotics Research, pages 75-84. Springer, 2020.
[35] Andrzej Ruszczyúski and Alexander Shapiro. Conditional risk mappings. Mahematics of operations research, 3143)5444-51, 2006.
[36] Georg Ch Pflug and Werner Romisch. Modeling, measuring and managing risk. World Scientific, 2007.
[37] Stefano P Coraluppi and Steven Marcus. Risk-sensitive and minimax control of discrete-time, finite-state markov decision processes. Automatica, 35(2):301-309, 1999.
[38] Christopher Gagne and Peter Dayan. Peril, prudence and planning risk, avoidance and worry. PsyArXiv, 2021.
[39] Angela J Yu and Peter Dayan. Uncertainty, neuromodulation, and attention. Neuron, 46(4):681- 692, 2005.
[40] Carl W Lejuez, Jennifer P Read, Christopher W Kahler, Jerry B Richards, Susan E Ramsey, Gregory L Sturrt, David R Strong, and Richard A Brown. Evaluation of a behavioral measure of risk taking: the balloon analogue risk task (bart). Journal of Experimental Psychology: Applied, 8(2):75, 2002.
[41] Alex Imas. The realization effect: Risk-taking after realized versus paper losses. American Economic Review, 106(8):0086-2009 2016.
[42] Luke Clark, Trevor W Robbins, Karen D) Ersche, and Barbara J Sahakian. Reflection impulsivity current and former substance users. Biological psychiatry, 60(5):515-522, 2006.
[43] Tobias Hauser, Michael Moutoussis, Peter Dayan, and Raymond Dolan. Increased de- cision thresholds trigger extended information gathering across the compulsivity spectrum. Translational psychiatry, 7(12):1-10, 2017.
[44] Roger Mayer, James H Davis, and F David Schoorman. An integrative model of organizational trust. Academy of management review, 20(3):709-734, 1995.
[45] Denise M Rousseau, Sim B Sitkin, Ronald S Burt, and Colin Camerer. Not different after all: A cross-discipline view of trust. Academy of management review, 23(3):393-400 1998.
[46] Quentin JM Huys, Mare Stuitart-Masip Raymond J Dolan, and Peter Dayan. Decision-theoretic psychiatry. Clinical Psychological Science, 3(3):400 421, 2015.
[47] Shaun Wang. Premium calculation by transforming the layer premium density. ASTIN Bulletin: The Journal of the IAA, 26(1):71-92, 1996.
[48] Shaun S Wang. A class of distortion operators for pricing financial and insurance risks. Journal of risk and insurance, pages 15-36, 2000.
[49] Hans Föllmer and Alexander Schied. Stochastic finance. de Gruyter, 2016.
[50] Carlo Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking & Finance, 26(7):1505113) 2002.
[51] Menahem Yaari. The dual theory of choice under risk. Econometrica: Journal of the Econometric Society, pages 95-115, 1987.
[52] Hans Peter Wächter and Thomas Mazzoni. Consistent modeling of risk averse behavior with spectral risk measures. European Journal of Operational Research, 229( 2013.
[53] Daniel Kahneman and Amos Tversky. Prospect Theory: An Analysis of Decision under Risk. Econometrica, 47(2):233-291, March 1979.
[54] Amos Tversky and Daniel Kahneman. Advances prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5(4):297-323, 1992.
12


Reviews:
[{"review_id": "y2XKyMCXhTe", "reviewer": "Reviewer_nvvN", "summary": "This paper suggests a new model of behaviour based on conditional-value at risk principle that can provide interesting alternative interpretations to the \"two step\" task used in many \"model-free\" vs. \"model-based\" RL studies. Although the scope of the framework is somewhat narrow (albeit with a variety of modifications to address time consistency), the paper provides a number of interesting explanations to the data that should be considered for the sake of accurate interpretation of computational model-based analyses. The analyses are also very detailed and interesting although I would have preferred to see more focus on empirical data as opposed to theoretical considerations. Overall a strong paper for NeurIPS.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Although the CVaR method is not new, its application to this kind of problem and analysis of all the consequences is original. The gridworld task used to illustrate differences between different model variations is novel (in this context).\n\nQuality: The study seems of really high quality with a good review of literature, critical analysis of relevant issues (notably likely misinterpretation of inferred parameters in the discussed tasks/studies), and detailed analysis of consequences of the proposed model, including time sensitivity, an important even a somewhat narrow issue. I think readers can be convinced of relevance and importance of this kind of model extension.\n\nClarity: Overall the paper is quite clear, even though Fig. 5 and its explanation is a bit confusing. The first paragraph of results refers to Fig. 3, not 2. The figures are nice and neat.\n\nSignificance: Although the proposed model extension is somewhat narrow (and there might be other more elegant ways to achieve the same outcomes, e.g. incorporating prospect theory as opposed to simply considering mean of the lower range of outcomes, which could be discussed more in detail), its consequences to interpretation of model parameter results are quite striking. The discussion is very interesting, putting the results in a good perspective. It's nice that the authors go beyond the simple two-step task boundaries but even the gridworld task is fairly simple (compared to what happens in the real world).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. Perhaps should discuss about other relevant models outside of CVaR framework a bit more.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "rTWfY-6ipGd", "reviewer": "Reviewer_DTEP", "summary": "The paper considers using CVaR in human behavior modeling of a two-stage task. In particular, the paper uses CVaR in modeling the choice probability in the two-stage decision-making setting. To work with the time-inconsistency of CVaR, the authors have tried three approaches, namely the fixed, precommitted, and nested CVaR. Using the choice model in analyzing a dataset, the authors show that some participants exhibit risk-averse behaviors.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is clearly written and the motivation of the paper is well-explained. However, it is unclear what the main contribution of the paper is. The current manuscript is an application of existing methods for choice model estimation and constructing time-consistent risk measures from time-inconsistent risk measures, e.g., CVaR. The main novelty of the paper comes from the proposed choice model. However, more explanation and experiments (especially on the sensitivity of the modeling assumption) need to be provided in order to show that such a choice model should be used.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "S6WOWuJVnqM", "reviewer": "Reviewer_xqSF", "summary": "This study proposes how we can integrate distributional RL and CVaR to account for seemingly irrational human behavior. CVaR is widely accepted in finance but not yet in other fields provides a new way to measure the level of subjective risk. Recent findings suggest that the parameter associated with CVaR may well capture the individual differences in risk sensitivity.  This study shows that the model combining the distributional RL and CVaR indeed explain better human choices in two-step task. Furthermore, this study provides a potential solution of how this method can be extended in multistep decisions in a more complicated task. While this model is better predicts the actual choices of human subjects, many parts of this model have not yet fully tested, as the author pointed out, because the data was acquired from too simple a two-step task. Despite this limitation, this might be appreciated by broad research fields (e.g. from investigating psychiatric diseases to designing robots or self-driving cars). I have few comments to make this study to be appreciated more by other filed of researchers.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. What is assumed to be known or unknown to participants? What is the condition that this model works? It seems that participants should know the task structure such as transition but they might not know the true probability of transition between states. However, it is rare that the graphical configuration is given to participants. Participants should infer the latent task structure, especially in a two- or n-step task. Having a good representation of the transition structure would be more difficult when the task is getting complicated. It might be necessary to mention what information is given to participants or pre-trained. If participants use a hybrid model and have a tendency to choose the previous option if they get the rewards (such win-stay-lose-switch strategy is different from stickiness or perseveration), how well alpha can be estimated?\n\n2. Is ρ={ρk, ... ρN-1} assumed to have the same value per subject? Participants might be more sensitive to the risk of the recently visited state. However, this temporal decay might already be achieved by the distribution - updating the distribution of unvisited state close to the uniform distribution -.  Can the authors confirm if this is correct?\n\n3. Since the models are not competing to explain variances of the data, I wonder how much fCVaR and nCvaR can be discriminated when comparing the model fits. Can it be distinguished which algorithm a participant uses, or they are introduced as two methods to model the behavior but give similar results? For example, in the grid example, these two models predict different behavior under the same alpha. However, can they have the same prediction but with different alpha? If pseudo data is generated by one of them, how much does the other model closely estimate the alpha? Do they give similar alpha values or does one tends to estimate alpha higher or lower than the other?\n\n4. What is the minimum number of trials required to estimate the stable alpha value and how it changes when the task gets complicated? What are factors should considered to compute the minimum number of trials? For example, does the number of minimum responses to estimate accurate parameters differ by the alpha values?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "9PDRuAqz7xb", "reviewer": "Reviewer_zeBh", "summary": "The authors adopt a risk measurement to a well-studied two-step RL task and show that many humans are risk-averse. This provides an alternative explanation (risk aversion) for previously assumed stickiness or perseveration. The authors also discussed time consistency issues with their models and provided a few better alternative models and a better experimental design to test it. ", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The task and data are not new, they are from previous publications. The method or the combination of task and method is somewhat new -- applying an existing measurement to a different setting (sequential decision-making).\n\nThe paper is very interesting and interdisciplinary, but overall I’m not very positive about the paper. The model comparison results indicate that the new risk-sensitive model might on average be as good as the mean model. The averaged BIC is not reported, nor is the percentage of subjects that are better by the risk-sensitive model. If what I suspected is true (two models are almost equally good), the paper contributes an alternative explanation for stickiness/perseverance, which can be interesting too. However, the results that uncertainty/risk affects decision-making(perseverance) or modulates learning rate are not very novel (e.g. Behrens et al, 2007), and both explanations (risk-aversion vs. perseverance) seem descriptive to me. The authors extensively discuss their model on time consistency and future directions on this line of research, which are very interesting and can be fruitful. However, they are a work in progress, which usually does not warrant an acceptance at neurips.\n\nThe paper is not hard to understand, but it is a bit hard to follow. I guess one reason is that the paper does not have the common structure that a ML paper has, which is totally fine. It’s great that the authors provide explanations, but then the most critical definition/information is kind of hidden in a dense paragraph with information that can go into intro&method&discussion. I think this might come from a different writing style of a science paper vs. a more technical paper. I think one quick fix would be to leave them (e.g. VaR) stand-alone, like equation (1). Some “dry” math definitions before explanation/intuition can help. Reducing the number of footnotes can also help.\nTime consistency can be better introduced, especially as it is one of the key concepts discussed in this paper. If possible, a real-life example might serve the best. A lot of concepts from psychology/economics can be very new to the majority of neurips readers. It would be impossible to explain them all, but maybe the very key ones can be further explained without assuming domain knowledge from the readers.\n\nSome additional questions and comments: \n1. Is the two-step task structure necessary/critical for this paper?\n2. What are the distributions below the second stage option in figure 2(a)? \n3. It seems that the reward probability has a Markov structure to me. What is the generative model of figure 1(b)? Is this Markov structure considered for MB learning?\n4. It would be good to briefly mention how the mean-model differs from the one used in [3]. And if they are quite different, it might be good to include the model in [3] as a baseline model.\n5. It’s good to include the percentage of subjects that are better fit by the risk-sensitive model.\n6. Can the model capture/generate risk-seeking behaviour?\n7. I’m not sure if this data is available, but it would be interesting to correlate alpha in the model with any risk attitude scale.\n8. I’m not a domain expert, but it seems to me that the risk in this paper might contain both risk and ambiguity, since the subjects also need to learn the reward probability. Please ignore this if I’m wrong.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was extensive discussion on this paper.  Perhaps the largest concern among reviewers was the significance of the model fit though I believe the author response on this issue is satisfactory.  While this paper ended up as a split decision among reviewers, some strong support and comments like the following particularly stood out to me: "[the paper shows] how a different kind of model can provide a substantially different interpretation than numerous other studies" and "[the paper could] encourage other researchers to study how the brain implements distributional RL to make everyday decisions".  Given that this paper proposes and defends a novel perspective in an interdisciplinary area highly germane to current NeurIPS discourse (also with high potential impact), I believe the paper has passed the bar for NeurIPS acceptance.
Decision: Accept (Spotlight)