Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper considers using CVaR in human behavior modeling of a two-stage task. In particular, the paper uses CVaR in modeling the choice probability in the two-stage decision-making setting. To work with the time-inconsistency of CVaR, the authors have tried three approaches, namely the fixed, precommitted, and nested CVaR. Using the choice model in analyzing a dataset, the authors show that some participants exhibit risk-averse behaviors.
main_review: The paper is clearly written and the motivation of the paper is well-explained. However, it is unclear what the main contribution of the paper is. The current manuscript is an application of existing methods for choice model estimation and constructing time-consistent risk measures from time-inconsistent risk measures, e.g., CVaR. The main novelty of the paper comes from the proposed choice model. However, more explanation and experiments (especially on the sensitivity of the modeling assumption) need to be provided in order to show that such a choice model should be used.
limitations_and_societal_impact: The authors have adequately addressed the limitations of their work.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors adopt a risk measurement to a well-studied two-step RL task and show that many humans are risk-averse. This provides an alternative explanation (risk aversion) for previously assumed stickiness or perseveration. The authors also discussed time consistency issues with their models and provided a few better alternative models and a better experimental design to test it. 
main_review: The task and data are not new, they are from previous publications. The method or the combination of task and method is somewhat new -- applying an existing measurement to a different setting (sequential decision-making).

The paper is very interesting and interdisciplinary, but overall I’m not very positive about the paper. The model comparison results indicate that the new risk-sensitive model might on average be as good as the mean model. The averaged BIC is not reported, nor is the percentage of subjects that are better by the risk-sensitive model. If what I suspected is true (two models are almost equally good), the paper contributes an alternative explanation for stickiness/perseverance, which can be interesting too. However, the results that uncertainty/risk affects decision-making(perseverance) or modulates learning rate are not very novel (e.g. Behrens et al, 2007), and both explanations (risk-aversion vs. perseverance) seem descriptive to me. The authors extensively discuss their model on time consistency and future directions on this line of research, which are very interesting and can be fruitful. However, they are a work in progress, which usually does not warrant an acceptance at neurips.

The paper is not hard to understand, but it is a bit hard to follow. I guess one reason is that the paper does not have the common structure that a ML paper has, which is totally fine. It’s great that the authors provide explanations, but then the most critical definition/information is kind of hidden in a dense paragraph with information that can go into intro&method&discussion. I think this might come from a different writing style of a science paper vs. a more technical paper. I think one quick fix would be to leave them (e.g. VaR) stand-alone, like equation (1). Some “dry” math definitions before explanation/intuition can help. Reducing the number of footnotes can also help.
Time consistency can be better introduced, especially as it is one of the key concepts discussed in this paper. If possible, a real-life example might serve the best. A lot of concepts from psychology/economics can be very new to the majority of neurips readers. It would be impossible to explain them all, but maybe the very key ones can be further explained without assuming domain knowledge from the readers.

Some additional questions and comments: 
1. Is the two-step task structure necessary/critical for this paper?
2. What are the distributions below the second stage option in figure 2(a)? 
3. It seems that the reward probability has a Markov structure to me. What is the generative model of figure 1(b)? Is this Markov structure considered for MB learning?
4. It would be good to briefly mention how the mean-model differs from the one used in [3]. And if they are quite different, it might be good to include the model in [3] as a baseline model.
5. It’s good to include the percentage of subjects that are better fit by the risk-sensitive model.
6. Can the model capture/generate risk-seeking behaviour?
7. I’m not sure if this data is available, but it would be interesting to correlate alpha in the model with any risk attitude scale.
8. I’m not a domain expert, but it seems to me that the risk in this paper might contain both risk and ambiguity, since the subjects also need to learn the reward probability. Please ignore this if I’m wrong.



limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper suggests a new model of behaviour based on conditional-value at risk principle that can provide interesting alternative interpretations to the "two step" task used in many "model-free" vs. "model-based" RL studies. Although the scope of the framework is somewhat narrow (albeit with a variety of modifications to address time consistency), the paper provides a number of interesting explanations to the data that should be considered for the sake of accurate interpretation of computational model-based analyses. The analyses are also very detailed and interesting although I would have preferred to see more focus on empirical data as opposed to theoretical considerations. Overall a strong paper for NeurIPS.
main_review: Originality: Although the CVaR method is not new, its application to this kind of problem and analysis of all the consequences is original. The gridworld task used to illustrate differences between different model variations is novel (in this context).

Quality: The study seems of really high quality with a good review of literature, critical analysis of relevant issues (notably likely misinterpretation of inferred parameters in the discussed tasks/studies), and detailed analysis of consequences of the proposed model, including time sensitivity, an important even a somewhat narrow issue. I think readers can be convinced of relevance and importance of this kind of model extension.

Clarity: Overall the paper is quite clear, even though Fig. 5 and its explanation is a bit confusing. The first paragraph of results refers to Fig. 3, not 2. The figures are nice and neat.

Significance: Although the proposed model extension is somewhat narrow (and there might be other more elegant ways to achieve the same outcomes, e.g. incorporating prospect theory as opposed to simply considering mean of the lower range of outcomes, which could be discussed more in detail), its consequences to interpretation of model parameter results are quite striking. The discussion is very interesting, putting the results in a good perspective. It's nice that the authors go beyond the simple two-step task boundaries but even the gridworld task is fairly simple (compared to what happens in the real world).


limitations_and_societal_impact: Yes. Perhaps should discuss about other relevant models outside of CVaR framework a bit more.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This study proposes how we can integrate distributional RL and CVaR to account for seemingly irrational human behavior. CVaR is widely accepted in finance but not yet in other fields provides a new way to measure the level of subjective risk. Recent findings suggest that the parameter associated with CVaR may well capture the individual differences in risk sensitivity.  This study shows that the model combining the distributional RL and CVaR indeed explain better human choices in two-step task. Furthermore, this study provides a potential solution of how this method can be extended in multistep decisions in a more complicated task. While this model is better predicts the actual choices of human subjects, many parts of this model have not yet fully tested, as the author pointed out, because the data was acquired from too simple a two-step task. Despite this limitation, this might be appreciated by broad research fields (e.g. from investigating psychiatric diseases to designing robots or self-driving cars). I have few comments to make this study to be appreciated more by other filed of researchers.
main_review: 1. What is assumed to be known or unknown to participants? What is the condition that this model works? It seems that participants should know the task structure such as transition but they might not know the true probability of transition between states. However, it is rare that the graphical configuration is given to participants. Participants should infer the latent task structure, especially in a two- or n-step task. Having a good representation of the transition structure would be more difficult when the task is getting complicated. It might be necessary to mention what information is given to participants or pre-trained. If participants use a hybrid model and have a tendency to choose the previous option if they get the rewards (such win-stay-lose-switch strategy is different from stickiness or perseveration), how well alpha can be estimated?

2. Is ρ={ρk, ... ρN-1} assumed to have the same value per subject? Participants might be more sensitive to the risk of the recently visited state. However, this temporal decay might already be achieved by the distribution - updating the distribution of unvisited state close to the uniform distribution -.  Can the authors confirm if this is correct?

3. Since the models are not competing to explain variances of the data, I wonder how much fCVaR and nCvaR can be discriminated when comparing the model fits. Can it be distinguished which algorithm a participant uses, or they are introduced as two methods to model the behavior but give similar results? For example, in the grid example, these two models predict different behavior under the same alpha. However, can they have the same prediction but with different alpha? If pseudo data is generated by one of them, how much does the other model closely estimate the alpha? Do they give similar alpha values or does one tends to estimate alpha higher or lower than the other?

4. What is the minimum number of trials required to estimate the stable alpha value and how it changes when the task gets complicated? What are factors should considered to compute the minimum number of trials? For example, does the number of minimum responses to estimate accurate parameters differ by the alpha values?

limitations_and_societal_impact: yes.
review_title: null

Meta-review: There was extensive discussion on this paper.  Perhaps the largest concern among reviewers was the significance of the model fit though I believe the author response on this issue is satisfactory.  While this paper ended up as a split decision among reviewers, some strong support and comments like the following particularly stood out to me: "[the paper shows] how a different kind of model can provide a substantially different interpretation than numerous other studies" and "[the paper could] encourage other researchers to study how the brain implements distributional RL to make everyday decisions".  Given that this paper proposes and defends a novel perspective in an interdisciplinary area highly germane to current NeurIPS discourse (also with high potential impact), I believe the paper has passed the bar for NeurIPS acceptance.
Decision: Accept (Spotlight)