A Unifying Normative Framework of Decision Confidence
Authors: Amelia Johnson
Michael A Buice
Koosha Khalvati
Venue: NeurIPS 2024 poster (2024)
Keywords: Decision making
cognitive science
computational neuroscience
planning as inference
confidence
Primary area: neuroscience_and_cognitive_science

Abstract: Self-assessment of one’s choices, i.e., confidence, is the topic of many decision neuroscience studies. Computational models of confidence, however, are limited to specific scenarios such as between choices with the same value. Here we present a normative framework for modeling decision confidence that is generalizable to various tasks and experimental setups. We further drive the implications of our model from both theoretical and experimental points of view. Specifically, we show that our model maps to the planning as an inference framework where the objective function is maximizing the gained reward and information entropy of the policy. Moreover, we validate our model on two different psychophysics experiments and show its superiority over other approaches in explaining subjects' confidence reports.

A Unifying Normative Framework of Decision Confidence
Amelia M. Johnson
Micheal Buice
Department of Computer Science University of Washington Seattle, WA 98195 ameliamg jalliammjuu...
Allen Institute Seattle, WA 98109
mi michel'buualleennssitttteee.
Koosha Khalvati
Allen Institute Seattle, WA 98 98109
koosha. www.wvaaailllleeenatttttteeeeee
Abstract
Self-assessment of one's choices, i..., confidence, is the topic of many decision neuroscience studies. Computational models confidence, however, are limited to specific scenarios such between choices with the same value. Here present a normative framework for modeling decision confidence that is generalizable to various tasks and experimental setups. We further drive the implications of model from both theoretical and experimental points of view. Specifically, we show that our model maps to the planning an inference framework where the objective function is maximizing the gained reward and information entropy of the policy. Moreover, we validate our model and show its superiority over other approaches in explaining subjects' confidence reports !.
different psychophysics experiments
Introduction
of one's choices, i.e., decision confidence, plays key role in long-term decision- making and learning [[] This assessment helps the decision maker improve their model of the outside world and consequently gain higher utility in the future 2 3l
Due to this critical role, confidence has been the topic of many theoretical and experimental decision neuroscience studies. However, primarily focused on in perceptual decision-making experiments, confidence is mainly mathematically defined only for scenarios where different choices have the same potential reward. In these situations, based on some perceptual cues, the goal is pick the "correct" choice instead of all other equally "incorrect" ones. Consequently, confidence is defined as the probability of choosing the correct option [ 4 In these setups, perception confidence, i..., "what is the probability that my observation was correct?" and decision confidence, "what is the probability that made the correct decision?" are inseparable because the choices do not differ in value.
In the real world, though, different actions lead to various potential rewards, and this variation in reward could influence confidence in decisions, differentiating it from perception confidence. A rustling sound is more likely to be because of the wind rather than predator approaching. Still, everyone becomes more vigilant when hearing that sound one of the possibilities is potentially life-threatening despite being unlikely. Moreover, everyone would be confident about the rationality
Thee majority of this research was done while the first author was intern at the Allen Institute.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

of this decision. Notably, most experiments where different choices vary in value, known as value- based recision-making do not involve uncertainty perception [5]. These works are mostly about memory retrieval and valuation of different objects and do not study the interaction between value and perceptual uncertainty [6]. Here, we present normative framework to formally define and assess decision confidence in general scenario involving uncertainty about the outside world, prior knowledge of the decision maker about the world, and different utility functions for available choices. We model decision confidence "probability of making the best decision". In mathematical language, decision confidence is the probability being optimal over a sequence of states and actions given the policy.
We further show that our approach modeling subjects' confidence in their decision equals to planning as inference framework 7 8 This framework maps to reinforcement learning agent whose objective function is to jointly maximize the reward and the information entropy of the policy (also called maximum entropy reinforcement learning) [9] [0] Moreover, we validate our framework by testing it on two different experiments on confidence evaluation and explaining its implications 5)
2
Modelling Background and Problem Definition
Markov Decision Processes are graphical models used for optimal sequential decision-making in artificial agents. Recently, these frameworks have been applied successfully in modeling the behavior of subjects across various decision-making tasks and species in cognitive neuroscience
12
3
14
15
2.1 Fully and Partially Observable Markov Decision Processes
Formally, a Markov Decision Process (MDP) is a tuple (S, R) describing a Markovian system where S' is the finite set of states, A is the finite set of actions, 7 = p(s'|a, s) is the transition function between states, and r(s, a). The goal of an MDP agent is to come up with the recipe of action selection, called policy, to maximize its total reward within horizon H. As the system is Markovian, each policy can be defined probability distribution of actions given the state, shown by (a]ss) The optimal policy "t" is the policy that attains the goal the agent:
a bounded function representing the reward of each action in each state,
H Σ rss, ai)
π arg max E (eenne)maaaalll
rr
The optimal policy can be obtained in polynomial time in the size of the state space, e.g., by using dynamic programming algorithms.
In most real-world situations, however, the environment is only partially observable, making the agent uncertain about the current state of the world. Partially Observable MDP (POMDP) models these situations by adding the concepts observation, observation function, and the belief state the MDP framework [16]. POMDP is formally defined as tuple (S, A, Z, T, P. R) where S, A. T and R have the same definition as MDP. Z is the finite set of observations. In addition, P' = p(z|s, ss a) is the observation function representing the probability of each observation, given the state and chosen action. A POMDP agent does not know the current state of the environment. Therefore, starting from a prior, called initial belief state b, it updates the posterior probability distribution over the states with each observation and action:
b((s) p(zols,
Σ p(s]s', s'ES
p(ssss/.a--1)bb1 an-1)bi-1(s))
(2 (2
(2)
In a POMDP, each policy can be represented a mapping from belief states to a probability distribution over actions, i.e., π (alb) with the optimal policy x¹ being the mapping that maximizes the expected total reward:
H Σ
arg max Eebr (r)
ΣΣ (s())(()) 1= lES
arg max Esor, )xx(()) (b))
r(b at)

---

where r(b,, az) = ExEs ((())r(s;,a) a.) ) is the expected reward of belief state if the agent choose action dr.
POMDP can be viewed as a fully observable MDP with the state space of the belief state space of the original environment. Therefore, the optimal policy can be obtained in polynomial time in the size of the belief state. Since the belief' state is probability distribution over states, its space's ' size is exponential in the size of the state space. This means that transforming the POMDP to MDP not helpful unless the number of states is extremely low (less than 10) or the belief state can always be represented by a distribution of few parameters. An example of the latter case is Kalman-filte filter-like environments where the belief state can always be represented with a Gaussian distribution with two parameters of pa and ot.
2.2 Models of confidence in Cognitive Neuroscience
One of the applications of POMDPs and similar Bayesian frameworks is modeling the behavior "perceptual decision making" where the subject should select the "correct" choice based on some sensory observations get reward [1].. As the term "correct" suggests, the reward function is symmetrical among different choices. In some of these studies, subjects also report their confidence in their choice. Numerous experiments have demonstrated that trained subjects perform similarly to an optimal agent such as POMDP. Moreover, their confidence in their choice closely matches the probability of choosing the correct option, i.e., the posterior probability of the most probable choice [ (or the sum of belief states that leads to the most probable choice). This close match is also called Bayesian confidence hypothesis 19.. Notably, due to reward symmetry these situations, perception confidence, modeled as the belief about the hidden state, is inseparable from decision confidence, which would be the belief about the decision. More specifically, these experiments cannot flesh out the interactions between value, perception, and confidence and cannot test theoretical models for these interactions.
There are a few confidence experiments with asymmetries in the reward function. Models and methods of these studies, however, are all descriptive/statistiaal, e.g., positive correlation between confidence report and reward value [20] [1 Therefore, as opposed to normative models, these methods do not explain the reason behind the relationship of confidence with perception, prior, and rewards in a systematic and generalizable manner. Moreover, some experimental and theoretical works have studied "value-based confidence" in the context of "value-based decision making" where the task is to choose an object between offered options [22] 6 5.. There is no perceptual ambiguity in these studies, and the variance in response is primarily due to the variability of the valuation of objects at different time points based on memory (6).
Finally, note that the optimality assumption in proposed models, such as the Bayesian confidence hypothesis, does not necessarily imply that the subject's confidence and performance are equal on average. As opposed to an AI POMDP agent, the subject is unaware of the exact generative functions of the environment (S, T, and P). The internal world model of the subject is built and learned by training. Therefore, the type and amount of training and feedback and the subject's capability and motivation in creating the accurate internal world model affect confidence. However, given the internal model, the behavior optimal, e.g., the perception confidence follows the maximum belief state [8 19
We will explain this in more detail in our empirical evaluation and discussions.
2.3 Problem definition
We aim to formally define and test a normative framework that explains the interaction of perception and reward/value with the subject's confidence about their decision. Before presenting the model, we discuss a thought experiment as the motivation. This experiment consists of a set of trials, each of which begins with the presentation of Gabor filter tilted left or right. The subject asked to report their perception of the direction of the Gabor filter (left or right) and their confidence in their decision. Choosing the incorrect direction leads to no reward. Moreover, if correct, the direction left produces reward 10 times the reward of the direction right. In an example trial, the subject believes the direction is right with a 0.7 probability. Assuming they want to gain higher reward, they will choose the direction left due to the higher expected value, namely 0.3 versus 0.7. However, their perception confidence or "probability of choosing the correct/rewarding direction" is 0.3. On the other hand, their confidence in their decision, "likelihood of making the best decision" is probably
3

---

higher than 0.5 (because they chose it). Notably. while being higher than 0.5, their confidence in this trial is likely lower than in trials where they think the stimulus direction is left. However, from the strict optimality point of view, the decision confidence is in both cases. Therefore, while strictly optimal framework like POMDP offers a normative model of confidence (by explaining why confidence equals I through the lens of optimality), such a model is not aligned with reality.
Our goal is systematically define and test decision confidence that includes all main aspects of decision-making including reward, priors, and perceptual cues (observations) from the perspective of an optimal probabilistic decision-makee combine reward and belief, such as the ratio of expected values of different actions, might work, but they are totally arbitrary and do not generalize to different tasks. We are looking for normative model that explains why such relationship exists between confidence and task parameters in generalizable fashion.
that also aligned with reality. Notably, heuristics that
3 Model
To formally model the decision confidence, we use the idea of optimality in the POMDP framework, where optimality is binary variable that reflects receiving the maximum reward. The probability of optimality describes subject's confidence or internal sense of whether their decision was optimal. This can be viewed as an agent choosing actions maximize their total reward, modeled by POMDP, and later evaluating the optimality of their decision.
Notably, the decision-making process might involve multiple actions. Therefore, what re- fer to as "decision" is, in fact, a sequence of actions, given an observation after each action a1 22, 02, 23 .... ar which is called trajectory As the agent knows the observation function and the system is Markovian, this trajectory could also be expressed with belief states instead of observations:
T == a1 b2, 02 ba..b.,... 0.1
Consequently, the probability of observing trajectory r in an optimal agent is p(T|O = 1, b1). This probability should be maximum for a trajectory that is generated by the optimal policy x².
With strict interpretation of optimality, only trajectories that are generated from the optimal policy n' are optimal. This means that confidence is 1 for these trajectories and for others. This "hard" definition of optimality punishes all non-optimal trajectories the same, meaning that a trajectory generated by suboptimal policy is considered as non-optimal trajectory with the minimum possible outcome. This is not iieealespeciallyy when the agents are humans, as humans are inherently suboptimal.
Given the fact that the system is Markovian and each policy is a mapping from belief states to actions, the probability of optimality can be expressed for each action given the belief state, i.e. p(o, 1|a,,bg (Fig. [ left plot). With this representation, the probability of optimality for the whole trajectory can be expressed as p(01.H 1/7). Consequently, the probability of trajectory being optimal is:
H
p(T(or.)) & o1.)) = p(b)) II p(o: = 1|b,, a, a.)p((b+11le a.).
(5)
If the belief state dynamics deterministic, i.e., each action in each state leads to exactly one state and one observation, we would have:
p(T(orH)) I(p(T) + 0) II p(o: az).
(6)
t=0
A high probability of optimality for a given belief state and action pair should reflect higher reward for that pair of belief state and action (as higher optimality denotes a higher reward). Optimality should then be defined so that the total probability of optimality reflects the total reward sum. To transit between this total probability of optimality product and total reward sum, the probability of optimality for a given belief state and action is set to the exponentiation of the reward of the same belief state and action:
p(orlbe, a1) (bc,)))

---

TeP
20 010
10A
Choblo
Graphical model of the framework that measures the probability of making the optimal decision confidence. Right: Experimental setup of perceptual decision making task with varying prior and reward distribution (picture from [T).
Figure Left:
To make p(o,lb, ax) between O and 1, we can make all rewards negative by subtracting the maximum possible reward from them.
Notably, the above choice still looks arbitrary as there are many other functions that satisfy the constraints of our problem. Beyond being intuitive way to relate probability and sum, if optimality defined for an agent as it is in equation then the agent will jointly maximize the total reward and the information entropy of the policy, which is also why our framework is normative model. This result can be shown by deriving the policy from the optimal trajectory as defined above. One way to derive such policy is approximate p(T[01.H) If the approximation of p(o, l|a, b,) expressed with policy π (ac the approximation of optimal trajectory will be:
H
p(T) = p(bo) II p(b,-1/b, a (aelb))
(8)
t=0
and the desired policy ((olbbe) can be obtained by minimizing the KL-divergence (maximizing negative KL-divergence) between the approximate and true distributions:
-L((()))P(TToLr))) BT
Σ r(br, ac) log o xaa
(9)
(b [r(be, ad) + H((((a(bb))
This derivation is known as "planning as inference" in the literature 7 8] and equivalent of soft Q-learning on the belief state [O]. Moreover, these equations also show that our definition of the probability of optimality works even when the dynamics of the belief state are stochastic.
3.1 Interpretation
According to our model, the agent makes decisions strictly optimally, like a POMDP. Its evaluation of optimality, however, allows for other trajectories through the concept of soft optimality. We only derived that policy of confidence judgment to demonstrate better why exponentiation of reward (equation 7 reasonable choice. This self-assessment is, to some degree, similar inverse reinforcement learning. Notably, entropy regularization has been proven to be practical approach in inverse reinforcement learning too (9. In this approach, the evaluator allows some suboptimality to consider noises and hidden information. Such allowance in our confidence model considers imperfect learning of the environment or possible trial-to-trial changes the computations. For example, arousal level affects perception and, consequently, performance. While in each trial, the most rewarding action should be chosen based on the received information, consider variation in the quality of this information gathering, e.g., "What if this decision was made under low arousal level?" Maximum Entropy regularization an agnostic approach to these variations and confounding factors.
evaluation should
5

---

Notably, the regularization usually accompanies a parameter 3 with positive value, and p(o=)6 bt, a²) = (b))) (3 2 0) still reflects the monotonic relationship between confidence (proba- bility of being optimal) and the accumulated reward. We did not use that parameter in our equations for simplicity, and the fact that such an extra free parameter did not improve our fits to experimental data. However, with the extra parameter B, this model also include the hard optimality B - inf.
3.2 Using the model on experimental data
Our framework proposes confidence model for any sequential decision decision-making task under uncer- tainty. Specifically:
H confidence x II p(o by, ax) = II c., br(eaa))
(10)
when the trajectory of the subject r = a1, b2, 02 :.......1 Oz in a given trial. Fitting this equation to subject's 's reported confidence could be extremely computationally expensive. One of the main reasons behind such computational cost is the belief state, with a space size exponential in the original state space size in the most general cases. Another reason is the need for normalization, which requires the calculation of the probability of optimality for all possible trajectories. In practice, however, fitting confidence is not intractable due to the simplicity of models of perception and the existing experimental setups. First, behavioral and even neural data on perception have been shown to be accurately modeled with Gaussian distributions. Second, in the current experimental setups, the dynamics of the task (transition function) are often very simple, and the number of actions is very minimal, e.g., one action of choosing one of the options after a perceptual cue. As result, the belief state can always be expressed with a Gaussian distribution with two parameters of mean and variance. In other words, b, could often replaced with two one-dimensional parameters of je and of Finally, if the task has only one step of action selection, which is often the case, the confidence is simply proportional to (F((r))))) where a is the chosen action. While this is convenient in terms of fitting, brings another challenge. Any monotonic function of reward as the probability of optimality aligns it with the maximum total reward concept (instead of the exponential we used in equation 7 to map summation to product). For example, one convenient heuristic is the ratio of the expected reward of different actions, i.e., confidence xx IE, [r(s, a)]. Although this is arbitrary. we tested this intuitive definition of confidence, called "expected value ratio", and compared our definition of confidence on experimental data.
4 Results
We tested our model in two experiments, each focusing on different aspects and potential issues of confidence modeling. The first experiment was a perceptual decision-making task with asymmetric priors and rewards [IT]. We focused on the interaction of reward and perception with confidence in our fits and analyses. The second experiment was value-based decision-making task with no perceptual ambiguity (23). in which we tested our definition of optimality and compared our results with "expected value ratio" as confidence hypothesis. To make our results more readable, we call our confidence model, "soft optimality" decision confidence.
4.1 Perceptual Decision Making Task with varying priors and rewards
This experiment was designed to study the interaction of reward and priors with perceptual confidence [IT]. In this experiment, 10 subjects were shown a Gabor filter tilted left or right and asked report their perception of the direction of the Gabor filter (left or right) and, subsequently, their confidence in their perception (low or high) (as seen in Fig. [ right plot). The difficulty of the trials (the extremity of the orientation of the Gabor filters) was constant across trials and was fitted to each subject before the main experiment to have approximately 70% accuracy. Each subject completed different sessions of this task where the prior probability distribution and the reward distribution for correction choices across the two directions were varied (prior probability was either 3:1, 1:1, or 1:3; payoffs were either 2:4, 3:3, or 4:2). The subjects were told the exact prior and reward distribution before each session and every 50 trials during the session. Each session was completed on separate days, with the fully symmetric trials being completed first, followed by the asymmetric trials in random order. Each session consisted of 700 trials, the first 100 of which were
Experiment:

---

Table 1: AIC values for the fit to all confidence models in perceptual maniing task
Model
Perception
1350.81 42.17
1668.53 + 2,98
1763.63 = 16.18
1302.93 15.63
1441.89 3.19
Soft Optimality
20.12
1674.94
2056.63
2300,59
32.36
Ohservation
1316,56 + 22.86
1675.12 + 6,34
1673.77 = 3,03
1302.33 2 15.6
[441.89 I 3.19
Expected Value Ratio
1346.6 + 40.16
1668.48 1.82
1695.27 = 7.89
1295.91
13.51
1441.34
2.37
Model
6
10
Perception
6.86
52.65
1777.82
Soft Optimality
1827.91 = 22.73
1889.75 + 20.14
1685.49 + 10.5
1656.61
6.59
1358,02 +: 1.07
Observation
1.368.44 3.58
1682.17 12.00
T743
TO.26
1673.23
18.8
T366.
T.47
Expected Ratio
1376.47 4.98
1875.34 42.0
1723.66
Table 2: Rate/Probability of each reporting high confidence in each subject and each model's prediction for the trials with value asymmetry the perceptual decision making task
Model
Experiment
80.28
Perception
60.78
48.58
37.17
16.22
26.93
17.57
51.13
29.77
26.34
68.1
Soft Optimality
79.29
68.72
52.74
61.96
69.84
55.78
68 68.42
50.17
47.0
69.5
Observation
Expected Value Ratio
61.06
49.17
43.4
16.89
27.42
18.82
51.76
32.77
29.69
83.87
discarded from analysis. Subjects received rewards of $0-$20 based on their performance. Notably, the subjects were explicitly instructed to report their confidence in the direction of the stimulus. In other words, they were explicitly requested to report their perception confidence. More details can be found in the original paper of this study [I].
First, we built each subject's model with a POMDP based on their choices. This POMDP contained 2 (hidden) states, each representing one of the directions. The perception of the subject (observation function) was modeled W(-1, and (1, of) from direction left and right respectively. Notably, the actual generative process observations could be different from the learned model by the subject [8]. Therefore, while observations were sampled from ((11, the subject's 's internal model was W(11, o. o? called external observational noise. o² is the internal observational noise. Using the POMDP framework, the choice in each trial was obtained by sampling from the true generative process (with external noise), updating the belief based on the prior and learned observation function (with internal noise), and finally, picking the direction with highest expected reward by combining belief and the reward each direction.
Fitting and Comparison:
Gaussian distribution. Observations came from
The external observational noise was first fit to each subject's choices in symmetric trials through gradient descent. Only asymmetrical trials would the internal observational noise impact the subject's choice. Therefore, the internal observational noise was fit to the subject's choices in subset the trials with prior distribution asymmetries using a grid search and maximum likelihood estimation with Bernoulli likelihood function. The prior distribution was set to the actual prior value communicated to the subjects. Therefore, our model had only two free parameters in the fitting process, i.e., internal and external noise.
Based on the noise parameters obtained from the choice data, we could predict the subject's confidence in each trial according to different confidence models. The main two models were soft optimality confidence (ours) and perception confidence, which is the belief about the choice. We also included two more models. One was the observation likelihood (belief but without considering the prior), which we called observation confidence. The other model was the expected value ratio, as discussed before. For each confidence model, the confidence criterion, the threshold at which confidence is binarized into "low" or "high", was fit to the subjects' confidence reports in trials with asymmetric prior and fully symmetric trials (same subset of trials that we obtained noise parameters from). This fitting process included grid search and maximum likelihood estimation with Bernoulli likelihood function.

---

Table 3: AIC values for the fit to all confidence models with an additional parameter accounting for the choice bias
Model
3
5
Perception
1670.7 +
1774.34
12.47
1443.22
4.41
Soll Optimality
1201.72 = 0.65
1818.99 1 16.09
1676.09 : 4.47
2072.39
42.81
2336.31 = 77.72
Expected Vālue Rati
T364.T5 = 33.0
16644.5 2388
[669.71 = 12.56
1315.6 = 17.07
1433.92 = 1I.62
Model
6
8
9
10
Perception
1386.16 = 12.54
1699.3 + 31.19
1938,9 = 41.8
1767.18 19.07
1371.14 2.06
Soft Optimality
1828.37 19.27
1972.15 + 93.6
1657.1 7.11
1659.87 + 5.6
1360.43 184
Expecteed Value Katio
1379.32 9.61
1669.3 23.49
T866.07 = 29.36
17091 16.48
T363.8 3.87
We tested the four mentioned models on the trials that had an asymmetry in the value distribution, which were not used to fit any of the parameters of these models. Notably, the model's values/rewards of actions were set the values communicated to the subjects and were not free parameters. The AIC value ranges in table I are the results of fitting the model 10 times on randomly shuffled trials. The bolded values are the lowest AIC scores for a given subject, factoring in the range of uncertainty found in the average process across 10 trials. Moreover, table 2 shows the rate/probabilliy of high- confidence reports by each subject and the prediction of the models in these trials (asymmetrical values; averaged over the 10 runs). Comparing these predictions to the actual high-confidence rate is not statistically reliable but more intuitive to humans.
As shown in table [ our decision confidence model (soft optimality) was a better fit for five subjects compared to the perceptual confidence model. This is especially important because the subjects were explicitly instructed to report their perception confidence, yet half reported their confidence in choice. To be more precise, half could not override the assessment mechanism of their decisions. Importantly, adding the other two models (observation and expected value ratio) did not change our result. In fact, they strengthened our claim. The AIC value of our model often differs more significantly from the other three models. In other words, when our model performs better, it usually outperforms others significantly (and vice versa). This shows that the phenomenon we are observing not an artifact of misuse of statistical tests (e.g.. test's assumptions not holding) or fitting a powerful function.
We also got the same results while considering a choice bias in subjects towards one of the directions (as seen in table 3). We modeled this bias by adding a free parameter to the prior of each subject. We did not include the observation model as a comparison for these results because the observation model does not make use of subject's 's prior distribution in its confidence computation. All subjects' behavior that our model fit best originally were also best explained by our model after the addition of the bias parameter.
4.2 Value Based Decision Making Task
To further test our soft-optimality assumption, especially compared to the expected value ratio heuristic, we applied our model to value-based decision making experiment [2]
Experiment: In this experiment, 33 subjects first did a rating task where they were shown pictures of different foods in succession and were asked to rate, on sliding scale. how much they would want to eat that food the end of the experiment (as seen in Fig. 2 once, they were shown each food again and asked to rate it second time to account for noise in their subjective assessment of the food's value. They were not told they would have to rate each food twice. this didn't affect their initial rating of the food allow them to try to memorize their first rating when rating a food for the second time. Subsequently, they were asked to do a decision decision-making task where they were presented with pictures of two of the foods they had previously rated and had to select which the two they would rather eat and rate their confidence their selection on a sliding scale. A "correct" decision defined as decision in which the subject chose the food object that they had rated higher on average between the two rating trials. Further task details are available in the original paper (33)
left plot). After rating each food
Fitting and Comparison: To generate predictions of subjects' confidence for the trials in this experiment, used their average rating of food during the rating trials to be an estimate of their
8

---

0.33
How much would you
like eut this Itam?
Corbuace baline: offfutt
oxpected vilue
0.86
420
0.84
How confident are your
200
0.83
-0.22
AIC soft
Quandaa Yake Offenance
Figure 2: Left: Experimental setup value-based decision task with rating and decision trials (picture from (23).. Middle: Visual demonstration of AIC values for the soft optimality model versus the expected value ratio model on the value-based decision making task. The red dashed line represents equal AIC values. Right: As the quantile value difference increases, soft optimality predicts a rate of confidence increase closer to the experimental rate as compared to the rate predicted by the expected value method.
Table 4: AIC values by subject for different confidence prediction models the value- decision making task
Model
2
3
5
Expected Value Ratio
258.87
224.61
219.06
589.85
210.31
333.47
206.60 60
202.14
190.56
799.15
219.69
Soft Optimality
216.79
225.63
220.58
212.84
210.84
118.28
206.40
218.32
191.94
224.71
219.48
Model
12
13
14
15
16
17
18
19
20
21
Expected Value Ratio Soft Optimality
185.37
261.71
297.92
220.79
223.33
215.82
260.02
417.31
199.57
229.26
199.56
200.20
221.03
Model
233
244
25
26
27
28
29
30
31
32
33
Expected Value Ratio Soft Optimality
207.01
215.36
104.43
343.86
220.60
185.43
484.97
436.96
276.38
76.54
593.16
230.68
219.77
105.06
228.51
220.16
206.52
230,81
227.34
215.55
71.99
228.66
expected value of reward for a given option during the choice tasks. We then used this expected value which of the two choices is the
of the belief state (where the belief state is their hypothesis "correct" choice) in our soft optimality equation and normalized this value to map our confidence prediction the sliding scale that subjects use report their confidence their decision.
To assess our model's ability to predict subjects' confidence reporting behavior, we compared the fit of our soft optimality model to the fit of the expected value ratio model (as seen in Table 4 and Fig. 2 middle plot). Notably, since we used the values obtained from the rating section, none of the models have any free parameters. The additional two models used for comparison in the perceptual decision- making experiment couldn't be used as metrics of comparison this task as there was no perceptual component, which is requirement of using the observation and perception models. On average, our soft optimality outperformed the expected value model with an average AIC value of 206.65 across all subjects as compared optimality model was also a better fit for the majority of the subjects; it best explained 33 subjects' confidence reporting behavior. Moreover, the difference in AIC values significantly larger in subjects whose behavior is better explained by our model (soft optimality) as demonstrated in Fig. [ middle plot. We also compared the relationship between the difference the value of the offered choices and the subject's confidence (relative to their minimum confidence level, i.e. confidence offset, for normalization across all subjects) with the predictions of the two models. Our method better predicts the rate at which the subjects' confidence will correspondingly increase (Fig. 2 right plot). Validating our model on this data set strengthens our conviction that can generalize across multiple types of common decision paradigms and that it can be descriptive of more subjects than other comparable Bayesian approaches. Moreover, it further confirms the practicality of the exponential function in explaining confidence opposed to heuristics like the expected value ratio.
decision-making
average AIC value of 282.11, 11. The soft
out of the

---

Discussion
Our analysis code is also available at We present a normative framework that measures "decision confidence" and captures its interaction with reward, perception, and prior. Our model is essentially planning as inference framework, developed and discussed before 7 10]. Our main contribution here is connecting this framework to the confidence judgment in humans. Notably, soft reinforcement learning has been used to fit subjects' choices in decision-makinn before. However, those were actual decisions in the context of
imperfect rationality. We used this approach in action evaluation. In fact, according to our model, actions themselves follow strict optimality (modeled by a POMDP). We validated our results on two different experiments from different groups. Both datasets are publicly available and accessible in their corresponding papers 11 233.
Current experiments, mostly including only 2 choices and step of action selection, are insufficient to flesh out different aspects of confidence and its models. We believe the generalizability of our framework makes it a great candidate for testing and modeling confidence in more complicated setups. Future experiments could go in multiple directions. One example is confidence assessment in a sequence of actions instead of one action. One important aspect of our model, which is not present in other models and even most experiments, is the ability to assess confidence in a sequence of actions (trajectory). Multiple common experimental setups in the field, such as two-step task for humans (24] and even maze navigation for rodents can be used to study confidence of sequence of actions (whole decision) confidence assessment is added to them. Another example direction is confidence assessment in multiple choices. Two-choice tasks are too simple distinguish between various models. Therefore, some researchers have started focusing on confidence assessment on multiple choices. For example, one study has shown that the difference between the probability of the top two choices explains confidence better than the posterior probability of the most likely state when three choices are presented (5). In that study, the reward the same for all options. It could also be illuminating to test confidence assessment when the rewards of different choices are not equal.
Confidence expression is an important factor in society and affects individuals' interactions. Under- privileged individuals often suffer from low confidence, which may further hurt them in their role in their community. Therefore, we believe accurate estimation of confidence and comparing it with actual performance will reduce the societal gaps and promote fairness. However, computational models of confidence and, in general human behavior are very recent. Therefore, no impactful decision should be made based on these models (e.g., in job applications) until they are tested extensively in numerous controlled setups.
References
[1] Alexandre Pouget, Jan Drugowitsch, and Adam Kepecs. Confidence and certainty: distinct probabilistic quantities for different goals. Nature Neuroscience, 19(3):366-3744 March 2016.
Number: Publisher: Nature Publishing Group.
[2] Roozbeh Kiani and Michael N. Shadlen. Representation of Confidence Associated with a Decision by Neurons in the Parictal Cortex. Science, 324(5928):59--766 May 2009. Publisher: American Association for the Advancement of Science
[3] Jan Drugowitsch, André G Mendonça, Zachary F Mainen, and Alexandre Pouget. Learn- ing optimal decisions with confidence. Proceedings of the National Academy of Sciences, 1 116(49):44822-444) 2019. Publisher: National Acad Sciences.
[4] Koosha Khalvati, Saghar Mirbagheri, Scongmin A. Park, Jean-Claude Dreher, and Rajesh PN Rao. A Bayesian Theory of Conformity in Collective Decision Making. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[5] Hsin Hsin-Hung Li and Wei Ji Ma. Confidence reports in decision decision-making with multiple alternatives violate the Bayesian confidence hypothesis. Nature Communications, 11(1):2004, April 2020. Number: Publisher: Nature Publishing Group.
[6] Michael Shadlen and Daphna Shohamy: Decision Making and Sequential Sampling from Memory, Newron, 90(5))227-93) June 2016.
10

---

[7] Mare Toussaint and Amos Storkey. Probabilistic inference for solving discrete and continuous state Markov Decision Processes. In Proceedings of the 23rd international conference on Machine learning ICMI. '06, pages 945-952, Pittsburgh, Pennsylvania, 2006. ACM Press.
[8] Matthew Botvinick and Mare Toussaint, Planning as inference. Trends Cognitive Sciences, 16(10):485-43) October 2012.
[9] Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd national conference Artificial intelligence Volume 3, AAAI'08, pages 1433-1438, Chicago, Illinois, July 2008. AAAI Press.
[10] Sergey Levine. Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review, May 2018. arXiv:1805.00909 [cs, stat].
[11] Shannon M. Locke, Elon Gaffin-Cahn, Nadia Hosseinizaveh, Pascal Mamassian, and Michael S. Landy. Priors and payoff's confidence judgments. Attention, Perception, & Psychophysics, 82(6):3158-3111 August 2020.
[12] Dayan and N. D. Daw. Decision theory, reinforcement learning, and the brain. Cognitive, Affective, & Behavioral Neuroscience, 8(4):429-453 December 2008.
[13] Rajesh P. N. Rao. Decision Making Under Uncertainty: A Neural Model Based on Partially Observable Markov Decision Processes. Frontiers in Compulational Neuroscience, 4:146, November 2010.
[14] Koosha Khalvati and Rajesh PN Rao. A Bayesian Framework for Modeling Confidence in Perceptual Decision Making. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
[15] Armin Lak, Emily Hueske, Junya Hirokawa, Paul Masset, Torben Ott, Anne E Urai, Tobias H Donner, Matteo Carandini, Susumu Tonegawa, Naoshige Uchida, and Adam Kepecs. Reinforce- ment biases subsequent perceptual decisions when confidence is low, a widespread behavioral phenomenon. eLife, 9:e49834, April 2020. Publisher: eLife Sciences Publications, Ltd.
[16] Anthony R. Cassandra, Leslie Pack Kaelbling, and Michael L. Littman. Acting optimally in partially observable stochastic domains. In Proceedings of the Twelfth AAAI National Conference on Artificial Intelligence, AAA194,, pages 1023-1028, Seattle, Washington, August 1994. AAAI Press.
[17] Joshua I. Gold and Michael N. Shadlen. The Neural Basis of Decision Making. Annual Review of Neuroscience, 30:535-574, 2007.
[18] Koosha Khalvati, Roozbeh Kiani, and Rajesh P. N. Rao. Bayesian inference with incomplete knowledge explains perceptual confidence and its deviations from accuracy. Nature Communi- cations, 12(1):5704, September 2021. Number: Publisher: Nature Publishing Group.
[19] Wei Ji Ma, Konrad Paul Kording, and Daniel Goldreich. Bayesian Models of Perception and Action: An Introduction. MIT Press, 2023.
[20] Benedetto De Martino, Stephen M. Fleming, Neil Garrett, and Raymond J. Dolan, Confidence in value-based choice. Nature Neuroscience, 16 (6(1):105-114) January 2013. Number: Publisher: Nature Publishing Group.
[21] Annika Boldt, Charles Blundell, and Benedetto De Martino. Confidence modulates exploration and exploitation in value-based learning. Neuroscience of Consciousness, (0ni(0): January 2019.
[22] Rubén Moreno-Bote. Decision Confidence and Uncertainty in Diffusion Models with Partially Correlated Neuronal Integrators. Neural Computation, 1787::176-1111, 2010.
[23] Jeroen Brus, Helena Aebersold, Marcus Grueschow, and Rafael Polania. Sources of confidence in value-based choice. Nature Communications, 12(1))7337, December 2021. Number: 1 Publisher: Nature Publishing Group.
[24] Nathaniel D. Daw, Samuel J. Gershman, Ben Seymour, Peter Dayan, and Raymond J. Dolan Model-Based Influences on Humans' Choices and Striatal Prediction Errors. Neu- ron, 104-))2204-12111 March 2011.
11

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the last paragraph of our introduction we outline the scope of what we are going to do in this paper and then in the following sections complete all of the objectives that we' ve outlined the introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
12

---

Justification: In our results section we discuss the limitations of our model through our discussion of how many subjects' behavior the model is most explanaotry for compared to other Bayesian models. We also discuss limitations of scaling of the model in the background section of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: There are full explanations for theoretical results and the references were provide also include detailed thcoretical proofs.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
13

---

Justification: We provide a lot of detail on the equations that we used to get our results and the fitting methods that we used for these equations and both of the datasets we used are freely accessible so one should have all of the information that they need to replicate results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g.. a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Both of the data sets that we used are freely accessible and the code that we used is attached in supplementary material and clearly commented, providing sufficient instruction to reproduce our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips.coo http:///www....eeeeee//////e/ssa
14

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The results section clear on what splits of the data were used for training and testing the model, which parameters were fitting, and how these parameters were fit.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For the main experimental results that had stochasticity in the fitting procedures for them, uncertainty bars shown in the tables to show the variability in fitting across different runs of the model.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
15

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
Justification: The model that we ran wasn't incredibly computationally expensive; cumula- tively, all of the results could be reproduced in couple of hours on a normal laptop so wasn't relevant mention in the main paper for sake of reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have viewed the NeurlPS Code of Ethics and our paper conforms to it in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts of the work preformed is discussed in the discussion
section of the main paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
16

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our model does not have a high risk for misuse as it is more the realm of basic research.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original creators of the 2 open data sets that we use in are experiments are properly credited in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code in included and well documented so that other people can understand how use it.
Guidelines:
17

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The instructions that were communicated to human subjects and the details about compensation for these human subjects are communicated best as know them from the original papers where we got the data sets from.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
While we used data from human subjects for the main results of this paper, we didn't 't not conduct these experiments ourselves and were not part of the ethical review process for these experiments. The papers we took these open data sets from didn't 't describe any potential harms for the study participants.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
18


Reviews:
[{"review_id": "OGWTUtq9LQ", "reviewer": "Reviewer_GT57", "summary": "The submission proposes a model of decision confidence that is applicable to value-based as well as perceptual decision making tasks, and is generally grounded in normative inference. The approach is applied to two real-world datasets.", "questions": "* L321-322 I cannot parse \"we compared the fit of our soft optimality model to the expected value ratio model explained away\". Can you reword? \n* I'm confused about the definition of probability of optimality. I would imagine that there's a function o:f(tau)->[0,1] that maps trajectories to optimal or not, which then induces a probability distribution P(o=1|a..., b...) because of the stochasticity in trajectories. Why is the conditional flipped instead on line 148?\n* L132 typo \"decision-make,r\"", "limitations": "Adequately discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think the motivation is compelling, grounding confidence out normatively is sensible, and a lot of detail is provided about the empirical evaluation, which is somewhat mixed but remains favorable to the proposed model in the value-based decision making dataset. I also think the introduction and background is easy to understand and exhaustively presented.", "weaknesses": "My two biggest concerns are [a] that the results don't favor the proposed model all that strongly or consistently, and [b] that the transformation of reward sum to probability of optimality seems no less heuristic than the alternatives.\n\nRegarding the latter: for all that the paper makes strong claims about deriving a normative notion of confidence from first principles, expression 7 seems a bit heuristic, I'm not clear on why this definition should hold (except if it is there to back into soft Q-learning later in the section -- if so, maybe the paper should just be up front about the fact that it's taking its inspiration from those set of approaches). I'm also a bit surprised that this indicator function is used instead of something more conventional like regret (which would sidestep the issue regarding penalizing all non-optimal trajectories the same as identified on the paragraph starting line 150, and not require this extra heuristic).\n\nRegarding the former: unless I'm missing something, the results in section 4.1 are not particularly conclusive w.r.t. which model best explains the data. I'm partial to the idea that different participants interpret the confidence prompt differently (cf. Ericsson and Simon 1984) but I'm not convinced by the argument that this is actually a result in favor of the proposed model because participants can't \"override their intrinsic mechanism\". The results in section 4.2 are a bit more conclusive, though it can be hard to interpret AIC (vs something like out of sample predictions, or even cross-validated log likelihoods). I'm wondering whether we're seeing an artifact of using an exponential instead of a ratio, where it's easier for the exponential model to saturate and predict very high / low confidences, yielding the relatively large AIC values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive feedback and comments. \n\nConcerns: \nAbout the results, we would like to emphasize that in the first experiment (4.1), subjects were explicitly instructed to report their perceptual confidence (lines 245-247). However, half of them reported their decision confidence (line 287). That’s why we talked about overriding. Notably, even the observation of different effects of prior and value on the subject’s confidence was important and interesting enough that the original paper was published in a prestigious venue in the field of psychophysics a few years ago. That paper was only about the different effects of prior and payoff on various subjects, and it didn’t talk about decision confidence nor a normative model for that. Our model explains that observation in a systematic way. In other words, there was no normative model for that subset of subjects, and we propose one in this paper.  \n\nFor the second experiment, we added another analysis. This result provided more evidence in favor of our model. As demonstrated in Figure 1 of the rebuttal pdf, we looked at how confidence rating changes based on the difference in the value of presented choices. While the confidence increases with the increase in the choice value difference in experimental data and both tested models, the predicted growth rate in our model is significantly closer to the experimental data. Specifically, our model predicts very slow growth of confidence with the value difference increase, just like the experimental data.   \n   \nRegarding the exponential function, which looks like a heuristic by itself, we agree with the reviewer that our framework is based on the soft-Q learning principle, especially as a normative model. We chose this narrative as we found it more intuitive for the concept of confidence. We will add more explanation upfront about expression 7 and emphasize that it seems arbitrary until we reach the soft-Q learning loss function.      \n\nQuestions: \n1-Sorry for the typos; The sentence should be “To assess our model’s ability to predict subjects’ confidence reporting behavior, we compared the fit of our soft optimality model to the fit of expected value ratio model.”\n\n2- The reviewer is right; line 148 should be replaced with “Consequently, the probability of observing trajectory τ in an optimal agent is p(τ |O = 1, b1), which should be maximum for a trajectory that is generated by the optimal policy π∗\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V85Tc3iTew", "reviewer": "Reviewer_Q2rg", "summary": "The present paper presents a normative framework for modeling decision confidence in humans that is generalizable to various tasks and experimental setups. In particular, the authors connect the planning as an inference framework to decision confidence. They validate their model in two different psychophysics experiments, where it is compared to other approaches in explaining subjects’ confidence reports.", "questions": "1. The authors write that \"confidence is mainly mathematically defined only for scenarios where different choices have the same potential reward\". That is not clear to me. Why is this true?\n\n2. What is meant by asymmetric and symmetric reward functions?\n\n3. The authors write that \"According to our model, the agent makes decisions strictly optimally, like a POMDP. Its evaluation of optimality, however, allows for other trajectories through the concept of soft optimality\". It is unclear to me why this distinction is necessary. If I understand it correctly, the acting part is never evaluated/considered in the experiments.\n\n4. It was unclear to me how priors come into the model (i.e., for the first experiment).\n\n5. How are observation likelihood and perception confidence different?\n\n6. How is the fitting to subjects' reported confidence done? Were there any free parameters? If so, are these fitted on choices, and then evaluated on the confidences, or are they directedly fitted on confidences?\n\n7. Is the POMDP formulation actually needed for experiment 2?", "limitations": "Not really discussed.", "rating": 3, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "In general, I believe that this work has many of the ingredients of a solid project. Linking planning as an inference to decision confidence has potential, and the goal to unify different notions of confidence is ambitious. The selected experiments allow to highlight the advantages of the proposed method.", "weaknesses": "1. Decision confidence is -- to a large extent -- about epistemic uncertainty, as also mentioned by the authors in their discussion of the Bayesian confidence hypothesis. Yet, any representation of epistemic uncertainty is absent in the framework at present.\n\n2. The authors argue that their framework provides normative justifications. However, Equation 7 is never motivated from a normative perspective but rather introduced in a heuristic manner. There could be a potential for a normative justification I believe but it is never formalized. \n\n3. The results presented in Tables 1 and 3 are rather mixed and do not show clear benefits of the proposed framework.\n\nFurthermore, the paper would benefit from further finetuning in terms of writing. To give a few examples:\n\n4. The authors write that \"One of the applications of POMDPs and similar Bayesian frameworks is modeling the behavior “perceptual decision making” [...]\". This implies that their framework is Bayesian, which it is not (see point 1).\n\n5. The authors write that \"we include the idea of optimality to the POMDP framework\". This sounds like the authors invented optimality in the context of POMDPs, which is obviously not the case.\n\n6. The optimality variable is sometimes denoted with O, sometimes with o_{1:H}. There could be more consistency.\n\n7. \"This is not ideal, especially when the agents are humans, which are inherently sub-optimal\" => sounds strange.\n\n8. Equation 5 is introduced as \"the probability of a trajectory being optimal\" but it rather is the probability of producing a trajectory, given that you are optimal. In general, Equation 5 could be untangled a bit more.\n\n9. End of page 5: beta is not introduced and there is a typo in the equation. Furthermore, it should be discussed that changing beta corresponds to scaling of the reward. Perhaps this could also be a footnote if beta is not used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and questions. \n\nOur framework is normative because the agent maximizes the reward and information entropy of the policy (eq 9). We presented our approach somewhat backward to make it more intuitive for confidence representations. As other reviewers also noted, we should emphasize in the paper that eq 7 seems arbitrary and heuristic until we reach eq 9. Importantly, maximizing entropy in the fitting is actually mainly related to epistemic uncertainty, where the agent does not know the underlying model and makes minimal assumptions about it.  \n\nIn response to your questions:\n\n1- Our intention with this sentence was just to communicate that the standard in the field is that confidence is mainly mathematically defined only for scenarios where different choices have the same potential reward. This is likely true because most perceptual decision-making paradigms (which is most frequently where in the past confidence has been defined) feature two choices which will give the exact same reward if they are the correct choice, given the perceptual stimulus, for the trial. We point this out because the goal of our model is to generalize to paradigms which have asymmetric distributions of reward which are much more closely related to real life scenarios. \n\n2- A symmetric reward function is meant to denote the situation where any choice will give the same reward if it is the correct choice while an asymmetric reward function means that is the subject is correct, the reward they will receive depends on the choice they picked. As an example, the perceptual decision-making task in Section 4.1 has trials with both symmetric and asymmetric reward functions. The subject must choose if they think the stimulus was tilted to the right or left based on a brief picture of it they saw; in symmetric trials, they will get a reward of 1 if they correctly choose the direction the stimulus was tilted, while in asymmetric trials, they could get a reward of 3 if they choose right and that is the direction of the stimulus they were shown versus only getting a reward of 1 if they choose left and that is the direction of the stimulus they were shown. In this case, the asymmetric reward would encourage someone who was more unsure of their perception to choose the direction with the higher reward because that might be the option with the greater expected reward.\n\n3- We evaluate the acting part to fit some of our model's parameters, specifically the internal and external observational noise. There is more information in lines 255-269 where we talk about fitting the subject's choices (to fit these subject’s choices we had to evaluate the acting part).\n\n4- If you refer to Equation 10, this is the equation that we use for confidence in the later sections on experimental data; the priors from the first experiment are used to calculate the expected reward (in the exponent) of this equation. For the first experiment the equation we used for expected reward was $E_{s}[r(s,a)]= \\int p (s│z)r(s,a)ds$ where $p(s│z)$  is the posterior probability of the state given the observation and so is defined as $p(s│z) \\propto p(s) p(z|s)$ where $p(s)$ is the prior.\n\n5- Observation likelihood doesn’t not include the prior information given to the subject about the trial while perception confidence does. So observation likelihood is $p(z|s)$ which is just the likelihood of the observation for that trial given the state the subject choose while perception confidence is $p(s)p(z|s)$ which combines the likelihood of the observation with the given prior information (which results in a Bayesian posterior, so this is the Bayesian confidence hypothesis).\n\n6- We didn’t fit any parameters for experiment 2, however, we did fit some for experiment 1. The details of our fitting methods can be found in lines 248-275. We fit 3 different parameters: internal observational noise, external observation noise, and the confidence cutoff (which we used to binarize the continuous values of confidence we generated to the “high” or “low” assessments of confidence that subjects gave). External observational noise was fit using gradient descent while the other two parameters were fit using grid search with maximum likelihood estimation. Internal and external observational noise were fit on subjects’ choices while confidence cutoffs were fit on subjects’ confidence assessments.\n\n7- The POMPD formulation is not strictly “necessary” for experiment 2 as the subject are tasked with only making one decision as opposed to the string of decisions represented in the POMDP framework. Our goal was to define a model that could generalize across many decision-making paradigms so even if the complexity is not fully utilized in experiment 2, it is included as a general framework for modeling decision-making tasks that are more complex than experiment 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iVA2JUayse", "reviewer": "Reviewer_rZps", "summary": "In this work, the Authors propose a normative model of decision confidence extending the planning-as-inference framework with an optimality variable proportional to the softmax over the reward distribution. The model is fitted to the data from two behavioral tasks featuring the participants’ confidence reports; comparisons with other models of confidence are provided.", "questions": "-Please discuss the relevance (or the lack thereof) of existing behavioral/neuronal data in animals to the validation of the proposed model. Additional analyses, if possible, may strengthen the results.\n\n-What experiment could further support the relevance of the proposed model to the confidence computation in humans/animals? What could be the additional steps to verify the proposed model?\n\n-Please discuss your results on why different models are best at explaining the confidence reports of different participants. Is it because different participants use different confidence models or is it because there might be a third model that universally explains their reports? What does the literature have to say about such scenarios?", "limitations": "Different models are best at explaining the data from different participants.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The text is written remarkably well, clear, and easy to follow.\n\nThe problem is well-motivated; the model is novel and, at the same time, rooted in literature.\n\nThe separation between perceptual confidence and decision confidence is principled, well-articulated, and put to a meaningful test through two experimental scenarios meant to distinguish the two.\n\nThe statistics are done in a robust, appropriate way.", "weaknesses": "In the value-based decision-making task, the proposed model best explains the confidence reports only for a fraction of the participants, although a large one. In the perceptual decision-making task, where the perceptual model was naturally expected to dominate, the proposed model best explains half of the participants’ reports. While these results are interesting in their own right, they call for a deeper dive into the issue.\n\nAn ultimate test for the model could be to look into neural activity, which naturally calls for looking into animal confidence data. Other published studies may offer such data and insights as to where to look for corresponding representations in the brain. Perhaps, a line of experimental + theoretical work by Paul Masset, Torben Ott, and Adam Kepecs (in various order/combinations) may be used to further distinguish the perceptual vs. decision confidence and to seek their representations in the brain. Either way, it would be nice to propose an experiment that would allow us to solidify the results in this paper, and derive testable predictions for such an experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for bringing up great points about testing the model in more and better experiments. We agree that these directions need to be discussed in the paper, and we will do so in the final version, which allows one more page. \nCurrent experiments, mostly including only 2 choices and 1 step of action selection, are insufficient to flesh out different aspects of confidence and its models. We believe the generalizability of our framework makes it a great candidate for testing and modeling confidence in more complicated setups. Future experiments could go in multiple directions. Here are a couple of examples:     \n1. Confidence assessment in a sequence of actions instead of one action: One important aspect of our model, which is not present in other models and even most experiments, is the ability to assess confidence in a sequence of actions (trajectory). Multiple common experimental setups in the field can be used to study confidence of a sequence of actions (whole decision) if confidence assessment is added to them:  \n\n1.1 Two-step task: One potentially interesting experiment is assessing the confidence of humans in the two-step task (Daw 2011). The two-step task is a well-known experiment in neuroscience that involves two binary choices in each trial. However, confidence has not been assessed in that experimental setup yet. It is interesting to see how the involvement of multiple actions for each decision affects confidence and whether such an effect is compatible with our model’s prediction.\n\n1.2 Maze navigation with perceptual cues (odor navigation): In one of the classic papers on confidence in neuroscience, Kepecs et al. used the waiting time of the mice for the “expected incoming reward” as a measure of their confidence (Kepecs 2008). Combined with the odor navigation task, one can assess the confidence of mice when multiple actions are involved (e.g. navigating the maze and sampling). Studying the role of different regions, such as the orbitofrontal cortex and hippocampus, in this confidence assessment could be an exciting area of research, too. \n\n2. Confidence in multiple choices (with different values): As also pointed out by another reviewer, two-choice tasks are too simple to distinguish between various models. Therefore, some researchers have started focusing on confidence assessment on multiple choices. For example, one study has shown that the difference between the probability of the top two choices explains confidence better than the posterior probability of the most likely state when three choices are presented (Li 2020). In that study, the reward was the same for all options. It is interesting to see how that works when the rewards of different choices are different. Notably, the softmax function can model the difference between the two best choices, by eliminating the least possible option. Therefore, we expect that such an experiment confirms our framework’s predictions.    \n\nRegarding additional results, we added another analysis in experiment 4.2, looking at the confidence rating based on the choice value difference (figure 1 one rebuttal pdf). While the confidence increases with the increase in the choice value difference in experimental data and both tested models, the predicted growth rate in our model is significantly closer to the experimental data. Specifically, our model predicts very slow growth of confidence with the value difference increase, just like the experimental data.       \n\nRegarding different strategies vs. one for confidence assessment, the literature points out both individual differences (Navajas 2017) and parameters that affect confidence, such as attention, confirmation bias, and various types of noises (sensory, motor, and decision) (Khalvati 2019, Li 2020). Notably, Bayesian frameworks could incorporate these effects, but the model would have more parameters. With the increased parameters affecting decision-making and confidence, one could expect even more significant differences between subjects’ behaviors.\n\nReferences: \n\n-\"Neural correlates, computation and behavioral impact of decision confidence,\"  Adam Kepecs, Naoshige Uchida, Hatim A. Zariwala & Zachary F. Mainen, Nature 2008\n\n-\"Model-based influences on humans’ choices and striatal prediction errors,\" Nathaniel D. Daw, Samuel J. Gershman, Ben Seymour, Peter Dayan, and Raymond J. Dolan, Neuron 2011\n\n-\"The idiosyncratic nature of confidence, \" Joaquin Navajas, Chandni Hindocha, Hebah Foda, Mehdi Keramati, Peter E Latham 5, Bahador Bahrami, Nature Human Behavior 2017\n\n-\"Bayesian inference with incomplete knowledge explains perceptual confidence and its deviations from accuracy\", Koosha Khalvati, Roozbeh Kiani, Rajesh P.N Rao, Nature Communication 2019\n\n-\"Confidence reports in decision-making with multiple alternatives violate the Bayesian confidence hypothesis,\" Hsin-Hung Li & Wei Ji Ma, Nature Communication 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z8VbaNZloU", "reviewer": "Reviewer_R2CF", "summary": "Great paper about confidence with payoffs but maybe not the unifying one yet. They follow the Bayesian Confidence Hypothesis and the optimality assumption to formulate in a POMDP the modelling of perceptual confidence and decision making. The title should be more like modeling decision confidence as soft Q-learning under the reward optimality assumption.", "questions": "Does planning as inference framework always incorporate the information entropy?\nThe summary on MDP, POMDP and belief-MDP is nice but is it actually needed?\nAn example of the latter case is Kalman-filter-like environments where the belief state can always be represented with a Gaussian distribution with two parameters of μt and σt. Not only in Kalman, we can always model the belief state into a variational (factorized) density formed with Gaussians. Furthermore, we can use an encoder to reduce dimensionality. This is the SOTA approach to model-based RL.\n“optimal agent such as POMDP” is a POMDP an optimal agent or a mathematical formalism to model decision making.\n\nWhy defining a trajectory à la optimal planning as inference if every trial is independent?\nWhile it is presented as that the derivation matches the planning as inference, it may be the other way around. So the authors forced the assumptions to end into soft Q-learning.\n“This self-assessment is, to some degree, similar to inverse reinforcement learning.” Is this not too biased to problems with rewards?\nWould be great to explain a mathematical intuition of perceptual confidence.\nAIC tables better explanation on what would be a perfect fit could improve the clarity.\n\nLiterature suggestions:\n\nFleming 2019. Self-Evaluation of Decision-Making: A General Bayesian Framework for Metacognitive Computation.\nMeera, A. A., & Lanillos, P. (2024). Confidence-Aware Decision-Making and Control for Tool Selection. arXiv preprint arXiv:2403.03808.\n\n\nTypo: decision-make,r", "limitations": "Limitations are missing.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1.\tThe motivation is clear and has impact in the decision making community\n2.\tThe example is illustrative.\n3.\tThe mathematical formalism is sound and the Bayesian Confidence Hypothesis is well known.\n4.\tThe evaluation with real human data.\n5.\tReplicability with databases available and code.", "weaknesses": "1.\tAssimilating rewards to different confidence is only valid under the RL framework. In essence is the task that has different confidence.\n2.\tWe model decision confidence as “probability of making the best decision”. This is a strong assumption that drives the whole work. But it may be not the only description. Thus, going against the unified framework. See Fleming 2019. Self-Evaluation of Decision-Making: A General Bayesian Framework for Metacognitive Computation. And its follow-up statement: “decision confidence is the probability of being optimal over a sequence of states and actions given the policy.” It may be in terms of the mathematical conceptualization of the authors, but what about the posterior entropy of the policy? Or the posterior uncertainty of the state-estimation? In the models, this is better clarified: “their confidence in their choice closely matches the probability of choosing the correct option, i.e., the posterior probability of the most probable choice”. I think optimality is a rational agent biased concept. Actually, the issues of optimality are mentioned in the problem definition. Equation 10 is then a strong assumption. \n3.\tThe approach may fail to go beyond more than binary forced choice scenarios. i.e. intuitively, we would expect our confidence in a selected action in a 3 choice scenario to be lower if another unchosen option was extremely close in probability.\n4.\tIn the results, the paper could be more interesting if comparing with posterior entropy of the belief state, the action model and the observation model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and literature suggestions. We will incorporate them into our paper. \n\nWe agree entirely that confidence could be different when more than 2 choices are available. In fact, there is a recent work showing that confidence is better modeled by the difference in probability of the top 2 choices as opposed to the posterior belief of the top choice, when three options are presented (Li 2020). Notably, we predict that this is very well modeled by the SoftMax function, where the probability of the least possible option converges to zero.  Nonetheless, testing the model on such an experiment would be a very interesting follow-up work on our model.\n\nWe also fit the posterior entropy of the belief to subjects’ behavior in experiment 4.1. All AIC values were higher than those of other models (Table 1, rebuttal pdf). This is consistent with this model's poor performance on the three-choice dataset explained above (Li 2020). \n  \nRegarding modeling with Markov Processes, we agree that other frameworks are also available. However, the core concepts of all of these models are the same: Markovian system and Bayesian Inference.\n\nReference: \n\n-\"Confidence reports in decision-making with multiple alternatives violate the Bayesian confidence hypothesis,\" Hsin-Hung Li & Wei Ji Ma, Nature Communication 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Content of PDF: Plot of additional analysis for experiment 2. AIC table of Entropy model for experiment 1.

Meta-review: Modeling subjective decision confidence is a popular topic in the cognitive sciences. From a psychological perspective, understanding how humans self-assess their behavior is of fundamental importance. The work describes and validates a new model formulation for decision confidence based on a POMDP model framework suitable for describing decision behavior in value-based and perceptual
decision tasks.

The strength of the paper is that in contrast to the vast majority of existing confidence models, it present a formulation for decision-confidence that takes into account the reward structure of the task. The model considers subject's confidence reports as indicating their beliefs in having made the "best" decision while defining "best" in terms of highest expected reward. The confidence formulation is normative within the POMDP framework assuming the decision objective is to jointly maximize expected reward and information entropy of the decision policy. Validation against data from multiple human confidence experiments support the model. Also during the rebuttal, the authors presented model fits/predictions for confidence ratings in a three-alternatives perceptual decision task, showing the generalizability of their model.

Reviewers raised minor concerns with regard to the claim of normativity and the model performance in one of the tested datasets. The rebuttal mostly resolved these issues although the formulation remains limited to the assumed optimal behavioral model. However, the paper is well written and describes work that is conceptually novel, technically solid, and has the potential to significantly impact the field of studying human decision confidence.
Decision: Accept (poster)