Hierarchical decision processes that operate over distinct timescales underlie choice and changes in strategy

Decisions are guided by available information and strategies that link information to action. Following a bad outcome, two potential sources of error—flawed strategy and poor information—must be distinguished to improve future performance. In a direction discrimination task where subjects decide by accumulating sensory evidence to a bound, we show that humans disambiguate sources of error by integrating expected accuracy and outcome over multiple choices. The strategy switches when the integral reaches a threshold. A hierarchy of decision processes in which lower levels integrate sensory evidence over short timescales, and higher levels interact with lower levels over longer timescales, quantitatively explains the behavior. Expected accuracy links these two levels and enables adaptive changes of decision strategy. Decision-making in a natural environment depends on a hierarchy of interacting decision processes. A high-level strategy guides ongoing choices, and the outcomes of those choices determine whether or not the strategy should change. When the right decision strategy is uncertain, as in most natural settings, feedback becomes ambiguous because negative outcomes may be due to limited information or bad strategy. Disambiguating the cause of feedback requires active inference and is key to updating the strategy. We hypothesize that the expected accuracy of a choice plays a crucial rule in this inference, and setting the strategy depends on integration of outcome and expectations across choices. We test this hypothesis with a task in which subjects report the net direction of random dot kinematograms with varying difficulty while the correct stimulus−response association undergoes invisible and unpredictable switches every few trials. We show that subjects treat negative feedback as evidence for a switch but weigh it with their expected accuracy. Subjects accumulate switch evidence (in units of log-likelihood ratio) across trials and update their response strategy when accumulated evidence reaches a bound. A computational framework based on these principles quantitatively explains all aspects of the behavior, providing a plausible neural mechanism for the implementation of hierarchical multiscale decision processes. We suggest that a similar neural computation—bounded accumulation of evidence—underlies both the choice and switches in the strategy that govern the choice, and that expected accuracy of a choice represents a key link between the levels of the decision-making hierarchy.

Related papers:

title: Executive control of visual attention in dual-task situations.
abstract: A theory of executive control is presented that proposes that executive processes control subordinate processes by manipulating their parameters, reconfiguring them to respond in accord with the current task set. It adopts C. Bundesen's (1990) theory of visual attention (TVA) and R. M. Nosofsky and T. J. Palmeri's (1997) exemplar-based random walk (EBRW) as the theory of subordinate processes. It assumes that a task set is a set of TVA and EBRW parameters sufficient to perform a task and that set switching involves changing those parameters. The theory solves 2 computational problems that emerge in dual-task situations: the binding problem and the serial order problem. It can perform dual tasks in series or in parallel but prefers the serial strategy because it is faster and it solves the binding problem naturally. The theory accounts for concurrence cost, set-switching cost, crosstalk between tasks, and the modulation of crosstalk by task set.

title: Hierarchically organized behavior and its neural foundations: A reinforcement learning perspective
abstract: Research on human and animal behavior has long emphasized its hierarchical structure-the divisibility of ongoing behavior into discrete tasks, which are comprised of subtask sequences, which in turn are built of simple actions. The hierarchical structure of behavior has also been of enduring interest within neuroscience, where it has been widely considered to reflect prefrontal cortical functions. In this paper, we reexamine behavioral hierarchy and its neural substrates from the point of view of recent developments in computational reinforcement learning. Specifically, we consider a set of approaches known collectively as hierarchical reinforcement learning, which extend the reinforcement learning paradigm by allowing the learning agent to aggregate actions into reusable subroutines or skills. A close look at the components of hierarchical reinforcement learning suggests how they might map onto neural structures, in particular regions within the dorsolateral and orbital prefrontal cortex. It also suggests specific ways in which hierarchical reinforcement learning might provide a complement to existing psychological models of hierarchically structured behavior. A particularly important question that hierarchical reinforcement learning brings to the fore is that of how learning identifies new action routines that are likely to provide useful building blocks in solving a wide range of future problems. Here and at many other points, hierarchical reinforcement learning offers an appealing framework for investigating the computational and neural underpinnings of hierarchically structured behavior.

title: Representation of Confidence Associated with a Decision by Neurons in the Parietal Cortex
abstract: www.sciencemag.org (this information is current as of May 26, 2009 ): The following resources related to this article are available online at http://www.sciencemag.org/cgi/content/full/324/5928/759 version of this article at: including high-resolution figures, can be found in the online Updated information and services, http://www.sciencemag.org/cgi/content/full/324/5928/759/DC1 can be found at: Supporting Online Material http://www.sciencemag.org/cgi/content/full/324/5928/759#otherarticles , 15 of which can be accessed for free: cites 55 articles This article  subject collections This article appears in the following http://www.sciencemag.org/about/permissions.dtl in whole or in part can be found at: this article permission to reproduce of this article or about obtaining reprints Information about obtaining

title: Choice Certainty Is Informed by Both Evidence and Decision Time
abstract: "Degree of certainty" refers to the subjective belief, prior to feedback, that a decision is correct. A reliable estimate of certainty is essential for prediction, learning from mistakes, and planning subsequent actions when outcomes are not immediate. It is generally thought that certainty is informed by a neural representation of evidence at the time of a decision. Here we show that certainty is also informed by the time taken to form the decision. Human subjects reported simultaneously their choice and confidence about the direction of a noisy display of moving dots. Certainty was inversely correlated with reaction times and directly correlated with motion strength. Moreover, these correlations were preserved even for error responses, a finding that contradicts existing explanations of certainty based on signal detection theory. We also contrived a stimulus manipulation that led to longer decision times without affecting choice accuracy, thus demonstrating that deliberation time itself informs the estimate of certainty. We suggest that elapsed decision time informs certainty because it serves as a proxy for task difficulty.

title: Neuronal Correlates of Metacognition in Primate Frontal Cortex
abstract: Humans are metacognitive: they monitor and control their cognition. Our hypothesis was that neuronal correlates of metacognition reside in the same brain areas responsible for cognition, including frontal cortex. Recent work demonstrated that nonhuman primates are capable of metacognition, so we recorded from single neurons in the frontal eye field, dorsolateral prefrontal cortex, and supplementary eye field of monkeys (Macaca mulatta) that performed a metacognitive visual-oculomotor task. The animals made a decision and reported it with a saccade, but received no immediate reward or feedback. Instead, they had to monitor their decision and bet whether it was correct. Activity was correlated with decisions and bets in all three brain areas, but putative metacognitive activity that linked decisions to appropriate bets occurred exclusively in the SEF. Our results offer a survey of neuronal correlates of metacognition and implicate the SEF in linking cognitive functions over short periods of time.

title: Neural correlates, computation and behavioural impact of decision confidence
abstract: Humans and other animals must often make decisions on the basis of imperfect evidence. Statisticians use measures such as P values to assign degrees of confidence to propositions, but little is known about how the brain computes confidence estimates about decisions. We explored this issue using behavioural analysis and neural recordings in rats in combination with computational modelling. Subjects were trained to perform an odour categorization task that allowed decision confidence to be manipulated by varying the distance of the test stimulus to the category boundary. To understand how confidence could be computed along with the choice itself, using standard models of decision-making, we defined a simple measure that quantified the quality of the evidence contributing to a particular decision. Here we show that the firing rates of many single neurons in the orbitofrontal cortex match closely to the predictions of confidence models and cannot be readily explained by alternative mechanisms, such as learning stimulus–outcome associations. Moreover, when tested using a delayed reward version of the task, we found that rats’ willingness to wait for rewards increased with confidence, as predicted by the theoretical model. These results indicate that confidence estimates, previously suggested to require ‘metacognition’ and conscious awareness, are available even in the rodent brain, can be computed with relatively simple operations, and can drive adaptive behaviour. We suggest that confidence estimation may be a fundamental and ubiquitous component of decision-making.

title: How to measure metacognition
abstract: The ability to recognize one's own successful cognitive processing, in e.g., perceptual or memory tasks, is often referred to as metacognition. How should we quantitatively measure such ability? Here we focus on a class of measures that assess the correspondence between trial-by-trial accuracy and one's own confidence. In general, for healthy subjects endowed with metacognitive sensitivity, when one is confident, one is more likely to be correct. Thus, the degree of association between accuracy and confidence can be taken as a quantitative measure of metacognition. However, many studies use a statistical correlation coefficient (e.g., Pearson's r) or its variant to assess this degree of association, and such measures are susceptible to undesirable influences from factors such as response biases. Here we review other measures based on signal detection theory and receiver operating characteristics (ROC) analysis that are “bias free,” and relate these quantities to the calibration and discrimination measures developed in the probability estimation literature. We go on to distinguish between the related concepts of metacognitive bias (a difference in subjective confidence despite basic task performance remaining constant), metacognitive sensitivity (how good one is at distinguishing between one's own correct and incorrect judgments) and metacognitive efficiency (a subject's level of metacognitive sensitivity given a certain level of task performance). Finally, we discuss how these three concepts pose interesting questions for the study of metacognition and conscious awareness.

title: Two-stage dynamic signal detection: a theory of choice, decision time, and confidence.
abstract: The 3 most often-used performance measures in the cognitive and decision sciences are choice, response or decision time, and confidence. We develop a random walk/diffusion theory-2-stage dynamic signal detection (2DSD) theory-that accounts for all 3 measures using a common underlying process. The model uses a drift diffusion process to account for choice and decision time. To estimate confidence, we assume that evidence continues to accumulate after the choice. Judges then interrupt the process to categorize the accumulated evidence into a confidence rating. The model explains all known interrelationships between the 3 indices of performance. Furthermore, the model also accounts for the distributions of each variable in both a perceptual and general knowledge task. The dynamic nature of the model also reveals the moderating effects of time pressure on the accuracy of choice and confidence. Finally, the model specifies the optimal solution for giving the fastest choice and confidence rating for a given level of choice and confidence accuracy. Judges are found to act in a manner consistent with the optimal solution when making confidence judgments.

title: Post choice information integration as a causal determinant of confidence: Novel data and a computational account
abstract: Confidence judgments are pivotal in the performance of daily tasks and in many domains of scientific research including the behavioral sciences, psychology and neuroscience. Positive resolution i.e., the positive correlation between choice-correctness and choice-confidence is a critical property of confidence judgments, which justifies their ubiquity. In the current paper, we study the mechanism underlying confidence judgments and their resolution by investigating the source of the inputs for the confidence-calculation. We focus on the intriguing debate between two families of confidence theories. According to single stage theories, confidence is based on the same information that underlies the decision (or on some other aspect of the decision process), whereas according to dual stage theories, confidence is affected by novel information that is collected after the decision was made. In three experiments, we support the case for dual stage theories by showing that post-choice perceptual availability manipulations exert a causal effect on confidence-resolution in the decision followed by confidence paradigm. These finding establish the role of RT2, the duration of the post-choice information-integration stage, as a prime dependent variable that theories of confidence should account for. We then present a novel list of robust empirical patterns ('hurdles') involving RT2 to guide further theorizing about confidence judgments. Finally, we present a unified computational dual stage model for choice, confidence and their latencies namely, the collapsing confidence boundary model (CCB). According to CCB, a diffusion-process choice is followed by a second evidence-integration stage towards a stochastic collapsing confidence boundary. Despite its simplicity, CCB clears the entire list of hurdles.

title: Modeling confidence judgments, response times, and multiple choices in decision making: recognition memory and motion discrimination.
abstract: Confidence in judgments is a fundamental aspect of decision making, and tasks that collect confidence judgments are an instantiation of multiple-choice decision making. We present a model for confidence judgments in recognition memory tasks that uses a multiple-choice diffusion decision process with separate accumulators of evidence for the different confidence choices. The accumulator that first reaches its decision boundary determines which choice is made. Five algorithms for accumulating evidence were compared, and one of them produced proportions of responses for each of the choices and full response time distributions for each choice that closely matched empirical data. With this algorithm, an increase in the evidence in one accumulator is accompanied by a decrease in the others so that the total amount of evidence in the system is constant. Application of the model to the data from an earlier experiment (Ratcliff, McKoon, & Tindall, 1994) uncovered a relationship between the shapes of z-transformed receiver operating characteristics and the behavior of response time distributions. Both are explained in the model by the behavior of the decision boundaries. For generality, we also applied the decision model to a 3-choice motion discrimination task and found it accounted for data better than a competing class of models. The confidence model presents a coherent account of confidence judgments and response time that cannot be explained with currently popular signal detection theory analyses or dual-process models of recognition.

title: Relation between Belief and Performance in Perceptual Decision Making
abstract: In an uncertain and ambiguous world, effective decision making requires that subjects form and maintain a belief about the correctness of their choices, a process called meta-cognition. Prediction of future outcomes and self-monitoring are only effective if belief closely matches behavioral performance. Equality between belief and performance is also critical for experimentalists to gain insight into the subjects' belief by simply measuring their performance. Assuming that the decision maker holds the correct model of the world, one might indeed expect that belief and performance should go hand in hand. Unfortunately, we show here that this is rarely the case when performance is defined as the percentage of correct responses for a fixed stimulus, a standard definition in psychophysics. In this case, belief equals performance only for a very narrow family of tasks, whereas in others they will only be very weakly correlated. As we will see it is possible to restore this equality in specific circumstances but this remedy is only effective for a decision-maker, not for an experimenter. We furthermore show that belief and performance do not match when conditioned on task difficulty – as is common practice when plotting the psychometric curve – highlighting common pitfalls in previous neuroscience work. Finally, we demonstrate that miscalibration and the hard-easy effect observed in humans' and other animals' certainty judgments could be explained by a mismatch between the experimenter's and decision maker's expected distribution of task difficulties. These results have important implications for experimental design and are of relevance for theories that aim to unravel the nature of meta-cognition.

title: Neural correlates of strategic reasoning during competitive games
abstract: Although human and animal behaviors are largely shaped by reinforcement and punishment, choices in social settings are also influenced by information about the knowledge and experience of other decision-makers. During competitive games, monkeys increased their payoffs by systematically deviating from a simple heuristic learning algorithm and thereby countering the predictable exploitation by their computer opponent. Neurons in the dorsomedial prefrontal cortex (dmPFC) signaled the animal’s recent choice and reward history that reflected the computer’s exploitative strategy. The strength of switching signals in the dmPFC also correlated with the animal’s tendency to deviate from the heuristic learning algorithm. Therefore, the dmPFC might provide control signals for overriding simple heuristic learning algorithms based on the inferred strategies of the opponent. Neuronal responses in the dorsomedial prefrontal cortex predict choices and switches in gaming strategies in monkeys. Smart monkeys can outwit a computer What happens in the brain when we are learning to compete against an opponent? Seo et al. observed monkeys competing against a computer that can adapt to the monkey's behavior. The monkeys switched their learning strategies when they worked out that their opponent was reacting to their behavior. The responses of the dorsomedial prefrontal cortex cells in the monkey brains predicted their choices and switches in strategies. Science, this issue p. 340

title: Rational regulation of learning dynamics by pupil–linked arousal systems
abstract: The ability to make inferences about the current state of a dynamic process requires ongoing assessments of the stability and reliability of data generated by that process. We found that these assessments, as defined by a normative model, were reflected in nonluminance-mediated changes in pupil diameter of human subjects performing a predictive-inference task. Brief changes in pupil diameter reflected assessed instabilities in a process that generated noisy data. Baseline pupil diameter reflected the reliability with which recent data indicate the current state of the data-generating process and individual differences in expectations about the rate of instabilities. Together these pupil metrics predicted the influence of new data on subsequent inferences. Moreover, a task- and luminance-independent manipulation of pupil diameter predictably altered the influence of new data. Thus, pupil-linked arousal systems can help to regulate the influence of incoming data on existing beliefs in a dynamic environment.

title: The rat as particle filter
abstract: The core tenet of Bayesian modeling is that subjects represent beliefs as distributions over possible hypotheses. Such models have fruitfully been applied to the study of learning in the context of animal conditioning experiments (and analogously designed human learning tasks), where they explain phenomena such as retrospective revaluation that seem to demonstrate that subjects entertain multiple hypotheses simultaneously. However, a recent quantitative analysis of individual subject records by Gallistel and colleagues cast doubt on a very broad family of conditioning models by showing that all of the key features the models capture about even simple learning curves are artifacts of averaging over subjects. Rather than smooth learning curves (which Bayesian models interpret as revealing the gradual tradeoff from prior to posterior as data accumulate), subjects acquire suddenly, and their predictions continue to fluctuate abruptly. These data demand revisiting the model of the individual versus the ensemble, and also raise the worry that more sophisticated behaviors thought to support Bayesian models might also emerge artifactually from averaging over the simpler behavior of individuals. We suggest that the suddenness of changes in subjects’ beliefs (as expressed in conditioned behavior) can be modeled by assuming they are conducting inference using sequential Monte Carlo sampling with a small number of samples — one, in our simulations. Ensemble behavior resembles exact Bayesian models since, as in particle filters, it averages over many samples. Further, the model is capable of exhibiting sophisticated behaviors like retrospective revaluation at the ensemble level, even given minimally sophisticated individuals that do not track uncertainty from trial to trial. These results point to the need for more sophisticated experimental analysis to test Bayesian models, and refocus theorizing on the individual, while at the same time clarifying why the ensemble may be of interest.

title: The perception of probability.
abstract: We present a computational model to explain the results from experiments in which subjects estimate the hidden probability parameter of a stepwise nonstationary Bernoulli process outcome by outcome. The model captures the following results qualitatively and quantitatively, with only 2 free parameters: (a) Subjects do not update their estimate after each outcome; they step from one estimate to another at irregular intervals. (b) The joint distribution of step widths and heights cannot be explained on the assumption that a threshold amount of change must be exceeded in order for them to indicate a change in their perception.  The perceived probability sometimes changes dramatically from one observation to the next. (g) Subjects sometimes have second thoughts about a previous change perception, after observing further outcomes. (h) The frequency with which they perceive changes moves in the direction of the true frequency over sessions. (Explaining this finding requires 2 additional parametric assumptions.) The model treats the perception of the current probability as a by-product of the construction of a compact encoding of the experienced sequence in terms of its change points. It illustrates the why and the how of intermittent Bayesian belief updating and retrospective revision in simple perception. It suggests a reinterpretation of findings in the recent literature on the neurobiology of decision making.

title: Learning the value of information in an uncertain world
abstract: Our decisions are guided by outcomes that are associated with decisions made in the past. However, the amount of influence each past outcome has on our next decision remains unclear. To ensure optimal decision-making, the weight given to decision outcomes should reflect their salience in predicting future outcomes, and this salience should be modulated by the volatility of the reward environment. We show that human subjects assess volatility in an optimal manner and adjust decision-making accordingly. This optimal estimate of volatility is reflected in the fMRI signal in the anterior cingulate cortex (ACC) when each trial outcome is observed. When a new piece of information is witnessed, activity levels reflect its salience for predicting future outcomes. Furthermore, variations in this ACC signal across the population predict variations in subject learning rates. Our results provide a formal account of how we weigh our different experiences in guiding our future actions.

title: Uncertainty, Neuromodulation, and Attention
abstract: Uncertainty in various forms plagues our interactions with the environment. In a Bayesian statistical framework, optimal inference and prediction, based on unreliable observations in changing contexts, require the representation and manipulation of different forms of uncertainty. We propose that the neuromodulators acetylcholine and norepinephrine play a major role in the brain's implementation of these uncertainty computations. Acetylcholine signals expected uncertainty, coming from known unreliability of predictive cues within a context. Norepinephrine signals unexpected uncertainty, as when unsignaled context switches produce strongly unexpected observations. These uncertainty signals interact to enable optimal inference and learning in noisy and changeable environments. This formulation is consistent with a wealth of physiological, pharmacological, and behavioral data implicating acetylcholine and norepinephrine in specific aspects of a range of cognitive processes. Moreover, the model suggests a class of attentional cueing tasks that involve both neuromodulators and shows how their interactions may be part-antagonistic, part-synergistic.

title: Reversal Learning and Dopamine: A Bayesian Perspective
abstract: Reversal learning has been studied as the process of learning to inhibit previously rewarded actions. Deficits in reversal learning have been seen after manipulations of dopamine and lesions of the orbitofrontal cortex. However, reversal learning is often studied in animals that have limited experience with reversals. As such, the animals are learning that reversals occur during data collection. We have examined a task regime in which monkeys have extensive experience with reversals and stable behavioral performance on a probabilistic two-arm bandit reversal learning task. We developed a Bayesian analysis approach to examine the effects of manipulations of dopamine on reversal performance in this regime. We find that the analysis can clarify the strategy of the animal. Specifically, at reversal, the monkeys switch quickly from choosing one stimulus to choosing the other, as opposed to gradually transitioning, which might be expected if they were using a naive reinforcement learning (RL) update of value. Furthermore, we found that administration of haloperidol affects the way the animals integrate prior knowledge into their choice behavior. Animals had a stronger prior on where reversals would occur on haloperidol than on levodopa (l-DOPA) or placebo. This strong prior was appropriate, because the animals had extensive experience with reversals occurring in the middle of the block. Overall, we find that Bayesian dissection of the behavior clarifies the strategy of the animals and reveals an effect of haloperidol on integration of prior information with evidence in favor of a choice reversal.

title: Detecting and predicting changes
abstract: When required to predict sequential events, such as random coin tosses or basketball free throws, people reliably use inappropriate strategies, such as inferring temporal structure when none is present. We investigate the ability of observers to predict sequential events in dynamically changing environments, where there is an opportunity to detect true temporal structure. In two experiments we demonstrate that participants often make correct statistical decisions when asked to infer the hidden state of the data generating process. However, when asked to make predictions about future outcomes, accuracy decreased even though normatively correct responses in the two tasks were identical. A particle filter model accounts for all data, describing performance in terms of a plausible psychological process. By varying the number of particles, and the prior belief about the probability of a change occurring in the data generating process, we were able to model most of the observed individual differences.

title: Bounded Integration in Parietal Cortex Underlies Decisions Even When Viewing Duration Is Dictated by the Environment
abstract: Decisions about sensory stimuli are often based on an accumulation of evidence in time. When subjects control stimulus duration, the decision terminates when the accumulated evidence reaches a criterion level. Under many natural circumstances and in many laboratory settings, the environment, rather than the subject, controls the stimulus duration. In these settings, it is generally assumed that subjects commit to a choice at the end of the stimulus stream. Indeed, failure to benefit from the full stream of information is interpreted as a sign of imperfect accumulation or memory leak. Contrary to these assumptions, we show that monkeys performing a direction discrimination task commit to a choice when the accumulated evidence reaches a threshold level (or bound), sometimes long before the end of stimulus. This bounded accumulation of evidence is reflected in the activity of neurons in the lateral intraparietal cortex. Thus, the readout of visual cortex embraces a termination rule to limit processing even when potentially useful information is available.

title: Neural basis of a perceptual decision in the parietal cortex (area LIP) of the rhesus monkey.
abstract: We recorded the activity of single neurons in the posterior parietal cortex (area LIP) of two rhesus monkeys while they discriminated the direction of motion in random-dot visual stimuli. The visual task was similar to a motion discrimination task that has been used in previous investigations of motion-sensitive regions of the extrastriate cortex. The monkeys were trained to decide whether the direction of motion was toward one of two choice targets that appeared on either side of the random-dot stimulus. At the end of the trial, the monkeys reported their direction judgment by making an eye movement to the appropriate target. We studied neurons in LIP that exhibited spatially selective persistent activity during delayed saccadic eye movement tasks. These neurons are thought to carry high-level signals appropriate for identifying salient visual targets and for guiding saccadic eye movements. We arranged the motion discrimination task so that one of the choice targets was in the LIP neuron's response field (RF) while the other target was positioned well away from the RF. During motion viewing, neurons in LIP altered their firing rate in a manner that predicted the saccadic eye movement that the monkey would make at the end of the trial. The activity thus predicted the monkey's judgment of motion direction. This predictive activity began early in the motion-viewing period and became increasingly reliable as the monkey viewed the random-dot motion. The neural activity predicted the monkey's direction judgment on both easy and difficult trials (strong and weak motion), whether or not the judgment was correct. In addition, the timing and magnitude of the response was affected by the strength of the motion signal in the stimulus. When the direction of motion was toward the RF, stronger motion led to larger neural responses earlier in the motion-viewing period. When motion was away from the RF, stronger motion led to greater suppression of ongoing activity. Thus the activity of single neurons in area LIP reflects both the direction of an impending gaze shift and the quality of the sensory information that instructs such a response. The time course of the neural response suggests that LIP accumulates sensory signals relevant to the selection of a target for an eye movement.

title: Neuronal basis of sequential foraging decisions in a patchy environment
abstract: Deciding when to leave a depleting resource to exploit another is a fundamental problem for all decision makers. The neuronal mechanisms mediating patch-leaving decisions remain unknown. We found that neurons in primate (Macaca mulatta) dorsal anterior cingulate cortex, an area that is linked to reward monitoring and executive control, encode a decision variable signaling the relative value of leaving a depleting resource for a new one. Neurons fired during each sequential decision to stay in a patch and, for each travel time, these responses reached a fixed threshold for patch-leaving. Longer travel times reduced the gain of neural responses for choosing to stay in a patch and increased the firing rate threshold mandating patch-leaving. These modulations more closely matched behavioral decisions than any single task variable. These findings portend an understanding of the neural basis of foraging decisions and endorse the unification of theoretical and experimental work in ecology and neuroscience.

title: The Wave Theory of Difference and Similarity
abstract: Psychophysical Origins of Difference and Similarity. The Elements. Challenges and Extensions. The Origin of the Psychometric Function. The Law of Comparative Judgment. Equality and Affective Value.  Discrimination and Response Bias. Discrimination and Response Strategy.  Response Probabilities and Response Times.   Accumulation of Wave Differences. Wald's Fundamental Identity. Origins of the Logistic Response Function. Gaussian Comparisons. Symmetric Stimuli.

title: The physics of optimal decision making: a formal analysis of models of performance in two-alternative forced-choice tasks.
abstract: In this article, the authors consider optimal decision making in two-alternative forced-choice (TAFC) tasks. They begin by analyzing 6 models of TAFC decision making and show that all but one can be reduced to the drift diffusion model, implementing the statistically optimal algorithm (most accurate for a given speed or fastest for a given accuracy). They prove further that there is always an optimal trade-off between speed and accuracy that maximizes various reward functions, including reward rate (percentage of correct responses per unit time), as well as several other objective functions, including ones weighted for accuracy. They use these findings to address empirical data and make novel predictions about performance under optimality.

title: Decision-making with multiple alternatives
abstract: Simple perceptual tasks have laid the groundwork for understanding the neurobiology of decision-making. Here, we examined this foundation to explain how decision-making circuitry adjusts in the face of a more difficult task. We measured behavioral and physiological responses of monkeys on a two- and four-choice direction-discrimination decision task. For both tasks, firing rates in the lateral intraparietal area appeared to reflect the accumulation of evidence for or against each choice. Evidence accumulation began at a lower firing rate for the four-choice task, but reached a common level by the end of the decision process. The larger excursion suggests that the subjects required more evidence before making a choice. Furthermore, on both tasks, we observed a time-dependent rise in firing rates that may impose a deadline for deciding. These physiological observations constitute an effective strategy for handling increased task difficulty. The differences appear to explain subjects' accuracy and reaction times.

title: Neural Mechanisms of Post-error Adjustments of Decision Policy in Parietal Cortex
abstract: Humans often slow down after mistakes (post-error slowing [PES]), but the neural mechanism and adaptive role of PES remain controversial. We studied changes in the neural mechanisms of decision making after errors in humans and monkeys that performed a motion direction discrimination task. We found that PES is mediated by two factors: a reduction in sensitivity to sensory information and an increase in the decision bound. Both effects are implemented through dynamic changes in the decision-making process. Neuronal responses in the monkey lateral intraparietal area revealed that bound changes are implemented by decreasing an evidence-independent urgency signal. They also revealed a reduction in the rate of evidence accumulation, reflecting reduced sensitivity. These changes in the bound and sensitivity provide a quantitative account of choices and response times. We suggest that PES reflects an adaptive increase of decision bound in anticipation of maladaptive reductions in sensitivity to incoming evidence.

title: A neural mechanism of speed-accuracy tradeoff in macaque area LIP
abstract: Decision making often involves a tradeoff between speed and accuracy. Previous studies indicate that neural activity in the lateral intraparietal area (LIP) represents the gradual accumulation of evidence toward a threshold level, or evidence bound, which terminates the decision process. The level of this bound is hypothesized to mediate the speed-accuracy tradeoff. To test this, we recorded from LIP while monkeys performed a motion discrimination task in two speed-accuracy regimes. Surprisingly, the terminating threshold levels of neural activity were similar in both regimes. However, neurons recorded in the faster regime exhibited stronger evidence-independent activation from the beginning of decision formation, effectively reducing the evidence-dependent neural modulation needed for choice commitment. Our results suggest that control of speed vs accuracy may be exerted through changes in decision-related neural activity itself rather than through changes in the threshold applied to such neural activity to terminate a decision.

title: Caudate Encodes Multiple Computations for Perceptual Decisions
abstract: Perceptual decision making is a complex process that requires multiple computations, including the accumulation of sensory evidence and an ongoing evaluation of the accumulation process to use for prediction and adjustment. Implementing these computations likely involves interactions among many brain regions. For perceptual decisions linked to oculomotor actions, neural correlates of sensory evidence accumulation have been identified in several cortical areas, including the frontal eye field and lateral intraparietal area, and one of their direct, subcortical targets, the superior colliculus. These structures are also connected indirectly, via the basal ganglia. The basal ganglia pathway has been theorized to contribute to perceptual decision making, but the nature of this contribution has yet to be examined directly. Here we show that in monkeys performing a reaction-time visual motion direction-discrimination task, neurons in a primary input structure of the basal ganglia, the caudate nucleus, encode three aspects of decision making: evidence accumulation, evaluation, and choice biases. These results indicate that the basal ganglia pathway can provide important signals to influence and assess perceptual decisions that guide oculomotor behavior.

title: A comparison of macaque behavior and superior colliculus neuronal activity to predictions from models of two-choice decisions.
abstract: Recently, models in psychology have been shown capable of accounting for the full range of behavioral data from simple two-choice decision tasks: mean reaction times for correct and error responses, accuracy, and the reaction time distributions for correct and error responses. At the same time, recent data from neural recordings have allowed investigation of the neural systems that implement such decisions. In the experiment presented here, neural recordings were obtained from superior colliculus prelude/buildup cells in two monkeys while they performed a two-choice task that has been used in humans for testing psychological models of the decision process. The best-developed psychological model, the diffusion model, and a competing model, the Poisson counter model, were explicitly fit to the behavioral data. The pattern of activity shown in the prelude/buildup cells, including the point at which response choices were discriminated, was matched by the evidence accumulation process predicted from the diffusion model using the parameters from the fits to the behavioral data but not by the Poisson counter model. These results suggest that prelude/buildup cells in the superior colliculus, or cells in circuits in which the superior colliculus cells participate, implement a diffusion decision process or a variant of the diffusion process.

title: Decision Making as a Window on Cognition
abstract: A decision is a commitment to a proposition or plan of action based on information and values associated with the possible outcomes. The process operates in a flexible timeframe that is free from the immediacy of evidence acquisition and the real time demands of action itself. Thus, it involves deliberation, planning, and strategizing. This Perspective focuses on perceptual decision making in nonhuman primates and the discovery of neural mechanisms that support accuracy, speed, and confidence in a decision. We suggest that these mechanisms expose principles of cognitive function in general, and we speculate about the challenges and directions before the field.

title: Using Time-Varying Evidence to Test Models of Decision Dynamics: Bounded Diffusion vs. the Leaky Competing Accumulator Model
abstract: When people make decisions, do they give equal weight to evidence arriving at different times? A recent study (Kiani et al., 2008) using brief motion pulses (superimposed on a random moving dot display) reported a primacy effect: pulses presented early in a motion observation period had a stronger impact than pulses presented later. This observation was interpreted as supporting the bounded diffusion (BD) model and ruling out models in which evidence accumulation is subject to leakage or decay of early-arriving information. We use motion pulses and other manipulations of the timing of the perceptual evidence in new experiments and simulations that support the leaky competing accumulator (LCA) model as an alternative to the BD model. While the LCA does include leakage, we show that it can exhibit primacy as a result of competition between alternatives (implemented via mutual inhibition), when the inhibition is strong relative to the leak. Our experiments replicate the primacy effect when participants must be prepared to respond quickly at the end of a motion observation period. With less time pressure, however, the primacy effect is much weaker. For 2 (out of 10) participants, a primacy bias observed in trials where the motion observation period is short becomes weaker or reverses (becoming a recency effect) as the observation period lengthens. Our simulation studies show that primacy is equally consistent with the LCA or with BD. The transition from primacy-to-recency can also be captured by the LCA but not by BD. Individual differences and relations between the LCA and other models are discussed.

title: Dynamics of postdecisional processing of confidence.
abstract: Most cognitive theories assume that confidence and choice happen simultaneously and are based on the same information. The 3 studies presented in this article instead show that confidence judgments can arise, at least in part, from a postdecisional evidence accumulation process. As a result of this process, increasing the time between making a choice and confidence judgment improves confidence resolution. This finding contradicts the notion that confidence judgments are biased by decision makers seeking confirmatory evidence. Further analysis reveals that the improved resolution is due to a reduction in confidence in incorrect responses, while confidence in correct responses remains relatively constant. These results are modeled with a sequential sampling process that allows evidence accumulation to continue after a choice is made and maps the amount of accumulated evidence onto a confidence rating. The cognitive modeling analysis reveals that the rate of evidence accumulation following a choice does slow relative to the rate preceding choice. The analysis also shows that the asymmetry between confidence in correct and incorrect choices is compatible with state-dependent decay in the accumulated evidence: Evidence consistent with the current state results in a deceleration of accumulated evidence and consequently evidence appears to have a decreasing impact on observed confidence. In contrast, evidence inconsistent with the current state results in an acceleration of accumulated evidence toward the opposite direction and consequently evidence appears to have an increasing impact on confidence. Taken together, this process-level understanding of confidence suggests a simple strategy for improving confidence accuracy: take a bit more time to make confidence judgments.

title: The construction of confidence in a perceptual decision
abstract: Decision-making involves the selection of one out of many possible courses of action. A decision may bear on other decisions, as when humans seek a second medical opinion before undergoing a risky surgical intervention. These “meta-decisions” are mediated by confidence judgments—the degree to which decision-makers consider that a choice is likely to be correct. We studied how subjective confidence is constructed from noisy sensory evidence. The psychophysical kernels used to convert sensory information into choice and confidence decisions were precisely reconstructed measuring the impact of small fluctuations in sensory input. This is shown in two independent experiments in which human participants made a decision about the direction of motion of a set of randomly moving dots, or compared the brightness of a group of fluctuating bars, followed by a confidence report. The results of both experiments converged to show that: (1) confidence was influenced by evidence during a short window of time at the initial moments of the decision, and (2) confidence was influenced by evidence for the selected choice but was virtually blind to evidence for the non-selected choice. Our findings challenge classical models of subjective confidence—which posit that the difference of evidence in favor of each choice is the seed of the confidence signal.

title: A Neural Implementation of Wald’s Sequential Probability Ratio Test
abstract: Difficult decisions often require evaluation of samples of evidence acquired sequentially. A sensible strategy is to accumulate evidence, weighted by its reliability, until sufficient support is attained. An optimal statistical approach would accumulate evidence in units of logarithms of likelihood ratios (logLR) to a desired level. Studies of perceptual decisions suggest that the brain approximates an analogous procedure, but a direct test of accumulation, in units of logLR, to a threshold in units of cumulative logLR is lacking. We trained rhesus monkeys to make decisions based on a sequence of evanescent, visual cues assigned different logLR, hence different reliability. Firing rates of neurons in the lateral intraparietal area (LIP) reflected the accumulation of logLR and reached a stereotyped level before the monkeys committed to a decision. The monkeys' choices and reaction times, including their variability, were explained by LIP activity in the context of accumulation of logLR to a threshold.

title: Linear-Nonlinear-Poisson models of primate choice dynamics.
abstract: The equilibrium phenomenon of matching behavior traditionally has been studied in stationary environments. Here we attempt to uncover the local mechanism of choice that gives rise to matching by studying behavior in a highly dynamic foraging environment. In our experiments, 2 rhesus monkeys (Macacca mulatta) foraged for juice rewards by making eye movements to one of two colored icons presented on a computer monitor, each rewarded on dynamic variable-interval schedules. Using a generalization of Wiener kernel analysis, we recover a compact mechanistic description of the impact of past reward on future choice in the form of a Linear-Nonlinear-Poisson model. We validate this model through rigorous predictive and generative testing. Compared to our earlier work with this same data set, this model proves to be a better description of choice behavior and is more tightly correlated with putative neural value signals. Refinements over previous models include hyperbolic (as opposed to exponential) temporal discounting of past rewards, and differential (as opposed to fractional) comparisons of option value. Through numerical simulation we find that within this class of strategies, the model parameters employed by animals are very close to those that maximize reward harvesting efficiency.

title: Decision processes in visual perception
abstract: Read more and get great! That's what the book enPDFd decision processes in visual perception will give for every reader to read this book. This is an on-line book provided in this website. Even this book becomes a choice of someone to read, many in the world also loves it so much. As what we talk, when you read more every page of this decision processes in visual perception, what you will obtain is something great.

title: Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control
abstract: A broad range of neural and behavioral data suggests that the brain contains multiple systems for behavioral choice, including one associated with prefrontal cortex and another with dorsolateral striatum. However, such a surfeit of control raises an additional choice problem: how to arbitrate between the systems when they disagree. Here, we consider dual-action choice systems from a normative perspective, using the computational theory of reinforcement learning. We identify a key trade-off pitting computational simplicity against the flexible and statistically efficient use of experience. The trade-off is realized in a competition between the dorsolateral striatal and prefrontal systems. We suggest a Bayesian principle of arbitration between them according to uncertainty, so each controller is deployed when it should be most accurate. This provides a unifying account of a wealth of experimental evidence about the factors favoring dominance by either system.

title: Effect of stimulus uncertainty on the pupillary dilation response and the vertex evoked potential.
abstract: Scalp recordings of slow baseline shifts (CNV) preceding auditory stimuli, as well as pupillary dilations and evoked potentials to these stimuli, were recorded and averaged in awake adult humans. The experimental situation manipulated the degree of advance knowledge by the subject as to which of two auditory stimuli would be presented next. In the certain condition the subject was told prior to each stimulus which of the two stimuli would be presented next; in the uncertain condition the subject made a guess prior to each stimulus and the occurrence of the stimulus confirmed or disconfirmed the subject's guess. In both the certain and uncertain conditions, the relative probability of occurrence of the two stimuli was varied in different blocks of trials so that each of the two stimuli occurred at 20, 40, 60 and 80% probabilities. Pupillary dilation was found to be larger and the CNV was found to be more negative in the uncertain condition than in the certain condition. Of the evoked potential components, P 3 was larger and N 2 was smaller in the uncertain condition. Somewhat less consistently, P 2 was larger and P 1 and N 1 were smaller in the uncertain condition. P 3 amplitude and peak dilation amplitude decreased monotonically as a function of the rareness of event occurrence (as measured by either stimulus or outcome probability). This relationship held for P 3 in both certain and uncertain conditions, but only in the uncertain condition for pupillary dilation. The P 2 component also showed similar relationships to probability, but less strongly than did the P 3 component. CNV did not vary as a function of stimulus probability in either condition. The results are consistent with the interpretation that the uncertain condition is characterized by a higher arousal level associated with greater task involvement on the part of the subject. The similarity between the pupillary dilation relationship to probability and the P 3 relationship to probability is consistent with an interpretation of P 3 as a "cerebral correlate of the orienting response". But this would require a broadening and redefinition of the orienting response concept in "informational" terms and in any case would fit all the available data on P 3 only with difficulty. A less ambitious formulation at this time would simply suggest that any operation which increases the "salience" of the stimulus to the subject tends to increase the amplitude of P 3 .

title: Heart rate and skin conductance analysis of antecendents and consequences of decision making.
abstract: The current study examined the pattern of heart rate and skin conductance changes preceding risky choices and following outcome for bad, moderate, and good performers on an analogue of the Iowa gambling task (Bechara, Damasio, Damasio, & Anderson, 1994). The task required a choice between four options; two options were followed by a high reward and, unpredictably, an even higher loss (disadvantageous options) and two other options were followed by a small reward but the unpredictable loss was also small (advantageous options). Anticipatory heart rate slowing and skin conductance level were higher preceding disadvantageous relative to advantageous options, but only for good performers. In contrast, heart rate slowed and skin conductance level increased following loss relative to reward outcomes, and these changes were similar for all performance groups. These findings were interpreted to suggest that decision-making impairments in bad performers arise from a weak somatic response generated by secondary inducers (i.e., somatic markers), rather than a weak somatic response generated by primary inducers of reward and punishment.

title: Reinforcement learning can account for associative and perceptual learning on a visual decision task
abstract: We recently showed that improved perceptual performance on a visual motion direction–discrimination task corresponds to changes in how an unmodified sensory representation in the brain is interpreted to form a decision that guides behavior. Here we found that these changes can be accounted for using a reinforcement-learning rule to shape functional connectivity between the sensory and decision neurons. We modeled performance on the basis of the readout of simulated responses of direction-selective sensory neurons in the middle temporal area (MT) of monkey cortex. A reward prediction error guided changes in connections between these sensory neurons and the decision process, first establishing the association between motion direction and response direction, and then gradually improving perceptual sensitivity by selectively strengthening the connections from the most sensitive neurons in the sensory population. The results suggest a common, feedback-driven mechanism for some forms of associative and perceptual learning.

title: Striatal activations signal prediction errors on confidence in the absence of external feedback
abstract: Research on the neural bases of learning has mainly focused on reinforcement learning where the central role of the dopaminergic system is well established. However, in everyday life many decisions are not followed by feedback, in which case humans have been shown to code the most probable outcome into memory. We used functional magnetic resonance imaging (fMRI) to examine the neural basis of internally generated signals on correctness and decision confidence in the complete absence of feedback in a categorization task. During test trials after observational training activation in dopaminergic target regions was modulated by the correctness of the answer similarly as during feedback-based training. Moreover, activation in the nucleus accumbens and putamen was correlated with the prediction error on confidence as estimated by a reinforcement learning model. In this model subjective confidence ratings acquired after each trial served as outcome measure. Activation in the striatum therefore follows a similar pattern in response to prediction errors on confidence as it does during reinforcement learning in response to reward prediction errors, but with respect to internally generated signals based on knowledge of the structure of the environment. Furthermore, ventral striatal activation decreased with stimulus novelty, which might support the allocation of attention to unfamiliar stimuli. These results provide a parsimonious account for the neural bases of learning, indicating overlapping neural substrates of reinforcement learning and learning when outcome information has to be internally constructed.

title: Mesolimbic confidence signals guide perceptual learning in the absence of external feedback
abstract: It is well established that learning can occur without external feedback, yet normative reinforcement learning theories have difficulties explaining such instances of learning. Here, we propose that human observers are capable of generating their own feedback signals by monitoring internal decision variables. We investigated this hypothesis in a visual perceptual learning task using fMRI and confidence reports as a measure for this monitoring process. Employing a novel computational model in which learning is guided by confidence-based reinforcement signals, we found that mesolimbic brain areas encoded both anticipation and prediction error of confidence—in remarkable similarity to previous findings for external reward-based feedback. We demonstrate that the model accounts for choice and confidence reports and show that the mesolimbic confidence prediction error modulation derived through the model predicts individual learning success. These results provide a mechanistic neurobiological explanation for learning without external feedback by augmenting reinforcement models with confidence-based feedback.

title: Hierarchical Models in the Brain
abstract: This paper describes a general model that subsumes many parametric models for continuous data. The model comprises hidden layers of state-space or dynamic causal models, arranged so that the output of one provides input to another. The ensuing hierarchy furnishes a model for many types of data, of arbitrary complexity. Special cases range from the general linear model for static data to generalised convolution models, with system noise, for nonlinear time-series analysis. Crucially, all of these models can be inverted using exactly the same scheme, namely, dynamic expectation maximization. This means that a single model and optimisation scheme can be used to invert a wide range of models. We present the model and a brief review of its inversion to disclose the relationships among, apparently, diverse generative models of empirical data. We then show that this inversion can be formulated as a simple neural network and may provide a useful metaphor for inference and learning in the brain.

title: Precision and neuronal dynamics in the human posterior parietal cortex during evidence accumulation
abstract: Primate studies show slow ramping activity in posterior parietal cortex (PPC) neurons during perceptual decision-making. These findings have inspired a rich theoretical literature to account for this activity. These accounts are largely unrelated to Bayesian theories of perception and predictive coding, a related formulation of perceptual inference in the cortical hierarchy. Here, we tested a key prediction of such hierarchical inference, namely that the estimated precision (reliability) of information ascending the cortical hierarchy plays a key role in determining both the speed of decision-making and the rate of increase of PPC activity. Using dynamic causal modelling of magnetoencephalographic (MEG) evoked responses, recorded during a simple perceptual decision-making task, we recover ramping-activity from an anatomically and functionally plausible network of regions, including early visual cortex, the middle temporal area (MT) and PPC. Precision, as reflected by the gain on pyramidal cell activity, was strongly correlated with both the speed of decision making and the slope of PPC ramping activity. Our findings indicate that the dynamics of neuronal activity in the human PPC during perceptual decision-making recapitulate those observed in the macaque, and in so doing we link observations from primate electrophysiology and human choice behaviour. Moreover, the synaptic gain control modulating these dynamics is consistent with predictive coding formulations of evidence accumulation.

title: Network Resets in Medial Prefrontal Cortex Mark the Onset of Behavioral Uncertainty
abstract: Changing Your Belief The ability to display behavioral flexibility depends on an internal representation of the environment—a framework of beliefs that can be adjusted on the basis of experience. Recording from multiple electrodes in the rat medial prefrontal cortex, Karlsson et al. (p. 135) investigated how ensembles of neurons changed their activity during the performance of a task in which the animal had to update its knowledge of reward contingencies. The results suggest that changes in perceived action-outcome contingencies were associated with abrupt switches in neuronal representations in the rat medial prefrontal cortex. Abrupt neuronal activity in the rat prefrontal cortex indicates a switch from exploitation to exploration. Regions within the prefrontal cortex are thought to process beliefs about the world, but little is known about the circuit dynamics underlying the formation and modification of these beliefs. Using a task that permits dissociation between the activity encoding an animal’s internal state and that encoding aspects of behavior, we found that transient increases in the volatility of activity in the rat medial prefrontal cortex accompany periods when an animal’s belief is modified after an environmental change. Activity across the majority of sampled neurons underwent marked, abrupt, and coordinated changes when prior belief was abandoned in favor of exploration of alternative strategies. These dynamics reflect network switches to a state of instability, which diminishes over the period of exploration as new stable representations are formed.

title: Posterior cingulate cortex: adapting behavior to a changing world
abstract: When has the world changed enough to warrant a new approach? The answer depends on current needs, behavioral flexibility and prior knowledge about the environment. Formal approaches solve the problem by integrating the recent history of rewards, errors, uncertainty and context via Bayesian inference to detect changes in the world and alter behavioral policy. Neuronal activity in posterior cingulate cortex - a key node in the default network - is known to vary with learning, memory, reward and task engagement. We propose that these modulations reflect the underlying process of change detection and motivate subsequent shifts in behavior.

title: Comparison of the Distortion of Probability Information in Decision Under Risk and an Equivalent Visual Task
abstract: Decision makers typically overweight small probabilities and underweight large probabilities. However, there are recent reports that when probability is presented in the form of relative frequencies, this typical pattern reverses. We tested this hypothesis by comparing decision making in two tasks: In one task, probability was stated numerically, and in the other task, it was conveyed through a visual representation. In the visual task, participants chose whether a “stochastic bullet” should be fired at either a large target for a small reward or a small target for a large reward. Participants’ knowledge of probability in the visual task was the result of extensive practice firing bullets at targets. In the classical numerical task, participants chose between pairs of lotteries with probabilities and rewards matched to the probabilities and rewards in the visual task. We found that participants’ probability-weighting functions were significantly different in the two tasks, but the pattern for the visual task was the typical, not the reversed, pattern.