A sampling-based circuit for optimal decision making

One of the central questions of perception is how organisms reliably estimate hidden or abstract quantities of interest using noisy and ambiguous sensory information . Almost equally important is representing the reliability of these estimates , especially in complex environments and situations of risk , where the uncertainty associated with a choice may radically change the optimal course of action . From basic functions , such as cue combination or motor control , to higher level cognitive tasks , such as decision making and planning , there is substantial evidence that both humans and animals represent and use uncertainty information to guide their actions [ 1 ] . Neural correlates have been identified in a number of regions , including the orbitofrontal cortex [ 2 ] , the cingulate cortex [ 3 ] , and the lateral intraparietal area ( LIP ) [ 4 , 5 ] , but the principles behind how uncertainty is represented in neural circuits to support circuit computations remain hotly debated . Since behavior has been shown to be Bayes-optimal in many situations [ 6 ] , the problem of perception can be modelled in the framework of Bayesian inference , whereby an observer combines prior information with current observations according to an internal model to arrive at a posterior estimate of a quantity of interest ( the ‘ latent variable ’ ) . Moreover , natural statistics are strongly non-Gaussian and there is evidence that human subjects use varied , non-Gaussian prior representations to support behavior [ 7 , 8 ] , so observers must be able to perform inference flexibly , efficiently and adaptively . This raises two questions : which neural organizations allow for these kinds of computations , and how are the associated neural representations mapped into behaviorally relevant action plans ? Currently there are several theories for how neural activity may represent probability distributions . One large class of models assume that neural responses encode parameters of underlying posterior distributions ; this includes probabilistic population codes [ 9 , 10 ] , their predecessors , kernel density estimators and distributional population codes [ 11 , 12 ] , and most recently , distributed distributional codes ( DDC ) [ 13 ] . A second class of models relies on stochasticity in recurrent circuits to approxi- 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , . mately encode probability distributions via sampling [ 14 , 15 ] . This idea was originally motivated by the practical success of Markov Chain Monte Carlo ( MCMC ) sampling when performing inference in complex graphical models [ 14 ] , and had the appeal of being able to flexibly represent complex probability distributions , something that parametric models could not achieve at the time.1 Sampling has been invoked to explain aspects of perceptual decision making [ 16 ] , response variability in V1 neurons [ 17 , 18 ] and structured spontaneous activity in the cortex [ 19 ] . Moreover , recent theoretical work has focused on improving the computational efficiency of neural sampling , by increasing sampling speed [ 20 , 21 ] and improving the robustness of the representation [ 22 ] . Nonetheless , sampling-based codes represent uncertainty only implicitly , distributed across time and neurons [ 22 ] . How the brain maps neural sampling dynamics into uncertainty-calibrated decisions remains a key open question . Here , we propose the first spiking circuit for mapping neural samples into appropriate actions . At the core of our idea is the observation that although inference may involve complex high-dimensional posteriors , individual decisions are usually based on low-dimensional marginals of these distributions , which can be represented explicitly using a traditional parametric code . The proposed recurrent network model takes as inputs samples from the task-relevant posterior marginal and integrates them over time in a procedure inspired by online kernel density estimation [ 23 ] . We demonstrate the ability of this circuit to perform decision making in several toy examples . We also analyze the spiking activity of the network to look for signatures of uncertainty and probabilistic computation at the level of single neurons and population activity and find that our model recapitulates a variety of empirical observations in neural data . The model also makes predictions for how uncertainty and prior biases affect neural activity and population latent dynamics in this decision circuit . 1 Spiking neural network for sampling-based marginalization . In a Bayesian framework , perceptual decision making involves several key computations . First , given a sensory stimulus , s , one needs to compute the posterior over the latent variables , x , that may have given rise to it , P ( x|s ) . Second , uncertainty about nuisance variables is averaged out , to obtain a marginal distribution over the task-relevant latent xi , P ( xi|s ) . Finally , this marginal is combined with a task-specific cost function to yield the final decision . The first two steps are straightforward under a sampling framework [ 22 ] , but parametric representations are needed for the final step to convert the information represented by samples into a spatially and temporally localized code [ 9 ] . Our approach uses spike-based distributed sampling for approximate inference and marginalization , then converts the resulting samples into an instantaneous parametric representation that can be used 1Since recent DDCs are also able to represent complex multidimensional posteriors , the arguments in favor of sampling have moved away from representational flexibility , to its ability to explain many nontrivial features of neural variability in the cortex . for decision making ( Fig.1 ) . For the inference portion of the model , we use the distributed sampling scheme developed by Savin and Deneve [ 22 ] , implemented using the Poisson version of the encoding model , proposed by Rullán Buxó and Pillow [ 24 ] . Briefly , the recurrent circuit embeds Langevin sampling dynamics into a population of spiking neurons so that samples can be read out from the instantaneous neural rates ( computed by low-pass filtering the spikes with an exponential kernel ) , r′t via a fixed linear decoder , D′ : xt = D′r′t ( see Suppl . Info . and Refs . [ 22 , 24 ] for details ) 2 . Given these dynamics , samples from the i-th marginal are obtained by projecting activity along the axis defined by row i of the decoding matrix , D′i . The core of our contribution is the second circuit , which receives inputs xi , t = D′ir ′ t from the inference circuit , and constructs a parametric approximation of the associated distribution as a mixture of Gaussian kernels , φk ( · ) , parametrized by kernel weights , c : P ( xi ) = ∑ k ckφk ( xi ) . For simplicity , here we assume a collection of K Gaussian basis functions , although alternative choices of kernels are also possible . The parameters c adapt dynamically with each input sample , following dynamics of the form : ċt = 1 α ( φ ( xit ) − f ( ct−1 ) ) ( 1 ) where vector φ ( xit ) concatenates all basis functions evaluated at the current input ; function f ( · ) ensures that the fixed point solution yields a properly normalized distribution . Throughout this paper , we define f ( ~ct ) = K〈φ〉~ct = β~ct , such that at the fixed point , the norm is 1 , ∑ k φk ( xt ) β =∑ k ck , t = 1 ( see Suppl . Info . for details ) . The α term in Eq . 1 can be thought of as a leak , or the discount rate of internal evidence . This term becomes important for studying the dynamics of neural activity and decision making when the marginal posterior is changing , but otherwise does not qualitatively affect our results . The dynamics implementing the parametric approximation of the marginal are embedded into a spiking neuron population , using the same encoding model used for the first module [ 24 , 25 , 27 ] , so that the parameters can themselves be read out from neural activity via another arbitrary , fixed linear decoder , D , of size K ×N . The communication between circuits involves a linear map along the relevant axis of the inference circuit decoder , followed by point nonlinearities , with a form determined by the kernel φk ( · ) ; this map could be implemented by local dendritic non-linearities . As a simple proof of concept , we first consider inference in a linear Gaussian graphical model ( Fig . 2A ; see Suppl . Info . for details and simulation parameters ) , which results in Gaussian posterior marginals . Before stimulus onset , the inference circuit samples from a broad Gaussian prior centered at zero . Once the stimulus is on ( t = 0.15s ) , the dynamics switch to sampling from a sharper and correlated posterior ( Fig . 2B ) . We take the first dimension x1 as the decision-relevant latent dimension , with corresponding samples forming the stream of inputs to the decision circuit of size N = 160 , where they are converted into a parametric representation of the marginal ( Fig . 2C ) . The spiking activity of the neurons in the second circuit reflect these probabilistic computations , both before ( spontaneous activity ) and after stimulus presentation ( Fig . 2D ) so that parameters ct can be read out from the neural responses ( Fig . 2E ) . They also closely match the ground truth ∼ 150− 200ms after stimulus onset , once the dynamics have stabilized ( Fig . 2F ) . To explore how this representation could be used for optimal decision making , we consider a slightly more involved scenario , in which the task-relevant marginal is a bimodal distribution ( formally , a two-component Gaussian mixture with unequal variance , Fig . 3A ) , a classic example where different choices of cost functions can have very different optimal decisions.3 For instance the task of estimating x1 has the posterior mean as the optimum , while reporting the most likely value would use a maximum a posteriori ( MAP ) estimate . Here , we consider a binary categorization task , which requires reporting the sign of x1 as an abstract analogue to binary discrimination ( for instance , deciding whether or not an oriented grating is tilted leftwards or rightwards relative to the vertical ) . The optimal decision in this case is to report the side of the decision boundary that has more mass ( integrating out the posterior ∫∞ 0 P ( xi|s ) dxi ) . For our concrete example , there is slightly more mass 2The decoder weights are pre-specified by the linear dynamical system , consistent with the encoding framework by [ 25 ] ; however , there exist extensions on how the weights could be learned through local plasticity [ 26 ] . 3Bimodal posteriors are common in probabilistic accounts of bistable percepts [ 28 , 29 ] on the negative side , such that with equal costs the optimal decision would be to report ‘ L ’ .4 We also define an asymmetric cost scenario where the cost of mistakenly reporting a negative value is higher than the alternative so the optimal decision changes to ‘ R ’ . The dynamics of the decision circuit are shown in Fig . 3B . In this example , the inference circuit happens to start its trajectory in the rightward mixture component ; since Langevin dynamics are slow at crossing regions of low probability [ 20 , 21 ] , initially the decision circuit only ‘ sees ’ samples from one component , but eventually both components are explored and the parametric marginal converges to the ground truth . In parallel , a cost-weighted linear projection of the corresponding neural responses tracks the evolution of the estimated cost associated to the two options ( Fig . 3C ) . Indeed , for equal costs option ‘ L ’ starts with a high value , as it looks improbable under the initial approximate posterior , with the estimate converging to the correct low value with more samples ; comparing the two options yields the optimal decision , ‘ L ’ . Similarly , when high costs bias the decision away from this option , the network labels ‘ R ’ as the preferred option . Any other choice of cost function can be accommodated by changing the linear readout from the decision circuit . The same is true for estimation tasks based on the posterior mean , which we simulated to reproduce a reduction in the slope of a psychometric function with increasing uncertainty ( Fig . 3D ) and a shift in due to prior biases ( Fig . 3E ) .

Summary: This work proposes to combine two main (usually competing) theories of how the brain encodes probability distributions - sampling-based and population coding. Both are used in this model of probabilistic computation. The sampling-based representation is used for potentially complex joint distributions. These feed into downstream areas, which represent simpler marginal distributions using parametric encoding. Finally, the article examines application of the parametric code for decision making.  Several experimental facts are reproduced and explained in toy examples.