Bias and variance of the Bayesian-mean decoder

1 Introduction . Perception has been described in neuroscience as resulting from a two-stage , ‘ encoding-decoding ’ process . In the encoding stage , an external stimulus elicits in the brain an internal representation ( in the form of a neural activity ) whose statistics depend on the physical properties of the stimulus . In the decoding stage , the brain makes use of the internal representation to achieve a goal : for instance , to estimate the magnitude of the stimulus , or to choose the best of two stimuli . It is generally understood that the encoding capacity of the brain is limited , i.e. , that there is some amount of imprecision in the internal representation . The theory of ‘ efficient coding ’ , however , proposes that the encoding is optimized , under a constraint on the encoding capacity of the brain ; and the theory of the ‘ Bayesian brain ’ puts forward the notion that the brain optimally decodes the imprecise representation , through Bayesian inference . Although these two theories emerged separately , recent models of perception combine the two into a single , normative account of perception [ 2–9 ] . There does not seem to be a general consensus , however , as to what objectives the encoding and decoding stages optimize [ 10 ] ; in some models ( though not all [ 8 ] ) , the two stages optimize different objectives [ 6 , 9 ] . Furthermore , in behavioral experiments involving one-dimensional , physical magnitudes ( such as some orientation or length ) , human perceptual decisions are often biased , and they exhibit variability ( ‘ noise ’ ) . Recent encoding-decoding models of perception account for these 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . patterns , and make specific predictions regarding how , in perceptual tasks , the bias and the variance of the estimates should relate to the distribution of stimuli ( the prior ) . They predict , in addition , that the bias should be proportional to the derivative of the variance . Although there is substantial support for this relation in experimental data [ 1 ] , the value of the constant of proportionality — including its sign — remains unclear ; it is thus uncertain , in a given task , whether the perceptual bias should be in one direction , or the other . Experimentally , biases towards the stimuli that are more probable as per the prior [ 11–14 ] , and the opposite , i.e. , biases towards the less probable stimuli ( ‘ anti-Bayesian percepts ’ ) [ 6 , 15–17 ] , have both been exhibited . Brayanov and Smith [ 17 ] simultaneously find the former in a task involving ( implicit ) weight estimation , and the latter in a task involving weight discrimination . The bias , in encoding-decoding models of perception , is in great part determined by the ‘ decoder ’ , i.e. , the decoding mechanism . A widely used decoder is the Bayesian mean . The Bayesian mean is the optimal decoder both in estimation tasks ( with squared-error loss ) , and in a common type of discrimination tasks ( which we describe in more detail below ) . It is , in the general case , biased ; its bias depends on the prior , but also on the characteristics of the encoding . Here , we present an analytical approximation to the bias of the Bayesian mean , obtained under very general assumptions about the encoding . In short , the main assumption is that the encoding takes the form of a series of independent signals . These signals need not be identically distributed . This corresponds , for instance , to a population of sensory neurons that are each characterized by a different tuning curve . We do not endeavor to model the individual properties of each neuron in a population : instead , we rely on the statistical properties resulting from the accumulation of independent signals , as summarized by their total Fisher information . This approach also provides an approximation to the variance of the Bayesian mean . A mathematical result similar to that presented here has been derived in the statistics literature [ 18 ] , but to our knowledge its importance for perception models has not been noted . Two terms drive the bias of the Bayesian mean : the variations of the prior across stimulus space , and the variations of the Fisher information . When the encoding is ‘ efficient ’ , these two terms have opposite effects . The resulting size — and sign — of the bias will depend on the specifics of the encoding , and on the objective it optimizes . We show , below , how the encoding objective function determines the relation between the bias and the prior , the direction of the bias , and the relation of proportionality between the bias and the derivative of the variance . Our results explain why opposite biases are obtained in different experiments : when the encoding is optimized for an estimation task ( with squared-error loss ) , the bias is directed towards the more probable stimuli ; while when the encoding is optimized for discrimination tasks , the opposite is predicted . Finally , we run simulations , and examine how the accuracy of our asymptotic approximations vary with the amount of imprecision in the encoding . 2 Bias and variance of the Bayesian mean . We consider an observer presented with a stimulus characterized by a magnitude , x ( for instance , its length , its orientation , or its speed ) , which is randomly sampled from a prior distribution , π ( x ) . We assume that the presentation of the stimulus elicits in the brain of the observer an internal representation , r , as a series of n random signals , r1 , . . . , rn , sampled independently from n distributions conditioned on the stimulus , fi ( ri|x ) . We assume in addition that moments of a sufficient order exist for all i , and that the rate of growth of these moments is limited in a way that allows us to use a central limit theorem ( see Supplementary Materials ) . This is the encoding stage . We emphasize that we assume the signals to be independent , but not necessarily that they are identically distributed . They can be understood as representing the activity of a neural population in which each neuron has a different tuning curve , for instance centered on a ‘ preferred ’ stimulus . ( The result we present is however a mathematical one , and is thus not tied to a particular physical implementation . ) The internal representation , r , is characterized by its Fisher information , I ( x ) , which is the sum of the Fisher informations of each signal ; it is thus of order n. Our calculations rely on the asymptotic statistical properties resulting from the accumulation of these signals , when n is large . The Fisher information plays here a central role , by providing a summary measure of the sensitivity of the representation to the stimulus , effectively abstracting away the specifics of each random signal ( such as , for instance , the individual properties of each neuron in a population ) . Turning to the decoding stage , we assume that our observer uses the Bayesian mean . We sketch , here , the derivation of the approximations to the bias and to the variance of this decoder ( a more detailed derivation is presented in the Supplementary Materials ) . Given an internal representation , r , the Bayesian mean is x̂ ( r ) ≡ E [ x|r ] = ∫ x̃π ( x̃ ) ∏n i=1 fi ( ri|x̃ ) dx̃∫ π ( x̃ ) ∏n i=1 fi ( ri|x̃ ) dx̃ . ( 1 ) Fixing r , we denote by L ( x ) the log-likelihood of a stimulus x , i.e. , L ( x ) = ∑n i=1 ln fi ( ri|x ) . The log-likelihood is thus of order n. We rewrite Eq . ( 1 ) using the log-likelihood , as x̂ ( r ) = ∫ x̃π ( x̃ ) eL ( x̃ ) dx̃∫ π ( x̃ ) eL ( x̃ ) dx̃ . ( 2 ) This equation is the ratio of two integrals of the type ∫ g ( x̃ ) eL ( x̃ ) dx̃ . With large n , the integrands are dominated by their values at the point where L ( x ) reaches its maximum , i.e. , at the maximumlikelihood estimator ( MLE ) . Through Laplace approximations of these integrals , we obtain an approximate expression of the Bayesian mean , equal to the MLE corrected by a term of order 1/n that depends on the prior , on the likelihood , and on their derivatives . The MLE has been widely studied in statistics : approximations to its bias and to its variance , as functions of the Fisher information , I ( x ) , are well known [ 19–21 ] . This allows us to obtain an approximation of order 1/n to the bias of the Bayesian mean , as E [ x̂− x ∣∣x ] ' b ( x ) = 1 I ( x ) [ π′ ( x ) π ( x ) − I ′ ( x ) I ( x ) ] , ( 3 ) where x is the true value of the stimulus , and π′ and I ′ are the derivatives of the prior and of the Fisher information . We emphasize that this is a very general result about the Bayesian mean , obtained under one main assumption , that a series of independent signals , not necessarily identically distributed , encodes the stimulus . Equation ( 3 ) shows , first , that the bias of the Bayesian mean is scaled by the inverse of the Fisher information , 1/I ( x ) . In other words , the more precise the internal representation about the stimulus , the smaller the bias . Second , the bias depends on the difference between the relative variation of the prior ( around x ) , π′ ( x ) /π ( x ) , and the relative variation of the Fisher information , I ′ ( x ) /I ( x ) . The former induces a ‘ prior effect ’ , by which the Bayesian mean tends to be biased towards more probable stimuli . The latter induces an ‘ encoding effect ’ , by which the bias tends to be in the direction of decreasing precision of the encoding . The resulting direction of the bias is the sum of these two effects . If more probable stimuli are encoded with a higher precision ( i.e. , if π′ and I ′ have the same sign ) , which is what efficient-coding models suggest ( see below ) , then these two effects are opposite . If in addition the encoding effect is stronger than the prior effect , then the Bayesian mean is biased towards stimuli of decreasing probability , a bias direction that has been qualified as ‘ anti-Bayesian ’ [ 6 , 17 ] , although here it is obtained with a Bayesian-mean decoder . Finally , it is well known that the variance of the MLE converges to the inverse of the Fisher information , 1/I ( x ) . It is thus the leading term of the variance of the Bayesian mean , and we find that the additional terms , in the approximation to this variance , are of orders higher than 1/n . Hence the inverse of the Fisher information is our approximation of order 1/n to the variance of the Bayesian mean , i.e. , Var [ x̂ ∣∣x ] ' v ( x ) = 1 I ( x ) . ( 4 ) The Bayesian-mean decoder is thus less variable for stimuli that are encoded with higher precision . Equations ( 3 ) and ( 4 ) underline how the bias and the variance of the Bayesian mean depend on the precision of the encoding , as measured by the Fisher information . Different choices of encoding thus result in different behaviors of the bias and of the variance . We now turn to the question of how the choice of encoding can be efficiently adapted to the specific task faced by the observer .

Summary: The paper investigates the bias (an the variance) of the Bayesian-mean decoder. The authors found that the bias is driven by two components: The variations of the priors and variations of the Fisher information, and the exact relation is driven by the encoding objective. Simulations are used to back the results.  Disclaimer: I am an experimental visual psychophysicist with experience in Deep Learning. Thus the paper falls a bit outside my area of expertise (as indicated to the AC). I am focusing on the general aspects of this paper.