Impression learning: Online representation learning with synaptic plasticity

1 Introduction . Sensory systems are faced with a task analogous to the scientific process itself : given a steady stream of raw data , they must extract meaningful information about its underlying structure . Because the true underlying structure of the data is rarely accessible , this “ representation learning ” must be largely unsupervised . Framing perception in the language of Bayesian inference has proven fruitful in perceptual and cognitive science [ 1–4 ] , but has been difficult to connect to biology , because we still lack a satisfactory account of how the machinery of Bayesian inference and learning is implemented in neural circuits [ 5 , 6 ] . Past work includes several examples of circuits that simultaneously learn a top-down generative model of incoming stimuli and perform approximate inference with respect to these models . These differ in the nature of the approximation , from maximum a posteriori estimation [ 7 ] , to efficient population codes that embed prior structure [ 8 ] to either parametric [ 9 , 10 ] or sampling-based [ 11 ] variational inference . Learning generally takes the form of optimizing a probabilistic objective , either by backpropagation [ 9 , 10 ] or through local parameter updates , which match biological learning more 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . closely [ 11 , 7 , 12 , 13 ] . While these models are mostly restricted to static stimuli , several instances also operate over time [ 14–16 ] . Developing biologically plausible learning rules that are applicable to temporally-structured data is hampered by the fact that optimizing a probabilistic objective function in such contexts requires access to non-local information across space and time . Previous research on local approximations to credit assignment in BP address spatial credit assignment by ascribing differential functions to the apical and basal dendrites of pyramidal neurons in cortex , where apical dendrites are hypothesized to receive top-down learning signals , and basal dendrites receive bottom-up sensory signals [ 17–25 ] . Locally implementing temporal credit assignment is a bigger challenge [ 26 , 27 ] . Our work , which we have dubbed ‘ impression learning ’ ( IL ) , combines the tradition of probabilistic learning [ 11 , 14 ] with these recent developments in local optimization , in order to learn dynamic stimuli concurrently with perception . We propose a network architecture in which top-down stimulus predictions arriving at the apical dendrites of neurons influence both network dynamics and synaptic plasticity , allowing the network to concurrently learn a probabilistic model of the stimuli and an approximate inference computation . We provide a mathematical derivation of synaptic plasticity rules that approximate gradient descent on a novel unsupervised loss function , along with detailed analyses of the biases induced by this approximation . We explore the empirical and mathematical relationships between IL and three other methods : backpropagation ( BP ) [ 28 ] , the Wake-Sleep ( WS ) algorithm [ 29 ] , and a specific form of neural variational inference ( NVI∗ ) [ 30 , 31 ] . We further demonstrate that IL scales to naturalistic stimuli and multilayer network architectures 1 . 2 Probabilistic inference and local learning in a recurrent circuit . We construct a network of neurons that aims to learn a generative model of the temporal sequence of stimuli that it receives , pm ( r , s ) = ∏T t=0 pm ( rt , st|rt−1 ) , in which s represents stimuli in an input layer.2 The latent variables r are not defined by a physical model of the stimulus environment , but are learned in an unsupervised manner to provide the best generative explanation of stimuli received . We assume that stimuli are generated by a true probability distribution p ( s|z ) , where s corresponds to the first layer of neural activations in an early sensory layer , and vector z ∼ p ( z ) corresponds to the environmental factors which jointly caused that activity . Because learning is unsupervised , we do not enforce explicit correspondence between the internal and true latent features , r and z , only a correspondence between model predictions and ground truth stimuli . We also assume that the network performs online inference with respect to its model , inferring the corresponding latent cause r using Bayes ’ rule : pm ( r|s ) = pm ( r , s ) /pm ( s ) . Because the network won ’ t , in general , be able to explicitly calculate Bayes ’ rule , we will assume that the network learns an approximate inference distribution q ( r|s ) , which it attempts to bring ‘ close ’ to pm ( r|s ) . This joint process of learning and inference , known as Bayesian latent feature extraction , provides a general framework for conceptualizing early sensory processing in the brain [ 5 ] . In subsequent sections , we will write a loss function for this general latent feature extraction objective , and show how local modifications at apical and basal synapses can perform approximate gradient descent on this loss . Loss function The loss function that we propose will produce a learning algorithm where neurons alternate between sampling from the model , pm , and performing approximate inference according to q in response to real stimuli received from p ( s|z ) . This alternation will allow the network to learn online in a way that minimally perturbs the continuity of perception . First , consider two families of hybrid probability distributions , which we denote in shorthand q̃θ and p̃θ : q̃θ = T∏ t=0 q̃t ( rt , st|zt , λt ; θ ) = T∏ t=0 ( q ( rt|st ; θq ) p ( st|zt ) ) λt pm ( rt , st|rt−1 , λt ; θp ) 1−λt p̃θ = T∏ t=0 p̃t ( rt , st|zt , λt ; θ ) = T∏ t=0 ( q ( rt|st ; θq ) p ( st|zt ) ) 1−λt pm ( rt , st|rt−1 , λt ; θp ) λt , ( 1 ) where a collection of binary random variables λt determines whether , at a given time step , sampling occurs due to q ( rt|st ; θq ) p ( st|zt ) or pm ( rt , st|rt−1 , λt ; θp ) , and the full parameter space is denoted 1Code provided at : https : //github.com/colinbredenberg/Impression-Learning-Camera-Ready . 2We use the shorthand notation ‘ s ’ to refer to the N × T matrix of stimuli across time . θ = [ θp , θq ] . We define an objective of the form : L = Eλ , z [ KL [ q̃θ||p̃θ ] ] = Eλ , z [ ∫ [ log q̃θ − log p̃θ ] q̃θ drds ] . ( 2 ) This loss provides a generalization of the widely-used evidence lower bound ( ELBO ) , which corresponds to the case λt = 1 ∀t . Importantly , we can show that L = 0 if and only if q ( rt|st ; θq ) p ( st|zt ) = pm ( rt , st|rt−1 , λt ; θp ) ∀t . If this equality were achieved , it would also imply pm ( r , s ) = q ( r|s ) p ( s|z ) . However , this absolute minimum will not be achievable unless zt is deterministic , because pm ( rt , st|rt−1 , λt ; θp ) has no dependency on the latent variables in the environment . Thus , our goal is inherently unachievable , and different choices of p ( λt ) and network architectures may lead to different local minima . However , each choice will incentivize learning a close correspondence between these distributions , and an approximation to gradient descent with respect to any choice will lead to local synaptic plasticity rules , making this objective particularly interesting for the computational neuroscience community . Update derivation We begin by taking the gradient of our new loss w.r.t . θ = [ θq , θp ] : −∇θL =−∇θEλ , z [ ∫ [ log q̃θ − log p̃θ ] q̃θ drds ] =− Eλ , z [ ∫ [ ∇θ ( log q̃θ − log p̃θ ) ] q̃θ drds + ∫ [ log q̃θ − log p̃θ ] ∇θ q̃θ drds ] , where the second equality follows from the product rule . Both integrals are analytically intractable , but if we can write both as expectations , they can be approximated by averaging over samples of r and s. To accomplish this , we note that ∇θ q̃θ = ∇θelog q̃θ = [ ∇θ log q̃θ ] q̃θ , which allows us to rewrite our expression as an expectation over r and s : −∇θL =− Eλ , z [ ∫ [ ∇θ log q̃θ −∇θ log p̃θ ] q̃θ drds + ∫ [ log q̃θ − log p̃θ ] ( ∇θ log q̃θ ) q̃θ drds ] . We also observe that ∫ [ ∇θ log q̃θ ] q̃θ drds = ∇θ ∫ q̃θ drds = ∇θ1 = 0 , allowing the elimination of two terms : −∇θL = Eλ , z [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ log p̃θ q̃θ ] ( ∇θ log q̃θ ) q̃θ drds ] ≈ Eλ , z [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ p̃θ q̃θ − 1 ] ( ∇θ log q̃θ ) q̃θ drds ] = Eλ , z [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ p̃θ q̃θ ] ( ∇θ log q̃θ ) q̃θ drds ] = Eλ , z [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ ∇θ log q̃θ ] p̃θ drds ] . ( 3 ) The approximation in the second line comes from a Taylor expansion of log p̃θq̃θ about 0 , i.e . when p̃θ q̃θ = 1 ( which introduces a bias to the parameter updates that we examine analytically in Appendix A ) . This expansion is the core of our derivation , and not all algorithms take this approach : for this reason , in Appendix B and C we show how the properties of our algorithm compare to alternatives ( NVI∗ , BP , or WS ) . At this point , we have not yet defined p ( λ ) . We ’ ll assume that λ0 ∈ { 0 , 1 } , that p ( λ0 = 0 ) = p ( λ0 = 1 ) = 0.5 , and that the λ values alternate deterministically with a ‘ phase duration ’ K , i.e . λk+1 = 1 − λk if mod ( k , K ) = 0 , and λk+1 = λk otherwise . Under these conditions , the two integrals in Eq . ( 3 ) are equivalent , and computing our parameter updates only requires sampling from q̃ . If we define λ′ = 1− λ , then we have p ( λ′ ) = p ( λ ) and q̃ ( r , s|z , λ ; θ ) = p̃ ( r , s|z , λ′ ; θ ) , which we make use of as follows : −∇θL ≈Ez [ ∑ λ [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ ∇θ log q̃θ ] p̃θ drds ] p ( λ ) ] =Ez [ ∑ λ ∫ [ ∇θ log p̃θ ] q̃θ drdsp ( λ ) + ∑ λ′ ∫ [ ∇θ log p̃θ ] q̃θ drds p ( λ′ ) ] =2Ez [ ∑ λ ∫ [ ∇θ log p̃θ ] q̃θ drds p ( λ ) ] . ( 4 ) Using the definitions for q̃θ and p̃θ and the properties of the logarithm gives us the following parameter update rule : ∆θ ∝ 2Eλ0 , z [ ∫ [ ∑ t ( 1− λt ) ∇θ log qt + ( λt ) ∇θ log pmt ] q̃θ drds ] . ( 5 ) As we will show below , this parameter update equation produces updates that require only information locally available to synapses , a necessary condition for any biologically-plausible algorithm . Basic model To make the above general learning procedure concrete , we need to specify how to sample from q̃θ , which in turn requires an architecture for performing approximate inference at each time step , q ( rt|st ; θq ) , and a joint model of stimuli and neural activations , pm ( rt , st|rt−1 , λt ; θp ) . We map these two model components onto neural circuitry , with their own local variables corresponding to s and r , and segregated synaptic parameters : the ‘ basal ’ compartment is dedicated to feedforward inference ( q , index ‘ inf ’ ) and the ‘ apical ’ compartment is dedicated to generative sampling from the model ( pm , index ‘ gen ’ ) ; this segregation allows their influence on neural dynamics to be selectively gated by λt ( Fig . 1a ) . First , the internal generative model of the circuit is implicitly defined by a set of currents to the apical dendritic compartment corresponding to generated samples for the next latent variable , rgent : rgent = ( ( 1− kt ) Dr + ktI ) rt−1 + σgenr ηt ( 6 ) sgent = f ( Dsrt ) + σ gen s ξt , ( 7 ) where Dr is a diagonal transition matrix ( constraining generated latent-variables to be independent AR ( 1 ) processes ) , Ds is a linear decoder , I is the identity function , ηt , ξt ∼ N ( 0 , I ) are independent white noise samples , and σgenr and σ gen s denote respectively the generative standard deviation for neurons at the stimulus and latent levels . We define kt = ( 1 − δ ( λt − λt−1 ) ) λt , with δ ( · ) the Dirac delta function ; kt is 1 only if λt = 1 and λt−1 = 0 . We chose a piecewise model ( gated by kt ) for r gen t because we observed that the statistics of stimuli st given previous activities rt−1 are different if a transition has just occurred ( λt = 1 and λt−1 = 0 ) , which will bias the training of the generative transition parameters Dr. We chose I for this case , but one could alternatively have a different parametric model for after transitions have occurred . As we will show , adding this condition to our model will never affect the dynamics of our network , but will cause learning for Dr to occur only on time steps when a transition has not just occurred . Nothing in our derivation requires the transition matrix Dr to be diagonal , but we constrained it in this way to allow for learning independent latent features . As is , Dr defines the leakiness of the apical dendritic compartment of the neuron ; off-diagonal components of the transition matrix would correspond to recurrent synapses . These dynamics define a probability distribution : pm ( r , s ) = ∏T t=0 pm ( rt , st|rt−1 , λt ; θp ) . Second , we define our inference model , a factorized conditional probability distribution q ( r|s ) =∏T t=0 q ( rt|st ; θq ) , which applies a feedforward nonlinear transformation to incoming stimuli : rinft = f ( Wst ) + σ inf r ηt , ( 8 ) where W denotes the feedforward weights , σinfr is the inference standard deviation for neurons at the latent level , and the nonlinearity f ( · ) is the tanh function . During inference mode , the stimulus layer receives latent-associated inputs from the environment , further corrupted by the same noise as the internal representation : sinft = s̄ ( zt ) + σ inf s ξt , ( 9 ) where σinfs denotes the standard deviation for neurons at the stimulus level , and s̄ ( zt ) is input from external stimuli . During simulations , samples are determined by a combination of pm and q , given by q̃θ : rt = λtr inf t + ( 1− λt ) r gen t ( 10 ) st = λts inf t + ( 1− λt ) s gen t . ( 11 ) We interpret these dynamics biologically as network of recurrently connected pyramidal neurons with two sources of input , one to the apical dendrites ( rgent or s gen t ) and one to the basal dendrites ( r inf t or sinft ) . The gating variable λt determines which input source controls the circuit dynamics . Plasticity rule interpretation Inserting our particular choice of qt and pmt into our approximate gradient descent derivation , the parameter updates can be interpreted as local synaptic plasticity rules at the basal ( for qt ) or apical ( for pmt ) compartments of our neuron model : log q ( rt|st ; θq ) =− 1 2 ( σinfr ) 2 ‖rt − f ( Wst ) ‖ 2 2 + cq ( 12 ) log pm ( rt , st|rt−1 , λt ; θp ) =− 1 2 ( σgenr ) 2 ‖rt − ( ( 1− kt ) Dr + ktI ) rt−1‖22 − 1 2 ( σgens ) 2 ‖st − f ( Dsrt ) ‖22 + cp , ( 13 ) where cq = −Nr log ( √ 2π ( σinfr ) 2 ) and cp = −Nr log ( √ 2π ( σgenr ) 2 − Ns log ( √ 2π ( σgens ) 2 are constants that do not depend on network parameters . We can use these equations to evaluate our weight updates , by using the general formula in Eq . 5 and calculating derivatives . For online parameter updates , we assume that weights change stochastically at each time step , based on samples from λ0 , z , r , and s ( instead of explicitly calculating the expectation in Eq . 5 ) : ∆W ( ij ) ∝ 1− λt ( σinfr ) 2 ( r ( i ) t − f ( Wst ) ( i ) ) f ′ ( Wst ) ( i ) s ( j ) t ( 14 ) ∆D ( ii ) r ∝ λt ( 1− kt ) ( σgenr ) 2 ( r ( i ) t − ( Drrt−1 ) ( i ) ) r ( i ) t−1 ( 15 ) ∆D ( ij ) s ∝ λt ( σgens ) 2 ( s ( i ) t − f ( Dsrt ) ( i ) ) f ′ ( Dsrt ) ( i ) r ( j ) t . ( 16 ) Each of these updates has the form of a local synaptic plasticity rule , under the following assumptions : W ( ij ) is a basal synapse from neuron j to neuron i , r ( i ) t and r ( j ) t correspond to the pre- and postsynaptic firing rates , respectively , and f ( Wst ) ( i ) corresponds to the local basal current injected into neuron i . Thus , assuming that a basal synapse has access to both the neuron ’ s firing rate and its local basal synaptic current at a particular moment in time , ∆W ( ij ) is local ; the same principle holds for the apical updates . If λt = 0 , then network activity is driven by the generative inputs , and so the parameter updates for basal synapses depend on apically-driven activity , as has been observed experimentally [ 32 ] ; similarly , apical synaptic plasticity should depend on basally-driven activity . The updates for the generative transition matrix , Dr–determining the leakiness of the apical dendritic compartments–are gated by 1 − kt , indicating that parameter updates are delayed upon entering ‘ inference ’ mode : this could reasonably be implemented biologically by a slow cascade of biochemical processes that delay changes in neural parameters , as has been proposed by previous plasticity models [ 33 , 34 ] .

Summary: The authors are interested in understanding how brain circuits are able to perform unsupervised learning. The work is an extension of the wake-sleep algorithm, where the algorithm is asked to perform two functions: i) infer latent variables from observations and ii) build a generative model of the received observations. They start by defining a loss function to be minimized for their system to perform both functions and then interpret this minimization as a plasticity rule between the weights of a network of neurons. Their learning algorithm is compared to other existing one in the literature, and its efficiency is probed on a real-world task.