Impression learning: Online representation learning with synaptic plasticity

1 Introduction . Sensory systems are faced with a task analogous to the scientific process itself : given a steady stream of raw data , they must extract meaningful information about its underlying structure . Because the true underlying structure of the data is rarely accessible , this “ representation learning ” must be largely unsupervised . Framing perception in the language of Bayesian inference has proven fruitful in perceptual and cognitive science [ 1–4 ] , but has been difficult to connect to biology , because we still lack a satisfactory account of how the machinery of Bayesian inference and learning is implemented in neural circuits [ 5 , 6 ] . Past work includes several examples of circuits that simultaneously learn a top-down generative model of incoming stimuli and perform approximate inference with respect to these models . These differ in the nature of the approximation , from maximum a posteriori estimation [ 7 ] , to efficient population codes that embed prior structure [ 8 ] to either parametric [ 9 , 10 ] or sampling-based [ 11 ] variational inference . Learning generally takes the form of optimizing a probabilistic objective , either by backpropagation [ 9 , 10 ] or through local parameter updates , which match biological learning more 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . closely [ 11 , 7 , 12 , 13 ] . While these models are mostly restricted to static stimuli , several instances also operate over time [ 14–16 ] . Developing biologically plausible learning rules that are applicable to temporally-structured data is hampered by the fact that optimizing a probabilistic objective function in such contexts requires access to non-local information across space and time . Previous research on local approximations to credit assignment in BP address spatial credit assignment by ascribing differential functions to the apical and basal dendrites of pyramidal neurons in cortex , where apical dendrites are hypothesized to receive top-down learning signals , and basal dendrites receive bottom-up sensory signals [ 17–25 ] . Locally implementing temporal credit assignment is a bigger challenge [ 26 , 27 ] . Our work , which we have dubbed ‘ impression learning ’ ( IL ) , combines the tradition of probabilistic learning [ 11 , 14 ] with these recent developments in local optimization , in order to learn dynamic stimuli concurrently with perception . We propose a network architecture in which top-down stimulus predictions arriving at the apical dendrites of neurons influence both network dynamics and synaptic plasticity , allowing the network to concurrently learn a probabilistic model of the stimuli and an approximate inference computation . We provide a mathematical derivation of synaptic plasticity rules that approximate gradient descent on a novel unsupervised loss function , along with detailed analyses of the biases induced by this approximation . We explore the empirical and mathematical relationships between IL and three other methods : backpropagation ( BP ) [ 28 ] , the Wake-Sleep ( WS ) algorithm [ 29 ] , and a specific form of neural variational inference ( NVI∗ ) [ 30 , 31 ] . We further demonstrate that IL scales to naturalistic stimuli and multilayer network architectures 1 . 2 Probabilistic inference and local learning in a recurrent circuit . We construct a network of neurons that aims to learn a generative model of the temporal sequence of stimuli that it receives , pm ( r , s ) = ∏T t=0 pm ( rt , st|rt−1 ) , in which s represents stimuli in an input layer.2 The latent variables r are not defined by a physical model of the stimulus environment , but are learned in an unsupervised manner to provide the best generative explanation of stimuli received . We assume that stimuli are generated by a true probability distribution p ( s|z ) , where s corresponds to the first layer of neural activations in an early sensory layer , and vector z ∼ p ( z ) corresponds to the environmental factors which jointly caused that activity . Because learning is unsupervised , we do not enforce explicit correspondence between the internal and true latent features , r and z , only a correspondence between model predictions and ground truth stimuli . We also assume that the network performs online inference with respect to its model , inferring the corresponding latent cause r using Bayes ’ rule : pm ( r|s ) = pm ( r , s ) /pm ( s ) . Because the network won ’ t , in general , be able to explicitly calculate Bayes ’ rule , we will assume that the network learns an approximate inference distribution q ( r|s ) , which it attempts to bring ‘ close ’ to pm ( r|s ) . This joint process of learning and inference , known as Bayesian latent feature extraction , provides a general framework for conceptualizing early sensory processing in the brain [ 5 ] . In subsequent sections , we will write a loss function for this general latent feature extraction objective , and show how local modifications at apical and basal synapses can perform approximate gradient descent on this loss . Loss function The loss function that we propose will produce a learning algorithm where neurons alternate between sampling from the model , pm , and performing approximate inference according to q in response to real stimuli received from p ( s|z ) . This alternation will allow the network to learn online in a way that minimally perturbs the continuity of perception . First , consider two families of hybrid probability distributions , which we denote in shorthand q̃θ and p̃θ : q̃θ = T∏ t=0 q̃t ( rt , st|zt , λt ; θ ) = T∏ t=0 ( q ( rt|st ; θq ) p ( st|zt ) ) λt pm ( rt , st|rt−1 , λt ; θp ) 1−λt p̃θ = T∏ t=0 p̃t ( rt , st|zt , λt ; θ ) = T∏ t=0 ( q ( rt|st ; θq ) p ( st|zt ) ) 1−λt pm ( rt , st|rt−1 , λt ; θp ) λt , ( 1 ) where a collection of binary random variables λt determines whether , at a given time step , sampling occurs due to q ( rt|st ; θq ) p ( st|zt ) or pm ( rt , st|rt−1 , λt ; θp ) , and the full parameter space is denoted 1Code provided at : https : //github.com/colinbredenberg/Impression-Learning-Camera-Ready . 2We use the shorthand notation ‘ s ’ to refer to the N × T matrix of stimuli across time . θ = [ θp , θq ] . We define an objective of the form : L = Eλ , z [ KL [ q̃θ||p̃θ ] ] = Eλ , z [ ∫ [ log q̃θ − log p̃θ ] q̃θ drds ] . ( 2 ) This loss provides a generalization of the widely-used evidence lower bound ( ELBO ) , which corresponds to the case λt = 1 ∀t . Importantly , we can show that L = 0 if and only if q ( rt|st ; θq ) p ( st|zt ) = pm ( rt , st|rt−1 , λt ; θp ) ∀t . If this equality were achieved , it would also imply pm ( r , s ) = q ( r|s ) p ( s|z ) . However , this absolute minimum will not be achievable unless zt is deterministic , because pm ( rt , st|rt−1 , λt ; θp ) has no dependency on the latent variables in the environment . Thus , our goal is inherently unachievable , and different choices of p ( λt ) and network architectures may lead to different local minima . However , each choice will incentivize learning a close correspondence between these distributions , and an approximation to gradient descent with respect to any choice will lead to local synaptic plasticity rules , making this objective particularly interesting for the computational neuroscience community . Update derivation We begin by taking the gradient of our new loss w.r.t . θ = [ θq , θp ] : −∇θL =−∇θEλ , z [ ∫ [ log q̃θ − log p̃θ ] q̃θ drds ] =− Eλ , z [ ∫ [ ∇θ ( log q̃θ − log p̃θ ) ] q̃θ drds + ∫ [ log q̃θ − log p̃θ ] ∇θ q̃θ drds ] , where the second equality follows from the product rule . Both integrals are analytically intractable , but if we can write both as expectations , they can be approximated by averaging over samples of r and s. To accomplish this , we note that ∇θ q̃θ = ∇θelog q̃θ = [ ∇θ log q̃θ ] q̃θ , which allows us to rewrite our expression as an expectation over r and s : −∇θL =− Eλ , z [ ∫ [ ∇θ log q̃θ −∇θ log p̃θ ] q̃θ drds + ∫ [ log q̃θ − log p̃θ ] ( ∇θ log q̃θ ) q̃θ drds ] . We also observe that ∫ [ ∇θ log q̃θ ] q̃θ drds = ∇θ ∫ q̃θ drds = ∇θ1 = 0 , allowing the elimination of two terms : −∇θL = Eλ , z [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ log p̃θ q̃θ ] ( ∇θ log q̃θ ) q̃θ drds ] ≈ Eλ , z [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ p̃θ q̃θ − 1 ] ( ∇θ log q̃θ ) q̃θ drds ] = Eλ , z [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ p̃θ q̃θ ] ( ∇θ log q̃θ ) q̃θ drds ] = Eλ , z [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ ∇θ log q̃θ ] p̃θ drds ] . ( 3 ) The approximation in the second line comes from a Taylor expansion of log p̃θq̃θ about 0 , i.e . when p̃θ q̃θ = 1 ( which introduces a bias to the parameter updates that we examine analytically in Appendix A ) . This expansion is the core of our derivation , and not all algorithms take this approach : for this reason , in Appendix B and C we show how the properties of our algorithm compare to alternatives ( NVI∗ , BP , or WS ) . At this point , we have not yet defined p ( λ ) . We ’ ll assume that λ0 ∈ { 0 , 1 } , that p ( λ0 = 0 ) = p ( λ0 = 1 ) = 0.5 , and that the λ values alternate deterministically with a ‘ phase duration ’ K , i.e . λk+1 = 1 − λk if mod ( k , K ) = 0 , and λk+1 = λk otherwise . Under these conditions , the two integrals in Eq . ( 3 ) are equivalent , and computing our parameter updates only requires sampling from q̃ . If we define λ′ = 1− λ , then we have p ( λ′ ) = p ( λ ) and q̃ ( r , s|z , λ ; θ ) = p̃ ( r , s|z , λ′ ; θ ) , which we make use of as follows : −∇θL ≈Ez [ ∑ λ [ ∫ [ ∇θ log p̃θ ] q̃θ drds + ∫ [ ∇θ log q̃θ ] p̃θ drds ] p ( λ ) ] =Ez [ ∑ λ ∫ [ ∇θ log p̃θ ] q̃θ drdsp ( λ ) + ∑ λ′ ∫ [ ∇θ log p̃θ ] q̃θ drds p ( λ′ ) ] =2Ez [ ∑ λ ∫ [ ∇θ log p̃θ ] q̃θ drds p ( λ ) ] . ( 4 ) Using the definitions for q̃θ and p̃θ and the properties of the logarithm gives us the following parameter update rule : ∆θ ∝ 2Eλ0 , z [ ∫ [ ∑ t ( 1− λt ) ∇θ log qt + ( λt ) ∇θ log pmt ] q̃θ drds ] . ( 5 ) As we will show below , this parameter update equation produces updates that require only information locally available to synapses , a necessary condition for any biologically-plausible algorithm . Basic model To make the above general learning procedure concrete , we need to specify how to sample from q̃θ , which in turn requires an architecture for performing approximate inference at each time step , q ( rt|st ; θq ) , and a joint model of stimuli and neural activations , pm ( rt , st|rt−1 , λt ; θp ) . We map these two model components onto neural circuitry , with their own local variables corresponding to s and r , and segregated synaptic parameters : the ‘ basal ’ compartment is dedicated to feedforward inference ( q , index ‘ inf ’ ) and the ‘ apical ’ compartment is dedicated to generative sampling from the model ( pm , index ‘ gen ’ ) ; this segregation allows their influence on neural dynamics to be selectively gated by λt ( Fig . 1a ) . First , the internal generative model of the circuit is implicitly defined by a set of currents to the apical dendritic compartment corresponding to generated samples for the next latent variable , rgent : rgent = ( ( 1− kt ) Dr + ktI ) rt−1 + σgenr ηt ( 6 ) sgent = f ( Dsrt ) + σ gen s ξt , ( 7 ) where Dr is a diagonal transition matrix ( constraining generated latent-variables to be independent AR ( 1 ) processes ) , Ds is a linear decoder , I is the identity function , ηt , ξt ∼ N ( 0 , I ) are independent white noise samples , and σgenr and σ gen s denote respectively the generative standard deviation for neurons at the stimulus and latent levels . We define kt = ( 1 − δ ( λt − λt−1 ) ) λt , with δ ( · ) the Dirac delta function ; kt is 1 only if λt = 1 and λt−1 = 0 . We chose a piecewise model ( gated by kt ) for r gen t because we observed that the statistics of stimuli st given previous activities rt−1 are different if a transition has just occurred ( λt = 1 and λt−1 = 0 ) , which will bias the training of the generative transition parameters Dr. We chose I for this case , but one could alternatively have a different parametric model for after transitions have occurred . As we will show , adding this condition to our model will never affect the dynamics of our network , but will cause learning for Dr to occur only on time steps when a transition has not just occurred . Nothing in our derivation requires the transition matrix Dr to be diagonal , but we constrained it in this way to allow for learning independent latent features . As is , Dr defines the leakiness of the apical dendritic compartment of the neuron ; off-diagonal components of the transition matrix would correspond to recurrent synapses . These dynamics define a probability distribution : pm ( r , s ) = ∏T t=0 pm ( rt , st|rt−1 , λt ; θp ) . Second , we define our inference model , a factorized conditional probability distribution q ( r|s ) =∏T t=0 q ( rt|st ; θq ) , which applies a feedforward nonlinear transformation to incoming stimuli : rinft = f ( Wst ) + σ inf r ηt , ( 8 ) where W denotes the feedforward weights , σinfr is the inference standard deviation for neurons at the latent level , and the nonlinearity f ( · ) is the tanh function . During inference mode , the stimulus layer receives latent-associated inputs from the environment , further corrupted by the same noise as the internal representation : sinft = s̄ ( zt ) + σ inf s ξt , ( 9 ) where σinfs denotes the standard deviation for neurons at the stimulus level , and s̄ ( zt ) is input from external stimuli . During simulations , samples are determined by a combination of pm and q , given by q̃θ : rt = λtr inf t + ( 1− λt ) r gen t ( 10 ) st = λts inf t + ( 1− λt ) s gen t . ( 11 ) We interpret these dynamics biologically as network of recurrently connected pyramidal neurons with two sources of input , one to the apical dendrites ( rgent or s gen t ) and one to the basal dendrites ( r inf t or sinft ) . The gating variable λt determines which input source controls the circuit dynamics . Plasticity rule interpretation Inserting our particular choice of qt and pmt into our approximate gradient descent derivation , the parameter updates can be interpreted as local synaptic plasticity rules at the basal ( for qt ) or apical ( for pmt ) compartments of our neuron model : log q ( rt|st ; θq ) =− 1 2 ( σinfr ) 2 ‖rt − f ( Wst ) ‖ 2 2 + cq ( 12 ) log pm ( rt , st|rt−1 , λt ; θp ) =− 1 2 ( σgenr ) 2 ‖rt − ( ( 1− kt ) Dr + ktI ) rt−1‖22 − 1 2 ( σgens ) 2 ‖st − f ( Dsrt ) ‖22 + cp , ( 13 ) where cq = −Nr log ( √ 2π ( σinfr ) 2 ) and cp = −Nr log ( √ 2π ( σgenr ) 2 − Ns log ( √ 2π ( σgens ) 2 are constants that do not depend on network parameters . We can use these equations to evaluate our weight updates , by using the general formula in Eq . 5 and calculating derivatives . For online parameter updates , we assume that weights change stochastically at each time step , based on samples from λ0 , z , r , and s ( instead of explicitly calculating the expectation in Eq . 5 ) : ∆W ( ij ) ∝ 1− λt ( σinfr ) 2 ( r ( i ) t − f ( Wst ) ( i ) ) f ′ ( Wst ) ( i ) s ( j ) t ( 14 ) ∆D ( ii ) r ∝ λt ( 1− kt ) ( σgenr ) 2 ( r ( i ) t − ( Drrt−1 ) ( i ) ) r ( i ) t−1 ( 15 ) ∆D ( ij ) s ∝ λt ( σgens ) 2 ( s ( i ) t − f ( Dsrt ) ( i ) ) f ′ ( Dsrt ) ( i ) r ( j ) t . ( 16 ) Each of these updates has the form of a local synaptic plasticity rule , under the following assumptions : W ( ij ) is a basal synapse from neuron j to neuron i , r ( i ) t and r ( j ) t correspond to the pre- and postsynaptic firing rates , respectively , and f ( Wst ) ( i ) corresponds to the local basal current injected into neuron i . Thus , assuming that a basal synapse has access to both the neuron ’ s firing rate and its local basal synaptic current at a particular moment in time , ∆W ( ij ) is local ; the same principle holds for the apical updates . If λt = 0 , then network activity is driven by the generative inputs , and so the parameter updates for basal synapses depend on apically-driven activity , as has been observed experimentally [ 32 ] ; similarly , apical synaptic plasticity should depend on basally-driven activity . The updates for the generative transition matrix , Dr–determining the leakiness of the apical dendritic compartments–are gated by 1 − kt , indicating that parameter updates are delayed upon entering ‘ inference ’ mode : this could reasonably be implemented biologically by a slow cascade of biochemical processes that delay changes in neural parameters , as has been proposed by previous plasticity models [ 33 , 34 ] .

Summary: This work combines probabilistic learning in the tradition of the Helmholtz machine with the recent hypothetical ideas about apical dendrites carrying top-down learning signals (expectation) to implement BP using local learning rule promoted earlier by numerous authors such as Larkum,  Lillicrap and Richards.  There are innovations. The impression learning mechanism proposed does sleep-wake in real-time.  It avoids the offline sleep phase in learning by using a global switching signal that can briefly substitute real incoming data with generative samples to enable learning continuously in a way that minimally perturbs natural neural trajectories.  This presents another new approach that allows the brain to learn generative models through local synaptic plasticity while concurrently performing sampling-based approximate inference wrt these models.  Incidentally, this work has been presented in COSYNE 2021 by the authors.