Learning rule influences recurrent network representations but not attractor structure in decision-making tasks

1 Introduction . Computational neuroscientists have increasingly used recurrent neural networks ( RNNs ) to model cortical computation and gain insight into how the brain performs cognitive and motor tasks . Typically , neuroscientists design RNNs to model a particular brain region during a behavioral task by generating an artificial dataset that abstracts the task . RNNs are trained to perform this task and often exhibit similar representations and low-dimensional dynamics to neural population recordings . This similarity is used to argue that the RNN is a good model of the brain circuit ’ s computations and neural dynamics . In this way , RNNs have been successfully used to study cognitive tasks , including decision-making and working memory [ 1–7 ] , as well as motor tasks [ 8–12 ] . Because RNN parameters are fully observed , trained RNNs can be analyzed to propose new hypotheses for how these computations are carried out in the brain [ 13 ] . One potential concern of this approach is that there are significant differences between RNNs and biological circuits . Further , RNNs may exhibit different representations and dynamics as a result of design choices , such as hyperparameters and architectures . Maheswaranathan , Williams , et al . [ 14 ] investigated whether there were any universal properties across recurrent architectures and activation functions . They found through a numerical study that RNNs tend to exhibit different representations across architectures ( RNN , GRU , LSTM , UGRNN ) and activation function ( relu , tanh ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . but exhibited universality in attractor topologies . These results encouragingly suggest that , although RNN architecture and activation function may result in different representations , the dynamical mechanisms of these networks appear to be conserved irrespective of these choices . A similarly important aspect of RNN modeling is how networks are trained , which differ in nature from biological neural circuits . Neural circuits are believed to employ Hebbian learning principles [ 15 ] , and while it is possible that they may employ a backpropagation-like algorithm , typical training approaches in RNNs are not necessarily biologically plausible [ 16 , 17 ] . For example , backpropagation-through-time assumes global knowledge of all the parameter weights in the network . Computational neuroscience studies utilize a diverse range of learning algorithms , from stochastic gradient descent with backpropagation-through-time [ 1–3 , 5 , 6 , 9–11 ] , Hebbian-inspired learning [ 18 ] , FORCE learning [ 19 , 4 , 20 ] , and evolutionary learning through genetic algorithms [ 21 ] . Different learning rules result in different learning trajectories through parameter space , leading to the possibility that RNN computation is learning rule dependent . If learning rule affects the learned solution , then conclusions of computational neuroscience studies may depend on the experimenterchosen learning rule . We therefore sought to empirically study the similarities and differences in RNN representations and dynamics across learning rules . In particular , we study the topological structure of RNN attractor ( or fixed point ) states and the representational geometry following training with different learning rules . We find that all learning rules resulted in similar attractor topologies across different decision-making tasks . However , while we find evidence that the attractor structure is invariant to the choice of learning rule , it is not invariant to task hyperparameters . We found that task hyperparameters for the contextual integration task from different studies resulted in RNNs with different attractor structure and dynamical mechanisms . Interestingly , we also find that as we train RNNs on progressively more complex tasks , the attractor topology remains invariant to the choice of learning rule even though representational geometries become more individualistic . The results of our study therefore provide insights into what properties of RNNs are likely conserved across design choices , which in turn supports the study of RNNs to probe dynamics in biological neural circuits . 2 Methods . 2.1 Model Architecture . We trained continuous time RNNs , which when discretized via Euler ’ s method , evolve as xt+1 = ( 1− α ) xt + α ( Wrecf ( xt ) + Winut ) ( 1 ) where xt ∈ R50 is the pre-activation state , ut ∈ RM is the input to the RNN , f ( · ) is the activation function , and α = ∆t/τ is the Euler time step , ∆t , divided by the network time constant , τ . When α = 1 , the continuous time RNN simplifies to a vanilla RNN . Unless noted otherwise , the activation function is f ( · ) = 1 + tanh ( · ) , and we define rt = f ( xt ) . We shifted the tanh ( · ) by 1 so that its output is always positive , as a non-negative activation is typically used for computational neuroscience studies [ 2 ] . The RNN output yt is a linear readout of the hidden states , yt = Woutf ( xt ) . ( 2 ) Wrec was initialized with weights drawn fromN ( 0 , g 2 N ) where g = 1 is the network gain andN = 50 is the number of artificial neurons in the RNN . Win weights were initialized from N ( 0 , 0.5 ) . Wout were initialized from N ( 0 , 0.1 ) . At the beginning of all trials , the initial hidden state x0 ∈ R50 was drawn from N ( 0 , 0.3 ) . All networks were trained using the mean-squared error ( MSE ) loss . The tasks and learning rules are described in the next sections . In total , we trained and analyzed 1,920 RNNs in this numerical study1 . 2.2 Tasks . We used three tasks based on commonly studied perceptual decision making tasks from the neuroscience literature [ 22 , 1 , 14 , 18 ] . 1Training was carried out in approximately 7 days on an RTX 2080 . H and FF RNNs were trained using CPUs both locally and on AWS Random dots motion ( RDM ) : RNNs received a 1-dimensional Gaussian noise input [ 22 ] . For each trial , we drew a random number µ ∼ U [ −0.1857 , +0.1857 ] , which modeled the “ coherence ” of a traditional RDM stimulus [ 23 , 1 ] . At each timestep , t = 0 , 1 , . . . , 750 ms , the input to the network was drawn from ut ∼ N ( µ , 1 ) . The RNN was constrained to output yt = 0 at t = 0 and yt = 1 ( or yt = −1 ) at t = 750 ms if µ was positive ( negative ) . These input and output values were chosen to be consistent with a previously implemented context-dependent integration task , described next [ 1 ] . Context-dependent integration ( CDI ) : We implemented the CDI task in the same way as [ 1 ] . RNNs received two static binary context inputs and two time-varying Gaussian noise inputs . For each trial , one context input was 0 and the other was 1 , providing a one-hot encoding of which Gaussian input stream should be integrated to compute the output . For each trial , two Gaussian noise inputs where both independently sampled fromN ( µ1 , 1 ) andN ( µ2 , 1 ) where µ1 and µ2 where both independently sampled from U [ −0.1857 , 0.1857 ] . Samples were drawn at t = 0 , 1 , . . . , 750 ms. RNNs were trained to output zero at t = 0 and ±1 at t = 750 ms depending on the sign of µ for the relevant input . N-input context-dependent integration ( N-CDI ) : We generalized the CDI task by considering N noisy input streams and N static context inputs , which we call the N-CDI task . Only one of the N static context inputs was 1 , indicating which input should be integrated to compute the output . Each noisy input stream was generated the same way as described for the CDI task . As in the CDI task , the RNN was trained to output zero at t = 0 , and ±1 at t = 750 ms depending on the sign of µ for the relevant input . We trained networks from N = 2 to N = 6 contexts . 2.3 Learning Rules . We trained RNNs with four different learning rules : stochastic gradient descent with gradients computed using backpropagation-through-time ( BPTT ) , a genetic learning rule ( GA ) [ 21 ] , a Hebbian ( H ) learning algorithm [ 18 ] , and the Full Force ( FF ) algorithm [ 20 ] . For all learning rules , training was terminated when accuracy on a validation dataset of 2,000 trials exceeded 90 % . BPTT : Gradients were computed using backpropagation through time over the entire trial length . After computing gradients , we updated parameters using the Adam optimizer with a mini-batch size of 500 trials and a learning rate of 0.0005 . GA : Each iteration of the GA involves producing a more accurate generation of RNNs from the prior generation . We started with an initial population of 50 RNNs with random initializations as described in Section 2.1 , termed generation zero . We then computed the loss of generation zero RNNs over 500 trials . The five RNNs with the lowest loss were then used to produce the next generation ( generation one ) . The next generation of RNNs was constructed by randomly choosing one of the five best RNNs from the prior generation and applying a small amount of noise to all the weight matrices . The noise was drawn from N ( 0 , 0.005 ) . This resulted in 50 new RNNs . The original 5 RNNs ( without perturbation ) were also included in the next generation . This process was repeated until our validation criterion was met . The final RNN was the RNN with the lowest MSE in the final generation . H : We used the learning rule described in Miconi [ 18 ] . In brief , during training trials , all weights , accumulated an eligibility potential . The weight wi , j , which is the ( i , j ) element of Wrec , had an eligibility potential ei , j , given by , ei , j ( t ) = ei , j ( t− 1 ) + [ rj ( t− 1 ) × ( xi ( t ) − x̄i ) ] 3 ( 3 ) where rj is the activation of artificial unit j and xi is the pre-activation value of artificial unit i. x̄i designated the average pre-activation value of artificial unit i across the trial . At the end of each trial , the weights were updated according to the eligibilities and the discrepancy between the current trial loss , R , and a running average of past losses for this trial type , R̄ , ∇wi , j = ηei , j ( R− R̄ ) ( 4 ) for a learning rate , η = 0.01 . This learning rule therefore alters synapses ( weights ) when the activity of post-synaptic and pre-synaptic neurons are correlated during training , mimicking Hebbian plasticity . In this learning rule , the RNN output is a readout of a single unit [ 18 ] . These Hebbian networks used f ( · ) = tanh ( · ) , since the shifted 1 + tanh ( · ) resulted in training difficulties . FF : We used the learning rule proposed by DePasquale et al . [ 20 ] . In short , this learning rule uses a recursive least squares algorithm to minimize the difference between the RNN ’ s output and the target output . Additionally , recurrent activations are constrained to match the activations of a second randomly initialized RNN that receives both task inputs and outputs . These FF networks used f ( · ) = tanh ( · ) , since the shifted 1 + tanh ( · ) resulted in training difficulties .

Summary: This paper studies the effect of learning rules on the algorithms learned by recurrent networks on context-dependent integration tasks. A number of continuous time vanilla RNNs are trained on integration tasks with different learning rules: backprop, genetic algorithms, hebbian learning, and full-force. The paper finds that across these learning rules, all networks learn consistent attractor topologies in order to solve the task. As the complexity of the task is increased (increasing number of contexts), the representations start to differ, but the attractor structure is preserved. The paper finds that using line attractors to solve integration seems to be a universal property of these networks, independent of learning rule. Finally, the paper shows how properties of the learned attractors (both the topology and whether the attractors are continuous manifolds or discrete points) vary with task input noise.