Cortico-cerebellar networks as decoupling neural interfaces
Authors: Joseph Oliver Pemberton
Ellen Boven
Richard Apps
Rui Ponte Costa
Venue: NeurIPS 2021 Poster (2021)
Keywords: Cortico-cerebellar loop
cerebellar function
decoupled neural interfaces
locking problem
temporal credit assignment
RNNs

Abstract: The brain solves the credit assignment problem remarkably well. For credit to be assigned across neural networks they must, in principle, wait for specific neural computations to finish. How the brain deals with this inherent locking problem has remained unclear. Deep learning methods suffer from similar locking constraints both on the forward and feedback phase. Recently, decoupled neural interfaces (DNIs) were introduced as a solution to the forward and feedback locking problems in deep networks.
Here we propose that a specialised brain region, the cerebellum, helps the cerebral cortex solve similar locking problems akin to DNIs. To demonstrate the potential of this framework we introduce a systems-level model in which a recurrent cortical network receives online temporal feedback predictions from a cerebellar module. We test this cortico-cerebellar recurrent neural network (ccRNN) model on a number of sensorimotor (line and digit drawing) and cognitive tasks (pattern recognition and caption generation) that have been shown to be cerebellar-dependent. In all tasks, we observe that ccRNNs facilitates learning while reducing ataxia-like behaviours, consistent with classical experimental observations. Moreover, our model also explains recent behavioural and neuronal observations while making several testable predictions across multiple levels.
Overall, our work offers a novel perspective on the cerebellum as a brain-wide decoupling machine for efficient credit assignment and opens a new avenue between deep learning and neuroscience.

Cortico-cerebellar networks as decoupling neural interfaces
Joseph Pemberton"
Ellen Boven"
Bristol Computational Neuroscience Unit Dept. of Computer Science, SCEEM University of Bristol, UK oq190428bristol. ac.uk
Bristol Computational Neuroscience Unit School of Phys., Pharm. and Neuroscience University of Bristol, UK
ellen wwwenbbristol...........o...o... ac.
Richard Apps
Rui Ponte Costa
School of Phys., Pharm. and Neuroscience University of Bristol, UK apps@bristol..ee ac
Bristol Computational Neuroscience Unit Dept. of Computer Science, SCEEM University of Bristol, UK rui. mustabbbbbiiit........ ac uk
Abstract
The brain solves the credit assignment problem remarkably well. For credit to be assigned across neural networks they must, in principle, wait for specific neural computations to finish. How the brain deals with this inherent locking problem has remained unclear. Deep learning methods suffer from similar locking constraints both on the forward and feedback phase. Recently, decoupled neural interfaces (DNIs) were introduced as solution the forward and feedback locking problems in deep networks. Here we propose that a specialised brain region, the cerebel- lum, helps the cerebral cortex solve similar locking problems akin to DNIs. To demonstrate the potential of this framework we introduce systems-level model in which a recurrent cortical network receives online temporal feedback predie- tions from cerebellar module. We test this cortico-cerebellar recurrent neural network (ccRNN) model on a number of sensorimotor (line and digit drawing) and cognitive tasks (pattern recognition and caption generation) that have been shown to be cerebellar cerebellar-dependent. In all tasks, we observe that ccRNNs facilitates learning while reducing ataxia-like behaviours, consistent with classical experi- mental observations. Moreover, our model also explains recent behavioural and neuronal observations while making several testable predictions across multiple levels. Overall, our work offers novel perspective on the cerebellum as a brain- wide decoupling machine for efficient credit assignment and opens a new avenue between deep learning and neuroscience.
Introduction
Efficient credit assignment in the brain is critical part of learning, However, how the brain solves the credit assignment problem remains mystery. One of the central issues of credit assignment across multiple stages of processing is the need to wait for later stages to finish their computation before learning can take place. deep artificial neural networks, where processing is divided into forward (i.e. prediction) and feedback (i.e. error backpropagation) phase, these constraints are explicit [1-5]. Given hidden activity at certain layer or timestep, network computations
'Equal contributions
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

at subsequent layers/tmesteps must first be computed before feedback gradients ōh are available for network parameters to be updated. Not only does this introduce computational inefficiency, in particular for the temporal case where backpropagation through several timesteps is required, but it also considered biologically unrealistic [6]. Recently, framework was introduced to decouple forward and feedback processing in artificial neural networks decoupled neural interfaces (DNI; Jaderberg et al. [5])² in which a connected but distinct neural network provides the main network with predicted forward activity or feedback gradients, thereby alleviating the locking problems.
Here, we propose that a specialised brain area, the cerebellum, performs a similar role in the brain. In the classical view the cerebellum is key for fine motor control and learning by constructing internal models of behaviour [7-11]. More recently, however, the idea that the cerebellum is also involved in cognition has gained significant traction [12-14]. An increasing body of behavioural, anatomical and imaging studies points to role of the cerebellum in cognition in both human and non-human primates [12, 14-16]. In particular cerebellar impairments have been observed in language [15], working memory [17], planning [18], and others modalities [19]. Coupled with its notoriously uniform structure, these observations suggests that the cerebellum implements universal function across the brain [7-9, 20J. Moreover, experimental studies looking at cortico-cerebellar interactions have directly demonstrated that cerebellar output is crucial for the development and maintenance of neocortical states [17, 21, 22]. However, to the best of our knowledge, no computational formulation exists which explicitly describes the function of such interactions between the cerebellum and cortical areas.
With this in mind and inspired by deep learning DNIs we introduce systems-level model of cortico- cerebellar loops. In this model and consistent with the cerebellar universal role, the cerehellum serves to break the apatio-temport locks inherent to both feedforward and feedback information processing in the brain, akin to DNI. In particular, we posit that the cerebellar forward internal model hypothesis of the cerebellum is equivalent to DNI-mediated unlocking of feedback gradients. Following this view the cerebellum not only provides motor or sensory feedback estimates, but also any other modality encoded by particular brain region. In this regard we introduce cortico-cerebellar RNN (ccRNN) model which we test on sensorimotor tasks: (i) a simple line drawing task [23-25) and (ii) drawing tasks with temporally complex imput based on the MNIST dataset, but also (iii) on cognitive task caption generation [15]. Our results support the decoupling view of cortico-cerebellar networks by showing that they improve learning while reducing ataxia-like behaviours in range of tasks, being qualitatively consistent with a wide range of experimental observations [15, 23-25].
2 Cerebellum as a cortical feedback prediction machine
We first describe DNIs following Jaderberg et al. [5] and then establish the link to cortico-cerebellar networks. We focus on "backward" (feedback) DNI which directly facilitates learning and is the model considered in this paper. but we also consider "forward" DNI to be strongly relevant to cerebellar computation (see SM A).
Although in this paper we focus on the DNI in a temporal setting, to better explain DNI we first briefly explain the spatial case. This case not only serves as a general basis for the temporal case (i.e. one can think of a recurrent neural network as a specific instance of a feedforward network), but also highlights the possibility place cerebellar modules between different pathways. Assume that a feedforward neural network consists of N layers, with the ith layer (1 < i S N) performing a "computational step" fa with parameters O,. Given input æ at layer 1, the output of the network at its final layer is therefore given by fN (fN-1(... fe(f((((()))))) (z)) Let Fi denote the composition of steps from layer i to layer (inclusively). Finally, let hs denote the (hidden) activity at layer i, so that h4 (((-11 with ho x.
We now illustrate the learning constraints of standard artificial neural networks used in deep learning. Suppose that network is in its feedback (or backpropagation) phase, with current ttaret pair al. is required, where L = L(u. 5) is the loss (Ir, 3). To update the layer parameters 0; the gradient d0; which compares the target value against the model output ý = F. (a)) under some loss function L: dl.
we then apply gradient descent on the parameters, 0, < 0, - a 28, with learning rate a > Suppose however that the network has only recently received the input x and is currently only at layer of
PNNs are related to earlier work on using network critics to train neural networks [2].

---

the forward computation. In order to update the corresponding parameters of that layer. 8,. the layer must first wait for all remaining layers to finish f, (j i) for the loss to be computed. Only then are the various gradients of the loss are backpropagated and dL is finally available. The enforced wait 20;
makes layer i "feedback locked"3 to F and imposes an complete dependence between learning of the layer in question and the computations in the remaining network.
The basis of DNI is to break the feedback lock by sending a copy of the hidden layer activity h2 to a separate neural network C, termed "synthesiser" in [5] and which we propose as being implemented by the cerebellum in the brain, which then returns synthetic gradient 8, an estimate of the real loss dL, We can then update our gradient with respect to the layer, i.c. (h,), = 8, Rs ó, where ó == 2h, original layer immediately according to gradient descent, effectively breaking the feedback lock with
0: t 0, aco; dh:; d0,
= C((b)
How we trust the synthetic gradient to improve performance? ? The parameters of C are themselves learnt so as to best approximate the observed feedback gradient. That is, once the standard forward and feedback computations are performed in the rest of the network (which may take some time) and dL is available, we minimise the objective function Le = 1|ō; ah, ||. Assuming C learns decent dh, approximation, the original layer parameters should then move (roughly) along the direction of the true loss gradient. Consistent with the need for this approximation a model with a fixed cerehellum C (i.e. not learnt) gives poor or often learning. indicating that C output is not simply facilitating learning via a stabilising effect (Fig. S7).
Temporal feedback in RNNs Using the same terminology but replacing layer indices with time I and setting 0, = 0 Vi. we can apply the above to recurrent neural networks (RNNs; Fig. 1a). Now the synthesiser predicts the effect of the current hidden activity he to future losses Ly>t = =Er>e Ly, effectively providing the RNN with (approximate) temporal feedback before the sequence may even be finished. As an optimization algorithm to obtain the true temporal feedback 6, we use backpropagation through time (BPTT). In principle, this feedback is only available once all the future feedback is available and C itself would be feedback locked. To circumvent this issue C(h,) learns using a mixture of nearby true feedback signals and bootstrapped (self-estimation term (Fig. lb), where for each we now minimise 1| - de with de = -ET= dL- term includes the nearby feedback signals backpropogated within some limited time horizon T, and the second term includes the bootstrapped cerebellar prediction which estimates gradient information beyond this horizon (analogous to temporal difference algorithms used to estimate value functions in reinforcement learning). In this case the cerebellum feedback facilitates learning by enabling the RNN to learn using (predicted) future feedback signals that would otherwise be inaccessible, enabling a future-aware online learning setup. In particular, this model has already been demonstrated to thrive in the harsh but more biologically reasonable setting of reduced temporal windows for BPTT (small T; of [5]).
dh. C(hr) der [5]. Here the first The
2.1 Mapping between cortico-cerebellar networks and decoupled neural interfaces
Building on the temporal feedback DNI we introduce a systems model for cortico-cerehellar com- putation, which we denote ccRNN (corttic--eerbeellar recurrent neural network; Fig. 1). The model includes two principal components: an RNN which models a recurrent cortical network (or area, e.g. motor cortex or prefrontal cortex), and a feedforward neural network the synthesiser C- which receives RNN activity before returning predicted feedback & and which we interpret as the cerebellum. The cerehellum receives the current cortical activity h, through its mossy fibres which then projects onto granule cells and Purkinje cells and sends back the predicted cortical feedback 8,
At the architectural level our model is in keeping with the observed anatomy of the cerebral-cerebellar circuitry. Recurrent connections are widespread in the neocortex and are considered functionally important for temporal tasks [26], whereas the cerebellum exhibits relatively simple feedforward architecture [7-9]. Moreover, an important defining feature of the cerebellum is its expansion at the granular layer: with > 50 billion granule cells (more than the rest of the brain's 's neurons combined)
3 We use feedback locked networks to incorporate both the "update" and "backward"' lock as described in [5].
3

---

TT
C
hr
Input
Cortical area
83
C
t
t2
t3
It
A
h1
h2
A
A
A
hi
Inferior Olive
8s
h1
h2
ha
82
8z
ht
c
C
C
time
Cerebellum
Figure 1: Cerebellum as a cortical feedback prediction machine. (a) Learning in a given recurrent cortical area A is modelled as feedback-encoding gradients (red arrow) originating from a loss or error region L at the end of a task (time 7). Meanwhile the cerebellum receives the current cortical activity he (black arrow) which projects
granule cells (GC; orange) and Purkinje cells (PC; orange) and sends back the predicted cortical feedback 8, (blue arrow). Cerebellar learning mediated by the inferior olive, which compares estimated feedback 8, with observed feedback &,, computing Lc ~ ||ó; - 8ell (see text for more details). (b) Example cortico-cerebellar model unfolded across three time steps. Starting at 23 the cortical network A receives feedback 83 (red), which is then transmitted to the cerebellar network as cortical feedback 63 (light blue). The cerebellum generates cortical feedback predictions 63 (blue) given cortical activity htg (black), and learns using inferior olive (diamond) error signals (red arrow). Before 13, cortical feedback is not readily available, thus the cerebellum learns through bootstrapping. In this case the inferior olive (diamond) compares the old cerebellar prediction (e.g. 82) with the new one (e.g. 83) to generate cerebellar learning signals (red arrow; see main text for details).
there is a considerable divergence from the ~ 250 million input mossy fibres. To replicate this, in experiments we include high divergence ratio 4-5) between the RNN network size and the hidden layer of the cerebellar network (see SM for network sizes). In our experiments this ratio helped the cerebellar module learn more quickly (not shown).
At the functional level, our model suggests that the cerebellum should encode a variable dependent error signals (in particular, the eπоо gradient with respect to neocortical activity), consistent with neural recordings showing cerebellar-encoding of kinematic errors during motor learning [27-29], but also reward prediction signals in cognitive tasks [30].
Finally, we conform to the classical view that the inferior olive would be the site which cerebellar prediction and feedback are compared to compute the cerebellar cost Le which drives cerebellar learning (7-9, 31]. Crucially, we also highlight the cerebellum's ability to learn the case delayed feedback which is necessarily the case under this model via its specially adapted timing rules for plasticity at the parallel fibre synapse [32].
2.1.1 Relationship to existing cerebellar computational models
For over half century computational neuroscientists have posited that the feedforward circuitry and available error signals (stemming from the inferior olive) make the cerebellum well suited for pattern recognition [7-9, 33, 34). This computational perspective is in line with our model, but which patterns should be learnt, and how is cerebellar output used by the rest of the brain has remained unclear. The most prevalent theory points towards the cerebellum nervous system [10], which may arise in two forms, an inverse model or a forward model.
"internal model" of the
We argue that ccRNN is in fact an instance of the forward model hypothesis (see SM A for link between DNIs and the inverse model). In the classical forward model of sensorimotor control, cerebellum receives an efferent copy of the motor command from the motor cortex [11, 35), and the

---

respective sensory feedback. With these two inputs the forward model learns to predict the sensory consequences of motor commands. Generalising to the non-motor domain. a similar predictive model can be applied to the brain activity between virtually any two brain areas [36]. The prefrontal cortex and the temporo-parietal cortex, for instance, both involved planning of cognitive behaviour and decision making, are possible vognitive-associated areas where forward "mental" model may be applied [12-14, 36). Our model also takes neural activity ht from any brain area and its associated feedback signals 6, which provides, to the best of our knowledge, the first explicit computational implementation of postulated brain-wide forward models. In addition, existing computational models have far only been used in simplistic sensorimotor tasks, whereas our model can be readily applied to a wide range of tasks as we demonstrate below.
2.1.2 Encoding feedback gradients in the brain
Our systems level model uses BPTT to generate temporal feedback signals. We use weak forms of ManncatedBPTTT (i.e. with a reduced BPTT window), using only one BPTT step in some cases, which mimics a more biologically realistic setting. It is in principle possible to use models capable of generating biologically plausible feedback [6, 37-41]. Computational models of spatial back- propagation of gradients (i.e. across layers) suggest that gradient-encoding feedback signals must be encoded in distal dendrites [37, 38.. In addition, these models demonstrate that explicit gradient feedback information is not needed, but rather that this can be reconstructed locally in dendritic microcircuits using cortical feedback [38]. The cerebellum provides feedback connections via higher order thalamocortical loops [42] that target apical dendrites of pyramidal cells in the neocortex [43-45]. These cerebellar feedback loops thus provide a good substrate for driving cortical learning through cortical feedback prediction, which is consistent with model. On the other hand Bellec et al. [46] have shown that temporal gradients as used in BPTT can be approximated by biologically plausible cligibility traces that transmit gradient information forward in time. Both of these solutions can, in principle, be incorporated into our framework, in which ccRNN would predict feedback activity originating from upstream brain areas (as in Sacramento al. (38)) and/or make forward predictions which are then integrated with locally computed eligibility traces [46].
3 Results
We contrast model with cerebellar feedback by comparing ccRNN to a purely cortical that is. without cerebellar predicted feedback RNN (cRNN) on a variety of tasks. Tasks are chosen so as to broadly approximate conditions in which cerebellar patients have shown deficits, including simple and then more complex sensorimotor control/planning paradigms before ending with a challenging language-based task.
Cortical recurrent neural networks are modelled as long short-term memory networks (LSTMs) [47] in line with previous work [48, 49]. A linear readout is added on top of the RNN to compute final model output. The window of cortical temporal feedback modelled using BPTT of specific truncation window. We refer to feedback originating from the cerebellum as feedbbck..
For the sensorimotor tasks we also test the models with different external feedback intervals. This defines how often the model has access to an external teaching signal (e.g. only every 2 timesteps). This mirrors biologically-relevan setting where visual feedback not continually available but can only be sampled at some rate. Therefore, the model is trained with external feedback only for some timesteps, and that it should ideally generalise to the remaining timesteps. We also evaluated the model performance under these testing conditions (i.e. in which external feedback was not available) to yield an ataxia score, a measure designed to quantify the irregularity of movement associated with cerebellar ataxia [50]. Concretely, we compute the ataxia score as the total deviation from the model output to the targets provided during training as well as those for which external feedback was unavailable.
ataxia score
Σ CV., Dc) + EE De)
tEFB
EEFB
where ye and ÿee denotes the target and model output at time of the sequence, respectively, L is the task-associated error function (e.g. mean squared error), and FB denotes the set of times at which error feedback is given. Note that the first term is simply the training error (which defines the model optimisation problem), whereas the second term quantifies the model's 's ability to generalise to unseen parts of the data (for example, intermediate points on line).
5

---

3.1
Simple sensorimotor task: line drawing
20-
cRNN
ooRNN
norm. arcor
Cortical area
hv
training
1.0
10-
0.8
0.8
h+s
C
200
300
420
900
69NN
cSRNN
52%
20%
50%
70%
spachs
cortical temporal fredbacck(5)
Figure 2: Line drawing task. (a) Schematic of the task. One out of six cues is given at ( and the network must learn to draw a straight line to the respective target. Sparse feedback used so that an explicit target is only provided at every other timestep during training. (b) Learning curves for cortical RNN (cRNN; grey) and cortico-cerebellar RNN (ccRNN; orange). These networks use cortical temporal feedback of 10% (ef. d), increasing the difficulty of learning in the RNN. Trajectories produced by the models towards 6 targets are given as insets. (c) Ataxia score for both models at the end of training. (d) Mean squared error of the ccRNN normalised to that of cRNN over different cortical temporal feedback windows (reported as a percentage of total task length); arrow indicates level of feedback used in (b.c). All experiments used 10 seeds.
Inspired by classical sensorimotor studies in the cerebellum we first test a simple line drawing task [23-25], in which given cue at time ( the network needs to produce straight line towards one of 6 targets over 10 timesteps (Fig. 2a; see SM for more details). In order to model a more biologically realistic setting the external feedback (i.e. how far the model output is from a straight line) is only provided at every other timestep (sparse external feedback). Although the cRNN model learns to perform the task it does so more slowly and with higher variability (Fig. 2b), in line with cerebellar experiments [23-25]. Importantly, we also observe differences in the model output (Fig. 2b). Even though the models were only trained with sparse external feedback the ccRNN generalises near-perfect straight line whereas the cRNN produces a much more irregular, oscillatory- output (Fig. 2b), leading to an increased ataxia score (Fig. 2c). Moreover, cerebellum module is beneficial when the main cortical recurrent network struggles to learn the task on its own (i.e. when using a short temporal feedback window; Fig. 2d, S1). This is consistent with the observation that the cerehellum is more important to correct for larger errors during motor task [51]. In addition to this prediction our model makes several other predictions from systems level down to cellular level. Below we highlight some of these predictions using the line drawing task and relate it to the experimental literature where possible.
model predicts that the
Facilitation of learning depends on feedback interval The benefits provided by the ccRNN model are stronger at intermediate feedback intervals (Fig. 3a). In particular, we find that when the external feedback is given continuously there are only minor benefits of having the cerebellar module, but if the external feedback interval increased then having predictive cerebellar module becomes more important until the feedback interval is just too large for the models to be able to learn the task (Fig. 3a). We observe these differences because the feedback predicted by the cerebellum is only useful once the external feedback is relatively infrequent. Although this is broadly in line with the important role of feedback serebellar-mediant learning [23, 52, 53) this exact prediction remains to be tested.
Cortico-cerebelin coupling is training-dependen As a measure of the cortico-cerebellar cou- pling we calculated the pairwise correlations between the activity of each neuron in the main cortical RNN and the activity of each hidden neuron (granule cell) in the cerebellar module (see SM). Our model reveals two distinct phases of cortico-cerebellar coupling during learning (Fig. 3b). During the fast phase of learning we observe noticeable rise the average cortico-cerebellar coupling (in line with [13]). In addition, our model predicts that as training continues the population correlation should gradually decrease. In the model these changes in coupling are explained by the fact that initially the model needs to output relatively high feedback due to the high errors, but as learning progresses

---

b
C
chort intervel
medium interval
long interval
traming oITor
trening
10
cortoo-conballer euuting
10
9.2-
opoch
oooch
opach
apocha
ror 6710r
12
49
107%
50%
40%
30%
20%
50
130
203
400
nsternal feedbwckk
early trariigg
Wdb trarring
ob ablatoo spoch
Figure 3: Model predictions using simple line drawing task. (a) Cerebellar feedback facilitates learning across a range external feedback intervals. (b) Cortico-cerebellar coupling measured as pairwise correlations for first 100 epochs of learning. Insets show the change in distribution of correlations at respective epochs (left: early epochs; right: later epochs). (c) Importance of cerebellar feedback reduces over learning revealed by ablation study. Top: Learning curves for few example points of ablation. Vertical lines represent points of ablation. Bottom: Summary plot of the training error (ccRNN normalised cRNN) across epoch of ablation. Error bars represent standard error of the mean using 10 seeds.
errors (and feedback) becomes gradually smaller, thus making the cerebellar module less correlated with the activity of the main cortical network.
Importance of cerebellar feedback reduces over learning The decrease in coupling over learning predicts that the importance of the cerebellar feedback should become weaker over learning. To test this idea and which phases of learning are more crucial for the cerebellar-mediated facilitation of learning we completely ablated the cerebellar module at different points during learning (Fig. 3c). Although these results are consistent with the classical observations made in cerebellar patients [23-25] this ablation study predicts nonlinear relationship between ablation time and performance impairment, which has not been tested experimentally. Our model predicts that ablating the cerebellum after learning has started actually has a more detrimental impact than there was not cerebellum to start with (Fig. 3c). This happens because when a cerebellum module is present the main cortical network starts learning by relying partially on the feedback predicted by the cerebellum, so that this component is suddenly removed the learning trajectory of the main network is perturbed and needs to be readjusted. After this first critical phase of learning the cerebellum becomes gradually less important (> 50 epochs). These results echo the existing literature [54, 55] in that though shared cortico-cerebellar dynamics might emerge during the acquisition of a novel motor sequence, these dynamics can become less interdependent once knowledge becomes consolidated.
3.2 Advanced sensorimotor tasks
To test how the previous line drawing task generalises to a more realistic setting with continuous input we introduce two sensorimotor tasks which build on the classical MNIST dataset [56]. Here we present the model with a given MNIST image sequentially row by row (28 rows, each with 28 pixels), whilst training the model to simultaneously draw digit-specific (i) straight line (ld-seqMNIST) or (ii) a digit template (dd-S)QMNIST) (Fig. 4a). The latter case is inspired directly from known cases of cerebellar agraphia [57]. As before we employ sparse external feedback at certain interval (every 4 timesteps).
As in the line drawing task ccRNN Iearns faster than cRNN while also showing reduced ataxia score (Fig. 4b, c). The positive effect of cerebellar feedback less apparent when the model was trained to write digits, possibly due to the less predictable, non-line non-linear nature of target output and subsequently external feedback. As with the line drawing task we find that predicted feedback is most valuable where strong constraints on BPTT are enforced (Figs. 4d, S3). We observed a similar dependency to the simple line drawing task on the level of external feedback (Fig. S4), cortico-cerebral coupling (Fig. $5). and ablations (Fig. S6). However, in contrast to the simple line drawing task, cerebellar feedback is still relatively important even in later stages of learning. This is due the non-zero error even after convergence in these tasks.
7

---

CRNN
10.0
Id-seq
urror (MSE)
7.5-
1.0
5.0
rorm.
2.5
0.8
0.0
CRNN
CLANN
10%
30%
dd-seqMNIST
40
12.5
I
BAA
3x
1.2
errer 045
2x
7.5
5.0
101
CRNN CCRNN
10%
20%
30%
Figure 4: Advanced sensorimotor tasks based on MNIST dataset. (a) Example model output given an MNIST digit as input (colour-coded by digit) from a trained ccRNN model for the line drawing sequential MNIST task (ld-seqMNIST, top) and the digit drawing sequential MNIST (dd-seqMNIST, bottom); underlying grey dots represent target output. Sparse feedback is used that an explicit target is only provided at every 4 timesteps during training. (b) Training mean-squared error. Ataxia score for both models at end of training. (d) ccRNN error (normalised to cRNN error) over different cortical temporal feedback windows (reported as a percentage of total task length); arrow indicates level of feedback used in (a,b,))). Error bars represent standard error of the mean using 10 seeds.
Finally, we also considered how ccRNN performs on the more standard sequential MNIST task, in which it is necessary to classify the digit at the end of the sequence [58]. In line with the regression tasks, we find significantly faster learning 10% higher accuracy after about 10 epochs) when using the ccRNN model (Fig. S2). This is consistent with a role of the cerebellum in decision making or sensory discrimination tasks [21, 59, 60).
3.3 Cognitive task: Caption generation
-isturtt a lange white Nespical prodee
C
white hed enting on a nodk fird perched on top of too brandh largs white tropical hind peeched on paiu tree
prediction erar (BPW)
3.28
3.26
3.24
A yan and meovso are standing to kitcher
is woman standing
people by bennnis waters
Temore
CORNN
grapes
epoch
10
Figure 5: Caption generation task. Model schematic with CNN (blue), cortical RNN (LSTMM in gray) and cerebellar module (orange). CNN is pretrained on a image dataset, while the RNN is trained predict the next word in caption. b Learning curves in bits per word (BPW). Sample test image with associated model produced captions (colour coded as in b; black denotes gold standard caption). More examples are given in the SM. Error bars represent standard error of the mean using seeds.
We emphasise that our framework does not only apply to sensorimotor tasks, but should generalise to virtually any task and learning paradigm (supervised, unsupervised and reinforcement learning). To demonstrate this and inspired by cognitive tasks in which cerebellar patients have shown deficits [61] we test our models in a caption generation task. In this task the network leamns directly from the data (i.e. unsupervised learning) to generate textual description for given image. All models have two components: pretrained convolutional neural network (CNN) to extract lower dimensional representation of the image, and an RNN to learn a simple model of language given visual representation from the CNN (Fig. 5a).

---

We use a standard dataset SLLVVR-2211-CLSS Russakovsky et al. [62]) and the networks are trained to maximise the likelihood of each word given an image (SM for more details). We find that also in this more challeging task the cerehellum module manages to learn good enough feedback to improve learning when compared with cRNN (Fig. 5b). Interestingly, the captions generated on test images are qualitatively more accurate for the ccRNN model (Figs. 5c, S8), suggesting that the ccRNN model better captures the contextual nature of this task, in line with experimental observations [61]. Moreover, ccRNN also outputs a better model of language as assessed by language metrics (Table S2).
4 Conclusions and discussion
We have introduced systems level model [5] of cortico-cerebellar function, where the cerebellum's s role is to provide the neocortex with predicted temporal feedback. We propose that an existing deep learning framework can mimic the function of one of the largest projection pathways in the brain, cortico-cerebellar networks. This systems level framework abides to the classical notion of the cerebellum pattern learning multi-layer perceptron in the context of the forward model hypothesis. In particular, our model suggests that the cerebellum encodes future errors (or analogously, rewards) which are then transmitted to the neocortex for efficient learning. It thus proposes how the brain might learn efficiently from future teaching signals: a key requirement of temporal credit assignment.
One of the advantages of our model (ccRNN) is that it can be readily applied to wide range of tasks as a systems-level model of cortico- cerebll-e interactions. Indeed, our model makes explicit the concept of "cognitive error" which is directly connected to growing empirical evidence for cerebellar role in working memory, planning. language and beyond [60, 63]. However, to make more specific predictions at the cellular, sub-cellular and cell-type level an important next step is model cortical feedback errors in a more biologically plausible fashion [38, 64).
Our model makes a number of predictions (see above). For example, that cerebellar involvement is most beneficial where neocortical temporal learning mechanisms are not long enough for the task at hand. This offers an explanation for the often detrimental impact cerebellar impairments can have on motor learning which are often temporally challenging. It is important to highlight that our results do not predict that cerebellar impairments will necessarily lead to an inability to learn, but merely a slower learning curve and perhaps lower performance threshold. There are however specific conditions in which the cerebellum is optimally placed to facilitate learning according to model, namely the regularity of the external feedback, the ability for learning of cortical networks, model also makes predictions
the difficulty of the task and the exact phase of learning. In addition about cortico-cerebelln coupling, predicting a steady decrease of the population correlation as learning progresses.
Our model puts forward a solution for how the brain deals with delayed feedback signals, by relying on the cerebellum to predict future feedback signals. A key experiment which could be used to test our model would be to demonstrate that brain areas responsible for computing prediction errors are not by themselves requisite for plasticity. The cerebellum by itself should be sufficient for cortical learning, provided that it has learnt a good predictive model of future feedback.
There are number of interesting directions to take forward with this model. For example, in the present study we do not explicitly model the long-range projections via the thalamus and pons that mediate cortico-cerebellar interactions [43-45], but for simplicity we have assumed direct connectivity. We predict that introducing these intermediate structures with low dimensionality has two potential important functions: (i) force the cerebellum to focus on learning only the most important feedback signals and (ii) gate in and out cerebellar feedback so that this is only allowed to modulate learning when beneficial. Moreover, here we have focused on a variant of the model that predicts feedback signals needed for learning, but is likely that the cerebellum is also involved in speeding up feedforward processing as in forward DNIs (see SM).
On the deep learning side there are also number of promising options to explore. Predicting feedback is hard task, due to its continuously changing nature. There are a number of architectural and cellular features of the cerebellum that may make learning feedback predictions more efficient in these models, namely mossy sparse connectivity (as observed between mossy fibres and granule cells 165, 66)) and modularity (as observed throughout the cerebellum [67]). At the same time, it may also be fruitful to consider decoupling neural processing more generally. Various other, potentially

---

non-cerebellar, methods for unlocking have been proposed in machine learning that may provide interesting avenues for neuroscience [68-70].
Overall, ccRNN provides a natural link between deep learning and existing cortico-cerebellar ideas and more classical cerehellar models. Moving forward we hope the framework to offer novel but concrete framework with which to study the cerebro-cerebellar loop for neuroscientists and as source of inspiration for machine learners.
Acknowledgments and Disclosure of Funding
We would like to thank the Neural & Machine Learning group. Paul Anastasiades, Paul Chadderton and Paul Dodson for useful discussions. JP was funded by EPSRC Doctoral Training Partnership award and EB by the Wellcome Trust (Neural Dynamics PhD Program). This work made use of the HPC system Blue Pebble at the University of Bristol, UK.
10

---

References
David E Rumelhart, Geoffrey E. Hinton, and Ronald Williams. Learning representations by back- propagating errors. Nature, 323 1986.
[2] Jürgen Schmidhuber. Networks adjusting networks. In Proceedings of" Distributed Adaptive Neural Information Processing", pages 197-208, 1990.
[3] Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propagation. In Joint european conference on machine learning and knowledge discovery in databases, pages 498-515. Springer, 2015.
[4] Adam II Marblestone, Greg Wayne, and Konrad P Kording. Toward an integration of deep learning and neuroscience. Frontiers in computational neuroxcience, 10:94, 2016.
[5] Max Jaderberg, Wojeiech Marian Czamecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In Proceedings of the 34th International Conference on Machine Learning- Volune 70, pages 1627-1635. JMLR. org, 2017.
[6] Timothy P Lillicrap and Adam Santoro. Backpropagation through time and the brain. Current opinion in neurobiology, 55:82-88 29,22999900 2019.
[7] David Marr. A theory of cerebellar cortex. The Journal of Physiology. 202(2)4377-70,, jun 1969. ISSN 00223751. doi: 10.1 113//p/yiool.16..s.p...... 1969 sp0088p00
URL http: //tt::://d.... weepii.iwlll......... com/10. 1113/jphysiol.
[8] James S Albus. A theory of cerebellar function. Mathematicul Biosciences, 10(1):25-61, 1971. ISSN 0025-5564. doi: http:///bb.orr//1000010000 11100/0000005557799000555511111 URL htt.aa////aarc/e//////////511110
[9] Jennifer I. Raymond and Javier F Medina, Computational principles of supervised learning in the cerchellum. Annual review of neuroscience, 41:233-2553 2018.
[10] Daniel M Wolpert, R Chris Miall, and Mitsuo Kawato. Internal models in the cerebellum. Trends in cognitive sciences, 2(9):338- 347, 1998.
[11] R. C. Miall, J. Weir, Wolpert, and Stein. the cerebellum a smith predictor? Journal of Motor Behavior, 25(3):003-216, 1993. ISSN 19401027. doi: 10.1080/00222895.. (1188999.1999... 1993.9942055
[12] Jeremy D. Schmahmann, Xavier Guell, Catherine J. Stoodley, and Mark A. Halko. The Theory and Neuroscience of Cerehellar Cognition. Annual Review of Neuroscience, 42(1):333-336 jul 2019. ISSN 0147-006X. doi: 10.1 144////muree-neuuo-00000 doi/10. 1146/
URL https://www annualrevievs org/
[13] Mark J. Wagner and Liqun Luo. Neurosciences, 43(1 ):42554, 2020. ISSN 1878108X. doi: 10. (t116).tiss2219.11..0. URL https:
Circuits for Cognitive Processing. Trends in
//doi.org/10. j. tins 2019. 11.002
[14] James A. Brissenden and David C. Somers. Cortico-cerebella. cerebellar networks for visual attention and working memory. Current Opinion in Psychology, 29:239-247, 2019. ISSN 2352250X. doi: 10.1016/j.copsy/ 2019.05.003. URL https:://dll org/ 1016/j copayc. 2019. 05. 0033.
[15] Xavier Guell, Franziska Hoche, and Jeremy D. Schmahmann. Metalinguistic Deficits in Patients with Cerebellar Dysfunction: Empirical Support for the Dysmetria of Thought Theory. Cerebellam, 14(1): 50 58, 2015. ISSN 14734230. doi: 10.1007/412311-01--0003
[16] Xavier Guell, John D.E. Gabrieli, and Jeremy D. Schmahmann. Triple representation of language, working memory, social and emotion processing in the cerebellum: convergent evidence from task and seed-based resting-state state fMRI analyses in a single large cohort. Neurolmage, 172:437 449, may 2018. ISSN 10959572.
doi: www.://nneooomee..22. 2018..00080000 0...00..
[17] Ben Deverett, Mikhail Kislin, David W Tank, S Samuel, and Н Wang. Cerebellar disruption impairs working memory during evidence accumulation. bioRxiv, page 521849, 2019.
[18] S C. Baker, R D Rogers, Adrian M Owen, C Frith, Raymond Dolan, R $ Frackowiak, and Trevor W Robbins. Neural systems engaged by planning: a PET study of the Tower of I London task. Neuropsycologiia, 34(6):515-526, 1996.
[19] Julie A Fiez. Steven E Petersen, Marshall K Cheney, and Marcus E Raichle. Impaired non-motor learning and error detection associated with cerehellar damage: A single case study. Brain, 11551)::55-1778 1992.

---

[20] Jöm Diedrichsen, Maedbh King, Carlos Hernandez-Castine Castillo, Marty Sereno, and Richard B. Ivry. Universal Transform or Multiple Functionality? Understanding the Contribution of the Human Cerebellum across Task Domains. Neuron, 102(5):91-992) 2019. ISSN 10974199. doi: 10. 10.1016/j.neuron.. 2019..4...110044... 0..0.21.
[21] Zhenyu Gao, Courtney Davis, Alyse Thomas, Michael Economo, Amada M Abrego, Karel Svoboda, Chris De Zeeuw, and Nuo Li. A cortico-cerchellar loop for motor planning. Nature, 563(7729)) 2018.
[22] Francois P Chabrol, Antonin Blot, and Thomas D Floic-Flogel, Cerebellar contribution to preparatory activity in motor neocortex. Neuron, 103(3):50--19, 2019.
[23] Jerome Sanes, Bozhidar Dimitrov, and Mark Hallett. Motor learning in patients with cerebellar dysfunction. Brain, 113(1):003-120, 1990.
[24] Peter A. Butcher, Richard B. Ivry, Sheng Han Kuo, David Rydz, John W. Krakauer, and Jordan A. Taylor. The cerebellum does more than sensory prediction error- based learning in sensorimotor adaptation tasks. Journal of Neurophysiology, 118(3):6622-1636 2017. ISSN 15221598. doi: 10. (0.1152/j.000451.200 201770
[25] Abdutraheem Nashef, Oren Cohen, Ran Harel, Zvi Israel, and Yifat Prut. Reversible Block of Cerebellar Outflow Reveals Cortical Circuitry Motor Coordination. Cell Reports, 27(9):2608 2619,64, e4,,, 2019. ISSN 22111247. doi: 10.1016/j.eerre.22013 www.ereepp.19...110.... 1000 URL httph::t///oot org/ 1016//////...0////.././/// celrep. 2019. 04. 100.
[26] Valerio Mante, David Sussillo, Krishna V Shenoy, and William T Newsome, Context-dependent computa- tion by recurrent dynamics in prefrontal cortex. nature, 503(7474):78- 84, 2013.
[27] Laurentiu S Popa, Angela L Hewitt, and Timothy Ebner. Predictive and feedback performance errors are signaled in the simple spike discharge of individual Purkinje cells. Journal of Neuroscience, 32(44): 15355,1 2012.
[28] Eric J Lang, Richard Apps, Fredrik Bengtsson, Nadia L Cerminara, Chris I De Zeeuw, Timothy J Ebner. Detlef H Heck, Dieter Jaeger, Henrik Jörntell, Mitsuo Kawato, and Others. The roles of the olivocerebellar pathway in motor learning and motor control. A consensus paper. The Cerebellum, 16(1):230-252, 2017.
[29] Laurentiu S Popa and Timothy J Ebner. Cerebellum, Predictions and Errors. Frontiers in Cellular Neuroscience, 12:524, 2019. ISSN 1662-5102, doi: (0.3389/fncee22211800222 URL https://wwat frontiersin, org/ wttg//rticee//ie
(0339//fncell 20w.cee........... 2018.00524.
[30] Mark Wagner, Tony Hyun Kim, Joan Savall, Mark Schnitzer, and Liqun Luo. Cerebellar granule cells encode the expectation of reward. Nature, 544(7648))96, 2017.
[31] Shogo Ohmae and Javier F Medina. Climbing fibers encode -ifffreeede prediction error during cerebellar learning in mice. Nature neuroscience, 18(12): 1983- 2015.
[32] Aparna Suvrathan, Hannah L Payne, and Jennifer Raymond. Timing rules for synaptic plasticity matched to behavioral function. Neuron, 92(5):959 2016.
[33] James C Houk, Jay T Buckingham, and Andrew G Barto. Models of the cerebellum and motor learning. Behavioral and brain sciences, 19(3):368-33)) -33,, 1996.
[34] Matthew Hausknecht, Wen-Ke Li, Michael Mauk, and Peter Stone. Machine learning capabilities of simulated cerebellum. IEEE transactions neural networks and learning 2ystems,2833):5100 28(3):10--522, 2016.
[35] M Ito. Neurophysiological aspects of the cerebellar motor control system. International journal neuralogy, 72):122-176, 1970. ISSN 0020-7446. URL http://euuropppacccoo abstxact/MED/ 5499516.
[36] Masao Ito. Control of mental activities by internal models in the cerebellum. Nature Reviews Neuroscience, 94))304-313, apr 2008. ISSN 1471 4011- doi: 10. m338/mmmm333.. URL www. wwwure.tre... wwt.wawwwweorreeeee/eo neurohttp://www http:://wwwwww/www/////paeaae nature. ctm/ attaaceeeeinnn........
[37] Jordan Guerguiev, Timothy P Lillicrap, and Blake A Richards. Towards deep learning with segregated dendrites. ELife, 6:c22901, 2017.
[38] João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, pages 8721-8732, 2018
12

---

[39] Blake A Richards and Timothy P Lillicrap. Dendritic solutions to the credit assignment problem. Current opinion neurobiology, 54:28-36, 2019.
[40] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake Richards, and Richard Naud. Burst- dependent synaptic plasticity can coordinate learning in hierarchical circuits, bioRxiv, 2020. doi: 10.1101/ 2020.03.30.015511 URL https: //wppp biorxiv. org/ 015511.
30.
[41] Nasir Ahmad, Marcel A J van Gerven, and Luca Ambrogioni. GATT-prop: A biologically plausible learning rule derived from backpropagation of error. arXiv preprint arXiv:2006. 06438, 2020.
[42] Simona V Gornati, Carmen B Schäfer, Oscar Eelkman Rooda, Alex L Nigg, Chris Zeeuw, and Freek E Hoebeek. Differentiating cerchellar impact on thalamic nuclei. Cell reports, 233)):690-2204 2018.
[43] KuangHua Guo, Naoki Yamawaki, Karel Svoboda, and Gordon MG Shepherd. Anterolateral motor cortex connects with medial subdivision of ventromedial thalamus through cell type-specific circuits, forming an excitatory maalamo-coricco-thalmme type neurons, Journal of Neuroscience, 38747):8777-877, 2018.
loop via layer apical tuft dendrites of layer 5b pyramidal tract
[44] Hirofumi Fujita, Takashi Kodama, and Sascha Du Lac. Modular output circuits of the fastigial nucleus for diverse motor and nommotor functions of the cerebellar vermis. Elife, 9:e58613, 2020.
[45] Paul G Anastasiades. David P Collins, and Adam G Carter. Mediodorsal and ventromedial thalamus engage distinet 11 eircuits in the prefrontal cortex. Neuron, 109(2):14--33, 2021.
[46] Guillaume Bellec, Franz Scherr, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets. arXiv preprint arXiv:1901.090% 2019.
[47] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98)::1335-1780 1997.
[48] Rui Ponte Costa, Ioannis Alexandros Assael, Brendan Shillingford, Nando de Freitas, and Tim Vogels. Cortical microcircuits gated-recurrent neural networks. In Advances in neural information processing systems, pages 272-283, 2017.
[49] Jane X Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as meta-rriifooremmen lcarning system. Nature neuroscience, 21(6):860-868 2018
[50] P Trouillas, Takayyanagi. M Hallett, RD Currier, SH Subramony, K Wessel, A Bryer, HC Diener, S Mas- saquoi, CM Gomez, et al. International cooperative ataxia rating scale for pharmacological assessment of the cerchellar syndrome. Journal of the neurological sciences, 145(2))005-21, 1997.
[51] Sarah E contributions to motor learning. Journal of neurophysiology, 103(4):225-22284 2010.
Amy Bastian, and Reza Shadmehr, Size of error affects ccrchellar
[52] H-C Diener and Dichgans. Pathophysiology of cerebellar ataxia. Movement disorders: official journal of the Movement Disorder Society, 7(2):95 109, 1992.
[53] Takuya Honda, Masaya Hirashima, and Daichi Nozaki. Adaptation visual feedback delay influences visuomotor learning. PLoS one, 777000, 2012.
[54] Julien Doyon, Virginia Penhune, and Leslie Ungerleider.
Distinct contribution of cortico-striatal and cortico-cerebellar systems to motor skill learning. Neuropsychologia, 41(3):552-26,, 2003.
[55] Elisa Galliano, Zhenyu Gao, Martijn Schonewille, Boyan Todorav, Esther Simons, Andreea S Pop, Egidio D' Angelo, Am M J M Van Den Maagdenberg, Freek E Hoebeek, and Chris De Zeeuw. Silencing the majority of cerchellar granule cells uncovers their essential role in motor learning and consolidation. Cell reports, 3(4):1239-123 2013.
[56] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database, 2010.
[57] Hyo Jung De Smet, Sebastiaan Engelborghs, Philippe F Paquier, Peter P De Deyn, and Peter Mariën. Cerebellar-indues apraxic agraphia: review and three new cases. Brain and cognition, 76(3):424 434,
2011. 1.
13

---

[58] Quoc V Le. Navdeep Jailly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint vcXiv:154400041,, 2015.
[59] Ben Deverett, Sue Ann Koay, Marlies Oostland, and Samuel H Wang. Cerebellar involvement in an evidence-accumulation decision-making task. Elife, 7:e36781, 2018.
[60] Maedbh King, Carlos R Hermandez-Castille Russell A Poldrack, Richard B Ivry, and Jörn Diedrich- sen. Functional boundaries in the human cerchellum revealed by a multi-domain task battery. Nature neuroscience, 22:1371 2019.
[61] Andrea L Gebhart, Steven E Petersen, and W Thomas Thach. Role of the posterolateral cerebellum in language. Annals of the New York Academy of Sciences, 978(1):318 333, 2002.
[62] Olga Russakovsky. Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252, 2015, doi: 10. 12637711-1----
[63] Michael P Alexander, Susan Gillingham, Tom Schweizer, and Donald T Stuss. Cognitive impairments duc to focal cerebellar injuries in adults. Cortex, 48(8):980 9900099000000 2012.
[64] Blake A Richards, Timothy Lillicrap. Philippe Beaudoin. Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, and Others. A deep learning framework for neuroscience. Nature neuroscience, 22(11)1171-11700 2019.
[65] Nicolas Schweighofer, K Doya, and F Lay. Unsupervised learning of granule cell sparse codes enhances cerebellar adaptive control. Neuroscience, 103(1):35-50, 2001.
[66] N Alex Cayco-Gajic, Claudia Clopath, and R Angus Silver. Sparse synaptic connectivity is required for decorrelation and pattern separation feedforward networks. Nature communications, 8(1):1-11, 2017.
[67] Richard Apps, Richard Hawkes, Sho Aoki, Fredrik Bengtsson, Amanda M Brown, Gang Chen, Timothy J Ebner, Philippe Isope, Henrik Jörntell, Elizabeth P Lackey, et al. Cerebellar modules and their role as operational cerebellar processing units. The Cerebellum, 17(5):654-622, 2018.
[68] Hojung Lee and Jong- Lee. Local critic training of deep neural networks. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. TEFE, 2019.
[69] Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. In International Conference on Machine Learning, pages 736-745. PMLR, 2020.
[70] Huiping Zhuang, Yi Wang, Qinglai Liu, and Zhiping Lin. Fully decoupled neural network learning using delayed gradients. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[71] Hirokazu Tanaka, Takahiro Ishikawa, Jongho Lee, and Shinji Kakei. The of Forward Model: A Review. Frontiers in Systems Neuroscience, 14:19, 2020. ISSN 1662-5137 doi: 3.338///nsss.2220.000 URL https:/ //wp:////wwu... www.rfrrrsiiinnnm wttp/rtiiee//....33e833383 00019.
a Locus
2020. http::/wwrrrrree.aoo.oooooaeo/./
[72] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[73] Kaiming He, Xiangyu Zhang, Shaoging Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[74] Mark J. Wagner, Tony Hyun Kim, Jonathan Kadmon, Nghia D. Nguyen. Surya Ganguli, Mark J. Schnitzer, and Liqun Luo. Shared (773))669-682..24 2019. ISSN 72. doi: 10. 6011106//.cel..20 20111.9..009 URI https://doi.onn org/ 1016/011jjjj felll//..elll.... cell.elll..... 2019.02000000 02...1..
Dynamics in the Execution and Learning of a Motor Task. Cell,
[75] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[76] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, BLEU: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 111-118. Association for Computational Linguisties, 2002.
14

---

[77] Chin- Yew Lin. (ROUGE): A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, pages 74-81, Barcelona. Spain, 2004. Association for Computational Linguistics. URL https://wws. aclweb. org/ anthology/W04-10133
[78] Michael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. Proceedings of the ninth workshop on statistical machine translation, pages 376-380, 2014.
[79] Ramakrishna Vedantam, Lawrence Zitmick, Devi Parikh. Cider: Consensus bassd image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566 4575, 2015.
[80] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European Conference on Computer Vision, pages 382-398. Springer, 2016.
15


Reviews:
[{"review_id": "tVg__DBpN4o", "reviewer": "Reviewer_pGCk", "summary": "The authors propose to interpret the function of the cerebellum through the recently developed theory of decoupled neural interfaces, referred to as DNI, allowing for online training of a deep neural network and not requiring locking. After introducing the DNI, the paper extensively explains the possible mapping between the model and observe cortico-cerebellar networks, leading to a clear explanation of how they conceive feedback to be implemented in the brain. Finally, the paper proposes to compare the results obtained with their architecture on a variety of tasks validating their model. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and proposes a rather uncommon way of addressing feedback in the ML/DL literature. It is highly appreciated that the paper clearly delineates structural properties, i.e., mapping onto known neural structures and the model of computation and learning.  \n\nHowever, an important weakness of the paper is that it does not try to understand the nature of the connections and neurotransmitters that would allow for such a model to be viable. The mapping only vaguely covers these aspects and weakens the paper. It would have been appreciated that either the paper itself a \"biologically plausible\" model and structure rather than forcefully mapping it onto the cerebellum and cortical networks. That does certainly not invalidates the point of the paper, but even with mitigating those arguments, the paper would remain strong while avoiding some oversimplification. \n\nOne could have also appreciated comparison to other models than itself in the experiments. It is not possible to know how well or poorly it performs compared to existing non-biologically plausible alternatives either SOTA or simply relevant to the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and possible research directions are nicely described. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Xwm42pQJM84", "reviewer": "Reviewer_cVWr", "summary": "This paper builds on past work that uses synthetic gradients (estimated by small networks) to decouple parts of a network so that they can run in parallel. The current paper proposes that the cerebellum may produce such synthetic gradients to deal with delayed feedback to recurrent networks. To illustrate how this could work, the authors model a cortico-cerebellar network as a LSTM (cortex) and feedforward gradient-estimating network (cerebellum). They show that the system can learn various tasks, including line and digit drawing and caption generation. The further show that disrupting the cerebellar input leads to something resembling ataxic behaviour due to cerebellar damage. ", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \nAs far as I know, this is a new idea. It combines the well-established neuroscience concept of the cerebellum as a shallow feedforward supervised network and a recent approach in deep learning. \n\nQuality: \nThe work seems to be technically sound and suitably thorough. \n\nI have a reservation related to the analogy between the synthetic gradient network and the cerebellum. Reversible cerebellar lesions not only affect learning, but induce ataxia in previously learned movements (e.g. Hore, J., & Flament, D. (1988). Changes in motor cortex neural discharge associated with the development of cerebellar limb ataxia. Journal of neurophysiology, 60(4), 1285-1302). This seems to conflict with Figure 3c, which shows that the model isn’t sensitive to cerebellar ablation after a certain point in learning. \n\nClarity: \nThe writing is very clear and the paper is well organized. Some methodological details could be moved to the main text, e.g. the network architecture and the number of time steps of the line-drawing movements. I don’t think the plant that carries out the movements was ever described. Is it a 2D point mass with force inputs? \n\nSignificance: \nAs the authors point out, if this idea turns out to be physiologically realistic, it will provide an important new perspective on the cerebellum. It could also point to the physiological details of the cerebellum as a source of inspiration for future machine learning work on decoupled neural interfaces. The paper gives some preliminary evidence of physiological relevance of the model, but much more would be needed to make a convincing case (more than would be expected in a single paper). So I think the significance is unclear at this point, but it is promising. \n\nMinor: \nI believe the cerebellum provides input mainly to frontal, cingulate, and a few parietal areas (Strick, P. L., Dum, R. P., & Fiez, J. A. (2009). Cerebellum and nonmotor function. Annual review of neuroscience, 32, 413-434.) Could the authors comment on whether this restriction relates to the proposed role of cerebellum? \n\nIn Figure 2d, my understanding is that the “cortical temporal feedback %” applies to both cRNN and ccRNN. So cRNN works better if there is a long enough feedback window, otherwise synthetic gradients are helpful. If this is right, I’d suggest expanding the caption slightly as I missed it the first time. \n\nLine 126: “our model suggests that the cerebellum should encode error signals, consistent with neural recordings showing cerebellar-encoding of kinematic errors during motor learning” I think the model suggests that the cerebellum should encode error gradients, which may only be consistent given additional assumptions. \n\nLine 226: “pairwise correlations between all the neural activity in the main cortical network and the neural activity of the cerebellar module (see SM).” I had to check the SM to understand this, but a slight rephrasing would make it clear in the main text. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The work has a minor societal impact. In the checklist, the authors say they have discussed limitations in sections 2.1.2 and 4, but I don’t see such a discussion. ", "review_text": "", "extra_scores": {"ethical_concerns": "The work does not raise ethical concerns.  ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "NQTeHJLORz", "reviewer": "Reviewer_YMU3", "summary": "In this work, the authors propose a neural architecture inspired by the anatomy of the cortico-cerebellar pathways in the brain.  The method relies on a previously proposed framework of decoupling neural interfaces. Here, the authors suggest that the cerebellar circuit calculates the future error of the cortical representation. It allows the implementation of Back Propagation Through Time without the phase-lock problem—where the network has to be wither in a feedforward of a feedback phase to calculate the local weight gradients. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- Considering the cerebellum as a circuit that calculates the expected error is a promising avenue that aligns with many current ideas about the cerebellum function.\n- The decoupling neural interfaces framework is a concrete model that could be used in the future to study cerebellar function.\n- The authors show the benefits of the architecture on various tasks that would (potentially) be associated with the cerebellum.\n\n**Concerns and weaknesses**\n- The authors show that the methods work well (or at all) only when there are intermediate inputs of external feedback. It seems to me that the role of the cerebellar layer here is to make sure the output is interpolated smoothly between the supervised feedback. It would also explain the higher error in the non-smooth outputs.\n- I found the details on the temporal feedback in RNNs (lines 96-108) hard to follow. In particular, the loss function for the temporal feedback. In here, delta-hat is the feedback from the cerebellar layer; why does the cerebellar activity also appear explicitly within the expression for delta-bar? This section should be revised and better explain the loss function being used; it is fundamental to this work and should not be left for a reference to make sense.\n- The feedforward-backward phases of training are only one problem of several problems of implementing  BP and BPTT in a biological realistic network. While the paper does not mention other substantial issues, I don't", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "IcXSZcC68j", "reviewer": "Reviewer_WvvH", "summary": "During a network update step all layers need to wait for the feedforward pass (or its temporal RNN analogue) to reach \"the end\" and only then the error is calculated, gradient computed, weights updates. So effectively most of the time the lower layers are just waiting for the feedback to trickle down. This is called \"feedback locking\" and is inefficient. The paper proposes to add a second network to the architecture that learn to predict intermediate layer gradient from their activations, allowing an immediate update. The overall architecture bears hypothesized similarity to the role of cerebellum in primate brain.\n\nI find the idea interesting and the execution faithful and detailed. My main concern is whether the chosen control experiment is doing a good job at eliminating confounding factors potentially responsible for the better performance of the proposed method (see below for detailed explanations and a proposal).\n\nI am reserving my final assessment till after authors' rebuttal.\n\n---- UPDATE ----\nAuthors' rebuttal has fully addressed my concerns and resolved them in a positive manner. I am now much more confident in the value of the contribution. Raising the score to \"8: Top 50% of accepted NeurIPS papers, clear accept\"", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "General: What is the supporting evidence for the brain not being afflicted by the \"feedback locking\" problem. Do we know for certain that there is such a mechanism in place in the brain? Or could it be the case that the brain offers no solution for the \"feedback locking\" problem and this locking is actually part of normal operation?\n\n88-89: Related to the question above. It would be ideal to see a reference to a in-vivo study where the up-stream portion of a certain brain \"circuit\" is being disabled so that we are certain the the signal is not reaching the higher level where the \"loss\" is computed (so that the credit assignment cannot happen). And then we feed some input to the lower portion of the \"circuit\" and confirm that is still learns.\n\nGeneral: If the cerebellum is believed to act as a producer of such a \"synthetic gradient\", doesn't this require the cerebellum to have direct connections to the part of the cortex where the \"synthetic gradient\" is (presumably) being used, say the motor cortex? But there is quite some distance, anatomically, between the motor cortex and cerebellum. Do such connections exist and if not, how it can still be responsible for carrying out the ascribed function?\n\nGeneral: There seems to be an issue of infinite regress - which brain structure will play the role of cerebellum for the cerebellum itself? Same applies to ccRNN - how do we avoid \"feedback locking\" when training the C network? I would imagine both the C and the Main network are being trained simultaneously so we cannot pre-train C in advance, can we?\n\n201: \"network needs to produce a straight line towards\" - is the output of the network at each time t an (x, y) tuple of coordinates for the next target on the line? Perhaps this could be explicitly stated here for clarity.\n\n209: Can we be certain that the positive effect of ccRNN here comes from the ability to avoid \"feedback locking\" and not because the C network is acting as a learning stabilizer / variance reducer, which then leads to more efficient learning? Here is a control experiment to consider: what if cRNN would also have a \"cerebellum\", but this time it would provide the learning signal only at the last time step? So instead of giving it directly to the RNN, we would pass it though the cerebellum network (but only at the last stage, preserving the \"feedback locking\" problem). If in this setup the resulting trajectories would still be more ataxious than ccRNN, then we would know that it is the precisely the ability to provide gradients and the intermediate steps is what is behind the improved performance, and not such factors as (a) the potential stabilizing effects of the C network, (b) the added overall system capacity (more weights), etc. What you imagine the outcome of such a control experiment would be? Could you run it in time for the rebuttal?\n\n226: \"pairwise\" refer here to the pairs of what? Nodes? So the correlation between the activity of each node in the \"cortical\" network with each node of the \"cerebellum\"? At each time step?\n\n252: Figure 4c: Error bars seem to be very small (compared to the ones on Figure 2c for examples). Given here the experiments was repeated only 3 times (compared to 10 before), one would expect longer error bars. Or was the results very consistent in this case? Why only 3 runs by the way, and not 10 as before?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do provide some disclaimer statements regarding the limitations of their model and its predictions. It would be curious however to also learn authors' opinion about the limitations in (a) model's actual correspondence with the biological counterparts, (b) \"failure\" modes where the this model would not be applicable and might even be detrimental (if such cases can be identified). Is it always a good idea to approximate the gradient and provide this information mid-update or could it lead to unintended consequences in both biological and artificial netoworks?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes that the previously published synthetic gradient approach, decoupled neural interfaces (Jaderberg et al. 2017), may serve as a model of the cortico-cerebellar learning. The reviewers found the proposed connection between DNI and cortico-cerebellar learning to be a promising lead for future biological investigation.  Since understanding cerebellar function and role in human cognition is a deep and important problem, the initial proposal of this possible connection to a recent AI approach is considered a valuable contribution.  While this connection is not validated in this work, the connection is considered in light of what is already generally known about the cerebellum, and the work explores the consequences of this comparison.

The results of the paper involve comparisons between a baseline RNN model which uses truncated BPTT and a ccRNN model that learns using the DNI mechanism.  While the DNI mechanism is not new, the tasks chosen are ones that are supposed to provide further illustrate the implications of the connection to cortico-cerebellar learning.

Personally, I found the submission to be creative in identifying the potential relationship between the existing ML model and the architecture of the brain.  I would emphasize that the ML portion is not new, so the contribution of this paper is specifically the proposal of how the DNI approach might serve as an analogy to cortico-cerebellar learning.  I initially found some details, especially around the presentation of the results, a bit unclear.  However, the authors and reviewers engaged during the discussion phase in a way that I believe will make the final version of the paper sufficiently clear.  Multiple reviewers raised their scores during the exchange.  I'm willing to endorse the reviewer consensus that this paper be accepted.

Decision: Accept (Poster)