A contrastive rule for meta-learning
Authors: Nicolas Zucchet
Simon Schug
Johannes von Oswald
Dominic Zhao
Joao Sacramento
Venue: NeurIPS 2022 Accept (2022)
Keywords: meta-learning
meta-plasticity
synaptic consolidation
biologically-plausible learning
equilibrium propagation
complementary learning systems

Abstract: Humans and other animals are capable of improving their learning performance as they solve related tasks from a given problem domain, to the point of being able to learn from extremely limited data. While synaptic plasticity is generically thought to underlie learning in the brain, the precise neural and synaptic mechanisms by which learning processes improve through experience are not well understood. Here, we present a general-purpose, biologically-plausible meta-learning rule which estimates gradients with respect to the parameters of an underlying learning algorithm by simply running it twice. Our rule may be understood as a generalization of contrastive Hebbian learning to meta-learning and notably, it neither requires computing second derivatives nor going backwards in time, two characteristic features of previous gradient-based methods that are hard to conceive in physical neural circuits. We demonstrate the generality of our rule by applying it to two distinct models: a complex synapse with internal states which consolidate task-shared information, and a dual-system architecture in which a primary network is rapidly modulated by another one to learn the specifics of each task. For both models, our meta-learning rule matches or outperforms reference algorithms on a wide range of benchmark problems, while only using information presumed to be locally available at neurons and synapses. We corroborate these findings with a theoretical analysis of the gradient estimation error incurred by our rule.

A contrastive rule for meta-learning
Nicolas Zucchet'
Simon Schug" Institute of Neuroinformatics University of Zürich & ETH Zürich sschug@ethz.co ch
Department of Computer Science
ETH Zürich nzucchet@inf www.f.ethz.ccm.m
Johannes von Oswald"
Dominic Zhao Institute of Neuroinformatics University of Zürich ETH Zürich dozhao@ethz.ch
Department of Computer Science
ETH Zürich voswaldj@ethz.c
João Sacramento
Institute of Neuroinformatics
University of Zürich & ETH Zürich rj rjoa0@ethz.c ch
Abstract
Humans and other animals are capable of improving their learning performance as they solve related tasks from a given problem domain, to the point of being able learn from extremely limited data. While synaptic plasticity is generically thought underlie learning in the brain, the precise neural and synaptic mechanisms by which learning processes improve through experience are not well understood. Here, present a general-purpose, biologically-plane plaausibe meta-learning rule which estimates gradients with respect to the parameters of an underlying learning algorithm by simply running twice. Our rule may be understood as a general- ization of contrastive Hebbian learning to meta-learning and notably, it neither requires computing second derivatives nor going backwards time, two charac- teristic features of previous gradient- based methods that are hard to conceive in physical neural circuits. We demonstrate the generality of our rule by applying it to two distinct models: complex synapse with internal states which consolidate task-shared information, and dual-system architecture which primary network is rapidly modulated by another one learn the specifics of each task. For both models, our meta-learning rule matches or outperforms reference algorithms on a wide range of benchmark problems, while only using information presumed to be locally available at neurons and synapses. We corroborate these findings with a theoretical analysis of the gradient estimation error incurred by our rule.
Introduction
The seminal study of Harlow [1] established that humans and non-human primates become better at learning when presented with a series of learning tasks which share a certain common structure. To achieve this, the brain must extract and encode whichever aspects common within a problem domain, in such way that future learning performance is improved. This capacity, which we refer meta-learning, confers great evolutionary advantage to organism another that must
"Equal contribution; arbitrary ordering. 'Code available https://github: wotm///oonssgg//cottrttttiii meta- learning
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

face new tasks starting from tabula rasa. The neural and synaptic basis of this higher-order form of learning is largely unknown and theories are notably scarce [2]. The present work focuses on developing one such theory.
Formally, we define learning as the optimization of data-dependent objective function with respect to learnable parameters, following the prevalent view in machine learning [3]. Meta-lcarning can be straightforwardly accommodated for in this framework by first specifying learning algorithm through a set of meta-parameters, and then measuring post-learning performance through meta-objective function [4-8]. Formulated as such, meta-learnng corresponds to a hierarchical optimization problem. where lower-level parameters are optimized to learn the specifics of each task, and meta perammerssssers are adapted over tasks to improve overall learning performance.
An essential question in this framework is how to optimize meta-parameterr, In current deep learning practice, meta-parameters are almost always learned by wwwprroppagatttottoog--eeeeeeeee an instance of
[9). While a number of biologically-plausiii designs [3, 10-13] have been developed for the standard error backpropagation algorithm for feedforward neural networks [14, 15], pagproopggaiiotttruu---eeeee
suffers from a number of issues which appear to be fundamentally difficult overcome in biological circuits. For example, when learning involves optimizing synaptic connection weights as it is presumed to be the case in the brain implementing backpropagattiontthounn carpiggtttioottuuuh--ho through-
would entail backtracking through a sequence of synaptic changes in reverse- timee--emeee order, while carrying out operations which would require knowledge of all synaptic weights be available at a single synapse. This clearly at odds with what is currently known about synaptic plasticity. Thus, calculating meta-parameter gradients by backpropagation is both computationally expensive, and hard to reconcile with biological constraints.
Here we present a meta-learning rule for adapting meta-parameters which does not exhibit such issues. Instead of backpropagating through a learning process, our rule estimates petaa-arrreete gradients by running the underlying learning algorithm twice: learning a task is followed by a second run to solve an augmented learning problem which includes the meta-objective. Our rule has number of appealing properties: (1) it runs forward in time, making the learning rule causal; (2) implementing it only requires temporarily buffering one intermediate state; (3) it does not evaluate second derivatives, thus avoiding accessing information that is non-local to a parameter; and (4) it approximates meta-gradients as accurately as needed. Furthermore, our rule is generically applicable and it can be used to learn any meta- meta-paramete which influences the meta-objective function.
The local and causal nature of our rule allows which slowly consolidate information over tasks in their internal hidden states or in their synaptic weights. We show through experiments that, when governed by our meta-learning rule, such slow adaptation processes result in improved learning performance in a variety of benchmark problems and network architectures, from deep convolutional to recurrent spiking neural networks, on both supervised and reinforcement learning paradigms. Moreover, we find that our meta-learnin learninn rule performs as well or better than reference methods, including backpropagation-trooe we provide a theoretical bound for its meta- grrid-grrdentt estimation error which is confirmed by our experimental findings. Thus, our results demonstrate that gradient- based meta-nearning is possible with local learning rules, and suggest ways by which slower synaptic processes in the brain optimize the performance of faster learning processes.
develop theory of meta-plastic synapses,
and
2 Background and problem setup
The goal of meta-learning improve the performance of a learning algorithm through experience. We begin by formalizing this goal as a mathematical optimization problem and outlining its solution with standard gradient-based methods. The approach we present below underlies a large body of work studying meta-learning in neural networks [e.g.. 7, 16-18). We also discuss why these standard methods may be deemed unsatisfactory as models of meta-leariing in the brain.
Problem setup. Formally, we wish to optimize the meta-parameters of an algorithm which learns to solve a given task r by changing the parameters ó of model. Each task is drawn from a distribution p(T) representing the problem domain and comes with an associated loss function T.learu (o, 0). which depends on some data Dlearn The goal of learning to minimize this loss while keeping the meta- mneaa-parameter. 0 fixed; we denote the outcome of learning task by ob,.. The subscript

---

0 in 0g.- is here to emphasize that the solution of task implicitly depends on the meta veeta-arameters 0 used during learning. Learning performance is then evaluated by measuring again a loss function L (Øo 0). defined on new evaluation data Deval from the same task. The meta-objective is this evaluation loss, averaged over tasks. Hence, we formalize meta-learning bilevel optimization problem, which can be compactly written as follows:
min Bxpp(r) [T, (0b.r:0) 0)] 8
s.t. 60.r E arg min I I. (ó, 0).
(1)
In this paper, we approach problem (1) with stochastic gradient descent, which uses meta-gradienl information to update reta-parameters after learning task (or minibatch of tasks) presented by the environment. For a given task T we thus need to compute the meta-gradies
d Loval (00.)).) 2
Verr
The implicit dependence of oG.r on the meta- pettaparrmettrs 0 complicates the computation of the meta-gradie meta-gradient: differentiating through the learning algorithm efficiently is central question in gradient-based meta- leaanne. We next review two major known ways of doing so.
A common strategy followed in previous work [cf. 19] is to replace the solution -6.- to learning task by the result 60,-, obtained after applying differ- entiable learning algorithm for 7 time steps, not necessarily until convergence. One advantage of this formulation is that the computational graph for 68,T,7 is explicitly available. Thus, backpropagation can be invoked to compute the meta-gradient Ve,r. yielding what we refer to as backpropagation- through-learning This approach is hardly biologically-plausiie
Review of halkppoppgattio-ttrooghlalaaaln
as it requires storing and revisiting the parameter trajectory {or ) backwards in time, from t T to O. Moreover, when the learning al algorithm which produces 60,-,7 is itself gradient-based, as it typically is in deep learning, differentiating through learning gives rise to second derivatives. These second-order terms involve cross-parameter dependencies that are difficult to resolve with local processes.
Review of implicit differentiation. An alternative line of methods [20-2]] approaches problem (1) through the implicit function theorem [24]. This theorem provides conditions under which the meta- gradient Vo,- is well-defined, while also providing formula for it. Over nackpropagation-throug learning, this approach has the advantages that it does not require storing parameter trajectories ( and that it is agnostic to which algorithm is used learn a task. However, the graddett formula provided by the implicit function theorem is difficult to evaluate directly for neural network models, as it includes the inverse learning loss Hessian. This makes it hard to design biologically- plausible meta-learning algorithms based directly on the implicit meta-gradient expression. We refer to Section S2 for more details and an expanded discussion on this class of meta- learnnen methods.
Contrastive meta-learning
Here present new meta-learnnng rule which is generically applicable to meta-learning problems of the form (I), Our rule is gradient-followingg, and therefore scalable to neural network problems involving high-dimensional perampea while being simpler conceive in biological neural circuits than the standard gradient-based methods reviewed in the previous section.
To derive meta-learning rule we first introduce auxiliary objective function which mixes the two levels of the bilevel optimization problem (1):
L, 3) = Ieerra (0,0) 0) + (c. 8).
(3)
We refer C, (g. as the augmented loss function. This auxiliary loss depends on new scalar parameter 3 E R, which we call the nudging strength. Positive values of 33 nudge learning towards the meta-objective associated with task т. Thus, we can define a family of auxiliary learning problems through the augmented loss L, by varying the nudging strength B away from zero. We denote the solutions to these auxiliary learning problems by
00, E arg min (. 0,
(4)
3

---

and we use 00. to distinguish approximate model parameters found in practice with some learning algorithm from the true minimizers 68,3,r- Note that for the special case of 0, we recover solution O of the original learning task defined by L 0).
Our contrastive meta-learning rule prescribes the following change to the meta-parameters 0 after encountering learning task T:
( ( O 3 00 (00,8,r, 0, (00,0..
1
OL. (ou (ōo.0.. 0.0)
(5)
0e
This rule contrasts information over two model parameter settings, Φ8,0.r and Φ9.8, may be understood as a generalization to meta-learning of a classical recurrent neural network learning algorithm known as contrastive Hebbian learning [25-29]. Intuitively, as we compute the solution to the augmented learning problem with 8 > 0, we nudge our learning algorithm towards a parameter setting 00.8.r that would have been better in terms of the meta-objective that we wish our algorithm had actually reached, without needing the meta-objective to influence the learning process.
Our rule implements meta-learning by gradient descent when the learning solutions d0.0.7 and 00,9.7 are exact and as 3 - 0. This important property can be shown by invoking the equilibrium propagation theorem [29, 30] discovered and proved by Scellier and Bengio; restate this result and present the technical conditions for applying it to meta-learning in Section SI. Critically, A estimates the aueta-gradics Ve.r using only partial derivative information and without ever directly calculating the total derivative in (2). Depending on the model, partial derivatives of the augmented loss C, may be easy to calculate analytically and implement, or they may require dedicated neural circuits for their evaluation; we return to this point in the next section.
We recall that the two points 60,0.7 and Φ0.3.r which appear in (5) respectively correspond to approximate solutions the original and the augmented learning problems. Thus, the information required to implement our rule can be collected causally by invoking the learning algorithm for second time, after the actual task has been learned, while buffering information across the two runs. In contrast to iikiproppgation-toooggllllaiiiie requires keeping single intermediate state in short-term memory. and entirely agnostic to which underlying learning algorithm is used. Moreover, as we will show in the theoretical results, its precision can be varied; the same rule can produce both coarse- and fine- mme-grained meta geaadggrd estimates as needed, by varying the amount of resources spent in learning and by controlling the nudging strength B.
this process runs forward in time, it only
4 Models
In the previous section, our contrastive meta-learning rule was presented in its general form. We now describe two concrete neural models that provide complementary views on how meta-learning could be conceived in the brain. We study the specific meta-learming rules arising from the application of the update (5) to each case and discuss their implementation with biological neural circuitry.
4.1 Synaptic consolidation as meta-learning
We first use our general contrastive meta-learning rule (5) to derive meta-plasticity rules for a complex synapse model which has been featured in prior meta-learning [22, 31] and continual learning [32, 33] work. Biological synapses are complex devices which comprise components that adapt at multiple time scales. Beyond changes induced by standard long-term potentiation and depression protocols lasting minutes to several hours, synapses exhibit activity-dependen plasticity at much longer time scales [34-36]. While previous work has focused characterizing memory retention in more realistic synapse models, here we study how such slow synaptic consolidation processes may support fast future learning through our contrastive meta-learning rule.
In the model we consider, besides a synaptic weight á which influences postsynaptic activity, each synapse has an internal consolidated state w towards which the weight is attracted whenever the synapse changes. We further allow the attraction strength to vary over synapses; reciprocal A-1 plays role similar to a learning rate. For this model the meta parameerssssers are therefore 0 = (),w),

---

We model the interaction between these three components through a quadratic function, which is added to the task-specific learning loss fearm (o))
1
T Jearn (1b 0) plearn (6)
+ Au - 6.)²
(6)
2
2=1
In machine learning terms, we regularize the learning loss with a quadratic regularizer. On the other hand, the evaluation loss function L (Ø) depends only on the synaptic weights & such that the meta-parameters only influence learning, not prediction.
The partial derivatives which appear in our contrastive meta-learning rule (5) can be analytically obtained for this synaptic model. A calculation yields the meta-plasticity rules
AW, À (00.9.- - 20.0.7) and 4x,-
2,3 (ô0.0.- w)² (0o, - w)²]
1
(7)
where all operations are carried out elementwise. Contrastive meta-learning thus offers a principled way to slowly (over learning tasks) consolidate information in the internal states of complex synapses to improve future learning performance. Critically, leads to meta-plasticity rules that are entirely local to a synapse and are independent of the method used to learn. Our meta-plasticity rules can thus be flexibly applied to improve the performance of any learning algorithm, including a host of learning rules, from precise neuron-specific error backpropagation circuits [37, 38] to stochastic perturbation reinforcement rules [39]. The only requirement our theory makes is that learning corresponds to the optimization of an objective.
4.2 Learning by top-down modulation
The second model that consider is inspired by the modulatory role that is attributed to top-down inputs from higher- to lower-order brain areas. Such modulatory inputs often feature in neural theories of attention and contextual processing [40-42]. Here, explore the possibility that they subserve fast learning of new tasks, We incorporate this insight into a simple meta-learning model, where learning a task 7 corresponds to finding the right pattern of task-specific modulation oū.- and meta-learning corresponds changing synaptic weights 0. Unlike in the complex synapse model presented in the previous section, here we interpret the task-specific parameters Φ as patterns of neural activity, not synaptic weights. This implies that, if meta-learning succeeds, it becomes possible to learn new tasks on the fast neural time scale without evoking synaptic plasticity.
More concretely, we take modulatory inputs multiplicative gain g and an adaptive threshold b per neuron, as done in previous work [43, 44]. Rapid (input-dependent) multiplicative and additive modulation of the sensitivity of the neural input-output response curve a (Ix) is typically observed in cortical neurons [45]. There exist a number of biophysical mechanisms which allow top-down inputs to modulate (xx [e.g., 46]. Assuming simple finear-threshole neuron model with weights 6, this yields the response o (x) = g0 x - b)+ to some input x, where (+)+ denotes the positive-part operation. In this model, there only few learnable parameters Φ (g. b). as they scale with the number of neurons and not with the number of synaptic connections.
We apply contrastive meta-learning to this model by changing synaptic weights 0 according to our rule (5). For this model, partial derivatives of the augmented loss function correspond to the usual derivatives with respect to model parameters that are routinely evaluated to learn deep neural networks; our rule simply asks to compute them twice. We therefore build upon existing theories of learning by packpropagation-of -f-rrrro in the brain and assume that some mechanism for neuron-specific spatial error backpropagation is available, for example via prediction error neural subpopulations [37] or dendritic error representations [11, 38, 47], or by invoking equilibrium propagation again [29].
5 Theoretical and experimental analyses
In the following, we theoretically analyze the approximation error incurred by our contrastive meta- learning rule before empirically testing it on a suite of meta-learning problems. The objective of our experiments is twofold. First, we aim to confirm our theoretical results and demonstrate the performance of contrastive meta-learning on standard machine learning benchmarks. Second, we want to illustrate the generality of our approach by applying to various supervised and reinforcement meta-learning problems well as more biologically realistic neuron and plasticity model.
5

---

5.1 Theoretical analysis of the meta-gradient approximation error
The contrastive meta-learning rule (5) only provides an approximation to the meta-gradient. This approximation can be improved by refining the two learning solutions ó0,0.r and 08,0,r through additional computation by using better learning algorithm, and by decreasing the nudging strength 3, prescribed by the equilibrium propagation theorem. In Theorem 1, we theoretically analyze how the mnea-gradientt estimate (5) benefits from such improvements (see Fig. 1A for visualization of the result, Section S3 for proof and empirical verification of our theoretical results). We find that the refinement of the learning solutions must be coupled to decrease in 3: too small / greatly detracts from the quality of the meta-ga graddintt estimate when the solutions are not improved, while better approximations are inefficient if B not decreased accordingly.
Theorem 1 (Informal), Let and ô be such that ||00,0.r 0 o0.0.-1 S ô and | ..r -b I 8. Then, under regularity and convexity assumptions, there exists a constant C such that
| Vo.rll SC (+ (1+ 8
B 3 1+
=: B))
5.2 Contrastive meta-lcarning is a high-performance meta-optimization algorithm
As first set of experiments, we study supervised meta-optimization problem based on the entire CIFAR-10 image dataset [48]. In these experiments the goal is to meta-learn set of hyperparameters ameta-parametery, such that generalization performance improves. This problem is a common testbed for assessing the ability of meta-learning algorithm to optimize given meta-objective [23]; it can be thought of as a limiting case of full meta-lcarning, as there are learnable meta- parameters, but only one task. As the meta-objective on a held-out dataset Deval, I, Teval (Ø) | ((a,y) [xx, y, Φ), where x is an image input and y its label. We equip a convolutional deep neural network with our synaptic model (6), meta- learning only the per-synapse regularization strength X, keeping w fixed at zero: T,learn (s) A) =
take the cross-entropy loss / evaluated
1 [ E( 1(z, Φ) + 1 E o,o7- We learn the weights & by stochastic gradient descent paired with backpropagation. Additional details and analyses may be found in Section 1..1.
We benchmark our meta-plasticity rule (7) against implicit gradient-based meta-learning methods, which are considered state-of-thh-art art for this type of problem [23] (see Section S2 a review). More concretely, recurrent back- propagation (RBP [49, 50]; also known as the Neumann series approximation [23, 51)) and the conjugate gradient method (CG) [21, 52] correspond to two different numerical schemes for calculating the meta-gradientt T1-T2 [53] is an approximate method which neglects compli- cated terms, thus introducing non-redueible bias in the meta- meta-gradin graddent estimate. Critically, unlike our contrastive meta-learning rule (CML), this method offers no control over the meta-gradient error.
Table 1: Meta-leaming per-synapse regular- ization strength meta-parameter (cff. Section 4.1) on CIFAR-10. Average accuracies (acc.) +
s.e.m. over 10 seeds.
Method
Evaluation acc. (%)
Test acc. (%)
T1-T2
64.77=0.40
62.57=:0.31
CG
57.65#1.51
57.51 #0.98
RBP
64 6..92+1.32
62.14 0.09
CML
74.43=00.5
66.94#0.25
No meta
60.06+00.7 #0.37
60.13 :t03333
TBPTL
73.17=0.27
65.35
We find that our meta-learning rule outperforms all three baseline implicit differentiation methods in terms of both evaluation-sel and actual generalization (test-set) performance, ef. Tab. 1. As a side result, we confirm the instability of CG in deep learning reported in ref. [51, 54]. We note that the hyperparameters of all four methods were independently and carefully set (cf. Section 14.1). These strong results on modern deep learning benchmark, involving stochastic approximate learning, demonstrate that contrastive meta-learning is scalable, highly effective meta-optimization algorithm. Moreover, Theorem 1 is in excellent qualitative agreement with our experiments, cf. Fig. 1.
To further contextualize our findings, we provide results for training the same network without meta-learning, where we performed a conventional hyperparameter search over scalar regularization strength hyperparameter shared by all synapses. This simple approach yields only a moderate evaluation and test accuracy.

---

A
Theoretical prediction
B
Empirical results
Bound grad. est error
10'
First phase approx (0)
5000
Validation loss
80
4000
phase steps
16"
10'
3000
10
60
2000
10 9
1000
10
40
10" 10 10"
10
10"
10"
Figure 1: (A) Visualization of the theoretical bound B on the meta-gradient estimation error from Theorem as a function of the nudging strength 3. Better approximations of the solutions (smaller 6) improve the quality of the meta-gradient, as they enable using smaller values B. (B) Confirmation of the qualitative findings of the theory on deep learning experiments. We show results for hyperparameter meta-learning problem, where per-synapse regularization strength is meta-learned (cf. Section 4.1) on CIFAR- 10 with rule (7). The validation loss is proxy for the quality of the gradient and the number of steps the first phase a proxy for por-ly log
As methods incur numerical errors when computing the meta-gradient, a comparison to using the analytical solution for the meta-gradient would be desirable. Since this is intractable in this case and running full paanopaggtiionttroug--e-eeee truncated backpropagation-thruuk
requires too much memory, we evaluate (TBPTL) with the maximal truncation window can fit on a single graphics processing unit (in our case 200 out 5000 steps). The resulting evaluation accuracy and test accuracy outperform other implicit gradient- baadd meta-learning methods but are still surpassed by our method.
5.3 Contrastive meta-learning enables visual few-shot learning
The ability to learn new object classes based on only a few examples a hallmark of human intelligence [55] and prime application of meta-learning We test whether our contrastive meta- learning rule is able to turn into a few-shot learner a standard visual system, a convolutional deep neural network learned by gradient descent and error backpropagation. Furthermore, we ask how our contrastive meta-learning rule fares against other gradient-based meta-learning algorithms which rely on backpropagation- through-learning and implicit differentiation to compute gradients. To that end, we focus on two widely-studied few-shot image classification problems based on minilmageNeet [56] and the Omniglot [57] datasets. To further facilitate comparisons, we reproduce exactly the experimental setup of ref. [18], which has been adopted in a large number of studies.
Briefly, during meta-learning N N-way K-shot tasks created on-the-fi fly by sampling N classes random from a fixed pool of classes, and then splitting the data into task-specific learning D (with K examples per class for learning) and evaluation Deval sets, used to define the corresponding loss functions I.learn and Loval, The meta-objective is then simply the task-averaged evaluation loss, measured after learning. The performance of the learning algorithm is tested on new tasks consisting of classes that were not seen during meta- neta-learning. We provide all experimental details in Section $4.2.
As reference methods, we compare against the well-known model-agnostic meta-learning (MAML)) algorithm [18],
Table 2: One-shot minilmageNet learn- ing. Averages over 5 seeds std.
meta- learn an initial set of weights, starting from which few gradient steps should succeed: this is conceptually similar to meta-learning the consolidated state of our complex synapses. We also include results obtained with its first- order approximation FOMAML (as well as a closely re- lated algorithm known as Reptile (58]), which, like the TI- T2 algorithm of the previous section, excludes all second- order terms from the meta-gradient estimate to simplify the update, at the expense of introducing bias. Finally,
which relies on backpropagation-ttrougee throgg---ttooouuueeeannn
Method
Test acc. (%)
MAML [18]
48.70 11.84
FOMAMI. [18]
48.07 +1.75
Synaptic
48.43
10.43
Modulatory
49.80 4:0.0
7

---

we compare to the implicit MAML (iMAML) algorithm [22], which corresponds exactly to meta- learning our consolidated synaptic state w. but with implicit differentiation methods.
When applied to the problem domain of minilmageNet. one-shot learning tasks, the performance of all meta-learning al- gorithms consider here is closely clustered together, cf. Tab. 2. In par- ticular, meta-learning the consolidated states of our complex synapses with implicit differentiation (iMAML) or our local update (7) leads to comparable per- formance. Interestingly, we further find that minilmageNet one-shot learning performance is significantly improved when using the modulatory model described in Section 4.2, despite the low dimensionality of the task-specific variable é. This is line with other results suggesting that highly efficient visual learning of new categories may be possible without necessarily engaging synaptic plasticity [43]. On Omniglot (see Section S4.2 for additional variants), the situation is comparable, except that on its 20- (0- 1-shot variant, the performance gap between first- and second-order methods widens. In line with our theory, our contrastive meta-learning rule performs close to (second-order implicit differentiation, showing that despite its simplicity and locality our rule is able to accurately estimate meta-gradients.
Table Omniglot character few-shot learning. Test set classification accuracy (%) averaged over 5 seeds + std.
Method
20-way 1-shot 20-way 5-shot
MAML [18]
95.8+0033
98,9
FOMAML (18)
89.4+0.5
97.9
Reptile [58]
89.43=0014
97.12=0.32
iMAML [22]
94.46-t0.42
98.69
CML (synaptic)
94.16+0.12
98.06=0.26
CML (modulatory)
94.24400.39
98.60=0.27
5.4 Contrastive meta-learning enables meta-plasticis in a recurrent spiking network
For the experiments described on the previous sections we used simple artificial neuron models and ofckpropagation-o-eerorr to learn. We now move closer biological neuron and plasticity model and consider meta-learning in a recurrently-connec neurons with plastic synapses. We study simple few-shot regression problem [18], where the aim is to quickly learn to approximate sinusoidal functions which differ in their phase and am- plitude (for additional details see Section S4.3). For each task, we measure the mean squared error on 10 samples for the learning loss and 10 samples for the evaluation loss. We imple- ment synaptic plasticity using the local e-prop rule [59] and
neural network of leaky integrate-and-fe
population of 100 Poisson neurons to encode inputs, see Fig. 2A. As our contrastive meta-learning rule (5) is agnostic to the specifies of the learning process, we can augment the model with our synaptic consolidation model and apply the meta-plasticity rules derived in (7), Fig. 2B illustrates how the learning process improves with increasing number of tasks encountered, eventually consolidating sinu- soidal prior that can be quickly adapted to the specifics of a task from few examples, cf. Fig. 2C.
We compare our method to standard base- line where updates are computed by back- propagating through the synaptic plasticity process (backpropagation-ttrouuu whroonntttooouueuuennng BPTL) using surrogate gradients to handle spiking nonlincarities [60] similar to previous work on spiking neuron meta-learning [61]. Since full BPTL requires reducing the num- ber of learning steps compared to our method due to memory constraints, we also include TBPTL with the same number of 500 learning steps and a truncation window of 100 steps. In both cases, we find competitive performance for our method, see Tab.
Table 4: Few-shot learning of sinusoidal functions with a recurrent spiking neural network. Avg. mean squared error (MSE) over 10 seeds + s.c.m.
Method
Validation MSE
Test MSE
BPTL + BPTT
0.17
0.41 10.10
BPTL + e-prop
0.52
0.72
TBPTL e-prop
0.27 10.07
0.50 20.11
CML c-prop 1.204
0.23 1.004
5.5 Contrastive meta-learning improves reward-basee learning
Finally, we demonstrate how contrastive meta-learning can be applied the challenging setting of reward-based learning, second nature to most animals. Reward-based learning clearly demonstrates hallmarks of meta-learning as animals are capable of flexibly remapping reward representations when task contingencies change [62, 63]. Inspired by this, we aim to meta-learn value function
8

---

A
B
Number of meta- meta-epochs
C
target conso prior
samples
200
400
fine tuned
Mean squared error
2.5
10"
rewro
-2.5
10
RSNN
2
10
5
Number training samples
Figure 2: (A) network of recurrently-ooneeee leaky-integrate and fire neurons is tasked with learning sinusoids on an input encoding of Poisson spike trains. Its prediction is the voltage of the output neuron averaged over time. (B) Learning performance from few examples measured as the mean squared error on evaluation examples during learning episode improves as more tasks are encountered over the course of meta-learning. (C) Meta-plasn veea--pasticcty encodes information on the consolidated synaptic component (dashed) which results in improved learning performance (purple), compared to a naive network learning from scratch (blue).
family of reward-based learning tasks that can be quickly adapted predict the expected reward of the actions available to the agent in a particular task.
Specifically. we consider the wheel bandit problem introduced by [64] with the meta-leamiing setup previously studied in refs. [65, 66]. On each task, an agent is presented with a sequence of context coordinates randomly drawn from unit circle for each of which it has choose among actions to receive a stochastic reward. Hidden to the agent, task-specific radius 5 tiles the context space into a low- and high-reward region depending which the optimal action to take changes (see Section S4.4).
The goal of meta-learning is to dis- cern the general structure of the low- and high-reward region across tasks whereas the goal of learning becomes to identify the task- task-specific radius 8 of the current task. During meta-learning we randomly sam- ple tasks ~ 2/(0, 1) and generate dataset by choosing actions ran- domly. Data from each task is split into training and evaluation data, ef-
Table 5: Cumulative regret on the wheel bandit problem for different 8. Values normalized by the cumulative regret of uniformly random agent. Avgs, over 50 seeds + s.c.m.
8
0.5
0.9
0.99
Neurall.neear [64]
0.95+0.02
4.65+0.18
49.63=2.11
MAML
0.45
1.02
15.21 11169
CML (synaptic)
0.40
0.82
12.27
CML (modulatory)
0.42
1.83+0.11
16.46=11880
fectively creating a sparse regression problem where only the outcome of a randomly chosen action can be observed for particular context. After meta-learning, we evaluate the cumulative regret obtained by an agent that chooses his actions greedily with respect to its predicted rewards and adapts its fast parameters on the observed context, action, reward triplets stored replay buffer.
We use both our synaptic consolidation and modulatory network models to meta-learn the value function using our contrastive rule. We compare learned baseline, NeuralLinear, from ref. [64], which performed among the best in their large-scale comparison. Tab. 5 shows the cumulative regret obtained different task parametrizations 5 in the online evaluation after meta-learning (extended table in Section S4.4). Meta-learning clearly improves upon the non-meta-learned baseline with both our models performing comparably to MAML. This improvement is more pronounced for tasks with larger within which it is more difficult to discover the hewarreward region.
models to MAML and the non-meta-
Discussion
We have presented general-purpont meta-learning rule which allows estimating meta-gradient from local information only, and we have demonstrated its versatility studying two neural models on a range of meta-learning problems. The competitive performance observed suggests that contrastive meta-

---

learning is worthy contender to ieglogicallyimplausiiii machine learning algorithms - especially for problems involving long learning trajectories, as demonstrated by the strong results on supervised meta-optimization. At its core, our method relies on contrasting the outcome of two different learning episodes. Despite its conceptual simplicity this requires complex synaptic machinery which is able to buffer these outcomes in way accessible to synaptic consolidation.
According to our top-down modulation model the goal of synaptic plasticity in primary brain areas is not to learn a specific task, in contrast to more traditional theories of learning. Instead, we postulate that the goal of synaptic plasticity is to make it possible learn any given task by modulating the sensitivity of primary-area neurons in a task-dependent manner. This view is consistent with the experimental findings of Fritz et al. [67]. who observed the rapid formation of task-dependent receptive fields in the primary auditory cortex of ferrets, as the animals learned several tasks, presumably due to changes in top-down signals originating in frontal cortex. Together with the strong results of the modulatory model in the challenging setting of visual one-shot learning and recent studies in continual learning problems [68-71] this shows the practical effectiveness of task- dependent modulation. Complementary to the interaction of the frontal cortex with primary cortical areas, the prefrontal cortex might similarly modulate the striatum during reward-based learning. Whereas classical dopamine- learning posits that reward prediction errors are used subcortically to learn the reward structure of a task, recent work has demonstrated that reward can similarly affect prefrontal representations to quickly infer the current task identity and switch the context provided to the striatum [72]. More broadly, viewing synaptic plasticity as meta-learning is also consistent with meta-reiinfrrcement learning system [73].
recent modeling work casting the prefrronaa cortex
Reflecting on how our meta-learning rule can be implemented in the brain, we conjecture that the hippocampal formation plays central role in coordinating the two phases as well as creating the augmented learning problem. First, some mechanism must signal that a switch from learning problem to augmented learning problem has occurred, corresponding to the sign switch in our rule (5). We argue that the hippocampus is well positioned for signaling such a switch to cortical synapses. A recent experimental study shows that the hippocampus is at least able to control cortical synaptic consolidation [74] but further evidence would be needed to support our hypothesis.
Second, we conjecture that the creation of the augmented learning problem at the heart of our meta- gradient estimation algorithm might itself critically rely on the hippocampus. In all our experiments, this second learning problem consisted simply of new data, presented to the learning algorithm to evaluate how well learning went. Transferring additional data into cortical networks, putatively during sleep and wakeful rest, fits well with the role that is classically attributed to the hippocampus in systems consolidation and complementary learning systems theories [75, 76]. We thus speculate that the hippocampus 'prescribes' additional learning problems to the cortex, which serve the purpose of testing its generalization performance. By showing that a second 'sleep' learning phase enables meta-learning with simple plasticity rules, our results lend further credit to complementary learning systems theory, as well as to the hypothesis that dreams have evolved to assist generalization [77].
Lastly, this view of the cortex as contrastive meta- meta-learning system aided by the hippocampus may also help elucidate how the brain learns from an endless, non-stationary stream of data. Current artificial neural networks notoriously struggle to strike a balance between learning new knowledge and retaining old one in such continual learning problems, particular when the data are not independent and identically distributed nor structured into clearly delineated tasks [78]. Interestingly, recent investigations have shown that meta-learning can greatly improve continual learning performance [79-83]. While details vary, the essence of these methods is to blend in past (replay) data with new data in a meta-objective function. This amounts to a different instantiation of our bilevel optimization problem (1), resulting in augmented learning problem in which past and present data are intermixed, for which the hippocampus would again appear to be ideally positioned.
Acknowledgments and Disclosure of Funding
This research was supported by an Ambizione grant (PZ00P3 1__118600 186027) from the Swiss National Science Foundation and an ETH Research Grant (ETH-23 21-1) awarded to João Sacramento. Johannes von Oswald is funded by the Swiss Data Science Center (J.v.O. P18-03). We thank Angelika Steger, Benjamin Scellier, Greg Wayne, Abhishek Banerjee, Blake A. Richards, Nicol Harper, Thomas Akam, Mohamady El-Gaby, Rafal Bogacz, Giacomo Indiveri, Jean-Pascal Pfister, Mark van Rossum, Maciej Wolezyk, Seijin Kobayashi and Alexander Meulemans for discussions and feedback, and Charlotte Frenkel for assistance in our implementation of e-prop.
10

---

References
[1] Harry F. Harlow. The formation of learning sets. Psychological Review, 56(1):51, 1949.
[2] Johanni Brea and Wulfram Gerstner. Does computational neuroscience need new synaptic learning paradigms? Current Opinion in Behavioral Sciences, 11:61-66, 2016.
[3] Blake A. Richards, Timothy P. Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J. Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W. Lindsay, Kenneth D. Miller, Richard Naud, Christopher C. Pack, Panayiota Poirazi, Pieter Roelfsema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna C. Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Konrad P. Kording. A deep learning framework for neuroscience. Nature Neuroscience, 22(11):
1761-1770, 2019.
[4] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-.. hook. Diploma thesis, Institut für Informatik, Technische Universität München, 1987.
[5] Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Technical report, Université de Montréal, Département d' Informatique de Recherche opérationnelle, 1990.
[6] David J. Chalmers. The evolution of learning: an experiment in genetic connectionism. In David S. Touretzky, Jeffrey L.. Elman, Terrence J. Sejnowski, and Geoffrey E. Hinton, editors, Connectionist Models, pages 81 81-90. Morgan Kaufmann, 1991.
[7] Sebastian Thrun and Lorien Pratt. Learning to learn. Springer US, 1998.
[8] Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning learn using gradient descent. In International Conference on Artificial Neural Networks, Lecture Notes in Computer Science. Springer. 2001.
[9] Paul J. Werbos. Backpropagation through time: what it does and how to do Proceedings of the IEEE, 35500); 1990.
[10] James C. R. Whittington and Rafal Bogacz. Theories of error back-propagation in the brain. Trends in Cognitive Sciences, 23(3)2235-250, 2019.
[11] Blake A. Richards and Timothy P. Lillicrap. Dendritic solutions to the credit assignment problem. Current Opinion Neurobiology, 54:28-366 -3,.,,,223 2019.
[12] Pieter R. Roelfsema and Anthony Holtmaat, Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience, 1963):166-1880 2018.
[13] Timothy Lillicrap, Adam Santoro, Luke Marris, Colin Akerman, and Geoffrey Hinton. Backpropagation and the brain. Nature Reviews Neuroscience, 21(6))335-346, 2020.
[14] Paul J. Werbos. Beyond regression: tools for prediction and analysis in the behavioral sciences. Ph.D. thesis, Harvard University, 1974.
[15] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back propaaappggg errors. Nature, 323(6088):533 536, 1986.
[16] Richard S. Sutton. Adapting bias by gradient descent: An incremental version of belta-bar-delt.
In National Conference on Artificial Intelligence, 1992.
[17] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, 2016.
[18] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, 2017.

---

[19] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey, Meta-learning in neural networks: survey. arXiv preprint WxXiv:2004.05439 2020.
[20] Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural Compulation, 12(8): 1889-1900, 2000.
[21] Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In International Conference Machine Learning. 2016.
[22] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, 2019.
[23] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, 2020.
[24] Asen L. Dontchev and R. Tyrrell Rockafellar. Implicit Functions and Solution Mappings. Springer, NY, 2009.
[25] Carsten Peterson and James R. Anderson. mean field theory learning algorithm for neural networks. Complex Systems, 1:995-1019, 1987.
[26] Javier R. Movellan. Contrastive Hebbian learning in the continuous Hopfield model. In Connectionist Models, pages 10-17, Elsevier, 1991.
[27] Pierre Baldi and Fernando Pineda. Contrastive learning and neural oscillations. Neural Computation, 3(4):526-545, 1991.
[28] Randall C. O'Reilly. Biologically plausible error-driven learning using local activation differ- ences: The generalized recirculation algorithm. Neural Computation, 8(5):895-953 1996.
[29] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 11, 2017.
[30] Benjamin Scellier. deep learning theory for neural networks grounded in physics. PhD Thesis, Université de Montréal, 2021.
[31] Yutian Chen, Abram L. Friesen, Feryal Behbahani, Arnau Doucet, David Budden, Matthew W. Hoffmann, and Nando de Freitas. Modular meta-learning with shrinkage. In Advances in Neural Information Processing Systems, 2020.
[32] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, 2017.
[33] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Ctrabsk--Barwingg Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 114(13)3321-3322 2017.
[34] Wickliffe C. Abraham. Metaplasticity: tuning synapses and networks for plasticity. Nature Reviews Neuroscience, 95):38773377 2008.
[35] Stefano Fusi, Patrick J. Drew, and Larry F. Abbott. Cascade models of synaptically stored memories. Neuron, 4594),599-611, 2005.
[36] Lorric Ziegler, Friedemann Zenke, David B. Kastner, and Wulfram Gerstner. Synaptic consol- idation: from synapses to behavioral modeling. Journal of Neuroscience, 35(3):1319-1334 1334. 2015.
[37] James C. R. Whittington and Rafal Bogacz. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity. Neural Computation, 29(5): 1229-11221126 2017.
12

---

[38] Alexandre Payeur, Jordan Guerguiev, Friedemann Zenke, Blake A. Richards, and Richard Naud. Burst-dependent synaptic plasticity can coordinate learning hierarchical circuits. Nature Neuroscience, 24(7):1010-1019, 2021.
[39] Xiaohui Xie and Sebastian Seung. Learning in neural networks by reinforcement of irregular spiking. Physical Review F, 69(4), 2004.
[40] Earl K. Miller and Jonathan D. Cohen. An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24(1):167-002, 2001.
[41] Rajeev V. Rikhye, Ralf D. Wimmer, and Michael M. Halassa. Toward an integrative theory of thalamic function. Annual Review of Neuroscience, 41(1):163-183, 2018.
[42] Heather K. Titley, Nicolas Brunel, and Christian Hansel. Toward a neurocentric view of learning. Neuron, 95(1):19-3,, 2017.
[43] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning In International Conference on Machine Learning, 2019.
[44] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: visual reasoning with general conditioning layer. In Proceedings of the Thirty- Secrty-eecond-- AAAI Conference on Artificial Intelligence, 2018.
[45] Katic A. Ferguson and Jessica A. Cardin. Mechanisms underlying gain modulation in the cortex. Nature Reviews Neuroscience, 21(2):80-92 2020.
[46] Matthew E. Larkum, Walter Senn, and Hans-R. Lüscher. Top-down dendritic input increases the gain of layer 5 pyramidal neurons. Cerebral Cortex, 14(10):0059-1070, 2004.
[47] João Sacramento, Rui P. Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcircuits approximate the backpropagation algorithm. In Advances in Neural Information Processing Systems, 2018.
[48] Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
[49] Luis B. Almeida. Backpropagation in perceptrons with feedback. In Rolf Eckmiller and Christoph v.d. Malsburg, editors, Neural Computers, pages 199-208. Springer Berlin Heidelberg. 1989.
[50] Fernando J. Pineda. Recurrent backpropagation and the dynamical approach to adaptive neural computation. Neural Computation, 1(2):161-172 1989.
[51] Renjie Liao, Yuwen Xiong, Ethan Fetaya, Lisa Zhang, KiJung Yoon, Xaq Pitkow, Raquel Urta- sun, and Richard Zemel. Reviving and improving recurrent back-propagation. In International Conference on Machine Learning, 2018.
[52] Chuan-sheng Foo, Chuong B. Do, and Andrew Y. Ng. Efficient multiple hyperparameter learning for log-lincs linelr models. In Advances in Neural Information Processing Systems, 2007.
[53] Jelena Luketina, Mathias Berglund, Klaus Greff, and Tapani Raiko. Scalable gradient-based based tuning of continuous regularization hyperparameters. In International Conference on Machine Learning, 2016.
[54] Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back- propagation for bilevel optimization. International Conference on Artificial Intelligence and Statistics, 2019.
[55] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 1332-6666):3332113333 2015.
[56) Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In Interna- tional Conference on Learning Representations, 2016.
13

---

[57] Brenden M. Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua B. Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2011.
[58] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprini arXiv: (803,02999 2018.
[59] Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Leg- enstein, and Wolfgang Maass. solution to the learning dilemma for recurrent networks of spiking neurons. Nature Communications, 11(1):3625, 2020.
[60] Emre O. Neftci, Hesham Mostafa, and Friedemann Zenke. Surrogate gradient learning in spiking neural networks: bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Processing Magazine, 36(6):51-66 636,, 2019.
[61] Guillaume Bellec, Darjan Salaj, Anand Subramoney, Robert Legenstein, and Wolfgang Maass. Long short-term memory and learning- -o-lennn in networks of spiking neurons. Advances in Neural Information Processing Systems, 2018.
[62] Abhishek Banerjee, Giuseppe Parente, Jasper Teutsch, Christopher Lewis, Fabian F. Voigt, and Fritjof Helmchen. Value-guided remapping of sensory cortex by lateral orbitofrontal cortex. Nature, 585(7824):245-2555 2020.
[63] Veronika Samborska, James Butler, Mark Walton, Timothy E.J. Behrens, and Thomas Akam. Complementary task representations in hippocampus and prefrontal cortex for generalising the structure of problems. bioRxiv, 2021.
[64] Carlos Riquelme, George Tucker, and Jasper Snoek. Deep Bayesian bandits showdown: an empirical comparison of Bayesian deep networks for Thompson sampling. In International Conference on Learning Representations, 2018.
[65] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 1rxiv:180..01622, 2018
[66] Sachin Ravi and Alex Beatson. Amortized Bayesian meta-learning. In International Conference on Learning Representations, 2019.
[67] Jonathan B. Fritz, Stephen V. David, Susanne Kadtke-Schullee Pingbo Yin, and Shihab A. Shamma. Adaptive, behaviorally gated, persistent encoding of taskrreevant auditory information in ferret frontal cortex. Nature Neuroscience, 1011:100,,, 2010.
[68] Nicolas Y. Masse, Gregory D. Grant, and David J. Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of National Academy of Sciences, 1 115(4))E10467 E10475, October 2018.
[69] Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
[70] Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetorrs. In International Conference Learning Representations, 2020.
[71] Ben Tsuda, Kay M. Tye, Hava T. Siegelmann, and Terrence J. Sejnowski. modeling framework for adaptive lifelong learning with transfer and savings through gating in the prefrontal cortex. Proceedings of the National Academy of Sciences, 117(47):9872229888) 2020.
[72] Marta Planco-Pozo, Thomas Akam, and Mark Walton. Dopamine reports reward prediction errors, but does not update policy, during inference-guided choice. preprint, Neuroscience, June 2021. URL http: htap::///iooooe bioxiii. wrgg//ookp//osi//// 10. 1010/22111... .0000.10.6666.. 25. 4499959
[73] Jane X. Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. Nature Neuroscience, 21eee.1161))0) 21(6))860-868, 2018.
14

---

[74] Guy Doron, Jiyun N. Shin, Naoya Takahashi, Moritz Drüke, Christina Bocklisch, Salina Skenderi, Lisa de Mont, Maria Toumazou, Julia Ledderose, Michael Brecht, Richard Naud, and Matthew E. Larkum. Perirhinal input to neocortical layer controls learning. Science, 370 (6523))eaaz33136 2020.
[75] James L. McClelland, Bruce L. McNaughton, and Randall C. O'Reilly. Why there are com- plementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102(3):
419 4999457, 1995.
[76] Dharshan Kumaran, Demis Hassabis, and James L. McClelland. What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends in Cognitive Sciences, 20(7):512-534, 2016.
[77] Erik Hoel. The overfitted brain: Dreams evolved to assist generalization. Patterns, 205):100244, 2021.
[78] Raia Hadsell, Dushyant Rao, Andrei A. Rusu, and Razvan Pascanu. Embracing change: continual learning in deep neural networks. Trends Cognitive Sciences, 24(12):1028-1044 2020.
[79] Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, 2019.
[80] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019.
[81] Gunshi Gupta, Karmesh Yadav, and Liam Paull. Look-ahead meta learning for continual learning. In Advances in Neural Information Processing Systems, 2020.
[82] Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. arXiv preprint arXiv:2002.0957/, 2020.
[83] Johannes Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where to learn: Gradient sparsity in meta and continual learning. In Advances in Neural Information Processing Systems, 2021.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We provide a theoretical analysis
of the meta-gradient approximation error incurred by our method and discuss yet-to-be confirmed biological requirements in section 6.
(c) Did you discuss any potential negative societal impacts of your work? [No] Our study aims developing a method and theory meta-learning in the brain. While advances in understanding meta-learning in the brain and improving meta-learning itself can lead to better learning systems and a lasting societal impact, we do not anticipate any immediate direct societal impact originating from our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] The full set of assumptions of all theoretical results is provided in the SM.
(b) Did you include complete proofs of all theoretical results? [Yes] All proofs for our theoretical results are included in the SM.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URI.)? [Yes] We include code and instructions to reproduce main results as part of our submission and will include an URL linking to the code upon publication.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Complete hyperparameters and training details are provided the SM.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] An estimate of the involved compute and description of the resources used can be found in the SM.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] License information may be found in the SM.
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We include code as part of our submission and will include an URL linking to the code upon publication.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "zQtKF3dgz2w", "reviewer": "Reviewer_rbQw", "summary": "The authors propose a contrastive meta-learning rule having several appealing properties: (1) biologically-plausibility, (2) locality, (3) causality, and (4) learning process agnostic. Specifically, the rule contrasts the outcome of two learning episodes with/without meta-objective to update meta-parameters. The proposed meta-learning rule estimates meta-gradient by simply running the learning algorithm twice compared with the existing meta-learning methods which require either second-order terms or inverse learning loss Hessian. The authors theoretically analyze the meta-gradient estimation error and confirm this through the experiment results. Also, the proposed meta-learning rule shows similar performance to the existing meta-learning methods on various tasks and architectures.", "questions": "- One of the main contributions of this paper is its simplicity. However, it is not clearly written whether such simplicity leads to better computational and memory efficiencies than the existing meta-learning rules. Therefore, it would be great to add a comparison table for computational and memory complexity.\n- In Fig. 2A, it looks like input and output are Poisson spike trains. However, since the input and output are continuous values, please redraw Fig. 2A.\n- In Section 5.5, the modulatory model performs much worse than the synaptic model and MAML as increases a task-specific radius while it outperforms the synaptic model on visual few-shot learning. However, CAVIA [1], which uses a modulatory approach as cited in the paper, outperforms MAML in the RL Cheetah experiments. So, it would be great to see the performance of CAVIA on the wheel bandit problem.\n\nReferences\\\n[1] Luisa Zintgraf et al. Fast context adaptation via meta-learning. In International Conference on Machine Learning, 2019.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The authors propose a novel method for meta-learning and applying this rule to the synaptic and modulatory models. The proposed method is simpler than the existing meta-learning methods including MAML and iMAML because it estimates the meta-gradients using only partial derivatives rather than second-order derivatives or inverse Hessian.\n\nQuality: The authors' claims are well supported in the paper. The authors theoretically prove the meta-gradient estimation error of the proposed meta-learning rule in terms of the nudging strength and demonstrate it via the experimental results. Also, the proposed meta-learning rule matches or outperforms the reference methods on various benchmarks even its simplicity. However, it is not clearly written whether the simplicity of the proposed rule leads to better computational and memory efficiencies.\n\nClarity: The paper is generally well-written and easy to follow.\n\nSignificance: The results in the paper are important because the proposed meta-learning rule is simple but its estimation error is theoretically proven and the performance matches the reference algorithms on various tasks. Therefore, I expect this paper is likely to be used in meta-learning literature.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mzcqQkW3gO4", "reviewer": "Reviewer_6moM", "summary": "The authors proposed a biologically inspired approach to meta-learning that avoids backpropagation through time. They change the optimization process and reorganize the loss function such that it mixes problem-specific and meta-specific parameters. ", "questions": "It would be helpful if the authors discuss how their approach would scale to more complex problems. ", "limitations": "I believe a discussion about the limitations of the experimental results would help to better understand the strengths and weaknesses of the proposed approach. For example, the authors could discuss the possibility of using this method on more challenging meta-learning RL tasks.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides a novel approach to meta-learning. The new approach is more biologically plausible than existing approaches: it is time local and does not require backpropagation through time. The paper is well written and the authors conducted an informative theoretical analysis of their approach. \n\nThe authors conducted experiments on several different types of datasets. They obtained better results than SOTA on meta-learning CIFAR-10, but comparable results on other datasets. While I think that the paper would be stronger with more experiments on more challenging datasets, I believe it is still a solid contribution. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "an8gJf69pNi", "reviewer": "Reviewer_1nAt", "summary": "The authors propose a novel meta-learning algorithm, effectively applying Equilibrium Propagation (EP, Scellier and Bengio, 2017) to the dynamical system defined by a learning algorithm minimizing a loss function (rather than the dynamical system defined by a model minimizing an energy function as in the original EP work). The authors define an augmented loss function (analogous to the \"total energy function\" of EP) consisting of a learning loss and an evaluation loss, weighted by a nudging strength. To estimate the total derivative with respect to the meta-parameters, they take the difference of the partial derivatives of the nudged and non-nudged loss, thus making a finite-difference approximation of the total derivative. Finally, the authors validate the algorithm with two specific meta-learning models (synaptic and neuronal) applied to four tasks (hyperparameter tuning, few-shot learning, regression, and reinforcement learning), demonstrating competitive performance, and provide a biological interpretation of their models. ", "questions": "- Are there cases where the meta-learning fails (e.g. does not converge to a viable solution for some/all of the learning tasks) due to the approximation? It would be instructive to include an example where CML entirely fails where another algorithm succeeds and vice versa (if such examples exist). \n- In section 5.2, the authors compare two methods (RBP, CG) which incur numerical errors in calculating the meta-gradient with CML's approximation. Is it possible to include an \"upper-bound\" result using the exact meta-gradient (computed with BPTL, or analytically as in section S3.6)?\n- In section 5.3, MAML perhaps plays the role of BPTL as an \"upper bound\" for the performance. However, it is difficult to compare these side-by-side as it is unclear whether the performance differences result from a superior meta-learning algorithm, or a superior learning algorithm, or differences in architectures (number of tuneable parameters, number of tuneable meta-parameters).\n- In section 5.4, can you clarify how the inputs are presented to the network, and how the outputs are read out? (e.g. for a given sinusoid amplitude/phase, is x_i a spike with probability p_i? Is y_k averaged across all 20 time steps? Is there only one y_k?)\n- In section 5.4, it is surprising that the BPTT+eprop performance is much worse than CML+eprop. If CML is an approximation of the exact meta-gradient from BPTT, why isn't BPTT strictly better?\n- Potential reference to include for section 4.2 or discussion: Titley et al., Toward a neurocentric view of learning, Neuron 2017\n", "limitations": "Technical limitations partially addressed through analysis of estimation error (see also Questions). Societal impact not discussed (but does not seem necessary for this work).", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality. The paper is an interesting application of Equilibrium Propagation to meta-learning, suggesting a biological implementation/approximation of backpropagation-through-learning (BPTL), analogous to how EP proposes a biological implementation of backpropagation-through-time. Although the contrastive rule itself is not novel, it is used in a different setting and its effectiveness there is thoroughly demonstrated. Furthermore, the authors extend the original theoretical analysis to include the approximation error incurred from the finite-difference step (i.e. \\beta) as well as from non-convergence to the fixed point. \n\nQuality. Overall, the paper is very thorough and rigorous, both theoretically with numerical validation of the theoretical results, and empirically in comparison to previous meta-learning benchmarks. A potentiall weakness is that for some of the empirical results on the various meta-learning tasks, it is unclear how to make a fair comparison to other methods (see Questions), equalizing architecture, computational resources, etc. \n\nClarity. Paper is very clearly written and well-organized. \n\nSignificance. From a machine learning perspective, this work provides a useful meta-learning approach which avoids storing the computational graph of the learning algorithm, thus using additional memory proportional to the number of meta-parameters (rather than the number of learning steps). The time complexity remains the same as BPTL. From a neurobiological perspective, it is an important step towards a biologically plausible meta-learning algorithm. Furthermore, the success of the modulatory CML algorithm is particularly interesting as it suggests a much more important role for learning via neuronal (rather than synaptic) changes than is typically considered. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KE9GJ9uqcni", "reviewer": "Reviewer_YWLg", "summary": "Paper addresses the problem of performing meta-learning when facing different tasks by introducing a general-purpose rule of contrastive kind that aims on biological plausibility. The main idea of contrastive meta-learning here is to run the underlying algorithm twice: once by learning the original presented task  and once to learn what the authors call \"augmented\" version of the task that includes the meta-objective. From this perspective, authors see their approach as generalization of classical Hebbian contrastive learning to meta-learning frame. When arguing for biological plausibility, authors state that \n1. the rule runs forward in time, avoiding backtracking through synaptic modifications in reverse time order,\n2. the rule does not evaluate second derivatives, avoiding using information non-local to an updated parameter\n3. implementing the rule requires only temporarily storing one intermediate state\n\nAuthors also derive an expression for estimating meta-gradient approximation error that arises by applying the rule, and state that meta-gradients can be consequently approximated as accurately as need, by investing more resources spent in learning and by controlling the nudging strength $\\beta$ that controls contribution of the meta-objective term in the total loss.\n\nTo show concrete implementations of their rule, the authors show two distinct versions using different meta-parameters. One is based on the complex synapse idea, where the synapse is allowed to have further internal variables apart from the effective synaptic weight. The other uses notion of fast top-down modulation, which affects two additional meta-parameters per neuron - its adaptive multiplicative gain $g$ and adaptive threshold $b$. \n\nThe authors then benchmark their proposed contrastive meta-learning algorithm using the two proposed implementations against alternatives using a number of tasks: supervised meta-learning on CIFAR-10; supervised few-shot meta-learning on miniImageNet and Omniglot; supervised meta-learning in a recurrent spiking network using sinusiodal functions dataset; and meta-learning in reward-based setting using wheel bandit task.\n\nOn the basis of the benchmarks, authors state that their contrastive meta-learning rule shows comparable of better performance than the alternative counterparts. Authors conclude that their biologically plausible rule has thus competitive or better performance than meta-learning methods that have to rely on non-local information. Finally, in the discussion section, authors elaborate on how the two phases of learning and the augmented version of the tasks that are required by their rule may be instantiated in the brain networks, taking hippocampus as possible candidate structure responsible for coordinating this learning form.", "questions": "1. It seems that the rule relies on knoweldge of the task identity $\\tau$. It would be good to discuss how learning algorithm may obtain this knoweldge in biological plausible way and still remain local.\n2. Does the training assume i.i.d shuffled data? If yes, how is that executed in biological plausible way? If no, what makes the algorithm perform continual learning?\n3. Why not combining both complex synapse and top-down modulation implementation in one rule? It seems both mechanisms are co-existing in the brain.\n4. What about benchmarking the algorithm on more realistic multi-task datasets? For instance, the standard ATARI game suit or MetaDataset (https://github.com/google-research/meta-dataset) would give a proper multi-task setting and allow for more generalizable comparision to other approaches.\n5. In the discussion, you point to hippocampus as a \"control\" structure that organizes contrastive learning - creating augmented task version and coordinating original and augmented task learning. It would be good to give a more grounded explanation - why necessary hippocampus (and not another structure, say basal ganglia or pre-frontal cortex, or insula, that also have access to most of cortical networks) and how it may subserve these functions? How sleep related activity may be relevant here, given that there will be a large time lag between learning episodes and interference with all possible tasks that happened during this period of time?\n6. Paper mentions one intermediate state that has to be temporary buffered after original task learning was executed, until augmented task learning has been performed. Is this state local to the synapses? Where is it in the update equations (is it $\\phi_{\\theta,0,\\tau}$)? It would be good to make this explicit for the reader. What is the biological plausible way to perform this short-term storage?\n7. It is stated that meta-gradient approximation error can be made arbitrary small by either running further learning or decreasing nudging factor $\\beta$. What may be a biologically plausible way to tune the nudging factor?\n8. It is mentioned that in case of implementing the rule by top-down modulation, the partial derivatives of the augmented loss function will be computed by automatic differentiation using standard backprop. To rescue biological plausibility, further mechanisms are invoked to replace backprop : prediction error neural subpopulations, dendritic error representations, or usage of equilibrium propagation. Would the same mechanisms then act for replacing backpropagation in general also for the original task learning, or is the idea here that those will be acting only for learning on augmented task?\n9. In general, how would usual approaches to replace backpropagation with local rules (again, for instance equilibrium propagation) relate to the proposed contrastive rule - are they complementary, or belong to the same class of methods, only applied for meta-learning in this case?\n", "limitations": "One limitation of the study that would be good to mention explicitly is the very small scale of conducted experiments (both network size and data size wise), compared to current state-of-the-art in deep learning.\n\nAuthors emphasize strongly benefits of top-down modulation based rule implementation. I think it would be good to cite corresponding classical works (eg Schmidhuber in 90s, Fast Weight Programmers, see for example discussion in Going Beyond Linear Transformers with Recurrent Fast Weight Programmers, https://proceedings.neurips.cc/paper/2021/hash/3f9e3767ef3b10a0de4c256d7ef9805d-Abstract.html ) that deal with fast slow learners, describing the same way of fast weight modulation by a slowly learning network (dual architecture as mentioned by authors), as alternative to slow weight change for task learning.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper introduces a novel take on biologically plausible meta-learning formulating a rule that attempts to use only locally available information for parameter updates. It uses the idea of contrastive learning in novel context by proposing to contrast between two different learning episodes based on the same task. I think this is an interesting idea to elaborate on in general - contrasting on the level of task instead of usual notion of contrasting on the level of data samples within a task. Authors motivate their rule in clear and conscise manner, emphasizing the point of biological plausibility, resulting locality and the differences to conventional meta-learning approaches. While there is some amount of work being done on deriving local plasticity rules that may replace backprop in standard learning, to my knowledge it is still rare to treat the same problem in meta-learning frame, and therefore significant for the community. Theoretical derivation of meta-gradient approximation error and the notion that approximation can be in principle made arbitrary accurate adds further value.\n\n## Weaknesses\nWhile the contrastive meta-learning rule aims on handling a setting with multiple various tasks streaming in, it is not clear for me after reading the manuscript how such a multi task setting should be handled in biologically plausible manner, which is one of central claims of the paper. It seems that rule cannot work when task do not have clear boundaries and needs some mechanism that identifies a task while it is learned. Further it is not clear what happens if there are concurrent tasks present at the same time. \n\nAnother point is handling of catastrophical forgetting and non-shuffled data: the authors do not elaborate whether their rule works under i.i.d assumption with respect to the tasks or whether it can also handle non-shuffled data (which would be another pre-requisite for biological plausible setting).\n\nIn the benchmarking, there is in my opinion lack of scenarios that test meta-learning in proper multi-task frame, where tasks have indeed different structure. So, meta-learning on CIFAR-10 is actually dealing with one task, while miniImageNet and Omniglot are used in rather classical few-shot scenarios. where the task is again the same across episodes. Other benchmarking tasks - sinusoidal function regression and wheel bandit - can be arguably considered as toy tasks, and it is not clear whether benchmarking on those can give us generic statements about the power of the algorithm. \n\nMy impression is also that networks used to benchmark the algorithm are very small compared to even small scale state-of-the-art. Recently derived scaling laws (e.g Scaling Laws for Neural Language Models, https://arxiv.org/abs/2001.08361) were pointing to the fact that increasing model and data scale during training improves generalization and transfer - I therefore wonder, whether testing algorithms that aim on strong generalization across tasks with very small scale networks and data may lead to distorted picture, where a more sophisticated algorithm may be superior on small scale, but substantially underperform against simpler algorithms on larger scale.\n\nFinally, one major unclear point to me is what authors attempt to elaborate on in discussion: how to organize and coordinate the contrastive learning which requires creation of the augmented version of the original task and careful execution of both original and augmented task learning in proximity to each other. I think authors attempt to bring in hippocampus as responsible structure falls short of explanation how it can manage to do those tasks - it feels to me like postulating a homunculus in form of hippocampus that has to figure out how to drive contrastive learning for all other brain networks involved. The authors should present at least some coarse explanation why they think that specifically hippocampus (and not, say, basal ganglia, prefrontal cortex, etc) is the candidate structure and how it may fulfull the required functions of creating augmented task and executing learning of both in coordinated way. Referring to \"transfer\" of \"additional data\" from hippocampus to cortex during sleep or resting state only adds to confusion here, in my opinion, as it introduces a very long time lag between original and augmented tasks and additional requirenment of specificity, where hippocampus should exactly to transfer data from a certain task to, and whether the target that should receive this data was not already exposed to data from various other tasks, which is not clear to me how to resolve.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept