Hebbian and Gradient-based Plasticity Enables Robust Memory and Rapid Learning in RNNs
Authors: Yu Duan
Zhongfan Jia
Qian Li
Yi Zhong
Kaisheng Ma
Venue: ICLR 2023 poster (2023)
Keywords: synaptic plasticity
meta-learning
Hebbian learning
few-shot learning
recurrent neural networks

Abstract: Rapidly learning from ongoing experiences and remembering past events with a flexible memory system are two core capacities of biological intelligence. While the underlying neural mechanisms are not fully understood, various evidence supports that synaptic plasticity plays a critical role in memory formation and fast learning. Inspired by these results, we equip Recurrent Neural Networks (RNNs) with plasticity rules to enable them to adapt their parameters according to ongoing experiences. In addition to the traditional local Hebbian plasticity, we propose a global, gradient-based plasticity rule, which allows the model to evolve towards its self-determined target. Our models show promising results on sequential and associative memory tasks, illustrating their ability to robustly form and retain memories. In the meantime, these models can cope with many challenging few-shot learning problems. Comparing different plasticity rules under the same framework shows that Hebbian plasticity is well-suited for several memory and associative learning tasks; however, it is outperformed by gradient-based plasticity on few-shot regression tasks which require the model to infer the underlying mapping.

Published as a conference paper at ICLR 2023
HEBBIAN AND GRADIENT-BASED PLASTICITY EN- ABLES ROBUST MEMORY AND RAPID LEARNING IN RNNS
Yu Duan, Zhongfan Jia, Qian Li, Yi Zhong", Kaisheng Ma" Tsinghua University, Beijing, China [duany19, j2f20)emails. wai.iiuustinguuuu.. edu. on (1iqian8, zhongyithu, etachennn@ttiigggini edu.cn
ABSTRACT
Rapidly learning from ongoing experiences and remembering past events with a flexible memory system are two core capacities of biological intelligence. While the underlying neural mechanisms are not fully understood, various evidence sup- ports that synaptic plasticity plays critical role in memory formation and fast learning. Inspired by these results, we equip Recurrent Neural Networks (RNNs) with plasticity rules to enable them adapt their parameters according ongoing experiences. In addition to the traditional local Hebbian plasticity, we propose a global, gradient- based plasticity rule, which allows the model to evolve towards its self-determined target. Our models show promising results on sequential and as- sociative memory tasks, illustrating their ability robustly form and retain mem- ories. In the meantime, these models can cope with many challenging few-shot learning problems. Comparing different plasticity rules under the same framework shows that Hebbian plasticity is well-suited for several memory and associative learning tasks; however, it is outperformed by gradient- based plasticity on few- shot regression tasks which require the model to infer the underlying mapping. Code is available at pttttps: /ttp:::://gttiie /ttpttgtttttt////////////t com/ www.yyuvenuaan...ocmomoommm PlasticNNS
INTRODUCTION
Biological neural networks can dynamically adjust their synaptic weights when faced with various
real-world tasks. The ability of synapses to change their strength over time is called synaptic plas- ticity, a critical mechanism that underlies animals' memory and learning {Abbott & Regehr 2004]] Stuchlik 2014 Abraham et al., 2019 Magee & Grienberger 2020). For example, synaptic plastic- ity is essential for memory formation and retrieval in the hippocampus (Martin et al.) 22000; Neves al. 2008 Rioult-Pedotti et al., 2000, Kim & Cho 2017, Nabavi et al.] [2014] Nakazawa Furthermore, recent results show that some forms of synaptic plasticity could be induced
al.
2004
within seconds, enabling animals form memory quickly and do one-shot learning (Bittner al.
2017
Magee & Grienberger 2020, Milstein et al. 2021).
To test whether plasticity rules could also aid the memory performance and few-shot learning ability in artificial models, we incorporate plasticity rules into Recurrent Neural Networks (RNNs). These plastic RNNs work like the vanilla ones, except that learned plasticity rule would update network weights according to ongoing experiences at each time step. Historically, Hebb's rule is a classic model for long-term synaptic plasticity; states that synapse is strengthened when there pos- itive correlation between the pre- and post-synaptic activity (Hebb) 1949). Several recent papers utilize generalized versions of Hebb's rule and apply to Artificial Neural Networks (ANNs) in different settings (Miconi et al.) (2088) Najarro & Risi 2020 Limbacher & Legenstein) 2020 TTyul-
mankov et al., 2022; Rodriguez et al. 2022). With redesigned framework, we apply RNNs with range of memory and few-shot learning tasks. Consistent
neuromodulated Hebbian plasticity
with the understanding in neuroscience (Magee & Grienberger) 2020; MMrtin et al., 2000 Neves
et al. 2008), find these plastic RNNs excel in memory and few-shot learning tasks.
"Corresponding authors

---

Published as a conference paper at ICLR 2023
Despite being simple and elegant, classical Hebbian plasticity comes with limitations. In multi-layer networks, the lack of feedback signals to previous layers could impede networks' ability to configure their weights in fine-graine manner and evolve to the desired target (Magee & Grienberger [220)] Marblestone et al. 2016). In recent years, some authors argue that other forms of plasticity rules in the brain could produce similar effects the back- prokk-ppppggatonnn algorithm, although the underlying mechanisms are probably different (Sacramento et al., 2018 Whittington & Bogacz] 2019, Roelf- sema & Holtmaat 2018). Inspired by these results, we attempt to model the synaptic plasticity in RNNs as reeff-eeerateddee gradient updates: at each time step, the RNN updates its parameters with a self-determined target. Allowing the RNN to generate and evolve to a customized target enables the RNN to configure its weights in flexible and coordinated fashion. Like Hebb's rule, the proposed gradient-based plasticity rule task-agnostic. It operates in an unsupervised fashion, allowing us to compare these two plasticity rules under the same framework.
In machine learning, learning a plasticity rule is one of the many meta-learning approaches (Schmid- huber et al.) 1997 Bengio et al., 2013). Although diverse collection of meta-learning methods have been proposed over the years (Huisman et al. 202T). these meta-learning methods are typically built upon specific assumptions on the task structure (e.g., assume the supervising signals explicitly given; see Sec. 2 for more detailed discussion). They thus could not be applied to arbitrary learning problems. In contrast, in our networks, the evolving direction of network parameters dWW /dl solely depends on the current network state, i.e., current network parameters and the activity of neurons. Since the designed plasticity rules do not rely on task- task-specific information (e.g., designated loss function and labels), they could be naturally applied to any learning problems as long as the input formulated as time series. Therefore, modeling biological plasticity rules also allows us build more general meta-leamers.
Our contribution can be summarized as follows. Based on previous work (Miconi et al. 2019p. we formulate a framework that allows us to incorporate different plasticity rules into RNNs. In addi- tion to the local Hebbian plasticity, we propose a novel gradient-based plasticity rule that allows the model to evolve towards self-determined targets. We show that both plasticity rules improve memory performance and enable rapid learning, suggesting that ANNs could benefit from synaptic plasticity similarly to animals. On the other hand, as computational models simulating biological plasticity, our models give insights into the roles of different forms of plasticity in animals' intel- ligent behaviors. We find that Hebbian plasticity is well-suited for many memory and associative learning tasks. However, the gradient- which requires the model to infer the underlying mapping instead of learning direct associations.
plasticity works better in the few-shot regression task,
2 RELATED WORK
Meta-Learning. Meta-learning, or "learning to learn", is an evolving field in ML that aims to build models that can learn from their ongoing experiences (Schmidhuber et al. 1997 Bengio et al.) 2013 A surprisingly diverse set of meta-learning approaches have been proposed recent years
Hospedales et al., 2021 Finn et al.) 2017 [antoro et al., 2016; Mishra et al. 2018 Lee et al.) 2019). In particular, one line of work proposes to meta-llarr learning rule capable of configuring network weights to adapt to different learning problems. This idea could be implemented by training an optimizer for gradient descent (Andrychowiez et al.) 2016] Ravi & Larochelle (017), training a Hypemnework that generates the weights of another network (Ha et al., 2017). or meta-learning plasticity rule which allows RNNs to modify its parameters at each time step (Miconi et al., 2019
Ba et al. 2016 Miconi et al. 2018). Our method belongs to the last category.
Compared to other meta-learning approaches, training plastic RNNs has some unique advantages. Plastic RNNs general meta-learners that could learn from any sequential input. In contrast, most meta-Icarning methods cannot deal with arbitrary learning problems due to their assumptions about task formulation. For example, methods that utilize gradient descent in the inner loop (e.g., MAML (Finn et al. 2017)) LSTM meta-learner (Ravi & Larochelle 2017) and GD2 (Andrychowicz et al. 2016)) typically assume that there exist explicit supervising signals (c.g.. ground truth) and a loss function that is used to update the base learner. However, such information is often implicit in the real world (e.g., when humans do few-shot learning from natural languages (Brown et al., (2020). In
contrast, plastic RNNs are task-agnostic: they can adapt their weights in unsupervised manner,
2

---

Published as a conference paper at ICLR 2023
and only a meta-objective is required for meta-training. Besides, the idea of evolving plasticity rules derives from animals, who are still the best meta-learners we have known so far.
Another line of work that is closely related to our gradient-based plasticity rule is meta-learning a loss function. This idea has been applied in reinforcement learning (Houthooft et al.) 2018 Oh et al. 2020 Kirsch et al. 2020) and supervised learning (Baik et al., 2021: Bechtle et al. 2021)_ However, as discussed above, these methods still depend on supervising signals and are thus less general compared to our methods. Moreover, our internal loss generation process is more flexibly determined by ongoing experience. The learning rule in the inner loop is also much more flexible than the usual gradient descent, as each connection has its own learning rate.
Synaptic Plasticity in ANNs. Previous work has incorporated synaptic plasticity into ANNs in different settings. For example, Hebbian networks can be explicitly used as storage of associative memories for ANNs (Limbacher & Legenstein 2020 SShhaag et al. 2021b). In addition, Hebb's ss rule alone can evolve random networks do simple reinforcement learning tasks (Najarro & Risi, 2020). |Miconi et (2.19) 2018) and [Ba et al. (2016) apply generalized Hebbian plasticity to RNNs and find plasticity helpful on tasks including associative learning, pattern memorization, and some simple reinforcement learning tasks. In Differentiable Plasticity (Miconi et al. 2018), the temporal moving average of outer products of pre- and post-synaptic activities is used as the plastic component of network weights. Miconi et al. (2019) extend this method by adding global neuromodulation. A recent paper further extends Hebbian plasticity with short-term dynamics (Rodriguez et al.) (022). Beyond Hebbian plasticity, some recent works explore other ways capture plasticity in RNNs. For example, some authors use Fast Weight Programmers to update part of the network with key-value mechanism (Schlag al. 2021a Irie al. 2021). Another line of work on key-value memory networks is also related to synaptic plasticity, where methods including gradient descent (Bartunov)
al. 2020 Munkhdalai et al. 2019) and three-factor plasticity rules (Tyulmankov al. 2021 1 are
proposed to update the memory network.
3 METHOD
3.1 MODEL FRAMEWORK
Following previous work on Hebbian plasticity (Miconi et al. 2018 2019 Tyulmankov al. 2022 Rodriguez et al.] 2022). we assume the weights for plastic layers to be the sum of static part w and plastic part We initialize the plastic part as 0 at the beginning of trial and update the plastic part throughout the trial. We adopt general architecture of RNNs as shown in Figure (left). Both the RNN and the last linear layer are plastic. The encoder is plastic linear layer in most tasks; the only exception is the one-shot image classification task, in which case the encoder is a non-plastic Convolutional Neural Network (CNN). The model output o, is the concatenation of three parts: scalar Tr. which modulates global plasticity by controlling how fast parameters change; a vector yr representing the model prediction; and an additional vector y, which allows more flexible control of weights for networks with gradient-based plasticity, as described in more detail in Sec. 3.3.
We summarize the general framework for training plastic RNNs in Algorithm In the inner loop, i... each time step of RNN, the network Iearns from ongoing experiences and adjusts its weights accordingly. In the outer loop, network parameters, including those that define the learning rules in the inner loop, are meta-trained with gradient descent. Conceptually, the outer loop corresponds to the natural evolution process where the biological synaptic plasticity is evolved. In our framework, the network updates its parameters in an unsupervised fashion, i.e., the computation of Aw in Algo- rithm I does not depend on the ground truth y.. The network must thus learn to adapt its parameters given only the input x,. x,. In some of our tasks, ground truth is given as part of the input for the model to learn the association between observations and targets (see Sec. 4). We choose not to use any external supervising signals follow the tradition of Hebbian plasticity, which does not depend on explicit supervising signals. Our formulation is thus a more realistic setting where the model must leam to identify the supervising signals from the input.
3.2 HEBBIAN PLASTICITY
We first discuss Hebbian plasticity with global neuromodulation. Recall that we assume the weight in plastic layer / be the sum of static part w, and plastic part wi. The plastic component
3

---

Published as a conference paper at ICLR 2023
Algorithm Training algorithm of plastic RNNS
Require: Trial Distribution T, loss function
Require: An RNN and its initial parameters w, comeetion-speciffc learning rates
while not converged do (Outer Loop)
2: Sample a batch of trials Ti
3: Initialize meta-loss
4: for each trial T; yx) do
5: Initialize the plastic weights wo t 0, initialize the hidden state hg <-
6: for time step 1, ... do (Inner Loop)
7: Get model prediction and new hidden state (y,, h,) < fx,,, h4- W w)
8: Compute the internal learning rate ny(t) according equation 2
9: Update the plastic parameters W, < (1 ( + n(t) Aw, where Aw is
computed with Hebbian or the gradient-based rule, Sec. 3.2 and 3.3 for details
10:
Accumulate meta-loss L + L Y().
end for
12:
end for
Perform gradient descent w.r.t. update w, a
14:
end while
qj
yx
Wij + Wij
Pi
Xt
Encoder
RNN
Linear
02
y/
Hobbian: AWij (t)) (a;jPiq; Wij))
Gradient: aw;; = n(()) atypi Waj
Modulate
Figure 1: Left: Model Architecture (see Sec. 3.1). Right: Comparison different learning rules in linear layer without nonlinearity. The difference between the two learning rules is highlighted. See Sec. 3.2 and 3.3 for more details.
wz(() is updated at each time step according the outer product of pre-synaptic activity pt()) and post-synaptic activity qr(():
(t) = σ (b, + (w) + w (t)),
(t 1) (1-)(t) (t) + (t) (pi(t)qT (t)), w) 0,
where is the activation function, o denotes element-wise product, and ay are learnable parameters that are initialized from 41|-1,1] 1]. ay acts as counection-speifee learning rates that allow each synapse to have different learning rules (e.g.. Hebbian or anti anti-Hebbian). Previous work on Hebbian plasticity has shown the benefit of having connection-spe speciffic plasticity over homogeneous plasticity (Miconi et al. 2018), and we found the same results in our experiments. The decay term, which is similar to the weight decay used in gradient descent algorithms, has been introduced in some recent Hebbian models (Miconi et al. [2018] Tyulmankov et al. 2022) to prevent the weight from exploding. (4) is the internal learning rate that controls the global plasticity, calculated as follows:
ntt) No Sigmoid(fj) min {1. max max_norm
where
||6,12
6, = Concat (Vec(pi(t)q (t)) S),
To a hyperparameter that controls the maximal learning rate, Vec(-) denotes the vectorization of matrix, Concat(-) denotes the concatenation of collection of vectors, and S is the set of plastic layers the network. We scale the internal learning rate according to the norm of 5, to prevent weights from changing too quickly. We use max norm 1 and 1)o 0.2 in all experiments.
Controlling the global plasticity with a self-generater signal 17(t) is well-motivated from the bio- logical perspective. In animals, neurotransmitters, especially dopamine, play an essential role in
4

---

Published as a conference paper at ICLR 2023
modulating plasticity and consequently influence animals' memory and learning (Cohn al. 2015
Katiuska et al. 2009 Kreitzer & Malenka 2008 Nadim & Bucher 2014). Theoretical models usu- global factor due to the volume transmission of neurotransmitters
ally model neuromodulation
(Magee & Grienberger 2020). Allowing the brain to adaptively modulate synaptic plasticity en-
ables reward-based learning (Gul 2002) Pignatelli Bonci 2013) and active control of forgetting
(Berry et al., 2012; 2015). Previous work has shown the benefit of such adaptive learning rates on Hebbian plasticity (Miconi et al., 2019). In our experiments, we empirically demonstrate that neuromodulation is helpful for both Hebbian and gradient- based plasticity, validating the biological understandings from computational perspective.
3.3 GRADIENT-BASED PLASTICITY
For RNNs with gradient- plasticity, at each time step t, we first calculate the internal loss on the model output o
L(t)= = dim(o:)
w
our 1e112 dim(o,)
| Wout Concat(y:, me, 1
(3)
where Wont are parameters initialized as 1 that are trained in the outer loop. All three components of the model output used to calculate the internal loss. Yt enables the model to meta-learn customized internal loss function that does not only depend on model prediction. In practice, we find a four-dimensional 7/: works well enough. Note that Yt does not affect network dynamics in Hebbian plasticity. The internal loss term does not involve the ground truth; it can thus be viewed as a self- gelf-generated target that the model wants to optimize. We then update the plastic parameters
as follows:
OL(()
wi + 1) = (i)n((t)(((t) + ntt)an
@wr(f) w)( 0:
@L(t)
bt(t + 1) (1 m(t)): + (t)BB
bi(0) = O; (b)
(4)
@L(t)
@t(()
8, Concat
Vec
(b) LES)
(wi
where 3 are learnable element-wise learning rates just like ax; ntt) is defined the same way in equation 2, except that 6, now denotes the concatenation of gradients of all plastic parameters. One difference with Hebbian plasticity is that the bias terms by are also plastic and updated similarly to weights. The gradient- gradient-based plasticity rule resembles the usual gradient descent, but the connection- specific learning rates and (3 allow additional flexibility.
Figure [ (right) shows a conceptual comparison between Hebbian and gradient- based plasticity. The main difference is that, for gradient- based plasticity, the gradient of the vost-synaptic activity replaces the activity itself, thus allowing update signals to propagate to previous layers. Two plastic- ity rules are particularly similar in the last linear layer, where the gradient- masdentt-baed plasticity rule takes the same form as Hebb's rule up to constant scaling vector (Sec. 1). In other words, the last linear layer will still follow Hebb's rule a network with the proposed gradient- plasticity.
EXPERIMENTS
Inspired by the hypotheses on the role of synaptic plasticity in animals (Magee & Grienberger 2020 Martin et al. 2000 Neves et al. 2008), we conduct experiments to test the following hypotheses:
1. Plasticity helps to form and retain memory and enlarges the memory capacity. In our network, we expect the plastic weights to act as extra memory storage in addition to the hidden states of RNN. We use the copying task and the cue-reward association task to test memory capability.
2. Plasticity helps the model to learn rapidly from their experiences and observations. We test this hypothesis with one-shot image classification and few-shot regression tasks.
To make fair comparisons between models, scale the size of hidden layers of different models so that all models have approximately the same number parameters. We use four different random seeds and average the results in all experiments. See Sec. A.2 for more implementation details.
5

---

Published as a conference paper at ICLR 2023
gradient
hebbian none
hebbian
none
Test Loss
Delay m
Sequence Length n
Figure 2:
Performance of LSTM models with different plasticity rules on the copying task. Error bars represent the SEM of four random runs. Left: Performance with different m when 5. Right: Performance with different n when m = 0.
4.1 COPYING TASK
We first test our models on sequential copying task. each trial, we generate random sequence of length After delay of m steps, the model must reproduce the sequence in its original order. The total length one trial thus + m. We calculate the MSE loss as the criterion for model performance, For more details, see Sec. A.3.I.
conduct two sets of experiments to compare our models with non-plastic baseline models. First, to test the ability of the models retain memory during delay, we set n 5 and vary the number of delay steps m. Second, to measure the models' sequential memory capacity, we set m = and vary the length of the sequence to be remembered. The results of LSTM models shown in 2 (see Figure for results RNNs). Indeed, we find plastic models exhibit larger memory capacity and able remember the sequence after long delay. In contrast, baseline models typically stuck on chance performance when the delay is large (see Figure for learning curves). The qualitative difference between plastic and non-plastic models shows the plastic RNNs' ability to capture long-term dependency when meta-trained with gradient descent.
Figure
4.2
ASSOCIATION
Associative memory refers the ability to remember the relationship between unrelated items. For animals, the dependency of associative memory on synaptic plasticity is well-documented in neuroscience literature (Morris al. 1986 Kim & Cho 2017 Nakazawa al. 2004). To evaluate if plasticity also helps the formation of associative memory in artificial RNNs, we train our models quickly associate cues with corresponding rewards. In each trial, we first sample n random cues and their corresponding rewards. We randomly choose a cue at each time step and present it to the model. The model is expected to answer the corresponding reward. During the first half of the trial, the ground truth is also given in input for the model to learn the association. Please refer to Sec. A.3.2 for more details.
The result is shown in Figure 3 Plastic models quickly converge to reasonable solutions with both RNN and LSTM backbone. The two plasticity rules perform similarly, the gradient-based plasticity appears more compatible with the LSTM backbone.
4.3 ONE-SHOT IMAGE CLASSIFICATION
To test models' ability of rapid learning, we train our models on the one-shot image classification task, a classic benchmark used in meta-learning. Here we consider the sequential version of 5-way one-shot image classification on MinilmageNet (Vinyals et al.) 2016) and CIFAR-FS (Bertinetto) et al. 2019). Similar to the previous task. in each trial, the model needs to learn the association between image embedding and the corresponding class in the training stage, then infer the class of novel images in the testing stage. Following previous work, choose regular 4-layer CNN or ResNet- 12 architecture as the image encoder (Lee et al. 2019). We describe more task details in Sec. A.3.3.

---

Published as a conference paper at ICLR 2023
grad
0.14-
0.14
hebbian
Validation Loss
0.12
0.12
none
gradient hebbian
0.10
Validation
0.10
0.08
none
0.08
0.06
0.06
0.04
0.04
5000 10000
15000
20000
5000
10000 15000 20000
Training Step
Training Step
Figure 3:
Validation loss curves in the cue-reward association task. The shaded area represents SEM on four random runs. Here and trial length 20. Left: RNN models. Right: LSTM models.
Conv-4
ResNet-12
Models
CIFAR-FS
minilmageNet
CIFAR-FS
minilmageNet
LSTM, Non-Plastic
49.9 - 0.5
46.8 1: 0.7
52.0 0.9
0.3
LSTM, Hebbian
50.0 0.7
46.6 0.8
4.3
33.2 +
LSTM. Gradient
50.5
47.0 0.3
50.
28.1 10.4
RNN, Non-Plastic
39.9
44.1
41.5
42.3 0.6
RNN, Hebbian
55.5 1.0
49,8 0.5
4.5
50.4 1.1
RNN, Gradient
51.2 2.6
47.9 + 1.2
52.8
50.5 0.4
MAML (Finn et al.) 2017)
58.9 + 1.9
48.7 + 1.8
ProtoNet (Snell et af. 2017
55.5 - 0.7
53.5 - 0.6
72.2 0.7
59.3 + 0.6
COSOC (Luo et al. 2021)
69.3 + 0.5
Table 1: Model performance (test accuracy) the one-shot image classification task compared to other methods. 95% confidence interval is shown. Data for ProtoNet is from (Lee et al. 2019).
The test performance of our models is reported in Table I Both plasticity rules improve the per- formance of RNN models by large margin, with the Hebbian plasticity performing slightly better. The learning curves (Figure 12) show that non- non-plastic RNNs can also overfit the training set, but the generalization gap is much larger than the plastic RNNs. These observations suggest that plasticity only increases representation power but also provides powerful inductive bias that inherently strengthens models' ability to learn from their environments quickly. Interestingly, even though the non-plastic LSTM consistently outperforms the non-plastic RNN, this is longer the case if plas- ticity is introduced. We infer that plastic weights provide stable memory storage like the cell states LSTMs. As result, the original advantages of LSTM models might no longer exist.
Plastic networks have comparable performance to other meta-learning methods when we limit the visual encoder be 4-layer CNN. However, we did find plastic networks significantly ben- efit from a deep vision encoder like the recent work on few-shot image classification (Lee et al., 2019 (Huisman et al. 202T). In recent years, methods that get the best results on few-shot learning benchmarks (e.g., MetaOptNet (Lee et al. (019)), COSOC (Luo et al.) (02T)) are also exclusively designed for few-shot image classification. Unlike our plastic RNNs, these methods are difficult to apply to other learning problems memory tasks. Instead of striving to get the best performance on specific task, our goal is to build a general architecture that tackles a wide range of memory and learning problems.
4.4 FEW-SHOT REGRESSION
We test our models on a regression task to further evaluate the performance of few-shot learning. In each trial, we randomly generate mapping f 1| - IR which is either a linear function or a small MLP. The model needs to learn the underlying mapping from K observations, i.e, K pairs of (x,, f(x))). then make predictions on f(x)) given x,. In experiments, use K 10 or 20. In the case of K 10, K is even smaller than the free parameters f. making the task more challenging. More details of the task described in Sec. A.3.4.

---

Published as a conference paper at ICLR 2023
Linear
MLP
Models
K 10
K 20
K = 10
K = 20
LSTM, Non-Plastic
.605 + .002
.402 .010
.241 .043
.087 - .006
LSTM, Hebbian
.446 + .003
.229 1001
.282
.013
.112 .002
LSTM. Gradient
.300 + .002
.107 .01
.142
.001
.036 .001
RNN. Non-Plastic
.605 + .002
.469 .001
.465 0015
.383 .007
RNN. Hebbian
.378 .059
.200 - .042
.165 = .001
.077 = .028
RNN, Gradient
.301 + .001
.108 .001
.142 .001
.036 + .001
Table 2: Test error on the K-shot regression task. 95% confidence interval is shown.
0.2
gradiant hebbian
modulated non- mod-laduldted
0.3
0.30
Validation Loss
0.25
0.1
Validetion
0.20
0.15
modulated
non-modulsted
0.0
20 60
60
2000
4000
6000 8000
10000
0
2500
5000
7500
10000
Training Step
Training Step
Figure 4: Left: Dynamics of it(t) in the copying task, we average the result across the 6400 trials in the test set. The shaded area reflects the SEM of models with different random seeds. LSTM models are shown. The sequence length n = 20 and delay m = 40. Middle and Right: Learning curves of different models in the copying task. Here compare using an adaptive internal learning rate ("modulated") and using a fixed one ("non-modulated") Middle: models with Hebbian plasticity. Right: models with gradient-based plasticity.
Model performance is shown in Table Unlike previous tasks, the proposed gradient- -baed plastic- ity consistently produces the best result, although Hebbian plasticity also improves the performance in most cases. We infer that such difference is caused by the need for inference in the few-shot regression task. In this task, the model not only needs to remember the observations but also infer the underlying mapping f from these observations, which necessitates complex calculations not quired in tasks such as associative learning. Models with gradient-basee based plasticity are more capable of learning the underlying rules, probably because they can leverage propaaatnnn to optimize their circuits over the desired target. In contrast, models with Hebbian plasticity are relatively weak evolving their network weights
reach any given target. local learning rule like Hebb's rule might be good enough for learning direct associations. However, the lack of feedback signals to prior layers makes it hard for the whole network evolve in a coordinated fashion.
4.5 ANALYSIS AND ABLATION STUDY
Recall that we use the internal learning rate 1)()) in our plastic networks to model biological neu- romodulation. rt)) is key quantity that controls how much information stored and discarded in plastic weights at each time step f. By tuning rtt) in an adaptive way, the network can effectively choose to learn from experience quickly retain the previously learned knowledge. Indeed, as shown in Figure 4 such adaptive behavior of 17(t) is observed in our experiments. In the copying task, the network learns to set large ry(t) when the sequence presented the model. 17(t) quickly decays to O during the delay, probably because the model learns to preserve the memory stored in plastic weights. In terms of task performance, we find an adaptive 1))4,, i.e., instead of setting ()t) to fixed value, consistently leads to improvements for both plasticity rules. Similar results are also observed in other tasks (Figure [O). These results suggest that neuromodulation the key to flexible and robust plasticity-based memory system.
We conduct additional ablation studies on the cue-reward association task; the detailed figures are shown in Sec. A.4. We find plastic RNNs consistently outperform non-plastic ones when larger or smaller learning rates are used (Figure R). and our default learning rate is appropriate for plastic and non-plastic models. We find the maximal learning rate тo to be key hyperparameter;
8

---

Published as a conference paper at ICLR 2023
RNN
Without
RNN
0.12
0.14
With
0,050
Velidtion Eree
Loys
0.12
Velidation Error
0.10
0.10
0.045
0.08
0.08
0.06
0,040
0.06
0.04
0.04
0.2
5000
10000
15600
20000
Training Step
dim
Figure 5: Ablation study on the cue-reward association task, gradient-based plasticity is used for all three panels. Left: Effect of no on final validation error, note that 7/ = 0 means no plasticity. Middle: Effect of connection-specifie learning rates a on validation loss curve. Right: Effect of dy)( on final validation error.
set = 0.2 for our main experiments to balance model capability and training stability (Figure connection-specifie learning rates a
left, also see Figure 99.. We show that handomly-initiaizze consistently work better than fixed global learning rate (Figure middle, also see Figure II]. For the gradient-based plasticity, we find that using y, improves performance, but dim(F.) good enough (Figure 5 right). In addition, on the copying task, when n 5 and m 40, we find that setting becomes nan) in 3 out of 4 random runs, illustrating the necessity of tuning down n7() when the norm change ||d:|| exceeds an appropriate threshold (equation 2).
works
100 instead of causes gradient explosion (so that the training loss
DISCUSSION
In this work, we draw inspiration from biological synaptic plasticity and propose incorporate different forms of plasticity into RNNs. We highlight the advantage of neuromodulated plasticity by comparing plastic RNNs against non-plastic ones on range of challenging memory and few- shot learning tasks. In resonance with hypotheses from neuroscience (Magee & Grienberger [2020] Martin al. 22000] Neves et al., 2008), we show that adopting plasticity in RNNs improves their memory performance and helps them to learn from observations quickly. Moreover, we go beyond the traditional Hebbian plasticity and design novel gradient-based plasticity where the model can flexibly adapt its weights with self-generated target. Our experiments illustrate the feasibility of training RNNs capable of doing gradient updates where both the learning rule and the internal loss function are meta-trained.
By comparing two plasticity rules under the same framework, we find both of them have pros and cons. The classical Hebbian plasticity, which is computationally more efficient, proved sufficient for robust memory storage and enabled the network to do simple forms of learning. However, results on the few-shot regression task show example of how networks could benefit from non-local learning rules where error signals are propagated to prior layers. Just like different forms of plasticity rules have been found in different brain regions accountable for different cognitive functions (Magee) & Grienberger 2020), we believe that different plasticity rules are suitable for different tasks in ANNs. For neuroscientists, our work also provides a computational framework that can potentially offer insights into the functions of different plasticity rules in the brain, which is still challenging to directly test animal experiments (Neves al., 2008 Magee Grienberger 2020).
Despite the promising results, training plastic RNNs with the current deep learning paradigm comes with challenges. Plastic weights are intrinsically unstable, and methods that stabilize the model (e.g., clip weights, normalization) could also limit the model's 's capability. In addition, training plastic RNNs with back proo-propngttion requires the plastic weight at each step to be stored in the computational graph, causing extensive memory usage GPUs. As result, applying plastic RNNs in more challenging settings (e.g., large-scale language modeling) takes more engineering effort. We hope our work can encourage future researchers further improve the engineering framework, explore other designs of plasticity rules, and investigate the benefit of synaptic plasticity in an even more comprehensive range of tasks.

---

Published as a conference paper at ICLR 2023
ACKNOW
We are thankful for Liyuan Wang and Yudi Xie for their helpful feedback. This work was supported by NSF of China Project (Research on the power supply of implanted neural dust clusters the sub-neural cell scale, 041302027).
REFERENCES
LF Abbott and Wade G Regehr. Synaptic computation. Nature, 431((7100):766 (000000))996-,,,,, 2004.
Wickliffe Abraham, Owen Jones, and David Glanzman. Is plasticity of synapses the mecha- nism of long-term memory storage? NPJ science of learning, 4(1):1-10, 2019.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z. Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. Advances in neural information processing systems, 29, 2016.
Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, and Kyoung Mu Lee. Meta- learning with task-adaptive loss function for few-shot learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14655-9477 2021.
Sergey Bartunov, Jack Rae, Simon Osindero, and Timothy Lillicrap. Meta-learning deep energy- based memory models. In International Conference on Learning Representations, 2020. URL
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta learning via learned loss. In 2020 25th International Conference on Puttern Recognition (ICPR), pp. 4161-4168. IEEE, 2021.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gescei. On the optimization of a synaptic learning rule. In Optimality in Biological and Artificial Networks?, pp. 281-303. Routledge, 2013.
Jacob A Berry, Isaac Servantes-Sandoval. Eric P Nicholas, and Ronald L Davis. Dopamine is required for learning and forgetting in drosophila. Neuron, 74(3):330-33 2012.
Jacob A Berry, Isaac Cervantes-Sandoval Molee Chakraborty, and Ronald Davis. Sleep facilitates memory by blocking dopamine meuron-mediated forgetting. Cell, 161(7):1666-1667, 2015.
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differen- tiable foomed-form solvers. In International Conference on Learning Representations, 2019. URL
https::/openrewe wwt.://wwwwweetteeeeee.m
net www.moormiiioooooo.o Ht
Katie C Bittner, Aaron D Milstein, Christine Grienberger, Sandro Romani, and Jeffrey C Magee. Behavioral time scale synaptic plasticity underlies cal place fields. Science, 357(635)):0333- 1036, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Raphael Cohn, lanessa Morantte, and Vanessa Ruta. Coordinated and compartmentalized neuro- modulation shapes sensory processing in drosophila. Cell, 163(7):1742-1755, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Golnaz Ghiasi, Tsung-Yo Yi Lin, and Quoc V Le. Dropblock: A regularization method for convolu- tional networks. Advances in neural information processing systems, 31, 2018
10

---

Published as a conference paper at ICLR 2023
Qiang Gu. Neuromodulatory transmitter systems in the cortex and their role in cortical plasticity. Neuroscience, 11 111(():815-88) 835, 2002.
David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetorrs.. In International Conference on Learn- ing Representations, 2017.
DO Hebb. The organization of behavior. a neuropsychological theory. 1949.
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey, IEEE transactions on pattern analysis and machine intelligence, 2021.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski, OpenAl Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. Advances in Neural Information Processing Systems, 31, 2018.
Mike Huisman, Jan N Van Rijn, and Aske Plaat. A survey of deep meta-lcarning, Artificial Intelli- gence Review, 44(6):44834451, 2021.
Kazuki Iric, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear trans- formers with recurrent fast weight programmers. Advances in Neural Information Processing Systems, 34:7703-7717, 2021.
M. L.. Katiuska, P. Ana, R. Sebastian, H. Benjamin, S. G. Maximilian, R. P. Mengia-Seraina, A. R. Luft, and L. Jan. Dopamine in motor cortex is necessary for skill learning and synaptic plasticity. PLoS ONE, 4(9)==7022- 2009.
Woong Bin Kim and Jun-Hyeong Cho. Encoding of discriminative fear memory by input-specific ltp in the amygdala. Neuron, 95(5): 111-111221111 1146, 2017.
Louis Kirsch, Sjoerd Steenkiste, and Juergen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. In International Conference on Learning Repre- sentations, 2020. URL
Lot.:t//ooouunnddddSleerreerr www.nnee//oouummee S1evierrYrr
A. C. Kreitzer and C. Malenka. Striatal plasticity and basal ganglia circuit function. Neuron, 60 (4):543-554, 2008.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kwonjoon Lee, Subhransu Maji. Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 10657-10665, 2019.
Thomas Limbacher and Robert Legenstein. H-mem: Harnessing synaptic plasticity with hebbian memory networks. Advances in Neural Information Processing Systems, 33:21627-21637, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
Xu Luo, Longhui Wei, Liangjian Wen, Jinrong Yang, Lingxi Xie, Zenglin Xu, and Qi Tian. Rectifying the shortcut learning of background for few-shot learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34. pp. 13073-13085. Curran Asso- ciates, Inc., 2021. URL. 2or2 06w. neur ps. cc/ppaper/221//f1ee .pdf]
Jeffrey C. Magee and Christine Grienberger. Synaptic plasticity forms and functions. Annual Re- newro-09099 view of Neuroscience,, 43(1):95-117, 2020. doi: 10. 166 neuro 0909 1900919-020 190119--2222 09-022842. URL
090919 022842
PMID: 32075520.
Adam H Marblestone, Greg Wayne, and Konrad P Kording. Toward an integration of deep learning and neuroscience. Frontiers in computational neuroscience, pp. 94, 2016.

---

Published as a conference paper at ICLR 2023
SP Martin, Paul D Grimwood, Richard GM Morris, al. Synaptic plasticity and memory: an evaluation of the hypothesis. Anmual review of neuroscience, 23(1):64--711, 2000.
Thomas Miconi, Kenneth Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. In International Conference on Machine Learning, pp. 3559- 3568. PMLR. 2018.
Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth O. Stanley. Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity. In International Conference on Learning Representations, 2019. URLbttps: httptpps///ppneeeeeeee..... net. nww.foruntffmm forum?
Id=rl1rAiASW
Aaron Milstein, Yiding Li, Katie Bittner, Christine Grienberger, Ivan Soltesz, Jeffrey C Magee, and Sandro Romani. Bidirectional synaptic plasticity rapidly modifies hippocampal representa- tions. Elife, 10, 2021.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018. URL https:
RGM Morris, Elizabeth Anderson, GS Lynch, and Michel Baudry. Selective impairment of learn- ing and blockade of long-term potentiation by an o-methyl-d-asparree receptor antagonist, ap5. Nature, 319(6056):77477777 776,,7777,,77 1986.
Tsendsuren Munkhdalai, Alessandro Sordoni, Tong Wang, and Adam Trischler. Metalearned neural memory. Advances in Neural Information Processing Systems, 32, 2019.
Sadegh Nabavi, Rocky Fox, Christophe Proulx, John Y Lin, Roger Tsien, and Roberto Malinow. Engineering a memory with Itd and ltp. Nature, 51 I (7509))3348 332-352, 2014.
Farzan Nadim and Dirk Bucher. Neuromodulation of neurons and synapses. Current opinion in neurobiology, 29:48 56, 2014.
Elias Najarro and Sebastian Risi. Meta-learning through hebbian plasticity in random net- works. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 20719 207311. Curran As- sociates, Inc., 2020. URL http:ttts///prceediinee http://eweeeeein.n...eeeeeeeeeee wwa.scc/// pctpppper//22///11eee.. Be23e7ad9b4 73dd0 72d5 aaa 9b222222222222222222222222222222 Papeerppprrppp..
Kazu Nakazawa, Thomas J McHugh, Matthew A Wilson, and Susumu Tonegawa. Nmda receptors, place cells and hippocampal spatial memory. Nature Reviews Neuroscience, 5(5):361-372, 2004.
Guilherme Neves, Sam F Cooke, and Tim VP Bliss. Synaptic plasticity, memory and the hippocam- pus: a neural network approach to causality. Nature Reviews Neuroscience, (1):65-75, 2008
Junhyuk Oh, Matteo Hessel, Wojciech M Czarneck, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 33:1060-1070, 2020.
Marco Pignatelli and Antonello Bonci. Role of dopamine neurons in reward and aversion: a synaptic plasticity perspective. Neuron, 86(5)::115--155 2015.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference Learning Representations, 2017. holLhttps://openeeete http:/ewwwwe.e..rrr.eeeeeee forum?
Kd=r
Mengia-S Rioult-Pedotti, Daniel Friedman, and John P Donoghue. Leamiig--indueed Itp in neocor- tex. science, 290(5491):5333555 2000.
Hector Garcia Rodriguez, Qinghai Guo, and Timoleon Moraitis. Short-term plasticity neurons leamn ing to learn and forget. In International Conference on Machine Learning, pp. 18704-18722 PMLR, 2022.
12

---

Published as a conference paper at ICLR 2023
Pieter R Roelfsema and Anthony Holtmaat. Control of synaptic plasticity in deep cortical networks. Nature Reviews Neuroscience, 1963):166-1 1106666-,,,0001 2018.
João Sacramento, Rui Ponte Costa, Yoshua Bengio, and Walter Senn. Dendritic cortical microcir- cuits approximate the backpropagation algorithm. Advances in neural information processing systems, 31, 2018.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory -aggeetd neural networks. In International conference on machine learn- ing, pp. 1842-1850, PMLR, 2016.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, 9355-9366. PMLR, 2021a.
Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative inference using fast weight memory. In International Conference on Learning Representations, 2021b.
URL [tRL[tttp:///perevieeee net -oree//oooonniiiTTKauuaccae
Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering. Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-imprrvement, Machine Learning, 28(1):
105-130, 1997.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Ad- vances in neural information processing systems, 30, 2017.
Ales Stuchlik. Dynamic learning and memory, synaptic plasticity and neurogenesis: an update.
Frontiers in behavioral neuroscience, 8:106, 2014.
Danil Tyulmankov, Ching Fang. Annapurna Vadaparty, and Guangyu Robert Yang. Biological learn- ing in key-value memory networks. Advances in Neural Information Processing Systems, 34: 22247-22258, 2021.
Danil Tyulmankov, Guangyu Robert Yang, and LF Abbott. Meta-lcarning synaptic plasticity and memory addressing for continual familiarity detection. Neuron, 110(3):544-557 2022.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
James C.R. Whittington and Rafal Bogacz. Theories of error baok-propagation in the brain. Trends in Cogmitive Sciences, 23(3):235-255) 2019. ISSN 1364-6613. doi: http:://doi.rrg//0.0111.... 2018. 12.05...05.. $1364661319300122 www.
A APPENDIX
A.1 COMPARISON OF TWO PLASTICITY RULES
Here we show that for the proposed gradient-based plasticity, the dynamic of the parameters in the last linear layer is Hebbian.
OT((I)
wt (t + 1) - wi( = 17())
ai @w(t)
wi(()
OL(!)
v1)
α (1(() daK (1)
- -w(())
(5)
)) wi())
2wout Wout
(t))
p((t)
(i()))
dim(o:)
(t) ay
(t) (qi(t) o v))) wi()))
where is scaling vector that does not change in trial. In comparison, with Hebbian plasticity, we have
wt + 1) (t) rttt) (ar (pi(t)qT (+) wt)))
Please refer Sec. 3 for the meaning of notations.
13

---

Published as a conference paper at ICLR 2023
A.2 IMPLEMENTATION DETAILS
A.2.1 ARCHITECTURE
For all tasks, the RNN and the last linear layer are plastic. For the one-shot image classification task, the encoder is CNN followed by a linear layer, both the CNN and the following linear layer are non- plastic. For other tasks, the encoder is plastic linear layer. In all experiments, the size of the RNN input layer is equal to the hidden size of the RNN. We apply the ReLU activation after the linear layer in the encoder. We ReLU activation in vanilla RNNs; we the customary activation functions in LSTMs. Layer weights and biases are initialized between [-1/N, 1/N], where N is the number of output neurons, i.c., layers are initialized in fan-out fashion. However, in the one- shot image classification task, considering the large dimension of the CNN embedding (1600 for Conv-4 and 16000 for ResNet- 12), the non-plastic linear layer following the CNN is initialized in fan-in fashion for better performance.
Due to the extra parameter of synapse-wisg wiie learning rate a, the number of parameters plastic models almost double. We thus scale the hidden size of the baseline model by 1.5x. We also scale hidden size of RNN models by 2x. See Table for details.
Models
Hidden size
Hidden size in One-shot Image Classification
LSTM, Non-Plastic
192
384
LSTM, Hebbian
128
256
LSTM, Gradient
128
256
RNN, Non-Plastic
384
768
RNN, Hebbian
256
512
RNN, Gradient
256
512
Table 3: Model size in different tasks. We scale the hidden size so that all models have a similar number of parameters,
A.2.2 TRAINING
Tasks
Training Steps
Validation Interval
Copying
10000
200
Cue-reward association
20000
200
One-shot image classification
40000
1000
Few-shot regression
10000
200
Table 4: Total number of training steps and the validation interval (i.e., how often do we perform validation) in different tasks.
For meta-trainne vreta-training, we use the Adam W optimizer (Loshchilov & Hutter 2017) with learning rate of 10-3 We use batch size 64 in all experiments. Gradients are clipped at norm 5.0. the one-shot image classification experiments, we use a weight decay of 5 x 10-1, we use the cosine annealing scheduler with the final learning rate set to 10-1, In other tasks, we set weight decay to zero since assume infinite training samples; learning rate schedulers the test set comprise 6400 trials (i.c., 100 batches). The number of training steps and the validation interval in different tasks is shown in Table 4 We test the model with the best validation error (or accuracy) on the test set and then report the test error (or accuracy).
not used. Both the validation and
Note that to meta-train plastic RNNs with gradient-basse based plasticity, need compute gradient- of of-gradients, which makes the training of gradient- basedd plasticity about 20% slower than Hebbian plasticity the cue-reward association task.
14

---

Published as a conference paper at ICLR 2023
A.3 TASK DETAILS
A.3.1 COPYING TASK
In each trial, we generate sequence of random numbers r; drawn from?([-1, 1| of length n. Recall that the model must reproduce the sequence after a delay of m steps. At each time step t, the input three dimensional vector X,, where:
If / [1, the first n steps;
(rr, 1, 0). In other words, we present the sequence to the model during
If E [n L,n + m). x = 0,,0,0);
If E [n + m + 1,22n + ml. X₁ = (0, 0, 1). So the last entry is a binary variable indicating
whether t m + n.
We calculate the MSE loss as the criterion for model performance, i.e.,
(re Un+m+t)?
where denotes the model prediction. The same loss is used for traintrr
A.3.2 CUE-REWARD ASSOCIATION
Let the trial length be m. Recall that we use sample n random cues C1, ..., On from [0, 1j4 where d 14. We also sample the corresponding rewards ri...., Ty from 00, 1]. At each time step we randomly sample i, E 21, nj and a small Gaussian noise & with o = 0.1. The input is a vector of size + 2, X2 = Concat + 8 Fr. bt), where:
20 in our experiments. In each trial, we first
.
lf E m1,m/2), Ft ria b. =0 0.
. If E [m/2 + 1,m. Fr 0, b: = 1.
We calculate the MSE loss over the whole trial as the criterion for model performance, i.e.,
(ru, )²)
1=1
The same loss is used for meta-training.
A.3.3 ONE-SHOT IMAGE CLASSIFICATION
We first give more details on the dataset. Previous work reported that Hebbian plasticity improves one-shot image classification performance on Omniglot dataset (Miconi et al., 2018). However, Omniglot is not a challenging image dataset and models can easily get near-perfect performance Mishra et al., 2018, Miconi et al.) 2018); therefore, recent works tend to choose MinilmageNet VVinyals al. 2016) and CIFAR-FS (Bertinetto et al.) 2019) instead. The minilmageNet dataset consists of 100 image classes, and there are 600 images of size x for each class. The CIFAR- FS dataset consists of the 100 image classes from CIFAR- 100 (Krizhevsky et al.) 2009); each class has 600 images of size 32 x 32. For both datasets, the 100 image classes are split into 64, 16. and 20 classes for meta-training, meta-validation, and meta-testing, respectively. We use the customary way to partition the datasets (Ravi Larochelle] 2017 Lee et al. 2019).
Now describe the details of the sequential version of 5-way 1-shot image classification. In each trial, we randomly sample five image classes from the dataset. Unlike previous tasks, the input in this task an image-label pair (a,, b.), where a is random image from of the five chosen classes, by is 5-dimensional vector.
For < < 5, (the training stage). b, is the one-hot encoding of the label Dr. where /e (1, 4,5).
15

---

Published as a conference paper at ICLR 2023
For < t 10, (the testing stage), b, the model needs to predict the label with learned associations.
In both the training and the testing stage, the model will see precisely one image from each chosen class, and the order of labels is randomized. During meta-training, we average the cross entropy loss on the whole testing stage, i.e.,
10
L
CrossEntropy(yr..l! Mt).
Since the model might utilize the prior knowledge that exactly one image from each class is pre- sented in the testing stage, we only consider the first test image when we calculate the test accuracy.
In order for the model process the paired input (ar, by), we first calculate the image embedding of ag, the embedding is then projected to lower dimension by non-plastic linear layer and con- catenated with b,- The dimension of the resulting vector is the same as the hidden size of the RNN. We adopt some common techniques to alleviate meta-overfitting, including data augmentations and dropblock (Ghiasi et al. 2018) ResNet models. We use the same data augmentations and drop- block configurations used in MetaOptNet (Lee al. 2019)
A.3.4 FEW-SHOT REGRESSION
Recall that in each trial, we first generate mapping f [-1, + IR. At each time step we sample a random vector V, [-1, 1] and expect the model answer f(ve). The input is a vector of size d + 2, x, Concat(c,, + de, St, be), where:
Ift E [1, KJ. De f(ve) + b = 1, where is small gaussian noise with a = 0.1.
Ift E [K + 1, + 20). 3t- 0, b =
The model should try infer the mapping given the noisy observations during the first K steps, then answer the queries during the next 20 steps. The exact form of f decides the task difficulty; here, we consider two different settings:
1. f is randomly initialized linear layer with bias, d 12.
2. f is two-layer MLP with tanh activation, the size of the hidden layer is 2, d 6.
After f is generated in each trial, we do an affine transformation f + af + so that f(v:) has zero mean and unit variance. We calculate the MSE loss over the whole trial the criterion for model performance, i.e.,
K +0
(J(v:) - yr)
K
+ 20
1=1
The same loss used for meta-training.
A.4 SUPPLEMENTARY FIGURES
16

---

Published as a conference paper at ICLR 2023
gradient hebbian none
gradient
hebbian
none
Test Loss
60
Delay m
Sequence Length n
Figure 6: Performance of RNN models with different plasticity rules the copying task. Error bars represent the SEM of four random runs. Left: Performance with different m when n 5. Right: Performance with different n when m 0.
gradient hebbian none
gradient
RNN Validation Loss
D.3
0.3
hebbian
none
LSTM Validation
0.0
2000
4000
6000
8000
10000
2000
4000
6000
8000
10000
Training Step
Training Step
RNN Validation Loss
0.30
0.3
0.25
LSTM Validation
0.20
0.15
gradient
0.1
gradient
hebbian
hebbian
0.10
none
none
2500
5000
7500
10000
0
2000
4000
6000
8000
10000
Training Step
Training Step
0.30
RNN Validation Loss
0.25
0.2
gradient
LSTM
0.1
gradient
0.10
hebbian
0.05
2500
5000
7500
10000
2000
4000 6000 adoo
10000
Training Step
Training Step
Figure 7: Validation loss curves for the copying task when n 20. From top to bottom: 0, Left: RNN models. Right: LSTM models.
m 20, m
17

---

Published as a conference paper at ICLR 2023
0.14
Validation Loss
D.12
gradient hebbian
0.12
gradient
0.10
Validation
0.10
0.08
0.06
0.08
0.04
0.06
10000
15000
20000
5000
10000
15000
20000
Training Step
Training Step
0.16
0.14
0.14
Val dation Loss
0.12
0.12
gradient
gradient hebbian
hebbian
0.10
none
0.08
none
0.08
0.06
0.06
0.04
5000
10000 15000
20000
5000
10000 15000 Training Step
20000
Training Step
Figure 8: Results the cue-reward association task when different learning rates are used. Note that the default learning rate 10-3 Top: learning rate x -4, Bottom: learning rate 10-2 Left: RNN models. Right: LSTM models.
RNN
0.12
LSTM
0.060
Validation Error
0.10
0.055
Validation
0.050
0.08
LSTM
0.045
0.06
0.040
0.04
0.035
0.1
dim
Figure 9: Ablation study on the cue-reward association task, Hebbian plasticity is used. Left: Effect of n/ final validation error. Note that 7/ = O) means plasticity. Right: Effect of dim(y;) on final validation error. For Hebbian plasticity. y. has no significant effect on performance since does influence the model dynamies. It only has minor effect on weight initialization (since the last linear layer is initialized in fan-out manner).
18

---

Published as a conference paper at ICLR 2023
0.2
gradient hebbion
0.14
modulated non-modulated
0.14
modulated
non-modulated
delidatioon Loes
Voll on Loss
0.12
0.12
0.10
0.10
0.08
0.08
0.06
0.0
0.06
0.04
5
10
15
20
5000
10000
15000
20000
5000
10000
15000 20000
Training Step
Training Stop
0.2
gradient
0.14
moduisted
modulsted
hebbian
non-modulsted
non-modulated
Vallcation Lees
0.12
Validetion Loss
0.12
0.10
0.10
0.08
0.08
0.06
0.0
0.04
0.04
5
10
15
20
5000
10000
15000
20000
0
5000
10000
15000
20000
Training Step
Training
0.2
0.5
0.5
gradiant hebblan
modulated
modulated
non-modulated
non-modulatee
0.4
Validation Lozs
Validation Loos
0.2
0.2
0.1
0.0
0.1
10
20
30
40
0
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
Training Step
Training Step
0.2
0.5
gradient hebbian
modulated
modulated
0.4
non-modee moduladdd
non-m adulated
0.4
Validation Loss
Validation
0.3
0.2
0.2
0.1
0.1
0.0
0
10
20
30
40
2000
4000
6000 8000
10000
0
2000
4000
6000 8000 10000
C
Training Stop
Training Stop
Figure 10: Trial-averaged curves of 17(t) (the left column) and the effect of ncuromodulation on performance (the middle and the right column) in different tasks. The value of ry(t) is averaged over 6400 trials in the test set. The middle column shows results with Hebbian plasticity and the right column shows results with gradient-based plasticity. From top to bottom: 1) results of LSTM models on the cue-reward association task; 2) results of RNN models on the cue-reward association task; 3) results of LSTM models on the few-shot regression task, where the underlying mapping is an MLP, K 20; 4) results of RNN models on the few-shot regression task.
19

---

Published as a conference paper at ICLR 2023
none
none
0.14
0.14
uniform
uniferm
Val Validatoon Lo55
0.12
neg, uniform
0.12
unifomm
ran dom
random
0.10
Validation
0.10
0.08
0.08
0.06
0.06
0.04
0.04
5000
10000
15000
20000
5000
10000
15000
20000
Training Step
Training
0.14
uniform
none
0.14
validation Loss
neg unifo m
0.12
0.12
random
Validation
neg unifomm 0.10
uniform
0.10
random
0.08
0.06
15000 20000
15000 20000
Training Step
Training Step
Figure 1I: Comparison different ways to initialize the learning rates ay on the cue-reward as- sociation task. 'none': connection-specifi learning rates. 'uniform': day 1. 'neg uniform':: aij = -1 'random': ai ~ 4[-1, Top: gradient-based plasticity. Bottom: Hebbian plasticity.
Left: LSTM models. Right: RNN models.
1.5
hebbian
Validation Accuracy
50
Training Loss
none
40
2.0
30
0.5
gradient
hebbian
20
none
0.0
10000
20000
30000
40000
10000
20000
30000
40000
Training Step
Training Step
2.5
gradient
hebbian
Validation Accuracy
Training Loss
none
1.5
gradient
0.5
hebbian
none
10000 20000 30000 Training Step
40000
10000 20000 Training Step
30000
40000
Figure 12: Training loss (left) and validation accuracy (right) curves of the one-shot image classifi- cation task on the CIFAR-FS (top) and minilmageNet (bottom) datasets. Here we use ResNet- 12 as the encoder and the vanilla RNN (instead of L.ST) as the backbone of the recurrent module.
20


Reviews:
[{"review_id": "ai2fvGem8w", "reviewer": "Reviewer_x3Sh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the Authors design and implement recurrent neural networks with two types of plasticity: classical Hebbian and gradient-based w.r.t. an internal loss function. The Authors then test their models on a set of memory/learning tasks including copying, cue-reward association, image classification, and regression. They find that the new gradient-based approach fares better on the regression task while the classical Hebbian approach is better in copying and image classification tasks. An ablation study is provided to assess the importance of the models’ parameters.", "summary_of_the_review": "This is a comprehensive study describing two mechanisms of synaptic plasticity in RNNs. The properties of the described models are thoroughly tested. In the future, some of the design choices made here (quite interesting!) may help build neuroscience-relevant models. The drawback is in high similarity to prior work (Hebb’s rule part) and high similarity to Hebb’s rule scores on benchmarks (gradient part).", "strength_and_weaknesses": "This is a thoroughly-designed and well-described study of plasticity in recurrent neural networks, which builds upon previous results by Jeff Clune, Ken Stanley, and colleagues, known to the ICLR community.\n\nThe paper introduces several interesting updates on top of the previous works. First, the “learning rate of plasticity” \\eta here is time-dependent, which supposedly allows the models to turn on the synaptic plasticity during one/few-shot learning, and to turn it off to retain the learned knowledge during testing. Second, in the gradient-based method, the Authors introduce internal loss. This loss relies on an extra output of the RNN, conditioned upon the RNN’s weights trained in the outer loop, and is used to guide the network’s plasticity in the inner loop. For both these updates, the Authors provide (in the Appendix) the ablation studies showing how different values of these parameters affect the models’ performance.\n\nThe internal loss part seems especially interesting because it may link this work to a biological phenomenon of motivation. Previously. motivations in machine learning were set to reflect observable quantities such as curiosity; conversely, the internal cost functions here are defined by the model itself to reflect the important task-related quantities.\n\nAnother strength of this work lies in the comprehensive testing of the models which includes standard benchmarks and thorough ablation studies. It would be nice though to move more figures to the main text, if possible, as they show important results.\n\nSpeaking of weaknesses, the Hebbian-plasticity part of the work is highly similar to that in a series of works by Miconi et al, mentioned by the Authors (the similarity is also acknowledged by the Authors). The differences (which need to be clearly summarized somewhere in the text!) seem to be limited to 1) the time-dependent internal learning rate for synaptic plasticity, 2) not using Oja’s rule, and 3) using a (more complex) image classification task instead of the Omniglot. Furthermore, the proposed gradient-based rule seems to be only efficient in the few-shot regression task among the used benchmarks. These two factors limit the contribution of this work to the field. I also found the biological insights a bit preliminary – though they do not constitute the main result of the work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZYUcbCnlP1G", "reviewer": "Reviewer_5CwK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inspired by the evolved role of plasticity in neural circuits, this paper proposes a meta-learning approach for training RNNs. The method consists of an inner loop with a learning rule that updates the parameters of an RNN, and an outer loop that learns a function for the update rule. ", "summary_of_the_review": "Although the proposed meta-learning algorithm is interesting (not novel), the comparisons with untrained (or non-plastic) RNNs is not sufficient for demonstrating the strengths of the proposed method. ", "strength_and_weaknesses": "The proposed meta-learning approach is interesting especially from a neuroscience perspective (though not novel, e.g. Najarro and Risi, NeurIPS, 2020). However, I’m not convinced that the proposed model has any advantages compared to the previously proposed methods. In particular, almost all the comparisons are between the proposed method and non-plastic RNNs which I found insufficient. \n\n$\\textbf{Questions}$:\n\n1- Almost all the comparisons are with what is called non-plastic RNNs in the paper. The motivation behind using non-plastic RNNs as the baseline model for comparisons is not explained in the paper. Specifically, the initialization of weights could place the untrained RNNs in a disadvantaged dynamical regime compared to the trained RNNs. It is hard to imagine a situation where non-plastic neural networks are considered as an alternative for how brains function, except in cases where, for example, the RNNs are initialized at the edge of chaos (criticality) which could give them specific advantages (better short-term memory, etc.) Can the authors elaborate more on the motivation behind comparing only with untrained RNNs? \n\n2- The update rules of the outer (meta-learning) rule are not shown in the paper. It was only mentioned that gradient descent was used for the outer loop, but the loss function and the target variables are not explained. Was the MSE loss shown in figures 2-3 used for the outer loop updates? Also, mentioned in section 3.2 “ [...] network parameters, including those that define the learning rules in the inner loop, are meta-trained with gradient descent.” A list of parameters meta-trained with gradient descent would be helpful.\n\n3- Related to comment # 1, it would be interesting to see comparisons with RNNs trained in a supervised (or unsupervised) fashion (no meta-learning). The main advantages of the meta-learning approach in associative memory and few-shot learning would be more clear with those comparisons. \n\n4- One general comment: Although a key concept in the paper, the connection with meta-learning did not become clear until the end of the introduction (not even mentioned in the abstract). Concepts like “self-determined target” and “neuromodulated plasticity” are used a few times throughout the introduction before mentioning that the proposed method is a meta-learning algorithm. Because of that, It's difficult to follow the introduction and understand the main question of the paper. I suggest that the authors rephrase the introduction and highlight the connections with meta-learning at the very beginning. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J7L7Qo10bF", "reviewer": "Reviewer_vDcd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper compares two forms of meta-learning, with inner-loop updates based either on modulated Hebbian plasticity, or on gradient descent over a network-generated synthetic loss function. \n\nVarious experiments show that both forms of episodic plasticity (Hebbian and gradient-based) improve performance over using fixed recurrent networks, as in RL^2/L2RL. Network-controlled modulation of plasticity is also shown to be important.\n\n", "summary_of_the_review": "== Update after reading the authors' response ==\n\nI thank the authors for their clarifications and I have updated my review and score towards acceptance.\n\n== Original review ==\n\nPlease cite more existing literature about the concept of meta-learned loss functions; explicitly state the novelty of the work; and clarify the working of the algorithm.\n\nI am willing to increase my score depending on the author's response to the above, especially regarding novelty.", "strength_and_weaknesses": "Strengths:\n\n- The paper tackles an interesting problem (learning to learn without necessarily having access to a supervised or even a reward  signal during each episode, i.e. the inner loop is unsupervised).\n\n- The paper is interesting and may propose one novel method, i.e. network-generated gradient loss, though see below.\n\n- The experiments, such as they are, offer reasonable evidence that the proposed methods are useful over the studied tasks\n\n\nWeaknesses:\n\nMy main concern is about novelty. AFAICT the Hebbian portion of the method is identical to previous proposals that are cited in the paper. \n\nSimilarly, the gradient-based  method seems to be an example of meta-learning a loss function to perform gradient descent on in the inner loop. This approach is clearly not new, see e.g. https://arxiv.org/abs/1802.04821 , https://arxiv.org/abs/1906.05374 , https://arxiv.org/abs/2110.03909 and  the many references therein.  (Disclaimer: I am not affiliated with any of these authors)\n\nSurprisingly, no previous paper on this approach (meta-learning a loss function) seems to be cited in the paper, unless I missed it.\n\nThere *may* be some novelty in the proposed approach, because here the synthetic loss  is an output of the learner network itself (as opposed to a separate dedicated network). IIUC this means that the synthetic loss itself is subject to learning and tuning during the inner loop (as  in Baik et al. mentioned above, but in a much more flexible manner). This also gives rise to interesting loops in the learning process (though the precise working of the algorithm is a bit unclear, see below).\n\nIf this is correct, and if this is the claim to novelty of the paper, it should be stated explicitly.\n\nAdditional possible  problems:\n- The paper is somewhat unclear on certain points, which is important to better understand the proposed method. See below. \n- The experiments are interesting, but perhaps a bit limited?\n- Not really concerns, but suggestions for future work: the two approaches (Hebbian and gradient-based) seem fully compatible and it might be interesting to see what  happens when a network uses both together; the weight decay parameter, which is now just 1-eta, could include a separate learnable parameter as is Tyulmankov et al. and Rodriguez, Guo && Moraitis ICML 2022 (I am not affiliated with these authors).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1u6-PVbvnO", "reviewer": "Reviewer_fkhw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript introduces a new way of learning task-specific unsupervised learning rule for RNNs in which the network computes not only its task output but also dynamically alters hyperparameters of its own learning process. The learning rule functional form is learned in a gradient-based outer loop based on task constraints provided via a validation set. The model is validated on a wide range of tasks and shows interesting few shot learning performance improvements relative to a static network.\n", "summary_of_the_review": "Overall an intriguing idea and interesting results, but the presentation of the core methodology and its justification is too sparse and needs substantial improvement before publication.", "strength_and_weaknesses": "\n+ Strength: impressive performance\n+ S: analogies to biology as a motivation for the general approach\n- Weakness: the logic of the optimization procedure is not spelled out as clearly as it should be\n- W: alternative model comparison is restricted to variations of the same scheme, hard to evaluate the benefits of the approach more broadly w.r.t. alternative few shot learning approaches\n- W: conceptually unclear to me if the hebbian vs gradient based distinction is a technical choice or if one of the two versions is championed in the paper\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster