Meta-Reinforcement Learning with Self-Modifying Networks
Authors: Mathieu Chalvidal
Thomas Serre
Rufin VanRullen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Meta-reinforcement learning
Synaptic plasticity
Hopfield networks
Control theory

Abstract: Deep Reinforcement Learning has demonstrated the potential of neural networks tuned with gradient descent for solving complex tasks in well-delimited environments. However, these neural systems are slow learners producing specialized agents with no mechanism to continue learning beyond their training curriculum. On the contrary, biological synaptic plasticity is persistent and manifold, and has been hypothesized to play a key role in executive functions such as working memory and cognitive flexibility, potentially supporting more efficient and generic learning abilities. Inspired by this, we propose to build networks with dynamic weights, able to continually perform self-reflexive modification as a function of their current synaptic state and action-reward feedback, rather than a fixed network configuration. The resulting model, MetODS (for Meta-Optimized Dynamical Synapses) is a broadly applicable meta-reinforcement learning system able to learn efficient and powerful control rules in the agent policy space. A single layer with dynamic synapses can perform one-shot learning, generalize navigation principles to unseen environments and demonstrates a strong ability to learn adaptive motor policies, comparing favorably with previous meta-reinforcement learning approaches.

Meta-Reinforcement Learning with Self-Moditying Networks
Mathieu Chalvidal
Thomas Serre
Artificial and Natural Intelligence Toulouse Institute Universite de Toulouse, France
Carney Institute for Brain Science Brown University, U.S. thomas_ www.seererrrrrbrreeeeee edu
mathieu_chalvee btabiee_challiid@brom.coo
Rufin VanRullen
Centre de Recherche Cerveau & Cognition CNRS, Universite de Toulouse, France rufin. vanrullen@cnrs... fr
Abstract
Deep Reinforcement Learning has demonstrated the potential of neural networks tuned with gradient descent for solving complex tasks in well-delimited environ- ments. However, these neural systems are slow learners producing specialized agents with no mechanism to continue learning beyond their training curriculum. On the contrary, biological synaptic plasticity is persistent and manifold, and has been hypothesized to play a key role in executive functions such working memory and cognitive flexibility, potentially supporting more efficient and generic learning abilities. Inspired by this, propose to build networks with dynamic weights, able continually perform self-reflexive modification as function of their current synaptic state and action-reward feedback, rather than a fixed network configuration. The resulting model, MetODS (for Meta-Optiman Optimized Dynamical Synapses) broadly applicable meta-reinforcemmnt learning system able learn efficient and powerful control rules the agent policy space. A single layer with dynamic synapses can perform one-shot learning, generalizes navigation principles to unseen environments and manifests a strong ability to learn adaptive motor policies.
Introduction
The algorithmic shift from hand-designed to learned features characterizing modern Deep Learning has been transformative for Reinforcement Learning (RL), allowing to solve complex problems
ranging from video games [ multiplayer contests 3 motor control 4 55) Yet, "deep" RL has mostly produced specialized agents unable to cope with rapid contextual changes or tasks with novel or compositional structure [6.8]. The vast majority of models have relied gradient- based optimization to learn static network parameters adjusted during a predefined curriculum, arguably preventing the emergence of online adaptivity. potential solution this challenge is meta-learn 9 computational mechanisms able to rapidly capture a task structure and automatically operate complex feedback control: Meta-Reine Reinforcement Learning constitutes a promising direction build more adaptive artificial systems [12) and identify key neuroscience mechanisms that endow humans with their versatile learning abilities B
In this work, we draw inspiration from biological fast synaptic plasticity, hypothesized to orchestrate flexible cognitive functions according context-dependent rules [14]77] By tuning neuronal selectivity at fast time scales - from fast neural signaling (milliseconds) to experience- based learning
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(seconds and beyond) fast plasticity can support in principle many cognitive faculties including motor and executive control. From dynamical system perspective, fast plasticity can serve as an efficient mechanism for information storage and manipulation and has led to modern theories of working memory 22] Despite the fact that the magnitude of the synaptic gain variations may be small, such modifications are capable of profoundly altering the network transfer function [23] and constitute plausible mechanism for rapidly converting reward and choice history into tuned neural functions (24]. From a machine learning perspective, despite having a long history [25/29), fast weights have most often been investigated in conjunction with recurrent neural activations 30 53]
and rarely function of external reward signal or of the current synaptic state itself. Recently, new proposals have shown that models with dynamic modulation related to fast weights could yield powerful rett-reiiforeemmet learners (6). In this work, explore an original self-referential update rule that allows the model to form synaptic updates conditionally on information present in its own synaptic memory. Additionally, environmental reward is injected continually in the model rich feedback signal to drive the weight dynamics. These features endow our model with a unique recursive control scheme, that support the emergence of self-contained reinforcement learning program.
Contribution: We demonstrate that neural network trained to continually self-modify its weights as a function of sensory information and its own synaptic state can produce a powerful reinforce- ment learning program. The resulting general- purpose meta-RL agent called 'MetODS' (for Meta- Optimized Dynamical Synapses) is theoretically presented as a model-free approach performing stochastic feedback control in the policy space. In our experimental evaluation, investigate the reinforcement learning strategies implemented by the model and demonstrate that a single layer with lightweight parametrization can implement wide spectrum of cognitive functions, from one- shot learning to continuous motor-control. We hope that MetODS inspires more works around self-optimizing neural networks.
The remainder of the paper is organised as follows: Section we introduce our mathematical formulation of the meta-RL problem, which motivates MetODS computational principles presented in Section 3 In Section 4 we review previous approaches of reneorreiforr learning and we discuss other models of artificial fast plasticity and their relation to associative memory. In Section 5 we report experimental results in multiple contexts. Finally, ,, in Section 6 we summarise the main advantages of MetODS and outline future work directions.
2 Background
2.1 Notation
Throughout, we refer to "tasks" as Markov decision processes (MDP) defined by the following tuple т = (S, A, P, r. Po), where and A are respectively the state and action sets, P S x x + [0. 1] refers to the state transition distribution measure associating probability to each tuple (state, action, new statc), r A S + I is a bounded reward function and Po is the initial state distribution. consider finite-horizon MDP with T time-steps although extended to the infinite horizon case as well as partially observed MDP. We further specify notation when needed by subseripting symbols with the corresponding task time-step t.
For simplicity,
discussion be
2.2 Meta-Reinforcemen learning optimal transport problem:
Meta Retta-Reinfrremmet learning considers the problem of building program that generates distribution y, of policies E IT that are "adapted" for distribution /4r of task 7 E T with respect to given metric R. For instance, R(T, T) can the expected cumulative reward for a task and policy a, ie where state transitions are governed by 8 P, (-Ist, at), actions are sampled according to the policy т: a; ~ я and initial state 8p follows the distribution po,r- Provided the existence of an optimal policy n² for any task T, we can define the distribution measure jrr of these policies over II. Arguably, an ideal system aims at associating to any task т its optimal policy n², i.e, finding the transport plan ry in the space T(UT, (xx) of coupling distributions with marginals μr and px that maximizes R.
max B(r,,) R(T, (ET(MT)))*)
where R(T,x) E.,...
Σ rr (a,, 8t)
(I)

---

b)
HT
TT
MTo
TI
7
T2
Figure 1: Meta-Reinforcement Learning as a transport problem and MetODS synaptic adaptation: a) Associating any task r in T to its optimal policy π. in II can be regarded as finding an optimal transport plan from for to fr" with respect to the cost -- RR Finding this transport plan is generally an intractable problem. b) Meta-RL approximates solution by defining a stochastic flow in the policy space II conditioned by the current task t and driving a prior distribution 1o of policies wo towards distribution f.r" &.r.t of policies with high score R. e) Density and mean trajectorics of our model dynamic weights principal components over several episodes of the Harlow task (see (cc1ion[3.1 reveal this policy specialization. Two modes colored with respect to whether the agent initial guess was good or bad emerge, corresponding two different policies solve the task.
Most generally, problem (1 is intractable, since Ma- is unknow or has explicit form. Instead, previous approaches optimize a surrogate problem, by defining a parametric specialization procedure which builds for any task sequence (Tz)): of improving policies (see Fig. [ Defining 0 some meta-paramenn parammeerssss governing the evolution of the sequences (Te) and 1..T. the distribution measure of the policy my after learning task r during some period 4, the optimization problem amounts to finding the meta-parameters A that best adapt Te ~ lig.i over the task distribution.
E BRN - [R(T, ( m)]
max 0
Equation (2) cast meia-Reeinforeement learning the problem of building automatic policy control in II. We discuss below desirable properties such control for which we show that our proposed meta-learnt synaptic rule has a good potential.
Efficiency: How "fast" the distribution p.. transported towards distribution of high- performing policies. Arguably, an efficient learning mechanism should require few interaction steps with the environment to identify the task rule and adapt its policy accordingly. This can be seen as minimizing the agent's 's cumulative regret during learning [37. This notion is also connected to the policy exploration-explotaaiiie foster rapid discovery of policy improvements while retaining performance gains acquired through exploration. We show that our learnt synaptic update rule able to change the agent transfer function drastically in few updates, and settle to policy when task structure has been identified, thus supporting one-shot learning of a task-contingent association rule in the Harlow task, adapting a motor policy in few steps exploring original environments quickly in the Maze experiment.
trade-off [38 [3], where the agent learning program should
That property defines the learner sensitivity to particular task structures and its ability to convert them into precise states in the policy space, which determines the achievable level of performance for a distribution of tasks μr. This notion is linked the sensitivity of the learner, i.e how the agent captures and retains statistics and structures describing a task. Precedent work has shown that memory-bar based Meta-RL models operate a Bayesian task regression problem 44), and further fostering task identification through diverse regularizations 42 39 benefited performance of the optimized learning strategy. Because our mechanism is continual, it allows for constant tracking of the environment information and policy update similar to memory- -based models. We particularly test this property in the maze experiment in Section 5 showing that tuned online synaptic updates obtain the best capacity under systematic variation of the environment.
Capacity:
Generality: We refer here to the overall ability of the meta-learnt policy flow drive μ ...T. towards high-performing policy regions for a diverse set of tasks (generic trainability) but also to how general the resulting reinforcement learning program and how well transfers to tasks unseen during training (raasferabilitt). In the former case. since the proposed synaptic mechanism is model -reee, allows for tackling diverse types of policy learning, from navigation to motor control. Arguably, to build reinforcing agents that learn in open situations, we should strive for generic and efficient computational mechanisms rather than learnt heuristies. Transferability corresponds to the ability of the meta-learned policy flow to generally yield improving updates even in unseen policy regions
3

---

of the space II or conditioned by unseen task properties: new states, actions and transitions, new reward profile. etc. We show in a motor-control experiment using the Meta- Woold benchmark that meta-tuned synaptic updates are a potential candidate to produce a more systematic learner agnostic to environment setting and reward profile. The generality property remains the hardest for current meta- approache, demonstrating the importance of building more stable and invariant control principles.
3
MetODS: Meta-Optimized Dynamical Synapses
3.1 Learning reflexive weights updates
What if a neural agent could adjust the rule driving the evolution of f.. based on own knowledge? In this work, we test a neural computational model that learns compress experience of a task r into its dynamic weights W. vt T, 7(a 8, W,) T.
V(
V(
V
W
V
W
W
Contrary to gradient- based rules whose analytical expression a static and predefined function of ac- tivations error (A(W,) aw. date rule that directly depends on the current weight state through parameterized non-linear recursive scheme, making the expression of the update much more sensitive to weight content itself.
W
da ) we define an up-
(4) Figure 2: A layer of MetODS updates its weights by recursive applications of read and write opera- tions based on neural activations 2)(*) and synaptic This self-reflexive evolution of the synaptic config- traces ((())
At TT, A(W.) = (W))
W=w.
uration W, allows for online variation of the learning rule during adaptation for a given task and opens an original repertoire of dynamics for the policy evolution u..... Specifically, at every time-step i, network inference and learning consists in recursive applications of read and write operations that we define below.
Read-write operations: The core mechanism consists in two simple operations that respectively linearly project neurons activation v through the dynamic weights W followed by an element-wise non-linearity, and build a local weight update with element- wise weighting a::
o W. v) = o (W.v) read
(5)
(b(b) о v write
Here, is matrix of RNNN, @ denotes the outer-product, @ is the element-wise multiplication, a non-linear activation function. The read and write operations are motivated by biological synaptic computation: Reading corresponds to the non-linear response of the neuron population to a specific activation pattern. Writing consists in an outer product that emulates a local Hebbian rule between neurons. The element- wise weighting allows locally tuning synaptic plasticity at every connection consistent with biology 43 44 which generates matrix update with potentially more than rank one as in the classic hebbian rule.
Recursive update: While single iteration of these two operations can only retrieve similar activation pattern (reading) or add unfiltered external information into weights (writing), recursively applying these operations offers a much more potent computational mechanism that mix information between current neural activation and previous iterates. Starting from an initial activation pattern (0) and previous weight state (0)) Wi-1, the model recursively applies equations (5) for [1 on (8)) and ((1) such that:
(+)) =
s-1
Σ
Ks (
(1)
+ K (s) di ((-1)), (o vlo-1))
for s E [] :
1-0
(6)
( Σ
3 W + /²:²) (((l-11))

---

Parameters K.s and d are scalar values learnt along with plasticity parameters a, and correspond to delayed contributions of previous patterns and synaptic states to the current operations. This is motivated by biological evidence of complex cascades of temporal modulation mechanisms over synaptic efficacy [45] 46]. Finally, (v($), (()) are respectively used as activation for the next layer, and as the new synaptic state W..
o Computational interpretation: We note that if S=1 in equation (6) the operation boils down to a simple hebbian update with synapse-specins weighting a² This perspective makes MetODS an original form of modern Hopfield network [47] with hetero-associative memory that can dynamically access and edit stored representations driven by observations, rewards and actions. While pattern retrieval from Hopfield networks has a dense litterature, our recursive scheme is an original proposal to learn automatic updates able articulate representations across timesteps. We believe that this mechanism is particularly benefitial for meta-RL to filter external information with respect to past experience. The promising results shown in our experimental section suggest that such learnt updates can generate useful self-modifications to sequentially adapt to incoming information at runtime.
In this work, test single dynamic layer for MetODS and leave the extension of the synaptic plasticity to the full network for future work.
Algorithm 1: MetODS synaptic learning
1: Require: 0 [f 9. a, k. B] and Wo
2: for I 1st S T do
3: v(0) f- f( rt-1)
In order for the model to
4: W Wi-1
learn credit assignment
5: for 1 1 8 < S do
strategy, state transition and previous reward informa- tion (2,, a:-1, rt-1] are em- bedded into a vector 2, by a feedforward map f as in previous meta-RI. ap- proaches 48 49. Action
o(s)_
Kg (1) (1) + kb
(6) (ooo
( + Σ B w()) + + (a w(0 8 o(o-1))
end for
9: ae, + g((
10: W,+ + (+)
and advantage estimate are read-out by feedforward
11: end for
policy map g. We meta-learn the plasticity and update coefficients, as well as the embedding and read-out function altogether: 0 = [f.g, a, K. B.. Additionally, the initial synaptic configuration Wo can be learnt, fixed priori or sampled from a specified distribution.
4 Related work
learning has recently flourished into several different approaches aiming at learning high-level strategies for capturing task rules and structures. A direct line of work consists in automatically meta-lcarning components or parameters of the RL arsenal to improve over heuristic settings (50 Orthogonally, work building on the Turing-completeness of recurrent neural networks has shown that simple recurrent neural networks can be trained to store past information in their persistent activity state to inform current decision, in such a way that the network implements a form of reinforcement learning over each episode [53 48 54] It is believed that vanilla recurrent networks alone are not sufficient to meta-learn the efficient forms of episodic control found in biological agents [55] 13. Hence additional work has tried to enhance the system with better or by modeling policy as an attention module over an explicitely stored set of past events 49 Optimization based approaches have tried to cast episodic adaptation
episodic memory model [56 58
as an explicit optimization procedure either by treating the optimizer as a black-box system (59] 60] or by learning synaptic configuration such that one or few gradient steps are sufficient to adapt the input/output mapping to a specific task [1].
Artificial fast plasticity: Networks with dynamic weights that can adapt function of neural activation have shown promising results over regular recurrent neural networks to handle sequential 33]. However, contrary to our work, these models postulate a persistent neural activity orchestrating weights evolution. On the contrary, we show that synaptic states are the sole persistent components needed to perform fast adaptation. Additionally, the possibility of optimizing synaptic dynamics with evolutionary strategies in randomly initialized networks [64] or through gradient descent [65] has been demonstrated, as well as in time-continuous setting
data 62 29
31
63
32
5

---

[66]. Recent results have shown that plasticity rules differentially tuned at the synapse level allow dynamically edit and query networks memory [3 67 However another specificity of this work is that our model synaptic rule is function of reward and synaptic state, allowing to drive weight dynamics conditionally on both an external feedhack signal and the current model belief.
Associative memory: As discussed above, efficient memory storage and manipulation is a crucial feature for building rapidly learning agents. To improve over vanilla recurrent neural network policies 48]. some models have augmented recurrent agents with content-addressabee
dictionaries able to reinstate previously encoded patterns given the current state [68 70 [3]. However these slot-based memory systems are subject to interference with incoming inputs and their memory cost grows linearly with experience. Contrastingly, attractor networks can be learnt to produce fast compression of sensory information into a fixed size tensorial representation [71] [2]. One class of such network are Hopfield networks 73 which benefit from large storage capacity [76], can possibly perform concept binding 77 34 and produce fast and flexible information retrieval 47
Experiments
In this section, we explore the potential of our meta-learnt synaptic update rule with respect to the three properties of the meta-RL problem exposed in section 2 Namely, 1) efficiency, 2) capacity and 3) generality of the produced learning algorithm. We compare with three state-of-the-an meta-RI. models based on different adaptation mechanisms: RI. [48 memory-based algorithm based on training a GRU-cell to perform Reinforcement Learning within the hidden space of its recurrent unit, MAML [61] which performs online gradient descent on the weights of three-layer MLP and PEARL (78), which performs probabilistic task inference for conditioning policy, Details for each experimental setting are further discussed in S.I. and the code can be found at https
//github. httpe//ww...oo/meaieecaaaaaaeeee
5.1 Efficiency: One-shot reinforcement learning and rapid motor control
Harlow Beraar training curve
2.
X
y
METOOS-
50
150
350
200
250
500
d)
AntDir- Reward
Antrr- Deline Adachation
700
R.2
R12
?
620
230
100
50
75
150
3225
150
175
200
Figure 3: a-b) Schemas of the Harlow and Mujoco Ant-directional locomotion task. c-d) Evolution of accumulated reward over training. number recursive iterations (S=1) or removing the trainable plasticity weights a resulting in sub-optimal policy. In Ant-dir we compare our agent training profie against MAMI. and RI. e) We can interpret the learned policy in terms of Hopfield energy adapting with experience. We show horizontally two reward profiles of different episodes and the energy Ew, (w, 22) -Di W.y along two principal components of the vector trajectory we In first episode, the error in the first presentation (red square) transforms the energy landscape which changes the agent policy, while on the other episode, the model belief does not change over time. Note the two modes for every energy map. which allows the model to handle the potential position permutation of the presented values. f) Average rewards per timestep during single episode of the Ant-dir task.
Harlow task, we conduct an ablation study by either reducing

---

To first illustrate that learnt synaptic dynamics can support fast behavioral adaptation, we use classic experiment from the neuroscience literature originally presented by Harlow 77 and recently reintroduced in artificial meta-RI. in [54 well as a heavily-henchmrred MuloCo directional locomotion task (see Fig- 3 To behave optimally in both settings, the agent must quickly identify the task rule and implement a relevant policy: The Harlow task consists of five sequential presentations of two random variables placed on a one-dimensional line with random permutation of their positions that an agent must select by reaching the corresponding position. One value is associated with positive reward and the other with a negative reward. The five trials are presented in alternance with periods of fixation where the agent should return neutral position between items. In the MuJoCo robotic Ant-dir experiment, a 4-legged agent must produce a locomotive policy given a random rewarded direction.
Since values location are randomly permuted across presentations within one episode, the learner cannot develop mechanistic strategy to reach high rewards based on initial position. Instead, to reach the maximal expected reward over the episode, the agent needs to perform one-shot learning of the task-contingent association rule during the first presentation. We found that even very small network of N=20 neurons proved to be sufficient to solve the task perfectly. We investigated the synaptic mechanism encoding the agent policy. A principal component analysis reveals policy differentiation with respect to the initial value choice outcome supported by synaptic specialization in only a few time-steps (see Figure[])) We can further interpret this adaptation in terms of sharp modifications of the Hopfield energy of the dynamic weights (FigureB)o Finally, the largest synaptic variations measured by the sum of absolute synaptic variations occur for states that carry a non-null reward signal (see S.I). These results suggest that the recursive hebbian update combined with reward feedback is sufficient to support one-shot reinforcement learning of the task association rule.
Harlow:
MuJoCo Ant-dir: We trained models for performing the task on an episode of 200 timesteps and found that MetODS can adapt in a few time-steps, similar to memory-based models such as RL2, (Figure thanks to its continual adaptation mechanism. By design, MAML and PEARL do not present such property, and they need multiple episodes before being able to perform adaptation correctly. We still report MAML performance after running its gradient adaptation at timestep t=100. We further note that our agent overall performance in single episode is still superior to MAML performance reported in 61 49 when more episodes are accessible for training.
5.2 Capacity: Maze exploration task
a)
b)
10a
194
cum Rawards
Cum. Rewards
101
0.5 1.0
2.0 Erv. steps (xe2)
3.5
255
3.0
170
0.5
3.0 1.5 2.0
23
Env. staps (x1e²)
Figure 4: a) Examples of generated maze configurations. The mazes consist in 8 x 8 pix. areas, with walls and obstacles randomly placed according to a variation of Prim's 's algorithm [80] with target location randomly selected for the entire duration of the episode (star). The agent receptive field is highlighted in red. b) Comparisons with MAML. and RI.² and effect clement-wise plasticity a. e) Variations of MetODS writing mechanism as well as depth S of recursivity yield different performances (see. S.1 for further details)
We further tested the systematicity of MetODS learnt reinforcement program on a partially observable Markov decision process (POMDP): An agent must locate a target in a randomly generated maze while starting from random locations and observing a small portion of its environment (depicted in Figure 4. While visual navigation has been previously explored in meta-RL focus on the mnemonic component of navigation by complexifying the task in two ways, we reduce the agent's s visual field small size of 3x3 pix. and randomize the agent's position after every reward encounter. The agent can take discrete actions in the set (up.down.left.ightt) which moves
48 49 31],
we here
7

---

it accordingly by one coordinate. The agent's reward signal is solely received by hitting the target location, thus receiving reward of 10. Each time the agent hits the target, its position is randomly reassigned on the map (orange) and the exploration resumes until 100 steps are accumulated during which the agent must collect as much reward as possible. Note that the reward is invisible to the agent, and thus the agent only knows it has hit the reward location because of the activation of the reward input. The reduced observability of the environment and the sparsity of the reward signal (most of the state transitions yield no reward) requires the agent to perform logical binding between distant temporal events to navigate the maze. Again, this setting rules out PEARL since its latent context encoding mechanism erases crucial temporal dependencies between state transitions for efficient exploration. Despite having no particular inductive bias for efficient spatial exploration or path memorization, a strong policy emerges spontaneously from training.
Agent Random
1st rew." (1)
Success (t)
Cum. Rew. (t)
Cum. Rew (Larger) (1)
96,8 + 0.5
5 %
3.8 + 8.9
3.7 + 6.4
MAML
64.3 + 39.3
45.2%
14.95 = 4.5
5.8 + 10.3
RL
16.2 + 1.1
96.2%
77.7 4:6.5
28.1 + 29.7
MetODS
14.7 = 1.4
96.6%
86.5 + 46.8
34.9 = 34.9
Figure 5: Performance of Meta-RL. models tested at convergence (Ic" env. steps). MctODS better explores the maze as measured by the average number of steps before Ist reward and the success rate in finding the reward at least once. It then better exploits the maze as per the accumulated reward. (* We assign 100 to episodes with no reward encounter.)
Ablation study and variations: We explored the contribution of the different features combined in MetODS update rule, showing that they all contribute to the final performance of our meta-learning model. First, we tested the importance of the element-wise tuning of plasticity in weight-based learning models and note that while it adversely affects MAML gradient update, it greatly improves MetODS performance, suggesting different forms of weights update. Second, we verified that augmenting recursivity depth S was beneficial to performance, consistently with Harlow results. Third, we noted that transforming the rightmost vector of the writing equation in with a linear projection (Linear in figure 4 see S.I for full experimental details) yields major improvement while non-linear (MLP) does not improve performance. Finally, we additionally test the capability of the learnt navigation skills to generalize to a larger maze size of 10 x 10 pix. unseen during training. We show that MetODS is able to retain its advantage (see figure and table for results).
5.3 Generality: Motor control
Finally, we test the generality of the reinforcement learning program learnt by our model for different continuous control tasks:
World: First, we use the dexterous manipulation benchmark proposed in 8 using the benchmark suite [82), in which Sawyer robot is tasked with diverse operations. full adaptation episode consists in N=10 rollouts of 500 timesteps of the same task across which dynamic weights are carried over. Observation consists in the robot's joint angles and velocities, and the actions are its joint torques. We compare MetODS with baseline methods in terms of meta-traine veta-ttaining and meta-testing success rate for 3 settings, push, reach and ML-10. We show in Fig. 6 the meta-training results for all the methods in the MetaWorke environments. Due to computational resource constraints, we restrict our experiment to budget of 10M steps per run. While we note that our benchmark does not reflect the final performance of previous methods reported in [81] at 300M steps, we note that MetODS test performance outperforms these methods early in training and keeps improving at 10M steps, potentially leaving room for improvement with additional training. (see S.I for additional discussion). Finally, we note that all tested approaches performed modestly on ML for test tasks, which highlights the limitation of current methods. We conjecture that this might be due to the absence of inductive bias for sharing knowledge between tasks or fostering systematic exploration of the environment of the tested meta-learning algorithms.
We also tested the robustness of MetODS learnt reinforcement programs by evaluating the agent ability to perform in a setting not seen during training: specifically, when partially impairing the agent motor capabilities. We adopt the same experimental setting as section 5.1 for the Ant and Cheetah robots and evaluate the performance when freezing of the robots
Robot impairment:
8

---

torque. We show that our model policy retains a better proportion of its performance compared to other approaches. These results suggest that fast synaptic dynamics are not only better suited to support fast adaptation of a motor policy in the continuous domain, but also implement a more robust reinforcement learning program when impairing the agent motor capabilities.
Hil-reach-v2
NL1-pushw2
MI.10
Mejoco Robot impairment
Irip.
Cheetal
Cheeten inih
METODS
3000L
PEAAL
Figure 6: Left Meta- Meta-training results for MetaWorld benchmarks. Subplots show tasks success rate over training timesteps. Average meta-test results for MetODS shown in dotted line. Right Cumulative reward of the Ant Cheetah directional locomotion task. each condition, results are normalized against the best performing policy
6 Discussion
In this work, we introduce a novel meta-RL system, MetODS, which leverages a self-referential weight update mechanism for rapid specialization at the episodic level. Our approach is generic and supports discrete and continuous domains, giving rise to a promising repertoire skills such as one- shot adaptation, spatial navigation motor coordination. MetODS demonstrates that locally tuned synaptic updates whose form depends directly on the network configuration can be meta-learnt for solving complex reinforcement learning tasks. We conjecture that further tuning the hyperparameters as well as combining MetODS with more sophisticated reinforcement learning techniques can boost its performance. Generally, the success of the approach provides evidence for the benefits of fast. plasticity in artificial neural networks, and the exploration of self-referential networks.
7 Broader Impact
Our work explores the emergence of reinforcement-leamingg programs through fast synaptic plas- ticity. The proposed pieces of evidence that information memorization and manipulation as well as behavioral specialization can be supported by such computational principles 1) help question the functional role of such mechanisms observed in biology and 2) reaffirms that alternative paradigms to gradient descent might exists for efficient artificial neural network control. Additionally, the proposed method is of interest for interactive machine learning systems that operates in quickly changing environments and under uncertainty (bayesian optimization, active learning and control theory). For instance, the meta RL approach proposed in this work could be applied to brain-computer interfaces for tuning controllers to rapidly drifting neural signals. Improving medical applications and robot control promises positive impact, however deeper theoretical understanding and careful deployment monitoring are required to avoid misuse.
Acknowledgments and Disclosure of Funding
This work was supported by ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR- 19-P13A-0004), OSCI-DEEP ANR (ANR-19-NEUC-00U) ONR (N00014-19-1-2029), and NSF (11--1922200 and EAR-1925481). Additional support provided by the Carney Institute for Brain Science and the Center for Computation and Visualization (CCV) via NIH Office of the Director grant S 1100D025181. The authors would like thank the anonymous reviewers for their thorough comments and suggestions that led to an improved version of this work.

---

References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
[2] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 518(7540):529-533 2015.
[3] Max Jaderberg, Wojciech M. Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castañeda, Charles Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuoglu, and Thore Graepel. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443))859-8635 2019.
[4] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1): (334)133333373, 2016.
[5] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra, Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
[6] Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel Gershman, Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017. 2017..
[7] Brenden Lake and Marco Baroni. Still not systematic after all these years: On the compositional skills of sequence-to-sequens recurrent networks, 2018.
[8] Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning, 2020.
[9] Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering. Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement. Machine Learning, 28, 01 1997.
[10] Sebastian Thrun. Lifelong Learning Algorithms, pages 181-209. Springer US, Boston, MA, 1998.
(11) Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artif. Intell. Rev. 18(2):77-95, oct 2002.
[12] Jeff Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. CoRR, abs/1905.10985, 2019.
[13] Matthew Botvinick, Sam Ritter, Jane X. Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in Cognitive Sciences, 23(5):408 422, 2019.
[14] S. J. Martin, P. D. Grimwood, and R. G. M. Morris. Synaptic plasticity and memory: An evaluation of the hypothesis. Annual Review of Neuroscience, 23(1):649-711, 2000.
[15] LF Abbott and Wade Regehr. Synaptic computation. Nature, 431(70100:796-883, 2004.
[16] Wade G Regehr. Short-term presynaptic plasticity. Cold Spring Harbor perspectives in biology, 4(7):a005702, 2012.
[17] Natalia Caporale and Yang Dan. Spike -eepenent plasticity: hebbian learning rule. Annual Review of Neuroscience, 31(1):25 2008. PMID: 18275283.
[18] Gianluigi Mongillo, Omri Barak, and Misha Tsodyks. Synaptic theory of working memory. Science, 14465869)115431155 2008
[19] Nicolas Masse, Guangyu Yang, H. Song, Xiao-Jing Wang, and David Freedman. Circuit mechanisms for the maintenance and manipulation of information in working memory, 04 2018.
10

---

[20] Omni Barak and Misha Tsodyks. Working models of working memory. Current Opinion in Neurobiology, 25:20-24, 24002242222 2014. Theoretical and computational neuroscience.
[21] Mark G. Stokes. "activity-silent" working memory in prefrontal cortex: a dynamic coding framework. Trends in Cognitive Sciences, 19(7):394 405, 2015.
[22] Sanjay G. Manohar, Nahid Zokaei, Sean J. Fallon, Tim P. Vogels, and Masud Husain. Neural mechanisms of attending to items in working memory. Neuroscience and Biobehavioral Reviews, 101:1-12, 2019.
[23] Pierre Yger, Marcel Stimberg, and Romain Brette. Fast learning with weak synaptie plasticity. Journal of Neuroscience, 35(39):13351-13366 2015.
[24] Räzvan V Florian. Reinforcement learning through modulation of teakee-tminn-deeppeee synaptic plasticity. Neural computation, 19(6): 146--1502, 2007.
[25] Geoffrey E Hinton and David C Plaut. Using fast weights to deblur old memories. In Proceedings of the 9th Annual Conference of the Cognitive Science Society, pages 177-186, 1987.
[26] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139) 1992.
[27] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4:131-139, 1992.
[28] Christoph von der Malsburg. The Correlation Theory of Brain Function, pages 95-119. Springer New York, New York, NY, 1994.
[29] David Ha, Andrew Dai, and Quoe V Le. Hypernetorrs.. arXiv preprint arXiv:1609.0911 2016.
[30] Jimmy Ba, Geoffrey Hinton, Volodymyr Mnih, Joel Z Leibo, and Catalin Ionescu. Using fast weights to attend to the recent past. Advances in Neural Information Processing Systems, 29:4331-4339, 2016.
[31] Thomas Miconi, Kenneth Stanley, and Jeff Clune. Differentiable plasticity: training plastic neural networks with backpropagation. In International Conference Machine Learning. pages 3559-3568. PMLR, 2018
[32] Imanol Schlag and Jürgen Schmidhuber. Learning to reason with third-order tensor products. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 10003-10014. 2018
[33] Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jürgen Schmidhuber. Going beyond linear transformers with recurrent fast weight programmers. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[34] Imanol Schlag, Tsendsuren Munkhdalai, and Jürgen Schmidhuber. Learning associative in- ference using fast weight memory. In International Conference on Learning Representations, 2020.
[35] Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In International Conference on Machine Learning, pages 9301- 9312. PMLR, 2021.
[36] Luckeciano C Melo. Transformers are meiforeeinorreemett learners. In International Conference on Machine Learning, pages 15340-15359) PMLR, 2022.
[37] Jane X Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: 1611,06611..766 2016.
[38] Shin Ishii, Wako Yoshida, and Junichiro Yoshimoto. Control of meta-parameter in reinforcement learning. Neural networks, 15(4-6):665-687 2002.
11

---

[39] Luisa Zintgraf. Kyriacos Shiarlis, Maximillan Igl, Sebastian Schulze, Yarin Gal. Katja Hofmann, and Shimon Whiteson. Varibad: very good method for bayes-adaptive deep via meta- learning. arXiv preprint arXiv: 1xXiv:99100084453 2019.
[40] Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, al. Meta-learning of sequential strategies. arXiv preprint arXiv: 030300 2019.
[41] Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A. Ortega, Yee Whye Teh, and Nicolas Heess, Meta reinforcement learning as task inference. CoRR, abs/1905.06422 2019.
[42] Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream control: 2019.
Learning behaviors by latent imagination. CoRR,
[43] Larry F Abbott and Sacha B Nelson. Synaptic plasticity: taming the beast. Nature neuroscience, 3(11):1178 101111888113331 1183, 2000.
[44] Tuning into diversity of homeostatic synaptic plasticity. Nenropharmacolooy, 78:31-37, 2014, Homeostatic Synaptic Plasticity.
[45] Chun Yun Chang. Impact of second messenger modulation on activity-dependent and basal properties of excitatory synapses. 2010.
[46] Rylan S. Larsen, Ikuko T. Smith, Jayalakshmi Miriyala, Ji Eun Han, Rebekah J. Corlew, Spencer L. Smith, and Benjamin D. Philpot. Synapse-specific control of experience-dependent plasticity by presynaptic nmda receptors. Neuron, 83(4):877-893, 2014.
[47] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael Kopp, et al. Hopfield networks is all you need. In International Conference on Learning Representations, 2020.
[48] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint 111.77161..02779. 2016.
[49] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018.
[50] Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 5405-5414, 2018.
[51] Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 2402-2413, 2018.
[52] Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. Advances in Neural Information Processing Systems, 31:5302-5311, 2018.
[53] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pages 87-94. Springer, 2001.
[54] Jane X. Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforreemen learning system. 2018.
[55] Máté Lengyel and Peter Dayan. Hippocampal contributions to control: The third way. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2008
[56] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmented neural networks. In International conference on machine learning, pages 1842-1850, PMLR, 2016.
12

---

[57] Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Conference Machine Learning, pages 2827-2836. PMLR, 2017.
[58] Samuel Ritter, Jane X Wang, Zeb Kurth-Nelson, and M Botvinick. Episodic control as meta- reinforcement learning. bioRxiv, page 360537, 2018.
[59] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmen tional Conference on International Conference on Machine Learning- Volume 48, ICML' 16, page 1842-1850. JMLR.org, 2016.
neural networks. In Proceedings of the 33rd Interna-
[60] Sachin Ravi and H. Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.
[61] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR, 06-11 Aug 2017.
[62] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pages 2554-2563 PMLR, 2017.
[63] Thomas Miconi, Aditya Rawal, Jeff Clune, and Kenneth Stanley. Backpropamine: training self- gelf-modifying neural networks with differentiable neuromodulated plasticity. In International Conference on Learning Representations, 2018.
[64] Elias Najarro and Sebastian Risi. Meta-learning through hebbian plasticity in random networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20719-20731. Curran Associates, Inc., 2020.
[65] Thomas Miconi. Learning to learn with backpropagation of hebbian plasticity, arXiv: Neural and Evolutionary Computing, 2016.
[66] Krzysztof Choromanski, Jared Davis, Valerii Likhosherstov, Xingyou Song, Jean-Jacques Slotine, Jacob Varley, Honglak Lee, Adrian Weller, and Vikas Sindhwani. An ode to an ode. ArXiv, abs/2006. 1421, 2020.
[67] Mathieu Chalvidal, Matthew Ricci, Rufin VanRullen, and Thomas Serre. Go with the flow:
Adaptive control for neural ODEs. In International Conference on Learning Representations, 2021.
[68] Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks arXiv preprint arXiv:1410.388 2014.
[69] Wojeiech Zaremba and Ilya Sutskever. Reinforcement learning neural turing machines revised, 2016.
[70] Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adrià Puigdomênech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2827-2836. PMLR, 06-11 Aug 2017.
[71] Sergey Bartunov, Jack Rae, Simon Osindero, and Timothy Lillicrap. Meta-learning deep energy-based memory models. In International Conference on Learning Representations, 2019.
[72] Wei Zhang and Bowen Zhou. Learning to update auto-associative memory in recurrent neural networks for improving sequence memorization, 2017.
[73] J J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences, 2955:2554-2 1982.
[74] Pascal Koiran. Dynamics of discrete time, continuous state hopfield networks. Neural Compu- tation, 6(3):459-468, 1994.
13

---

[75] Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 02 2017.
[76] Dmitry Krotov and John Hopfield. Dense associative memory for pattern recognition. In NIPS, 2016.
[77] Imanol Schlag and Jürgen Schmidhuber. Learning to reason with third order tensor products. In NeurlPS, pages 10003-10014, 2018.
[78] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.
[79] Harry Frederick Harlow. The formation of learning sets. Psychological review, 56 1:51-65, 1949.
[80] Robert Clay Prim. Shortest connection networks and some generalizations. The Bell System Technical Journal, 36(6):1389-14011 1957.
[81] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning (CoRL), 2019.
[82] The garage contributors. Garage: A toolkit for reproducible reinforcement learning research. https: /tttp:://iithub.... wwt.cooooollooe www.pub..oooooooooorrrooo/////// ggarge 2019.
[83] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal represen- tations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
[84] Michael Betancourt, Charles C Margossian, and Vianey Leos-Barajar Barajas. The discrete ad- joint method: Efficient derivatives for functions of discrete sequences. arXiv:2002.00336. 2020.
arXiv preprint
[85] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference machine learning, pages 1928-1937. PMLR, 2016.
[86] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[87] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation, 2018
14


Reviews:
[{"review_id": "J-yjRplRa8G", "reviewer": "Reviewer_7zkD", "summary": "This paper presents a new neural network that is able to modify its own weights, known as the self-modifying network.\nBy incorporating self-modifying networks into the meta reinforcement learning (RL) framework, the new model Meta-Optimized Dynamical Synapses (MetODS) shows better empirical performance in terms of efficiency, capacity, and generality, compared to several meta RL baseline algorithms.", "questions": "What neural network structures are used for baseline algorithms such as MAML and RL2?", "limitations": "I may misunderstand something here, but in equation 5, it seems that computing $v^{(s)}$ and $W^{(s)}$ requires whole trajectories of $v^{(l)}$ and $W^{(l)}$ to be stored, which leads to a large memory requirement.\nThe discrete adjoint sensitivity method only helps with the backward pass but not the forward pass (i.e. computing $v^{(s)}$ and $W^{(s)}$). How about the forward pass?", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work is well motivated. The proposed algorithm MetODS shows significantly better performance across different kinds of tasks, from maze navigation to motor control.\n\nNevertheless, It is hard for me to understand the reasons behind the success of self-modifying networks and MetODS.\nIn Section 3, it is claimed that \"Our model learns to train itself by updating its weights through interaction with the environment and its own current weight state. This mechanism enables MetODS to rapidly compress experience of a task $\\tau$ into a particular synaptic configuration ......\" Arguably, an RL model using traditional neural networks optimized with SGD also update its weights through interactions with the environment, based on its current weights. What is the critical difference between self-modifying networks and classical neural networks? More explanations and analytical experiments are needed to show the advantage of self-modifying networks.\n\nSmall typos:\n- Line 48, conjonction --> conjunction.\n- Line 11 in Algorithm 1.\n- Missing one citation in Line 296. \n- Missing x & y labels in Figure 6.\n\nFinally, I do not find Figure 1 very helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3KVlDNW2XWv", "reviewer": "Reviewer_aCER", "summary": "Post-response update\n----------------\n\nThe authors have improved the paper and addressed some of my concerns; while I still find the assessment to be problematic in some areas, I think the present results, together with the interesting idea, make for a paper that could be a useful contribution to the field. I have updated my score accordingly.\n\nOriginal review\n--------------\n\nThis work presents an interesting and fairly novel perspective on meta-RL: having an agent adapt to a task by modifying its weights through an iterated Hebbian update process. The paper demonstrates that this approach outperforms vanilla baselines (RL^2, MAML, and sometimes PEARL) across a variety of settings that stress different aspects of adaptation. This is overall an intriguing approach that I hope to see published either here or in the future.\n", "questions": "See above. Briefly:\n* Which aspects of each experiment might be unfair to the baselines? Would the baselines outperform the proposed algorithm if the experiments were run for longer?\n* Which aspects of the approach are necessary (justified with ablation experiments)?\n* Can the authors justify the policy transport perspective further, e.g. by comparing to the baselines with this perspective?\n\n", "limitations": "See above. Briefly:\n* Unless I am misunderstanding something, the experiments are a limitation because the baselines are handicapped (e.g. by far less training).\n* The conclusions we can draw from the paper are limited by the lack of ablation experiments, to identify which aspects of the approach are essential.  \n* The authors mention briefly that they leave extending their plasticity rule to multiple layers to future work. Of course, Hebbian rules don't tend to work well for training deep networks, and this point might deserve slightly more emphasis (e.g. a brief sentence in the discussion to highlight the challenge). ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\nThe idea is interesting—it’s nice to see a fresh perspective on adaptation drawing on dynamical systems perspectives from neuroscience.\nThe breadth of demonstrations is fairly compelling; it’s great that the authors articulate different aspects of the meta-RL problem and test each of them.\nThe results on impared robots and with continual adaptation (in the appendix) are particularly intriguing, because they suggest an increased robustness inherent in this approach; however, this difference would be more compelling if the comparison algorithms were trained to matching train performance (see below). \n\n\nWeaknesses (in order of importance):\n* Some of the evaluation choices seem like they could unfairly handicap the baselines; therefore I am not sure how much to trust the overall conclusions of the paper. \n  - The authors limit to 10M steps for the MetaWorld ML1 and ML10 environments; but the original MetaWorld paper achieved much higher performance with RL^2 and MAML after much longer training (300M steps). So the evaluation seems to artificially handicap the baselines, by training for much less time than the baseline approaches require. Indeed, in ML1-reach and ML10 the curves for one of the baselines appear to be passing the proposed MetODS approach just where the plot ends. Would the baselines perform better than MetODS if the experiments were run for longer?\n  -The authors limit their approach to a single layer of adaptable weights—did they make a similar limitation for MAML? I wasn’t entirely clear from either the main text or the supplement. If the baselines are restricted compared to the original work this limitation would be a major caveat that should be stated much more explicitly in the text. Algorithms like MAML are generally developed and tested with deep networks; it would not be appropriate to test them with a different architecture without doing full hyperparameter tuning; and even with hyperparameter tuning the conclusion should come with caveats if the authors are artificially limiting the baseline approaches. It’s possible that the authors are not doing this; I am happy to be corrected if so, but in either case I would suggest they be more explicit about precisely what the baseline approaches were in the paper when they revise it.\n  - Unless I’m misunderstanding both the above points, I find statements like “producing better agents than previous meta-RL approaches” or “compares favorably with prior meta-RL algorithms” to be somewhat misleading. Ideally the baseline experiments should be run using architectures comparable to the original papers, for as long as the original paper in at least a subset of the domains. At the very least the plots should show e.g. the max performance figures achieved in the original MetaWorld paper as a baseline comparison in each case (e.g. a star or dashed line with label like “Fully trained MAML” etc.), and the statements about comparisons (e.g. in intro and conclusions) should qualify that MetODS performs better “[for adapting a single layer of weights] at the beginning of training.”\n* MetODS consists of a complicated set of changes to the architecture; some ablation experiments should be performed to demonstrate the contributions of different changes. A few such changes are tested for Harlow (S=1 and no element-wise weights), but it would be ideal to see these on a few other tasks, as well as conditions like the following: \n  - It wasn’t entirely clear whether beta and kappa are element-wise parameters or scalar; assuming the latter, switching to the scalar version would be a useful ablation. \n  - What if the authors kept all aspects of MetODS but switched from the Hebbian update rule to an alternative rule. Ideally this would include something like a hypernetwork (see below), but if that is too computationally expensive there are a variety of changes that could be explored, e.g. instead of the updating based on the outer product of v with itself, what if learned linear projection weights from v to two new vectors q and p and then used a (weighted) outer product of q and p? This may not be the best experiment; the point is just that more exploration of which aspects of the update rule are important (e.g. whether the Hebbian update is somehow intrinsically useful)  would help future researchers understand what to explore.\n  - Adding features to the baselines could be useful too in determining where the real benefits are; e.g. if the authors used MAML with S (smaller) updates per timestep rather than 1, how does it compare? What if they augmented MAML with parameter-specific meta-learned update weighting and smoothing constants alpha and beta? Etc.\n* I found the connection to the policy transport perspective somewhat unsatisfying. A substantial portion of the paper (~1.5 pages) is spent presenting this perspective, but it seems like the efficiency, capacity, and generality can be (and have been) previously defined without adopting this perspective.\n  - I believe that the policy transport perspective could be removed or substantially reduced; indeed, I think that this would perhaps make the paper clearer by focusing more directly on the algorithmic contributions.\n  - The three concepts defined have prior precedents in the literature; for example the notion of cumulative regret has a long history in RL, has often been used in meta-RL (e.g. Wang et al., 2016), and is effectively a measure of efficiency. The paper should connect these concepts to the relevant prior ideas.\n  - If the authors include the policy perspective in the paper, it would be ideal to *compare* their approach to the baselines with respect to policy transport. For example, if they visualized policy transport for other approaches (e.g. MAML) like they do for their own approach in Fig. 2C, and then *quantified* the advantage of their approach in terms of policy transport (e.g. showing that their method takes a more direct path in policy space than the baselines), that would be much more compelling.\n  - As one additional note of interest, the policy transport perspective reminded me a bit of a recent paper I saw on how fast policies update with a gradient update to the weights (Schaul et al., 2022); it seems potentially relevant to thinking about the policy transport perspective, although it is not primarily focused on optimal changes in the policy so much as noisy ones (to my understanding; I’ve not actually read Schaul et al. yet).\n* The discussion of the prior literature seemed a little skewed to me in places; while it is true that few prior approaches have pursued fast weights “as a function of the current synaptic state or external reward signals” per se, there has been work on using fast weights in Meta-RL (e.g. Sarafian et al., 2021) that probably should be discussed as relevant background.\n\nThe authors do not necessarily have to run every experiment suggested above (which might be infeasible) for me to consider the paper publication worthy. However, the paper would need to would need to compare to stronger baselines (including the original MetaWorld results, if it is computationally infeasible to train their own strong baselines), would need to more clearly describe the experiments they performed, and would need to outline the limitations of the experiments and approach in more detail.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-c2alWfalf", "reviewer": "Reviewer_dG2z", "summary": "The authors propose MetODS: Meta Optimized Dynamical Synapses, a meta-RL algorithm that learns by updating its weights through interaction with the environment and its own current weight state. The update rule is recursively applied and allow the algorithm to learn relations between stored patterns and incoming information. MetODS notably uses an element-wise weighting, that allows for different plasticity amplitudes at every connection. The authors then experimentally compare their algorithm to RL2 and MAML and analyze the performances of their algorithms against these baselines in terms of efficiency (One-shot learning and rapid motor control), capacity (achievable level of performance for a distribution of task) and generality (how well the policy transfers to tasks unseen during training).  ", "questions": "What is the (high level) intuition behind the \"read\" and \"write\" operation of the update rule ?\n\nwriting suggestion/typos:\nl 99 and l 296: broken citation\nl 139: Synapses (with capital S)\nl 174: s instead of S \nl 290: \"to\" is repeated", "limitations": "I haven't seen any discussion on potential negative societal impact. I, however, cannot really find non-generic remarks on this topic for this work. ", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well motivated by biological and existing artificial neural methods. It clearly presents a novel meta RL approach and compares its performances to Meta RL baselines (namely RL2 and MAML). While the claim on efficiency and generality seems to me well-supported by the experimental evaluation, I have difficulties to understand how the evaluation on the randomly generated grid world helps to evaluate the capacity of the learner. Are different generation patters used between training and evaluation? \nOtherwise, I think that the authors could sometimes add high level motivational details on the design choices of the algorithm (notably on the multi-step scheme).  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept