Experiment details
General information: As specified in section 3 and 5 , we test a single model definition for all experiments in this work, with one layer of dynamic weights \bm{W}_{t} . This layer consists in a dense matrix of size \bm{n}\in\mathbb{N} with learnt or random initialization. Our model lightweight parametrization of the synaptic update rule makes it a very parameter efficient technique, which can perform batch-computation and be ported to GPU hardware to accelerate training. We refer readers to table 1 for specific details of each experiment presented in section 5. In addition to this pdf file, we provide the code for training METODS agents in experiments presented in section 5 at https:/

Feature - Experiment Harlow Gym Mujoco Maze Meta-World Dynamic layer size \bm{n} 20 100 200 100 Input size \bm{i} 12 134 (Ant) / 27 (Cheetah) 15 i Output size \bm{o} 2 8 (Ant)/ 6 (Cheetah) 4 8 Embedding \bm{f} [\bm{i}\times 32,\sigma,32\times\bm{n}] [\bm{i}\times 64,\sigma,64\times\bm{n}] [\bm{i}\times 32,\sigma,32\times\bm{n}] [\bm{i}\times 32,\sigma,32\times 64,\sigma,64\times\bm{n}] Read-out \bm{g} [\bm{n}\times 32,\sigma,32\times\bm{o}] [\bm{n}\times 64,\sigma,64\times\bm{o}] [\bm{n}\times 32,\sigma,32\times\bm{o}] [\bm{n}\times 64,\sigma,64\times\bm{o}] Non-linearities tanh tanh tanh tanh Init. \bm{W}_{0} \mathcal{N}(0,1e-3) Learned Learned Learned # Num. of episodes 5 (max) 1 1 10 Lenght of 1 episode 250 (max) 200 100 500 Meta-training Alg. A2C A2C A2C PPO learning rate 5e-4 1e-4 5e-4 5e-4 meta-batch-size 50 50 20 25 discount factor \lambda 9e-1 9e-1 9.9e-1 9.9e-1 gae 1. 9.5E-1 9.5e-1 9.5e-1 value function coeff. 4e-1 4e-1 4e-1 - entropy reg. factor 3e-2 3e-2 1e-2 1e-2

Meta-training algorithm: We show in our experiments, that the all meta-parameters \bm{\theta}=[\bm{\alpha},\bm{\kappa},\bm{\beta},\bm{f},\bm{g},] can be jointly optimized with two policy gradient algortihms. We use policy gradient methods to meta-train the synaptic parameters: Advantage Actor-critic algorithm (A2C) [ 85 ] where we consider that temporal dependancies are crucial to solve the task, and we use PPO [ 86 ] as a sequential policy optimization over fixed rollouts for the motor control experiment. Additionally, to reduce noise in policy gradient updates, we further show that it is possible to learn a dynamic advantage estimate of the Generalized Advantage Estimation (GAE) [ 87 ] and that we can meta-learn it as a second head of the MetODS layer output \bm{v}^{(s)} . This fact confirms that tuned hebbian-updates are also a sufficient mechanism to keep track of a policy value estimate.

Plasticity parameters: In all our experiments, \bm{\alpha} consists in a real-valued matrix of \mathbb{R}^{\bm{n}\times\bm{n}} initialized with independent normal distribution \mathcal{N}(\mu=0,\sigma=1e-3) . Similarly, multi-step weighting parameters (\smash{\bm{\kappa}_{s}^{(l)})_{l\leq s}} and (\smash{\bm{\beta}_{s}^{(l)})_{l\leq s}} can be stored as entries of triangular inferior matrices of \mathbb{R}^{\bm{S}\times\bm{S}} and are initialized with \mathcal{N}(\mu=0,\sigma=1e-2) .

Embedding and read-out: At each time-steps, inputs to the feed-forward embedding function \bm{f} consist in a concatenation of new observable/state \bm{s}_{t} as well as previous action and reward \bm{a}_{t-1} and \bm{r}_{t-1} . After adaptation procedure, policy is read-out from the last activation vector \smash{\bm{v}^{(S)}} by a feed-forward function \bm{g} which output statistics of a parameterized distribution in the action space (categorical with Softmax normalization in the discrete case or Gaussian in the continuous case). Both input and output mappings \bm{f} and \bm{g} consist in 2-layer Perceptrons with dense connections initialized with orthogonal initialization and hyperbolic tangent non-linearities.