Introduction
The algorithmic shift from hand-designed to learned features characterizing modern Deep Learning has been transformative for Reinforcement Learning (RL), allowing to solve complex problems ranging from video games [ 1 , 2 ] to multiplayer contests [ 3 ] or motor control [ 4 , 5 ] . Yet, "deep" RL has mostly produced specialized agents unable to cope with rapid contextual changes or tasks with novel or compositional structure [ 6 , 7 , 8 ] . The vast majority of models have relied on gradient-based optimization to learn static network parameters adjusted during a predefined curriculum, arguably preventing the emergence of online adaptivity. A potential solution to this challenge is to meta-learn [ 9 , 10 , 11 ] computational mechanisms able to rapidly capture a task structure and automatically operate complex feedback control: Meta-Reinforcement Learning constitutes a promising direction to build more adaptive artificial systems [ 12 ] and identify key neuroscience mechanisms that endow humans with their versatile learning abilities [ 13 ] .

In this work, we draw inspiration from biological fast synaptic plasticity, hypothesized to orchestrate flexible cognitive functions according to context-dependent rules [ 14 , 15 , 16 , 17 ] . By tuning neuronal selectivity at fast time scales – from fast neural signaling (milliseconds) to experience-based learning (seconds and beyond) – fast plasticity can support in principle many cognitive faculties including motor and executive control. From a dynamical system perspective, fast plasticity can serve as an efficient mechanism for information storage and manipulation and has led to modern theories of working memory [ 18 , 19 , 20 , 21 , 22 ] . Despite the fact that the magnitude of the synaptic gain variations may be small, such modifications are capable of profoundly altering the network transfer function [ 23 ] and constitute a plausible mechanism for rapidly converting reward and choice history into tuned neural functions [ 24 ] . From a machine learning perspective, despite having a long history [ 25 , 26 , 27 , 28 , 29 ] , fast weights have most often been investigated in conjunction with recurrent neural activations [ 30 , 31 , 32 , 33 ] and rarely as a function of an external reward signal or of the current synaptic state itself. Recently, new proposals have shown that models with dynamic modulation related to fast weights could yield powerful meta-reinforcement learners [ 34 , 35 , 36 ] . In this work, we explore an original self-referential update rule that allows the model to form synaptic updates conditionally on information present in its own synaptic memory. Additionally, environmental reward is injected continually in the model as a rich feedback signal to drive the weight dynamics. These features endow our model with a unique recursive control scheme, that support the emergence of a self-contained reinforcement learning program.

Contribution: We demonstrate that a neural network trained to continually self-modify its weights as a function of sensory information and its own synaptic state can produce a powerful reinforcement learning program. The resulting general-purpose meta-RL agent called ’MetODS’ (for Meta-Optimized Dynamical Synapses) is theoretically presented as a model-free approach performing stochastic feedback control in the policy space. In our experimental evaluation, we investigate the reinforcement learning strategies implemented by the model and demonstrate that a single layer with lightweight parametrization can implement a wide spectrum of cognitive functions, from one-shot learning to continuous motor-control. We hope that MetODS inspires more works around self-optimizing neural networks.

The remainder of the paper is organised as follows: In Section 2 we introduce our mathematical formulation of the meta-RL problem, which motivates MetODS computational principles presented in Section 3 . In Section 4 we review previous approaches of meta-reinforcement learning and we discuss other models of artificial fast plasticity and their relation to associative memory. In Section 5 we report experimental results in multiple contexts. Finally, in Section 6 we summarise the main advantages of MetODS and outline future work directions.