Learning reflexive weights updates
What if a neural agent could adjust the rule driving the evolution of \mu^{\bm{\theta},\tau,t}_{\pi} based on its own knowledge? In this work, we test a neural computational model that learns to compress experience of a task \tau into its dynamic weights \bm{W}_{t} .

\forall t\leq T,\hskip 14.45377pt\pi(\bm{a}|\bm{s},\bm{W}_{t})\sim\mu^{\bm{\theta},\tau,t}_{\pi} (3)

Contrary to gradient-based rules whose analytical expression is a static and predefined function of activations error ( \Delta(\bm{W}_{t})=\frac{\partial\bm{a}}{\partial\bm{W}}.\frac{\partial\mathcal{L}}{\partial\bm{a}} ), we define an update rule that directly depends on the current weight state through a parameterized non-linear recursive scheme, making the expression of the update much more sensitive to weight content itself.

\forall t\leq T,\hskip 14.45377pt\Delta(\bm{W}_{t})=\mathcal{F}_{\theta}(\bm{W}_{t})\Bigr{|}_{\bm{W}=\bm{W}_{t}} (4)

This self-reflexive evolution of the synaptic configuration \bm{W}_{t} allows for online variation of the learning rule during adaptation for a given task \tau and opens an original repertoire of dynamics for the policy evolution \mu^{\bm{\theta},\tau,t}_{\pi} . Specifically, at every time-step t , network inference and learning consists in recursive applications of read and write operations that we define below.

\diamond Read-write operations: The core mechanism consists in two simple operations that respectively linearly project neurons activation \bm{v} through the dynamic weights \bm{W} followed by an element-wise non-linearity, and build a local weight update with element-wise weighting \bm{\alpha} :

\begin{cases}\phi(\bm{W},\bm{v})=\sigma(\bm{W}.\bm{v})\quad\textit{read}\ \psi(\bm{v})=\bm{\alpha}\odot\bm{v}\otimes\bm{v}\quad\textit{write}\ \end{cases} (5)

Here, \bm{\alpha} is a matrix of \mathbb{R}^{N\times N} , \otimes denotes the outer-product, \odot is the element-wise multiplication, \sigma is a non-linear activation function.
The read and write operations are motivated by biological synaptic computation: Reading corresponds to the non-linear response of the neuron population to a specific activation pattern. Writing consists in an outer product that emulates a local Hebbian rule between neurons. The element-wise weighting \bm{\alpha} allows locally tuning synaptic plasticity at every connection consistent with biology [ 43 , 44 ] which generates a matrix update with potentially more than rank one as in the classic hebbian rule.

\diamond Recursive update: While a single iteration of these two operations can only retrieve a similar activation pattern (reading) or add unfiltered external information into weights (writing), recursively applying these operations offers a much more potent computational mechanism that mix information between current neural activation and previous iterates. Starting from an initial activation pattern \bm{v}^{(0)} and previous weight state \bm{W}^{(0)}=\bm{W}_{t-1} , the model recursively applies equations ( 5 ) for s\in[1,S] on \bm{v}^{(s)} and \bm{W}^{(s)} such that:

for s\in[1,S]\quad:\quad\begin{cases}\bm{v}^{(s)}=\sum\limits_{l=0}^{s-1}\bm{\kappa}_{s}^{(l)}\bm{v}^{(l)}+\bm{\kappa}_{s}^{(s)}\phi(\bm{W}^{(s-1)},\bm{v}^{(s-1)})\ \bm{W}^{(s)}=\sum\limits_{l=0}^{s}\bm{\beta}_{s}^{(l)}\bm{W}^{(l)}+\bm{\beta}_{s}^{(s)}\psi(\bm{v}^{(s-1)})\end{cases} (6)

Parameters \smash{\bm{\kappa}_{s}^{(l)}} and \smash{\bm{\beta}_{s}^{(l)}} are scalar values learnt along with plasticity parameters \bm{\alpha} , and correspond to delayed contributions of previous patterns and synaptic states to the current operations. This is motivated by biological evidence of complex cascades of temporal modulation mechanisms over synaptic efficacy [ 45 , 46 ] . Finally, (\bm{v}^{(S)},\bm{W}^{(S)}) are respectively used as activation for the next layer, and as the new synaptic state \bm{W}_{t} .

\diamond Computational interpretation: We note that if S=1 in equation ( 6 ), the operation boils down to a simple hebbian update with a synapse-specific weighting \bm{\alpha}^{i,j} . This perspective makes MetODS an original form of modern Hopfield network [ 47 ] with hetero-associative memory that can dynamically access and edit stored representations driven by observations, rewards and actions. While pattern retrieval from Hopfield networks has a dense litterature, our recursive scheme is an original proposal to learn automatic updates able to articulate representations across timesteps. We believe that this mechanism is particularly benefitial for meta-RL to filter external information with respect to past experience. The promising results shown in our experimental section suggest that such learnt updates can generate useful self-modifications to sequentially adapt to incoming information at runtime.

In this work, we test a single dynamic layer for MetODS and leave the extension of the synaptic plasticity to the full network for future work. In order for the model to learn a credit assignment strategy, state transition and previous reward information [\bm{s}_{t},\bm{a}_{t-1},\bm{r}_{t-1}] are embedded into a vector \bm{v}_{t} by a feedforward map \bm{f} as in previous meta-RL approaches [ 48 , 49 ] . Action and advantage estimate are read-out by a feedforward policy map \bm{g} . We meta-learn the plasticity and update coefficients, as well as the embedding and read-out function altogether: \bm{\theta}=[\bm{f},\bm{g},\bm{\alpha},\bm{\kappa},\bm{\beta}] . Additionally, the initial synaptic configuration \bm{W}_{0} can be learnt, fixed a priori or sampled from a specified distribution.