Meta Learning Backpropagation And Improving It

1 Introduction . The shift from standard machine learning to meta learning involves learning the learning algorithm ( LA ) itself , reducing the burden on the human designer to craft useful learning algorithms [ 43 ] . Recent meta learning has primarily focused on generalization from training tasks to similar test tasks , e.g. , few-shot learning [ 11 ] , or from training environments to similar test environments [ 17 ] . This contrasts human-engineered LAs that generalize across a wide range of datasets or environments . Without such generalization , meta learned LAs can not entirely replace human-engineered variants . Recent work demonstrated that meta learning can also successfully generate more general LAs that generalize across wide spectra of environments [ 20 , 1 , 31 ] , e.g. , from toy environments to Mujoco and Atari . Unfortunately , however , many recent approaches still rely on a large number of human-designed and unmodifiable inner-loop components such as backpropagation . Is it possible to implement modifiable versions of backpropagation or related algorithms as part of the end-to-end differentiable activation dynamics of a neural net ( NN ) , instead of inserting them as separate fixed routines ? Here we propose the Variable Shared Meta Learning ( VSML ) principle for this purpose . It introduces a novel way of using sparsity and weight-sharing in NNs for meta learning . We build on the arguably simplest neural meta learner , the meta recurrent neural network ( Meta RNN ) [ 16 , 10 , 56 ] , by replacing the weights of a neural network with tiny LSTMs . The resulting system can be viewed as many RNNs passing messages to each other , or as one big RNN with a sparse shared weight matrix , or as a system learning each neuron ’ s functionality and its LA . VSML generalizes the principle behind end-to-end differentiable fast weight programmers [ 45 , 46 , 3 , 41 ] , hyper networks [ 14 ] , learned learning rules [ 4 , 13 , 33 ] , and hebbian-like synaptic plasticity [ 44 , 46 , 25 , 26 , 30 ] . Our mechanism , VSML , can implement backpropagation solely in the forward-dynamics of an RNN . Consequently , it enables meta-optimization of backproplike algorithms . We envision a future where novel methods of credit assignment can be meta learned while still generalizing across vastly different tasks . This may lead to improvements in sample efficiency , memory efficiency , continual learning , and others . As a first step , our system meta 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learns online LAs from scratch that frequently learn faster than gradient descent and generalize to datasets outside of the meta training distribution ( e.g. , from MNIST to Fashion MNIST ) . Our VSML RNN is the first neural meta learner without hard-coded backpropagation that shows such strong generalization . 2 Background . Deep learning-based meta learning that does not rely on fixed gradient descent in the inner loop has historically fallen into two categories , 1 ) Learnable weight update mechanisms that allow for changing the parameters of an NN to implement a learning rule ( FWPs / LLRs ) , and 2 ) Learning algorithms implemented in black-box models such as recurrent neural networks ( Meta RNNs ) . Fast weight programmers & Learned learning rules ( FWPs / LLRs ) In a standard NN , the weights are updated by a fixed LA . This framework can be extended to meta learning by defining an explicit architecture that allows for modifying these weights . This weight-update architecture augments a standard NN architecture . NNs that generate or change the weights of another or the same NN are known as fast weight programmers ( FWPs ) [ 44 , 45 , 46 , 3 , 41 ] , hypernetworks [ 14 ] , NNs with synaptic plasticity [ 25 , 26 , 30 ] or learned learning rules [ 4 , 13 , 33 ] . Often these architectures make use of local Hebbian-like update rules or outer-products , and we summarize this category as FWPs / LLRs . In FWPs / LLRs the variables VL that are subject to learning are the weights of the network , whereas the meta-variables VM that implement the LA are defined by the weight-update architecture . Note that the dimensionality of VL and VM can be defined independently of each other and often VM are reused in a coordinate-wise fashion for VL resulting in |VL| |VM | , where | · | is the number of elements . Black-box learning in activations ( Meta RNNs ) It was shown that an RNN such as LSTM can learn to implement an LA [ 16 ] when the reward or error is given as an input [ 47 ] . After meta training , the LA is encoded in the weights of this RNN and determines learning during meta testing . The activations serve as the memory used for the LA solution . We refer to this as Meta RNNs [ 16 , 10 , 56 ] ( Also sometimes referred to as memory-based meta learning. ) . They are conceptually simpler than FWPs / LLRs as no additional weight-update rules with many degrees of freedom need to be defined . In Meta RNNs VL are the RNN activations and VM are the parameters for the RNN . Note that an RNN with N neurons will yield |VL| = O ( N ) and |VM | = O ( N2 ) [ 46 ] . This means that the LA is largely overparameterized whereas the available memory for learning is very small , making this approach prone to overfitting [ 20 ] . As a result , the RNN parameters often encode task-specific solutions instead of generic LAs . Meta learning a simple and generalizing LA would benefit from |VL| |VM | . Previous approaches have tried to mend this issue by adding architectural complexity through additional memory mechanisms [ 53 , 29 , 40 , 27 , 42 ] . 3 Variable Shared Meta Learning ( VSML ) . In VSML we build on the simplicity of Meta RNNs while retaining |VL| |VM | from FWPs / LLRs . We do this by reusing the same few parameters VM many times in an RNN ( via variable sharing ) and introducing sparsity in the connectivity . This yields several interpretations for VSML : A VSML as a single Meta RNN with a sparse shared weight matrix ( Figure 1a ) . The most general description . B VSML as message passing between RNNs ( Figure 1b ) . We choose a simple sharing and sparsity scheme for the weight matrix such that it corresponds to multiple RNNs with shared parameters that exchange information . C VSML as complex neurons with learned updates ( Figure 1c ) . When choosing a specific connectivity between RNNs , states / activations VL of these RNNs can be interpreted as the weights of a conventional NN , consequently blurring the distinction between a weight and an activation . Introducing variable sharing to Meta RNNs We begin by formalizing Meta RNNs which often use multiplicative gates such as the LSTM [ 12 , 15 ] or its variant GRU [ 6 ] . For notational simplicity , we consider a vanilla RNN . Let s ∈ RN be the hidden state of an RNN . The update for an element j ∈ { 1 , . . . , N } is given by sj ← fRNN ( s ) j = σ ( ∑ i siWij ) , ( 1 ) where σ is a non-linear activation function , W ∈ RN×N , and the bias is omitted for simplicity . We also omit inputs by assuming a subset of s to be externally provided . Each application of Equation 1 reflects a single time tick in the RNN . We now introduce variable sharing ( reusing W ) into the RNN by duplicating the computation along two axes of size A , B ( here A = B , which will later be relaxed ) giving s ∈ RA×B×N . For a ∈ { 1 , . . . , A } , b ∈ { 1 , . . . , B } we have sabj ← fRNN ( sab ) j = σ ( ∑ i sabiWij ) . ( 2 ) This can be viewed as multiple RNNs arranged on a 2-dimensional grid , with shared parameters that update independent states . Here , we chose a particular arrangement ( two axes ) that will facilitate the interpretation C of RNNs as weights . VSML as message passing between RNNs The computation so far describes A ·B independent RNNs . We connect those by passing messages ( interpretation B ) sab ← fRNN ( sab , −→ma ) , ( 3 ) where the message −→mb = ∑ a′ f−→m ( sa′b ) with b ∈ { 1 , . . . , A = B } , f−→m : R N → RN ′ is fed as an additional input to each RNN . This is related to Graph Neural Networks [ 51 , 58 ] . Summing over the axis A ( elements a′ ) corresponds to an RNN connectivity mimicking those of weights in an NN ( to facilitate interpretation C ) . We emphasise that other schemes based on different kinds of message passing and graph connectivity are possible . For a simple f−→m defined by the matrix C ∈ RN×N , we may equivalently write sabj ← σ ( ∑ i sabiWij + ∑ a′ f−→m ( sa′a ) ) = σ ( ∑ i sabiWij + ∑ a′ , i sa′aiCij ) . ( 4 ) This constitutes the minimal version of VSML with VM : = ( W , C ) and is visualized in Figure 1b . VSML as a Meta RNN with a sparse shared weight matrix It is trivial to see that with A = 1 and B = 1 we obtain a single RNN and Equation 4 recovers the original Meta RNN Equation 1 . In the general case , we can derive an equivalent formulation that corresponds to a single standard RNN with a single matrix W̃ that has entries of zero and shared entries sabj ← σ ( ∑ c , d , i scdiW̃cdiabj ) , ( 5 ) where the six axes can be flattened to obtain the two axes . For Equation 4 and Equation 5 to be equivalent , W̃ must satisfy ( derivation in Appendix A ) W̃cdiabj = Cij , if d = a ∧ ( d 6= b ∨ c 6= a ) . Wij , if d 6= a ∧ d = b ∧ c = a. Cij +Wij , if d = a ∧ d = b ∧ c = a . 0 , otherwise . ( 6 ) This corresponds to interpretation A with the weight matrix visualized in Figure 1a . To distinguish between the single sparse shared RNN and the connected RNNs , we now call the latter sub-RNNs . VSML as complex neurons with learned updates The arrangement and connectivity of the subRNNs as described in the previous paragraphs corresponds to that of weights in a standard NN . Thus , in interpretation C , VSML can be viewed as defining complex neurons where each sub-RNN corresponds to a weight in a standard NN as visualized in Figure 1c . All these sub-RNNs share the same parameters but have distinct states . The current formulation corresponds to a single NN layer that is run recurrently . We will generalize this to other architectures in the next section . A corresponds to the dimensionality of the inputs and B to that of the outputs in that layer . The role of weights in a standard neural network is now assigned to the states of RNNs . This allows these RNNs to define both the neural forward computation as well as the learning algorithm that determines how the network is updated ( where the mechanism is shared across the network ) . In the case of backpropagation , this would correspond to the forward and backward passes being implemented purely in the recurrent dynamics . We will demonstrate the practical feasibility of this in Section 3.2 . The emergence of RNN states as weights quickly leads to confusing terminology when RNNs have ‘ meta weights ’ . Instead , we simply refer to meta variables VM and learned variables VL . With this interpretation , VSML can be seen as a generalization of learned learning rules [ 4 , 13 , 33 ] and Hebbian-like differentiable mechanisms or fast weights more generally [ 44 , 46 , 25 , 26 ] where RNNs replace explicit weight updates . In standard NNs , weights and activations have multiplicative interactions . For VSML RNNs to mimic such computation we require multiplicative interactions between parts of the state s. Fortunately , LSTMs already incorporate this through gating and can be directly used in place of RNNs . Stacking VSML RNNs and feeding inputs To get a structure similar to one of the non-recurrent deep feed-forward architectures ( FNNs ) , we stack multiple VSML RNNs where their states are untied and their parameters are tied.1 This is visualized with two layers in Figure 2 where the states s ( 2 ) of the second column of sub-RNNs are distinct from the first column s ( 1 ) . The parameters A ( k ) and B ( k ) describing layer sizes can then be varied for each layer k ∈ { 1 , . . . , K } constrained by A ( k ) = B ( k−1 ) . The updated Equation 3 with distinct layers k is given by s ( k ) ab ← fRNN ( s ( k ) ab , −→m ( k ) a ) where −→m ( k+1 ) b : = ∑ a′ f−→m ( s ( k ) a′b ) with b ∈ { 1 , . . . , B ( k ) = A ( k+1 ) } . 1The resultant architecture as a whole is still recurrent . Note that even standard FNNs are recurrent if the LA ( backpropagation ) is taken into account . To prevent information from flowing only forward in the network , we use an additional backward message s ( k ) ab ← fRNN ( s ( k ) ab , −→m ( k ) a , ←−m ( k ) b ) , ( 7 ) where ←−m ( k−1 ) a : = ∑ b′ f←−m ( s ( k ) ab′ ) with a ∈ { 1 , . . . , A ( k ) = B ( k−1 ) } ( visualized in Figure 3 ) . The backward transformation is given by f←−m : RN → RN ′′ . Similarly , other neural architectures can be explicitly constructed ( e.g . convolutional NNs , Section B.2 ) . Some architectures may be learned implicitly if positional information is fed into each sub-RNN ( Appendix C ) . We then update all states s ( k ) in sequence 1 , . . . , K to mimic sequential layer execution . We may also apply multiple RNN ticks for each layer k. To provide the VSML RNN with data , each time we execute the operations of the first layer , a single new datum x ∈ RA ( 1 ) ( e.g . one flattened image ) is distributed across all sub-RNNs . In our present experiments , we match the axis A ( 1 ) to the input datum dimensionality such that each dimension ( e.g. , pixel ) is fed to different RNNs . This corresponds to initializing the forward message −→m ( 1 ) a1 : = xa ( padding −→m with zeros if necessary ) . Similarly , we read the output ŷ ∈ RB ( K ) from ŷa : = −→m ( K+1 ) a1 . Finally , we feed the error e ∈ R B ( K ) at the output such that ←−m ( K ) b1 : = eb . See Figure 2 for a visualization . Alternatively , multiple input or output dimensions could be patched together and fed into fewer sub-RNNs .

Summary: This paper proposes VSML RNN, an architecture for RNNs that is suitable for meta-learning problems in which the RNN activations represent task-specific quantities and RNN weights represent task-independent quantities. The authors use the insight that RNNs for meta learning (meta RNNs as called in the paper) perform better when the meta-parameters are constrained relative to the task-specific parameters. VSML RNN is essentially an RNN with a particular weight sharing and sparsity structure. The authors demonstrate that VSML RNN can implement *within its hidden state* backpropagation on a smaller model, where the parameters of the smaller model are represented in the VSML RNN's hidden state. Empirically, the authors show that the learning algorithm implemented by a VSML RNN can outperform baselines, and can learn faster than gradient descent. The authors also demonstrate that the learning algorithm is transferable to unseen problems.