Meta Learning Backpropagation And Improving It

1 Introduction . The shift from standard machine learning to meta learning involves learning the learning algorithm ( LA ) itself , reducing the burden on the human designer to craft useful learning algorithms [ 43 ] . Recent meta learning has primarily focused on generalization from training tasks to similar test tasks , e.g. , few-shot learning [ 11 ] , or from training environments to similar test environments [ 17 ] . This contrasts human-engineered LAs that generalize across a wide range of datasets or environments . Without such generalization , meta learned LAs can not entirely replace human-engineered variants . Recent work demonstrated that meta learning can also successfully generate more general LAs that generalize across wide spectra of environments [ 20 , 1 , 31 ] , e.g. , from toy environments to Mujoco and Atari . Unfortunately , however , many recent approaches still rely on a large number of human-designed and unmodifiable inner-loop components such as backpropagation . Is it possible to implement modifiable versions of backpropagation or related algorithms as part of the end-to-end differentiable activation dynamics of a neural net ( NN ) , instead of inserting them as separate fixed routines ? Here we propose the Variable Shared Meta Learning ( VSML ) principle for this purpose . It introduces a novel way of using sparsity and weight-sharing in NNs for meta learning . We build on the arguably simplest neural meta learner , the meta recurrent neural network ( Meta RNN ) [ 16 , 10 , 56 ] , by replacing the weights of a neural network with tiny LSTMs . The resulting system can be viewed as many RNNs passing messages to each other , or as one big RNN with a sparse shared weight matrix , or as a system learning each neuron ’ s functionality and its LA . VSML generalizes the principle behind end-to-end differentiable fast weight programmers [ 45 , 46 , 3 , 41 ] , hyper networks [ 14 ] , learned learning rules [ 4 , 13 , 33 ] , and hebbian-like synaptic plasticity [ 44 , 46 , 25 , 26 , 30 ] . Our mechanism , VSML , can implement backpropagation solely in the forward-dynamics of an RNN . Consequently , it enables meta-optimization of backproplike algorithms . We envision a future where novel methods of credit assignment can be meta learned while still generalizing across vastly different tasks . This may lead to improvements in sample efficiency , memory efficiency , continual learning , and others . As a first step , our system meta 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learns online LAs from scratch that frequently learn faster than gradient descent and generalize to datasets outside of the meta training distribution ( e.g. , from MNIST to Fashion MNIST ) . Our VSML RNN is the first neural meta learner without hard-coded backpropagation that shows such strong generalization . 2 Background . Deep learning-based meta learning that does not rely on fixed gradient descent in the inner loop has historically fallen into two categories , 1 ) Learnable weight update mechanisms that allow for changing the parameters of an NN to implement a learning rule ( FWPs / LLRs ) , and 2 ) Learning algorithms implemented in black-box models such as recurrent neural networks ( Meta RNNs ) . Fast weight programmers & Learned learning rules ( FWPs / LLRs ) In a standard NN , the weights are updated by a fixed LA . This framework can be extended to meta learning by defining an explicit architecture that allows for modifying these weights . This weight-update architecture augments a standard NN architecture . NNs that generate or change the weights of another or the same NN are known as fast weight programmers ( FWPs ) [ 44 , 45 , 46 , 3 , 41 ] , hypernetworks [ 14 ] , NNs with synaptic plasticity [ 25 , 26 , 30 ] or learned learning rules [ 4 , 13 , 33 ] . Often these architectures make use of local Hebbian-like update rules or outer-products , and we summarize this category as FWPs / LLRs . In FWPs / LLRs the variables VL that are subject to learning are the weights of the network , whereas the meta-variables VM that implement the LA are defined by the weight-update architecture . Note that the dimensionality of VL and VM can be defined independently of each other and often VM are reused in a coordinate-wise fashion for VL resulting in |VL| |VM | , where | · | is the number of elements . Black-box learning in activations ( Meta RNNs ) It was shown that an RNN such as LSTM can learn to implement an LA [ 16 ] when the reward or error is given as an input [ 47 ] . After meta training , the LA is encoded in the weights of this RNN and determines learning during meta testing . The activations serve as the memory used for the LA solution . We refer to this as Meta RNNs [ 16 , 10 , 56 ] ( Also sometimes referred to as memory-based meta learning. ) . They are conceptually simpler than FWPs / LLRs as no additional weight-update rules with many degrees of freedom need to be defined . In Meta RNNs VL are the RNN activations and VM are the parameters for the RNN . Note that an RNN with N neurons will yield |VL| = O ( N ) and |VM | = O ( N2 ) [ 46 ] . This means that the LA is largely overparameterized whereas the available memory for learning is very small , making this approach prone to overfitting [ 20 ] . As a result , the RNN parameters often encode task-specific solutions instead of generic LAs . Meta learning a simple and generalizing LA would benefit from |VL| |VM | . Previous approaches have tried to mend this issue by adding architectural complexity through additional memory mechanisms [ 53 , 29 , 40 , 27 , 42 ] . 3 Variable Shared Meta Learning ( VSML ) . In VSML we build on the simplicity of Meta RNNs while retaining |VL| |VM | from FWPs / LLRs . We do this by reusing the same few parameters VM many times in an RNN ( via variable sharing ) and introducing sparsity in the connectivity . This yields several interpretations for VSML : A VSML as a single Meta RNN with a sparse shared weight matrix ( Figure 1a ) . The most general description . B VSML as message passing between RNNs ( Figure 1b ) . We choose a simple sharing and sparsity scheme for the weight matrix such that it corresponds to multiple RNNs with shared parameters that exchange information . C VSML as complex neurons with learned updates ( Figure 1c ) . When choosing a specific connectivity between RNNs , states / activations VL of these RNNs can be interpreted as the weights of a conventional NN , consequently blurring the distinction between a weight and an activation . Introducing variable sharing to Meta RNNs We begin by formalizing Meta RNNs which often use multiplicative gates such as the LSTM [ 12 , 15 ] or its variant GRU [ 6 ] . For notational simplicity , we consider a vanilla RNN . Let s ∈ RN be the hidden state of an RNN . The update for an element j ∈ { 1 , . . . , N } is given by sj ← fRNN ( s ) j = σ ( ∑ i siWij ) , ( 1 ) where σ is a non-linear activation function , W ∈ RN×N , and the bias is omitted for simplicity . We also omit inputs by assuming a subset of s to be externally provided . Each application of Equation 1 reflects a single time tick in the RNN . We now introduce variable sharing ( reusing W ) into the RNN by duplicating the computation along two axes of size A , B ( here A = B , which will later be relaxed ) giving s ∈ RA×B×N . For a ∈ { 1 , . . . , A } , b ∈ { 1 , . . . , B } we have sabj ← fRNN ( sab ) j = σ ( ∑ i sabiWij ) . ( 2 ) This can be viewed as multiple RNNs arranged on a 2-dimensional grid , with shared parameters that update independent states . Here , we chose a particular arrangement ( two axes ) that will facilitate the interpretation C of RNNs as weights . VSML as message passing between RNNs The computation so far describes A ·B independent RNNs . We connect those by passing messages ( interpretation B ) sab ← fRNN ( sab , −→ma ) , ( 3 ) where the message −→mb = ∑ a′ f−→m ( sa′b ) with b ∈ { 1 , . . . , A = B } , f−→m : R N → RN ′ is fed as an additional input to each RNN . This is related to Graph Neural Networks [ 51 , 58 ] . Summing over the axis A ( elements a′ ) corresponds to an RNN connectivity mimicking those of weights in an NN ( to facilitate interpretation C ) . We emphasise that other schemes based on different kinds of message passing and graph connectivity are possible . For a simple f−→m defined by the matrix C ∈ RN×N , we may equivalently write sabj ← σ ( ∑ i sabiWij + ∑ a′ f−→m ( sa′a ) ) = σ ( ∑ i sabiWij + ∑ a′ , i sa′aiCij ) . ( 4 ) This constitutes the minimal version of VSML with VM : = ( W , C ) and is visualized in Figure 1b . VSML as a Meta RNN with a sparse shared weight matrix It is trivial to see that with A = 1 and B = 1 we obtain a single RNN and Equation 4 recovers the original Meta RNN Equation 1 . In the general case , we can derive an equivalent formulation that corresponds to a single standard RNN with a single matrix W̃ that has entries of zero and shared entries sabj ← σ ( ∑ c , d , i scdiW̃cdiabj ) , ( 5 ) where the six axes can be flattened to obtain the two axes . For Equation 4 and Equation 5 to be equivalent , W̃ must satisfy ( derivation in Appendix A ) W̃cdiabj = Cij , if d = a ∧ ( d 6= b ∨ c 6= a ) . Wij , if d 6= a ∧ d = b ∧ c = a. Cij +Wij , if d = a ∧ d = b ∧ c = a . 0 , otherwise . ( 6 ) This corresponds to interpretation A with the weight matrix visualized in Figure 1a . To distinguish between the single sparse shared RNN and the connected RNNs , we now call the latter sub-RNNs . VSML as complex neurons with learned updates The arrangement and connectivity of the subRNNs as described in the previous paragraphs corresponds to that of weights in a standard NN . Thus , in interpretation C , VSML can be viewed as defining complex neurons where each sub-RNN corresponds to a weight in a standard NN as visualized in Figure 1c . All these sub-RNNs share the same parameters but have distinct states . The current formulation corresponds to a single NN layer that is run recurrently . We will generalize this to other architectures in the next section . A corresponds to the dimensionality of the inputs and B to that of the outputs in that layer . The role of weights in a standard neural network is now assigned to the states of RNNs . This allows these RNNs to define both the neural forward computation as well as the learning algorithm that determines how the network is updated ( where the mechanism is shared across the network ) . In the case of backpropagation , this would correspond to the forward and backward passes being implemented purely in the recurrent dynamics . We will demonstrate the practical feasibility of this in Section 3.2 . The emergence of RNN states as weights quickly leads to confusing terminology when RNNs have ‘ meta weights ’ . Instead , we simply refer to meta variables VM and learned variables VL . With this interpretation , VSML can be seen as a generalization of learned learning rules [ 4 , 13 , 33 ] and Hebbian-like differentiable mechanisms or fast weights more generally [ 44 , 46 , 25 , 26 ] where RNNs replace explicit weight updates . In standard NNs , weights and activations have multiplicative interactions . For VSML RNNs to mimic such computation we require multiplicative interactions between parts of the state s. Fortunately , LSTMs already incorporate this through gating and can be directly used in place of RNNs . Stacking VSML RNNs and feeding inputs To get a structure similar to one of the non-recurrent deep feed-forward architectures ( FNNs ) , we stack multiple VSML RNNs where their states are untied and their parameters are tied.1 This is visualized with two layers in Figure 2 where the states s ( 2 ) of the second column of sub-RNNs are distinct from the first column s ( 1 ) . The parameters A ( k ) and B ( k ) describing layer sizes can then be varied for each layer k ∈ { 1 , . . . , K } constrained by A ( k ) = B ( k−1 ) . The updated Equation 3 with distinct layers k is given by s ( k ) ab ← fRNN ( s ( k ) ab , −→m ( k ) a ) where −→m ( k+1 ) b : = ∑ a′ f−→m ( s ( k ) a′b ) with b ∈ { 1 , . . . , B ( k ) = A ( k+1 ) } . 1The resultant architecture as a whole is still recurrent . Note that even standard FNNs are recurrent if the LA ( backpropagation ) is taken into account . To prevent information from flowing only forward in the network , we use an additional backward message s ( k ) ab ← fRNN ( s ( k ) ab , −→m ( k ) a , ←−m ( k ) b ) , ( 7 ) where ←−m ( k−1 ) a : = ∑ b′ f←−m ( s ( k ) ab′ ) with a ∈ { 1 , . . . , A ( k ) = B ( k−1 ) } ( visualized in Figure 3 ) . The backward transformation is given by f←−m : RN → RN ′′ . Similarly , other neural architectures can be explicitly constructed ( e.g . convolutional NNs , Section B.2 ) . Some architectures may be learned implicitly if positional information is fed into each sub-RNN ( Appendix C ) . We then update all states s ( k ) in sequence 1 , . . . , K to mimic sequential layer execution . We may also apply multiple RNN ticks for each layer k. To provide the VSML RNN with data , each time we execute the operations of the first layer , a single new datum x ∈ RA ( 1 ) ( e.g . one flattened image ) is distributed across all sub-RNNs . In our present experiments , we match the axis A ( 1 ) to the input datum dimensionality such that each dimension ( e.g. , pixel ) is fed to different RNNs . This corresponds to initializing the forward message −→m ( 1 ) a1 : = xa ( padding −→m with zeros if necessary ) . Similarly , we read the output ŷ ∈ RB ( K ) from ŷa : = −→m ( K+1 ) a1 . Finally , we feed the error e ∈ R B ( K ) at the output such that ←−m ( K ) b1 : = eb . See Figure 2 for a visualization . Alternatively , multiple input or output dimensions could be patched together and fed into fewer sub-RNNs .

Summary: This paper proposes variable shared meta learning,  and implements the network forward and backward computation with RNN.  The proposed method can also be used for learning general learning algorithms. Experiments on MNIST and Fashion MNIST dataset demonstrate the effectiveness of the proposed method.