NORML: Nodal Optimization for Recurrent Meta-Learning
Authors: David van Niekerk
Venue:  (2020)
Keywords: meta-learning
learning to learn
few-shot classification
memory-based optimization

Abstract: Meta-learning is an exciting and powerful paradigm that aims to improve the effectiveness of current learning systems. By formulating the learning process as an optimization problem, a model can learn how to learn while requiring significantly less data or experience than traditional approaches. Gradient-based meta-learning methods aims to do just that, however recent work have shown that the effectiveness of these approaches are primarily due to feature reuse and very little has to do with priming the system for rapid learning (learning to make effective weight updates on unseen data distributions). This work introduces Nodal Optimization for Recurrent Meta-Learning (NORML), a novel meta-learning framework where an LSTM-based meta-learner performs neuron-wise optimization on a learner for efficient task learning. Crucially, the number of meta-learner parameters needed in NORML, increases linearly relative to the number of learner parameters. Allowing NORML to potentially scale to learner networks with very large numbers of parameters. While NORML also benefits from feature reuse it is shown experimentally that the meta-learner LSTM learns to make effective weight updates using information from previous data-points and update steps.

Under review as a conference paper at ICLR 2020
NORML: NODAL OPTIMIZATION FOR RECURRENT META-LEARNING LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning is an exciting and powerful paradigm that aims to improve the ef- fectiveness of current learning systems. By formulating the learning process as an optimization problem, a model can learn how to learn while requiring significantly less data or experience than traditional approaches. Gradient- based meta-learning methods aims to do just that, however recent work have shown that the effective- ness of these approaches are primarily due to feature reuse and very little has to do with priming the system for rapid learning (learning to make effective weight updates on unseen data distributions). This work introduces Nodal Optimization for Recurrent Meta-Learnin (NORML), novel meta-learning framework where an LSTM-based meta-leurner performs neuron-wise optimization on learner for efficient task learning. Crucially, the number of mela-learner parameters needed in NORML, increases linearly relative to the number of learner parameters. Al- lowing NORML potentially scale learner networks with very large numbers of parameters. While NORML also benefits from feature reuse it is shown ex- perimentally that the meta-learner LSTM learns to make effective weight updates using information from previous data-points and update steps.
INTRODUCTION
Humans have remarkable capability to learn useful concepts from a small number of examples or a limited amount of experience. In contrast most machine learning methods require large, labelled datasets to learn effectively. Little is understood about the actual learning algorithm(s) used by the human brain, and how it relates to machine learning algorithms like backpropagation (Lillicrap & Körding (2019)) [Botvinick et al. (2019) argues that inductive bias and structured priors are some of the main factors that enable fast learning in animals. In order to build general-purposs systems we must be able to design and build learning algorithms that can quickly and effectively learn from limited amount of data by utilizing prior knowledge and experience.
Supervised few-shot learning aims challenge machine learning models to learn new tasks by lever- aging only a handful of labelled examples. |Vinyals et al.] (2016) introduces the few-shot learning problem for image classification, where model tasked to classify number of images while being provided either 1 or 5 examples of each class (hereafter referred to as 1-shot and 5-shot learning). One way approach this problem is by way of meta-learning, broad family of techniques that aim to learn how to learn (Thrun & Pratt (1998)). One particularly powerful group of approaches, known as memory-based methods, use memory architectures that can leverage prior information to assist in learning (Santoro et al. (2016); Ravi & Larochelle (2017).. Optimization- based methods (Finn al. (2017) is another exciting area that aim to learn an initial set of parameters that can quickly adapt to new, unseen tasks with relatively little training data.
This work introduces a novel technique where a recurrent neural network based meta-learner learns how to make parameter updates for a task learner. The meta-learner and the learner are jointly trained so as to learn how to learn new tasks with little data. This approach allows one to utilize aspects from both optimization-ban based and memory-based meta-learning methods. The recurrent architecture of the meta-learner can use important prior information when updating the learner, task. The
while the learner can learn a set of initial task parameters are easily optimized for vanishing gradient challenge faced by gradient- gradient-based optimization is solved by using a Long short- term memory (LSTTM) based meta-learner (Hochreiter & Schmidhuber (1997)) Memory-based

---

Under review as a conference paper at ICLR 2020
methods (Ravi & Larochelle (2017)) that all of the learner's parameters input to meta- learner tend to break down when using a learner with a large number parameters (Andrychowicz et. 2016). The technique proposed in this work, Nodal Optimization for Recurrent Mleta-Learning (NORML., solves this scaling problem by learning layer-wise update signals used to update the learners's parameters.
NORML is evaluated on the Mini-ImageNet dataset and is shown to improve on existing optimization-based and memory-based methods. An ablation study is done showing the effects of the different components of NORML. Furthermore a comparison is done between NORML and Model Agnostic Meta-Leaminn (MAML) using the Omniglot dataset. The comparison demonstrates that NORML makes superior parameter updates than the updates made by gradient descent.
2
PRELIMINARIES
2.1
BACKPROPAGATION
The backpropagation algorithm (Rumelhart et al. 1986) was developed in the 1980's s and has since been the status quo for training neural networks. By recursively applying the chain rule, backprop- agation sends gradient signals back through stacked layers of a deep network. Each gradient signal at a particular layer is used to calculate the gradient of the weights connected to that layer.
Consider neural network with N hidden layers. Let the rows of matrix Wi denote the weights connecting the nodes in the layer below to a hidden node in layer and let the vector b, denote the biases connected to each node in layer Then:
a Wih- bi, h4 f((a;)
Where fi is the activation function used layer I, a; is the layer's pre-activation output, and h; is referred to as the activations of layer Note that hg = x is the input to the network. The output of the network is given by ý (aour). When choosing to use softmax the output activation and a cross-entropy loss function, the loss and the output layer's gradient are calculated as follows:
n Σ Yn log ÿn. + (1 yn) log(1 9n)
0L
8 8aout
Da
where I. is the cross cross-entropy loss, n is the network's output size, and de.. is the gradient at the pre-activation of the output layer. Note that 8a... is an n-dimensional vector where each element of 5 6a... describes the corresponding node's contribution to the loss. By propagating ö 6a... back through the network one obtains the gradients at each the hidden layers' nodes:
OL (W, da,) (ai)( 8a. daz-1
(4)
where f(() the derivative of the activation function, an element-wise multiplication and da, is the pre-activation gradient at layer I. In order to calculate the gradient with regards to the weights, 6a, matrix multiplied with the activations of the previous layer:
aL aw;
5a, h4-1
(5)
2.2 META-LEARNING PROBLEM DEFINITION
The N- N-way K-shot problem is defined using episodic formulation as proposed in Vinyals et al. 2016). A task T, is sampled from a task distribution p(T) and consists of an N N-way classification problem. A meta-dataset is divided into training meta-set Str, validation meta-set Sual and testing meta-set gtest where the classes contained each meta-set is disjointed (i.e. none of the classes in Steat is present in frr and visa versa).
2

---

Under review as a conference paper at ICLR 2020
The task T; consists of training set D and validation set D The training set D contains K' examples from each of T'' N classes. The validation set D usually contains a larger set of examples from each class in order to give an estimate of the model's generalization performance task T, Note that a task's validation set D (used to optimize the meta training objective) should not be confused with the meta validation set Sual, which is used for model selection.
2.3 MODEL AGNOSTIC META- LEARNIG
Optimization- based meta-learning is approach to meta-leariing where an inner loop is used for fast adaptation to a new task, and an outer loop is used to optimize the inner loop's training steps. Model Agnostic Meta-Learning (MAML) is one such approach that contains all the key ingredi- ents used in optimization- gradient descent on a few-shot learning task, T;
meta-learning. The inner loop consists of training a model, fo via
8' = G(O, D
(6)
where G is often implemented as single step of gradient descent on the training set D 0g nf VoL4, (fo). It is often advantageous for G to consist of multiple sequential update steps.
The validation set of T; then used to evaluate for and the task-specific validation loss, L calculated. The outer loop consists of optimizing the base-parameters 0, using the sum of M different tasks' validation losses:
t 0
V
Σ
L
Leal(fo) (fn:)
TTxp(T)
where M is referred as the meta-batch size.
This approach allows a model to learn a set of base- base- parameters 8, that can quickly adapt to a new unseen task. After meta-training model, the inner loop procedure specific parameters 0g using only a few examples of the new task. By differentiating through the inner loop, MAML learns update the aase-parameters in such way that the task-specific para- maters, 8', generalizes to unseen examples of task T,
update 0 and return task-
In many cases it is preferred have inner loop that consists of multiple sequential updates. However the inner loop's computational graph can become quite large if too many steps are taken. This often results in exploding and vanishing gradients since the outer loop still needs to differentiate through the entire inner Aravind Rajeswaran (2019) Antoniou et al. (018). This limits MAML domains where small amount of update steps are sufficient for learning. The LSTM-based meta- learner proposed in this work, allow gradients to effectively flow through a large number of update steps. NORML can therefore be applied to a wide array of domains.
MODEL
3.1 TRAINING
NORML incoporates neural network based learner with N layers and an LSTM-based meta- learner that is used to optimize the learner's inner loop parameter updates. Figure depicts the process by which the meta-learner calculates the parameter updates for single layer learner. When multiple layers are optimized the additional gradient signals and layer inputs are added to the meta learner's imput. The meta-learner outputs two update signals for every layer, so when 4-layer network is used as the learner, the meta-learner outputs 8 update signals.
The ligh-level operation is as follows (Algorithm 1). First the learner's 's task-specific loss, L (ffo, and mnyer-gradient da are calculated via cross-entropy and backpropagation. The meta-leamer takes say h4-1 and the loss, L, as input, and outputs the current cell state and the update signals, h and sa,:
8a: h
=mo (5a, h1-1, c.))
(8)
C1,++1
3

---

Under review as a conference paper at ICLR 2020
The update signals matrix multiplied to determine the parameter updates for layer 1:
A W 17ō h
A n
(10)
Gradiant updist signel
Matrix
Multiplication
Hidden layer update signal
Meta-Learner
Gradiel at Reeacc
Loamning Rate:
Parematier Updates
Layer input
Fully- Connected
Pro acthotioo
Softmax
Fred
Entropy Loss Function
Gradient Calculation
Figure 1:
Training process of NORML.
The learner's updated parameters, 87; are then used the next training example and the whole pro- cess gets repeated for U update steps. The inner loop terminates after calculating the task validation loss using for
Leel CrossEntropy for (Fvai))
(11)
The outer loop consists of optimizing both Oo and the meta-learner parameters & via gradient de- scent:
0 8
Σ
L
(fo:)
(12)
p(T)
+ 1
V
Σ
L7, (fo:)
(13)
T. p(T)
3.2 META-LEARNER
The meta-learner, m is implemented using a modified LSTM cell as shown in Figure 2.
The input to the meta-learner is normalized, flattened and concatenated. is used to denote con- catenating the loss, the layer gradients, and the activations the previous layer.
The meta-learner consists of a forget gate, fi, an input gate, é and an update gate, é. The cell state at update step is denoted via Ct,,, and is calculated as follows:
fi. o Wi, + bi,))
(14)
f o [W,, + bt,s)
(15)
c.. tanh Wi, + b,
(16)
C₁ fi. O C/,-1 i čt,t
(17)
In order to determine the update signals, the cell state, CLa is used as input to two separate fully- connected layers followed by sigmoid activation function and a pointwise multiplication with the original layer gradient and the previous layer's activations:
8a: sa, c @o(Wi + b..9)
(18)
h, h O (WLCC br.))
(19)

---

Under review as a conference paper at ICLR 2020
Algorithm Nodal Optimization for Rapid Meta-Learning
Require: Learner f with parameters 0
Require: Meta-learner m with parameters +
Require: p(T): distribution over tasks
Require: 17. 3: step size hyperparameters
1: randomly initialize 0,
2: while not converged do
3: Sample batch tasks T, ~ (())
4:
5:
for all T; do
6:
Let (D,", D.al)= T; Initialize 0/
7:
8:
9:
for / = 1, Leenn do
aLT; Ba:
10:
Compute learner's training loss LT, (fo;_.) and layer gradients
for all layers /
11:
Compute the update signals: f mo (L4, (fo;_,): Ba, h(-1)
01.7,
12:
Update task parameters: 0g 01-1 nighh--1 end for
13:
14:
Compute validation loss end for
(fa:) (fa)
15:
Update - V Ex, p(T) Leak (ee)(Se:)
17:
Update & - Φ- V ET, p(T) L7 (So))
18: end while
State
Cell State art+1
PC Layer
Layer
Forget
sigmnic
sigmond
sigmoid
sigmold
Layer
Concatt and Norm
State DEDDS
Shate
prooe.ono Cradient
Laytr
Layer Input
Gradhett
Stgnal
Figure 2: Meta-learner LSTM.
The sigmoid activation and pointwise mutliplication operations allow the meta-learner to scale the update signal received by each node the network. This node-wise scaling let the meta-learner control how the weights connected to a particular node will get changed at any given update step. By scaling the layer gradients, the meta-learner can dynamically control by how much each node's input should be changed at each update step. Likewise, by scaling the hidden layer activations, the meta-learner can control the change in weighting each the previous nodes' output. By controlling the magnitude of both the activations and the layer gradients, the meta-learner can learn a dynamically adaptive learning rate for each individual weight and bias of the learner. This can be achieved with meta-learner that scales linearly in size relative to the size of the learner, i.c. increasing the number of learner parameters by a factor of p. will result in the same factor increase in the number of meta-learner parameters.
5

---

Under review as a conference paper at ICLR 2020
4 RELATED WORK
Learning how to learn has a long history and variety of approaches have been proposed. Memory- based methods memory architectures either store key training examples to encode fast adaptation algorithms (Santoro et al. (016) [Andrychowicz et al., 2016) (Ravi & Larochelle] 2017 Metric-based methods are designed to learn similarity metrics for examples of the same class (Koch)
2015) (Vinyals et al. 2016) (Snell et al. 2017). Optimization based methods aim to learn set of initial parameters that can quickly adapt to new tasks (Finn al. 2017).
A subset of memory-based methods use recurrent neural networks to learn fast adaptation algo- rithms. In Ravi & Larochelle (2017) an LSTM-basee meta-learner is trained iteratively optimize neural network. At a given timestep the meta-learner's s hidden state consists of the optimizee's 's cur- rent parameters. At each timestep a training datapoint is used to calculate the optimizee's 's parameter gradients. The parameter gradients and the task loss is used input to the meta-learner. The hidden state of the following timestep is then used as the updated parameters for the learner. A challenge of this approach is that if you want to optimize tens of thousands of parameters, you would have massive hidden state and input size, and will therefore require an enormous number of parameters for the meaa-leerner.
In Andrychowicz et al. (2016) an alternative approach is introduced that avoids the aforementioned scaling problem by individually optimizing each parameter using separate LSTM. This vastly decreases the number of meta-parameters required, however each meta-learner only given infor- mation of a single parameter and therefore use information of other parts of the network when doing optimization. The approach introduced in this work uses a single meta-learner to opti- mize all the parameter's the learner, while the meta-learner require less than times the number of parameters used by the learner.
Optimization-basee methods (Finn optimize learning. Improvements to MAML include using pre-trained network to extract features and to then only optimize the final fully-connecte layer via gradient based optimization (?). Using pre-trained feature extractor assists in overcoming task overfitting and have been demonstrated to give large boost in performance(Sun
2017). differentiate through the inner loop in order to
al. 2018). In this work a pre-trained feature extractor was also used when evaluating NORML on Mini-lmagenet, but is not requirement for implementing NORML.
Other work that custom layer-wise update signals include Nø kland (2016) and Nkland & Eidnes (019)) In feedback alignment layer gradient approximating update signals are used to train a neural network by passing the error signal through fixed random matrices. Backpropagation in contrast passes the eтror signal through the transpose of the subsequent layer's weight matrix. Surprisingly feedback alignment demonstrates that learning can still occur even when using random matrices for the backward pass.
5 EXPERIMENTAL RESULTS
The proposed approach is evaluated on the Mini- Mini-Imagenet benchmark dataset. An ablation study comparing the performance against. MAMI. is conducted the Omniglot dataset. The evaluation aims to compare the performance of NORML against existing methods and to specifically test node-wise optimization improves the update signals provided by backpropagation.
5.1 MINI-IMAGENET CLASSIFICATION
The Mini- mini-Imagenet dataset was proposed by Vinyals et al. (2016) and is a popular benchmark for few-shot learning methods. The dataset consists of 100 classes, each containing 600 images. The meta-training, meta- validaaaiittinnn and meta-testing sets each contain 64, 16 and 20 classes respectively. has been demonstrated that by using a pre-trained feature extractor the performance on few-shot learning tasks can be significantly improved (Sun et al. (2018)) For evaluation of NORML on the Mini- Imageneteeet benchmark, a Resnet-12 12 network is pre-trdined using only data and classes from the meta-training set. The pre-train model is trained to perform 64-way elassification. After pre- training. the final FC-layer is replaced by a randomly initialized FC-leyer and the convolutional layers are frozen.

---

Under review as a conference paper at ICLR 2020
The meta-learner network and the final layer of the learner optimized during meta-training The inner loop consisted of epochs for both the 5-shot and 1-shot cases, while using a meta- meta-batch size of 4. The model was trained for 10 000 meta-iterations and the meta-validation set was used for model selection for testing.
Classification accuracies for NORML and other baselines shown in Table 1.
Model
Mini- Mini-Imagennt Test Accuracy
1-shot
5-shot
Matching network (Vinyals (2016))
43.56 10.84%
55.31 +0.73%
Meta-learner LSTM (Ravi & Larochelle (2017)
43.44 =0.77%
60.60 =0.71%
MAML Finn al.
2201
48.70 #1.84%
63.11 =0.92%
LLAMA Grant et al.
2211
49,40 +1.83%
REPTILE (Nichol et 2018
49.97 +0.32%
65.99
+..58%
PLATIPUS (Finn et al.
2018
50.13 #1.66%
SNAIL. (Mishra et al.
2017)
55.71 +0.99%
68.88
+0.92%
Gidaris & Komodakıs
2018
56.20 +0.86%
73.00 +0.64%
[Bauer et al. (0018
56.30 10.40%
73.90 10.30%
(Munkhdalar et
(217))
57.10 10.70%
70.04 +0.63%
Zhou et al.
20
58.49 #0.91%
71.28 +0.69%
Qiao 2017
59.60 =0.41%
73.74 10.19%
NORMI
60.2 + 0.91%
74.1 + 0.71%
Table 1: Test accuracies on the Mini-ImageNet benchmark. The first set of results use convolutional networks, while the second much deeper residual networks, predominantly in conjunction with pre-training.
5.2 ABLATION STUDY
For the Mini-Imagenet benchmark the learner consisted of single FC-layer which was proceeded by a pre-trained feature extractor. NORML can however be applied to networks with multiple layers. To assess the effect of the different components of the meta-learner and to compare NORML to MAML, an ablation study was conducted on the Omniglot 20-way dataset. The learner consisted of fully-connected network with 4 hidden layers, each including batch normalization and ReLU activations, followed by a linear layer and softmax. The hidden layer sizes were 256, 128, 64 and 64. The same learner network used for all approaches while only changing the setup of the meta-learner between experiments.
The cell and hidden state of the meta-learner carries information of all the previous losses, gra- dients and update steps that was calculated for the task at hand. This enables NORML to potentially improve on the update signals calculated via backpropagation, since the gradient for particular data-point only contains information from the current time step. To assess the contribution of the cell state and hidden state the performance of NORML, the following three setups were tested: (1) Setting the cell input state and hidden input state to zero, (2) only setting the hidden state to zero and (3) only setting the cell state zero.
The meta-learner used in NORML can learn to dynamically control the learning rates of each learner node's input and output weights. To assess the effect of the node-wise learning rates and to compare NORML to MAML, three different training setups are used for MAML: (1) MAML with a constant learning rate, (2) MAML with a single learnable learning rate and (3) MAML with learn able par-layer learning rates. The results for the 1-shot and 5-shot tasks are shown in Table 2. Note that the results of the ablation study are obtained using a fully--onnected network; a convolutional neural network will give higher accuracies.

---

Under review as a conference paper at ICLR 2020
Model
Omniglot 20-way Test Accuracy
1-shot
5-shot
MAML (constant Ir)
86.3%
95.7%
MAML (global Ir)
86.5%
95.7%
MAML (layer-wise Ir)
87.1%
96.0%
NORMI (zero hidden & zero cell)
86.3%
95.7%
NORMI. (zero cell)
86.3%
95.7%
NORML (zero hidden)
87.9%
96.5%
NORML
90.3%
97.2%
Table 2: Ablation study and comparison to MAML using different training setups
NORML learns to dynamically control the learning rates of each learner node's input and output weights. This is shown to give a significant improvement when compared to using constant, learn- able, and per-layer learnable learning rates. Intuitively it makes sense, since NORML can effectively control the learning rates of each parameter in the learner network. Additionally NORML can con- trol the learning rates dynamically. Dynamic control seems crucial to NORML's success, since a parameter that receives large update step i, might need to make a small update at step t 1. The importance of being able to dynamically adapt the learning rate using prior update information, confirmed in the experiments where the input hidden state and input cell state of the meta-learner are set to zero. Although the meta-learner can still learn nodal learning rates, it is unable to use information of previous update steps and ends up performing the same as MAML with a constant learning rate.
CONCLUSION AND FUTURE WORK
Recent work has shown that the successes of most meta- meta-learning techniques are primarily due to feature reuse (Raghu et al. 2019). Ideally meta-learning techniques should be learning to also make efficient and effective parameter updates. This work introduces novel meta-learning method that allows node-wise scaling of learner update signals, thereby enabling the meta-leamer to dynami- cally control the learning rate of each of the learner's parameters. Importantly this is done by meta-learner who's s size scales linearly relative to the size of the learner.
Future work would focus on scaling NORML to optimize large learner networks. Furthermore, applying NORML to domains where large number of update steps are required, such as Deep Reinforcement Learning, could prove to be an exciting avenue of further research since the LSTM- based meta-learner allow gradients to propagate back much farther than gradient-based optimization methods (Finn et al.) 2017)
REFERENCES
Marein Andrychowicz, Misha Denil, Sergio Gómez. Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning to learn by gradient de- scent by gradient descent. In D. D. Lee, M. Sugiyama, V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3981-
nips cc/paper/ 6461 learning to learn by gradient deeerttddeeenttoee by gradient desteet.
3989. Curran Associates, Inc., 2016. URL
pdf
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How train your MAML. CoRR, abs/1810.09502, 2018. URL Utt::tt///arxi.ccoo org./ abs/
wttp://wwww...p////////... 011./10010.900222 09502
Sham Kakade Sergey Levine Aravind Rajeswaran, Chelsea Finn. Meta-leaming with implicit gra- dients. 2019.
Matthias Bauer, Mateo Rojas-Carulla, Jakub Bartlomiej Swiatkowski, Bernhard Schölkopf, and Richard E. Turner. Discriminative k-shot learning using probabilistic models. ArXiv, abs/1706.00326, 2018
Matthew Botvinick, Sam Ritter, Wang Jane X., Zeb Kurth-Nelson Charles Blundell, and Hassabis Demis. Reinforcement learning, fast and slow. Trends Cognitive Science, 23:408-421 2019.
8

---

Under review as a conference paper at ICLR 2020
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. CoRR, abs/1703.03300, 2017. URL htttp://axxivorrt wtt.:r/g/bbs//...//
03400
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta- learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp- 9516-9527, Curran Associates, Inc., 2018. URL http:: 8161 poo.-probbbilittcc---oo modei-agnotticc meta Learning... pdf
ni nips. https.cc/ppprr/
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. CoRR, abs/1 00/11804.9455 2018.
wttpooggg.abg//////////// 09458
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas L. Griffiths. Recast-
ing gradient-based meta-lcarning as hierarchical bayes. CoRR, abs/1801.08930, 2018. URL
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Gregory R. Koch. Siamese neural networks for one-shot image recognition. 2015.
Timothy Lillicrap and Konrad P. Körding. What does mean understand neural network? CoRR, abs//107.063377 2019. URL
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. Mcta-lcarning with temporal con- volutions. CoRR, abs/1707.03141, 2017. UR/Attp////xxv..oorr///////
03141
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, Tong Wang, and Adam Trischler. Learning rapid-temporal adaptations. 12 2017.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. CoRR, abs/ 0bss/803.00999) 2018. URL http: wtt.g/iwiiii...../aaaooooo abs/1803. 1803.0299999.
Arild No kland. Direct feedback alignment provides learning in deep neural net- works. In D. D. Lee. M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems 29, pp. 1037- 1045. Curran Associates, Inc., 2016. URL otpp
pdf
Arild Nkland and Lars Eidnes. Training neural networks with local error signals, 01 2019.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. CoRR, abs/ 706.03366, 2017. URL http:://aaxxx..xrggg/g//o//
1706. 03466
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 5th In- ternational Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings, 2017. URL htttps://ppereviiii wtw.heruuwne//fommmmmmmm
i @d=rJY0-Kcl1l
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning Representations by Back Back propagarpgg Errors. Nature, 323(6088):533-534 1986. doi: 10.1038/323533a0. URL
http:://wwe. httt:://ww.naturee nature. com/ httm//ariiees/333333333
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. One- shot learning with memory- augmented neural networks. CoRR, NDw/160500666 2016. URL
http://arxivw wwwpiivworgggbb.. hwtpg///////// /ttp///abs1115.00000 06065

---

Under review as a conference paper at ICLR 2020
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. httg://b/1173.05111
CoRR, abs/1703.05115, 2017.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot
learning. CoRR, abs/1812.02391, 2018.
Sebastian Thrun and Lorien Pratt. Learning to Learn: Introduction and Overview, pp. 3-17.
Springer US, Boston, MA, 1998. ISBN 978 978-1-4615--552222 doi: 10.1007/998-114415--55999 URL
10. -tt.::/.......100771------------
Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. CoRR, abs/1666.044080 2016. URL http::/arxxx
Fengwei Zhou, Bin Wu, and Zhenguo Li. Deep meta-leamming Learning to learn in the concept space. CoRR, abs/1802.03596, 2018. URL bttp:://rxiv..rg//aaatt
03596
A
APPENDIX
You may include other additional sections here.
10


Reviews:
[{"review_id": "rkgxcuLcYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The paper targets the scalability issue of certain meta-learning frameworks, and is therefore addressing an important and interesting topic. \n\n- Theoretical and technical novelty is rather minimal. \n\n- The paper writing is well beyond the ICLR level, and is honestly beyond the level required by a scientific manuscript in general. Writing needs a massive overhaul.\n\n- There are way too many grammatical mistakes. In addition, there are citations written the wrong way -e.g. Rajeswaran (2019)-, and also the flow of the ideas has room for improvement. \n- For the aforementioned reference, the author ordering is wrong, and not consistent with the way it is cited in the text.\n\n- page 1: \"while the learner can learn a set of initial task parameters are easily optimized for a new task. \": Please fix and/or clarify. \n\n- page 2: \"where the classes contained in each meta-set is disjointed\". \n\n- page 3: \"The LSTM-based meta-learner proposed in this work, allow gradients to\" \n\n\n- \"Memory-based Under review as a conference paper at ICLR 2020 methods (Ravi & Larochelle (2017)) that use all of the learner’s parameters as input to a meta-learner tend to break down when using a learner with a large number of parameters (Andrychowicz et al., 2016).\": Just to clarify: The latter paper, which is criticising the former category, is older than the paper representing the former category. Is that right? \n\n- page 2: \"superior parameter updates\". I see that the result has been based on classification accuracy. Notwithstanding the ablative study in the experiments secion, maybe superior parameter updates can be either replaced by a more unequivocal description of the mentioned comparison, or formally defined from there onwards. \n\n- At the ICLR level, I do not think that all this detailed description of backpropagation would be necessary. \n\n- Equation 1 and the description that follows: \"a_l is the layer’s pre-activation output\". Is a_l the post-activation output as well? Equation 1 implies so, doesn't it?\n\n- page 3: \"This limits MAML to domains where a small amount of update steps are sufficient for learning.\": What do you mean by \"This\"? Is it to have inner loops consisting of multiple sequential updates, which is what was referred to as \"preferable\" in the beginning of the same paragraph? i.e. Is the MAML limitation noted in this paragraph a limitation of the vanilla MAML (plus the other versions) as well or solely due to the \"preferred, yet detrimental\" extension of adopting multiple sequential updates?\n\n\n\nMinor:\n- page 1: Supervised few-shot learning \"aims to challenge machine learning models to ...\": It does not challenge ML models; it is a specific ML paradigm. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "H1eDLjf4cr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a meta-learner that learns how to make parameter updates for a model on a new few-shot learning task. The proposed meta-learner is an LSTM that proposes at each time-step, a point-wise multiplier for the gradient of the hidden units and for the hidden units of the learner neural network, which are then used to compute a gradient update for the hidden-layer weights of the learner network. By not directly producing a learning rate for the gradient, the meta-learner’s parameters are only proportional to the square of the number of hidden units in the network rather than the square of the number of weights of the network. Experiments are performed on few-shot learning benchmarks. The first experiment is on Mini-ImageNet. The authors build upon the method of Sun et al, where they pre-train the network on the meta-training data and then do meta-training where the convolutional network weights are frozen and only the fully-connected layer is updated on few-shot learning tasks using their meta-learner LSTM. The other experiment is on Omniglot 20-way classification, where they consider a network with only full-connected layers and show that their meta-learner LSTM performs better than MAML.\n\nThe closest previous work to this paper is by Ravi & Larochelle, who also propose a meta-learner LSTM. The submission states about this work that “A challenge of this approach is that if you want to optimize tens of thousands of parameters, you would have a massive hidden state and input size, and will therefore require an enormous number of parameters for the meta-learner…In Andrychowicz at al. an alternative approach is introduced that avoids the aforementioned scaling problem by individually optimizing each parameter using a separate LSTM…” I don’t believe this is true.\n\nAs stated in the work by Ravi & Larochelle, “Because we want our meta-learner to produce updates for deep neural networks, which consist of tens of thousands of parameters, to prevent an explosion of meta-learner parameters we need to employ some sort of parameter sharing. Thus as in Andrychowicz et al. (2016), we share parameters across the coordinates of the learner gradient. This means each coordinate has its own hidden and cell state values but the LSTM parameters are the same across all coordinates.” Thus, Ravi & Larochelle employ something similar to Andrychowicz at al., meaning that the number of parameters in the LSTM meta-learner there is actually a constant relative to the size of the learner network. \n\nThus, the paper’s contribution relative to Ravi & Larochelle is to propose a LSTM meta-learner with more parameters relative to the learner model. Firstly, I think this comparison should be stated and explained clearly in the paper. Additionally, in order to prove the benefit of such an approach, I think a comparison to the work of Ravi & Larochelle with the exact experimental settings used in the submission (pre-training the convolutional network and only using the meta-learner LSTM to tune the last fully-connected layer) would be helpful in order to validate the usefulness of the extra set of parameters in their proposed meta-learner LSTM.\n\nThe submission also states that “In many cases it is preferred to have an inner loop that consists of multiple sequential updates. However the inner loop’s computational graph can become quite large if too many steps are taken. This often results in exploding and vanishing gradients since the outer loop still needs to differentiate through the entire inner loop (Aravind Rajeswaran (2019), Antoniou et al. (2018)). This limits MAML to domains where a small amount of update steps are sufficient for learning. The LSTM-based meta-learner proposed in this work, allow gradients to effectively flow through a large number of update steps. NORML can therefore be applied to a wide array of domains.” I think this statement should be validated if it is stated. Can it be shown that when making a lot of inner-loop updates, the LSTM meta-learner performs better than MAML because of overcoming the stated issues with differentiation through a long inner loop? The experiments done in the paper involve very few inner loop steps and so I don’t believe the claim is supported.\n\nLastly, the experimental results are not very convincing. Though the authors say they modify the method proposed in Sun et al, the results from Sun et al. are not shown in the paper. Sun et al actually seem to achieve better results than the submission with the same 1-shot and better 5-shot accuracy. Was there a reason these results are not shown in the submission? Additionally, for the Omniglot experiment, is there any reason why it was not performed with the typical convolutional network architecture? Since the original MAML results on 20-way Omniglot are with a convolutional network, using the convolutional network would make the results more meaningful relative to previous work and show that the method is more broadly applicable to convolutional networks.\n\nI believe there are several issues with the paper as stated above. Because of these issues, it is hard to evaluate whether the idea proposed is of significant benefit.\n\nReferences\nAndrychowicz et al. Learning to learn by gradient descent by gradient descent. NIPS 2016.\nRavi & Larochelle. Optimization as a Model for Few-Shot Learning. ICLR 2017.\nSun et al. Meta-transfer learning for few-shot learning.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByeW_VS5Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes NORML, a meta-learning method that 1) learns initial parameters for a base model that leads to good few-shot learning performance and 2) where a recurrent neural network (LSTM) is used to control the learning updates on a small support set for a given task. The method is derived specifically for full connected neural networks, where the meta-learner produces gating factors on the normal gradients (one for each neuron that the parameter is connecting). The method is compared with various published few-shot learning methods on miniImageNet, and an ablation study and detailed comparison with MAML is presented on Omniglot.\n\nUnfortunately, I believe this submission should be rejected, for the following reasons:\n\n1. Limited novelty: it is a fairly incremental variation compared to the Meta-Learner LSTM (Ravi & Larochelle). I actually kind of like the proposed variant, but then I would at least expect a direct comparison with Meta-Learner LSTM. And though the authors try to make a case for why NORML is better than Meta-Learner LSTM, unfortunately they don't actually provide a fair comparison. Indeed, the results for Meta-Learner LSTM on miniImageNet use a very different base learner (i.e. not a ResNet-12) which wasn't pretrained. Same in fact can be said about many of the results reported in Table 1.\n\n2. Missing baselines: though the comparison in Table 2 with MAML is a good step, I actually believe the 3 versions of MAML considered aren't appropriate. Instead, I believe that 1) at a minimum, a comparison with a version of MAML where a separate learning rate is learned \"per hidden unit\" AND \"per inner loop step\" is necessary, as it is closer to what NORML can achieve, and 2) a comparison with MAML++ (Antoniou et al.) would be ideal. Finally, I think this study should also be done on miniImageNet, not only Omniglot.\n\n3. Limited applicability: NORML assumes that the base learner is a fully connected network. That strongly limits the applicability of the method, given that most few-shot learners usually have convolutional layers.\n\n4. Inaccurate descriptions of some prior work: The related work includes certain statements that I believe aren't accurate. For example, it is stated that the Meta-Learner LSTM requires \"an enormous number of parameters\". I believe this is not true: there is a single, small LSTM that is used for all parameters of the base learner (however, the authors are right that running this LSTM requires a very large cell state and input size, since the batch size of the Meta-Learner LSTM is essentially the number of parameters of the base learner). Similarly, it is stated that Andrychowicz et al. proposes to \"individually [optimize] each parameter using a separate LSTM\". I believe this is also not true, and that a single LSTM is used for all parameters.  In fact, ironically, NORML on the contrary appears to be using different LSTMs for each layer of the base learner (at least based on Equations 14-19, where the LSTM parameters are indexed by layer ID l, thus implying different LSTMs), unlike what is claimed in the Relate Work section (\"this work uses a single meta-learner\").\n\nFor me to consider increasing my rating, I would expect all points above to be addressed. \n\nFinally, here are a few minor issues I've found:\n- In Equation 16, \"b_{i,\\tilde{c}}\" should be \"b_{l,\\tilde{c}}\" (i.e. i should be l)\n- The submission doesn't mention whether gradients are propagated into the base learner gradients (i.e. whether a first-order version of the method is used)\n- 4th paragraph of the Relate Work section has a missing reference (\"?\")\n- Table 2 is missing confidence intervals", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a LSTM-based meta-learning approach that learns how to update each neuron in another model for best few-shot learning performance.

The reviewers agreed that this is a worthwhile problem and the approach has merits, but that it is hard to judge the significance of the work, given limited or unclear novelty compared to the work of Ravi & Larochelle (2017) and a lack of fair baseline comparisons.

I recommend rejecting the paper for now, but encourage the authors to take the reviewers' feedback into account and submit to another venue.
Decision: Reject