Online Meta-Learning via Learning with Layer-Distributed Memory
Authors: Sudarshan Babu
Pedro Henrique Pamplona Savarese
Michael Maire
Venue: NeurIPS 2021 Poster (2021)
Keywords: Meta-learning
Memory models
Online learning

Abstract: We demonstrate that efficient meta-learning can be achieved via end-to-end training of deep neural networks with memory distributed across layers.  The persistent state of this memory assumes the entire burden of guiding task adaptation.  Moreover, its distributed nature is instrumental in orchestrating adaptation.  Ablation experiments demonstrate that providing relevant feedback to memory units distributed across the depth of the network enables them to guide adaptation throughout the entire network.  Our results show that this is a successful strategy for simplifying meta-learning -- often cast as a bi-level optimization problem -- to standard end-to-end training, while outperforming gradient-based, prototype-based, and other memory-based meta-learning strategies.  Additionally, our adaptation strategy naturally handles online learning scenarios with a significant delay between observing a sample and its corresponding label -- a setting in which other approaches struggle.  Adaptation via distributed memory is effective across a wide range of learning tasks, ranging from classification to online few-shot semantic segmentation.

Online Meta-Learning via Learning with Layer-Distributed Memory
Sudarshan Babu
Pedro Savarese
Michael Maire University of Chicago mmaire@uchicago.. edu
TTI-C
TTI-C savarese@ttic.edu
sudarshanOttic.dd
Abstract
We demonstrate that efficient meta-learning can be achieved via end-to-end training of deep neural networks with memory distributed across layers. The persistent state of this memory assumes the entire burden of guiding task adaptation. Moreover, its distributed nature is instrumental orchestrating adaptation. Ablation experiments demonstrate that providing relevant feedback to memory units distributed across the depth the network enables them guide adaptation throughout the entire network. Our results show that this is successful strategy for simplifying meta- learning - often cast as a bi-level optimization problem to standard end-to-ent training, while outperforming gradient-based, prototype-based, and other memory- based meta-learning strategies. Additionally, our adaptation strategy naturally handles online learning scenarios with significant delay between observing sample and its corresponding label - a setting in which other approaches struggle. Adaptation via distributed memory is effective across wide range of learning tasks, ranging from classification to online few-shot semantic segmentation.
Introduction
Meta-learning or learning-to-Uee to-learn a paradigm that enables models to generalize to a distribution of tasks rather than specialize to just task [1, 2]. When encountering examples from new task, we would like the model to adapt to the new task after seeing just a few samples. This is commonly achieved via episodic training of deep neural networks, where, in each episode, the network is exposed to variety of inputs from the same distribution [3, 4], and the distribution shifts over episodes. The ability of deep networks to adapt to new task within just a few samples or iterations is central the application of meta-learning methods in few-shot and online learning scenarios [5, 6].
A recent surge of interest directed towards meta-learning using neural networks has spurred develop- ment of a variety of methods [7-9]. In standard episodic training framework, a network must adapt to sampled task (or collection of tasks) and incurs generalization loss for that task (or collection); this generalization loss is backpropagated to update the network weights. Methods differ in the underlying architecture and mechanisms they to support adaptation. Strategies include using gradient descent in an inner loop, storing and updating prototypes, parameterizing update rules by another neural network, and employing neural memory [3, 10-12]. Section 2 provides overview.
focus on memory-based meta-lcarning, and specifically investigate the organization of neural memory for meta-learning. Motivating this focus is the generality and flexibility of memory-base approaches. Relying on memory for adaptation allows one to cast meta- meta-learning as merely learning problem using straightforward loss formulation (viewing entire episodes examples) and standard optimization techniques. The actual burden of adaptation becomes an implicit responsibility of the memory subsystem: the network must learn to use its persistent memory manner that facilitates task adaptation. This contrasts with explicit adaptation mechanisms such as stored prototypes.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure I: Adaptation in activation space. A trained memory- bassed model F' adapting two different tasks (red and blue path) using the corresponding per- sistent states hy and A the 7th time step of both tasks. xi and x!! are samples of task I (red) and (bluc) at time stop i.
F Training
F u1
" = F* (x (1) 1
21 = F* (a (2) h (2))
Fn
In this implicit adaptation setting, memory architecture plays crucial role in determining what kind of adaptation can be learned. We experimentally evaluate the effectiveness of alternatvee neural memory architectures for meta-learning and observe particular advantages distributing memory throughout network. More specifically, we view the generic LSTM equations, Wx + Wh-1, as adaptation induced by hidden states in activation space (see Figure 1). By distributing LSTM memory cells across the depth of the network, each layer is tasked with generating hidden states that are useful for adaptation. Such memory organization is compatible with many standard networks, including CNNs. and can be achieved by merely swapping LSTM memory cells in place of existing filters.
Our simple approach also contrasts with several existing memory- based meta-learning methods used in both generative and classification tasks [13-17]. These methods view memory as a means to store and retrieve useful inductive biases for task adaptation, and hence focus on designing better read and write protocols. They typically have a feature extractor that feeds into a memory network that performs adaptation, whereas our architecture makes no such distinction between stages.
We test the efficacy of network architectures with distributed memory cells on online few-shot and continual learning tasks as in Santoro et al. [13], Ren et al. [18] and Javed and White [6]. The online setting is challenging for two reasons: 1) It is empirically observed that networks are not well suited for traniiggadaptatioon with batch size of one [19]: 2) In this setting the model has adapt to one image at a time step, thus having to deal with a prolonged adaptation phase. For these reasons, we these tasks as suitable for evaluating the adaptation capabilities of the hidden states generated by the network.
We empirically observe that our method outperforms strong gradient-based and prototypical baselines, delineating the efficacy of the local adaptation rule learnt by each layer. Particularly important is the distributed nature of our memory, which allows every network layer to adapt when provided with label information; in comparison, restricting adaptability to only later network layers delivers far less compelling performance. These results suggest that co-design of memory architecture and meta- training strategies should be a primary consideration in the ongoing development of memory- based meta-learning, We further test our model in harder online few-shot Icarning scenario. wherein the corresponding label to a sample arrives after a long delay [20]. Our method adapts seamlessly, without requiring any changes to the model, while, in this setting, other adaptation strategies struggle. These results highlight promising directions for advancing and simplifying meta-learning by relying upon distributed memory for adaptation.
2 Related Work
Early work on meta-learning introduces many relevant concepts. Schmidhuber [21] proposes using task specific weights, called fast weights, and weights that are adapted across tasks, called slow weights. Bengio et al. [2] updates the network via a learning rule which is parameterized by another neural network. Thrun [22] presents meta-learning in a life-long scenario, where the algorithm accrues information from the past experiences to adapt effectively for the task at hand. Hochreiter et al. [23] train a memory network to learn its own adaptation rule via just its recurrent states. These high level concepts can be seen in more recent methods. We group current meta-learning methods based on the nature of adaptation strategy and discuss them below.
Gradient-based Adaptation Methods. Methods that adapt via gradients constitute prominent class of meta- learning algorithms [9]. Model-agnostic meta- leta-lea (MAML) [4] learns an initialization that can efficiently be adapted by gradient descent for new task. Finn et al. [24] focus on learning a network that can use experience from previously seen tasks for current task adaptation. They adapt to the current task by using a network that is MAML pre-trained on the samples from the previous task. Nagabandi al. [25], Caccia et al. [26] perform online adaptation under non-stationary distributions, either by using a mixture model or by spawning MAML pre-traince network when the

---

input distribution changes. Javed and White [6], Beaulieu et al. [27] employ a bi-level optimization routine similar to MAML, except the outer loop loss is catastrophic forgetting. They thereby learn representations that are robust to forgetting and accelerate future learning under online updates.
Memory and Gradient-based Adaptation. Andrychowiez al. [28], Ravi and Larochelle [10] learn an update rule for network weights by transforming gradients via LSTM, which outperforms human-designed and fixed SGD update rules. Munkhdalai and Yu [29] learn a transform that maps gradients to fast (task specific) weights, which are stored and retrieved via attention during evaluation. They update slow weights (across task weights) at the end of each task.
These methods learn an encoder which projects training data to a metric space, and obtain blass-wise prototypes via averaging representations within the same class. Following this, test data is mapped to the same metric space, wherein classification is achieved via a simple rule (e.g,, nearest neighbor prototype based on either euclidean distance cosine similarity) [5, 30, 31]. These methods are naturally amenable for online learning as class-wise prototypes can be updated in an online manner as shown by Ren al. [18].
Prototypical Methods.
Memory-based Adaptation. Santoro et al. [13] design efficient read and write protocols for a Neural Turning Machine [32] for the purposes of online few-shot learning, Rae et al. [33] design sparse read and write operations, thereby making them scalable in both time and space. Ramalho and Garnelo [11] use logits generated by the model to decide if certain sample is written to neural memory. Mishra et al. [7] employ an attention-
mechanism to perform adaptation, and use a CNN to generate features for the attention mechanism. Their model requires storing samples across all time steps explicitly, thereby violating the online learning assumption of being able to access each sample only once. All of these methods mainly focus on designing better memory modules either via using more recent attention mechanisms or by designing better read and write rules to neural memory. These methods typically use CNN which is not adapted for the current task. Our approach differs from these methods, in that we study efficient organization of memory for both online few-shot learning and meta-learning more generally, and show that as consequence of our distributed memory organization, the entire network is capable of effective adaptation when provided with relevant feedback.
Kirsch and Schmidhuber [34] introduce an interesting form of weight sharing wherein LSTM cells (with tied weights) distributed throughout the width and depth of the network, however each position has its own hidden state. Further, they have backward connections from the later layers to the earlier layers, enabling the network implement its own learning algorithm clone human- designed learning algorithm such as backprop. Both our model and theirs implement an adaptation strategy purely using the recurrent states. The difference, however, is in the nature of the adaptation strategy implemented in the recurrent states. Similar to conventional learning algorithms, their backward connections help propagate error from the last layer to the earlier layers. In our architecture, the feedback signal is presented as another input, propagated from the first layer to the last layer.
In addition to being used in classification settings, Guez al. [35] employ memory- based meta- learning approach to perform adaptation for reinforcement learning tasks indicating the generality of using memory as a means for adaptation.
Few-shot Semantic Segmentation. Few-shot segmentation methods commonly rely on using proto- types [36, 37], though recent approaches include gradient-based methods analogous to MAML [38]. The methods that use neural memory typically employ it in final network stages to fuse features of different formats for efficient segmentation: Li et al. [39] use ConvLSTMs [40] to fuse features from different stages of the network: Valipour et al. [41] to fuse apatio-tempore features while segmenting videos; Hu et al. [42] use ConvLSTM to fuse features of query with the features of support set; Azad et al. [43] use a bidirectional ConVLSTM to fuse segmentation derived from multiple scale space representations. We differ from these works in organization,
of, and information provided to memory module: 1) Memory is distributed across the network as the sole driver of adaptation; 2) Label information is provided to assist with adaptation.
Meta-learning Benchmarks. Caccia et al. [26] present benchmarks that measure the ability of a model to adapt to new task, using the inductive biases that it has acquired over solving previously seen tasks. More specifically, the benchmark presents an online non-stationary stream of tasks, and the model's ability to adapt to new task at each time step is evaluated. Note that they do not measure
3

---

the model's s ability to remember earlier tasks; they only want the model to adapt well on a newly presented task.
Antoniou et al. [44] present henchmarks for continual few-shot learning. The network is presented a number of few-shot tasks, one after the other, and then is expected generalize even to the previously seen tasks. This is a challenging and interesting setup, in that, the network has to show robustness to catastrophic forgetting while learning from limited data. However, we are interested in evaluating the online adaptation ability of models, while Antoniou et al. [44] feed data in a batch setting. We follow experimental setup as in Javed and White [6], where in, the model required to remember inductive biases acquired over longer time frame when compared to the experimental setup used by Antoniou al. [44].
3 Methodology
3.1 Problem: Online Few-shot Learning
This setting combines facets of online and few-shot learning: the model is expected to make predic- tions stream of input samples, while it sees only a few samples per class in the given input stream. In particular, we use a task protocol similar to Santoro et al. [13]. At time step i, an image zy is presented the model and it makes a prediction for z; In the following time step, the correct label yi is revealed to the model. The model's performance depends on the correctness of its prediction at each time step. The following ordered set constitutes task: T ((x1, null), (I2,, y1 y1),,. (Irz, It)-)). Here relll indicates that no label is passed at the first time step, and i is the total number of time steps (length) of the task. For k-way N-shot task / k N. The entire duration of the task is considered as the adaptation phase, as with every time step the model gets a new sample and must adapt on it to improve its understanding of class concepts.
3.2 Memory as Adaptation in Activation Space
Consider modulating the output of network F for input x with a persistent state h: u F(x, h). Now, if adding h aids in realizing a better representation u than otherwise (F(x)), we could view this as adaptation in activation space. In Figure 1, model F adapts to tasks using its persistent states h. Specifically let us consider the generic LSTM equations Wx + W h-1, we could view Wa as the original response and W h- as modulation by a persistent state (memory) the activation space. So, for the online learning task at hand, we seek to train LSTM which learns to generate hidden state hú at each time step i, such that it could enable better adaptation ensuing time steps. We note that adaptation in activation space has been discussed in earlier works. We use this perspective to organize memory better and to enable effective wase adaptation across the network.
3.3 Model
Architecture, We distribute memory across the layers of the network, in order to enable the layers to learn local layer-wiss wise adaptation rules, In particular, we use model in which each layer of the feature extractor is a convolutional LSTM (CL) [40] followed by a LSTM [45] and a classifier, as shown in Figure 2.
Similar to the LSTM, each convolutional LSTM (CL) layer consists of its own input, forget, and output gates. The key difference is that convolution operations (denoted by *) replace matrix- vector multiplication. In this setup, we view the addition by Wh, * fz-1 as adaptation in the ins time step within the input gate. The same view could be extended to other gates as well. The cell and hidden state generation are likewise similar LSTM, but use convolution operations:
4 a WW i * + Who h(-1) (1)
4 = ft Cr- + O tanh(W,, * xt + Wng * h:-
1)
f = a (,, * xt + Why * 14-1) (2)
(4)
0, o (,,, I, Who h (3)
h = o2 C tanh(ci)
(5)
In initial experiments, we observe that for tasks with time steps these models did not train well. We hypothesize that this could be due to the same network being repeated 50 times, thereby inducing

---

ComLSSMM
ComLSTTO
ComLSTMM
Com
LSTM
Clasilte
(c, Ax]
CavLSTMM
ConLLSTM
ComL
CanLSTM-
LSTM
Chassfier
h,
Figure 2: Methodology. An example distributed memory architecture consists of four layers of convolutional LSTMs (gray), followed by an LSTM (gray). and a classifier (gray). At the 4 time step, the sample 2; (green) and the previous sample's 's label y-1 different label injection modes are shown. Pink: label is fed to the first CL layer only; Violet: label is fed to each CL layer; Red: label is fed to the LSTM layer. Label information is provided every time step; sake of clarity we avoid showing at time state of the Jeh layer at the in time step. We use cyan, gray, orange, and green to denote persistent states, network parameters, label information, and input sample, respectively. Best viewed color.
presented network. Three
here. 115), cg are the hidden and cell
an effectively very deep network. We resolve this issue by adding skip connections between the second layer and the fourth layer (omitted Figure 2). Further discussion on this is Appendix B.
Label Encoding, As label information is essential for learning an adaptation rule, we inject labels offset by one time step the ConvLSTM feature extractor and the LSTM. This provides the opportunity for each layer to learn an adaptation rule. For k-way classification problem involving images of spatial resolution 8, we feed the label information th row if e is indeed the true label. We reshape this matrix as k x X 8 tensor and concatenate along the channel dimension of image at the time step. To the LSTT layer, feed the label in its one-hot form by concatenating with the flattened activations from the previous layer.
k x s² matrix with all ones in the
Training and Evaluation
Following Santoro et al. [13], we perform episodic training by exposing the model to variety of tasks from the training distribution P(Trrain). For a given task, model incurs a loss L; at every time step of the task; we sum these losses and backpropagate through the sum at the end of the task. This is detailed in Algorithm in Appendix A. We evaluate the model using partition of the dataset that is class-wis disjoint from the training partition. The model makes prediction every time step and adapts to the sequence by using its own hidden states, thereby not requiring any gradient information for adaptation. Algorithm 2 in Appendix A provides details.
4 Experiments
4.1 Online Few-Shot Learning
We use CIFAR-FS [46] and Omniglot [47] datasets for our few-shot learning tasks; see Appendix A for details. We adopt the following methods to serve as baselines for comparison.
LSTM and NTM. Santoro et al. [13] use a LSTM and a NTM [32] with read and write protocols for the task of online few shot learning. Both aim meta-learn tasks by employing a neural memory.
Adaptive Posterior Learning (APL). Ramalho and Garnelo [11] propose a memory-auggeent model that stores data point embeddings based measure of surprise, which is computed by the loss incurred by each sample. During inference, they retrieve a fixed number of nearest-neighbor data embeddings, which are then fed classifier alongside the current sample.
Online Prototypical Networks (OPN). Ren al. [18] extend prototypical networks to the online case, where they sequentially update the current class-wise prototypes using weighted averaging.
5

---

Contextual Prototypical Memory (CPM), Ren et al. [18] improve on OPN by learning represen- tation space that is conditioned on the current task. Furthermore, weights used to update prototypes are determined by newly-introduced gating mechanism.
Table 1 shows that our model outperforms the baselines in most settings. These results suggest that the adaptation rules emergent from our design are more efficient than adaptation via prototypes, and adaptation via other memory-based architectures. In the CIFAR-FS experiments, the prototypical methods outperform our method only in the I-shot scenario. As the 5-shot and 8-shot scenarios have a longer fine-tuning or adaptation phase, this shows that our method is more adept at handling tasks with longer adaptation phases. One reason could be that the stored prototypes which form the persistent state of OPN and CPM are more rigid than the persistent state of our method. The rigidity stems from the predetermined representation size of each prototype. which thereby prevents allocation of representation size depending upon classification difficulty. In our architecture, the network has the freedom to allocate representation size for each class as it deems fit. Consequently, this may help the network learn more efficient adaptation strategies that improve with time.
We examine the importance of distributed adaptation through ablation experiments that vary the layer into which we inject label information. Table 2 shows that models with feature extractors that do not receive label information are outperformed by the model whose earlier layers do receive label information (injecting into CL-1): the latter is even better than pre-trained models. By distributing memory across each layer and allowing label information to flow to each memory module, we enable every layer to learn its own adaptation rule. Here, the CNN baselines are pre-trained with MAML; these pre-trained networks replace the ConvLSTM part and are jointly trained with the L.TTM (which receives the labels) and classifier. In these cases, we just replace the CSTVL.TTM in Figure 2 with a CNN. During the meta-testing phase, the CNN is a just feature extractor and the burden of adaptation falls entirely on the LSTM. In CNN-F, we freeze the weights during meta-training. Our CL+LSTM, restricted to adapt only in the final layer (3rd row; label injection into final LSTM layer only) performs comparably to the CNN baselines. The same model, with full adaptivity (last row) outperforms.
4.2 Delayed Feedback
We consider task similar to online few-shot classification (Section 3.1), except instead of offset by one timestep, labels are offset by delay parameter. Supposing the label delay is 3, then the task T is presented to the model as the sequence: T ((11. wull), (z2, realt), (rg, wull), (r)........ ... (xt. where is the sequence length. The model must discern and account for the time delay.
Table 3 shows that our network can learn under these conditions, though performance decreases with increase in delay. This could imputed to difficulty in associating the hidden representation of a sample with the correct label, consequently creating a noisy environment for learning adaptation rules. We see that pre-training helps: we take our network pre-trained for label delay of I and meta-train for tasks with label delay of 5. This improves the model accuracy to outperform the model directly trained with label delay of 4. This could be because the necessary adaptation rules are already learnt by the pre-trained model, and it only has to learn the quantum of delay. Furthermore, from Tables I and 3, even with a delay of 2 our method outperforms CPM with no delay.
Omniglot 5-shot
CIFAR-FS
Model
1-shot
8-shot
1-shot
5-shot
8-shot
LSTM
85.3 (0.2)
94.4 (0.1)
95.8 (0.6)
NTM
88.7 (0.5)
96.8 (0.1)
97.3 (0.1)
APL
89.1 (0.0)
94.9 (0.0)
95.7 (0.1)
(0.6
45.8 (0.4)
46.9 (0.7)
OPN
91.2 (1.1)
94.6 (1.1)
95.8 (0.5)
19.9(0.2)
54.9(0.4)
56.8 (0.1)
CPM
94.5 (0.1)
97.0 (0.1)
97.4 (0.4)
50.2 (0.2)
55.8 (0.4)
57.8 (0.4)
CL+LSTM
96.8 (0.5)
99.4 (0.2)
99.7 (0.2)
47.6 (1.0)
56.7 (1.6)
61.0 (1.3)
Table Omniglot and CIFAR-FS results for 5-way online few-shot learning. Shown are average and standard deviation across 3 runs. Methods with hand-designed memory mechanisms, like CPM and APL, benefit less from increased number of samples. Distributed memory CL+LSTM comfortably outperforms other adaptation methods. These methods fail to train on CIFAR-FS.

---

Model
Pre-training
Label Injection Layer
1-shot
5-shot
8-shot
CNN-F LSTM
LSTM
89.1 (0.2)
96.4 (0.0)
97.0 (0.2)
CNN+ + LSTM
V
LSTM
93.9 (0.1)
97.9 (0.2)
98.2 (0.4)
CL+LSTM
X
LSTM
94.7 (0.6)
97.3 (0.7)
97.9 (0.6)
CL+LSTM
X
CL-1, LSTM
96.8 (0.5)
99.4 (0.2)
99.7 (0.2)
Table 2: Ablation experiments on Omniglot 5-way online few-shot learning. Results in (%)
(average and deviation across runs), comparing different feature extractors (pre-trained or not) while providing labels offset by 1 time step. Outputs of feature extractors are given to a LSTM (Figure 2). Our distributed memory model (CL+LSTMM) with offset labels provided at the first layer, outperforms other models: since memory is distributed across layers, and since label information is provided to the entire network, each layer has sufficient information to learn its own adaptation rule.
In this setting, our model can be used in a seamless manner, without having to make any adjustments. Gradient-baid based and prototypical methods cannot be used as is, and would require storing the samples (violating online assumption) for the time period of delay, causing memory usage to grow linearly with delay; in contrast, it constant for our method. Further, to use prototypical or gradient-based methods, we would have to know the delay parameter in advance; our network learns the delay.
4.3 Online Continual Learning
We address the problem of continual learning in the online setting. In this setup, the model sees a stream of samples from non-stationary task distribution, and the model is expected to generalize well even while encountering samples from a previously seen task distribution. Concretely, for a single continual learning task construct subtasks from underlying dataset and first present to the model samples from the first subtask, then the second subtask, so on until the n subtask that order. Once the model is trained on all n subtasks sequentially, it expected to classify images from any of the subtasks, thereby demonstrating robustness catastrophic forgetting [48].
Task Details. We use the Omniglot dataset for our experiments. Following [6]. we define each subtask as learning single class concept. So this protocol a single online 5-way 5-shot continual learning task is defined as the following ordered set: T (Ti, T2. T3, T1, T5)- Here, subtask T contains 5 samples from 1 particular Omniglot class. After adaptation is done on these subtasks (25 samples) we expect the model to classify samples from a query set consisting of samples from all of the subtasks. The performance of the model is the prediction accuracy on the query set. We experiment by varying the total number of subtasks from 5 to 20 in Figure 3.
Training Details. We perform episodic training by exposing our model to a variety of continual learning tasks from the training partition. At the end of each continual learning task, the model incurs a loss on the query set. We update our model by backpropagating through this query set loss. Note that during evaluation on the query set, we freeze the persistent states of our model in order to prevent any information leak across the query set. Since propagating gradients across long time steps renders training difficult, we train our model using simple curriculum of increasing task length every 5K episodes. This improves generalization and convergence. Appendix C presents more details. Further, we shuffle the labels across tasks in order to prevent the model from memorizing the training classes. During evaluation, we sample tasks from classes the model has not encountered. The model adapts to the subtasks using just the hidden states and then acquires the ability to predict on the query set, which contains samples from all of the subtasks. We use the same class wise disjoint train/test split as in Lake [47].
Delay
Pre-training
6-shot
7-shot
8-shot
Table 3: CIFAR-FS results (%) for 5- way online few-shot learning with de- layed labels (average and deviation, in parentheses, across 2 runs). As expected, longer delays degrade the model's performance. Further, we ob- serve that pre-training model with 1- step delay improves the performance in tasks with 5-step and 4-step delays.
2
X
56.8(0.1)
58.4(0.2)
60.0(0.0)
4
x
52.2(0.4)
53.1(0.3)
54.1(0.4)
4
/
56.1(0.6)
57.6(0.3)
59.0(0.1)
5
X
51.1(0.8)
51.7(0.7)
52.7(0.3)
5
55.2(0.3)
56.4(0.0)
57.4(0.4)
7

---

110
Figure 4: Accuracy of subtasks within a 20 task con- tinual learning task. Typically in earlier tasks (first 10), CL+LSTM seems to have higher accuracy than ANML and OML, suggesting that it is more immune forgetting. ANML performs poorly in the 20th task, about 56 % we impute that to momentum issues in training. Note: Exclud- ing the 20th task, the average ANML performance in the first tasks is 91.78, while CL+LSTM averages 93. in the first 19 tasks.
Figure 3: Online few-shot continual learning. Accuracy vs task length on Omniglot. CL+LSTM model all outper- forms the baselines across tasks lengths, strongly suggesting that the CL+LSTM model is adept storing inductive biases required to solve the subtasks within a given continual learning task.
Baseline: Online Meta Learning (OML). Javed and White [6] adopt meta-training strategy similar to MAML. They adapt deeper layers in the inner loop for the current task, while updating the entire network in the outer loop, based on a loss measuring forgetting. For our OML experiments we use 4-yyeer CNN followed by two fully connected layers. Appendix C provides implementation details.
Baseline: A Neuromodulated Meta-Learning Algorithm (ANML). Beaulieu et al. [27] use a hypernetwork to modulate the output of the trunk network. In the inner loop, the trunk network is adapted via gradient descent. In the outer loop, they update both the hypernetwork and the trunk network on a loss measuring forgetting. For our ANML experiments, we use 4-layer CNN followed by a linear layer as the trunk network, with a 3-layer hypernetwork modulating the activations of the CNN. They use 3 times as many parameters as our CL+LSTM model. Appendix C provides details.
Results. Figure 3 plots average accuracy on increasing the length of the continual learning task. Task length is the number of subtasks within each continual learning task, which ranges from 5 to 20 subtasks in our experiments. As expected, we observe that the average accuracy generally decreases with increased task length for all models. However, the CL+LSTM model's performance degrades slower than the baselines, suggesting that the model has learnt an efficient way of storing inductive biases required to solve each of the subtasks effectively.
From Figure 4, we see that CL+LSTM is robust against forgetting, as the variance on performance across subtasks is low, This suggests that the CL+LSTM model learns adaptation rules that minimally interfere with other tasks.
Analysis of Computational Cost. During inference, our model does not require any gradient computation and fully relies on hidden states to perform adaptation. Consequently, it has lower computational requirements compared to gradient-baset models assuming adaptation is required at every time step. For a comparative case study, us consider three models and their corresponding GFLOPs per forward pass: OMI. baseline (1.46 GFL.Pss:: CL+LSTM (0.40 GFL.Oss;: 4-layer CNN (0.30 GFLOPs) with parameter count similar to CL+LSTM. Here, we employ standard methodology for estimating of compute cost [49], with a forward and backward pass together incurring three times the operations in a forward pass alone.
We can extend these estimates to compute GFLOPs for the entire adaptation phase. Suppose we are adapting/updating our network on task of length iterations. The OML baseline and the 4- CNN (adapting on gradient descent) would consume 4.38t GFLOPs and 0.9t GFLOPs, respectively. Our CL+LSTM model would consume only 0.40z GFLOPs; here, we drop the factor of three while computing GFL.Oss for the CL+LSTM model, since we do not require any gradient computation for adaptation. During training, we lose this advantage since we perform backpropagation through time, making the computational cost similar to computing meta-gradients.
8

---

Time Stop-1
"S-2
T5-6
15-7
15-0
15-9
15-10 TS-11 15.12 TS-11 T4-14 15-15
TS. 16
15.17
75. 1B
Figure 5: Sample online few-shot segmentation task with distractors. At each time step, the model gets one image, and its corresponding ground truth in the subsequent time step. The model is tasked with either segmenting masking, depending on whether or not the image is distractor. Here, the fish are distractors and the ducks are objects we want to segment. The portion highlighted in green is the predicted segmentation, and the red portion is the predicted mask. These are results from our 10-ConyLSTM model with label injection at the first layer.
4.4 Online few-shot Semantic Segmentation
These experiments investigate the efficacy and applicability of adaptation via persistent states to challenging segmentation task and analyze the effectiveness of label injection for segmentation.
Task Details. We consider binary segmentation task: we present the model a sequence of images, one at each time step (as in Figure 5), and the model must either segment or mask out the image based on whether it is distractor. Similar to the classification tasks, we augment the ground truth segmentation information along the channel dimension. The ground truth is offset by time step. so at the first time step we concatenate to the channel dimension an all -1 matrix as a null label, at the next time step we concatenate to the channel dimension the actual ground truth of the image at time step 1. If it is an image to be segmented, we concatenate the ground truth binary mask of the object and the background in the form channel dimension an all zeros matrix indicating that the entire image should be masked out. k-shot scores for segmentation is the IoU of the predicted segmentation on the k + 1th time the model sees the object we want to segment out. For k-shot masking scores, we compute the fraction of the object that has been masked, when model sees the distractor image for the + 1th time step. We sample episodes from the dataset $SS1000 [50]; more dataset details are in Appendix D.
binary matrix. If it is distractor image, we concatenate to the
The construction of this task avoids zero-shot transfer of inductive biases required for segmentation and forces the model to rely on the task data to learn which objects are to be segmented.
Training Details. We augment 10 layer U-Net [51] like CNN with memory cells in each layer, by converting each convolution into a convolutional LSTM-referred to as CL U-Net (architecture details in Appendix D. We utilize episodic training. where each episode is an online few-shot segmentation task, as in Figure with 18 time steps in total (9 segmentation images and 9 distractors). We follow a simple training curriculum train: the first 100k episodes we train without any distractors; in the next 100k episodes we train with distractors as in Figure 5. Further training details are in Appendix D. The episodes presented during evaluation contain novel classes.
We use a 10 layer U-Net like CNN pre-trained with MAML for segmentation without any distractors (architecture details in Appendix D). We use this model as fine-tuning CNN baseline, in that we fine-tune the model on the online stream of images using gradient descent at each time step. From Table 4, we see that the model fails mask out the distractors, indicating its inability to ability to adapt to the online feed.
Baselines.
From Table 4, we see that CL U-Net variants are capable of effective online adaptation; both models are capable of segmenting and masking images. However we observe that providing label information at the first layer significantly boosts our performance, thereby bolstering our claim that effective task adaptation can be achieved by providing relevant feedback to a network containing distributed memory.
4.5 Standard Supervised Learning
Finally, we assess whether our proposed model can be directly employed in a classic supervised learning setting i.e., without requiring modifications in terms of architecture design. The central

---

Segmentation 6-shot
Masking
Model
Variants perform comparably original models, Label Injection
1-shot
1-shot
6-shot
Fine-tuned CNN
No
65.3
64.0
23.1
23.6
CL U-Net
9th layer
38.7
45.7
73.8
77.8
CL U-Net
1st layer
49.2
56.0
84.6
91.7
Table 4: Segmentation and Masking results on CL U-Net. Fine-tuning CNN fails to adapt to distractors, while CL U-Net variants demonstrate adept capacity for adaptation. Further label injection at the first layer outperforms label injection at the 9th layer, suggesting that label injection enables network wide adaptation.
Model
Params (M)
C10
C100
Table 5:
Test accuracy (% of supervised learning on CIFAR. Shown are averages across runs. Parentheses following model
VGG-I1
91.5
66.9
CL-VGG-11 (0.5x)
20.0
91.5
67.3
CL-VGG-11 (0.35x)
9.5
90.7
64.3
names indicate shrinkage factor of filter sets.
ResNei-20
0.3
9I.9
68.
CL-ResNet-20 (0.5x)
90.9
63.2
showing that they can be employed off-the-shelf
2L-ResNet-20 (0.35x)
0.3
89.5
58.8
in both meta and supervised learning settings.
motivation behind these experiments is see if meta-learning methods can be applied to standard supervised learning tasks without requiring any change in methodology. Hence, in a setting when a priori knowledge whether the task hand standard supervised learning task or meta-learning task is unavailable, we could use ConvLSTM models. This is similar to the experiments done in [11], where they try to close the gap between standard supervised learning approaches and their meta-learning method applied to standard supervised learning tasks.
We use CIFAR data our standard supervised learning benchmark [52]; further dataset details are in Appendix E. We use standard networks such as VGG [53] and ResNet [54] as our baselines. In Table we observe that CL variants perform comparably in most cases. This affirms that the ConvLSTM model is capable of handling a conventional supervised learning scenario without any change training procedure. Even in the absence of temporal signal, ConvLSTMs can still operate well. This is interesting since direct application of gradient-base based meta-learners to the conventional supervised learning setting would require optimizing through a prohibitively long inner loop.
5
Conclusion
Our results highlight distributed memory architectures as a promising technical approach to recasting the problem of meta-learning as simply learning with aegmer--uggmented models. This view has potential to eliminate the need for ad-hoc design of mechanisms or optimization procedures for task adaptation, replacing them with generic and ggneral-purposs memory modules. Our ablation studies show the effectiveness of distributing memory throughout deep neural network (resulting in an increased capacity for adaptation), rather than limiting to a single layer or final classification stage.
We demonstrate that standard LSTM cells, when provided with relevant feedback, can act as a basic building block of a network designed for meta-learning On wide variety of tasks, a distributed memory architecture can learn adaptation strategies that outperform existing methods. The applica- bility of purely memory-bar based network to online semantic segmentation points to the untapped versatility and efficacy of adaptation enabled by distributed persistent states.
Acknowledgments and Disclosure of o Funding
We thank Greg Shakhnarovich and Tri Huynh for useful comments. This work was supported in part by the University of Chicago CERES Center. The authors have no competing interests.
References
[1] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-.. hook. PhD thesis, Technische Universität München, 1987.
10

---

[2] Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality Artificial and Biological Neural Networks, volume 2, 1992.
[3] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, and Daan Wierstra. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3633333 3638, 2016.
[4] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.
[5] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, Curran Associates, Inc., 2017.
[6] Khurram Javed and Martha White. Meta-learning representations for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[7] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. simple neural attentive meta-learner. In 6th International Conference on Leurning Representations, ICLR 2018, Vun- couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.ner, 2018.
[8] Joaquin Vanschoren. Meta-learning: survey. arXiv preprint
2018.
[9] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. arXiv preprint arXiv:2004. 05439, 2020.
[10] Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In Sth International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[11] Tiago Ramalho and Marta Garnelo. Adaptive posterior learning: few-shot learning with surprise-based memory module. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[12] Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. 8th International Conference on Learn- ing Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[13] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta-learning with memory-augmenter neural networks. In Proceedings of The 33rd Inter- national Conference on Machine Learning. volume 48 of Proceedings of Machine Learning Research, pages 1842-1850. PMLR, 20-22 Jun 2016.
[14] Sergey Bartunov, Jack Rae, Simon Osindero, and Timothy Lillicrap. Meta-learning deep energy-based memory models. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[15] Yan Wu, Greg Wayne, Alex Graves, and Timothy P. Lillicrap. The kanerva machine: A generative distributed memory. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Truck Proceedings. OpenReview.net, 2018.
[16] Yan Wu, Gregory Wayne, Karol Gregor, and Timothy Lillicrap. Learning attractor dynamics for generative memory. In Advunces in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 9401-9410, 2018

---

[17] Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals. Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In International Conference Machine Learning, pages 2827-2836. PMLR, 2017.
[18] Mengye Ren, Michael Louis Iuzzolino, Michael Curtis Mozer, and Richard S. Zemel. Wandering within a world: Online contextualized few-shot learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[19] Yoshua Bengio. Practical recommendations for gradient- based training of deep architectures. In Neural networks: Tricks of the trade, pages 437-478. Springer, 2012.
[20] Chris Mesterharm. On-line learning with delayed label feedback. In International Conference on Algorithmic Learning Theory, pages 399-413. Springer, 2005.
[21] Jürgen Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139, 1992.
[22] Sebastian Thrun. Lifelong learning algorithms. In Learning to learn, pages 181-209. Springer, 1998.
[23] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pages 87-94. Springer, 2001.
[24] Chelsea Finn, Aravind Rajeswaran, Sham M. Kakade, and Sergey Levine. Online meta-learming In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 1920-1930. PMLR, 2019.
[25] Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model- based RL. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[26] Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexande Lacoste, David Vazquez, et al. Online fast adap- tation and knowledge accumulation: a new approach to continual learning. arXiv preprint arXiv:2003.05856 2020.
[27] Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to continually learn. In ECAI 2020 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 September 8, 2020 Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications. pages 992-1001. IOS Press, 2020.
(28) Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3981 13899, 2016
[29] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pages 2554-2563. PMLR, 2017.
[30] Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pages 719-729, 2018.
[31] Yingwei Pan, Ting Yao, Yehao Li, Yu Wang, Chong- Wah Ngo, and Tao Mei. Transferrable prototypical networks for unsupervised domain adaptation. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pages 2239-2247, 2019.
12

---

[32] Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXhv:1410.540/ 2014.
[33] Jack Rae, Jonathan Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, and Timothy P Lillierap. Scaling memory reads and writes. arXiv preprint arXiv:1610.09027, 2016.
memory-augmenter -ugmented neural networks with sparse
[34] Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. arXiv preprint 10X.v:2012.1490000 2020.
[35] Arthur Guez, Mchdi Mirza, Karol Gregor, Rishabh Kabra, Sébastien Racanière, Théophane Weber, David Raposo, Adam Santoro, Laurent Orscau, Tom Eccles, et al. An investigation of model-free planning. In International Conference on Machine Learning, pages 2464-2473. PMLR, 2019.
[36] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In British Machine Vision Conference 2017, BMVC 2017, London, UK, September 4-7, 2017. BMVA Press, 2017.
[37] Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha A. Efros, and Sergey Levine. Con- ditional networks for few-shot semantic segmentation. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Work- shop Track Proceedings. OpenReview.net, 2018.
[38] Atmadeep Banerjee. Meta-dmn Meta-learning for 1-shot image segmentation. arXiv preprint arXiv:2008.00244 002477 2020.
[39] Ruiyu Li, Kaican Li, Yi-Chun Kuo, Michelle Shu, Xiaojuan Qi, Xiaoyong Shen, and Jiaya Jia. Referring image segmentation via recurrent refinement networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 5745-5753, 2018
[40] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wii-Kin Wong, and Wang-chun Woo. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebee, Canada, pages 802-8 810, 2015.
[41] Sepehr Valipour, Mennatullah Siam, Martin Jagersand, and Nilanjan Ray. Recurrent fully con- volutional networks for video segmentation. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 29-36. IEEE, 2017.
[42] Tao Hu, Pengwan Yang, Chiliang Zhang, Gang Yu, Yadong Mu, and Cees GM Snock. Attention- based multi-context guiding for few-shot semantic segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8441-8444 2019.
[43] Reza Azad, Abdur R Fayjie, Claude Kauffmann, Ismail Ben Ayed, Marco Pedersoli, and Jose Dolz. On the texture bias for few-shot cnn segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2674-2683. 2021.
[44] Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, and Amos Storkey, Defining benchmarks for continual few-shot learning. arXiv preprint arXiv:2004.1196) 2020.
[45] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
[46] Luca Bertinetto, João F. Henriques, Philip H. S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[47] Brenden Lake. Omniglot git repo, 2015. URL.
13

---

[48] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinn et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526, 2017.
[49] OpenAi. Ai and compute, 2018. URL
wtt.ccoo////./.eecc and compute/.
[50] Xiang Li, Tianhan Wei, Yau Pun Chen, Yu- YunWing Tai, and Chi-Keung Tang. Fss-1000: A 1000-class dataset for few-shot segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2869-2878, 2020.
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234 241. Springer, 2015.
[52] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[53] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[54] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778, IEEE Computer Society, 2016.
[55] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332-133333 2015.
[56] Jake Snell. Github implementation of prototypical-networks, 2017.
URL
https://gtt thub.
[57] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[58] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICMI., 2015.
[59] Tim Cooijmans, Nicolas Ballas, César Laurent, Çağlar Gülçehre, and Aaron Courville. Recur- rent Batch Normalization. ICLR, 2017.
[60] R. J. Williams and J. Peng. An efficient gradient- algorithm for on-line training of recurrent network trajectories. Neural Computation, 1990.
14


Reviews:
[{"review_id": "guneoPm0W5", "reviewer": "Reviewer_oAcd", "summary": "The paper proposes to adapt Meta RNNs (memory-based meta-learning) by instantiating multiple layers of LSTMs where some of them are convolutional.\nThe authors refer to this as distributed memory due to each layer having its own LSTM-based memory.\nThey demonstrate good performance in few-shot learning and continual learning.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality and Significance\n\nThe authors suggest that instead of using a single LSTM in MetaRNNs (memory-based meta-learning) multiple layers should be added and some should be convolutional.\nWhile this seems like a fairly incremental change, they demonstrate convincingly that this helps in few-shot learning and continual learning.\nIn particular, the improvement / similar performance over ANML and OML is interesting as no inner gradients are required.\n\n## MetaRNNs (memory-based meta-learning) not correctly cited\n\nThe proposed architecture is very similar to Meta RNNs (supervised [1], RL [2,3]) that the authors do not cite. They only discuss other forms of external memory in the related work section. The author's contribution of adding multiple layers of LSTMs and making some of them convolutional should be highlighted more clearly. This contribution tends to get lost in the paper over discussions that also apply to previous work.\n\n[1] Hochreiter, S., Younger, A. S., & Conwell, P. R. (2001). Learning to learn using gradient descent. International Conference on Artificial Neural Networks.\n\n[2] Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., & Abbeel, P. (2016). RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning. ArXiv Preprint ArXiv:1611.02779.\n\n[3] Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., & Botvinick, M. (2016). Learning to reinforcement learn. ArXiv Preprint ArXiv:1611.05763.\n\n## Connections to other recent work on distributed memory / fast weights should be added\n\nRecent work [4] also introduced a form of distributed memory for meta-learning online adaptation strategies / learning algorithms. In that case, each weight in a neural network is replaced by an LSTM, which is a different but related architectural choice compared to the proposed work. The connection should be discussed and cited.\n\n[4] Kirsch, L., & Schmidhuber, J. (2020). Meta Learning Backpropagation And Improving It. ArXiv Preprint ArXiv:2012.14905.\n\n## Conclusion\n\nOverall I think it is a good paper with interesting results on meta-learning with distributed memory. Unfortunately, it is missing some crucial relationships to previous work. I would like to give the paper a good score based on appropriate adjustments in the rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and potential negative societal impact is adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Tj77AR2-DS", "reviewer": "Reviewer_U3Jn", "summary": "In this paper the authors propose a distributed memory system to tackle the online few-shot setting. The system is based on layers of conv-LSTMs that take as input an image and the labels at previous steps to generate activations and store information. \n\nOverall I think that the technical contribution is marginal. Both the system structure and the problem formulation make the method impractical and difficult to scale to more complex settings. Experiments are limited to small datasets and do not show the potential of the proposed solution. Limitations are not appropriately discussed and a better framing w.r.t. more recent online few-shot learning work is missing. In the current form the paper is not ready to be accepted. I am open to change my mind and increase my score if the authors provide a strong rebuttal and a satisfactory answer to the concerns I have outlined below.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1) Clarification on the importance of having a memory. I have checked algorithm 1 in the appendix and the details of how task are sampled and aggregated in appendix A.1. The authors wrote: \"we sample 10-shot 5-way online few-shot tasks by sampling 5 classes and 10 samples per class (a total of 50 images), which are fed to the model at a rate of one image per time step\". My understanding is that those 50 images are shuffled and presented in a random order. I am wondering if there are simpler ways to manage this setting, for instance by just accumulating the classes seen so far into bins and then pass the histogram instead of the one-hot label. This could be done by simply summing the one-hot vectors seen so far (and optionally normalize by the total number of images in the sequence). One could have a standard neural network that makes a prediction by using the image at time t and the accumulated histogram. If I am correct in my assumptions, then this seems to be a trivial baseline to try in order to justify the use of a memory. In other words, why we need a complex memory mechanism if the only information that matters is the current image and the labels accumulated so far in the sequence? The authors should clarify this point and provide empirical evidences that this trivial baseline is not as competitive as the proposed method.\n\n2) Problem setting. The particular formulation used in this work has been used previously, for instance in [1], and it is based on the key assumption that labels are delayed by a given factor. However, this formulation is just one of the possible ways meta-learning can be applied to the online setting, and it may suffer several shortcomings when used in practice. For instance, in many real-world settings the feedback can be delayed by a variable factor, it can change between train and evaluation stages, and even be overwritten by inputs belonging to different classes. Recent work has investigated online few-shot learning in some of those challenging settings but the authors did not mention it or discuss it. I suggest including these papers [2,3,4] with an explanation of the advantages/disadvantages of the formulation used by the authors.\n\n3) Computational cost. A satisfactory discussion of computational costs and memory issues has not been provided. The system seems quite expensive in terms of computational resources. The authors pointed out in lines 154-157 some stability issues that they assume are due to: \"the same network being repeated 50 times\". If backpropagation is performed from the last step to the first one, it will require the storage of all the activations along the way. This is a serious limitation, the model may not scale well to large images and deeper backbones due to this bottleneck. A discussion of all these issues is expected.\n\n4) Datasets. The datasets used for the evaluation of the proposed method are Omniglot and CIFAR-FS. I think that Omniglot is starting to be outdated and has been replaced by more sophisticated benchmarks in recent publications (e.g. Meta-Dataset [5], SlimImageNet [2]). CIFAR-FS has also some serious limitations (limited diversity, small resolution, etc) when compared to recent benchmarks. The authors are encouraged to move towards more recent benchmarks if they want to fully showcase the potential of the proposed solution.\n\n\nReferences\n----------\n\n[1] Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016). One-shot learning with memory-augmented neural networks. arXiv preprint arXiv:1605.06065.\n\n[2] Antoniou, A., Patacchiola, M., Ochal, M., & Storkey, A. (2020). Defining benchmarks for continual few-shot learning. arXiv preprint arXiv:2004.11967.\n\n[3] Caccia, M., Rodriguez, P., Ostapenko, O., Normandin, F., Lin, M., Caccia, L., ... & Charlin, L. (2020). Online fast adaptation and knowledge accumulation: a new approach to continual learning. arXiv preprint arXiv:2003.05856.\n\n[4] Ren, M., Iuzzolino, M. L., Mozer, M. C., & Zemel, R. S. (2020). Wandering within a world: Online contextualized few-shot learning. arXiv preprint arXiv:2007.04546.\n\n[5] Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., ... & Larochelle, H. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Hfeikn5YK5C", "reviewer": "Reviewer_s4fJ", "summary": "In this paper, the authors propose a novel network architecture for online few-shot learning. They propose to use an LTSM framework and store the previous information in the hidden states of different layers. They also evaluate their methods on other tasks, such as online continual learning, online few-shot semantic segmentation, and standard supervised learning. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- ***This paper is poorly organized and very hard to read.*** I have been working on few-shot learning for several years and have read lots of papers in this field. However, after reading the abstract and introduction, I totally don’t understand what they want to do. For example, many concepts occur before they are defined. In Line 2, the authors mention “the persistent state”. What is it? It is not defined. In Line 134, the authors also use “h”. I cannot find the definition for h before this paragraph. Is that a value, a vector, or a matrix? \n\n- ***The motivation for using memory-based methods is unclear.*** I think the authors try to motivate their method in Lines 30-36. However, I have lots of questions about this motivation, e.g., (1) what are the definitions of generality and flexibility in few-shot learning, and (2) why using a straightforward loss formulation and standard optimization techniques is better? After reading the introduction, I still don’t know what is the main challenge in this paper and how the authors address it.\n\n- ***The claim “their method outperforms gradient-based, prototype-based, and other memory-based meta-learning strategies” seems incorrect.*** Most of the meta-learning methods, such as MAML and ProtoNets, evaluate the methods in the standard few-shot classification setting. However, the authors evaluate their methods in the online few-shot learning setting. Besides, the comparing methods (LSTM, NTM, APL, OPN, and CPM) don’t contain a gradient-based method. \n\n- ***The experiments are conducted on very small datasets.*** The authors use CIFAR-FS and Omniglot, which are two small datasets. In standard few-shot classification, we usually use miniImageNet and tieredImageNet. In [18], they use RoamingImageNet, which is based on tieredImageNet. So it is not reasonable to run experiments on such small datasets. \n\n- ***The method section is too simple.*** The authors only use one page for the method section. Many details about the implementation are unclear. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have included the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4L_xu0a1WC", "reviewer": "Reviewer_9VUm", "summary": "The authors approach the solution to the problem of meta-learning as simply running a recurrent network through various tasks and back-propagating to train the RNN to adapt to new tasks (as in some previous works). The main advance of this work is proposing an architecture - a Deep Convolutional LSTM (with few architectural details) and showing that it works very well. The simplicity of this advance is a plus. The authors should change initial writing and clarify what they mean by distributed memory (see below in more detail) but otherwise the paper is easy to read. A good number of experiments is conducted, though it would be good if it was tested in harder tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are the basic cons of this work: The first one should be easy to fix: The description from the start with very mysterious - talking about distributed memory, makes a reader wonder what kind of new paradigm have they devised, just to later find out it is just a different RNN architecture. Taking about this as “memory” is misleading because while the RNN activations do store information, often times one considers weights of RNN to be the memory, and learning being the adaptation of the weights - which is not what they are meta-learning (the weights update has a fixed algorithm - back-propagation). The authors should make it clear from the beginning what they mean by memory (=activations of an RNN) and what they mean by distributed (=across layers in deep RNN). Once the reader knows that, the paper is well written and easy to read.\n\nThe second issue is that the tasks they use are have quite a short time spans - I am reluctant to call this meta-learning. Nevertheless, these types of problems have been studied before, and this paper makes a nice contribution with a number of experiments. However it is not clear how well would this learning to adapt the activations work in problems with much larger time scales.\n\nDetails: \n\n- I would say in the abstract what the basic idea is: e.g. something like “We device a variant of deep convolutional LSTM architecture, the feature of which is a large capacity state of activations, and use standard end-to-end training by back-propagating through sequence of tasks, …”\n\nI would even ideally change a title - distributed memory can mean many things, like adapting weights of a networks and neither I feel like this should even be referred to as meta-learning (due to shot time scale nature of the problems).\n\n- This kind of architecture is indeed quite capable - very similar one was used in RL: “An Investigation of model-free planning”, Guez at al., - you could add the reference. \n\n- 134-142 - May be add here some of you previous relevant references of works that do this (train RNN end-to-end) - end of the second last sentence.\n\n- 202: You mention pre-training but what it is is only mentioned in the sentence after the next one.\n\n- 243: In 240 you said that task has 5 samples, but here you are changing to 5-20 - does it have more examples or the same 5 are repeated more. If the former, shouldn’t the performance of the model, Figure 3, theoretically increase since you see more examples of a class?\n\n- 247: Meaning that every query example is started with the same state? (That after training set of tasks)\n\n- Figure 4: I don’t understand what is on the x axis. I thought we have a sequence of tasks, and then test the model on the query set containing samples from all the subtasks.\n\n- Section 4.5: How do you do the training here? Do you sample examples from training set at random and treat them as a long sequence, back-propagating every n steps, or how is this done? What is the architecture of CL-ResNet-20 say? Resnet followed by convLSTM followed by LSTM or something else?\n\nI would be happy to raise my score if the issues are addressed and there are no major other issues I have missed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper operates in a simple approach to meta-learning, of running a recurrent network through tasks and back-propagating to learn the adaptation. They find that simple deep-convolutional lstm networks (with details) outperforms many of the previous architectures studies before on standard benchmarks and interestingly also works in classic supervised setting at state of the art level (as a single system). 

On the positive side, the method is simple and the experiments are done carefully. The major drawback is that it is not being tested on harder problems and more modern online learning settings. Pushing the system this would we provide a major improvement to the paper. Despite this, the currently results are still sufficiently interesting. The are few points in terms of writing the paper that should be improved and I hope the authors will address that for the final version. 
Decision: Accept (Poster)