Wandering within a world: Online contextualized few-shot learning
Authors: Mengye Ren
Michael Louis Iuzzolino
Michael Curtis Mozer
Richard Zemel
Venue: ICLR 2021 Poster (2021)
Keywords: Few-shot learning
continual learning
lifelong learning

Abstract: We aim to bridge the gap between typical human and machine-learning environments by extending the standard framework of few-shot learning to an online, continual setting. In this setting, episodes do not have separate training and testing phases, and instead models are evaluated online while learning novel classes. As in the real world, where the presence of spatiotemporal context helps us retrieve learned skills in the past, our online few-shot learning setting also features an underlying context that changes throughout time. Object classes are correlated within a context and inferring the correct context can lead to better performance. Building upon this setting, we propose a new few-shot learning dataset based on large scale indoor imagery that mimics the visual
experience of an agent wandering within a world. Furthermore, we convert popular few-shot learning approaches into online versions and we also propose a new model that can make use of spatiotemporal contextual information from the recent past.

Published as a conference paper at ICLR 2021
WANDERING WITHIN A WORLD: ONLINE CONTEXTUALIZED FEW-SHOT LEARNING
Mengye Ren 1.3 Michael L. Iuzzolino"
Michael C. Mozer
Richard S. Zemel 1,3,5 1.355
University of Toronto Google Research University of Colorado, Boulder "CIFAR mmren, zemel ) @ss. toronto... edu (mi micchel wuwzoliiiii ino, mozer @colorado.edu
Vector Institute
ABSTRACT
We aim to bridge the gap between typical human and machine-learning environ- ments by extending the standard framework few-shot learning to an online, continual setting. In this setting, episodes do not have separate training and testing phases, and instead models are evaluated online while learning novel classes. As in the real world, where the presence of spatiotemporal context helps retrieve learned skills in the past, our online few-shot learning setting also features an underlying context that changes throughout time. Object classes correlated within a context and inferring the correct context can lead to better performance. Building upon this setting, propose a new few-shot learning dataset based on large scale indoor imagery that mimics the visual experience of an agent wandering within world. Furthermore, we convert popular few-shot learning approaches into online versions and we also propose a new contextual prototypical memory model that can make use of spatiotemporal contextual information from the recent past.
INTRODUCTION
In machine learning, many paradigms exist for training and evaluating models: standard train-then- evaluate, few-shot learning, incremental learning, continual learning, and so forth. None of these paradigms well approximates the naturalistic conditions that humans and artificial agents encounter as they wander within a physical environment. Consider, for example, learning and remembering peoples' names in the course of daily life. We tend see people a given environment- gym, etc. We tend to repeatedly revisit those environments, with different environment base rates, nonuniform environment transition probabilities, and nonuniform base rates of encountering a given person in given environment. We need to recognize when we do not know person, and need to learn to recognize them the next time we encounter them. We are not always provided with name, but we can learn in semi-supervised manner. And every training trial is itself evaluation trial as we repeatedly use existing knowledge and acquire new knowledge. In this article, we propose a novel paradigm, online contextualized few-shot learning, that approximates these naturalistic conditions, and we develop deep-learning architectures well suited for this paradigm.
home,
In traditional few-shot learning (FSL.) (Lake et al., 2015 Vinyals et al., 2016), training is episodic. Within an isolated episode, a set of new classes is introduced with a limited number of labeled examples per class-the support set-followed by evaluation on an unlabeled query set. While this setup has inspired the development of a multitude of meta-learning algorithms which can be trained to rapidly learn novel classes with a few labeled examples, the algorithms are focused solely on the few classes introduced in the current episode; the classes learned are not carried over to future episodes. Although incremental learning and continual learning methods (Rebuffi et al. 2017 Hou et al. 2019) address the case where classes are carried over, the episodic construction of these frameworks seems artificial: in our daily lives, we not learn new objects by grouping them with five other new objects, process them together, and then move on.
Our code and dataset relcased at: https: / httpg/hhh...../eeeeeeeenneeeeeee htt.gtttt...o.eeeeeeeeeee/eeeaae

---

Published as a conference paper at ICLR 2021
Online evaluation with old and new classes
Context switching
Evelustion
Kiihhen
Env Cifice
New
Neew
New
Cls?
Uving
World
40A
2
Lne 3: Office
Life 1: Kitchen
Figure I: Online contextualized few-shot learning. A) Our setup is similar to online learning, where there is separate testing phase; model training and evaluation happen at the same time. The input at each time step is an (image, class-label) pair. The number of classes grows incrementally and the agent is expected to answer "new" for items that have not yet been assigned labels. Sequences can be semi-supervised; here the label is not revealed for every input item (labeled/unlabele shown by red solid/grey dotted boxes). The agent evaluated on the correctness of all answers. The model obtains learning signals only on labeled instances, and is correct if it predicts the label of previously-see classes, or 'new" for new ones. B) The overall sequence switches between different learning environments. While the environment ID is hidden from the agent, inferring the current environment can help solve the task.
To break the rigid, artificial structure of continual and few-shot learning, we propose a new continual few-shot learning setting where environments are revisited and the total number of novel object classes increases over time. Crucially, model evaluation happens on each trial, very much like the setup in online learning. When encountering a new class, the learning algorithm is expected to indicate that the class is "new," and it is then expected to recognize subsequent instances of the class once a label has been provided.
When learning continually in such a dynamic environment, contextual information can guide learning and remembering. Any structured sequence provides temporal context: the instances encountered recently predictive of instances to be encountered next. In natural environments, spatial context- information in the current input weakly correlated with the occurrence of a particular class-car can be beneficial for retrieval as well. For example, we tend to see boss in an office setting, not in bedroom setting. Human memory retrieval benefits from both spatial and temporal context (Howard) 2017 Kahana| 2012). In our online few-shot learning setting, presentation of each instance and temporal structure to sequences, enabling an agent to learn from both spatial and temporal context. Besides developing and experimenting on a toy benchmark using handwritten characters (Lake et al., 2015), we also propose a new large-scale benchmark for online contextualized few-shot learning derived from indoor panoramic imagery (Chang et al., 2017). In the toy benchmark, temporal context can be defined by the co-occurrence of character classes. In the indoor environment, the context-tempee and spatial- is a natural by-product as the agent wandering in between different rooms.
provide spatial context in the
We propose a model that can exploit contextual information, called contextual prototypical memory (CPM), which incorporates RNN to encode contextual information and a separate prototype memory to remember previously learned classes (see Figure 4 This model obtains significant gains on few-shot classification performance compared to models that do not retain memory of the recent past. We compare to classic few-shot algorithms extended to an online setting, and CPM consistently achieves the best performance.
The main contributions of this paper are as follows. First, we define an online contextualized few-shot learning (OC-FSL) setting mimic naturalistic human learning. Second, we build three datasets: 1) Roaming RommingOmniglot is based on handwritten characters from Omniglot (Lake et al.) 2015): 2) DoaminglmageNNe is based on images from ImageNet (Russakovsky et al. 20155) ; and 3) Roam- ingRooms is our new few-shot learning dataset based on indoor imagery (Chang et al. 2017). which resembles the visual experience a wandering agent. Third, we benchmark classic FSL methods and also explore our CPM model, which combines the strengths of RNNs for modeling temporal context and Prototypical Networks (Snell et al. 2017) for memory consolidation and rapid learning.
RELATED WORK
In this section, we briefly review paradigms that have been used for few-shot learning (FSL) and continual learning (CL).
2

---

Published as a conference paper at ICLR 2021
A) RoamingOmniglot
B) RoamingRooms
Cls
Chair
Taole
457
bank
Applience 102
App 434
Applicce 402
Appliance 402
Figure 2: Sample online contextualized few-shot learning sequences. A) RoamingOmniglor, Red solid boxes denote labeled examples of Omniglot handwritten characters, and dotted boxes denote unlabeled ones. Environments are shown in colored labels in the top left corner. B) Image frame samples of a few-shot learning sequence in our RoamingRooms dataset collected from a random walking agent. The task here to recognize and classify novel instance IDs in the home environment. Here the groundtruth instance masks/bounding boxes are provided. C) The growth of total number of labeled classes in a sequence for RoamingOmniglot (top) and RoamingRooms (bottom).
Few-shot learning: FSL Lake et al., 2015 Li et al. 2007 Koch et al. 2015 Vinyals et al., 2016 considers learning new tasks with few labeled examples. FSL models can be categorized as based
on: metric learning (Vinyals et al. 2016; Snell et al. 2017), memory (Santoro et al., 2016), and gradient adaptation (Finn et al., 201 7, Li al. 2017). The model we propose, CPM, lies on the boundary between these approaches, as we use an RNN to model the temporal context but we also use metric-learning mechanisms and objectives to train.
Several previous efforts have aimed to extend few-shot learning to incorporate more natural constraints. One such example is semi-supervised FSL (Ren et al., 2018), where models also learn from a pool of unlabeled examples. While traditional FSL only tests the learner novel classes, incremental FSL GGiarrs & Komodakis 2018 Ren et al.] 2019) tests on novel classes together with a set of base classes. These approaches, however, have not explored how iteratively add new classes.
In concurrent work, Antoniou et al. (2200) extend FSL to a continual setting based on image sequences, each of which is divided into stages with fixed number of examples per class followed by a query set. It focuses on more flexible and faster adaptation since the models are evaluated online, and the context is soft constraint instead of a hard separation of tasks. Moreover, new classes need to be identified as part of the sequence, crucial to any learner's incremental acquisition of knowledge.
Continual learning: Continual (or lifelong) learning is a parallel line of research that aims to handle a sequence of dynamic tasks (Kirkpatrick et al., 2017: Li & Hoiem 2018 Lopez-Paz. & Ranzato 2017: Yoon et al.] 2018). A key challenge here is catastrophic forgetting (McCloskey & Cohen 1989), where the model "forgets" task that has been learned in the past. Incremental
learning (Rebuffi et al. 2017, Castro et al., 2018 Wu et al. [2019; Hou et al.) 20199] He et al.. 2020 is form of continual learning, where each task is an incremental batch of several new classes. This assumption that novel classes always come in batches seems unnatural.
Traditionally, continual learning is studied with tasks such as permuted MNIST (Lecun et al. 1998) or Split-CIFAR (Krihevsky) (209)). Recent datasets aim to consider more realistic continual learning, such as CORe50 (Lomonaco & Maltoni 2017) and OpenLORIS (She et al.) 2019). We summarize core features of these continual learning datasets in Appendix A First, both CORe50 and OpenL.ORIS have relatively few object classes, which makes meta-learning approaches inapplicable; second, both contain images of small objects with minimal occlusion and viewpoint changes; and third, OpenLORIS does not have the desired incremental class learning.
In concurrent work, Caccia et al. (2020) proposes setup to unify continual learning and meta-learning with a similar online evaluation procedure. However, there are several notable differences. First, their models focus on general loss function without specific design for predicting new classes; they predict new tasks by examining if the loss of exceeds some threshold. Second, the sequences of inputs are fully supervised. Lastly, their benchmarks based on synthetic task sequences such as Omniglot or Imagedet,,,Ne,, which are less naturalistic than our RoamingRooms dataset.
Online meta-learning:
Some existing work builds on early approaches Thrun 1998 Schmidhuber
1987) that tackle continual learning from meta-learning perspective. Finn et al. (2119) proposes storing all task data in a data buffer; by contrast, Javed & White (2019) proposes to instead learn a good representation that supports such online updates. In Jerfel et al. (2019). ) hierarchical Bayesian mixture model is used to address the dynamic nature of continual learning.
3

---

Published as a conference paper at ICLR 2021
A) Instance Category Distribution
B) Frames Between Occurrences
Unique Viewpoints Frames
2500-
10"
1500-
101
1000
10
7
Figure 3: Statistics for our RoamingRooms dataset. Plots show a natural long tail distribution of instances grouped into categories. An average sequence has 3 different view points. Sequences are highly correlated time but revisits are not uncommon.
Connections to the human brain: Our CPM model consists of multiple memory systems, con- sistent with claims of cognitive neuroscientists of multiple memory systems in the brain. The complementary learning systems (CLS) theory (McClelland et al., 1995) suggests that the hippocam- pus stores the recent experience and is likely where few-shot learning takes place. However, our model more closely related to contextual binding theory (Yonelinas et al. 2019), which suggests that long-term encoding of information depends on binding spatiotemporal context, and without this context as a cue, forgetting occurs. Our proposed CPM contains parallels to human brain memory components (Cohen & Squire| 1980). Long term statistical learning is captured in CNN that produces deep embedding. An RNN holds a type of working memory that can retain novel objects and spatiotemporal contexts. Lastly, the prototype memory represents the semantic memory, which consolidates multiple events into a single knowledge vector (Duff' et al. (020). Other deep learning researchers have proposed multiple memory systems for continual learning. In Parisi et al. (2018). the learning algorithm is heuristic and representations come from pretrained networks. In Kemker Kanan (0018). prototype memory is used for recalling recent examples and rehearsal from generative model allows this knowledge to be integrated and distilled into a long-term memory.
3 ONLINE CONTEXTUALIZED FEW-SHOT LEARNING
In this section, we introduce our new online contextualized few-shot learning (OC-FSI.) setup in the form of a sequential decision problem, and then introduce our new benchmark datasets.
Continual few-shot classification as sequential decision problem: In traditional few-shot learn- ing, an episode is constructed by support set S and a query set Q. A few-shot learner f is expected to predict the class of each example in the query set x² based on the support set information: ge = f(x2; (x², .v¹).. )................. fit for continual learning, since it unclear when to insert query set into the sequence.
..... (x yN)), where x RD, and y E [1 K]. This setup is not a natural
Inspired by the online learning literature, we can convert continual few-shot learning into a se- quential decision problem. where every input example also part of the evaluation: de 1x it),)), for t = 1 1... T, where ÿ here further allows that the sequence of inputs to be sumi-supervised: ÿ equals 3/s if labeled, or otherwise 1. The setup in Santoro et al. (2016) and Kaiser et al. (2017) is similar; they train RNNs using such temporal sequence input. However, their evaluation relies on a "query set" at the end. We instead evaluate online while learning.
f(x;;
Figure A illustrates these features, using example of an input sequence where agent is learning about new objects in a kitchen. The model is rewarded when it correctly predicts a known class or when it indicates that the item has yet to be assigned label.
Contextualized environments: Typical continual learning consists of a sequence of tasks, and models are trained sequentially for each task. This feature is also preserved in many incremental learning settings (Rebuffi et al. (017). For instance, the split-CIFAR task divides CIFAR-100 into 10 learning environments, each with 10 classes, presented sequentially. In our formulation, the sequence returns to earlier environments (see Figure B). which enables assessment of long-term durability of knowledge. Although the ground-truie environment identity is not provided, we structure the task such that the environment itself provides contextual cues which can constrain the correct class label. Spatial cues in the input help distinguish one environment from another. Temporal cues implicit because the sequence tends to switch environments infrequently, allowing recent inputs to be beneficial in guiding the interpretation of the current input.
The Omniglot dataset (Lake et al. 2015) contains 1623 handwritten characters from 50 different alphabets. We split the alphabets into 31 for training, 5 for validation, and 13 for testing. We augment classes by 90 degree rotations to create 6492 classes in total. Each contextualized sho- learning image sequence contains 150 images, drawn from random sample
RoamingOmnigloo
4

---

Published as a conference paper at ICLR 2021
Contextual prototypical memory. Temporal contextual features are extracted from an RNN. The prototype memory stores vector per class and does online averaging. Examples falling outside the radii of all prototypes classified "new." 2
Figure 4:
CIN
Protstope Menury
of 5-10 alphabets, for total of 50 classes per sequence. These classes are randomly assigned to 5 different environments; within an environment, the characters are distributed according to a Chinese restaurant process (Aldous) 1985) to mimic the imbalanced long-tail distribution of naturally occurring objects. We switch between environments using a Markov switching process; i.e., at each step there is a constant probability of switching to another environment. An example sequence is shown in Figure 2 A.
RoamingRooms: As none of the current few-shot learning datasets provides the natural online learning experience that we would like to study, we created our own dataset using simulated indoor environments. We formulate this as a few-shot instance learning problem, which could be use case for a home robot: needs to quickly recognize and differentiate novel object instances, and large viewpoint variations can make this task challenging (see examples in Figure B). There are over 7,000 unique instance classes in the dataset, making it suitable to meta-learning approaches.
Our dataset is derived from the Matterport3D dataset (Chang al. 2017) with 90 indoor worlds captured using pamoramic depth cameras. We split these into 60 worlds for training. 10 for validation and 20 for testing. MatterSim (Anderson al. 2018) is used to load the simulated world and collect camera images and HabitatSim (Savva et al., 2019) is used to project instance segmentation labels onto 2D image space. We created random walking agent collect the virtual visual experience. For each viewpoint in the random walk, we randomly sample one object from the image sensor and highlight with the available instance segmentation, forming an input frame. Each viewpoint provides environmental context- the other objects present in the room with the highlighted object.
Figure 3 shows the object instance distribution. We see strong temporal correlation, as 30% of the time the same instance appears in the next frame (Figure 3 B), but there is also a significant proportion of revisits. On average, there are three different viewpoints per 100-image sequence (Figure C).. Details and other statistics of our proposed datasets are included in the Appendix.
To make the classification problem more challenging, we also report results on RoaminglmageNet, which uses the same online sequence sampler as RoamingOmniglot but applied on the fiered-ImageNee dataset (Ren et al., 2018)) a subset of the original ImageNet dataset (Russakovsky et al. 2015). It contains 608 classes with a total of 779K images of size 84x84. We use the same split of classes as |Rnn et al.] (2018). The high-level categories are used for sampling different "environments" just like the notion of alphabets in RoamingOmniglot.
RoamingImageNet:
4 CONTEXTUAL PROTOTYPICAL MEMORY NETWORKS
In the online contextualized few-shot learning setup, the few-shot learner can potentially improve by modeling the temporal context. Metric learning approaches (Snell et al. 2017) typically ignore temporal contextual relations and directly compare the similarity between training and test samples. sradient-based approaches (Javed & White 2019). on the other hand, have the ability to adapt to new contexts, but they do not naturally handle new and unlabeled examples. Inspired by the contextual binding theory of human memory (Yonelinas et al. 2019), we propose simple yet effective approach that uses an RNN to transmit spatiotemporal context and control signals to a prototype memory (Figure
Prototype memory: We start describing our model with the prototype memory, which is an online version of the Prototypical Network (or ProtoNer) (Snell et 2017). ProtoNet can be viewed as a knowledge base memory, where each object class le is represented by a prototype vector plk:) computed as the mean vector of all the support instances of the class in sequence. It can also be applied to our task of online few-shot learning naturally, with some modifications. Suppose that at time-step i we have already stored few classes in the memory, each represented by their current prototype Pa [kt, and we would like to query the memory using the input feature h4, We model our prototype memory Uı, softmax( -dn, (h, Ps [k]).. (c.g. squared Euclidean distance or cosine dissimilarity) parameterized by a vector m, that seales each dimension with a Hadamard product. To predict whether example is of a new class, we use separate novelty output ân
5

---

Published as a conference paper at ICLR 2021
Table 1: RoamingOmniglot OC-FSL results. Max 5 env, 150 images, 50 cls, with 8x8 occlusion.
Supervised 1-shot Acc.
Semi-supervised
Method
AP
3-shot Acc.
AP
1-shot Acc.
3-shot Acc.
LSTM
64,34
61.00 + 0.22
81.85
0.21
54.34
68.30 = 0.20
76,38 0.49
DNC
81.30
78.87 + 0.19
91.01
+ 0.15
81.37
88.56
+ 0.12
93.81
+ 0.26
OML-U
77.38
70.98 +: 0.21
89.13
66.70
74.65 = 0.19
90.81
0,34
OML-UU++
86.85
88.43
+: 0.14
92.01
81.39
71.64
0.19
93.72 + 0.27
O-MN
88.69
84.82
1: 0.15
95.55
0.111
84.39
88.77
: 0.13
97.28
1: 0.17
90.15
0.15
0.09
81.62
97.09
0.19
O-PN
90.49
85.68
+: 0.15
96.95 4: 0.09
84,61
88.71
= 0.13
97.61
0.17
CPM (Ours)
94.17
91.99
1: 0.11
97.74
0.08
90.42
93.18
0.16
97.89 0.15
Table 2: RoamingRooms OC-FSL results. Max 100 images and 40 classes.
Supervised 1-shot Ace.
Semi-supervised 1-shot Acc.
Method
AP
3-shot Acc.
AP
3-shot Acc.
LSTM
45.67
59.90
0.40
61.85
0.45
33.32
52.71
0.38
55.83 + 0.76
DNC
80.86
82.15
0.32
87.30
0.30
73.49
80.27
0.33
87.87 + 0.49
OML-U
76.27
73.91
+ 0.37
83.99
0.33
63.40
0.38 70.67
85.25 + 0.56
OML-UE- U++
88.03
0.27
89.61
0.29
81.90
84.79
89.80 1: 0.47
O-MN
85.91
82.82
2 0.32
89.99
0.26
78.99
80.08 + 0.34
92.43 + 0.41
0-IMP
87,33
85.28
0.31
90.83
0.25
75.36
84.57 0.31
91.17 + 0.43
O-PN
86.01
84.89
0.31
89.58
0.28
76.36
80.67
0.34
88.83 + 0.49
CPM (Ours)
89.14
88.39
0.27
91.31 0.26
84.12
86.17
+ 0.30
91.16 + 0.44
with sigmoid activation, similar the approach introduced in Ren al. (2018). where Sy. and 7f are vet-to-be-specifier thresholding hyperparameters (the superscript r stands for read):
a o ((min dm, (h,, Pt [k] 8/)/7().
Memory consolidation with online prototype averaging: Traditionally, ProtoNet uses the aver- age representation of a class across all support examples. Here, we must be able adapt the prototype memory incrementally at each step. Fortunately, we can recover the computation of a ProtoNet by performing simple online averaging:
1 (p--1ce1++h) Ct
C C--1 +1; A(ba: A(h::P--1,c.)
(2)
where is the input feature, and p is the prototype, and c is a scalar indicating the number of examples that have been added to this prototype up to time t. The online averaging function A can also be made more flexible to allow more plasiticity, modeled by a gated averaging unit (GAU):
(3) When the current example is unlabeled, Ue is encoded as 1, and the model's own prediction Åt will determine which prototype to update; in this case, the model must also determine a strength of belief, 4t" that the current unlabeled example should be treated new class. Given ú, and dt. the model can then update a prototype:
AGAU (h,: Pt-1) = (1 fi) P fo h where f. o (W,[h, Pc-1, + by) E E.
atfh σ ((min dmn (h,, Pelk)) //2")
(4)
A[k], 1 Lie = + ve, 0x 1ūe -11.
(5)
Supervised
Unsupervised
clk], c(k),- + Alkle
(6)
p|k]] A(h, pl c(()). or pík] = AGAU (h, p()))-1).
(7)
As-yet-unspeciffedd hyperparameters 3 and 7t" are required (the superscript w for write). These parameters for the online-updating novelty output næ" are distinct from 8 and oo in Equation D The intuition is that for "self-teaching" to work, the model potentially needs to be more conservative in cre- ating new classes (avoiding corruption of prototypes) than in predicting an input as being new class.
Contextual RNN: Instead of directly using the features from the CNN h hÇNN as input features to the prototype memory. we would also like to use contextual information from the recent past. Above we introduced threshold hyperparameters , 22' 19 7tt as well as the metric parameter M,. We let the contextual RNN output these additional control parameters, so that the unknown thresholds and metric function can adapt based on the information in the context. The RNN produces the context vector h and other control parameters conditioned on h CNN,
Iz,, h .m, 3r 7h's 8 7 RNN (WCCN, CNN, Zz-1);
(8)

---

Published as a conference paper at ICLR 2021
Table 3: RoaminglmageNe OC-FSL results. Max 150 images and 50 classes. denotes CNN
pretrained using regular classification.
Supervised 1-shot Ace.
Semi-supervised
Method
AP
3-shot Acc.
Acc.
3-shat Acc.
LSTM
22.54
28.14
0.20
$2.07
0.27
13.50
30.02 0.20
46.95 +: 0.56
DNC*
26.80
33.45
+ 0.19
55.78
0.27
16.50
39.53 0.19
54,10
+ 0.54
OML-U
21.89
15.06
::06:000 0.14
52.52
= 0.27
10.16
22.74 0.17
55.81 + 0.55
O-MN
13.05
20.61
0.15
38.73
0.24
9.32
25.96 0.16
55.32 0.51
0-IMP
14.25
22.92
41.01
0.25
4,55
20.70
51.23 0.53
O-PN*
23.10
32.82 = 0.19
49.98
00000.2 0.25
15.76
36.69 0.18
55.47 + 0.53
CPM (Ours)
34.43
40.40
0.21
60.29
0.26
24.75
44.58
+ 0.21
58.72 +: 0.53
where 2, is the recurrent state of the RNN, and m, is the scaling factor in the dissimilarity score. The context, hąNN, serves an additive bias on the state vector used for FSL: h, h,5 + h This addition operation in the feature space can help contextualize prototypes based on temporal proximity, and is also similar to how the human brain leverages spatiotemporal context for memory storage (Yonelinas et al. 2019).
Loss function:
The loss function is computed after an entire sequence ends and all network parameters are learned end-to-end. The loss is composed of two parts. The first is binary cross- entropy (BCE), for telling whether each example has been assigned a label or not, i.c., prediction of new classes, and uy is the ground-truth binary label. Second we use multi-class cross-entropy for classifying among the known ones. We can write down the overall loss function as follows:
T
Σ T
u u log((f) - (1 - ue) log a(5
Σ
13v. = k] - ue) log(Or.)).
(9)
Binary cross entropy old vs. new
Cross entropy old classes
Training and evaluation: During training, we sample learning sequences and for each sequence, we perform one iterative update to minimize the loss function (Eq. (9). At the beginning of each sequence, the memory reset. During training, the model learns from set of training classes. During test time, the model recognizes new classes that have never been seen during training.
EXPERIMENTS
In this section, we show experimental results for our online contextualized few-shot learning paradigm, using RoamingOmniglo and RoamingRooms (see Sec. (3 to evaluate our model, CPM, and other -ffte-of-the-art methods. For Omniglot, we apply an 8x8 CutOut (Devries & Taylor 2017) to each image to make the task more challenging.
Implementation details: For RoamingOmniglor, we use the common 4-layer CNN for few-shot learning with 64 channels in each layer. For RoaminglmageNet, we also use ResNet- 12 with input resolution 84 84 (Oreshkin et al., 22188). For the RoamingRooms, we resize the input to 120x 160 and use ResNet-12. 12. To represent the feature of the input image with an attention mask, we concatenate the global average pooled feature with the attention ROI feature, resulting in 512d feature vector. For the contextual RNN, in both experiments we used an LSTM (Hochreiter Schmidhuber) [997) with 256d hidden state. The best CPM model is equipped using GAU and cosine similarity for querying prototypes. Logits based on cosine similarity are multiplied with a learned scalar initialized at 10.0 Orrehkkin et al. 2018). We include additional training details in Appendix B
In order to compute single number that characterizes the learning ability over sequences, we propose to use average precision (AP) to evaluate both with respect old versus new and the specific class predictions. Concretely, all predictions are sorted by their old vs. new scores, and we compute AP using the area under the precision-recat curve. A true positive is defined as the correct prediction of multi-class classification among known classes. We also compute the "N-shot" accuracy; i.c., the average accuracy after seeing the label N times in the sequence. Note that these accuracy scores only reflect the performance are reported with an average over 2,000 sequences and for N-shot accuracy standard error is also included. Further explanation of these metrics in Appendix A.3
Evaluation metrics:
known class predictions. All numbers
Comparisons: To evaluate the merits of our proposed model, implement classic few-shot learning and online meta-learnin methods. More implementation and training details of these baseline methods can be found in Appendix B
7

---

Published as a conference paper at ICLR 2021
Booriyy oer Semk-Sup
sooringkeem Supt
Ream roro Ceer S
95
05
10mm
75
128
Figure 5: Few-shot classification accuracy over time. Left: RoamingOmniglot. Right: Roamin- gRooms. Top:
Supervised. Bottom: Semi-supervised. An offline logistic regression (Offline LR) baseline is also included, using pretrained ProtoNet features. It is trained on all labeled examples except for the one at the current time step.
Orlnn PM
CPN (Dars)
-50
-0
-Temrpora
+Spais!
-Both
Figure 6: Effect of spatiotemporal context. Spatiotemporal context are added separately and together in RoaningOnniglot, by introducing texture background and temporal correlation. Left: Stimuli used for spatial cue of the background environment. Right: Our CPM model benefits from the presence of temporal context ("+Temporal" and "+Both")
OML (Javed & White 2019) : This is an online version of MAML (Finn et al. 2017). It performs one gradient descent step for each labeled input image, and slow weights are learned via back- propagation through time. On top of OML, we added an unknown predictor û; - max; Dr, (OML-U), We also found that using cosine classifier without the last layer ReLU is usually better than using the original dot-product classifier, and this improvement is denoted as OML-U++
LSTM (Hochreiter Schmidhuber 1997) & DNC (Graves et al. 2016): We include RNN methods for comparison as well. Differentiable neural computer (DNC) is improved version of memory augmented neural network (MANN) (Santoro et al.) 2016).
Online MatchingNet (O-MN) (Vinyals et al., 2016). IMP (0-IMP) (Allen et al. 2019) & Pro- toNet (O-PN) (Snell et al. 2:17): We used the same negative Euclidean distance as the similarity function for these three metric learning based approaches. In particular, MatchingNet stores all examples and performs nearest neighbor matching, which can be memory inefficient. Note that Online ProtoNet is variant of our method without the contextual RNN.
Main results: Our main results are shown in Table and including both supervised and semi-supervised settings. Our approach achieves the best performance on AP consistently across all settings. Online ProtoNet is a direct comparison without our contextual RNN and it clear that CPM is significantly better. Our method is slightly worse than Online Matching in terms of s-oho accuracy on the RoamingRoons semisupervised benchmark. This can be explained by the fact that MatchingNet stores all past seen examples, whereas CPM only stores one prototype per class. Per timestep accuracy is plotted in Figure 5 and the decaying accuracy is due to the increasing number of classes over time. In RoamingOmnigloo, CPM is able to closely match or even sometimes surpass the offline classifier, which re- trains at each step and uses all images in a sequence except the current model is able to leverage information from the current context.
one. This is reasonable
Effect of spatiotemporal context: To answer the question whether the gain in performance is due to spatiotemporal reasoning, we conduct the following experiment comparing CPM with online ProtoNet. We allow the CNN to have the ability to recognize the context in RoamingOmniglot by adding texture background image using the Kylberg texture dataset (Kylberg) (011)) (see Figure 6 left). As a control, we can also destroy the temporal context by shuffling all the images in a sequence. We train four different models on dataset controls with or without the presence of spatial or temporal context, and results are shown in Figure 6. First, both online ProtoNet and CPM benefit from the inclusion of a spatial context. This is understandable the CNN has the ability to learn spatial cues, which re- re-confirms our main hypothesis that successful inference of the current context is beneficial novel object recognition. Second, only our CPM model benefits from the presence of temporal context, and it receives distinct gains from spatial and temporal contexts.
We tried few other ways and this is found he the best.
8

---

Published as a conference paper at ICLR 2021
Table 4: Effect of forgetting over a time interval on RoamingOmniglot. Average accuracy vs. the number of time steps since the model has last seen the label of particular class.
Supervised
Semi- Sunnii-ueerised
Interval
1-2
3-5
5
6-
10
11-20 20
-0-50 50
51
100
3-5
6
10
11
20
21-50
51
100
OPN 1-Shat
88.8
86.9
85.2
84.7
83.6
81.1
90.1
88.9
88.4
87.6
87.3
85.1
CPM 1-Shot
96.1
94.0
93.0
91.6
88.2
84.6
95.9
93.8
92.8
91.8
89.4
85.7
OPN 3-Shot
97.2
97.1
96.6
96.7
96.5
95.3
97.8
97.3
97.1
97.8
97.7
96.8
CPM 3-Shol
98.5
98.2
97.5
97.2
95.4
95.5
98.7
97,5
97,5
96.5
96.3
92.9
Table Ablation of CPM architectural com- ponents on RoamingOmniglot
Table 6: Ablation of semi-supervised learning components on KoamingOOmiglle
Method
Method
RNN
Prototype
GAU
Val AP
No tox
O.PN
90.83
1, only
O-PN
89.10
metric mi
93.61
O-PN
/
91.22
No No 76
91.98
CPM
92:57
hit
93.70
CPM
V
93.16
CPM Ang. Euc
94.00
CPM
93.20
CPM GAU Fas
CPM Ang. Cos
/
94.11
94.57
CPM
/
94.08
CPM GAU Cus
94.65
CPM
94.65
Effect of forgetting: As the number of learned classes increases, we expect the average accuracy to drop. To further investigate this forgetting effect, we measure the average accuracy in terms of the number of time steps the model has last seen the label of a particular class. It is reported in Table and and in Appendix Tabll T3] 14, where we directly compare CPM and OPN to see the effect of temporal context. CPM is significantly better than OPN on 1-shot within short interval, which suggests that the contextual RNN makes the recall of the recent past much easier. On RoamingImageNer OPN eventually surpasses CPM on longer horizon, and this can be explained by the fact that OPN has more stable prototypes, whereas prototypes in CPM could potentially be affected by the fluctuation of the contextual RNN over longer horizon.
We ablate each individual module we introduce. Results are shown in Tables 6 Table different ways use the RNN, including the context vector hNNN the predicted threshold parameters 33, 72, and the predicted metric scaling vector Table 6 studies various ways to learn from unlabeled examples, where separately disable the RNN update, prototype update, and distinct write-thresher parameters : (vs. using cead-threshold parameters), which makes it robust to potential mistakes made in semi-supervised learning. We verify that each component has a positive impact on the performance.
Ablation studies:
CONCLUSION
We proposed online contextualized few-shot learning, OC-FSL. a paradigm for machine learning that emulates a human or artificial agent interacting with physical world. It combines multiple properties create a challenging learning task: every input must be classified or flagged novel, every input is also used for training, semi-sur sumi-supeeeseed learning can potentially improve performance, and the temporal distribution of inputs is non-IID and comes from a generative model in which input and class distributions are conditional on a latent environment with Markovian transition probabilities. We proposed the RoamingRooms dataset to simulate an agent wandering within a physical world. We also proposed a new model, CPM, which uses an RNN to extract spatiotemporal context from the input stream and to provide control settings to a prototype- naturalistic domains like RoamingRooms, CPM is able to leverage contextual information to attain performance unmatched by other state-of-the-arr FSL. methods.
FSL model. In the context of
Acknowledgments: We thank Fei Sha, James Lucas, Eleni Triantafillou and Tyler Scott for helpful feedback on an earlier draft of the manuscript. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute (www. vector ientoorinssiii tttte.tttee. a htttttaatterrreeeeeeeeeeeeeeeeee This project is supported by NSERC and the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/Interior Business Center (Dol/IBC) contract number D16PC00003. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies endorsements, either expressed or implied, of IARPA, DoI/IBC, or the U.S. Government.

---

Published as a conference paper at ICLR 2021
REFERENCES
David J Aldous. Exchangeability and related topics. In Ecole d' Été de Probabilités de Saint-Flowe XII1-1933, pp. 1-198. Springer, 1985.
Kelsey R. Allen, Evan Shelhamer, Hanul Shin, and Joshua B. Tenenbaum. Infinite mixture prototypes for few-shot learning. In Proceedings of the 36th International Conference on Machine Learning, ICML, 2019.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. and-Language Navigation: Interpreting visually- grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018.
Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, and Amos J. Storkey. Defining benchmarks for continual few-shot learning. CoRR, abs/2004. 1967, 2020.
Massimo Caccia, Pau Rodríguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam H. Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation: a new approach to continual learning. CoRR, abs/2003. 0.055,, 2020.
Francisco M. Castro, Manuel J. Marín- Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In /5th European Conference on Computer Vision, ECCV, 2018.
Angel X. Chang, Angela Dai, Thomas A. Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng. and Yinda Zhang. Matterport3d: Learning from RGB-D data in indoor environments. International Conference on 3D Vision, 3DV, 2017.
NJ Cohen and LR Squire. Preserved learning and retention of pattern-analyzing skill in amnesia: dissociation of knowing how and knowing that. Science, 210(4466):207-210, 1980. ISSN 0036-8075. doi: 10.1 235/sccience.741433..
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. CORR, abs/1 Ne/11708.04552. 2017.
Melissa C. Duff, Natalie V. Covington, Caitlin Hilverman, and Neal J. Cohen. Semantic memory and the hippocampus: Revisiting, reaffirming, and extending the reach of their critical relationship. Frontiers in Human Neuroscience, 13:471, 2020. ISSN 1662-5161. doi: 10.3389/fnhum.2011 00471.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Mgdel-agmostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning, ICMI., 2017.
Chelsea Finn, Aravind Rajeswaran, Sham M. Kakade, and Sergey Levine. Online meta-learning. In Proceedings of the 36th International Conference on Machine Learning, ICMI, 2019.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting, In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette. Tiago Ramalho, John Agapiou, Adrià Puigdomenech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using neural network with dynamic external memory. Nature. 538(7626): 471-476, Oct 2016. ISSN 1476-4687. doi: 10. 10.1038.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. CoRR, abs/19 1912.9918666 2019.
Jiangpeng He, Runyu Mao, Zeman Shao, and Fengqing Zhu. Incremental learning in online scenario. In IEEF/CV Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
10

---

Published as a conference paper at ICLR 2021
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8): 1735-1780, nov 1997. ISSN 0899-7667. doi: 191.:/ccc..999...8.7777.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
M. W. Howard. Temporal and spatial context in the mind and brain. Current opinion in behavioral sciences, 17:14-19, 2017.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, NeurIPS, 2019.
Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine A. Heller. Reconciling meta-learning and continual learning with online mixtures of tasks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, NewrlPS, 2019.
Michael Kahana. Foundations human memory. Oxford University Press, New York, 2012. ISBN 9780195333244. OCLC: ocn744297060.
Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. In 5th International Conference on Learning Representations, ICLR, 2017.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning, In Proceedings of 6th International Conference on Learning Representations (ICLR), 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International
Conference on Learning Representations, ICLR, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 11((13)3521-3352, 2017.
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In JCML deep learning workshop, volume 2, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Gustaf Kylberg. Kylberg Texture Dataset 1.0. Centre for Image Analysis, Swedish University of Agricultural Sciences an...., 2011.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266): 1332-1338, 2015.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278-23) Nov 1998. ISSN 0018-9219. doi: 10.1 79.17999..7
Fei-Fei Li, Robert Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput. Vis. Image Underst., 106(1):59-70, 2007.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. CoRR, abs/1 707.09337. 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting, IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2935-29477 2018.
Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In 1st Annual Conference Robot Learning, CoRI., 2017.
11

---

Published as a conference paper at ICLR 2021
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, NeurIPS, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (NNov): 2008.
James McClelland, Bruce McNaughton, and Randall O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102:419-57, 08 1995. doi:
10.1037/0033-295X. 290.333-9995.......44 102.3.419.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Marius Muja and David Lowe. Flann-fit fast library for approximate nearest neighbors user manual. Computer Science Department, University of British Columbia, Vancouver, BC, Canada, 2009.
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems, NeurIPS, 2018
German I. Parisi, Jun Tani, Cornelius Weber, and Stefan Wermter. Lifelong learning of spatiotemporal representations with dual-memory recurrent self-orgamization. Frontiers in neurorobotics, 12: 78-78, Nov 2018. ISSN 1662-5218. doi: 20099//fnboo.218.0007077.
Solvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In 6th International Conference on Learning Representations, ICLR, 2018
Mengye Ren, Renjie Liao, Ethan Fetaya, and Richard S. Zemel. Incremental few-shot learning with attention attractor networks. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems, NeurlPS, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211-25) 2015.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. Meta-leaming with memory augmente neural networks. In Proceedings of the 33nd International Conference Machine Learning, ICML, 2016.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2019.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: The mela-meta-... hook. Diplomarbeit, Technische Universität München, München, 1987.
Qi She, Fan Feng, Xinyue Hao. Qihan Yang, Chuanlin Lan, Vincenzo Lomonaco, Xuesong Shi, Zhengwei Wang, Yao Guo, Yimin Zhang, Fei Qiao, and Rosa H. M. Chan. Openloris-object A dataset and benchmark towards lifelong object recognition. CoRR, abs/191 101199110000 1.06487, 2019.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, NeurIPS, 2017.
12

---

Published as a conference paper at ICLR 2021
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. Few-shot class-incremental learning. CoRR, abs/2004.10956, 2020.
Sebastian Thrun. Lifelong learning algorithms. In Learning to learn, pp. 181-209. Springer, 1998.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Advances Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems, NeurlPS, 2016.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye. Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
Andrew Yonelinas, Charan Ranganath, Arne D. Ekstrom, and Brian J. Wiltgen. A contextual bind- ing theory of episodic memory: systems consolidation reconsidered. Nature reviews. Neuroscience, 2066):364-375, Jun 2019. ISSN 1471-0048. doi: 10.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In 6th International Conference on Learning Representations, ICLR, 2018.
13

---

Published as a conference paper at ICLR 2021
A DATASET DETAILS
A.1 BENCHMARK COMPARISON
We include Table g
to compare existing continual and few-shot learning paradigms.
A.2 ROAMINGOMNIGLOT & ROAMINGIMAGENET SAMPLER DETAILS
For RoamingOnniglot and the DoaminglmageNee experiments, we use sequences with maximum 150 images, from 5 environments. For individual environment, we use Chinese restaurant process to sample the class distribution. In particular, the probability of sampling a new class is:
ka + 0
Pnew
(10)
m7
where the number of classes that we have already sampled in the environment, and m is the total number of instances have in the environment. is set to 0.2 and 9 is set to 1.0 in all experiments.
The environment switching is implemented by a Markov switching process. At each step in the sequence there is constant probability Pswichh that switches to another environment. For all experi- ments, we set Pswitch to 0.2. We truncate the maximum number of appearances per class to 6. the maximum appearance is reached, we will sample another class.
A.3 METRICS
Average precision: We chose to use AP (average precision or area under the precision-recall curve) way of integrating two aspects of performance:
the binary accuracy whether an instance belongs to a known or unknown class (KU-Assign for short), and
2. the accuracy assigning an instance the correct class label given is from a known class (Class-Assign for short).
The procedure calculate AP as follows. We first sort all the KU-Assign, Class-Assign predictions across all sequences in descending order based KU-Assign probability, where the high ranked predictions should be known (not novel) classes. For the N top ranked instances in the sorted list, we compute:
1. precision @ N conrect @N/N
2. recall@N N = correctCClas-Assiinnnee K.
where K is the true number of known instances and correct(Class-Assiii @ssign) is the count of the number of correct class assignments among the top N. (The class assignment for an unknown instance is always incorrect.) To obtain the AP, compute the integral of the function (y=precision @ N, x=recall N) across all N's.
N-shot accuracy: We define N-shot accuracy as the number of times an instance that has been seen times thus far in the sequence is classified correctly. We compute the mean and standard error of this over all sequences.
A.4 ADDITIONAL ROAMINGROOME STATISTICS
Statisties of the RoamingRooms are included in Table comparison to other few-shot and continual learning datasets. Note that since RoamingRooms is collected from simulated environment, with 90 indoor worlds consisting of 1.2K panorama images and 1.22M video frames. The dataset contains about 6.9K random walk sequences with maximum of 200 frames per sequence. For training randomly crop 100 frames to form a training sequence. There are 7.0K unique instance classes.
Plots of additional statistics of RoamingRooms shown in Figure 7 In addition to the shown in the main paper, instances and viewpoints also follow long tail distributions. The number objects in each frame follows exponential distribution.
14

---

Published as a conference paper at ICLR 2021
Table 7: Continual & few-shot learning datasets
Images
Sequences
Classes
Content
Permuted MNIST Locun ct al.
1998
60K
Hand written digits
Omniglot (Lake al. (2015)
32.4K
1.6K
Hand written characters
CIFAR- 100 (Knzhevsky 2009)
50K
100
Common objects
mini-ImageNet ( Vinyals cl al. 2016)
50K
100
Common objects
diered-ImageNet (Ren et al.. 2018
779K
608
Common objects
OpenLORRIS (She al. 2019)
98K
69
Small table-top obj.
CORe50 (Lomonaco & Malton
2017)
164.8K
11
50
Hand-helle obj.
RoamingRooms (Ours)
1.22M
6.9K
7.0K
General indoor instances
A.5 ROAMINGROOMS SIMULATOR DETAILS
We generate our episodes with a two-stage process using two simulators HaabitatSi (Savva et al. 2019) and MatterSim (Anderson et al.) (0188) because HabitatSim is based on 3D meshes and using HabitatSim alone will result poor image quality due incorrect mesh reconstruction. Therefore we sacrificed the continuous movement of agents within HabitatSim and base our environment navigation on the discrete viewpoints MatterSim, which is based on real panoramic images. The horizontal field of view is set to 90 degrees for HabitatSim and 100 degrees for MatterSim, and simulate with 800 600 resolution.
The first stage of generation involves randomly picking sequence of viewpoints the connectivity graph within MatterSim. For each viewpoint, the agent scans the environment along the yaw and pitch axes for random period of time until navigable viewpoint is within view. The time spent in single viewpoint follows a Gaussian distribution with mean 5.0 and standard deviation 1.0. At the start of each new viewpoint, the agent randomly picks direction rotate and takes 12.5 degree steps along the yaw axis, and with 95% probability, a 5 degree rotation along the pitch axis is applied in randomly chosen direction. When a navigable viewpoint detected, the agent will navigate to the new viewpoint and reset the direction of scan. When multiple navigable viewpoints are present, the agent uniformly samples one.
In the second stage, agent in HabitatSim retraces the viewpoint path and movements of the first stage generated by MatterSim, collecting mesh-rendered RGB and instance segmentation sensor data. The MatterSim RGB and HabitatSim RGB images are then aligned via FLANN-based feature matching (Muja Lowe 2009), resulting in an alignment matrix that is used to place the MatterSim RGB and HabitatSim instance segmentation maps into aligmment. The sequence of these MatterSim RGB and HabitatSim instance segmentation maps constitute an episode.
We keep objects of the following categories: picture, chair, lighting, cushion, table, plant, chest of drawers, towel, sofa, bed, appliances, stool, tv monitor, clothes, toilet, fireplace, furniture, bathtub, gym equipment, blinds, board panel. We initially generate 600 frames per sequence and remove all the frames with object. Then store every 200 image frames into separate file.
During training and evaluation, each video sequence is loaded, and for each image we go through each object present in the image. We create the attention map using the segmentation groundtruth of the selected object. The attention map and the image together form a frame in our model input. For training, we randomly crop 100 frames from the sequence, and for evaluation we use the first 100 frames for deterministic results.
Please visit our released code repository to download the RoamingRooms dataset.
A.6 SEMI-SUPERVISED LABELS:
Here we describe how we sample the labeled vs. unlabeled flag for each example in the semi- supervised sequences in both RoamingOmniglot and RoamingRooms datasets. Due to the imbalance in our class distribution (from both the Chinese restaurant process and real data collection), directly masking the label may bias the model to ignore the rare seen classes. Ideally, we would like to preserve at least one labeled example for each class. Therefore, designed the following procedure.
15

---

Published as a conference paper at ICLR 2021
Table 8: Comparison of past FSL and CL paradigms vs. our online contextualized FSL (OC-FSL).
Few Shol
Semi-sup. Supp. Set
Online Eval.
Predict
Soft Context
Tasks
Continual
New
Switch
Incremental Learning (IL) (Rchuffi al. 2017
O
Few-shot (FSL) Vinyals al 20
Incremental FSL Ren ct al 2
Cls. Incremental FSI.
[ão et al. 2020
Semi-supv. FSL. Ren sl. 2018
MOCA (Harrison
Online Mixture Jerfel al.
2019
Online Meta Javed & White 2019
Continual FSL (Antomou
2020
OSAKA* Cuocia al.
2020
OC-FSL (Ours)
denotes concurrent work.
First, for each class k, suppose my is the number examples in the sequence that belong to the class. Let be the target label ratio. Then the class label ratio ag is:
(1 a) exp(-0.5(m) 1)) + a.
(11)
We then for each class k we sample a binary Bernoulli sequence based on Ber(ax). ). If a class has all zeros in the Bernoulli sequence, we flip the flag of one of the instances to I make sure there is least one labeled instance for each class. For all experiments, we set a = 0.3.
A.7 DATASET SPLITS
We include details about our dataset splits in Table 9 and 10
B EXPERIMENT DETAILS
B. NETWORK ARCHITECTURE
For the RoamingOmniglot experiment we used the common 4-layer CNN for few-shot learning with 64 channels in each layer, resulting in 64-d feature vector (Snell et al. 2017). For the RoamingRooms experiment we resize the input to 120x 160 and we use the Re architecture (Oreshkin al. 2018) with (32.64, 124,266) channels per block. To represent the feature of the input image with an attention mask, we concatenate the global average pooled feature with the attention ROI feature, resulting in a 512d feature vector. For the contextual RNN, in both experiments we used an LSTM (Hochreiter & Schmidhuber 1997) with a 256d hidden state.
We linear layer map from the output of the RNN the features and control variables. We obtain -= by adding 1.0 to the linear layer output and then applying the softplus activation. The bias units for gr. are initialized 10.0. We also apply the softplus activation from the linear layer output.
B.2 TRAINING PROCEDURE
We use the Adam optimizer (Kingma & Ba 2015) for all of our experiments, with gradient cap of 5.0. For RoamingOmniglot we train the network for 40k steps with a batch size 32 and maximum sequence length 150 across 2 GPUs and an initial learning rate 2c-3 decayed by 0. x at 20k and 30k steps. For RoamingRooms we train for 20k steps with batch size 8 and maximum sequence length 100 across 4 GPUs and an initial learning rate le-3 decayed by 0.1 x at 8k and 16k steps. We use the BCE coefficient À = 1 for all experiments. In semi-supervised experiments, around 30% examples are labeled when the number of examples grows large (a 0.3, see Equation IT). Early stopping is used RoamingRooms experiments where the checkpoint with the highest validation AP score chosen. For RoamingRooms, allow cupi-supe writing to the prototype memory and we find it helps training stability. The probability starts with 0.0 and increase by 0.2 every 2000 training steps until reaching 1.0.
sample Bernoulli sequences
unlabeled inputs to gradually
16

---

Published as a conference paper at ICLR 2021
Table 9: Split information for RoamingOnnigloo. Each column an alphabet and we include all the characters in the alphabet in the split. Rows continuation of lines.
Angelic
Grantha
Ko
Aurek-Besh
Japanese (hiragana)
Malay
Asomtavruli
Sanskrit
Ojibwe
Korean
Arcadian
Greek
Alphabet the Magi
Blackfoot
Futurama
Train
Tagalog
Anglo-Saxon Futhore Burnese
Braillc
Cyrillic
Avesta
Gujarati
Ge ez
Syriac (Estrangelo)
Atlantean
Japanese (katakana) Glagolitic
Balinese
Atemayar Qelisayer
Ti Tifinagh
latin
Inuktitut
Hebrew
Mkhedruli
Armenian
Val
Early Aramaic
Bengali
Gurmukhi
Kannada
Keble
Malayalam
Manipuri
Mongolian
Test
Old Church Slavonic
Oriya
Syriac (Sertc)
Sylheti
Tengwar
Tibetan
ULOG
Table 10: Split information for RoamingRooms, Each column is the ID of an indoor world. Rows continuation of the lines.
r1c124BcV1o JnbYfDe2gKZ 29hnd4uzFmx
ULsKaCPVEJR v2xKFyx4Asd
ESuDoFAP3SH
sWUmblawc2A
UXmj2M21
mJXg=FtmKg4
EU6Fwg7SyZv
gYvKGZ5eRqb gxdoqLA6rwa
YFuZgd05vW) gIV8FGeVJC9
sI4fr6TAbpF
WVfe2iggaN
12yaW3q2PVE WYY7iVyf5p@ VFua26m2Qom YmJkgBEsHnn
2t7WUnJeko7 plie4gee7qrG cV4RVe2vu51 XcA2TqTSSA
ur6pPg6Qu1A
!pXnuDyAj8r b8CTxDM8gDG x8r5xyUWy9e X7HyMhZNoso
ааygHfaÑo7d
Train
IbHürupSAjP SKIMIppTHHU 2azg1b91c7z 2n8KARJN3HM
59hNv5qa7GM EDJbREaghzI qoiz81JEWZ2 qivSolVnCic Vt2gJaWjCF2
VæqibhrpDEA D7G3Y4RVNz 2MojNKEp431 uNb90FRL6hY
5IpN3gDmAK7
zqfAleλoiTq e9zR4mvMiw* yqstnuAEVhm zaNo4HB9u12 JF19kD82Mey
wc2JMjhGNZB rPc6DW41Mge jh4fc5c5qoQ HxpKθynjfin
GdvgFv5R125
kEZ7cmS4wCh VyŁNrZiPKCB D7N2EKCX4S
PX4nDJXEHrG
s8pcmisQ38h dhjSzFoUFzz RPaz2sHmrrY 1LXtFKjw3qL 8194nk5LbLH
Val
jtcxE69GiPV QUCTe63B5sX
p5wJjkokbxx
JCFG25nY
82sE5b5pLXE
oLBMNvg9in8 :47D5H71a5s 26MF0CV1Bux WUC4YcD=cY PRbA3pwrgk9
SN83YJsR3w2 g26f7yhEvPG ac262MMG7af 7y3aRwLe3Va B6ByNegPMKs
Test
UWV835sGsW3 VLzqgDo31VF 17DRE5sb8ly Pa4otMBVnkk
SzKstлWn8Ze
PuKPg4mmafe
Fm678kyY322 15noydFURCK
ARN2Jeq3xxb
5q/pvUzZiYa
B.3 DATA AUGMENTATION
For RoamingOnniglot, we pad the 28 28 image to 32x32 and then apply random cropping.
For RoamingRooms, we apply random cropping in the time dimension to get chunk of 100 frames input example. We also apply random dropping of 5% of the frames. We pad the 120x 160 images to 126 x 168 and apply random cropping in each image frame. We also randomly flip the order of the sequence (going forward or backward).
B.4 SPATIOTEMPORAL CONTEXT EXPERIMENT DETAILS
We use the Kylberg texture dataset Kylberg 2011 without rotations. Texture classes are split into train, val, and test, defined in Table 12 We resize all images first to 256x256. For each Omniglot image, a 28 x8828 patch is randomly cropped from texture image to serve as background. Random Gaussian noises with mean zero and standard deviation 0.1 are added to the background images.
17

---

Published as a conference paper at ICLR 2021
Table 11: RoaningRooms dataset split size
Split
Worlds
Sequences
Frames
Train
4,699
823,444
Val
20
725
125,823
Test
10
1,547
271,335
Total
90
6,971
1,220,602
A) Instance Category Distribution
B) Lniqque Viewpoints / 100 Frames
10a
2000
10P
1000
10
C) Objects/ / Frame
DJ Frames Between Occurrences
E) Instance Distribution
F) Viewpoint Distribution
10*
16a
102
10
107
10a
107
Figure 7: Additional statistics about our RoamingRooms dataset.
For spatial background experiments, we added an additional learnable network of the same size as the main network take the background image as input, and output the same sized embedding vector. This embedding vector is further concatenated with the main embedding vector to form the final embedding of the input. We also found that using spatially overlayed images with a single CNN can achieve similar performance well. The final numbers are reported using the concatenation approach since is less prone to overlay noises and more similar the implementation we use in the RoamingRooms experiments.
B.5 BASELINE IMPLEMENTATION DETAILS
Online meta-learning (OML): The OML model performs gradient descent step for each input. order for the model to predict unknown, we use the probability output from the softmax layer summing across the unused units. For example, if the softmax layer has 40 units and have only seen classes so far, then we sum the probability from the 6th to the last units. This summed probability separately trained with binary cross entropy, same as in Equation 9
The inner learning rate is set to 1e-2 and we truncate the number of unrolled gradient descent steps to 5/20 CattminngOmmiio///oamnnnii GoamingOmnight the network is trained with a batch size 32 across 2 GPUs, for a total of 20k steps, with an initial learning rate 2e-3 decayed by 0.1 10k and 16.7k steps. For RoamingRooms, the network is trained with batch size 8 across 4 GPUs, for total of 16k steps, with an initial learning rate 1e-3 decayed by 0.I at 6.4k and 12.8k steps.
in order to make the computation feasible. For
Long short-term memory (LSTM): We apply a stacked two layer LSTM with 256 hidden dimen- sions. Inputs are hÇNN concatenated with the label one-hot vector. If example is unlabeled, then the label vector all-zero. We directly apply a linear layer on top the LSTM to map the LSTM memory output into classification logits, and the last logit is the binary classification logit reserved for unknown. The training procedure is the same as our CPM model.
Differenniable neural computer (DNC): order to make the DNC model work properly, we found that it is sometimes helpful to pretrain the CNN weights. Simply initializing from scratch and train CNN+DNC end-to-end sometimes results in poor performance. We hypothesize that the attention structure in the DNC model is detrimental to representation learning. Therefore,
18

---

Published as a conference paper at ICLR 2021
Table 12: Split information for the Kylberg texture dataset. Each column is an texture type. Rows continuation of lines.
blanket2 pearlsugarl sesamescedsi
ceiling2
floor2 scarf2
grassl screen1
linseedsi
Train
rice2
seat2
stonel
stoneslabl
Val
blanket1 ricel
canvasi stone2
ceilingl
floorl
scartl
walll
lentilsi stone3
cushionl
rugl
sandl
Test
oatmeall
seatl
for RoamingOnniglot experiments, we use pretrained ProtoNet weights for solving 1-shot 5-way episodes to initialize the CNN, and we keep finetuning the CNN weights with 10% of the full learning rate. For RoamingRooms experiments, we train the full model end-to-end from scratch.
The DNC is also modified that it is more effective using the label information from the input. In the original MANN paper (Santoro et al. (016) for one-shot learning, the input features hÇSN and the label one-hot IDD simply concatenated to feed into the LSTM controller of MANN. We find that it is beneficial to directly add label one-hot vector as an input to the write head that generates the write attention and the write content. Similar the LSTM model, the memory readout is also sent to a linear layer in order to get the final classification logits, and the last logit is the binary classification logit reserved for the unknowns. Finally we remove the linkage prediction part of the DNC due to training instability.
The controller LSTM has 256 hidden dimensions, and the memory dimensions. There are 4 read heads and 4 write heads. The training procedure is the same as CPM.
slots each with 64
Online ProtoNet: Online ProtoNet is our modification the original ProtoNet (Snell et al. 2017). It is similar to CPM model without the contextual RNN. The feature from the CNN is directly written to the prototype memory. In addition, we do not predict the control hyperparameters 36 ofro)) 7t (r,w7} from the RNN and they are learned regular parameters. The training procedure is the same as CPM.
Online MatchingNet: Online MatchingNet our modification of the original Match- ingNet (Vinyals et al.) (2016). We do not consider the context embedding in the MatchingNet paper since it was originally designed for the entire episode using an attentional RNN encoder. is similar to online ProtoNet but instead of doing online averaging, it directly stores each example and its class. Since it an example-base storage, we did not extend it to learn from unlabeled examples, and all unlabeled examples are skipped. We use similar decision rule to determine whether an example belongs to a known cluster by looking at distance to the nearest exemplar stored memory, shifted by 3 and scaled by 1//. Note that online MatchingNet is not efficient at memory storage since scales with the number of steps in the sequence. In addition, we use the negative Euclidean distance the similarity function. The training procedure is the same CPM.
Online infinite mixture prototypes (IMP): Online IMP is proposed as a mix of prototype and example-based storage by allowing class to have multiple clusters. If an example is classified as unknown or it is unlabeled, we will assign its cluster based on our prediction, which either assigns it to one the existing clusters or creates a new cluster, depending on its distance the nearest cluster. If cluster with an unknown label later is assigned with an example with known class, then the cluster label will also be updated. We use the same decision rule as online ProtoNet to determine whether example belongs to known cluster by looking the distance to the nearest cluster, shifted by 3 and scaled by 1/7. As described above, online IMP has the capability of learning from unlabeled examples, unlike online MatchingNet. However similar to online MatchingNet, online IMP is also not efficient at memory storage since in the worst case also scales with the number of steps in the sequence. Again, the training procedure is the same CPM.
19

---

Published as a conference paper at ICLR 2021
Table 13: Effect of forgetting over a time interval on RoamingRooms. Average accuracy vs. the number of time steps since the model has last the label of particular class.
Supervised
Semi-Supervised
1-2
3-5
6-
10
11
20
21 50
51
100
1-2
3-5
6- 10
11
20
21-50
51 100
OPN I-Shot
93.5
89.3
79.4
67.2
60.3
60.1
86.5
83.6
76.3
68.4
64.7
61.5
CPM 1-Shot
95.7
92.2
85,7
75.2
70.0
66.4
91.0
88.7
82.9
77.0
72.2
66.5
OPN 3-Shot
95.1
91.8
85.6
78.2
74.6
73.8
92.6
88.0
85.1
81.1
80.6
76.7
CPM 3-Shot
96.1
93.8
81.4
79.1
78.2
94.8
91.0
86.9
83.1
82.7
79.2
Table 14: Effect of forgetting over time interval on RoaminglmageeNet. Average accuracy vs.
the number of time steps since the model has last seen the label of a particular class.
Supervised
Supervised
1-2
3-5
6-
10
11
20
-0-50
51
100
3-5
6-
10
11
20
21-50
51
100
OPN 1-Shul
40.8
35.9
33.0
30.7
27.0
21.4
40.5
37.7
35,9
33.7
31.5
28.4
CPM 1-Shot
67.5
52.9
35,5
24.2
18.3
13.8
60.4
51.3
39.5
26.6
21.8
15.4
OPN 3-Shot
52.5
50.3
48.8
47.2
44.4
42.3
57.6
55.1
54.6
52.3
52.1
49,5
CPM 3-Shot
77.8
64.5
46.6
32.9
24.7
17.9
76.1
61.8
48.6
30.5
24.1
15.6
C ADDITIONAL EXPERIMENTAL RESULTS
C.I EFFECT OF FORGETTING
We report the effect of forgetting of RoamingRooms and RoaminglmageNet in Table 13 and 14
C.2 EMBEDDING VISUALIZATION
Figure 8 shows the learned embedding of each example in Online ProtoNet our CPM model RoamingOmnigloo sequences, where colors indicate environment IDs. Online ProtoNet, the example features does not reflect the temporal context, and result, colors are scattered across the space. By contrast, in the CPM embedding visualization, colors are clustered together and we see a smoother transition of environments in the embedding space.
C.3 CONTROL PARAMETERS VS. TIME
Finally we visualize the control parameter values predicted by the RNN in Figure 9 we indeed need two sets of and y for read and write operations separately as they learn different values. 3 is smaller than 9" which means that the network is more conservative when writing to prototypes. 70h grows larger over time, which means that the network prefers softer slope when writing prototypes since in the later stage the prototype memory has already stored enough content and it can grow faster, whereas in the earlier stage, the prototype memory is more conservative to avoid embedding vectors to be assigned to wrong clusters.
We verify that
20

---

Published as a conference paper at ICLR 2021
Online ProtoNet
CPM (Ours)
Figure 8: Embedding space visualization of RoamingOmmigloo sequences using t-SNE (Maaten Hinton, 2008). Different color denotes different environments. Text labels (relative to each environment) annotated beside the scatter points. Unlabeled examples shown in smaller circles with lighter colors. Left: Online ProtoNet; Right: CPM. The embeddings learned CPM model shows smoother transition of classes based their temporal environments.
beta "M
10
Roamingome ngmmnigl
RoamingRooms
20
40
80
100
120
140
20
80
100
Time
Timo
gamma "r
gamma
1.8
1.10
1.8
1.35
1.4
12
Roaming Ornnng lot
RoamingRaoms
0.90
Figure CPM control parameters (8 "> vs. time. Left: RoamingOmniglot sequences;
Right: RoamingRooms sequences: Top: gr.00 the threshold parameter; Bottom: 7 the tempera- ture parameter.
21


Reviews:
[{"review_id": "o10wFYO7j0o", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new learning paradigm that combines both few-shot learning(FSL)  and continual learning (CL) to provide a more realistic learning environment rather than the traditional train-test-retrain approach in FSL. Two environments are proposed, along with a novel dataset. The evaluation seems to be thorough, with strong baselines (conventional approaches adapted to the proposed setting). A novel approach is proposed based on augmenting ProtoNets with contextual memory and is shown to have consistently strong performance compared to the baselines on both tasks.\n\nStrengths:\n+ The paper is very well written and reads very nicely. I particularly liked the motivation for the task.\n+ The use of contextual memory (incorporating both spatial and temporal context) is very interesting and is a promising approach for both FSL as well as a general learning architecture for visual event perception.\nConcerns:\n- While very excited by the potential of the proposed learning environment, I am a bit confused about how the actual implementation/evaluation maps to the motivation. For example, from what I can see, all baselines (including the proposed model) have access to the number of classes that are present in the data (k). A softmax-based decision function forces the model to choose one of these classes, either based on some online learning-based features or just through storing examples. Now, the premise (that of knowing when and what to learn) is not quite satisfied here and thus the metric (Average Precision) doesn't quite capture the entire picture. I think a better metric, IMHO, would be the harmonic mean of novel class detection and known class prediction. This would allow us to actually ascertain whether the models are learning novel instances and not just getting \"lucky\".\n- The baselines that are chosen are classic approaches to few-shot learning. Not many continual learning approaches are tested and I think that would make for a better comparison. Again, the actual learning setting is quite a bit more simplified than what is claimed.\n\nOverall, I think it is a very nicely written paper with some issues with evaluation settings that might be exaggerating the performance of baselines.\n\n==== Post discussion Update ====\n\nI am updating my score to accept after the discussion.", "extra_scores": {}, "extra_text": {}}, {"review_id": "W4Ku2cMDVPZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work aims to make a realistic learning setting by combining few-shot learning and continual learning in the online setting. Similar to few-shot learning, the model needs to adapt to new classes with a few samples (at least in the beginning). Similar to continual learning, the model needs to learn new classes over time while being tested on the older classes as well. When encountering a new class, the model is expected to recognize that. Similar to the online setting, model evaluation happens on each trial, after which the model can be updated with that data (labeled or unlabeled). This new paradigm is called Online Contextualize Few-Shot Learning.\n\nThe authors recognize the importance of spatio-temporal context in human learning. Building on this, they propose a new dataset, RoamingRooms, that incorporates such context. The authors propose a new method, Contextual Prototypical Memory, to tackle this problem. It makes use of an RNN to encode contextual information and a prototype memory to remember previously learned classes.\n\nPros:\n1. The new setting proposed, Online Contextualized Few-Shot Learning, is a very realistic setting. Few-shot learning misses that classes are repeated over time. Continual learning misses that new classes are not processed and learnt in groups. This new paradigm combines the two settings and improves on their shortcomings to make it more realistic. Additionally, this is all done in an online setting.\n2. The use of spatio-temporal context in creating the dataset and the model is realistic and interesting.\n3. The proposed model is simple, with components added specifically to make use of the additional information in the dataset (the RNN) or to output additional information required for the task (the new class detection branch).\n\nCons:\n1. Authors mention that there exist some datasets under a very similar setting, namely CORe50, OpenLORIS, and synthetic task sequences of Omniglot and Tiered-ImageNet. If something similar does exist, the authors should report numbers on these datasets rather than RoamingOmniglot.\n2. Authors mention that [1] proposed a model for a setting very similar to Online Contextualized Few-Shot Learning. Even if this method detects new classes by thresholding the probabilities for novel class detection, it should be used as a baseline method.\n3. Clever fine-tuning is competitive, if not the state-of-the-art, for continual learning [2] and few-shot learning [3]. How does this perform as a baseline?\n4. Average precision is used as the metric for this setting. What about the maximum F-1 scores?\n\nNotes:\n1. There is a lot going on in this paper. The writing can be made less redundant and more to the point to incorporate more details.\n\n[1] Massimo Caccia et al. Online Fast Adaptation and Knowledge Accumulation: A New Approach to Continual Learning.\n[2] Hang Qi et al. Low-Shot Learning with Imprinted Weights.\n[3] Guneet S. Dhillon et al. A Baseline for Few-Shot Image Classification.", "extra_scores": {}, "extra_text": {}}, {"review_id": "S219z92c1i7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "############## Summary ##############\n\nThis paper presents a new definition of the continual learning problem which attempts to bridge the gap between artificial settings typically used to evaluate continual learning and the way the real world requires us humans to perform. Concretely, this setting involves changing environments, where the agent is expected to autonomously detect when new classes are encountered. The changes in the environment include both spatial and temporal cues to enable the agent to differentiate between environments. Additionally, this work creates a new data set and adapts an existing one to this novel setting, and provides a method that can deal with it, as validated empirically.\n\n############## Strengths ##############\n\n1. The provided evaluation setting is close to what we would want in a continual learning setting.\n2. The provided data sets are potentially useful for evaluating future algorithms.\n3. The presented method is effective in handling this challenging evaluation setting.\n\n############## Weaknesses ##############\n\n1. While the evaluation setting is closer to a human setting, it appears that there is a pre-training stage where the agent prepares for that evaluation setting, which is not described in detail.\n2. The techniques used to create the Roaming version of Omniglot could potentially be applied to different existing data sets, but this is not described in the paper, limiting the usefulness of it.\n3. Since the authors do not create additional benchmarks following this procedure, the evaluation is only on two data sets, which limits the reader's ability to assess the benefits of the proposed approach.\n\n############## Recommendation ##############\n\nI recommend this paper for acceptance. I believe that problem settings that become increasingly realistic is a valuable direction of work, and although this manuscript perhaps oversells how realistic their proposed setting is, I believe it is still a step in the right direction. The introduction of a novel data set on Matterport3D could be impactful to future researchers working in this field, and provides a more realistic benchmark than many existing ones. Finally, the proposed approach is novel and effective.\n\n\n############## Arguments ##############\n\nThe main contribution of this work is to propose a novel problem formulation for continual few-shot learning, which is closer to realistic continual learning. I believe this in itself is interesting. The evaluation setting requires the agent to automatically detect when new classes are encountered, and simultaneously classify objects into existing classes. However, from the experimental section, it appears that the authors' proposed method requires a pre-training phase, where the agent encounters many evaluation scenarios like this one, in order to perform well in the final evaluation scenario. This is never explicitly stated or explained in detail, but if it is the case, it is highly unrealistic, as it requires the agent to essentially live multiple lives, before being able to perform well in a \"test\" life. I would encourage the authors to explain this in more detail, and more honestly assess how realistic their problem formulation is. Even if the pre-training phase is unrealistic, the evaluation phase is still realistic, which is valuable. This should be explicitly stated to make the contributions of the paper clearer. \n\nI very much liked the proposed RoamingRooms data set, and I also believe that to be a valuable contribution. The fact that a similar data set can be created from Omniglot interesting. Would it be possible to create \"Roaming\" versions of other synthetic benchmarks? What are the limitations for this? Since we typically expect continual learning methods to be evaluated in various benchmarks, it would be valuable to see diverse benchmarks for this new setting, or at least the details for how to create them. This could also enable a more comprehensive evaluation of the proposed approach in this submission.\n\nThe idea of using the recurrent net to output the thresholds used to make decisions of when to detect instances as novel is quite interesting, but I was left lacking an intuitive description of what this should do and how.\n\nIn terms of the empirical results, I believe them to demonstrate a substantial margin of improvement with respect to baselines, and the set of baselines to be sufficient. The ablative tests show the usefulness of each part of the proposed approach.\n\n############## Additional feedback ##############\n\nThe following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.\n\nIntro\n- What is a trial?\n- Hyperref to Figure 4 leads to Section 4.\n- This section states that comparisons will be made against few-shot baslines, which made me think that no comparisons would be made against continual learning baselines. Please clarify that you indeed compare against continual learning baselines.\n\nSec 2\n- Last paragraph of FSL has an incomplete sentence, and it remains unclear how Caccia et al. (2020) compares to the submission. Since this is clearly the most closely related work (as evidenced in Table 6), it is very important to make this comparison as complete as possible.\n- Omniglotor --> Omniglot or\n- Very comprehensive and well-written related work section, clearly outlining various lines of related work, including recent (and even concurrent) efforts that handle very similar settings to this one.\n\nSec 3\n- Figure hyperrefs point to captions (below the figure).\n- At this point, the problem formulation should clearly state how training will be carried out. This section does not mention the fact that the agent will first encounter various evaluation sequences and use those to generalize to unseen evaluation sequences. It mentions multiple few-shot sequences and train/val/test splits, but it is largely unclear at this point how these various sequences are supposed to be used.\n- The way the problem setting is presented, it makes it hard to differentiate it substantially from few-shot or continual learning, since it mainly focus on evaluation being done continually. It seems like a crucial part of the distinction is that the agent is evaluated even on unseen classes.\n    - This seems to be closely connected to work on open world and open set learning. Could the authors provide some guidance into how it compares to that setting? [1]\n\nSec 4\n- When doing online average, the method seems to assume that the current prototype p_{t-1} is fixed. But, if the model parameters change over time, which I expect they do in a continual learning setting, this prototype would be different if recomputed. How is this taken this into account?\n\nSec 5\n- I would've liked to see the ablation tests on RoamingRooms as well.\n- My only concern is that the evaluations are only on two different data sets. \n- In terms of the baselines, it is unclear which can leverage previous sequences to learn the new ones. I believe this is only possible with OML and variants, whereas all the other methods can only leverage data from the current sequence. \n\n\nAppendices\n- Thanks for providing lots of details on the experimental setting for baselines.\n- The additional visualizations are useful, especially the one of the learned thresholds over time (Figure 9).\n- These appendices should be referred to in the main text so the reader knows to look for them. Same for the provided videos.\n\n\n[1] Geng, C., Huang, S. J., & Chen, S. (2020). Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.", "extra_scores": {}, "extra_text": {}}, {"review_id": "35oPnOr_ZIO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#################################\n\nSummary:\n\nThe paper presented a new setting of online contextualized few shot learning to mimic human learning. This setting combines continual learning and few shot learning, and additionally considers context switch. Specifically, a learning method is presented with a sequence of samples that might come with labels. The method is then tasked to classify the current input into known categories, or recognize the input as belonging to a “new” category, while at the same time updating the model for known and new categories. Two new datasets (hand-written characters and indoor images) were constructed to support the learning setting. An extension of Prototypical Network (Snell et al.) was explored for this new setting. The results were compared against several baselines and were quite promising. \n\n#################################\n\nPros:\n\n* A novel setting of continual few-shot learning that considers context switching. The motivation is well articulated (naturalistic human learning). The setting has great potential to address some of the key challenges in AI (e.g., embodied vision, robotics, etc).\n* New datasets to support the proposed setting. Those dataset might facilitate future research in this direction.\n* The paper explored several baselines for the proposed setting, including an interesting extension of ProtoNet. The experiments are solid and the results are promising.  \n\n#################################\n\nCons:\n\n* The evaluation metric will need some thoughts\n\nProper evaluation metric is a critical component for the proposed learning setting. While the authors did provide a short description of the evaluation metrics (AP and N-Shot accuracy), those metrics lack some details and are not well justified. My understanding is that both metrics are accumulated from the starting to the current step and across all sequences. This was not particularly clear from the text. Also the definition of AP is different from standard average precision (the TP/TN/FP/FN definitions are different). Some more description is needed in the text. \n\nHow do these metrics capture catastrophic forgetting? For example, the accuracy / AP vs. the time interval between the current label and the last time the same label was observed.  \n\n\n#################################\n\nMinor comments:\n\nPage 3 paragraph 2: “focuses on more flexible ...” not a sentence. \n\nTable 1 and 2: Why the std of AP metric is not included?\n\nFigure 5 caption does not match the layout. Is this 1-shot accuracy? I did not find the description in the paper. \n\n#################################\n\nJustification for score:\n\nA good paper proposing an interesting learning paradigm. I’d expect some more discussion of the evaluation metric. Otherwise, I am happy to accept the paper. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new online contextualized few-shot learning setting, with two associated datasets (notably, including one obtained from trajectories within the real-world Matterport3D reconstructions). A simple recurrent contextualized extension of Prototypical Networks is also proposed as a stronger baseline, demonstrating the need for incorporating such context. The reviewers all agreed that this is an interesting setting combining continual and few-shot learning, offering a more realistic problem that mirrors those that might be encountered by embodied agents. The authors provided very detailed rebuttals, answering some of the questions and concerns raised by the reviewers. In the end, all reviewers agreed that this paper would contribute a significant novel setting, and so I recommend acceptance. I encourage the others to include modifications related to some of the comments, such as strengthening/clarifying the setting including metrics, details of the method, etc.
Decision: Accept (Poster)