Contextual Transformation Networks for Online Continual Learning
Authors: Quang Pham
Chenghao Liu
Doyen Sahoo
Steven HOI
Venue: ICLR 2021 Poster (2021)
Keywords: Continual Learning

Abstract: Continual learning methods with fixed architectures rely on a single network to learn models that can perform well on all tasks.
As a result, they often only accommodate common features of those tasks but neglect each task's specific features. On the other hand, dynamic architecture methods can have a separate network for each task, but they are too expensive to train and not scalable in practice, especially in online settings.
To address this problem, we propose a novel online continual learning method named ``Contextual Transformation Networks” (CTN) to efficiently model the \emph{task-specific features} while enjoying neglectable complexity overhead compared to other fixed architecture methods. 
Moreover, inspired by the Complementary Learning Systems (CLS) theory, we propose a novel dual memory design and an objective to train CTN that can address both catastrophic forgetting and knowledge transfer simultaneously. 
Our extensive experiments show that CTN is competitive with a large scale dynamic architecture network and consistently outperforms other fixed architecture methods under the same standard backbone. Our implementation can be found at \url{https://github.com/phquang/Contextual-Transformation-Network}.

Published as a conference paper at ICLR 2021
CONTEXTUAL TRANSFORMATION NETWORKS FOR ONLINE CONTINUAL LEARNING
Quang Pham', Chenghao Liu", Doyen Sahoo", Steven C.H. Hoi 1.2
1 Singapore Management University hgpham. 2017 smu. egu.sg
2 Salesforce Research Asia
{ [chenghao. liu, dsahoo, shoi shoi]Jsaleeffrcee.com
ABSTRACT
Continual learning methods with fixed architectures rely on a single network to learn models that can perform well all tasks. As result, they often only accommodate common features of those tasks but neglect each task's specific features. On the other hand, dynamic architecture methods network for each task, but they are too expensive to train and not scalable in practice, especially in online settings. To address this problem, we propose a novel online continual learning method named "Contextual Transformation Networks" (CTN) to efficiently model the task mask-specificc specifii while enjoying neglectable complexity overhead compared to other fixed architecture methods. Moreover, inspired by the Complementary Learning Systems (CLS) theory, we propose a novel dual memory design and an objective to train CTN that can address both catastrophic forgetting and knowledge transfer simultancously. Our extensive experiments show that CTN is competitive with large scale dynamic architecture network and consistently outperforms other fixed architecture methods under the same standard backbone. Our implementation be found at https: //////tttt //githu.. Trans
have separate
com/
INTRODUCTION
Continual learning is promising framework towards building Al models that can learn continuously through time, acquire new knowledge while being able to perform its already learned skills (French, 1999; 1992; Parisi et al., 2019; Ring, 1997). On top of that, online continual learning is particularly interesting because it resembles the real world and the model has to quickly obtain new knowledge on the fly by levering its learned skills. This problem is important for deep neural networks because optimizing them in the online setting has been shown to be challenging (Sahoo et al., 2018; Aljundi et al., 2019a). Morcover, while is crucial to obtain new information, the model must be able to perform its acquired skills. Balancing between preventing catastrophic forgetting and facilitating knowledge transfer is imperative when learning on a stream of tasks, which is ubiquitous in realistic scenarios. Thus, in this work, focus on the continual learning setting in an online learning fashion, where both tasks and data of each task arrive sequentially (Lopez-Paz & Ranzato, 2017).
In the literature, fixed architecture methods employ shared feature extractor and a of classifiers, one for each task (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a;b; Aljundi et al., 201 2019a). Although using a shared feature extractor has achieved promising results, the common and global features are rather generic and not well-tailored towards each specific task. This problem is even more severe when old data are limited while learning new tasks. As result, the common feature extractor loses its ability to extract previous tasks' features, resulting in catastrophic forgetting. On the other hand, while dynamic architecture methods such as Rusu et al. (2016); Li et al. (2019); Xu & Zhu (2018) alleviate this problem by having a separate network for each task, they suffer from the unbounded growth the parameters. Moreover, the subnetworks' design not trivial and requires extensive resource usage (Rusu et al., 2016; Li et al., 2019), which is not practical in many applications. These limitations motivated us to develop novel method that can facilitate continual learning with fixed architecture by modeling the task-specific features.

---

Published as a conference paper at ICLR 2021
Task
Semantic Memory
Controller 0
Task 2
Sampling
Episodic Memory
Yi
Base
h (x.x
t
Mxx
Classifier
C
Task
Data for Task
not
Data Sources
Used in Outer-loops
Used in Inner-loops
Overview of the Contextual Transformation Networks (CTN). CTN consists of a controller O that modifies the features of the base model Φ. The base model is trained using experience replay on the episodic memory while the controller is trained to generalize to the semantic memory, which addresses both alleviating forgetting and facilitating knowledge transfer. Best viewed in colors.
Figure I:
To achieve this goal, we first revisit a popular result in learning multiple tasks that each task's features are centered around a common vector (Evgeniou & Pontil, 2004; Aytar & Zisserman, 2011 I; Pentina& & Lampert, 2014; Liu al., 2019b). This result motivates us develop a novel framework of Contextual Transformation Networks (CTN), which consists of base network that learns the common features of given input and a controller that efficiently transforms the common features to become task-specific, given task identifier. While one can train CTN using experience replay, it does not explicitly aim at achieving a good trade-off between stability and plasticity. Therefore, we propose novel dual memory system and a learning method that encapsulate alleviating forgetting and facilitating knowledge transfer simultaneously. Particularly, we propose two distinct memories: the episodic memory and the semantic memory associated with the base model and the controller, respectively. Then, the base model is trained by experience replay on the episodic memory while the controllers is trained to learn task-specific features that can generalize to the semantic memory. As a result, CTN achieves good trade-off between preventing catastrophic forgetting and facilitating knowledge transfer because the task-specific features can generalize well to all past and current tasks. Figure gives an overview of the proposed Contextual Transformation Network (CTN).
Interestingly, the designs of our CTN and dual memory are partially related the Complementary Learning Systems (CLS) theory in neuroscience (McClelland et al., 1995; Kumaran et al., 2016). Particularly, the controller acts as neocortex that learns the structured knowledge of each task. In contrast, the base model acts as hippocampus that performs rapid learning to acquire new information from the current task's 's training data. Following the naming convention of memory in neuroscience, CTN is equipped with two replay memory types. (i) the episodic memory (associated with the hippocampus) caches a small amount of past tasks' training data, which will be replayed when training the base networks. (ii) the semantic memory (associated with the neocortex) stores another distinct set old data only used to train the controller such that the task-specific features can generalize well across tasks. Moreover, the CLS theory also suggests that the interplay between the neocortex and the hippocampus attributes to the ability to recall knowledge and generalize to novel experiences (Kumaran & McClelland, 2012). Our proposed learning approach closely characterizes such properties: the base model focuses on acquiring new knowledge from the current task while the controller uses the base model's ' knowledge generalize to novel samples.
In summary, our work makes the following contributions. First, we propose CTN, a novel continual learning method that can model task-specific features while enjoying neglectable complexity overhead compared to fixed architecture methods (please refer to Table 4). Second, we propose a novel objective that can improve the trade-off between alleviating forgetting and facilitating knowledge transfer to train CTN. Third, we conduct extensive experiments on continual learning benchmarks to demonstrate the efficacy CTN compared to a suite of baselines. Finally, we provide a comprehensive analysis to investigate the complementarity of each CTN's component.
2 METHOD
Notations. We denote & as parameter of the base model that extracts global features from the input and 0 as the parameter of the controller which modifies the features from Φ given a task identifier The task identifier can be set of semantic attributes about objects of that task (Lampert et al., 2009) or simply an index of the task, which we use in this work as a one-hot vector. A prediction is given as Ip, (h (xx, f)), where ge, (-) is the task T/'s classifier with parameter so: such as fully
2

---

Published as a conference paper at ICLR 2021
connected layer with softmax activation. And he. (e, t) is the final feature after transformed by the controller. We denote D as the training data of task Tr,M, and M; as the episodic memory and the semantic memory of task Ty respectively. The episodic memory and semantic memory maintains two distinct sets of data obtained from task Tr. The episodic memory of task as M similarly, 2t denotes the semantic memory of the first t tasks.
TT- is denoted
Remark, Both the M and M are obtained from D through the learner's internal memory management strategy and contains distinct samples from each other such that their combined sizes do not exceed a pre-defined budget.
2.1 LEARNING PASK-SPECIFL FEATURES FOR CONTINUAL LEARNING
Given a backbone network, one can implement the task-specific features by employing a set of task-specific filters and applying them to the backbone's 's output. However, this trivial approach is not scalable, even for small networks. In the worst case, results in storing an additional network per task, which violates the fixed architecture constraint. Since we want to obtain task-specific features with minimal parameter overhead, we propose to use a feature-wise transformation (Perez et al., 2018) to efficiently extract the task-specific features h(x,1) t) from the common features h(x) as follows:
M(x [ t) = 7t
@ h(x)) - |/
Bt and (Te,31) = co
(1)
lmlll2
where @ denotes the element- multiplication operator, and cD) is the controller implemented as a linear layer with parameter that predicts the transformation coefficients (7r, 9r) given the task identifier t. Since the task identifiers are one-hot vectors, which are sparse and make training the controller difficult, we also introduce an embedding layer to map the task identifiers to dense, low dimensional vectors. For simplicity, we will use f to refer to both the embedding and the linear layer parameters. In addition, instead of storing a set of coefficients fyr, B,} for each task, we only need fixed set of parameter 0 to predict these coefficients, which results in the fixed parameters in the controller. The coefficients (7t, } are fo-noraalieed and then transforms the common features fax t) to become task-specific features h(:;; t). Finally, both feature types are combined by a residual connection before passing to the corresponding classifier ge(-) to make the final prediction:
9up. (h(x,, t))), and h(x, t) h(xx,t) + h(xxt),
(2)
where a-)) is nonlinear activation function such as ReLU. Importantly, when the task task-specific features removed, i.e., M(x,tt) O, Eq. 2 reduces the traditional experience replay. Lastly, for each incoming task, CTN has to allocate new classifier, which is the same for all continual learning methods, and new embedding vector, which usually low dimensional, e.g. 32 or 64. Therefore, CTN enjoys almost the same parameter growth as existing continual learning methods.
2.2 TRAINING THE CONTROLLER
While one can train CTN with experience replay (ER), does not explicitly address the trade-off between facilitating knowledge transfer and alleviating catastrophic forgetting. This motivates us to develop a novel training method that can simultaneously address both problems by leveraging the controller's task-specific features. First, we introduce a dual memory system consisting of the semantic memory M. associated with the controller and the episodic memory M associated with the base model. We propose to train only the base model using experience replay with the episodic memory to obtain new knowledge from incoming tasks. The controller is also trained so that the task-specific features can generalize to unseen samples to the base model stored in the semantic memory. As a result, the task-specific features can generalize to both previous and current tasks, which simultaneously encapsulate both alleviating forgetting and facilitating knowledge transfer. Formally, given the current batch of data for task T, as B,, the training of CTN can be formulated as the following bilevel optimization problem (Colson et al., 2007):
Outer problem:
ming Letrl M
Inner problem:
S.l s" = arg min L ((Φ, B.UU.
(3)
where d² denotes the optimal base model corresponding to the current controller 0. Since every CTN's prediction always involves both the controller and the base model, we use
3

---

Published as a conference paper at ICLR 2021
(" 0). Br U M2T) to denote the training loss of the pair {Ø, 0) on the data Be U. U. Sim- ilarly, (.) denotes the controller's ' loss. For simplicity. omitted the dependency of the the loss on the classifiers' parameters and imply that the classifiers are jointly updated with the base model. Since we do not know the optimal transformation coefficients of any task, the controller is trained to minimize the classification loss of the samples via Φ. We implement both the training and controller's losses as the cross-entropy loss. Notably, Eq. 3 characterizes two nested optimization problems: the outer problem, which trains the controller to generalize, and each controller parameter parameterizes an inner problem that trains the base model to acquire new knowledge via experience replay. Moreover, only Φ is trained in the inner problem, while only @ is updated in the outer problem.
The bilevel optimization objective such as Eq. 3 has been successfully applied in other machine learning disciplines such as hyperparameter optimization, meta learning (Franceschi et al., 2018 8; Finn et al., 2017), and AutoML (Liu et al., 20)9 In this work, we extend this framework to continual learning to train the controller. However, unlike existing works (Franceschi et al., 2018; Finn et al., 2017; Liu et al., 2019a), our Eq. 3 has to be solved incrementally when a new data sample arrives. Therefore, we consider Eq. 3 as an online learning problem and optimize it using the follow the leader principle (Hannan, 1957). Particularly, we relax the optimal solutions of both the inner and outer problems to be solutions from a few gradient steps. When a new training data arrives, we first train the base model & using experience replay for a few SGD steps with an inner learning rate o, each of which is implemented as:
- Φ aV ((0, O,, U M
(4)
Then, we optimize the controller 0 such that it can improve o' performance on the semantic memory:
0 с 0 -V (L MzT1))
where 3 is the outer learning rate. As result, Eq. 3 is implemented as an alternative update procedure involving several outer updates to train 0, each of which includes an inner update train Φ. Moreover, performing several updates per incoming sample does not violate the online assumption since we will not revisit that sample in the future, unless it stored in the memories.
2.3 TRAINING THE BASE NETWORK
Despite using task-specific features, the base network may still forget previous tasks because of the small episodic memory. To further alleviate catastrophic forgetting in 6, we regularize the training loss (*r(-) with behavioral cloning (BC) strategy based on knowledge distillation (Hinton et al., 2015; van de Ven & Tolias, 2018). Let ý be the logits of the model's prediction before the softmax layer (-)), we regularize the training loss on the episodic memory data in Eq. as:
1
Uk
C C² (2. = (()())))) + ADKL
(6)
where À is the trade-off parameter, is the softmax's 's temperature, and ÿt is snapshot of the model prediction on the sample (æ, k) at the end of task Tx. While the behavioral cloning strategy requires storing Úk. the memory increase is minimal since ÿR is vector with dimension bounded by the total classes, which is much smaller than the image x dimension. Importantly, the behavioural cloning strategy used to alleviate catastrophic forgetting, which only happens in the base model, not the controller. Particularly, the controller's inputs are task identifiers such as one-hot vectors, which are fully available during learning. In summary, our episodic memory stores the input image æ. its corresponding label y and the soft label J. while the semantic memory stores the input-label pair æ, y.
3 RELATED WORK
3.1 CONTINUAL LEARNING
Prior works in continual learning can be grouped into three main categories: (1) regularization methods, (2) episodic memory based methods, and (3) dynamic architecture methods.
Regularization approaches (Kirkpatrick et al., 2017; 77 Zenke et al., 2017; Aljundi et al., 2018; Ritter et al., 2018) penalize the changes of important parameters to previous tasks using variant of
4

---

Published as a conference paper at ICLR 2021
knowledge distillation (Li & Hoiem, 2017) or via quadratic constraints. However, such methods usually isolate parameters or find a common solution to all tasks, limiting the model's capacity.
Episodic memory based approaches store a small amount of data from previous tasks and interleave it with data from the current task. Old data can be used as a constraint to optimize the model Lopez- Paz & Ranzato (2017); Chaudhry et al. (2018), representation learning Rebuffi et al. (201 7b), or simply just perform experience replay (ER) Chaudhry et al. (2019b); Aljundi et al. (2019a); Rolnick et al. (2019); van de Ven & Tolias (2018). While regularization and episodic memory-based methods have achieved promising results, they only use a shared feature extractor. Moreover, they do not consider the goal of improving the generalizability across tasks, which CTN explicitly addresses via the proposed bi-level optimization with the dual memory design.
Dynamic architecture approaches address catastrophic forgetting by having a subnetwork for each task (Rusu et al., 2016; Serra et al., 2018 8; von Oswald et al., 2020) or being able to grow its structure over time (Yoon et al., 2018; Li et al., 2019; Xu & Zhu, 2018; Hung et al., 2019). Such methods approximate training a full, separate network per task by reducing the number of additional parameters. However, most of them require growing the backbone network during training (Rusu et al., 2016; Yoon et al., 2018; Xu & Zhu, 2018; Li et al., 2019) or extensive resource usage (Rusu et al., 2016; Li et al., 2019). which is not scalable and undesirable for many applications. Notably, the idea of conditioning the task identifiers were explored in Serra et al. (2018); von Oswald et al. (2020). However, Serra et al. (2018) uses the task identifiers to gate the network's is activations, which limits the representation capability. On the other hand, von Oswald et al. (2020) employs hypernetwork (Ha et al., 2017) to generate a whole prediction network for each task and catastrophic forgetting is avoided by performing experience replay in the hypernetwork's output space. However, this approach requires storing the hypernetwork's i output for each task, which is equivalent to a prediction network's s parameter. Therefore, while Serra et al. (2018); von Oswald al. (2020) have achieved promising results, they requires larger memory and might not be suitable for the online setting.
3.2 FEATURE-WISE TRANSFORMATION
Early works in Bertinetto et al. (2016); Rebuffi et al. (2017a) showed that instead of using a task- specific network on the input, one can employ a set of 1 x filters to extract the task-specific from the common features. However, such approaches still require a quadratic complexity overhead in the number of channels, which can be expensive. Another compelling solution is the feature- wise transformation, FiLM (Perez al., 2018), which only requires a linear complexity. Thanks to its efficiency, FiLM has been successfully applied in many problems, including meta learning (Requeima et al., 2019; Zintgraf et al., 2019), visual reasoning (Perez et al., 2018), and others fields (Dumoulin et al., 2018) with remarkable success. Notably, CNAPs (Requeima et al., 2019) proposed an adaptation network to generate the FiLM's parameters and quickly adapt to new tasks. CNAPs has showed promising results when having access to a large amount of tasks to pre-train the common features. However, this setting is different from continual learning where the learner has to obtain new knowledge on the fly. Therefore, CNAPs are principally differs from CTN in that CNAPs assume having access to a well-pretrained knowledge source and uses FiLM to quickly adapt this knowledge to new task. On the other hand, CTNs use FiLM to accelerate the knowledge acquisition when learning progressively. Lastly, we emphasize that the CTN's design is general. If more budget is allowed, the proposed CTN is readily compatible with the aforementioned feature transformation methods such as Rebuffi et al. (2017a) by adjusting the controller's 's output dimension.
3.3 META LEARNING
Meta learning (Schmidhuber, 1987), also learning to learn, refers to learning paradigm where an algorithm learns to improve the performance of another algorithm. Our CTN design related to such learning to learn architectures where the controller is trained to improve the base model's performance. Importantly, we note that there exist other continual learning variants that intersect with meta learning, such as meta-continual learning (Javed & White, 2019) and continual-meta learning (He et al., 2019: Caccia et al., 2020). However, they consider different goals and problem settings, such meta pre-training (Javed & White, 2019) or rapid recovering the performance at test time given a finetuning step before inference is allowed (He et al., 2019), which is not the conventional online continual learning problem (Lopez-Paz & Ranzato, 2017) we focus in this study.
5

---

Published as a conference paper at ICLR 2021
Table Evaluation metries on continual learning benchmarks considered. All methods use the same backbone network and 50 memory slots per task, denotes a dynamic architecture method that has separate network per task
PMNIST
CORe50
Method
ACC(1)
FM(L)
LA(1)
ACC(1)
FM(L)
LA(1)
GEM
7..84:0.95
8.57
81.74:0.77
42.56=0.86
7.36=0.90
46.84:12.22
AGEM
68.67-10.71
13.98+00.68
81.54
40.28:13.15
11.08
1..6
MER
76.59-10.77
6.88
82.09-10.33
39.28
9.08
45.52
ER-Ring
76.02-10.59
8.57+20.3
83.6910.44
41.72+#130
9.10=0.80
48.18100.1
MIR
76,58=10.10
8.34
83.57=00.07
43.50=1.92
6.14
1..98
CTN (ours)
79.01=00665
6.69
85.11=10.45
54.17
5.50=1.01
55.32-10.34
Independent"
81.05-0.29
0.00
81.05
53.54
0.00
1..54
Offline
84.95-10.95
58.69:1.41
Split CIFAR
Split minilMN
Method
ACC(1)
FM(L)
LA(t)
ACC(t)
FM(1)
LA(t)
GEM
07.77
10.93
66.45:10.06
55.04
1.81
1..13
AGEM
58.27=1086
8.76:10.67
66.12.11.17
51.1442.16
6.99
55,11:0.76
MER
61.32:+0.86
11.90=10.86
72.51=0.41
57.94+1.08
8.98=0.79
66.11+0.76
ER- ER-Ring
61.36+11001
7.20
67.05=11008
53.43+1.18
11.21
63.46:11.05
MIR
63.37=1.99
10.53+1.63
73.27=0.77
51.97=1.58
10.37+2.72
60.63+13.43
CTN (ours)
0..45
6.33
1.44
65.82:
3.02
6..43
Independent"
67.21
0.00
67.21
65.85
0.00
65.85
Offline
74.11+00.66
71.15
Meta learning has been an appealing solution to learn a good initialization from a large amount of tasks (Finn al., 2017), even in an online manner: Online Meta Learning (OML) (Finn et al., 2019). However, we emphasize that OML fundamentally differs from our CTN in two aspects. First, OML requires all data of previous tasks and aims to improve the performance of future tasks, which is different from continual learning. Second, OML learns an initialization and requires finetuning at test time, which is not practical, especially when testing on learned tasks. contrast, CTN a continual learning method that maximizes the performance of the current task as well as all previous tasks. Moreover, CTN can make a prediction at any time without requiring an additional finetuning step.
EXPERIMENTS
4.1 BENCIMARK DATASETS AND BASELINES
We consider four continual learning benchmarks in experiments. Permuted MNIST (pMNIST) (Lopez-Paz& & Ranzato, 2017): each task is a random but fixed permutation of the original MNIST. We generate 23 tasks with 1,000 images for training and the testing set has the same amount of images as in the original MNIST data. Split CIFAR-100 (Split CIFAR) (Lopez-Paz & Ranzato, 2017) is constructed by splitting the CIFAR (Krizhevsky & Hinton, 2009) dataset into 20 tasks, each of which contains 5 different classes sampled without replacement from the total of 100 classes. Split Mini ImageNet (Split minilMN) (Chaudhry et al., 2019a), similarly, we split the minilMN dataset ( (Vinyals et al., 2016) into 20 disjoint tasks. Finally, we consider the CORe50 benchmark by constructing a sequence of 10 tasks using the original CORe50 dataset (Lomonaco & Maltoni, 2017).
Throughout the experiments, we compare CTN with suite of baselines: GEM (Lopez-Par & Ranzato, 2017), AGEM (Chaudhry et al., 201 2019a), MER (Riemer et al., 2019), ER-Ring (Chaudhry et al., 2019b), and MIR (Aljundi et al., 2019a). We also consider the independent model (Lopez-Paz & Ranzato, 2017), a dynamic architecture method that maintains a separate network for each task, and each has the same number of parameters other baselines. While the independent model is unrealistic, it is highly competitive and was used as an upper bound of a state-or of-the-an art dynamic architecture method in Hung al. (2019). Finally, include the Offline model, which does not follow the continual learning setting and performs multitask training on all tasks' data. Due to space constraints, we provide the results of less competitive methods in Appendix. C.2.

---

Published as a conference paper at ICLR 2021
75
Averaged Accuracy
Averaged Accuracy
70
70-
65
60-
195
Memory size per task
Memory size per task
(a) Split CIFAR
(b) Split minilMN
Figure 2: ACC(T) as function of the episodic memory size on the Split CIFAR-100 and Split miniMMN benchmarks. Best viewed colors.
We use a multilayer perceptron with two hidden layers of size 256 for PMNIST, reduced ResNet18 with three times fewer filters (Lopez-Paz & Ranzato, 2017) for Split CIFAR and Split minilMN, and a full ResNet18 on CORE50. Following (Lopez-Paz & Ranzato, 2017), we use a Ring buffer as the memory structure for all methods and random sampling select data from memory, including the episodic and semantic memories of CTN. The exceptions are MER (Riemer et al., 2019), which uses reservoir sampling, and MIR (Aljundi al., 2019a), which their sampling strategies by the authors. For CTN, the episodic memory and semantic memory are implemented as two Ring buffers with sizes equal to 80% and 20% of the total budget. This configuration is also cross-validated from the validation tasks. For each incoming batch of data, we randomly push 80% samples to the current task's 's episodic memory and the other 20% are for the current task's semantic memory.
proposed
We follow the procedure proposed in Chaudhry et al. (2019a) to cross- voos-validatt all hyperparameters using the first three tasks. Then, the best configuration is selected to perform continual learning on the remaining tasks. During continual learning, the task identifier is given to all methods. We optimize all models using SGD with a mini-batch of size ten over one epoch. We run each experiment five times, each has the same task order but different initialization seed, and report the following metrics: Averaged Accuracy (Lopez-Pazz & Ranzato, 2017): ACC(1) (higher is better), Forgetting Measure (Chaudhry et al., 2018): FM(L) (lower is better), and Learning Accuracy (Riemer et al., 2019): LA(1) (higher is better).
4.2 RESULTS OF CONTINUAL LEARNING BENCHMARKS
Table I reports the evaluation metrics of the models on four continual learning benchmarks considered with 50 samples per task. We observe that CTN is even comparable with the independent method and outperforms other baselines by a large margin. We remind that the independent method has T times more parameters than the remaining methods, where 7 is the total number of tasks. Moreover, CTN can exploit the relationship across tasks via the task identifiers to improve its performance. For example, learning to classify "man" and "woman" may be helpful to classify "boy" and "girl" because they belong to the same superclass "people". Finally, CTN significantly outperforms the baselines by achieving a better trade-off between alleviating catastrophic forgetting and facilitating knowledge transfer, as shown by lower FM(J) and higher LA(t). Overall, CTN achieves state-of tfate-of-the-ar results, even comparable with arge scale dynamic architecture method, while enjoying neglectable model complexity overhead compared to fixed architecture methods.
ACC(') as function of the episodic memory size.
We study the models' performances as the memory size increases. We consider the Split CIFAR 100 and Split minilMN benchmarks and train the models of CTN, ER, MIR, and GEM with the total memory size per task increasing from 50 to 200. Fig. 2 plots the ACC(t) curves as a function of the memory size. Generally, the performances of all methods increase with larger memory sizes. Overall, CTN consistently outperforms the competitors across all memory sizes. Notably, in both benchmarks, CTN can achieve comparable performances to the Offline model when the memory size per task is only 175. The results show that CTN not only excels in the low memory regime but also scales remarkably well when more memory budget is allowed.
7

---

Published as a conference paper at ICLR 2021
Table Evaluation metrics on the Small Split CIFAR benchmarks, M denotes the memory per task
Reduced Split CIFAR 25%, M = 50
Reduced Split CIFAR 25%, M = 25
Method
ACC(T)
FM())
LA(t)
ACC(1)
FM(L)
LA(1)
GEM
51.01+-0.95
5.65+11.09
53.39++0.98
47.33-00.9
8.77++1.58
53.79
ER-Ring
52.02::0.90
4.31
53.9710.65
48.15
8.22
53.88::0.93
MIR
50.82:10.83
5.22:10.68
53.27
47.19=0.54
8.41
53.51
CTN
61.27:10.93
4.19:10.78
61.92:1.15
56.17
7.71
61.40
Reduced Split CIFAR 10%, M = 50
Reduced Split CIFAR 10%, M 25
Method
ACC(1)
FM())
LA(t)
ACC(1)
FM(()
LA(1)
GEM
44.06+11.11
6.97
48.67+0.84
42.67=1.62
8.39+11.5
49.46-+0.00
ER- ER-Ring
14.60+11.65
6.07+
0..36
43.09=1122
7.68+1194
49.40
MIR
46.63:10.56
4.38:10.4
48.35
44.12=0.94
6.84
48.48:10.76
CTN
56.61110.74
0.43
58.77
52.64=0.63
6.74
57.27
Table 3: ACC(1) of each component in CTN on Split CIFAR and Split mini Imagenet with 50 memory slots per task. BC: behavioral cloning (Eq. 6), C: controller, BO: Bilevel optimization (Eq. 3)
Split CIFAR
Split minilMN
BC C BO
ACC(1)
FM(J)
LA(f)
ACC(1)
FM(J)
LA(t)
CTN
V
V
0..65
6.33
73.43::0:45
65.8210.559
3.02
1..73
V
V
66.37-0.53
9.6410.98
75.40:10.60
60.04
10.48:1.99
69.8711.60
V
64.46=1.16
8.51#1.53
72.23+0.554
61.01#1.09
5.31
64.35+
62.76=0049
10.10:10.8
72.12+0.41
58.95+1.76
9.08+1.61
66.94+10.3
ER
61.36=1.01
7.20=0.72
67.05=1.08 53,43+1.18 11.21 13.4
4.3 RESULTS ON LEARNING WITH LIMITED TRAINING DATA
One important goal of continual learning is to be able to learn with a limited amount of training data per task. This setting is much more challenging because it tests the leamner' ability to quickly acquire knowledge only with limited training samples by utilizing its past experiences. explore how different memory-basee based methods perform with only limited training samples per task and memory size. We consider the Split CIFAR benchmark; however, we reduce the amount of training data per task significantly. Particularly, we only consider 25% and 10% of the original data per task while the test data remains the same. We name the new benchmarks Reduced Split CIFAR 25% and Reduced Split CIFAR 10%. respectively. Notably, the Reduced Split CIFAR 10% only has five samples per class, which is extremely challenging. We compare CTN with GEM, ER, and MIR on these benchmarks with the memory size of 50 and samples per task.
this experiment,
Table 2 shows the results of this experiment. When the training data are scarce, the baselines performances drop significantly, even below 50% ACC(t) in three settings. CTN, on the other hand, consistently outperforms the baselines by large margin, from 8% to 10% across benchmarks, even the challenging Reduced Split Cifar 10%. Moreover, the three baselines have similarly low LA. showing that they struggle in acquiring new knowledge when the training data of each task limited. On the other hand, CTN can leverage information about the task-specific features to improve knowledge transfer and the learning outcomes. It is worth noting that even with 25% training data and 50 memory slots per task, CTN already outperforms several baselines that are trained with full data by cross-referencing the results with Table 1.
4.4 ABLATION STUDY
We study the contribution of each component in CTN in its overall performance and consider the Split CIFAR and Split minilMN benchmarks with an episodic memory of 50 samples per task. Particularly, we are interested how (1) the controller, (2) the bi-level optimization, and (3) the behavioral cloning strategy contribute to the base model. We implement variants of CTN with different combinations
8

---

Published as a conference paper at ICLR 2021
Table 4: Model complexity of CTN with various backbone architectures
Backbone
Controller
Total
Increase
Structure
Params
Structure
Params
MI.P (784-256-256-10
269,322
Linear model 17,728
287,050
6.58%
ResNet18 Lopez--Paz & Ranzato, 2017)
1,095,555
Linear model 20,992
1,116,547
1.92%
ResNet18 (He et al., 2016)
11,202,162
Linear model
59,200
11,261,362
0.53%
Table 5: Averaged running time (in seconds) of compared methods on the task-aware continual learning benchmarks. All methods use M=50 memory slots per task, Ring buffer, and up to four gradient updates per samples
Benchmark \\  Method
ER-Ring
MIR
AGEM
CTN
GEM
PMNIST
61
92
90
110
103
Split CIFARIO0
632
1030
680
910
1700
Split minilMN
1320
2130
1700
1890
2850
of these components and report the results in Table 3. Notably, CTN with only the controller (C) equivalent to training the base network and the controller using the vanilla experience replay approach. Despite this, the controller can offer significant improvements over ER: over 5% ACC(1) in Split minilMN. When the controller is optimized by our proposed bilevel optimization (C + BO), the performances are further improved, showing that proposed bilevel objective achieves better trade-off between alleviating forgetting and facilitating knowledge transfer. Lastly, the behavioral cloning strategy can help alleviate forgetting and further strengthen the results. Overall, each of the proposed components adds positive contributions to the base model, and they work collectively as holistic method and achieved state-of-the-ar results in continual learning.
4.5 COMPLEXITY ANALYSIS
In this section, we study the CTN's complexity with the backbones used in experiments and report the results in Table 4. In all cases, the controller only adds minimal additional parameters, almost neglectable in complex deep architectures such ResNets (He et al., 2016; Lopez-Pai & Ranzato, 2017). Therefore, we can safely compare CTN with other fixed architecture methods using the same backbone because they have nearly the same number of parameters.
Table reports the averaged running time (in seconds) of considered methods. All methods are implemented using Pytorch (Paszke al., 2019) version 1.5 and CUDA 10.2. Experiments are conducted using single K80 GPU and all methods are allowed up to four gradients steps per sample. Clearly, ER-Ring ER-Ring has the most efficient time complexity thanks to its simplicity. On the other hand, GEM has high computational costs because of its quadratic constraints. MIR also exhibits high running time because of its virtual update, which doubles the total gradient updates. CTN, in general, is slightly faster MIR and more efficient than GEM. Overall, CTN achieves great trade-off between model/compuuational complexity and performance: CTN's performances are significantly higher than considered baselines with only minimal memory and computational overhead.
5 CONCLUSION
In this work, we study the online continual learning problem and propose Contextual Transformation Networks (CTN), where a fixed architecture network can model both the common features and specific features of each task. CTN works by employing controller that modifies features of the base network conditioning on the task identifiers. To optimize CTN, we further propose novel dual memory system equipped with a bilevel optimization objective that can efficiently transfer knowledge and alleviate forgetting simultaneously. Moreover, we discuss the relationship of CTN to the Complementary Learning Systems theory in neuroscience and meta learning from different perspectives, showing that CTN is related to other disciplines. Through extensive experiments, our results demonstrate that CTN consistently outperforms fixed architecture methods and achieves ffate-of-thh-aa results. Moreover, CTN is even comparable with a large scale dynamic architecture network, while enjoying almost additional model complexity.

---

Published as a conference paper at ICLR 2021
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 139-154 2018.
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pp. 11849-11860 2019a.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 11254-11263, 2019b.
Yusuf Aytar and Andrew Zisserman. Tabula rasa: Model transfer for object category detection. In 2011 international conference on computer vision, pp. 2252-2259 IEEF, 2011.
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. Advances neural information processing systems, pp. 523-531, 2016.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko. Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexande Lacoste, David Vazquez, et al. Online fast adaptation and knowledge accumulation: new approach to continual learning. arXiv preprint urXiv22003.05856 2020.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. International Conference on Learning Representations (ICLR), 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint 1902: 10486, 2019b.
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153(1) 2:3-255,, 2007.
Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature- wise transformations. Distill, 2018. doi: 10.2391 1//ditiill.0000111111111 https://distill.uubb2
Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 109-117, 2004.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic veta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp.
JMLR. org, 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In Proceedings of the 36th International Conference on Machine Learning- Volume 97, pp. 1920-1930. JMLR. org, 2019.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learnin. In International Conference on Machine Learning, pp. 1568-15771 2018.
Robert M French. Semi-distributed representations and catastrophic forgetting in connectionist networks. Connection Science, 4(3-4):36--377, 1992.
10

---

Published as a conference paper at ICLR 2021
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, (0):128-33,, 1999.
David Ha, Andrew Dai, and Quoc Le. Hypernetworks. International Conference on Learning Representations (ICLR), 2017.
James Hannan. Approximation to bayes risk in repeated play. Contributions the Theory of Games, 3:97-139, 1957.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition. pp. 770-778, 2016.
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv:1906.05201, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URL http: http://axivvoooooee
abs/ 1103. 02531.
Saihui Hou, Xinyu Pan, Chen Change Loy. Zilei Wang, and Dahua Lin. Lifelong learning via progressive distillation and retrospection. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 437-452, 2018.
Ching Yi Hung, Cheng-Hat Hao Tu, Cheng- Bieng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, pp. 13669-13679, 2019.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, pp. 1818-1828, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinst Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Dharshan Kumaran and James L McClelland. Generalization through the recurrent interaction of episodic memories: a model of the hippocampal system. Psychological review, 119(3):573, 2012.
Dharshan Kumaran, Demis Hassabis, and James McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 20(7): 512-534, 2016.
Christoph Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 951-958. IEEE, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEFF, 86(11):2278-2333 1998.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pp. 3925-3934, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. Interna- tional Conference on Learning Representations (ICLR), 2019a.
Shikun Liu, Edward Johns, and Andrew Davison. End-to-end multi-task learning with attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1871-1880, 2019b.
11

---

Published as a conference paper at ICLR 2021
Vincenzo Lomonaco and Davide Maltoni. Core50: dataset and benchmark for continuous object recognition. Proceedings of Machine Learning Research, pp. 17-26. PMLR, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning, In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
James L McClelland, Bruce L McNaughton, and Randall C O'Reilly. Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint (eXXiv1803.020000 2018.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-peeformance deep learning library. In Advances Neural Information Processing Systems, pp. 8024-8035, 2019.
Anastasia Pentina and Christoph Lampert. A pac-bayesian bound for lifelong learning, In Interna- tional Conference on Machine Learning, pp. 991-999, 2014.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Quang Pham, Doyen Sahoo, Chenghao Liu, and Steven CH Hoi. Bilevel continual learning, arXiv preprint arXiv:2007. 1555, 2020.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, pp. 506-516, 2017a.
Sylvestre-AIvee Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017b.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In Advances in Neural Information Processing Systems, pp. 7959-7970, 2019.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. International Conference on Learning Representations (ICLR), 2019.
Mark B Ring. Child: A first step towards continual learning. Machine Learning, 28(1):77-104, 1997.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems, pp. 3738-3744 2018
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, pp. 348-358, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606666666471 2016.
12

---

Published as a conference paper at ICLR 2021
Doyen Sahoo, Quang Pham, Jing Lu, and Steven C. H. Hoi. Online deep learning: Learning deep neural networks on the fly. Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, 2018.
Jürgen Schmidhuber. Evolutionary principles in self-referential learning, on learning how learn: the meta-mela-... hook. PhD thesis, Technische Universität München, 1987.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. Proceedings of the 35th International Conference on Machine Learning- Volume 80, pp. 4548-4557. JMLR. org, 2018.
Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. In The biology and technology of intelligent autonomous agents, pp. 165-196. Springer, 1995.
Gido M van de Ven and Andreas S Tolias. Generative replay with feedback connections as general strategy for continual learning. arXiv preprint 1rX9v:1809.100 10635, 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. International Conference on Learning Representations (ICLR), 2020.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Processing Systems, pp. 899-908, 2018.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. International Conference on Learning Representations (ICLR), 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference Machine Learning-Volie Volume 70, pp. 3987- 3995. JMLR. org, 2017.
Luisa Zintgraf. Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, pp. 7693-7702. PMLR, 2019.
13

---

Published as a conference paper at ICLR 2021
APPENDIX
This Appendix is organized as follows. In Appendix A, we provide the details of the continual learning protocol and evaluation metrics used in this work. Appendix provides pseudo-code of CTN and its implementation on standard deep learning architectures such as MLP and Residual networks. Appendix provides additional experiment details, including the summary of our benchmarks, results of additional baselines, model and computational complexity, and hyperparameter settings.
A CONTINUAL LEARNING PROTOCOLS
Continual learning, a.k.a. lifelong learning, McCloskey & Cohen (1989); Thrun & Mitchell (1995); Ring (1997) has been extensive studied over the past decades. this work, we consider the problem of online continual learning studied by (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 201 9a).
Specifically, at time step i, a learner receives an input pair (æ. t) and makes a prediction y = fx, 8; w) Note that here the input belongs to underlying task Tr, which is also given to the learner. Each task Tr comprises a training dataset D'r, whose data will be sequentially presented to a learner, and a separate testing set D.r. Following (Chaudhry et al., 2019a), we also assume having access to a small amount of tasks prior to learning for hyperparameter validation and episodic memory M can be used. We assume that the stream of data (,,, 44), ) _ arrives sequentially and the goal is to optimize a model that can perform well on all observed tasks far.
by predictor f()) parameterized some parameter
To measure the model performance, we adopt three standard metrics: Average Accuracy ACC(t) (Lopez-Paz & Ranzato, 2017), Forgetting Measure FM(() Chaudhry et al. (2019a), and Learning Accuracy LA(1) (Riemer et al., 2019). Denote 0i as the model's accuracy evaluated on the test set Dj after it has been trained on the most recent sample in dataset D; of task T. Then, the above metrics are defined as:
Average Accuracy (higher is better): the average accuracy of all observed tasks:
1 ACC(1) T
arz
Forgetting Measure (lower is better): the average forgetting of all previous tasks:
FM(J) = T 1-1.
max
a.j
QT.
Learning Accuracy (higher is better): measures the performance of model on task right after finishes training that task:
1 LA(t) - Σ 02
In the literature, there exists several different continual learning protocols. Here we categorize them based on questions: (i) Is information about the task of sample given during training and testing? (ii) Does training within each task is performed online? For question (i), when we do not know which task does the sample belong to, evaluation called "single-head and there is a shared classifier for all tasks (Aljundi et al., 2019b). In question (ii). data of a task can either be fully available when task changes or can arrives sequentially. When all the task data is available, training within tasks can be done in an offline fashion with multiple epochs through data. Our protocol used is in this work is proposed Lopez-Paz & Ranzato (2017) in which data of each task arrives sequentially and task identifier is also given. Moreover, hyperparameter cross-validation is also important problem in continual learning, regardless of the protocol considered. Particularly, we must not use data of future tasks when searching for the hyperparameter. Here follow Chaudhry et al. (2019a) and assume that we have access to a small amount of tasks prior to continual learning. Such tasks will not be encountered again during actual continual learning and only be used for cross-validation.
14

---

Published as a conference paper at ICLR 2021
B IMPLEMENTING CONTEXTUAL TRANSFORMATION NETWORKS
B.1 PSEUDO-CODE
We provide the details algorithm our CTN and its subroutines in Alg. 1. For simplicity, we drop the dependency of the losses the parameters and use (B.( denote [ (Φ, 40, B,; and L(B.) to denote ( 4,, B,; 0)
Algorithm 1: Contextual Transformation Networks (CTN)
Algorithm Tra i nCTN (0, Φ, Digr Require: base model Φ, controller 0, classifier y
Init: 0, Φb MM, + MM +
for t +- 1 to I do
for j < to Nbbtchee do Receive a mini batch of data B, from Di
I/ Receive the dataset D sequentially
x² y t Random sampling from Bj
I/ Sampling for the semantic memory // Update the semantic memory
M Memory MemoryUpdatel (M {x" }))
M + Memory Update (M ( BB)
// Update the episodic memory
for < to nouter do
for n < 1 to Niinner do
10
8em + Sample(M
B, Bem UB;
& < Φ (B.)
// Inner update the base model s // Inner update the classifier
s + & VoL (Bn)
8 + Sample (Mar)
0 + 0 - VoL(Bam)
// Outer update the controller
15
M + M U (ū(/T)}
// Calculate the behavioural cloning outputs // Update the total episodic memory
M M UM
16
return 0,0 ④
Procedure Forward (0, Φ, a.
Tt. + (o()) M);;)) - Te
// Calculate the transforming coefficients I/ Calculate the task-specific features
24 & h(xx) +
&
Toell
return 9p. (h(x, 1))
Procedure MemoryUpdate (M,
Require: Implement M as a queue (FIFO) data structure for (a, y) in B do M.append y)
return... M
B.2 IMPLEMENTING CTN ON COMMON ARCHITECTURES
In this section, we provide the implementation details of CTN on feedforward network bases that we use in our experiments. We implement the context model as single regression layer. Moreover, we share the parameter of the scale and shift models 7, 8, resulting in one set of parameters that takes task embedding as input and outputs both scale and shift values for particular layer of the CTN with the base network as MLP
base network. Next, we will describe our implementation and ResNet (He et al., 2016). For CTN, we will use h as the original features, as the task specifcc features, and the combine features.
15

---

Published as a conference paper at ICLR 2021
CTN with Multilayer Perceptron. Consider L L-layers MLP with the form:
ho =
h1 (W, 1111), -=
1,
hr. - Softmax (W (,,hh-11)
where the last layer is the softmax classifier lu. Since the last classification layer is already conditioned on the task information, here we are interested in conditioning the intermediate layers ht4L. The CTN with MLP implemented as:
hn =
h ReeLU(W, h1-1), -.N.-,,.... L-
h4 =ReLU(n @ Wi h1-1 + Be), -1,1, 1,...,L-
hu =h hn hg 9 Softmax (W,,he-))
We condition each hidden layer of MLP by using context network for each layer. Each context network does not share parameters, however, the scale and shift models for one layer is shared.
CTN with Deep Residual Network. Unlike MLP, we apply the task conditioning after the residual blocks instead of each convolution layer. Particularly, given a residual block defined as:
h1 = ReLLU(BN(oon(())) h3 = =B((cc(((()) -ha + h4
h2 BN(conv(h:))
h4 = conv(z) (z)
The task-conditioned residual block is computed as:
h = ReLU(h) ReLU(r & B
While in principle, it is possible to have context network for each of the residual block, empirically found that this does not offer significant improvements over using only one controller on the last residual block. Therefore, only controller the last residual block in all experiments that use ResNet.
C EXPERIMENT DETAILS
C.I DATASET SUMMARY
We summary the datasets used in our experiments in Table 6.
Table Summary datasets used our experiments
Dataset
Classes
Train
Test
Dimension
MNIST (LeCun et al., 1998)
10
1,000
10,00
28x28
CIFAR100 100 (Krizhevsky & Hinton, 2009)
100
50,000
10,000
3x32x32
minilMN (Vinyals et al., 2016)
100
50,000
10,000
3x84x84
CORe50 (Lomonaco & Maltoni, 2017)
50
119.894
44,971
3x84x84
For each benchmark, we normalize the pixel values to [o, by dividing their values by 255.0 as used in Lopez-Paz & Ranzato (2017), other data preprocessing steps performed.
C.2 ADDITIONAL BASELINES
In Table 7, we provide a more comprehensive comparison with more baselines in the four benchmarks considered: Permuted MNIST, Split CIFARI00 and Split minilMN, and CORe50. Some of these baselines are less competitive, thus, were not included in the main paper due to space constraints. We provide brief descrption of each baselines in the following.
16

---

Published as a conference paper at ICLR 2021
Table Evaluation metries on continual learning benchmarks considered. All methods use the same backbone network for all benchmarks, episodic memory size is M-50 samples per task
PMNIST
CORc50
Method
ACC(1)
FM(L)
LA(1)
ACC(1)
FM(L)
LA(t)
Finetune
1.56611.50
20.67+1.64
80.89=0.45
4.3810.10
49.66#1.14
1..08
LwF
63.51
14.2943.05
75.76=1.43
31.20:10.66
20.44=1.37
49.20
EWC
67.3413.00
11.00+2.36
76.59=1.49
31.86:3.90
14.34+3.08
42.98+2.50
GEM
74.84=00.5
8.37
81.74+0.77
42.56=+0.86
7.36+0.90
46.84+2.22
KDR
72.97
9.20+4..4
81.40=0.41
OOM
0OM
OOM
AGEM
68.67100.7
13.98
81.54
4..18
11.08:1..01
46.68
MER
7..59
6.88
82.09-0.33
39.28+1.225
9.08+
45.52
ER-Ring
76.02=
8.5740.33
83.69=0.44
41.72+#1.30
9.10+10.0
48
MIR
76.58=0.10
8.3410.11
83.57=0.07
43.50+1.92
6.1410.91
45.98
BCL
7.91+0.34
6.23+0.14
83.75=0.28
44.72+1.31
5.97+0.88
47.68+0087
CTN (ours)
7.601
6.69
85.11=0.45
54.1771085
5.50+1.001
55.32++0.4
Independent"
81.05=0.22
0.00
81.0550.29
53.54+1.10
0.00
53.54
Offline
84.95=0.95
89.73=0.91
Split CIFAR
Split minilMN
Method
ACC(t)
FM(4)
LA(1)
ACC(t)
FM(L)
LA(t)
Finetune
33.52:1.113
33.88+2.78
65.15=1.18
31.51+2.00
26.00=12.12
5..83
EWC
39.46=3.775
24.69-13.84
64.54=1.20
32.52+0.53
25.74=2.778
56.39+22455
ICARL
50.27=0.84
16.55=0.82
65.83=1.53
44.95:10.08
17.59=0.40
61.46:1..0
GEM
57.77-0.86
10.93-11.03
66.45-10.06
55.04.11.88
7.81+1.70
10.13
KDR
62.75+1080
5.01
66.11:0.70
56.89412.45
4.83+1123
59.29
AGEM
58.2740.86
8.76+10.7
66.12=1.17
51.14=2.16
6.99+1.96
55.11
MER
61.32-0.86
11.90-10.86
72.51=0.41
57.94+1.08
8.9810.79
66.11
ER-Ring
6..36=11001
7.20
67.05=1.08
53.43+1.118
11.21#1.35
63.46:11055
MIR
63.37+1.99
1..53++1.63
73.27=0.77
51.97+11.58
10.37+2.72
60.63++3.43
BCL
63.87=2.27
0.93
67.73=1.99
62.20++043
4.85++0.95
65.23
CTN (ours)
0.465:10.43
6.33
7..43:0.45
65.82.1..59
3.02
67.43
Independent"
67.21+0.51
0.00
67.21
65.85
0.00
65.85+10.8
Offline
74.11
71.15
Finetune: a naive method that learns sequentially without any regularization.
LwF (Li & Hoiem, 2017): prevents forgetting by distillation loss of the previous model on current data.
EWC (Kirkpatrick et al., 2017): penalizes the changes of important parameters to previous tasks prevent forgetting.
GEM (Lopez-Paz & Ranzato, 2017): uses episodic memory to store some data and prevents the losses of old tasks from increasing during learning new tasks.
KDR (Hou et al., 2018): uses knowledge distillation and task-specific experts to balance
between learning new tasks and alleviating forgetting.
AGEM (Chaudhry al., 2019a): an efficient version GEM by averaging the constraints in GEM.
MER: (Riemer et al., 2019) maximizes the gradient inner product between every sample pair in the memory variant of the Reptile algorithm. We use MERAlgó with mini batch of size 10 for consistency with remaining methods.
ER-Ring (Chaudhry al., 2019b): simply mixes data of previous and current tasks during training and optimizes multitask loss.
MIR (Aljundi et al., 9a): is a variant of ER which selects the samples in the episodic memory that maximizes the model's forgetting replay.
17

---

Published as a conference paper at ICLR 2021
ACC
+
LA
FM
25
75
20
ACC. LA
70
65
60
55
10
20
30
40
50
60
70
80
90
Semantic memory size (%)
Figure 3: Effect of memory size on CTN's performance. For every semantic memory size m x 50, the corresponding episodic memory size is (1 m) x 50.
BCL (Pham et al., 2020): bilevel-optimization method using Reptile update (Nichol et al., 2018) such that the base model can generalize a separate memory units. Unlike BCL, our CTN can model the task-specific features and does not need approximations solve the bilevel optimization problem.
Independent" (Lopez-Paz & Ranzato, 2017): maintains a separate model for each task, each has the same number of parameters other methods. While being unrealistic, this model was used as the upper bound model in Hung et al. (2019) thanks to its impressive performance.
Offline: upper bound model that performs multitask training on all data. Note that this model does not follow the continual learning setting. We implement the offline model by training the network three epochs over all data of all tasks.
C.3 CTN MODEL COMPLEXITY AND COMPUTATION CosT
Model complexity. Recall the interaction between the controller and the base model is described as:
h(x;t)
Y
X f(xx))
3x and {Th, Br] = co (e(()))
(7)
|reell2
where h(x) is a feature map with dimension (C' x H x W) where C' is the number of channels, H and W are the spatial dimension of this feature map. feature-wise affine transformation Tts is only required to have dimension (C' x 1 x 1) for each t and 8, In our implementation, we predict both % and Be from the task embedding e(t) by a parameter A. As a result, let e be the task embedding dimension, the embedding layer will cost (T' x e) and the controller (linear regression model) will cost (2C x e), resulting in the total (T' x + 2C x e) parameters in the controller for all tasks. In practice, this term is dominated by C because T and are the number of tasks and the embedding dimension, which are quite small. When a new task arrives, we only need to allocate parameters in the embedding matrix.
Overall, CTN offers significantly performance improvements with only minimal memory overhead.
C.4 EFFECT OF THE SEMANTIC MEMORY SIZE
We study how the semantic memory size effects CTN performance. For this experiment, we consider the validation tasks in the Split CIFAR- 100 benchmark (the first three tasks) and vary the semantic memory size and episodic memory size such that their total sizes equals to 50 samples per task.
Fig. 3 reports the results of this experiment. We can see that when the semantic memory size is 10 (20% of the total memory), CTN achieves the highest ACC, FM(J) and lowest FM(J) and these evaluation metrics degrades when the semantic memory sizes increases. Generally, we have to balance the amount of memory for controller and the base network. Since the controller is only a simple model, only requires small amount of data in the semantic memory.
18

---

Published as a conference paper at ICLR 2021
C.5 HYPERPARAMETHR SELECTION
We provide the haper-parameters values of methods considered in our task-aware experiments. For brevity,
MNIST denote both the Permuted MNIST and Rotated MNIST benchmarks. The Small Split CIFAR experiments use the same hyper-parameter settings as the original Split CIFARI00. For each method, we use the same hyper-parameter notation and description as provided in the corresponding original papers.
GEM
Learning rate: 0.03 (MNNIST, Split CIFARI0O), 0.05 (Split minilMN)
Gradient noise yy: 0.5 (all experiments)
Number of gradient updates: (all experiments)
AGEM
Learning rate: 0.03 (MNNIST), 0.1 (Split CIFARI00) 0.3 (Split minilMN)
Number of to estimate gradient constraints: 1000 (MNIST), 850 (Splti CIFARI00, Split minilMN)
Number of gradient updates: (all experiments)
MER
Learning rate: 0.03 (MNIST), 0.05 (Split minilMN), 0.1 (Split CIFAR100)
Replay batch size: 64 (Permuted MNIST, Split CIFAR), 128 (Split minilMN)
Number of gradient updates: (all experiments)
Across batch leanring rate y: 0.3 (all experiments)
ER
Learning rate: 0.03 (MNIST, Split CIFARI00, Split mini Imagenet)
Replay batch size: 10 (all benchmarks)
Number of gradient updates: (all experiments)
MIR
Learning rate: 0.03 (MNIST, Split CIFARIO0, Split mini Imagenet)
Replay batch size: 10 (all benchmarks)
Number of gradient updates: (all experiments)
CTN
Inner learning rate a: 0.01 (all benchmarks)
Outer learning rate 3: 0.05 (all benchmarks)
Regularization strength A: 100 (all benchmarks)
Temperature T: (all benchmarks)
Replay batch size: 64 (all benchmarks)
Number of inner and outer updates: (all benchmarks)
Semantic memory size in percentage of total memory: 20% (all benchmarks)
Each hyper- parameter is cross validaeddd using grid search the three validation tasks, which will not be encountered during continual learning. The grid for each
is provided below.
Learning rate, including inner, outer (CTN) and across batch (MER) learning rates: [0.01, 0.03, 0.05, 0. 0.3)
Number of gradient updates, including inner and outer updates (CTN): 2, 3,
Replay batch size: [10, 32, 64, 128]
Temperature 7: 11,2,5, 10]
Regularization strength À (CTN):
19

---

Published as a conference paper at ICLR 2021
Table 8: Alternative strategies to reduce forgetting in CTN's inner optimization. BC: behavioural cloning strategy in Eq. 6
Split CIFAR
Split minilMN
Method
ACC(1)
FM(4)
LA(t)
ACC(1)
FM(L)
LA(t)
CTN-BC
67.65:10.43
6.73
73.43=0.45
65.82=0.59
3.02:11.13
67.73:11.73
CTN-EWC 60.33
9.33
68.78=0.24
57.69
5.59:00.45
61.53
CTN-GEM
64.40+2.52
8.06+1.92
71.49=0.46
6..65=0.80
5.83+0.84
64.42::0.46
À (CTN): [1, 10, 25, 50, 100)
7 (GEM): [0, 0.5, 1]
Semantic memory size in percentage of total memory (CTN): [10%, 20%, 40%]
D VARIANTS OF CTN
In this section, we explored alternative strategies for alleviating catastrophic forgetting in CTN's inner optimization problem, which is experience replay (ER) to train the base model Φ. Particularly, instead of the behavioural cloning strategy in Eq. 6, we consider two strategy alleviate forgetting in ER by combining ER with EWC (Kirkpatrick et al., 2017) and GEM (Lopez-Paz & Ranzato, 2017). Table 8 show the results of this experiment on the Split CIFAR 100 and Split minilMN benchmarks. We can see that the behavioural cloning strategy significantly outperforms its competitors, EWC and GEM. Notably, using CTN with EWC requires larger episodic memory to store the previous tasks' parameters and their importance. Moreover, using CTN with GEM results in slower running time since GEM has the slowest training time as shown in Table 5. The results show that the behavioural cloning strategy is more suitable for alleviating forgetting in ER, while enjoying less memory overhead or faster running time compared to other alternatives.
20


Reviews:
[{"review_id": "wj05RrcJJNU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary of paper**\n\nThis paper introduces a continual learning method called Contextual Transformation Networks (CTNs). CTNs consist of a base network and a controller, which outputs task-specific feature modulators. Both these have independent memories used to reduce forgetting. Additionally, the base network has an additional regularisation term. These two networks are trained together (formulated as a bi-level optimisation problem). Experiments on many different benchmarks are provided, with CTNs outperforming competing baselines on different metrics, often by large amounts. Some good additional experiments are also provided (different memory sizes, smaller datasets, ablations of the three major parts of CTNs).\n\n**Review summary**\n\nFor me, the great experimental results/section means I am suggesting this paper be accepted. I do not see the intuition behind many of the design considerations in CTN: there is essentially a shared base network, with task-specific scaling/shifting (given by a controller network) and task-specific heads; a key difference to previous works is the use of two different memories for the two networks. But CTN has very strong performance in continual learning benchmarks. \n\n**Pros of paper**\n\n1. The strongest part of this paper for me is the experiments section. The results are extremely good and consistent over many different benchmarks. The authors provide different metrics for continual learning (not just average accuracy), and compare against some strong baselines. The additional experiments in Tables 3 and 4 are also very nice to see. \n2. The method is presented well in Section 2, and Table 1 (comparing number of additional parameters) is nice.\n\n**Cons of paper**\n\n3. There are related works that I think the authors can mention, which have some similar ideas as in CTN (although all the ideas are never all put together as in CTN):\n(a) FiLM layers for meta-learning / continual learning / multi-task learning:\n[1] Requeima et al., 2019, \"Fast and Flexible Multi-Task Classification using Conditional Neural Adaptive Processes\"\n[2] Loo et al. 2020, \"Combining Variational Continual Learning with FiLM Layers\"\n[3] Rebuffi et al., 2018, \"Efficient parametrization of multi-domain deep neural networks\"\n(b) Soft targets for continual learning\n[4] van de Ven and Tolias, 2019, \"Generative replay with feedback connections as a general strategy for continual learning\"\n4. How long does it take to perform the bilevel optimisation in CTN (can the authors provide training times ideally, or else complexity analysis)?\n5. Section 3.1: \"Dynamic architecture approaches... suffer from the unbounded growth of the network size and extensive resource usage\". Is this true for all methods? Don't some methods try and achieve sublinear growth in parameters? Doesn't CTN also grow in a similar way?\n\n\n**Additional comments/suggestions**\n\n6. I found Figure 1 more difficult to understand than I think it needs to be. I think it would be useful for the authors to think how they can simplify it further for a future version of the paper.\n7. Did the authors try some other continual learning algorithms to avoid forgetting in the base network other than \"behavioral cloning\"? (Eg EWC or GEM or any other method.)\n\n**Update to review**\n\nI shall keep my rating at 7. The authors updated the paper and I think it is good enough to be accepted. I still note how Figure 1 is difficult to parse (what are all the variables? Why are there so many arrows and boxes? It is not possible to understand what is going on without reading the text in detail, by which time it does not add much to the reader's understanding); in my opinion it requires starting from scratch again.", "extra_scores": {}, "extra_text": {}}, {"review_id": "iu0-QGq_8sY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes CTN (Contextual Trasnformer Networks) for online continual learning. In particular, the authors introduce a dual memory framework that contains an episodic memory for base networks and semantic memory for task controllers. The overall framework is optimized with bi-level optimization. In addition, the base network also uses a KL-divergence loss to prevent catastrophic forgetting. Experiments are conducted on multiple datasets and the authors demonstrated that the proposed framewok outperforms other alternative approaches.\n\n####### Strengths######\n+ The paper is addressing an important problem, i.e., continual learning.\n+ The motivation is clear.\n+ Good results have been shown compared to other incremental learning methods.\n\n#######Weakness######\n- The novelty is a bit limited. The framework seems like a loose combination of existing well-explored techniques. For example, the modulation part  conditioned on tasks (Eqn 4) have been widely used before to modulate feature maps. For example.\n[1] TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning\n[2] TADAM: Task dependent adaptive metric for improved few-shot learning\nThe KL divergence to prevent forgetting has also been used before.\n\n- The semantic memory and the  episodic memory are a bit confusing to me. What exactly are stored into the memory? Are image samples stored in the memory? From Algorithm 1,  L8 is updating the episodic memory with a batch of samples. But L16, suggests the predictions are added.\n\nMinor:\n+ Page 4 L3, the symbols are the same for the semantic memory and the  episodic memory.\n+ Algorithm L8, the episodic memory is denoted as M^{tr} rather than M^{em}\n\n########After rebuttal#########\n\nI appreciate the author's effort in addressing my concerns. After reading the rebuttal and other reviews, I am raising my scores to 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gjcT2T978lY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles online continual neural network learning (following Lopez-Paz & Ranzato, 2017) with a combination of techniques: (1) a controller (or base parameter modulator, or hypernetwork) is introduced which produces task-specific scale and shift parameters, which modulate the feature maps of a base model (Perez et al., 2017); (2) as training data set samples are only provided once, the authors maintain experience replay buffers using soft label targets (by now standard techniques in online learning); (3) a bilevel optimization scheme where controller parameters are updated in an outer loop and base parameters adapted in an inner loop is employed. Task identities are available both during learning and at test time. This information is used to pick the correct task embeddings and the correct task-specific final classifier (\"head\").\n\nThe method achieves strong performance on appropriate benchmarks; hyperparameter tuning is handled carefully, which is not always the case in continual learning studies; the authors include a comparison to many relevant methods.\n\nThis paper's main weakness is low novelty. The method is essentially a careful application and combination of existing techniques. Some important references are missing or not correctly cited. In particular, the CNAPs method (Requeima et al., 2019) should be carefully discussed, and the differences to it highlighted. Also, the description of HAT (Serra et al.,2018) and task-conditioned hypernetworks (von Oswald et al., 2020) is not very accurate, as neither method requires \"unbounded growth of the network\"; in fact, both are quite related to the authors' approach, and both deserve more attention.\n\nThat being said, the experiments are well executed, and the study is comprehensive, as discussed above. I think that these merits outweigh the lack of novelty.\n\nSome additional comments to the authors may be found below.\n- The presentation of the method (which spans roughly two pages and a half) is lengthy and not very clear. For example the paragraph around eq. 1 is rather long, mixing related work with the actual method. There are a few imprecise statements, e.g., \"We add an embedding layer on the task identifiers to avoid sparsity issues from using one-hot vectors\". Is this embedding layer learned, and if so, how are sparsity issues avoided?\n\n- Eq. 3 and SGD description below: in the outer problem, do the authors assume that $\\phi^*$ is constant with respect to $\\theta$? Here, CAVIA (Zintgraf et al., 2018) should be cited and discussed.\n\n- The temperature hyperparameter $\\tau$ is undefined in eq. 6.\n\n- Should line 7 of the algorithm be omitted? If not, how are $B_M$ and $B_n$ defined?\n\n- Conflicting notation issues: the authors use $\\mathbf{c}_\\theta(\\cdot)$ to denote the controller (e.g., eq. 1), but also $\\theta(t)$ (e.g., line 2 of procedure Forward, algorithm 1).\n\n- Appendix B.2: I don't fully follow the equations below \"CTN with Multilayer Perceptron\". Why is $h_l$ obtained from $\\tilde{h}_{l-1}$?\n\n- The paper must be screened for typos and grammatical errors, there are plenty throughout.\n\n---\nEdit: after the authors' response I maintain my recommendation to accept the paper and keep my score of 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2yQpzCrl2Ct", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I have previously reviewed this submission for NeurIPS 2020 and cannot find substantial reason to alter my original review. I will therefore copy my original review along with the original rating. \n\nHowever, as I do no longer have access to the file submitted by the authors for NeurIPS 2020, I encourage the authors to correct me by detailing modifications to the manuscript made for ICLR 2021, in which case I will be open to changing my score provided sufficient reason.\n\n---\n\nThe authors introduce Contextual Transformation Networks (CTNs), a replay-based method for continual learning based on a dual-memory design and a controller that modulates the output of a shared based network to task-specific features.\n\nPros:\n- Results are generally strong in comparison to other memory based CL techniques on accepted benchmarks.\n- CTNs are a novel (albeit simple) architecture that might inspire future work.\n\nCons:\n- The manuscript fails to describe the exact sampling strategies used for Semantic and Episodic Memory in sufficient details.\"semantic memory that stores a tiny set of data from all observed tasks\" and \"episodic memory that caches a small amount of past tasks’ training data\". It's not clear to me what the difference between the two descriptions is. To me \"observed\" and \"past\" appear to be synonyms in this setting. Notation used later in the text uses no time index for the semantic memory (writing $M^{sm}$ as opposed to $M_t^{sm}$), indicating that the semantic memory remains fixed. If so, is the semantic memory initialised with data from all tasks at the beginning of training? Please describe this in more details.\n- While the presented results on Image datasets are good, the CL community has to start considering more challenging and realistic tasks to make impact on other areas of Machine Learning. The perfect setting for \"Online Continual Learning\" is Reinforcement Learning and I was somewhat disappointed not to see this as an experiment.\n- Baselines for this work are almost entirely focused on rehersal-based methods. It would have been nice to see modern Regularization or Dynamic architecture methods considered.\n- The design of CTNs is somewhat ad-hoc and appears mainly based on heuristics and intuition with a rather loose neuroscience inspiration.\n- Unfortunately I found the description of the proposed method somewhat confusing. Furthermore I suggest stating fundamental properties of the method more clearly.\n\n---\n\nPost-rebuttal: Raising my score from 5->6 after authors provided a detailed list of changes.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work develops a novel framework for online continual learning, which they authors name Contextual Transformation Networks (CTN). This framework comprises a base network, which learns to map inputs to a shared feature representation, and a controller that efficiently transforms this shared feature vector to task specific features given a task identifier. Both of these components have access to their own memory. The optimization of the both the controller and base network parameters is framed as a bi-level optimization framework.

Pros:
- important and challenging problem
- strong results

Cons:
- Currently the writing creates the impression of limited novelty from a technical perspective. I would encourage the authors to more crisply highlight the technical novelty of their method. 

Decision: Accept (Poster)