Efficient Architecture Search for Continual Learning
Authors: Qiang Gao
Zhipeng Luo
Diego Klabjan
Fengli Zhang
Venue:  (2021)
Keywords: continual learning
reinforcement learning
recurrent neural network
deep neural network

Abstract: Continual learning with neural networks is an important learning framework in AI that aims to learn a sequence of tasks well. However, it is often confronted with three challenges: (1) overcome the catastrophic forgetting problem, (2) adapt the current network to new tasks, and meanwhile (3) control its model complexity. To reach these goals, we propose a novel approach named as Continual Learning with Efficient Architecture Search, or CLEAS in short. CLEAS works closely with neural architecture search (NAS) which leverages reinforcement learning techniques to search for the best neural architecture that fits a new task. In particular, we design a neuron-level NAS controller that decides which old neurons from previous tasks should be reused (knowledge transfer), and which new neurons should be added (to learn new knowledge). Such a fine-grained controller allows finding a very concise architecture that can fit each new task well. Meanwhile, since we do not alter the weights of the reused neurons, we perfectly memorize the knowledge learned from previous tasks. We evaluate CLEAS on numerous sequential classification tasks, and the results demonstrate that CLEAS outperforms other state-of-the-art alternative methods, achieving higher classification accuracy while using simpler neural architectures.


Under review as a conference paper at ICLR 2021
EFFICIENT ARCHITECTURE SEARCH FOR CONTINUAL LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Continual learning with neural networks an important learning framework in AI that aims to learn a sequence of tasks well. However, it is often confronted with three challenges: (1) overcome the catastrophic forgetting problem, (2) adapt the current network to new tasks, and meanwhile (3) control its model complexity. To reach these goals, we propose novel approach named as Continual Learning with Efficient Architecture Search, or CLEAS in short. CLEAS works closely with neural architecture search (NAS) which leverages reinforcement learning techniques to search for the best neural architecture that fits new task. In particular, we design a neuron-level NAS controller that decides which old neurons from previous tasks should be reused (knowledge transfer), and which new neurons should be added (to learn new knowledge). Such a fine-grained controller allows finding very concise architecture that can fit each new task well. Meanwhile, since we do not alter the weights of the reused neurons, we perfectly memorize the knowledge learned from previous tasks. We evaluate CLEAS sequential classification tasks, and the results demonstrate that CLEAS outperforms other state-of-the alternative methods, achieving higher classification accuracy while using simpler neural architectures.
numerous
INTRODUCTION
Continual learning, or lifelong learning, refers to the ability of continually learning new tasks and also performing well on learned tasks. has attracted enormous attention in Al as it mimics human learning process constantly acquiring and accumulating knowledge throughout their lifetime (Parisi
al., 2019). Continual learning often works with deep neural networks (Javed & White 2019
Nguyen et al. 2017 Xu & Zhu 2018) as the flexibility in network design can effectively allow knowledge transfer and knowledge acquisition. However, continual learning with neural networks usually faces three challenges. The first one is overcome the so-called catastrophic forgetting problem (Kirkpatrick et al. 2017), which states that the network may forget what has been learned on previous tasks. The second one is to effectively adapt the current network parameters architecture to fit a new task, and the last one is to control the network size so as not to generate an overly complex network.
In continual learning, there are two main categories of strategies that attempt to solve the aforemen- tioned challenges. The first category example, (Rebuffi et al. 2017 Lopez-Paz & Ranzato 2017; Aljundi et al. (018)) replay some old samples with the new task samples and then learn a new network from the combined training set. The drawback is that they typically require a memory system that stores past data. (Kirkpatrick et al., 2017: Liu al. 2018) employ some regularization terms prevent the re-opiimieed parameters from deviating too much from the previous ones. Approaches using fixed network architecture, however, cannot avoid fundamental dilemma they must either choose to retain good model perfor- mances learned tasks, leaving little room for learning new tasks, compromise the learned model performances to allow learning new tasks better.
train all tasks within a network with fixed capacity. For
To overcome such dilemma, the second category is to expand the neural networks dynamically (Rusu et al. 2016 Yoon et al. 2018 Xu & Zhu 2018). They typically fix the parameters of the old neurons (partially fully) in order eliminate the forgetting problem, and also permit adding neurons to adapt to the learning of a new task. In general, expandable networks can achieve better model performances all tasks than the non-expandable ones. However, new issue appears: expandable

---

Under review as a conference paper at ICLR 2021
networks can gradually become overly large or complex, which may break the limits of the available computing resourees and/or lead to over-fitting.
In this paper, aim to solve the continual learning problems by proposing new approach that only requires minimal expansion of a network so as to achieve high model performances on both learned tasks and the new task. At the heart of our approach we leverage Neural Architecture Search (NAS) to find a very concise architecture to fit each new task. Most notably, we design NAS to provide a neuron-level control. That is, NAS selects two types of individual neurons compose new architecture: (1) a subset of the previous neurons that are most useful to modeling the new task; and (2) a minimal number of new neurons that should be added. Reusing part of the previous neurons allows efficient knowledge transfer; and adding new neurons provides additional room for learning new knowledge. Our approach is named as Continual Learning with Efficient Architecture Search, or CLEAS in short. Below are the main features and contributions of CLEAS.
CLEAS dynamically expands the network to adapt to the learning of new tasks and uses NAS determine the new network architecture;
CLEAS achieves zero forgetting of the learned knowledge by keeping the parameters of the previous architecture unchanged;
NAS used in CLEAS able to provide a neuron-level control which expands the network minimally. This leads to effective control of network complexity;
The RNN-based controller behind CLEAS is using an entire network configuration (with all neurons) as a state. This state definition deviates from the current practice in related problems that would define a state as an observation of a single neuron. Our state definition leads to improvements of 0.31%, %, 0.29% and 0.75% on three benchmark datasets.
If the network a convolutional network (CNN), CLEAS can even decide the best filter size that should be used in modeling the new task. The optimized filter size can further improve the model performance.
We start the rest of the paper by first reviewing the related work in Section 2
Then we detail our Experimental evaluations and the results are presented in Section 4
CLEAS design in Section 3
2 RELATED WORK
Continual Learning Continual learning is often considered as an online learning paradigm where new skills or knowledge are constantly acquired and accumulated. Recently, there are remarkable advances made in many applications based on continual learning: sequential task processing (Thrun 1995), streaming data processing (Aljundi et al. 2019), melf-management of resources (Parisi et al. 2019 Diethe et al. 2019), etc. A primary obstacle in continual learning, however, is the catastrophic forgetring problem and many previous works have attempted to alleviate it. We divide them into two categories depending on whether their networks are expandable.
The first category uses a large network with fixed capacity. These methods try to retain the learned knowledge by either replaying old samples (Rebuthi et al., 2017 Rolnick al.) 2019 Robins 1995) or enforcing the learning with regularization terms (Kirkpatrick et al., 2017 Lopez-Paz & Ranzato 2017 Liu et al.] 2018 Zhang al., 2200)). Sample replaying typically requires a memory system which stores old data. When learning new task, part of the old samples are selected and added the training data. As for regularized learning, a representative approach is Elastic Weight Consolidation (EWC) (Kirkpatrick et al. (017) which uses the Fisher information matrix to regularize the optimization parameters so that the important weights for previous tasks are not altered too much
Other methods like PLopez-Paz. & Ranzato 2017 Liu et al., 2018 Zhang et al.) 2020) also address the optimization direction of weights to prevent the network from forgetting the previously learned knowledge. The major limitation of using fixed networks is that it cannot properly balance the learned tasks and new tasks, resulting in either forgetting old knowledge or acquiring limited new knowledge.
To address the above issue, another stream of works propose to dynamically expand the network, providing more room for obtaining new knowledge. For example, Progressive Neural Network (PGN) (Rusu et al. 2016) allocates fixed number of neurons and layers to the current model for a new task. Apparently, PGN may end up generating an overly complex network that has high redundancy and it can easily crash the underlying computing system that has only limited resources.
2

---

Under review as a conference paper at ICLR 2021
Another approach DEN (Dynamically Expandable Network) (Yoon et al. 2018 partially mitigates the issue of PGN by using group sparsity regularization techniques. It strategically selects some old neurons to retrain, and adds new neurons only when necessary. However, DEN can have the forgetting problem due to the retraining of old neurons. Another drawback is that DEN has very sensitive hyperparameters that need sophisticated tuning. Both of these algorithms only grow the network and do not have a neuron level control which is a significant departure from our work. Most recently, novel method RCL (Reinforced Continual Learning) (Xu & Zhu 2018) also employs NAS to expand the network and it can further decrease model complexity. The main difference between RCL and CLEAS is that RCL blindly reuses all the neurons from all of the previous tasks and only uses NAS to decide how many new neurons should be added. However, reusing all the old neurons has two problems. First, it creates lot of redundancy in the new network and some old neurons may even be misleading and adversarial; second, excessively many old neurons reused in the new network can dominate its architecture, which may significantly limit the learning ability of the new network. Therefore, RCL does not really optimize the network architecture, thus it is unable to generate an efficient and effective network for learning a new task. By comparison, CLEAS designs fine-graine NAS which provides neuron-level control. It optimizes every new architecture by determining whether to reuse each old neuron and how many new neurons should be added to each layer.
Neural Architecture Search NAS is another promising research topic in the Al community. It employs reinforcement learning techniques to automatically search for a desired network architecture for modeling specific task. For instance, Cai et al. (Cai et al. 2018) propose EAS to discover superb architecture with a reinforced meta-controller that can grow the depth or width of a network; Zoph al. (Zoph & Le| (016)) propose an RNN-based controller generate the description of a network, and the controller is reinforced by the predicting accuracy of a candidate architecture. Pham et al. (Pham al., (018)) propose extension of NAS, namely ENAS, to speed up training processing by forcing all child networks to share weights. Apart from algorithms, NAS also has many valuable applications such as image classification (Real et al. 20199) Radosavovic al. 2019), video segmentation (Nekrasov et al. 2020). text representation (Wang et al. (2019) and etc. Hence, NAS is a demonstrated powerful tool and especially useful continual learning scenarios when one needs to determine what is good architecture for the new task.
3
METHODOLOGY
There two components in the CLEAS framework: one is the lask network that continually learns sequence of tasks; and the other is controller network that dynamically expands the task network. The two components interact with each other under the reinforcement learning context the task network sends the controller a reward signal which reflects the performance of the current architecture design; the controller updates its policy according to the reward, and then generates a new architecture for the task network to test its performance. Such interactions repeat until good architecture is found. Figure [ illustrates the overall structure of CLEAS. On the left is the task network, depicting an optimized architecture for task t 1 (it is using gray and pink neurons) and a candidate architecture for task t. They share the same input neurons but use their own output neurons. Red circles are newly added neurons and pink ones reused neurons from task t 1 (or any previous task). To train the network, only the red weights that connect new-old or new-new neurons are optimized. On the right the controller network which implements an RNN. It provides a neuron-level control generate description of the task network design. Each blue square is an RNN cell that decides to use or drop a certain neuron in the task network.
output
configuration
Action
Nidden layers
reward
State
Input
controllor network
Figure 1: The framework of CLEAS.
3

---

Under review as a conference paper at ICLR 2021
3.1 NEURAL ARCHITECTURE SEARCH MODEL
Task Network The task network can be any neural network with expandable ability, for example, fully connected network or CNN, etc. We use the task network to model sequence of tasks. Formally, suppose there are T tasks and each has a training set D; {(1,, Ji) validation set Vi = {(x,, y/ y1)) Mr and a test set T. = { yi) y:)} 1=11 for I = 1,2,...,5 T. We denote by A, the network architecture that is generated to model task L. Moreover, we denote A, = (NN, W:) where N₂ are the neurons or filters used the network and W are the corresponding weights. We train the first task with basic network A1 by solving the standard supervised learning problem
W1 arg min L1 (W1: D1),
(1)
W1
where 11 is the loss function for the first task. For the optimization procedure, we use stochastic gradient descent (SGD) with constant learning rate. The network is trained till the required number of epochs or convergence is reached.
When task (t > 1) arrives, for every task k < we already know its optimized architecture Aş and parameters W Now we use the controller to decide network architecture for task t. Consider a candidate network A, (N, W.). There are two types of neurons in Ny: Nodd are used neurons from previous tasks and Nr = N, \ Nyold are the new neurons added. Based on this partition, the weights W, can be also divided into two disjoint sets: Wold are the weights that connect only used neurons, and Wnew = W; \ Wolld are the new weights that connect old-new or new-new neurons. Formally, Nopld {n N, there exists k such that n NN and Woold {w E We there exists n2 E Nuli such that w connects f1, n2). The training procedure for the new task is to only optimize the new weights Wmew and leave Wold unchanged, equal to their previously optimized values Wild Therefore, the optimization problem for the new task reads
W new = -arg min (,W,, Wwot" Dz). Whow
Then we set We = (W Whew) Finally, this candidate network A, with optimized weights W, is tested on the validation set V₁. The corresponding accuracy and network complexity is used to compute reward R (described in Section 3.2 The controller updates its policy based on R and generates a new candidate network Ai to repeat the above procedure. After enough such interactions, the candidate architecture that achieves the maximal reward is the optimal one for task f, ie. A,- = (Ne, z), where N2 finally denotes the neurons of the optimal architecture.
Controller Network The goal of the controller is to provide neuron-level control that can decide which old neurons from previous tasks can be reused, and how many new neurons should be added. In our actual implementation, we assume there is large hyper-network for the controller to search for a task network. Suppose the hyper-network has layers and each layer i has maximum of u; neurons. Each neuron has two actions, either "drop" or "use" (more actions for CNN, to be described later). Thus, the search space for the controller would be 2n where n = E.= u, is the total number of neurons. Apparently, it is infcasible to enumerate all the action combinations and determine the best one. To deal with this issue, we treat the action sequence as a fixed-length string a1... = a1, Q2,..... on that describes a task network. We design the controller as an LSTT network where each cell controls one a; in the hyper-network. Formally, we denote by I (1:::|$1:n: policy function of the controller network as
$1::; 0c) the
T T(ain| 0c) = P(a (1:m: 0.) II P(ay(s1n:0e))
The state $imn is sequence that represents one state; the output is the probability of task network described by aimi and 0g denotes the parameters of the controller network. At this point we note that model is departure from standard models where states are considered individual sj and an episode is comprised of $1:n- In our case we define $1m, as one state and episodes are created by starting with different initial states (described below).
Recall that in [. the two components in CLEAS work with each other iteratively and there are H 2 such iterations where H is the number of episodes created and 2f the length of each episode.
4

---

Under review as a conference paper at ICLR 2021
) The initial state sim is either generated randomly copied from the terminal state s/ of the previous episode. The controller starts with some initial 0z. For any u = 1,2..... 1.2..... 2, the controller generates the most probable task network specified by di.. from si. by following LSTM. To this end, we use the recursion aj = [(5j. h where h is the hidden vector and f standard LSTM equations to generate ain from si. Let us point out that our RNN application aj = f( j. hi-1)) differs from Action @i.. is obtained from di by selecting the maximum probability value for each j. J n. The task trains this task network, evaluates on the validation set and returns reward R". We then construct g+++] from the previous neuron action aj together with the layer index as b for each 1 S S n. More concretely, 8y =a @ by where ay. by have been one-hot encoded, and @ is the concatenation operator. Finally, a new network architecture 1 is generated from $ At the end of each episode, the controller updates its parameter 0g by a policy gradient algorithm. After all H. U total iterations, the task network that achieves the maximum reward used for that task.
Consider episode (s1m R² , si.. σ] R² ..... L a 1im RM,, ,
the standard practice that uses a = h h 1).
The choice for treating the state as Si:2 and not s, has the following two motivations. In standard NAS type models after updating s; the network is retrained. This is intractable in our case as the number of neurons is typically large. For this reason we want to train only once per $1:n- The second reason is related and stems from the fact that the reward is given only at the level of $1mh For this reason it makes sense to have 81: as the state. This selection also leads to computational improvements as is attested later Section
CLEAS-C for CNN The design of CLEAS also works for CNN with fixed filter sizes where one filter corresponds to one neuron. However, we know that filter sizes in a CNN can have huge impact on its classification accuracy. Therefore, we further improve CLEAS so that it can decide the best filter sizes for each task. In particular, we allow a new task to increase the filter size by one upon the previous task. For example, a filter size 3 x used in some convolutional layer in task t can become 4 x 4 in the same layer in task t. Note that for one task all the filters in the same layer must use the same filter size, but different layers can use different filter sizes.
We name the new framework as CLEAS-C. There are two major modifications to CLEAS-C. First, the output actions in the controller are now encoded by 4 bits and their meanings are "only use," "use & "ttend,""olly drop" and "drop & extend" (see Fig. 2 Note that the extend decision is made at the neuron layer, but there has to be only one decision the layer level. To this end, we apply simple majority voting of all neurons at a layer to get the layer level decision. The other modification regards the training procedure of the task network. The only different case we should deal with is how to optimize a filter (e.g. 4 X 4) that is extended from a previous smaller filter (e.g. 3 x 3). Our solution is to preserve the optimized parameters that are associated with the original smaller filter (the 9 weights) and to only optimize the additional weights (the 16 9 = 7 weights). The preserved weights are placed in the center of the larger filter, and the additional weights are initialized as the averages of their surrounding neighbor weights.
usesextend usedextend uscaextend only use drop&cxtend
only use
only drop
only use
e
0
Q
0
D
0
Action
D
Controlk lor
State
layer
1-1
loyer 1
layer i+1
Figure 2: The controller design for convolutional networks.
3.2 TRAINING WITH REINFORCE
Lastly, we present the training procedure for the controller network. Note that each task has an independent training process so we drop subscript t here. Within an episode, each action string af.... represents task architecture and after training gets validation accuracy A". In addition to accuracy,
5

---

Under review as a conference paper at ICLR 2021
also penalize the expansion of the task network in the reward function, leading the final reward
Rd R(aim)) = A(ai..) aC(a
(4)
where C is the number of newly added neurons, and ar is a trade-off hyperparameter. With such episodes we train
J( E. Ba - by using REINFORCE. We use an exponential moving average of the previous architecture accuracies the baseline.
We summarize the key steps of CLEAS in Algorithm | where H is the number of iterations, 24 is the length of episodes, and p the exploration probability. We point out that we do strictly follow the usual e-greedy strategy; an exploration step consists of starting an epoch from a completely random state as opposed perturbing an existing action.
Algorithm 1: CLEAS.
Input: A sequence of tasks with training sets (Di, D2, Dr validation sets {Vi, V2,,.,, Vr) Output: Optimized architecture and weights for each task: Ag = (N,, W.) for t. = 1,2, .,. for f 1.2..... do
if = 11 then
Train the initial network A₁ on D1 with the weights optimized as W1;
else
Generate initial controller parameters tz:
for h = 1.2..... 7. do
/* A new episode */ Bernouli(p)): w or h then
/* Exploration */
Generate random state string $ but keep layer encodings fixed;
else
44+1 Set initial state string s $1n i.e. the last state of previous episode (h 1);
for u= 2, ... 24 do
Generate the most probable action string d.. from s by the controller;
Configure the task network as A based on ai. and train weights W on Dz;
Evaluate A" with trained W on V and compute reward R"; x++ from a.... and bi:n where by... is the layer encoding; Construct $1:7
Update 0, by REINFORCE using (si.n> al. R2 ,t.,,...................... s.. a4.m 184 $1
Store A (N W") where arg max,, R and R max, R"; Store A₁ A where h arg max, Rh:
4 EXPERIMENTS
We evaluate CLEAS and other sate-of-the-ard continual learning methods MNIST and CIFAR-100 datasets. The key results delivered are model accuracies, network complexity and training time, All methods are implemented in Tensorflow and ran on a GTX1080Ti GPU unit.
4.1 DATASETS AND BENCHMARK ALGORITTMM
We use three benchmark datasets follows. Each dataset is divided into T = 10 separate tasks. MNIST-associated tasks are trained by fully-connected neural networks and CIFAR- 100 tasks are trained by CNNs.
(a) MNIST Permutation (Kirkpatrick al. 2017): Ten variants the MNIST data, where each task transformed by different (among tasks) and fixed (among images in the same task) permutation of pixels. (b) Rotated MNIST (Xu & Zhu 2018): Another ten variants MNIST, where each task is rotated by different and fixed angle between 0 to 180 degree. (c) Incremental CIFAR-100 (Rebuffi et al. 2017): : The original CIFAR- 100 dataset contains 60,000 x 32 colored images that belong to 100 classes. We divide them into 10 tasks and each task contains 10 different classes and their data.

---

Under review as a conference paper at ICLR 2021
We select four other continual learning methods to compare. One method (MWC) uses fixed network architecture while the other three use expandable networks.
extent of correlation between consecutive tasks uses regularization terms to prevent large deviation of the network weights when re-optimized. (2) PGN: Expands the task network by adding a fixed number of neurons and layers (Rusu et al. 2016). (3) DEN: Dynamically decides the number of new neurons by performing selective retraining and network split (Yoon al. 2018). (4) RCL: Uses NAS to decide the number of new neurons. It also completely eliminates the forgetting problem by holding the previous neurons and their weights unchanged (Xu & Zhu 2018).
(1) MWC: An extension of EWC (Kirkpatrick et al. 2017). By assuming
2018 to use three-layer fully-connected network. We start with 784-312-128-10 neurons with RELU activation for the first task. For CIFAR- 100, we develop modified version of LeNet (LeCun et al., 1998) that has three convolutional layers and three fully-connecto and 4 x 4 and stride of 1 per layer. Besides, to fairly compare the network choice with (Xu & Zhu 2018 Yoon et al. 2018), we set: u, 1000 for MNIST and u, = 128 for CIFAR- 100. We also use H 200 and U 1. The exploration probability is set to be 30%. We select the RMSProp optimizer for REINFORCE and Adam for the training task.
For the two MNIST datasets, we follow (Xu & Zhu
layers. We start with 16 filters each layer with sizes of 3 x 3, 3 x
We also implement version with states corresponding to individual neurons where the controller is following aj = f(aj; 1,hh)) We configure this version under the same experimental settings of CLEAS and test it the three datasets. The results show that compared to CLEAS, this version exhibits an inferior performance of -0.31 %, -0.29 % -0.75% % in relative accuracy, on the three datasets, respectively. Details can be found in Appendix.
4.2
EXPERIMENTAL RESULTS
PEN
DEN
RCL
Improvement agsinstt EWC(%)
200-
150-
300-
1970
0-
MNIST permuuaaioo
Rutated MINIST
CIFAR-100
MNSST persratacins
Rotated SENISTT datasets
CIFAR-100
datasets
Figure 3: Average test accuracy across all tasks.
Figure 4: Average number of parameters.
Model Accuracy We first report the averaged model accuracies among all tasks. Fig shows the relative improvements of the network -ewand-bxaa methods against EWC (numbers on the top are their absolute accuracies). We clearly observe that methods with expandability can achieve much better performance than MWC. Furthermore, we see that CLEAS outperforms other methods. The average relative accuracy improvement of CLEAS vs RCL (the otate-of-the-arr method and the second best performer) is 0.21 %, 0.21% % and 6.70%, respectively. There are two reasons: (1) we completely overcome the forgetting problem by not altering the old neurons/filters; (2) neuron-level control can precisely pick useful old neurons as well as new neurons to better model each new task.
Network Complexity Besides model performance, we also care about how complex the network is when used to model each task. We thus report the average number of model weights across all tasks in Fig. 4 First, no surprise to see that MWC consumes the least number of weights since its network non-expandable. But this also limits its model performance. Second, among the other four methods that expand networks we observe CLEAS using the least number of weights. The average relative complexity improvement of CLEAS RCL is 29.9 %. 19.0% and 51.0% reduction, respectively. It supports the fact that our NAS using neuron-level control can find a very efficient architecture to model every new task.
Network Descriptions We visualize some examples of network architectures the controller gener- ates. Fig. Fillustrates four optimal configurations (tasks 2 to 5) of the CNN used to model CIFAR- 100. Each task uses three convolutional layers and each square represents filter. A white square means it is not used by the current task; a red square represents it was trained by some earlier task and now reused by the current task; light yellow square means it trained before but not reused; and
7

---

Under review as a conference paper at ICLR 2021
dark yellow square depicts a new filter added. According the figure, we note that CLEAS tends to maintain a concise architecture all the time. As the task index increases it drops more old filters and only reuses small portion of them that useful for current task training, and it is adding fewer new neurons.
CLEAS-C We also test CLEAS-C which decides the best filter sizes for CNNs. the CIFAR- 100 experiment, CLEAS uses fixed filter sizes 3 x 3. 3 and 4 4 in its three convolutional layers. By comparison, CLEAS-C starts with the same sizes but allows each task to increase the sizes by one. The results show that after training the 10 tasks with CLEAS-C the final sizes become 4 8 x 8, and 8 x 8. It achieves much higher accuracy of 57.4% than CLEAS (66.9%) ), i.e. a 0.7 % improvement. It suggests that customized filter sizes can better promote model performances. On the other hand, complexity of CLEAS-C increases by 92.6%.
rask
tak's
Dearon Index
Figure 5: Examples of CNN architectures for CIFAR-100.
Neuron Allocation We compare CLEAS to RCL neuron reuse and neuron allocation. Fig. visualizes the number of reused neurons (yellow and orange for RCL; pink and red for CLEAS) and new neurons (dark blue for both methods). There are observations. On one hand, CLEAS successfully drops many old neurons that are redundant or useless, ending up maintaining much simpler network. On the other hand, we observe that both of the methods recommend similar number of new neurons for each task. Therefore, the superiority of CLEAS against RCL lies more on its selection of old neurons. RCL blindly reuses all previous neurons.
number neareas
runtime (x10A))
#00-
100
MNIST partaaato
Ruteed SINIST datasets
CIFAK 100
Figure 6: Neuron allocation for MNIST Permulation.
Figure 7: Training time
Training Time We also report the training time in Fig 7 It is expected that CLEAS' running time is on the high end due to the neuron-level control that results in using a much longer RNN for the controller. On the positive note, the increase in the rumning time is not substantial.
Hyperparameter Sensitivity We show the hyperparameter analysis in Appendix. The observation that the hyperparameters used CLEAS are not as sensitive as those of DEN and RCL. Under all hyperparameter settings CLEAS performs the best.
CONCLUSIONS
We have proposed and developed novel approach CLEAS tackle continual learning problems. CLEAS is a network-expandern architecture for each task. NAS able to provide a neuron-level control that decides the reusing of old neurons and the number of new neurons needed. Such fine-grained control can maintain very concise network through all tasks. Also, we completely eliminate the catastrophic forgetting problem by never altering the old neurons and their trained weights. With demonstration by means of the experimental results, we note that CLEAS can indeed use simpler networks to achieve yet higher model performances compared to other alternatives. In the future, plan to design more efficient search strategy or architecture for the controller such that it can reduce the runtime while not compromising the model performance network complexity.
approach that uses NAS to dynamically determine the optimal
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision, pp. 139-154, 2018.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In The IEEE Conference on Computer Vision and Pattern Recognition, June 2019.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Thirty-Second AAAI conference on artificial intelligence, 2018.
Tom Diethe, Tom Borchert, Eno Thereska, Borja de Balle Pigem, and Neil Lawrence. Continual learning in practice. arXiv preprini arXiv:1903.05202 2019.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, pp. 1818- 1818-1828, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Baabbkka-aarinkka, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114 (13):3521-3526, 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-2322 1998.
Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez. and Andrew D Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In 2018 24th International Conference on Puttern Recognition, pp. 2262-2222222222 2268.. IEEE, 2018.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Vladimir Nekrasov, Hao Chen, Chunhua Shen, and Ian Reid. Architecture search of dynamic cells for semantic video segmentation. In The IEEE Winter Conference on Applications of Computer Vision, March 2020.
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint 1110::1110.10220 2017.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 2019.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International Conference on Machine Learning, pp. 4095-4104, 2018.
Ilija Radosavovic, Justin Johnson, Saining Xie, Wan- Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1882-1890, 2019.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Lee. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
SyliestreeAAlise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2): 123-146, 1995.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual leaming. In Advances in Neural Information Processing Systems, pp. 348-358, 2019.

---

Under review as a conference paper at ICLR 2021
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.00 04671, 2016.
Scbastian Thrun. A lifelong learning perspective for mobile robot control. In Intelligent Robots and Systems, pp. 201-214. Elsevier, 1995.
Yujing Wang, Yaming Yang, Yiren Chen, Jing Bai, Ce Zhang, Guinan Su, Xiaoyu Kou, Yunhai Tong, Mao Yang. and Lidong Zhou. Textnas: neural architecture search space tailored for text representation. arXiv preprint 19I. 2019.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Processing Systems, pp. 899-908, 2018.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representation, 2018
Jie Zhang, Junting Zhang, Shalini Ghosh, Dawei Li, Jingwen Zhu, Heming Zhang, and Yalin Wang. Regularize, expand and compress: Nonexpansive continual learning. In The IEEE Winter Conference on Applications of Computer Vision, pp. 854-862, 2020.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:6611.0157., 2016.
10


Reviews:
[{"review_id": "wa8gYEDg1tZ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper falls into a class of continual learning methods which accommodate for new tasks by expanding the network architecture, while freezing existing weights. This freezing trivially resolves forgetting. The (hard) problem of determining how to expand the network is tackled with reinforcement learning, largely building upon a previous approach (reinforced continual learning, RCL). Apart from some RL-related implementation choices that differ here, the main difference to RCL is that the present method learns a mask which determines which neurons to reuse, while RCL only uses RL to determine how many neurons to add. Experiments demonstrate that this allows reducing network size while significantly improving accuracy on Split CIFAR-100. The runtime is, however, increased here.\n\nThe obvious downside to this approach (and to RCL) is the potentially very increase in runtime stemming from RL, which requires fully training many networks just to solve one additional task. This renders the approach impractical for large models; consistent with this, the authors only study models of modest dimensions.\n\nThe present paper is mostly an extension of RCL. Thus, limited novelty is its main weakness. But the experimental gains are significant, the extension to RCL is meaningful and the paper is easy to follow.\n\nI leave some questions and comments for the authors below:\n- Did the authors re-implement the RCL objective using their own RL algorithmic choices? The RL implementation in the RCL paper differs in many ways (for example, actor-critic learning is used), which leaves some doubt as to whether (part of) the benefits stem from the changes to RL, or from the actual neuron-level control proposed here. I would like to hear the authors' reply to this point.\n\n- \"We point out that we do not strictly follow the usual $\\epsilon$-greedy strategy; an exploration step consists of starting an epoch from a completely random state as opposed to perturbing an existing action.\"\nIsn't this still $\\epsilon$-greedy? I do have a question on this point, though: is the exploration probability annealed? Picking random states (with a high probability of 30%) seems very extreme. Can the authors provide learning curves for the controller?\n\n- Permuted MNIST is admittedly not a great dataset to study transfer learning. Can the authors repeat the analysis of Figure 6 on Split CIFAR-100? Is allocation decreasing as new tasks are learned, suggesting that some form of transfer is occurring at the architecture search level, or does it remain roughly constant, on that dataset? Still on transfer learning: it would be good to report the actual training curve of the resulting nework (not the controller) over tasks and investigate whether learning becomes faster as more CIFAR splits are learned, compared to a baseline which does not benefit from architecture search. This would make the paper much stronger, in my opinion.\n\n- MWC: how is this method different from standard EWC? I couldn't find any explanation.\n\n- While EWC is a relevant baseline, it would be good to report as well the performance of simple joint multitask training as an upper baseline.\n\n- \"The results show that compared to CLEAS, this version exhibits an inferior performance of -0.31%, -0.29%, -0.75% in relative accuracy\"\nWhat is relative accuracy? Relative to MWC, as in Fig. 3? In any case, as these improvements are somewhat modest, how does runtime compare for the two options?\n\n- Regarding training time, Fig. 7: while runtime in seconds is important, can the number of (controller network) training iterations of RCL vs. CLEAS be provided as well? \n\n- Readability could be improved by using \\citep{} instead of \\citet{}.\n\n--\nPost-rebuttal edit: I read the authors' reply and thank them for the clarifications. I maintain my score of 6.", "extra_scores": {}, "extra_text": {}}, {"review_id": "X2qwseVr0pf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors tackle the problem of efficient architecture search for continual learning. They propose to use reinforcement learning-based neural architecture search to efficiently expend layers, select which neurons trained on previous tasks to reuse and which new neurons to train from scratch. They validate the approach using standard experiments on MNIST and Cifar-100. The paper is well written, clearly explaining each component and the reasons for their choices.\n\nMy main concern is about efficiency, the choice of RNN generating architecture propositions and using the performance of each proposition as a reward can't really be called efficient as it requires multiple (200 in the article) full trainingC on each task.  It would be interesting to see a training time comparison of the proposed method against the other baselines (e.g. measuring the training time as (i)wall-clock time or (ii)total number of parameter updates including the NAS part of the training).\n\nQuestions:\n- If my understanding is correct, the RNN is trained on the validation set (section 3.1). What is the size of this validation set for each task and what is the protocol to prevent overfitting? Are the HP tuned on the same set?\n- For the controller network training, why using an additional parameter for the exploration instead of directly following the policy parametrized by $\\theta_c$ ?\n- Why using LeNet architecture as the backbone architecture instead of a more recent and commonly used model in CL (e.g. smaller Resnets-18 or 34)?\n\nI would also like to see how a simple baseline like random search perform, using the same number of models as the NAS approach (i.e. training 200 networks sampled from the search space used by the NAS procedure on each task).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IwrYLYHtaNd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper presents a new method for continual learning by examining for each new task the NN created so far, deciding for each neuron whether to keep it (with the same weights), and how many new neurons should be added. This is done by a controller networks, consisting of LSTM units.\n\nPros: They demonstrate improved performance in comparison with 3 previously proposed methods, in particular MWC, an improvement (invented and tested by whom?)  over EWC of (Kirkpatrick et al., PNAS 2017), while using about the same number of parameters.\n\nCons: Their method is quite complicated and somewhat opaque to me, and it requires substantially more compute, see Fig. 7 (although it is stated on p. 8 that the „increase in running time is not substantial“; I do not understand that). Also the controller network needs to be apparently be trained for the whole task sequence. If this is correct, the full task sequence has to be known in advance.\n\nI do not understand the statement in the Discussion „we completely eliminate the catastrophic forgetting problem by never altering the old neurons and their trained weights“. If one removes some neurons in a network (only feedforward networks are considered), one changes the input to downstream neurons. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6gxj0JxV-v1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Efficient Architecture Search for Continual Learning \n- Summary\nThis paper aims to solve continual learning problems with minimal expansion of network parameters. The authors propose Continual Learning with Efficient Architecture Search (CLEAS), which is equipped with a neuron-level NAS controller. The controller selects 1) the most useful previous neurons to model the new task (knowledge transfer) and 2) a minimum number of additional neurons. The experimental results show that the proposed method outperforms state-of-the-art methods on several continual learning benchmark tasks such as MNIST Permutation, Rotation MNIST, and Incremental CIFAR-100.\n\n- Strong points\n\t1. The proposed framework selectively comprises neurons for new tasks, and training only newly added weights enables zero-forgetting of previously learned tasks.\n\t2. The experimental results showed performance improvements compared with the previous algorithms (PGN, DEN, RCL) while preserving or reducing the number of parameters, especially in the case of CIFAR-100.\n\n- Weakness\n\t1. From the perspective of real-world problems, the neuron-level decision through the RNN controller costs a long training time. The authors in this paper demonstrated small-scale neural networks such as 3-layers. Although the authors mentioned like “On the positive note, the increase in the running time is not substantial.”, I’m wondering it is plausible, and I couldn’t find what the running time in the figure 7 means.\n\t2. It is not clear for me the rationale behind the sequential states of neurons and the authors’ claim that “This state definition deviates from the current practice in related problems that would define a state as an observation of a single neuron.”. Also, what does “standard model” mean in page 4? Is the sequential state invariant to permutations in neuron topology? \n\n- Questions\n\t1. How did you set the maximum of u_i neurons in your experiments?\n\t2. Do you have any plan to publish the source codes for reproducibility? \n\t3. Please address and clarify the cons above.\n\n- Additional feedback\n\t- Typo: wrong citation for ENAS (line 7, “Neural Architecture Search” paragraph in section 2)\n\t- I would like to recommend denoting the accuracy of MWC in section 4.2.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a network-expandable approach to tackle NAS in the continual learning setting. More specifically, they use a RNN controller to decide which neurons to use (for a new task) and the additional capacity required (i.e., number of new neurons to add). This work can be viewed as an extension of RCL and as such suffers from the large runtime. This was a concern for most reviewers. While reviewers highlighted the gains in the experiments conducted, several questions remained regarding the efficiency of the proposed approach and how it compares to other strategies. The practical relevance of the proposed approach was also a concern as its application requires to restrict it to models of modest size.

Decision: Reject