Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Efficient Architecture Search for Continual Learning 
- Summary
This paper aims to solve continual learning problems with minimal expansion of network parameters. The authors propose Continual Learning with Efficient Architecture Search (CLEAS), which is equipped with a neuron-level NAS controller. The controller selects 1) the most useful previous neurons to model the new task (knowledge transfer) and 2) a minimum number of additional neurons. The experimental results show that the proposed method outperforms state-of-the-art methods on several continual learning benchmark tasks such as MNIST Permutation, Rotation MNIST, and Incremental CIFAR-100.

- Strong points
	1. The proposed framework selectively comprises neurons for new tasks, and training only newly added weights enables zero-forgetting of previously learned tasks.
	2. The experimental results showed performance improvements compared with the previous algorithms (PGN, DEN, RCL) while preserving or reducing the number of parameters, especially in the case of CIFAR-100.

- Weakness
	1. From the perspective of real-world problems, the neuron-level decision through the RNN controller costs a long training time. The authors in this paper demonstrated small-scale neural networks such as 3-layers. Although the authors mentioned like “On the positive note, the increase in the running time is not substantial.”, I’m wondering it is plausible, and I couldn’t find what the running time in the figure 7 means.
	2. It is not clear for me the rationale behind the sequential states of neurons and the authors’ claim that “This state definition deviates from the current practice in related problems that would define a state as an observation of a single neuron.”. Also, what does “standard model” mean in page 4? Is the sequential state invariant to permutations in neuron topology? 

- Questions
	1. How did you set the maximum of u_i neurons in your experiments?
	2. Do you have any plan to publish the source codes for reproducibility? 
	3. Please address and clarify the cons above.

- Additional feedback
	- Typo: wrong citation for ENAS (line 7, “Neural Architecture Search” paragraph in section 2)
	- I would like to recommend denoting the accuracy of MWC in section 4.2.
review_title: Solving continual learning problems with minimal expansion of network parameters.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors tackle the problem of efficient architecture search for continual learning. They propose to use reinforcement learning-based neural architecture search to efficiently expend layers, select which neurons trained on previous tasks to reuse and which new neurons to train from scratch. They validate the approach using standard experiments on MNIST and Cifar-100. The paper is well written, clearly explaining each component and the reasons for their choices.

My main concern is about efficiency, the choice of RNN generating architecture propositions and using the performance of each proposition as a reward can't really be called efficient as it requires multiple (200 in the article) full trainingC on each task.  It would be interesting to see a training time comparison of the proposed method against the other baselines (e.g. measuring the training time as (i)wall-clock time or (ii)total number of parameter updates including the NAS part of the training).

Questions:
- If my understanding is correct, the RNN is trained on the validation set (section 3.1). What is the size of this validation set for each task and what is the protocol to prevent overfitting? Are the HP tuned on the same set?
- For the controller network training, why using an additional parameter for the exploration instead of directly following the policy parametrized by $\theta_c$ ?
- Why using LeNet architecture as the backbone architecture instead of a more recent and commonly used model in CL (e.g. smaller Resnets-18 or 34)?

I would also like to see how a simple baseline like random search perform, using the same number of models as the NAS approach (i.e. training 200 networks sampled from the search space used by the NAS procedure on each task).

review_title: Correct paper that still need some work

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: This paper presents a new method for continual learning by examining for each new task the NN created so far, deciding for each neuron whether to keep it (with the same weights), and how many new neurons should be added. This is done by a controller networks, consisting of LSTM units.

Pros: They demonstrate improved performance in comparison with 3 previously proposed methods, in particular MWC, an improvement (invented and tested by whom?)  over EWC of (Kirkpatrick et al., PNAS 2017), while using about the same number of parameters.

Cons: Their method is quite complicated and somewhat opaque to me, and it requires substantially more compute, see Fig. 7 (although it is stated on p. 8 that the „increase in running time is not substantial“; I do not understand that). Also the controller network needs to be apparently be trained for the whole task sequence. If this is correct, the full task sequence has to be known in advance.

I do not understand the statement in the Discussion „we completely eliminate the catastrophic forgetting problem by never altering the old neurons and their trained weights“. If one removes some neurons in a network (only feedforward networks are considered), one changes the input to downstream neurons. 

review_title: A method for continual learning in feedforward neural networks that requires only moderate network growth.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper falls into a class of continual learning methods which accommodate for new tasks by expanding the network architecture, while freezing existing weights. This freezing trivially resolves forgetting. The (hard) problem of determining how to expand the network is tackled with reinforcement learning, largely building upon a previous approach (reinforced continual learning, RCL). Apart from some RL-related implementation choices that differ here, the main difference to RCL is that the present method learns a mask which determines which neurons to reuse, while RCL only uses RL to determine how many neurons to add. Experiments demonstrate that this allows reducing network size while significantly improving accuracy on Split CIFAR-100. The runtime is, however, increased here.

The obvious downside to this approach (and to RCL) is the potentially very increase in runtime stemming from RL, which requires fully training many networks just to solve one additional task. This renders the approach impractical for large models; consistent with this, the authors only study models of modest dimensions.

The present paper is mostly an extension of RCL. Thus, limited novelty is its main weakness. But the experimental gains are significant, the extension to RCL is meaningful and the paper is easy to follow.

I leave some questions and comments for the authors below:
- Did the authors re-implement the RCL objective using their own RL algorithmic choices? The RL implementation in the RCL paper differs in many ways (for example, actor-critic learning is used), which leaves some doubt as to whether (part of) the benefits stem from the changes to RL, or from the actual neuron-level control proposed here. I would like to hear the authors' reply to this point.

- "We point out that we do not strictly follow the usual $\epsilon$-greedy strategy; an exploration step consists of starting an epoch from a completely random state as opposed to perturbing an existing action."
Isn't this still $\epsilon$-greedy? I do have a question on this point, though: is the exploration probability annealed? Picking random states (with a high probability of 30%) seems very extreme. Can the authors provide learning curves for the controller?

- Permuted MNIST is admittedly not a great dataset to study transfer learning. Can the authors repeat the analysis of Figure 6 on Split CIFAR-100? Is allocation decreasing as new tasks are learned, suggesting that some form of transfer is occurring at the architecture search level, or does it remain roughly constant, on that dataset? Still on transfer learning: it would be good to report the actual training curve of the resulting nework (not the controller) over tasks and investigate whether learning becomes faster as more CIFAR splits are learned, compared to a baseline which does not benefit from architecture search. This would make the paper much stronger, in my opinion.

- MWC: how is this method different from standard EWC? I couldn't find any explanation.

- While EWC is a relevant baseline, it would be good to report as well the performance of simple joint multitask training as an upper baseline.

- "The results show that compared to CLEAS, this version exhibits an inferior performance of -0.31%, -0.29%, -0.75% in relative accuracy"
What is relative accuracy? Relative to MWC, as in Fig. 3? In any case, as these improvements are somewhat modest, how does runtime compare for the two options?

- Regarding training time, Fig. 7: while runtime in seconds is important, can the number of (controller network) training iterations of RCL vs. CLEAS be provided as well? 

- Readability could be improved by using \citep{} instead of \citet{}.

--
Post-rebuttal edit: I read the authors' reply and thank them for the clarifications. I maintain my score of 6.
review_title: Extension of reinforced continual learning (RCL)

Meta-review: The authors propose a network-expandable approach to tackle NAS in the continual learning setting. More specifically, they use a RNN controller to decide which neurons to use (for a new task) and the additional capacity required (i.e., number of new neurons to add). This work can be viewed as an extension of RCL and as such suffers from the large runtime. This was a concern for most reviewers. While reviewers highlighted the gains in the experiments conducted, several questions remained regarding the efficiency of the proposed approach and how it compares to other strategies. The practical relevance of the proposed approach was also a concern as its application requires to restrict it to models of modest size.

Decision: Reject