Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an approach for continual learning that builds dynamic network structures such that it minimizes catastrophic forgetting but maximizes forward transfer. The approach involves neural architecture search where an LSTM policy optimizes for selection of network layers (essentially their size) and their initialization with a reward that calculates forward improvement (from the architecture) over iterations as well as backward transfer (across tasks). The results on some standard (albeit toy-ish) CL datasets show that the approach mitigates catastrophic forgetting while also inducing forward transfer in tasks that can use it.
main_review: Pros 
- The paper is well written. Even though the approach is quite involved, the details are well explained and easy to follow along.
- Experimental results are strong and confirm that the approach can learn architectures for forward transfer while also minimizing forgetting. However, some recent related approaches and baselines are not discussed (see below).
- Ablations are insightful. I also liked the qualitative analysis in the supplementary of a sequence of architectures found by the agent.

Cons
- The datasets used are really small toy datasets, with only a maximum of 10 tasks in a sequence. While these are common datasets used in CL literature, it will be better to move beyond these for more impactful/useful results.
- A recent work [1] seems very closely related. It will be good to compare with this or at the least discuss differences with it. I also see additional baselines in that paper, such as RCL (Xu and Jhu, 2018) and DEN (Yoon et al., 2018) that are also relevant here.
- The approach seems to keep architectures for all previous tasks in memory which would give a prohibitively high memory requirement (order of number of tasks)

Questions for Authors
- The “reuse” action seems to require all parameters of previous tasks in memory. That will have an order of magnitude larger memory footprint than the baselines. Can you comment on this?
- The "new" action seems to double the number of parameters in a layer. This has the potential to blow up the number of parameters in the architecture. How is this managed? Is there a cap on max number of parameters?
- Please clearly characterize the memory requirement, in terms of both the number of parameters as well as the size of replay buffer, that the approach requires.
- Task representations seem to be computed by averaging all the representations of all instances in the task, regardless of their labels (see eq 1). This seems weak and uninformative. Can you comment on what similarities you hope to capture through such representations? Since this is the most important part of the input to the policy, it could be helpful to visualize this similarity for tasks to understand what it is capturing.
- Can you comment on the relevance of [1] and the baselines of RCL and DEN in the context of the proposed approach? 

[1] EFFICIENT CONTINUAL LEARNING WITH MODULAR NETWORKS AND TASK-DRIVEN PRIORS, ICLR 2021

limitations_and_societal_impact: Since the architecture required to solve each task could be expensive (in terms of memory and compute), the authors should potentially discuss this as a limitation.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper focuses on transfer of knowledge across tasks in a continual learning setting without catastrophic forgetting. The proposed method dynamically builds neural network structures. For each new task an agent samples a sequence of actions that integrates knowledge transfer from the old tasks to construct the continual learner. Old data are stored in a replay buffer, which are used for reward calculation rather than in training. The model thus incrementally learns a sequence of $N$ tasks. The continual learner is trained in a reinforcement learning setting where the reward is a combination of performance increase compared to a previous iteration and the forgetting rate of all previous tasks promoting the accuracy on all previous tasks.

main_review: The paper provides a clear definition of task-continual learning and class-continual learning, although the paper focuses on the easier problem of task-continual learning where at test time the task id is provided. 

The model exploits task similarities to enable the model to transfer knowledge across tasks. 
The task similarity is based on the current data of the new task and the data stored in the memory buffer of the old tasks and is computed as the combination of three classical similarity metrics applied on the representation of these data. The paper would benefit from adding a clear motivation of the way this similarity is computed and the intuition behind it. 

Now, the overview of the model and the training of the continual learner are described in text. For clarity, this text could be replaced by an algorithm in pseudocode. 

The work has value as transfer learning model where you learn to dynamically build your model for a new task exploiting properties of older learned models classified by a number of layer-wise discrete action labels (e.g., reuse, fuse, mask). These action labels are predicted by means of an LSTM that decides which action for which layer. The transfer learning capabilities of the proposed model are clearly shown in the experiments. 

The model is evaluated with several datasets commonly used in continual learning and the results of several metrics (accuracy, forgetting metric and transfer ability metric) are convincing that the claims of the paper are accomplished when compared with many baselines.  

Remarks to further improve the paper:
-	Line 122: Clarify better why the agent for a task $t$ is trained for $N$ iterations until it converges, where $N$ is the number of tasks.
-	Training time is giving in the supplementary material. An analysis of the computational complexity of the training procedure and ideas for reducing this complexity would strengthen the paper. 
-	Figure 1 (A): the font size of the legend could be improved; Figure 1 (B): More explanation in the caption would aid the reader to understand this figure early on when reading the paper.
-	Are the validation sets for reward computation released after publication of the paper? 

limitations_and_societal_impact: Yes, the authors have addressed the limitations of their work in the supplementary material.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes BNS  for task continual learning, which can dynamically adjust the network structure and parameter initialization method of each task by exploiting the similarities and differences of the tasks to achieve both objectives of Task-CL, overcoming catastrophic forgetting and transferring knowledge across tasks to improve the performance.
The contribution of the paper is to propose a knowledge transfer method across different tasks for continual learning.
main_review: 1. The paper address the problem of knowledge transfer across tasks in continual learning, which has received relatively little attention.  This problem is worth to be studied.
2. The paper formulates continual learning as a reinforcement learning problem and defines a novel RL setting.
3. The experiment results show the proposed method is effective.
4. The proposed method combines the structure search and element-wise action to better find the effective subnetwork for each task.
limitations_and_societal_impact: 1. The main concern is the RL usually needs large trial and error, and thereby low-efficient. The paper should give the learning cost.
2. The number of datasets is relatively small and hard to verify the effectiveness of continual learning.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies continual learning when the model is required to fit a sequence of tasks (with task identifiers) and maximize the average performance on all tasks. The goal is to prevent both catastrophic forgetting (CF), and improve forward knowledge transfer. It presents a neural architecture search (NAS) approach, namely BNS, to train an agent to design the new architecture for each new task. The idea of applying NAS in lifelong learning setting is interesting and novel, and the experimental results do show performance gains.
main_review: Overall, I think the paper is novel with good results. But there are certain technical details that are not immediately clear to me (and the link to code is empty), that I would appreciate if the author can clarify a bit in the rebuttal, and I am happy to adjust my score based on that.

1. How do you evaluate the score for, say the first task, after training on all tasks? Do you save the best model for it in the memory buffer? Or do you utilize the agent to create a new model? If it's the former, wouldn't that requires linear memory in terms of number of tasks? If it's the later, what is the environment, or the base learner for the input to the agent? Does that mean the agent can utilize knowledge learned in the later task to construct model for earlier tasks?
2. It seems to me that the agent is the most prone to CF. Essentially, it is trained to optimize the current task performance in a lifelong learning setup. Although preventing CF is included as part of the reward, I am not super convinced that this is a signal that is strong enough. Have you tried use other lifelong learning methods to regularize the agent?
3. A big concern of NAS is its efficiency, and this is even more serious the case here since you are learning to search for not just one but many tasks. Given the lifelong learning setting is often used in a online setting in reality, it seems to me that the computational complexity may be prohibitively large for your method. Can you provide the runtime compared with other methods for your experiments?
limitations_and_societal_impact: See above.
review_title: null

Meta-review: This paper proposes an NAS approach (namely BNS) for continual learning that builds dynamic network structures such that it minimizes catastrophic forgetting but maximizes forward transfer. The experiment results on several CL tasks show the effectiveness of the proposed method.

Overall, all reviewers found the paper to be well written and easy to follow, and focuses on an important research problem with relatively little attention from the area. Based on the reviewer comments, the authors further provided missing details pointed out by the reviewers and some reviewers increased the review score. After the rebuttal, all the reviewers remained positive. I recommend accepting this paper.

Decision: Accept (Poster)