Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a lifelong learning algorithm across multiple tasks that automatically learns which layers need to be optimized using an EM learning strategy for each task. In the expectation step, the algorithm updates the next best configuration and in the M step, it optimizes the model parameters.

Some details of the algorithm are not explained well. It could help a lot if the authors could provide the objective function they are trying to optimize. As examples: (1) The motivation behind Equation 3 is not clear at all. Based on Equation 3, It seems that the choice of different configurations is mini-batch dependent and not task dependent. I have a difficult time to understand why this is the case. I might have misunderstood the algorithm but providing the objective function helps a lot. (2) It is not clear why n_{c_i} in Equation 1 is the number of previous mini-batches for which C_{(t)} is the most probable configuration. It makes sense to use a soft version of this (sum of probabilities) to have it more compatible with the rest of the algorithm. Again, understanding what objective function the authors are targeting to optimize helps to understand this better too.

Even though, some aspect of the proposed algorithm is not clear, the algorithm works in practice and outperform multiple suggested baselines. 

Minor type: Page 5: 5% percent  5%

review_title: Algorithm needs more clarification

final_score: rating: 4: Ok but not good enough - rejection
review_content: -Summary-
The paper proposes a method for selective weight sharing per layer during continual learning. The authors show observations that sharing all layers can not be optimal for lifelong learning. Hence, they adopt a layerwise transfer configuration vector which decides activated layer-sharing at specific tasks. The problem is solved by EM algorithm-based approach. 

-Pros-
- Observations are reasonable and give many inspirations for solving continual learning issues and developing the existing methods.
- The paper is well written and easy to follow.
- The point of view connected to neural architecture search is understandable.
- The model outperforms old baselines.

-Cons-
- The problem is task-incremental which clearly gives task oracle during training and inference. Recent continual learning works obviously tackle class-incremental learning problems that are more challenging and applicable to a broader area [1]. 
- Baselines are too weak. If the paper targets task-incremental learning problems, the authors should compare their methods with recent works, rather than with 3-5 past years' works. I recommend to include further strong baselines like [2,3,4].
- The strengths of the methods may not be impressive on modern deeper networks, like ResNet-50. Since it requires massive computation time. The paper didn't show the results/analysis of modern deep architectures.
- The model inevitably requires much capacity for not-shared task-specific layers. But the authors didn't include it.

-Comments-
- Why do the authors only use a fraction of datasets?


===============
- [1] van de Ven, Gido M., and Andreas S. Tolias. "Three scenarios for continual learning." arXiv preprint arXiv:1904.07734 (2019).
- [2] Titsias, Michalis K., et al. "Functional Regularisation for Continual Learning with Gaussian Processes.", ICLR 2020.
- [3] Yoon, Jaehong, et al. "Scalable and Order-robust Continual Learning with Additive Parameter Decomposition.", ICLR 2020.
- [4] Davide Abati, et al.  "Conditional Channel Gated Networks for Task-Aware Continual Learning.", CVPR 2020.

review_title: Review of the Paper 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the problem of lifelong learning of a sequence of tasks by selectively transferring knowledge of some layers across tasks. The proposed approach, LASEM, uses EM to dynamically adjust transfer configuration between tasks by performing architecture search. The authors present results in benchmark datasets for continual learning. 

As strengths of the paper I would remark: 
-	The paper is well-motivated and demonstrates experimentally why simply transferring all layers does not lead to the best transfer configuration. 
-	The proposed approach in section 3 is clearly explained and the main design choices are well argued. In general, the paper is well-written and easy to follow. 
-	The paper includes thorough results of the computational complexity of the proposed method, which could be a concern given the proposed approach of maintaining both shared and specific sets of parameters for each task. 

The weaknesses I observe in this paper are: 
-	I am concerned on the extensibility of the proposed method to a large number of tasks, which I would think would pose a challenge in the proposed setting similar to how it does in multitask learning. What would be the impact of a large number of tasks, specifically on the set of shared parameters? The experiments provide results for a limited number of tasks (up to 20).

Questions for authors: 
- Please address the question regarding the effect of number of tasks on the proposed approach.
review_title: Review for Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer

final_score: rating: 7: Good paper, accept
review_content: Summary: authors look into life long learning setting (when task arrive one after another) and try to understand which layers of the source model need to be reused (transfered) and which should be re-learnt. Authors argue that this decision should be task specific, and come up with an algorithm (EM like) that for each task derives indices of the layers to reuse and also updates reusable and non reusable layers.

Overall, this paper is really well executed. Easy to follow, enough of background and motivation is provided, and the algorithm for the most part is clear and intuitive.
My main complaint is that it is somewhat thin on the experiments.

Additional questions/comments:
1) How is update 13 from Algo 1 happening? (how P(C|data) is updated? is it  working by recalculating the counts from step (1) (so it is not a smooth function that is differentiable and updated via optimization)
2) In Algo 1: while IsMoreTrainingDataAvailable is not clear to me. I assume that you do several epochs over task specific data, don't you? Also do you monitor anyhow EM convergence?
3) Experiments: One baseline would be useful to just randomly select c_t for each task and see how it does. EM is pretty expensive
4) Finally, I feel that related work should really go after the into
review_title: Nicely executed paper

Meta-review: The reviewers enjoyed reading about an interesting take on lifelong learning, encapsulating an EM methodology for selecting a transfer configuration and then optimizing the parameters. R3 made valid concerns regarding comparison with previous, recent work. R2 also would prefer to see more thorough experiments (ideally in settings where multiple tasks exist, as also commented by R4). During the rebuttal phase the authors made a good effort to run additional experiments which cover the related work aspect better. These experiments and the overall paper were discussed extensively among reviewers after the rebuttal phase.

In the discussions, the reviewers agreed that an interesting idea can be publishable even if it does not achieve SOTA results in all scenarios, as long as it brings new perspectives and shows at least comparable results. However, in the particular case of this paper, there exist remaining concerns regarding the usefulness and applicability of the method. Specifically, the paper could benefit from a more convincing demonstration about how the method can scale (e.g. R3 and R4’s comments), especially since training time and model capacity are important factors to consider for practical continual learning scenarios. Furthermore, it is not clear how the proposed method can be used in combination with other machine learning tools within a continual learning application, for example by leveraging modern deep architectures or by complementing existing adaptive knowledge approaches (as discussed by R3). 

Although the opinions of the reviewers are not fully aligned, this borderline paper seemed to lack an enthusiastic endorsement by a reviewer to compensate for the concerns discussed above and the relatively weak experimental results. Therefore I recommend rejection. 

Decision: Reject