Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a continual learning algorithm that enforces the convolutional filter in each layer to a low-rank filter subspace defined by a small set of filter atoms. For each task, each convolutional layer is defined by a new filter subspace but subspace coefficients are shared among the tasks. The algorithm is validated on multiple benchmark datasets. 
main_review: (+) Interesting idea of considering filter subspace with the construction of filter atoms and sharing the subspace coefficient fixed while changing the small set of filter atoms corresponding to the task. 
(-) Different distance metrics other than Grassmann should be considered to see whether it leads to consistent performance. the computational required to compute the distance between filter subspaces requires calculating an SVD which is computationally heavy.  
(-) The performance of the proposed approach will depend on the dataset at hand. For very diverse tasks, an inter-task ensemble would not be very effective. 
(-)The performance of the proposed algorithm can vary heavily on the diversity of tasks. 
(-) (Introduction) I do not agree with the first sentence of the introduction. Humans do forget concepts learned in the past. 


summary_of_the_review: The paper is written and organized adequately. The paper is based on extensive experimental results. The idea is relatively simple and performs well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper, motivated by the task subspace modeling literature, enforced a low-rank filter structure to each CNN layer across time in continual learning. It not only ensures that the knowledge of the past tasks is not lost but also saves a lot of computing memory. Meanwhile, the paper proposes novel intra-task ensembles and inter-task ensembles for class-incremental settings and task-incremental settings, respectively. 
main_review: In general, the idea of the article is concise and clear. The survey of related work and experiment settings are sufficiently detailed. More importantly, there is a huge improvement in memory saving compared with other methods.

However, there are two main problems that I am concerned about:
i. In Sec 5.2.2, there are some recent methods that show better performance on Cifar100(20 tasks) in Paper with code. But the author doesn't consider them, such as "Understanding Catastrophic Forgetting and Remembering in Continual Learning with Optimal Relevance Mapping", "Compacting, Picking and Growing for Unforgetting Continual Learning", etc.
ii. I think only the results of the two datasets are not enough to support the author's conclusion strongly. The author would better add a few more experiments, and there are plenty of datasets shown in other papers for your choices, such as CUBS, Stanford Cars and Flowers, etc.
summary_of_the_review: The paper idea is creative and has made a significant improvement in saving memory, but more experiments are still needed to support the conclusion.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper tackles the continual learning via enforcing a low-rank filter structure to each CNN layer.  They first perform atom-coefficient filter decomposition and then learn each task with a new filter subspace, so that the method only needs to save the new filters for each task. The contribution of this paper includes the low-rank filter scheme and the designed intra-task and inter-task model ensemble performing on the filters. The proposed method also achieves SOTA performance on several datasets with tiny size of model memory.

main_review: Strengths:
1. This paper is well-written and easy to follow. 
2. Learning new tasks on low-rank filter subspace seems interesting and novel for solving continual learning problems. 
3. The proposed method achieves good performance on the benchmark dataset. 

Weaknesses:
1. For the experimental results, we can see that the SOTA performance of the proposed method is based on the ensemble scheme. However, it may not be fair when adopting the model ensemble strategy when compared with other methods, especially the intra-task ensemble. The intra-task ensemble will linearly increase the memory and computation consumption with the E_w. 

2. The proposed low-rank filter strategy would allow highly efficient model storage while it seems that it does not reduce the computation during testing. The authors may need to compare the inference time among different methods. 

3. The proposed method is based on the over-parameterization of deep models. If we use a lightweight network architecture, the advantage of the proposed method compared with other expansion-based methods would narrow down.



summary_of_the_review: This is an interesting paper by introducing the idea of subspace modeling of tasks into continual learning and further proposing two model ensemble strategies to improve the performance. The proposed method also achieve good performance on benchmark datasets.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a model for continual learning based on the decomposition of linear filters into low-rank components, called atoms. Specifically, the authors decompose convolutional filters shaped (c,c',k,k) into two components: i) alpha, shaped (c,c',m) and D, shaped (m,k,k). The former is learned on the first task and then frozen, whereas for D every task has its own and they are do not conflict during optimization. On top of that, the authors envision two different ensembling schemes that improve performances. i) First, in task-incremental settings, they retrieve atoms from the task of interest and from similar tasks as well (based on SVD decomposition of D matrices and Grassman distance) and ensemble them. ii) Furthermore, in class-incremental settings, they explicitly setup multiple atoms per task, building a task ensemble. During inference, all ensembles are queried and their predictive variance is used to "discover" the relevant task, for which a prediction is carried out. Experiments are carried out on 3 datasets in both task-incremental and class-incremental learning settings.

main_review: PROs:
- The paper is for most parts well written. With very few exceptions, all the text is clear at a first read. Figures, mathematical notation and equations are properly used and truly help in understanding several technical aspects.
- The method proposed is technically sound. By keeping separate atoms for every task, they do not interfere with each other and therefore do not imply forgetting of past knowledge.
- The results presented for class-incremental learning are very encouraging (Tab. 1), especially considering the model does not involve any buffer, that is considered almost a standard for such settings. Even in its absence, the model can outperform a number of competing methods that store examples from past tasks, at a fraction of the memory overhead.
- To my knowledge, the low-rank decomposition of filters (a standard in other fields such as neural network compression) is novel in the continual learning field. This allows the network to grow as new tasks are encountered, like in other models such as Progressive Neural Networks [1]. However, by restricting the growth to the D components only, memory is impacted in a much more limited way.

---
CONs:
- The class-incremental experiment, the most interesting in my judgement, is only conducted on a single dataset (CIFAR-100). It would be interesting to check whether the same findings hold on miniImageNet as well. Moreover, the authors explain such a setting by stating "does not provide task-id during training". However, the exact definition is that it does not rely on task-id during inference[5,6]. Can the authors confirm this is the case for the results presented in Fig.4?
- A substantial part of Sec. 3 describes several ensembling techniques that are used in different settings within experiments. Although I understand ensembling is beneficial for performances on a single task, I consider this contribution somewhat orthogonal to the main aim of the paper. Indeed, it does not involve any interaction between different tasks during training and it is therefore not related to the sequential nature of the continual learning problem. It seems the authors included that as their model is structurally more prone to ensembling (which I acknowledge it is) that regular CNNs. For this reason, it is a bit difficult to assess this contribution. Couldn't other ensembling techniques be used, both for the proposed model and for competing methods? 
- The derived excess risk bound is not commented. The authors should add text describing what the derived bound means and why it is meaningful. Moreover, it is derived under a couple of strong assumptions, namely a single linear layer and a MSE objective, that are not met in the experiments carried out. As such, it is a bit difficult to assess whether that would generalize to deep networks used in practice.
- To my understanding, the model as is does not allow for any forward knowledge transfer, as past Ds are not used during training of the current task. This seems suboptimal, as other "architectural methods" such as [1,2,3,4,5] envision this possibility by allowing past filters to be employed during future tasks. Here, this possibility is only granted by the ensembling mechanism, which is however not explicitly trained for the purpose of forward transfer.
- The authors employ the Grassman distance in ensembling, in order to retrieve past task parameters that are somewhat similar to the parameters of the current task. This choice is not validated in experiments, and other types of distances should be tested, such as L2 or cosine similarity in parameters space.
- In authors' proposal, the matrix alpha is trained on the first task only. This raises the question whether the overall model is susceptible to the choice of the first task. A good way to check this would be to have a set of first tasks $F=\{T_1,...,T_m\}$ and a set of remaining tasks $R=\{T_{m+1}, ..., T_n\}$. If we repeat an experiment where the first task is sampled from F, and then all tasks from R are trained in a sequence, would the outcomes (i.e. performances on $R$, excluding the first warmup task) have a high variance?

Minor, questions:
- After Eq. 5, a $()_+$ sign is explained. However, that does not appear in the equation.
- Referring to Tab. 1, are all regularization-based methods used in a task-incremental way, by excluding classes belonging to unrelated tasks from the prediction?
- In Tab.1, why does HAT have (slight) backward transfer? To my knowledge, weights for the previous tasks are fixed and therefore it should have exactly 0 forgetting and BWT (same as PNN and the model proposed by authors.)

---
References:
- [1] Rusu, Andrei A., et al. "Progressive neural networks." arXiv preprint arXiv:1606.04671 (2016). 
- [2] Serra, Joan, et al. "Overcoming catastrophic forgetting with hard attention to the task." International Conference on Machine Learning. PMLR, 2018. 
- [3] Mallya, Arun, and Svetlana Lazebnik. "Packnet: Adding multiple tasks to a single network by iterative pruning." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018. 
- [4] Mallya, Arun, Dillon Davis, and Svetlana Lazebnik. "Piggyback: Adapting a single network to multiple tasks by learning to mask weights." Proceedings of the European Conference on Computer Vision (ECCV). 2018. 
- [5] Abati, Davide, et al. "Conditional channel gated networks for task-aware continual learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020. 
- [6] Van de Ven, Gido M., and Andreas S. Tolias. "Three scenarios for continual learning." arXiv preprint arXiv:1904.07734 (2019).
summary_of_the_review: In my opinion, the paper introduces a very simple model for continual learning and showcases extremely encouraging results in class-incremental learning. The submission could further improve by including class-incremental experiments on a second dataset (miniImagenet) and clarifying the points raised above about model ensembling and risk bound.

---- POST REBUTTAL COMMENT ----
As detailed in my answer below, I will increase my score to 8 after reading the author's feedback.
review_title: null

Meta-review: The authors propose a memory-based continual learning method that decomposes the models' parameters and that shares a large number of the decomposed parameters across tasks. In other words, only a small number of parameters are task-specific and the memory usage of storing models from previous tasks is hence a fraction of the memory usage of previous approaches. The authors take advantage of their method to propose specific ensembling approaches and demonstrate the strong performance of their methods using several datasets.

In the rebuttal, the authors were very reactive and provided many useful additional results during including a comparison of the computational cost of their method vs. others, results using two new datasets (CUBS & Flowers), and additional results on mini-ImageNet. They also answered, through additional experiments, several reviewer questions including the robustness to different first tasks in the sequence.

Overall, the reviewers after the rebuttal/discussion period agree that this is a strong contribution: novel and fairly simple method with some theoretical justification, thorough empirical evaluation, well-written and easy to follow manuscript. It also opens a few interesting avenues some of which the authors have already explored in their paper (e.g., ensembling).
Decision: Accept (Spotlight)