Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: I found the idea quite novel. Lately in continual learning the focus has been more on NAS type ideas and algorithms, but this work is a nice divergence from this direction. The idea of optimizing in a space orthogonal to the previous task is novel. The execution of the idea is nothing special since it's using standard linear algebra, but I gave the authors full merit to the idea itself. 

My higher score is mostly due to the fact that the experiments are limited. The benchmark algorithms definitely miss some recent works from 2019 and 2020. They should be included as otherwise the superior performance of the algorithms is questionable. 

See for example:
https://arxiv.org/abs/2006.04027
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In Advances in Neural Information Processing Systems, pages 899–908, 2018
review_title: Nice idea

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:

This work targets learning multi-class classifiers in the continual learning setting. The key idea is to learn new tasks by taking gradient steps in directions orthogonal to the gradient subspaces marked as crucial for previous past tasks. The method employs SVD after learning each task to find the crucial subspaces (which it calls as Core Gradient Subspaces) and stores them in a memory. Reasonable quantitative and qualitative evaluation has been performed to compare the method against existing SOTA baselines.


Review:

1. The paper is overall clearly written and the method is adequately described.

2. The work is relevant to audience at ICLR and provides reasonable details for reproducibility.

3. Relationship to previous works has been explained and relevant literature has been cited appropriately.


Strengths:

1. GPM shows very little degradation in performance on past tasks and is very resilient to forgetting.

2. It provides explicit control over forgetting via the \eps_{th} parameter.


Weaknesses:

1. GPM performs fairly close to or marginally better than HAT and EWC on most datasets.

2. GPM trades-off between ACC and BWT metrics by strictly controlling forgetting. Hence, it may not be able to identify tasks with very similar structure (or repeated tasks) and may not re-use the Core Gradient Space from previous tasks to improve performance on them (BWT). So, while it minimizes negative BWT, it also restricts positive BWT.


Potential improvements and clarifications:

1. The authors have repeatedly emphasized that their method performs task-free inference. However, doesn't this just mean that the tasks used are such that the inputs encode the task identity and do not require a separate task identifier. Further, on page 6, under Network Architectures, the authors state that apart from permuted MNIST tasks, they use the multi-head setting, i.e., each task has a separate classifier. How does this allow task-identifier free inference?

2. All models have been trained using plain SGD. Would it be possible to extend the method to other optimizers, e.g., Adam?

3. 3 runs are too few to average over. These are supervised learning experiments, so it should definitely be possible to use more runs (at least 5, but preferably 10).

4. At first glance, it seems that the method may have scalability issues since it relies on performing SVD. However figure 2 counter-intuitively shows GPM to be both fast and memory efficient. Is this because these graphs in figure 2 are per-epoch? What about in between tasks when GPM needs to run SVD on all layers? Some more explanation and results are required to better understand how GPM achieves computational efficiency despite relying heavily on SVD per layer after every task.

5. Lastly, no experiments with a single class per task have been performed. This setting is known to be quite challenging in general and induces significantly more catastrophic forgetting (see Kamra et al, 2017). In this setting it is also generally harder to just set batch-norm parameters using just the first task.

[Kamra et al, 2017] Deep Generative Dual Memory Network for Continual Learning. Nitin Kamra, Umang Gupta and Yan Liu. ArXiv 2017.

Please respond to the 5 potential improvements and clarifications mentioned above. I will be happy to raise the score further if the above concerns are addressed.

-------- Post-rebuttal edit -------

The authors have answered all my questions satisfactorily and provided additional experiments wherever it was required including settings where GPM may not outperform other baselines. I believe that the paper is strong and makes a significant technical contribution to the field of CL. Hence, I recommend acceptance and I am updating my score to reflect the same.
review_title: New method for continual learning, but needs more thorough evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This manuscript proposes a new approach for continual learning problems. The key idea is to maintain bases of subspaces using SVD in the Gradient Projection Memory (GPM), in which the update direction is orthogonal to the gradient subspaces deemed important for the past tasks. Image classification experiment was conducted to justify its better empirical performance in practice. Overall, the paper is well-written. I have the following comments. 

1. The contribution of this manuscript is kind of incremental compared with OGD (Farajtabar et al. 2020). From my understanding, the main improvement is using SVD to store the bases, which basically trades computational efficiency for memory. In addition, it is claimed that OGD only works for small learning rate. Why using SVD helps using a larger learning rate? It is not explained in the paper.
2.  Is it possible to report running time of the proposed approach and compare with other approaches? SVD is very expensive in the high-dimensional setting such as deep neural networks. It might be impractical due to running time concerns.
3. What is the value of $k$ in the experiments (in inequality (9)) for different $\epsilon_{th}^l$? It is better to report these values. In addition, when $k$ becomes large, the update rule (6) and (7) may become computationally expensive.
4. It is mentioned that “our proposed approach can perform inference without the task identity of test samples”, but with no explanations. It is desired to illustrate the meaning in the main paper.
5. The literature review is not sufficient. There are several recent papers which also change the update direction using gradient projection. It is desired to see the difference between the proposed method and other relevant approaches. For example,

	[1]. Yu et al. Gradient surgery for multi-task learning. NeurIPS 2020.

	[2]. Guo et al. Improved schemes for episodic memory-based lifelong learning. NeurIPS 2020.

========POST REBUTTAL=========

I would like to thank the authors to answer my questions. It addressed most of my concerns. Hence I increase my score to 6.
review_title: Interesting approach but kind of incremental. Experiments are good, but it lacks running time comparison.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The paper proposes one of the most scalable approaches to sequential continual learning with known task boundaries and related tasks, while taking steps towards enforcing data privacy and removing some of the task label constraints. At all levels in expressive deep models, SVD is used on learned representations to identify important bases of task gradient spaces and a memory is populated with such directions. Learning progresses only in directions orthogonal to gradient memory. Several recent evaluation methodologies are used to empirically validate the approach with significant success. 


Strong points: 
- Principled and relatively simple approach, yet scalable to interesting deep models.
- Manageable computational overhead.
- Memory of gradients introduces a layer of data privacy, and advantage compared to many other memory-based approaches.
- Strong empirical results, although not apples-to-apples in all cases, see the comment posted earlier.
- Clearly written paper.
- Analysis of scalability in terms of memory and computation is a welcome bonus.


Weak points:
- The sequential task learning setup is of limited practical use and not too realistic.
- It’s hard to say how the algorithm would be used in practical continual learning settings, e.g. reinforcement learning of single tasks without forgetting, or where clear task boundaries do not exist.
- Little effort is made to see the approach relevance for non-similar sequential task learning, where there could be interference between learned representations in terms of negative forward transfer.


Recommendation and Rationale:

I strongly recommend acceptance because the method is simple, practical and the paper is well written both in terms of clarity and also analysis.


Further questions:
- Do you see any positive forward transfer, e.g. later tasks are learned quicker due to previous tasks?
- What are the limitations and roadblocks to extending this method to sequences of hundreds of tasks which are not necessarily related? Please discuss in the manuscript.

review_title: Great method and paper applied for one of the most basic continual learning settings

Meta-review: The paper proposes a new approach to continual learning with known task boundaries that is scalable and highly performant, while preserving data privacy.  To mitigate forgetting the proposed approach restricts gradient updates to fall in the orthogonal direction to the gradient space that are important for the past tasks. The main novelty of the approach is to estimate these subspaces by analysing the activations for the inputs linked for each given task. 

All reviewers give accepting scores. R2, R3 and R4 strongly recommend accepting the paper, while R1 considers it borderline.

The authors provided an extensive response carefully considering all reviewers' comments. New experiments were introduced (training time analysis and comparisons with expansion-based methods), and several clarifications were added.

All reviewers agree that the paper is well written and its literature review adequate.

The main concern of R1 was the similarities with OGD (Farajtabar et al. 2020). R1 considered the authors’ response acceptable. R2, R3 and R4 consider the contribution well motivated and significant and highlight its simplicity. The AC agrees with this assessment.

The empirical evaluation covers most of the typical benchmarks in CL. Very strong results are reported on a variety of tasks both in terms of performance and memory efficiency, as agreed by R2, R3 and R4.

Overall the paper makes a strong contribution to the field of CL.

Decision: Accept (Oral)