Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: **Summary:** The paper focuses on gradient projection (GP) for incremental learning. The authors motivate their work by stating that while approaches based on GP lead to superior performance in overcoming catastrophic forgetting, they suffer from a significant drawback. In GP, once one calculates the subspaces spanned by layerwise inputs for a task, say Task A, the network is then forced to only update the weights in an orthogonal direction to these subspaces for learning a subsequent task, say Task B (hence keeping important parameters for Task A intact and overcome catastrophic forgetting). However, suppose Task B and Task A are similar (an extreme case is when Task B is the continuation of Task A!). In that case, we know that the essential parameters for Task A are likely to be important for Task B and that the network could benefit by continuing to update the important parameters for Task A, which is not allowed in GP algorithms. This behavior has two consequences: 1) intransigence, i.e., the network won't be able to learn Task B as effectively as possible, and 2) the network will have reduced backward transfer. The backward transfer issue is apparent in the extreme case when Task B is a continuation of Task A, and the network's performance on Task A would have improved if it was able to learn Task B using the important weights for Task A!  The paper addresses this issue, with a simple, yet practical, solution. For a new task, the authors calculate the correlation between the subspaces calculated for old tasks and the new task (layerwise), keep track of the most correlated previous tasks for each layer, and denote them as (layerwise) Trust Regions. Next, the authors propose a scaled weight projection that allows for unfreezing the important parameters in the Trust Region of the new task, while learning this task, and learn the corresponding weights as part of the optimization process. Finally, the authors report results on Permuted MNIST (PMNIST), CIFAR100 Split, CIFAR100 Sup, and a sequence of 5-Datasets with 10-class classification which includes CIFAR-10, MNIST, SVHN, not-MNIST, and Fashion MNIST, in comparison with GP methods, regularization-based methods, and memory replay method. They show consistent improvement in accuracy, and more interestingly in backward transfer. 

main_review: **Strengths:** 
The paper:
* addresses an important problem with a good solution
* is well motivated 
* is theoretically sound
* is well written and easy to follow
* does a good job at concisely reviewing the important recent work on the topic

**Weaknesses:**
I don't see any major weaknesses in this paper. I have a few questions that I would like the authors to address (See below). Here are a few points that the paper can improve upon:

* providing the memory footprint of the algorithm
* comparing wall-clocks of TRGP/GP

**Questions for authors:** I appreciate it if the authors can clarify the following points and provide additional information.

1. The scaling matrix $Q_{j,t}^l$ plays a crucial role in your approach. This matrix is balancing the freezing/unfreezing process. Why wouldn't the network always choose $Q$s such that all parameters in the Trust Region are unfrozen? This would presumably lead to the least loss for the current task (as it provides the maximum capacity for the task). If so, why should one optimize $Q$ as opposed to simply fixing it to unfreeze all parameters in the trust region? 

2. Could you please comment on the memory footprint of your algorithm? Also, could you please provide a head-to-head wall-clock comparison between GP and TRGP?

3. Last question is also about unfreezing and matrix $Q$. If my understanding is correct, all tasks in the Trust Region (for each layer)  are currently treated as equally important. For instance, if, for a certain layer, Task A and Task C have a correlation of 1.0 (same tasks), and Task B and Task C have a correlation of 0.8,  they are both added to the trust region of Task C, but the information that Task A is more important to unfreeze is missing and is left up to the optimization on $Q$ to decide this by itself.  Do you think there is a benefit in incorporating the correlation values into your Trust Region?

summary_of_the_review: **Overall evaluation:**  I think the paper addresses a very important problem. Generally speaking, non-memory-replay-based methods for overcoming catastrophic forgetting could suffer from intransigence, which is the inability to learn new tasks due to increased stiffness/rigidity of the network. This reduces both forward and backward transfer, especially when the tasks are similar (e.g., revisiting an old task). This paper provides a rational solution to this problem, which provides consistent numerical improvement over the state-of-the-art. From an editorial point of view, the paper is well-written, easy to follow, and it provides a good overview of the recent literature on the topic. Hence, I think this is a good paper and vote for its acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Some existing methods put restrictive constrains on the optimization space of the new task to prevent catastrophic forgetting, which may lead to unsatisfactory performance for the new tasks.
This paper aims to facilitate the forward knowledge transfer based on an efficient characterization of task correlation. Main contributions can be summarized as follows:
1.Introduce a novel notion of ‘trust region’ based on the norm of gradient projection onto the subspace spanned by task inputs to measure task correlation.
2.Proposed a novel approach for the new task to leverage the knowledge of the strongly correlated old tasks in the Trust Region through a scaled weight projection.
3.Developed a continual learning approach, trust region gradient projection(TRGP) based on the introduced Trust Region, scaled weight projection and a module to construct task input subspace.
4.Compared to related state-of-the-art approaches, TRGP achieves substantial performance improvement on all Benchmarks.
main_review: 
Strengths: 
1.This paper put forward the problems of the existing methods and provides a detailed discussion.  
2.Propose a novel solution to this problem, which achieves substantial performance improvement on all benchmarks and overwhelmed its baseline method.

Weakness: 
1.Toy example 1 seems not convincing. 
In this example, the only difference between task 1 and task 2 is the sign of the input. Since each task has a separate classifier, we can suppose a simple scenario where network has only one Linear Layer followed by a classifier. Since S2=S1, W2=W1 after learning task 2. Though W1x1 = -W2x2, we can adapt the classifier of task 2 which is task-specific to ensure Wc2 = -Wc1  and  make sure task 2 can be classified correctly
The example seems want to illustrate that the ideal output of task 2 should be the same as task 1, i.e., W1x1 = W2x2. But since the input has changed, why cannot the output change?
2.The way to measure the correlation between tasks.
Denote ΔX’∈R^(m×n), X∈R^(n×b), Δ∈R^(m×b) as the gradient spanned by the input batch of task t, where  is the input dimension and  is the label dimension. It’s projection onto the subspace of task j can be written as ΔX’ BjBj’. Perform SVD on X, the gradient projection can be rewritten as Δ(U∑V’)’ BjBj’, which is δU’BjBj’. Why use the projection of a vector spanned by U to measure the correlation instead of the base U?
3.As said in the paper, trust region aims to select strongly correlated tasks. But in Session ‘Impact of selected tasks in trust region’, it is obvious that trust region only selects most close tasks even if they are not strongly correlated, and this phenomenon can be seen in the task-wise experiment and higher layers in the layer-wise experiments, which makes me doubt the effectiveness of the trust region selection.
4.In the Session 4.2, this paper says that if task t is strongly correlated with task j, the weight projection on subspace of Sj is important for task t. This view seems like an assumption which is not proved. It would be more than welcome if more convincing proof could be provided.
summary_of_the_review: This paper put forward the main problem of existing methods and provides a detailed discussion. The author proposes a novel solution to this problem, which achieves substantial performance improvement on all benchmarks and overwhelms its baseline method. But some of the claims seem not convincing and some seem lack illustration.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the problem of forward knowledge transfer in continual learning settings is explored. The idea is to measure correlations between the learned tasks based on the notion of "trust region" which helps to identify the most similar learned tasks to the current task. The core idea is that frozen weights for similar past tasks can be relaxed to reuse them to learn the current task. Since task similarities are used for this purpose, this will not lead to catastrophic forgetting and at the same time helps to transfer knowledge. Experiments on four benchmarks are provided to demonstrate that the method is effective.
main_review: Strengths:

    1. The problem of forward knowledge transfer is an unexplored problem in the continual learning literature.

    2. The paper reads well.

    3. Experiments are somewhat convincing.

Weaknesses:

    1. Experiments are not extensive enough and important benchmarks are missing.

    2. Thorough comparison with recent works is missing.


summary_of_the_review: Current works in continual learning focus mostly on tackling catastrophic forgetting and overlook accumulative learning. Hence, I think the authors have selected a good area and the proposed idea also seems to be sound. However, I have some reservations about this work:

1. Given the expectation in the recent literature on continual learning, only relatively simple datasets are considered in the experiments. I think results on more complex datasets such as split-Sub-ImageNet and Split-ImageNet should be added. 

2. I was wondering why forward transfer metric is not used for comparison? I think it is as informative as backward transfer metric.

3.  An area of interest is to analyze the learning curves and dynamics of learning, i.e., performance vs learning epoch. It is helpful to see if the proposed method enables better jumpstart performance when a new task is learned.

4. Comparison with more works is missing. For example, EWC and HAT are both outdated for regularization-based methods and many recent follow-ups exist. For a realistic comparison against prior works, state-of-the-art methods for each group of methods should be included. Please check the recent literature.

5. Computational complexity is overlooked. Whereas it is an important factor for continual learning. I think it is necessary to include how much additional computational load should be performed to benefit from TRGP.

6. For a more informative comparison, the standard deviation in results should be reported. Also, BWT metric can be reported more accurately by including more decimals.

 In conclusion, I think this work is in a good direction but further improvement is necessary to make it suitable for a venue similar to ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a continual learning method based on gradient projection memory (GPM) of Saha et al., which projects the gradient of each layer to be orthogonal to the input subspace of previous tasks.
Motivated by the fact that the orthogonal projection can harm the performance by being too restrictive, the authors propose a heuristic algorithm to reduce the restriction.
Specifically, the authors choose a subset of most "correlated" tasks and let the model change along the subspace of the correlated tasks.
main_review: ## Strengths
- This paper points out a major issue of GPM (Saha et al.): the orthogonal projection is too restrictive.
- The proposed heuristic empirically improves the performance.

## Weaknesses
### The writing is hard to follow
Since this paper introduces multiple new concepts, it was hard for me to understand the overall algorithm and the intuition behind it.
I think the writing can be improved a lot.
For example,
- It was hard to find out what the authors are trying to do with the trust region. I was confused if they are going to allow or restrict the model update in the trust region.
- In Figure 2, $S_l^f$ seems to be the 2D input subspace of layer $l$ in 3D input space. But why is there $\nabla_{W^l} \mathcal L_t$, which should be in the parameter space? According to Eq.(2), $\nabla_{W^l} \mathcal L_t$ and $\mathrm{Proj} (\nabla_{W^l} \mathcal L_t)$ are matrices, but why are they represented as a vector?

### Some of the reasoning steps are controversial
- Toy example 1 is not a good example. The authors claim that orthogonal projection is problematic because the optimal model for task 2 should be $W_2^l = -W_1^l$. However, that will lead to complete forgetting of the first task. In fact, this example is impossible to solve with a single linear layer.
- Therefore, I do not agree with the conjecture motivated by this example: *naive orthogonal projection could possibly compromise the learning performance of the new task that is strongly correlated with old tasks, especially when the correlation is “negative” as in the toy example 1*. Toy example 1 is a situation where the model has to choose between two conflicting options: (i) perfectly memorize task 1 and ignore task 2, (ii) completely forget task 1 and learn task 2. Orthogonal projection is a method for option (i), which is one of the best things a linear model can do.
- Considering my previous arguments, the claim that the weight projection to the subspace of trust region should be modified is somewhat arbitrary. I think more theoretical justification is needed.

### Lack of justification and empirical evidence for the scaled weight projection
Although the authors claim scaled weight projection as one of their main contributions, there is no explanation of how the scaling matrix $Q_{j,t}^l$ in Eq.(6) is initialized and trained.
Also, there is no ablation study to prove its effectiveness.
Without further information, I cannot acknowledge this component as a contribution.

### Limited novelty
In the end, the authors perform a relaxed orthogonal projection of the gradient.
The proposed method is just one way of relaxing the orthogonal projection, and there is not enough justification for why this particular algorithm should be effective.
I think there can be various ways to relax the orthogonal projection.
It would be better if the authors could provide a thorough empirical analysis of multiple relaxing schemes or theoretical justification for the proposed method.
summary_of_the_review: This paper proposes a heuristic to improve GPM (Saha et al.).
However, the proposed method lacks theoretical justification, useful insights, and some essential experiments.
Also, the overall writing should be improved.
Therefore, I do not think this paper meets the ICLR standards.
review_title: null

Meta-review: The submission addresses the problem of whether or not to update weights for a previous task in continual learning.  The approach is to specify a trust region based on task similarity and update weights only in the direction of the tasks that are similar enough to the current one.  The paper was on the balance well received (3/4 reviewers recommended acceptance, 2 with scores of 8) and complemented for its simple but effective approach, and good discussion of related literature.  The submission attracted a reasonable amount of engagement and discussion between reviewers and authors, which should be taken into account in the final version of the paper.
Decision: Accept (Spotlight)