Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Regularizing the subspace of the network representation has been used in continual learning to alleviate forgetting. To mitigate learning capacity degradation caused by orthogonal projection/regularization, this paper proposes to learn in low-coherence subspace. It learns task-specific projectors to represent each task in subspace with low coherence to others. To learn in low-coherence subspace, the projectors are learned on the Oblique manifold. The proposed method performs better than the compared methods (especially the orthogonal subspace learning method) on the task-incremental setting (with known task identifiers). 
strength_and_weaknesses: Strength
- This paper proposes to learn the projections in the low-coherence subspace, instead of the orthogonal subspace, which lets the model capacity be used better, while reducing the parameter interference, with less capacity degradation. This is well motivated.

- The experiments prove the claim about the benefits of learning in the low-coherence subspace, compared to learning with the orthogonal subspace. 

- The paper is generally written clearly, with some points that can be improved, as discussed in the following. 

Weakness
- The work can be seen as an extension of the orthogonal subspace based continual learning paper (Chaudhry et al. 2020). The main difference is the projectors are learned on the Oblique manifold instead of the Riemannian manifold, which enables learning in the low-coherence subspace with some overlaps between the tasks. It brings limited novelty for continual learning. 
- Intuitively, the low-coherence subspace learning can be seen as a “relaxation” of the orthogonal subspace learning. The mechanism of low-coherence learning and the difference with orthogonal learning is not clearly analyzed and discussed. 
-- Specifically, some questions can be asked and discussed. For example, comparing with the orthogonal subspace learning, how the method alleviates the forgetting issue while not totally isolating the subspaces. 
- The proposed method is limited to only the task-incremental setting where the task identifiers are required in both training and testing phases, as shown on page 3 and experiments. It further influences the significance of the work, considering the insignificant technical novelty, although this is with the same setting as (Chaudhry et al. 2020). The authors may further 
- Some detailed questions:
-- What is the gap between the implementation and the analysis on page 6? 
-- How is the efficiency of the proposed method? How is the running time compared to other methods? 

clarity,_quality,_novelty_and_reproducibility: The whole paper is generally written clearly, and the work is with good reproducibility. The details on the task-specific BN can be improved for better reproducibility. The analyses, discussion, and introduction for the settings can be improved. The novelty and contribution are not significant enough. 
summary_of_the_review: The whole paper is with a clear motivation and representation overall. The novelty and the significance of the contribution can be improved. Although the experiments can prove the method can work well and better than the orthogonal subspace learning method, the experiments are limited to a very specific and even simple setting in CL, which influences the generality and significance. 

After checking all the reviews and responses, I would like to keep my original score - below the acceptance line. 
The concept of using low-coherence subspace projection in continual learning is interesting. However, the contribution is relatively limited, considering the relationship with the methods working on orthogonal subspace and the restricted task-incremental setting (i.e., the requirement on task id). The technical details and explanations of the experimental results (in Table 1) are unclear and confusing, as indicated by 8QtN. It is blurry what is the main role of the low-coherence projection on performance improvement, compared with the normalization trick.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn new tasks in low-coherence subspaces rather than orthogonal subspaces to mitigate catastrophic forgetting in continual learning. The authors believe that Gradient Orthogonal Projection (GOP) (though helps battle catastrophic forgetting) causes  learning capacity degradation and its learning capacity "is gradually degraded as the number of tasks increases and eventually becomes unlearnable". 

To learn new tasks in low-coherence subspaces, the authors propose a unified cost function to seek projectors and develop a gradient descent algorithm (on the Oblique manifold) by jointly minimizing inter-task coherence and intra-task coherence. The original mutual coherence (often used in compressive sensing) of a matrix is the maximum absolute value of the cross-correlations between the matrix's columns. The authors extend the concept of coherence to two matrices to capture the maximum absolute value of the cross-correlations for the columns from two matrices. To deal with catastrophic forgetting heightened by batch normalisation (BN), the authors use two strategies: (1) learning specific BN for each task, or (2) using group normalisation (GN) instead of BN. 

The authors did experiments on Permuted MNIST, Rotated MNIST, Split CIFAR100 and Split miniImageNet. For 20 tasks (a typically setting), the proposed method achieved best accuracies on Permuted MNIST and Rotated MNIST, and 3nd best forgetting on Permuted MNIST and 2rd best forgetting on Rotated MNIST. The proposed method was reported to achieve best accuracies and zero forgetting (surprising to see zero forgetting) on Split CIFAR100 and Split miniImageNet. 

To show other methods would struggle with increasing tasks, the authors did experiments on 64 tasks and 150 tasks where the proposed method shows a clear advantage on accuracies, but not necessarily on forgetting (the proposed method is reported to achieve  best accuracies on both Permuted MNIST and Permuted CIFAR10, and 2nd best forgetting on Permuted MNIST and 3rd best forgetting on Permuted CIFAR10). 
strength_and_weaknesses: Strength: 
1) learning in low-coherence subspaces can be an alternative to GOP. 
2) the experimental results are impressive.

Weakness:
1) The authors believe that GOP causes learning capacity degradation and its learning capacity "is gradually degraded as the number of tasks increases and eventually becomes unlearnable". This should be more thoroughly analysed and supported by ablation study (curves on 64 tasks and 150 tasks with other methods are not sufficient to serve this purpose). 
2) The proposed method's forgetting performance on different datasets and settings vary widely from 0 forgetting to the state of being beaten by one or two competitors (e.g. see Fig. 1 (b) (d)).
3) Coherence metric can be restrictive. Babel function which extends cross-correlation between pairs of columns to the cross-correlation from one column to a set of other columns might be a better alternative.

clarity,_quality,_novelty_and_reproducibility: Low-coherence subspaces projection can be a good alternative to GOP. It is novel. 
More thorough analysis can be provided for the severity of GOP's learning capacity degradation issue. The authors' claim "their learning capacity is gradually degraded as the number of tasks increases and eventually becomes unlearnable" needs solid analysis and evidence. Likewise analysis and evidence for the proposed method does not suffer from this should be provided too. 

GOP should be used when the network has sufficient learning capacity as Chaudhry et al. (2020) pointed out that “We assume that the network is sufficiently parameterized, which often is the case with deep networks, so that all the tasks can be learned in independent subspaces.” If learning capacity can not catch up with the demands of increasing tasks, perhaps we can try to increase capacity on the fly? One way is to grow the network's depth adaptively [1*]. 

[1*] Online Deep Learning: Learning Deep Neural Networks on the Fly, by Doyen Sahoo, Quang Pham, Jing Lu, Steven C. H. Hoi, IJCAI 2018. 

summary_of_the_review: At this stage, I am happy for this paper to be accepted.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a continual learning method by constructing low-coherence subspace projector for each new task. Given a new task, the projector matrix is constructed by minimizing the coherence to the previous task projectors and within the current projector to be optimized, in the oblique manifold. The proposed approach is evaluated for task incremental learning tasks, especially when the number of tasks is large (e.g., 64, 150). The results show improvements over the baseline methods, including, EWC, OWM, GPM, etc. 
strength_and_weaknesses: Strength:

(1) The proposed approach tackles the challenging continual learning task, and proposed a gradient projection method, based on low-coherence, instead of orthogonality. 

(2) The experiments on Permuted MNIST, Rotated MNIST, Split CIFAR100 and Split miniImageNet show the improved results compared with some baseline methods, e.g., EWC, GPM, OWM, etc. 

Weakness:

(1)  The proposed method is based on construction of the projection matrix by minimizing the coherence of the projection matrix with respect to the projection matrices of previous task, and also within the current task projection matrix.  Based on the sect. 4.1, it is unclear how to generate the projection matrix for the first task. Moreover, for the projection matrix formulation in eq. (6), it seems that the current O_t is purely determined based on the previous task O_i (i = 1, ..., t-1). It seems that the task data (or their features) are not utilized in the determination of O_t. How to guarantee that that learned O_t is optimal without seeing the task data or their features?

(2) According to the results in Table I, the LcSP achieves significantly lower results than the compared methods, on Split CIFAR100, Split miniImageNet. But LcSP-GN and LcSP-BN are better than the compared methods. How to explain this results? It seems that the learning BN for each task is essential to the good results on these two tasks. Are these compared methods also use the normalization technique as the LcSP-GN and LcSP-BN? Moreover, the comparisons of LcSP, LcSP-GN and LcSP-BN should be also included for other datasets.

(3) The comparisons are insufficient, e.g., the Adam-NSCL is also an typical continual learning method based on orthogonal projection, which should be also compared. The compared methods in Table 1 for different datasets are not consistent, for example, GPM and OWM are not compared for the two versions of MNIST dataset.
clarity,_quality,_novelty_and_reproducibility: Writing clarity can be improved. For example, how to set the O_1 for the first task? The algorithm 1 should be revised in a more clear way. For example, line 1 in Alg. 1 is not necessary. The settings of \alpha and \beta in algorithm 1 should be clear. 
summary_of_the_review: My major concerns on this work are some unclear details, e.g., the question (1) in weakness, and insufficient comparisons in experiments. Moreover, there are some unclear points in the experiment results, especially on the effect of normalization techniques in the results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the issue of learning capacity degradation in gradient orthogonal projection methods that are used to combat catastrophic forgetting in continual learning.  The idea is to move from orthogonal subspaces to low-coherence subspaces, and a loss function is designed that accounts for gradient projections on the oblique manifold.  Experimental results show that the proposed approach can effectively combat forgetting while retaining good learning capacity, especially when there are many tasks.

 The paper addresses an important problem and is well written.  The use of low-coherence subspaces appears effective.  There is some concern on the novelty of using low-coherence subspaces, but its application in continual learning appears new.  It is slightly surprising that the projection matrices are not specific to the tasks, and better clarifications are needed on why this still proffers good performance.  Empirically, the batch normalization seems to play an important role in keeping the good performance, and it will be important to understand its interaction with low-coherence.  

justification_for_why_not_higher_score: There is some concern on the novelty of using low-coherence subspaces, but its application in continual learning appears new.  It is slightly surprising that the projection matrices are not specific to the tasks, and better clarifications are needed on why this still proffers good performance.  Empirically, the batch normalization seems to play an important role in keeping the good performance, and it will be important to understand its interaction with low-coherence.
Decision: Reject