Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a continual learning method that looks at tasks’ correlation (the authors define the correlation measure as well) to improve positive backward transfer across similar tasks. Towards this, first the authors define the notions of tasks’ similarity through subspace projections and positive correlations, and then theoretically analyze how if positive correlation and sufficient projection constraints are satisfied, an SGD update without any projection can improve the learning of previous task(s). Inspired by the theoretical analysis, the authors propose a method, CUBER, that stores the gradients of the previous tasks, and either make an update orthogonal to the gradients subspace of the previous tasks (if the previous task is not positively correlated) or make un (approximately) unchanged update if the previous task is positively correlated with the current task. Experiments are conducted on 4 standard CL benchmarks and it has been shown that the proposed method consistently provides a positive backward transfer for different benchmarks. 

strengths_and_weaknesses: Strengths
-----------

* The paper is written really well and the final model is well motivated. 
* The paper develops nice intuitions regarding different components of the proposed method. 
* Experiments are thoroughly conducted to establish the claims of the paper. 

Weaknesses
-----------

* Comparison with GEM/ A-GEM: It seems to me that at least some ideas around checking the positive correlation and making the gradient updates based on that are similar to what GEM/ A-GEM inspired. While the authors of the present manuscript mention A-GEM and state that ‘... In stark contrast, the task gradient correlation characterization in [AGEM] involves the gradient evaluation of old tasks with respect to the current model weight, and hence requires the data of old tasks when learning the new task’, I feel the difference is superficial. My understanding of GEM/ A-GEM is that the authors chose to keep the data of previous tasks and compute the gradients of previous tasks at each new step of the current task, because storing previous data is cheap in terms of memory compared to storing the gradients. One could equivalently store the gradients in GEM/ A-GEM and their gradient projection step will not change. I feel that the proposed CUBER is an inefficient version of GEM/ A-GEM (in terms of memory). It is not clear to me what was the replay buffer size that the authors of the present manuscript used with A-GEM and ER. For a fair comparison, I would fix the memory size in MBytes and use equivalent memory for CUBER and A-GEM/ ER and then compare their numbers in Table 1. 

* First method to achieve positive forward transfer with a fixed capacity network: This claim is wrong! As stated above, since the update procedure of facilitating the positive backward transfer is very similar to GEM, GEM would also achieve the positive backward transfer. In fact, GEM indeed achieves the positive backward transfer on MNIST permutation and CIFAR100 (cf. Fig. 1 in https://arxiv.org/pdf/1706.08840.pdf). 

* Memory/ time comparison: I would like to see the memory and time comparison with other baselines. 

While I appreciate the mathematical elegance of the paper, it feels a bit distracting. For example, it is intuitively very easy to see when the learning of a new task could help in learning the previous task (in fact this is precisely the GEM’s update rule). It is not clear to me if Sec 3 is needed in all its glory. I also feel the learnings of Sec 3, for example the question about when Rule #2 can safely be used to update the model, are not applied in the final model – in regime 3 the authors still don’t use the rule #2 even when there is a positive correlation among the present and previous tasks. Given this, I would appreciate it if the authors could focus more on the intuition of the method itself – why the scaled projection would lead to forward transfer, how their method is different from GEM/ A-GEM etc. 

questions: Please respond to the weaknesses section above. 

limitations: See the weaknesses section above. 


--------
I have revised my score post-rebuttal. For detailed justification of this revision, see my response to the authors' rebuttal.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: To tackle the problem of incrementally learning a sequence of tasks with a deep neural network, this paper proposes the establishment of three different “regimes” with each their own update rule. At the start of learning a new task, each layer of the network is assigned to one of these regimes based on a carefully motivated set of decision rules and thresholds. Regime 1 (most protective of old tasks) uses an update rule similar to the rule proposed by Saha et al. (2021, ICLR), regime 2 uses an update rule similar to the rule proposed by Lin et al. (2022, ICLR) and regime 3 (least protective of old tasks) uses a newly proposed update rule that is hoped to allow for positive backward transfer. In an empirical comparison, the proposed method is shown to achieve positive backward transfer on two task-incremental learning benchmarks, while none of the compared methods do so.
strengths_and_weaknesses:  believe this is an interesting and insightful paper, and as such potentially suitable for publication at NeurIPS. However, as will become clear below, at the moment there are too many issues preventing me to support acceptance of this paper, but I would be happy to reconsider a suitably revised version.

Due to insufficient experimental details, I would not be able to reproduce the experiments in this paper. For example, the following important experimental details are not reported (or at least I was not able to find them):
- What optimizer / learning rate is used for each of the experiments, and based on what was this selected?
- How were “method-specific hyperparameters” selected for the various compared methods (e.g., “lambda” for EWC)?
- What was the size of replay buffer for AGEM and ER_Res? Why do the authors select this buffer size?

Another question about the experimental details: it is stated that on several of the benchmarks early stopping is used based on the validation loss. Which validation loss is used for this? (e.g., the loss on the current task? the loss on the tasks seen so far?)

It seems that most of the experimental results reported in Table 1 have been taken from (an)other paper(s) without proper acknowledgement. Many of the results are exactly the same as those reported by Lin et al. (2022, ICLR) and Saha et al., (2021, ICLR). Code also seems to be provided only for the method proposed in this paper, and not for the other methods included in the experimental comparison. If the authors did not run all of the reported results themselves, this should be clearly stated (and it should be acknowledged where the results come from) and it should be explained how it is ensured that all comparisons are still fair.

I believe the most interesting method to compare against is Natural Continual Learning proposed by Kao et al. (2021, Neurips; https://arxiv.org/abs/2106.08085). This method also tackles continual learning by putting restrictive constraints on the optimization of new tasks, but, unlike the present paper, uses an approximation of the Fisher Matrix of old tasks for this. Intuitively, this should allow for more positive backward transfer than using an orthogonal projection onto the input subspaces of olds tasks. I would like to encourage the authors to compare against this paper, ideally both conceptually and empirically.

A central claim of the paper relates to backward transfer: that the proposed method is the first method to achieve this on some of the considered benchmarks. The backward transfer that is achieved however appears to be very small. It would be helpful if the authors could provide an intuition as to what the reported backward transfer values mean. In particular, does the amount of achieved backward transfer have any practical significance?

Other issues:
- The authors do not discuss or reflect on the limitations of their proposed approach.
- I got confused by the term “Single-head” in Table 1. In the literature this term is typically used as a synonym for class-incremental learning, but this does not seem to be the case here. I thought that here this term might instead be used as a description of the type of output layer, but as the method HAT is not included with this benchmark, that also does not seem to be the case. (Or at least it is not the entire explanation, as I can’t see why HAT cannot be run with a single-headed output layer?) It seems the term is used to indicate that task labels are not available at test time, but for that I recommend the authors to instead use the terms task-incremental, domain-incremental and class-incremental (e.g., as in Kao et al., 2021; NeurIPS), to avoid the ambiguity of the term single-head.
- In L63-L64 the authors talk about “all the baseline CL methods”. This is too strong and should be moderated.

---

Following the rebuttal, I raised my "rating" from 3 to 4. See my response to the authors' rebuttal for more details.

questions: Please see the comment above for several questions.

limitations: It appears the authors have made no attempt at all to address the limitations of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper investigates the gradient space for backward knowledge transfer. Starting from the question of how can improve the previously learned model using new incoming data, it studies the criteria for backward transfer. It proposed a novel continual learning method with backward knowledge transfer. The experiments showed the effectiveness of the method. 
strengths_and_weaknesses: Strengths:
- strong theoretical analysis of backward knowledge transfer. 

Weaknesses: 
- large storage consumption for saving the network gradients for backward transfer. 
questions: Could you explain the storage consumption to save gradient information of the previous model?  
limitations: large storage consumption.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a continual learning model in which the model focuses on backward knowledge transfer and learns new tasks while, when relevant, improving the learning of old tasks. The latter relies on identifying the similarities between new tasks and old tasks by analyzing the gradient projections. The model is applied to a fixed capacity network without replay. The controls and optimizations are applied to each layer separately. 

This is an interesting work and, as the authors have mentioned, focuses on an area of continual learning which is less explored compared with several existing methods for forward learning. The model has been applied to several imaging benchmarks, and the performance of the model is compared with several baseline models.


strengths_and_weaknesses: *Strengths


The idea of backward learning and improving (the generalisability) of the old tasks is interesting, and the evaluations show that the model achieves better results than the existing solutions. 

The assumptions, experimental settings, and the results are well presented and discussed. 

The paper is well written and easy to follow. The difference between the current approach and existing solutions and related work are explained well. 


*Weaknesses  

This model is largely designed based on the assumption that at least some of the tasks are similar or very similar, which makes sense; otherwise, there won’t be much backward knowledge transfer. However, this seems to also depend on having clear boundaries between the tasks. In other words, it is unclear if the model would perform well in the drifts and task changes that happen gradually. With imaging benchmarks often, it is not easy to evaluate this use case. 

In construction, the base extraction an SVD technique is applied; for wide network and high dimensional data, this could be computationally very expensive to calculate this for the test of inputs; considering some of the improvements are incremental; what would be the trade-off between this high complexity and the overall improvement? 

The authors are right that the backward learning models are relatively less explored; however, it would be helpful if they could compare their work with an existing model that explicitly addresses backward learning and task similarity in continual learning (e.g. Zixuan et al., 2022) – ref [10] in the paper. 

questions: Is there any difference between the gradient project analysis in this work and the technique that is proposed in trust region gradient projection (TRGP) (Lin et al., 2022)? – ref [16] in the paper. 

What is HBK in the denominator of the equation shown in Theorem 1? 

For an L-layer network, CUBER characterizes the task correlation for each layer and then uses different strategies to decide how to learn a new task. Is there a way to quantify the increased complexity when the number of tasks grows? One may assume that if there are several tasks and if there are several layers in the network and then a subset of these layers become specifically relevant to one task, then this should require an increasing number of parameters to be stored; would it be possible to elaborate on or quantify the complexity? 

limitations: A limitation of the work could be the complexity of calculating SVD and assuming clear boundaries between the tasks. It would have been helpful to include a brief section to discuss these and other limitations of the work. 

The societal or impact are not discussed explicitly in the paper; however, the gain in this model comes are at a relatively significant level of complexity; it would be helpful to explore what real-world cases would benefit from this additional complexity and what level of compute/memory will require to train a model with several tasks.
review_title: null

Meta-review: metareview: This paper develops a method to improve backward transfer in continual learning without replay. In particular, upon learning a new task the proposed method also updates parts of the model responsible for the performance of the old tasks. 

The initial reviews for this paper were quite mixed. All reviewers agreed that parts of the method were novel, but reviewers also found important limitations of the initial manuscript. In the discussion, the authors were able to clarify many questions/criticisms notably with respect to related work as welll as to the exact experimental setup. The updated version of the manuscript now provides much needed details (which should help with reproducibility).

After the discussions, some questions regarding the significance of the results and the differences with respect to GEM remain. 

Regarding significance of the results, reviewer nsyh outlines that while improving backward transfer is significant, the level of improvements shown in the paper are modest and could be a result of overfitting. In particular, the method seems to outperform baselines, but mostly does not achieve positive backward transfer. The problem of backward transfer seems difficult and it is unclear if positive backward transfer is even achievable in most cases (e.g., it seems to rely on having sufficiently high similarity between tasks). In that sense, I find the results to be reasonable.

Regarding GEM, there is some disagreement about the level of similarity of this work compared to GEM and whether a simple extension of GEM could compare favourably to this method. Studying this extension of GEM would be improve this paper, but I don't see it as a requirement. This perspectivee is shared by two reviewers.

Overall, this remains a very borderline paper, one that studies an important problem, but for which reviewers disagree on significance and even to a certain extent novelty given existing work. Nonetheless, methods for improving backward transfer have not been studied much in continual learning and so this paper might spark additional work in the area. Further, the paper has no major weaknesses. Based on this I am happy to recommend this paper be accepted. 

I strongly encourage the authors to incorporate elements of the discussion you had with the reviewers into account when preparing the camera-ready version of their work. In particular, it would be useful to provide more intuitions regarding the method especially in light of related works (Saha et al.'21, Lin et al.'22, Kao et al.'21, and GEM Lopez-Paz et al.'17 + A-GEM Chaudhry et al.'19). Considering the extension of GEM would also provide an interesting baseline.
confidence: Less certain
Decision: Accept