Title: Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer
Authors: Sen Lin
Li Yang
Deliang Fan
Junshan Zhang
Venue: NeurIPS 2022

Abstract: By learning a sequence of tasks continually, an agent in continual learning (CL) can improve the learning performance of both a new task and `old' tasks by leveraging the forward knowledge transfer and the backward knowledge transfer, respectively. However, most existing CL methods focus on addressing catastrophic forgetting in neural networks by minimizing the modification of the learnt model for old tasks. This inevitably limits the backward knowledge transfer from the new task to the old tasks, because judicious model updates could possibly improve the learning performance of the old tasks as well. To tackle this problem, we first theoretically analyze the conditions under which updating the learnt model of old tasks could be beneficial for CL and also lead to backward knowledge transfer, based on the gradient projection onto the input subspaces of old tasks. Building on the theoretical analysis, we next develop a ContinUal learning method with Backward knowlEdge tRansfer (CUBER), for a fixed capacity neural network without data replay. In particular, CUBER first characterizes the task correlation to identify the positively correlated old tasks in a layer-wise manner, and then selectively modifies the learnt model of the old tasks when learning the new task. Experimental studies show that CUBER can even achieve positive backward knowledge transfer on several existing CL benchmarks for the first time without data replay, where the related baselines still suffer from catastrophic forgetting (negative backward knowledge transfer). The superior performance of CUBER on the backward knowledge transfer also leads to higher accuracy accordingly.

TL;DR: We propose a novel continual learning method to facilitate backward knowledge transfer, which can even achieve positive backward transfer on standard CL benchmarks for the first time.

Review discussion:
*Strengths
The idea of backward learning and improving (the generalisability) of the old tasks is interesting, and the evaluations show that the model achieves better results than the existing solutions.
The assumptions, experimental settings, and the results are well presented and discussed.
The paper is well written and easy to follow. The difference between the current approach and existing solutions and related work are explained well.
*Weaknesses
This model is largely designed based on the assumption that at least some of the tasks are similar or very similar, which makes sense;
otherwise, there won’t be much backward knowledge transfer.
However, this seems to also depend on having clear boundaries between the tasks.
In other words, it is unclear if the model would perform well in the drifts and task changes that happen gradually. With imaging benchmarks often, it is not easy to evaluate this use case.
In construction, the base extraction an SVD technique is applied; for wide network and high dimensional data, this could be computationally very expensive to calculate this for the test of inputs; considering some of the improvements are incremental; what would be the trade-off between this high complexity and the overall improvement?
The authors are right that the backward learning models are relatively less explored;
however, it would be helpful if they could compare their work with an existing model that explicitly addresses backward learning and task similarity in continual learning (e.g. Zixuan et al., 2022) – ref [10] in the paper.
Is there any difference between the gradient project analysis in this work and the technique that is proposed in trust region gradient projection (TRGP) (Lin et al., 2022)? – ref [16] in the paper.
What is HBK in the denominator of the equation shown in Theorem 1?
For an L-layer network, CUBER characterizes the task correlation for each layer and then uses different strategies to decide how to learn a new task. Is there a way to quantify the increased complexity when the number of tasks grows? One may assume that if there are several tasks and if there are several layers in the network and then a subset of these layers become specifically relevant to one task, then this should require an increasing number of parameters to be stored; would it be possible to elaborate on or quantify the complexity?
A limitation of the work could be the complexity of calculating SVD and assuming clear boundaries between the tasks. It would have been helpful to include a brief section to discuss these and other limitations of the work.
The societal or impact are not discussed explicitly in the paper;
however, the gain in this model comes are at a relatively significant level of complexity; it would be helpful to explore what real-world cases would benefit from this additional complexity and what level of compute/memory will require to train a model with several tasks.
No
Q6: Would it be possible to elaborate on or quantify the complexity?
A6: As the reviewer mentioned, the number of stored parameters will increase with the number of tasks. This is actually a common issue in many CL methods.
For example, in parameter-isolation based methods (e.g., [24][31]) where separate masks are usually learnt and stored for each task to protect the task-specific weights, the required memory size will clearly increase with the number of tasks. And in memory-based methods (e.g., [6][17][25]) where task-specific information (e.g., bases and scaling matrices) needs to be stored, the number of stored parameters will also increase with the number of tasks. For CUBER, one way to alleviate this issue is to gradually remove the least important bases for each old task in the memory when learning more tasks. We have also added the memory and training time comparison in Figure 4 and Table 6 in Appendix D.4 of the revision.
Q7: Impact and limitations.
A7: Thank you for the insightful suggestions! We have added a paragraph on impact and limitations in section 6 of the revision.
Thank you for your thorough reviews and constructive comments.
Per your suggestions, we have made the following major revision: (1) added experimental details including the list of hyperparameters in Appendix D.1-D.3,  (2) added the memory comparison in Figure 4 in Appendix D.4, (3) added training time comparison in Table 6 in Appendix D.4, (4) added discussion of impact and limitations in Section 6, and (5) made various revisions throughout the paper based on all reviewers’ comments (the reference indices have changed). All our changes are highlighted with blue-colored texts. New comments on these changes are very welcome!
Q1: It is unclear if the model would perform well in the drifts and task changes that happen gradually.
A1: Following a lot of CL studies in the literature, we consider the scenario where the task boundaries exist and are known.
When the data distribution drifts gradually, how to appropriately define and quantify the task correlation is very important and clearly nontrivial. We thank the reviewer for pointing out this interesting direction, and will study the backward knowledge transfer under more general CL setups in the future work.
Q2: What would be the trade-off between the complexity and the overall improvement?
A2: We have the following clarifications about the tradeoff between complexity and performance:
- The SVD computation indeed does not increase the complexity a lot as shown in Figure 4 and Table 6 in Appendix D.4 of the revision about the performance of GPM (Saha et al., 2021 [25]) and TRGP (Lin et al., 2022 [17]). This is mainly because the SVD computation for extracting bases is only performed once for each task and the gain in the simple and fast model learning compensates the time cost of SVD.
- CUBER has comparable complexity with most considered baseline methods, but achieves much better performance. Compared to TRGP (Lin et al., 2022 [17]), while some of the improvements are incremental, the increase in the computational complexity due to storing gradient information is actually marginal and controllable. More detailed comparisons can be found in Appendix D.4 of the revision.
Q3: Comparison with Zixuan et al. 2022 [11].
A3: Zixuan et al. 2022 [11] employs a complicated task similarity detection mechanism before learning each new task, where 1) a reference model is trained for the new task from scratch and 2) a transfer model is trained for each old task.
In contrast, our method only needs one forward-backward pass for all layers to evaluate the task similarity based on the stored bases and gradient information of the old tasks. Because additional networks are needed in [11], it is difficult to ensure fair comparisons between our method and the method in [11], and also impossible to implement the method in [11] with a fixed capacity network without nontrivial modifications. Per the reviewer's suggestion, we will try to find an appropriate way to compare CUBER with [11] in terms of the performance about backward knowledge transfer.
Q4: Difference between the gradient project analysis in this work and the technique in TRGP.
A4: Lin et al., 2022 [17] proposes TRGP to facilitate forward knowledge transfer to the current task by reusing the frozen weights of most correlated old tasks through a scaling matrix, where the model is still updated along the direction orthogonal to the input subspaces of old tasks to mitigate forgetting. In this case, the learnt model for all old tasks will not be modified.
In contrast, our method seeks to identify the old tasks for which modifying the learnt model using the data of the current task can be actually beneficial, so as to address forgetting and achieve backward knowledge transfer.
TRGP is more related to regime 2 where we should not modify the learnt model for the selected old tasks but be able to leverage the scaling matrix to facilitate forward transfer.
Q5: What is HBK in the denominator of the equation shown in Theorem 1?
A5: It is the product among the Lipschitz constants $H$, $B$, and the number of iterations $K$ for each task.
Strengths:
- strong theoretical analysis of backward knowledge transfer.
Weaknesses:
- large storage consumption for saving the network gradients for backward transfer.
Could you explain the storage consumption to save gradient information of the previous model?
large storage consumption.
No
Thank you for your thorough reviews and constructive comments.
Per your suggestions, we have made the following major revision: (1) added experimental details including the list of hyperparameters in Appendix D.1-D.3,  (2) added the memory comparison in Figure 4 in Appendix D.4, (3) added training time comparison in Table 6 in Appendix D.4, (4) added discussion of impact and limitations in Section 6, and (5) made various revisions throughout the paper based on all reviewers’ comments (the reference indices have changed). All our changes are highlighted with blue-colored texts. New comments on these changes are very welcome!
Q1: The storage consumption to save gradient information of the previous model.
A1: We compute the average gradient over a batch of data after learning each task together the bases extraction. After learning each task, most elements in its gradient matrix for each layer can be close to zero.
When evaluating the conditions for regime 3, we flatten the gradient matrix into a vector for each layer, e.g., a gradient matrix in $\mathbb{R}^{m\times n}$ to a gradient vector in $\mathbb{R}^{mn}$. The inner product between gradient matrices hence becomes the inner product between gradient vectors, where the near-zero elements have little impact on the value of the product.
Therefore, to save the computation cost and memory, we prune the layer-wise average gradient vector after learning each task with a certain sparsity ratio, and only store a gradient vector with non-zero elements and the corresponding indices in the original gradient vector.
Therefore, compared to the previous studies (e.g., TRGP) which also store the bases for the input subspaces, the memory increase introduced by saving the gradient information is marginal and controllable. More details can be found in Appendix D.4 of the revision.
Thanks for the rebuttal. After reading other reviews and your rebuttal, I found that I missed the similarity between the proposed CUBER and the previous GEM/A-GEM. I have changed my rate to borderline rejection.
Thanks for your reply.
In terms of the comparison with GEM, the reviewer nMSj is clearly wrong and has some misunderstanding of our approach.
- First, it is not fair and reasonable to compare two approaches that study different setups.
- Second, our approach provides a more efficient and fine model update, i.e., updates different components of the model with respect to different old tasks, based on characterizing the task correlation between the current task and old tasks once before learning the current task.
For example, for old tasks in regime 3, we modify their corresponding models using plain SGD. Without characterizing the task similarity, GEM has to employ an inefficient model update by computing multiple gradient inner products using raw data for all previous tasks at each step.
Thanks for the clarification. I see the difference. The proposed method CUBER works without data reply, while GEM/A-GEM is a replay-based approach. I will return to my original score.
Thank you for agreeing with us! And we also hope that our clarifications on the storage consumption and computational cost can address your previous concern.
believe this is an interesting and insightful paper, and as such potentially suitable for publication at NeurIPS.
However, as will become clear below, at the moment there are too many issues preventing me to support acceptance of this paper, but I would be happy to reconsider a suitably revised version.
Due to insufficient experimental details, I would not be able to reproduce the experiments in this paper.
For example, the following important experimental details are not reported (or at least I was not able to find them):
- What optimizer / learning rate is used for each of the experiments, and based on what was this selected?
- How were “method-specific hyperparameters” selected for the various compared methods (e.g., “lambda” for EWC)?
- What was the size of replay buffer for AGEM and ER_Res? Why do the authors select this buffer size?
Another question about the experimental details: it is stated that on several of the benchmarks early stopping is used based on the validation loss. Which validation loss is used for this? (e.g., the loss on the current task? the loss on the tasks seen so far?)
It seems that most of the experimental results reported in Table 1 have been taken from (an)other paper(s) without proper acknowledgement. Many of the results are exactly the same as those reported by Lin et al. (2022, ICLR) and Saha et al., (2021, ICLR). Code also seems to be provided only for the method proposed in this paper, and not for the other methods included in the experimental comparison. If the authors did not run all of the reported results themselves, this should be clearly stated (and it should be acknowledged where the results come from) and it should be explained how it is ensured that all comparisons are still fair.
I believe the most interesting method to compare against is Natural Continual Learning proposed by Kao et al. (2021, Neurips; https://arxiv.org/abs/2106.08085). This method also tackles continual learning by putting restrictive constraints on the optimization of new tasks, but, unlike the present paper, uses an approximation of the Fisher Matrix of old tasks for this. Intuitively, this should allow for more positive backward transfer than using an orthogonal projection onto the input subspaces of olds tasks. I would like to encourage the authors to compare against this paper, ideally both conceptually and empirically.
A central claim of the paper relates to backward transfer: that the proposed method is the first method to achieve this on some of the considered benchmarks. The backward transfer that is achieved however appears to be very small. It would be helpful if the authors could provide an intuition as to what the reported backward transfer values mean.
In particular, does the amount of achieved backward transfer have any practical significance?
Other issues:
- The authors do not discuss or reflect on the limitations of their proposed approach.
- I got confused by the term “Single-head” in Table 1. In the literature this term is typically used as a synonym for class-incremental learning, but this does not seem to be the case here. I thought that here this term might instead be used as a description of the type of output layer, but as the method HAT is not included with this benchmark, that also does not seem to be the case. (Or at least it is not the entire explanation, as I can’t see why HAT cannot be run with a single-headed output layer?) It seems the term is used to indicate that task labels are not available at test time, but for that I recommend the authors to instead use the terms task-incremental, domain-incremental and class-incremental (e.g., as in Kao et al., 2021; NeurIPS), to avoid the ambiguity of the term single-head.
- In L63-L64 the authors talk about “all the baseline CL methods”. This is too strong and should be moderated.
Following the rebuttal, I raised my "rating" from 3 to 4. See my response to the authors' rebuttal for more details.
Please see the comment above for several questions.
It appears the authors have made no attempt at all to address the limitations of their work.
No
Q6: Limitations.
A6: Thank you for the suggestions. We have discussed the impact and limitations in section 6 of the revision.
- (1) As in recent orthogonal-projection based CL methods, e.g., Saha et al. (2021 ICLR) and Lin et al. (2022 ICLR),
CUBER extracts the bases of the task subspaces based on SVD, which may lead to high computational cost for large dimensional data, and the complexity also increases with the number of tasks. How to reduce the complexity is an interesting direction. (2) As mentioned by Reviewer B6XQ, we assume that clear task boundaries exist between different tasks. In the future work, we will study how to extend CUBER to more general CL settings.
Q7: ''Single-head" in Table 1.
A7: Thank you for the suggestion. The concept of "Single-head" means that all the tasks share one output layer and there is no task identifier during testing. It could refer to the class incremental learning. We have clarified this in the first paragraph of section 5.1 of the revision.
Q8: Claim about ''all the baseline CL methods".
A8: Thank you for pointing this out. We have modified the claim to 'all the considered baseline CL methods', as shown in section 1 of the revision.
Q4: Comparison against Natural Continual Learning proposed by Kao et al. (2021 NeurIPS).
A4:  Thank you for pointing out this related work. We would like to 'conceptually' compare with this work, in the sense that our objective and strategy are both different from those in Kao et al..
- Kao et al. seeks to restrict the model update for the current task to directions (subspace) which do not interfere strongly with previous tasks. This indeed follows the same line as in OWM (Zeng et al., 2019 [32]) and GPM (Saha et al., 2021 [25]), but studies the problem about how to select the projected subspace in a way that appropriately balances learning on previous and current tasks. To this end, their method not only updates the model in the direction orthogonal to the steep directions of the prior on the current task, but also includes an additional regularization term to incorporate prior information into the stationary points. Intuitively, they still try to avoid the model changes for the old tasks.
- In stark contrast, out study does not avoid the model changes for all old tasks, but seek to selectively modify the learnt model for old tasks to facilitate the backward knowledge transfer. The theoretical results in Section 3 justified that when the old task has strong positive correlation with the current task in terms of the correlation notions we defined, modifying the learnt model for that old task is actually beneficial. This consequently motivates our algorithm design, where we first characterize the layer-wise task correlation into three different regimes to determine the subspaces on which we should either freeze or modify the gradient projection, and then take different strategies to learn the new task for better forward and backward knowledge transfer.
In fact, the method in Kao et al. is more related to the component where we modify the gradient by subtracting the gradient projection on the subspaces for the old tasks in regime 1 and 2 to minimize the interference to those old tasks.
However, our method is more than that by providing a finer model update for the current task based on the characterization of layer-wise task correlations before learning the current task.
We have included this work in the related work section in the revision per the reviewer's suggestion, and will conduct the experimental comparison with it in the future revision.
Q5: Intuition behind backward transfer.
A5: We have the following clarifications about backward knowledge transfer. Related modifications can also be found in section 5.2 in the revision.
- In CL, the learning agent is expected to be able to 1) not only learn a new task more easily by leveraging the accumulated knowledge from old tasks (forward knowledge transfer), 2) but also further improve the learning performance of old tasks based on the gained knowledge of related new tasks (backward knowledge transfer). Given a fixed neural network without data replay, the state-of-the-art CL methods still suffers from catastrophic forgetting with negative backward knowledge transfer, which has not met the second goal of CL. Clearly, achieving backward knowledge transfer is very challenging in this setup and it remains unclear if and how the backward knowledge transfer can be achieved.
To the best of our knowledge, this work makes the first attempt to study the problem above under a fixed capacity neural network without data replay, and proposes a method that meets the goal of CL. More importantly, we hope that this work will serve as initial steps and motivate further research in CL community on the important while less explored problem, i. e. , how to design algorithms that can provide targeted treatments to achieve backward knowledge transfer.
- The value of backward knowledge transfer indicates the average accuracy improvement for each old task after learning all tasks, which implies that the new task learning provides additional useful information for learning features of similar old tasks. Intuitively, this value (and also the value of forward knowledge transfer) should depend on the task similarity in CL.
For the experimental setup in this work, we follow the literature in CL and consider the standard benchmarks, e.g., for Split CIFAR-100 the tasks belong to the same dataset CIFAR-100 but have non-overlapping classes. In this setup, the tasks should share similar lower-level features but differ in high-level features, and the corresponding task similarity is lower compared to the task setup in OL-CIFAR100 (where we allow some tasks to have overlapping classes).
Therefore, the value of backward transfer in Split CIFAR-100 is smaller than that in OL-CIFAR100 (as shown in Table 1 and Table 2(a)). For task sequences with more similar tasks, we expect larger values of backward transfer, as also pointed out by Reviewer B6XQ.
Thank you for your thorough reviews and constructive comments.
Per your suggestions, we have made the following major revision: (1) added experimental details including the list of hyperparameters in Appendix D.1-D.3,  (2) added the memory comparison in Figure 4 in Appendix D.4, (3) added training time comparison in Table 6 in Appendix D.4, (4) added discussion of impact and limitations in Section 6, and (5) made various revisions throughout the paper based on all reviewers’ comments (the reference indices have changed). All our changes are highlighted with blue-colored texts. New comments on these changes are very welcome!
Q1: Insufficient experimental details.
A1: Per the reviewer's suggestions, we have included more experimental details, which can be found in Appendix D in the revision.
(Optimizer/learning rate): For all experiments, we adapt the SGD optimizer by modifying the gradient based on the optimization problem (3). This is consistent with the orthogonal-projection based methods (e.g., [6, 25, 17]) where the gradient direction used in the plain SGD optimizer is modified to minimize the interference to the old tasks.
The learning rates for different datasets are shown in Table 5 of Appendix D.3 in the revision. More specifically, for PMNIST we use a fixed learning rate of 0.01, for CIFAR-100 we use a decaying learning rate with initial rate of 0.01, while for both MiniImageNet and 5-Dataset we use a decaying learning rate with initial rate of 0.1.
(Method-specific hyperparameters): We have added a full list of hyperparameters of all methods on all benchmarks in Table 5 of Appendix D.3 in the revision.
(Size of replay buffer for AGEM and ER_Res): Following the experimental setups in Saha et al. 2021 [25], the reported results of A-GEM and ER_Res both use a sample size of 1000, 2000, 500, 3000 for PMNIST, CIFAR-100, miniImageNet and 5-Dataset, respectively. We have included the detailed hyperparameters in Table 5 of Appendix D.3 in the revision.
Q2: Which validation loss is used for early stopping?
A2: For all experiments, the minimum learning rate is set to 1e-5 and the learning rate decay factor is set to 2. After each model update, we evaluate the validation loss for the current task using its validation dataset w.r.t. the current model, and count the times whenever the validation loss increases.
When the counter is greater than 6, we divide the learning rate by 2 and reset the counter to 0. The training process will be stopped when the learning rate decays to the minimum learning rate.
Q3: Experimental results based on Lin et al. (2022 ICLR) and Saha et al. (2021 ICLR).
A3: Thank you for the suggestions. To ensure fair comparisons, we implement CUBER based on the released code of GPM (Saha et al. 2021) and TRGP (Lin et al. 2022), and follow the exactly same experimental setups. We reproduced the reported results of GPM and TRGP using the provided hyperparameters in the papers.
Therefore, for  other baseline methods considered in this work, we directly follow these two papers and use the reported results therein.  For reference, as mentioned in Saha et al. 2021 [25], the implementations of  A-GEM, ER_Res and OWM were based on their released official code, respectively, while EWC and HAT were implemented based on Serra et al. 2018 [26].
We have clarified this and added the acknowledgement accordingly in Appendix D.2 in the revision.
I would like to thank the authors for their response. I need a bit more time for a more detailed response, but there are already two points that I am able to highlight now:
- My comment about why the term “singlehead” is confusing in the context of this paper was not very clear, apologies for that. What I meant was that in the literature the term “singlehead” is typically used to refer to a class-incremental learning type problem (e.g., as it is used here: https://openaccess.thecvf.com/content_ECCV_2018/html/Arslan_Chaudhry__Riemannian_Walk_ECCV_2018_paper.html).
However, in the current paper, unless I am mistaken, the term is used to refer to a domain-incremental learning type problem. These are fundamentally different types of incremental learning problems, and I think it is important to be clear about this. A useful reference for the authors might be https://arxiv.org/abs/1904.07734
- I disagree with the authors’ interpretation that the Kao et al. paper only tries to avoid model changes for the old tasks. In principle, the Bayesian framework used by this paper is capable of using knowledge observed in new tasks to improve performance on old tasks. Note that this also means that, at least in theory, a method such as EWC also does not, as the authors suggest, solely focusses on avoiding model changes for the old tasks. I would be willing to accept a statement along the lines that the current paper *more explicitly* attempts to update the models of past tasks when relevant new information becomes available, but as currently formulated several of the statements are too strong.
Thank you so much for your quick response. We have further revised the paper based on your helpful suggestions.
Q1: About the term "single-head".
A1: Since we consider the same experimental setup as in GPM (Saha et al. 2021), we used the same term "single-head" following GPM. As a follow-up of the clarification by the reviewer, we also agree that the term here refers to domain-incremental learning (following the suggested reference https://arxiv.org/abs/1904.07734) and have revised the paper accordingly.
Q2: Comparison with Kao et al..
A2: Thank you for your insightful comments. In principle, the Bayesian framework proposed in Kao et al. implicitly leverages the knowledge of new task to improve the performance for the old tasks (because of the Bayesian nature). Taking one step further, to achieve backward knowledge transfer, our method seeks to modify the learnt model of old tasks in a more explicit way, by characterizing the conditions under which modifying the learnt model using the current task data is beneficial for the old tasks. Per the reviewer's suggestions, we have further clarified this in the revision.
Thanks to the authors for taking the time to address my questions and concerns.
I appreciate the added experimental details, the provided clarification and acknowledgements for re-use of previous results and the discussion of limitations.
I have increased my “rating” for this paper, but I’m afraid that I still do not support acceptance. This is partly because I agree with some of the points raised by reviewer nMSj, but also because I am not convinced by the significance of the results. As raised in my initial review, the reported values for BWT are rather small. In my interpretation, the achieved backward transfer is not substantial or of any real-world value. It could be countered that at least some of the reported BWT values are statistically significant above zero (the first ones in a particular setting, as the authors claim), but I think this might very well be a reflection of the overfitting on this particular benchmark (where the overfitting is not the just the results of the efforts of the current paper, but also the efforts of the papers it builds upon).
In particular, I have now had the chance to have a better look at the Saha et al. (2021, ICLR) and Lin et al. (2022, ICLR) papers, and I realize there is more of a similarity than I initially appreciated.
Although the proposed modifications relative to these papers are reasonably well motivated by the current paper, now that I am more familiar with these previous papers the modifications also appear somewhat hacky, while some of the things of the current paper I considered elegant where more or less directly borrowed from these previous papers.
Re-reading the above comment I realize this might sound harsher than I meant. I want to stress that I think that this is a decent paper with a contribution to the literature, but in my opinion just not good enough to support acceptance at NeurIPS.
Thank you for your detailed reply. We have the following clarifications:
(1) The value of backward transfer:
- It has been widely shown in the literature that achieving zero forgetting is already very challenging under a fixed-capacity network without data replay, whereas achieving positive backward knowledge transfer is clearly more challenging. We disagree with the reviewer that "the achieved backward transfer is not substantial or of any real-world value".
**We believe that even a small step towards achieving positive backward knowledge is meaningful and encouraging in the setup considered in this paper (especially when the tasks have non-overlapping classes), which also helps us to further understand the capacity limit for the neural network in CL and improve the performance by facilitating better backward knowledge transfer. More importantly, we hope that our work could serve as the initial step and motivate the CL community to explore the backward knowledge transfer more directly, which is obviously less explored but critical for achieving the ultimate goal of CL.**
- As we explained earlier, the value of backward knowledge transfer highly depends on the level of task similarity. It is not reasonable to justify the importance of the value without considering the task similarity. In the experimental setup considered in this work, which is also the standard benchmark used in many CL papers, the tasks have non-overlapping classes, in which case expecting a large value of backward knowledge transfer is simply non-realistic. As shown in our ablation study, the value of backward knowledge transfer increases when the tasks become more similar, which clearly justifies the intuition above.
(2) Comparison with Saha et al. 2021 and Lin et al. 2022: We are very familiar with these two papers, "perhaps as good as the authors". Generally speaking,
- Saha et al. 2021 seeks to minimize the interference from the current task to all old tasks by using orthogonal gradient projection (the idea of orthogonal gradient projection has already been explored before in [6, 32]), but without considering anything about the task similarity.
- Lin et al. 2022 follows the same line of orthogonal gradient projection because of its superior performance in a fixed capacity network without data replay, but proposes to leverage the task similarity to facilitate the forward knowledge transfer. This can help to improve the learning of the current task, but intuitively not benefit the learning of the old tasks because of the orthogonal gradient projection (protect the weights of old tasks by minimizing the introduced interference from the current task).
- Our work takes one step further beyond these two papers, by leveraging a more comprehensive characterization of task similarity to facilitate the backward knowledge transfer. In the meanwhile, we build our method on the powerful orthogonal gradient projection by following the line of this strategy in the research community, to minimize the interference to the old tasks that are less correlated with the current task. **The objective of directly improving the backward knowledge transfer has not been explored in Saha et al. 2021 and Lin et al.
2022 (and even less explored in the CL community), and in fact the idea of explicitly and selectively modifying the learnt models for old tasks to facilitate better backward knowledge transfer is also relatively new in the literature for the setup with a fixed capacity network and no data replay. Our paper also provides a theoretical foundation on the conditions under which we should modify the learnt model of old tasks, which is also new to the community in the setup considered here. **
(3) Comparison with GEM: The reviewer nMSj is clearly wrong and has some misunderstanding of our approach.
First, it is not fair and reasonable to compare two approaches that study different setups.
Second, our approach provides a more efficient and fine model update, i.e., updates different components of the model with respect to different old tasks, based on characterizing the task correlation between the current task and old tasks once before learning the current task.
For example, for old tasks in regime 3, we modify their corresponding models using plain SGD. Without characterizing the task similarity, GEM has to employ an inefficient model update by computing multiple gradient inner products using raw data for all previous tasks at each step.
Strengths
* The paper is written really well and the final model is well motivated.
* The paper develops nice intuitions regarding different components of the proposed method.
* Experiments are thoroughly conducted to establish the claims of the paper.
Weaknesses
* Comparison with GEM/ A-GEM: It seems to me that at least some ideas around checking the positive correlation and making the gradient updates based on that are similar to what GEM/ A-GEM inspired.
While the authors of the present manuscript mention A-GEM and state that ‘. In stark contrast, the task gradient correlation characterization in [AGEM] involves the gradient evaluation of old tasks with respect to the current model weight, and hence requires the data of old tasks when learning the new task’, I feel the difference is superficial. My understanding of GEM/ A-GEM is that the authors chose to keep the data of previous tasks and compute the gradients of previous tasks at each new step of the current task, because storing previous data is cheap in terms of memory compared to storing the gradients.
One could equivalently store the gradients in GEM/ A-GEM and their gradient projection step will not change. I feel that the proposed CUBER is an inefficient version of GEM/ A-GEM (in terms of memory). It is not clear to me what was the replay buffer size that the authors of the present manuscript used with A-GEM and ER. For a fair comparison, I would fix the memory size in MBytes and use equivalent memory for CUBER and A-GEM/ ER and then compare their numbers in Table 1.
* First method to achieve positive forward transfer with a fixed capacity network: This claim is wrong! As stated above, since the update procedure of facilitating the positive backward transfer is very similar to GEM, GEM would also achieve the positive backward transfer.
In fact, GEM indeed achieves the positive backward transfer on MNIST permutation and CIFAR100 (cf. Fig. 1 in https://arxiv.org/pdf/1706.08840.pdf).
* Memory/ time comparison: I would like to see the memory and time comparison with other baselines.
While I appreciate the mathematical elegance of the paper, it feels a bit distracting.
For example, it is intuitively very easy to see when the learning of a new task could help in learning the previous task (in fact this is precisely the GEM’s update rule). It is not clear to me if Sec 3 is needed in all its glory. I also feel the learnings of Sec 3, for example the question about when Rule #2 can safely be used to update the model, are not applied in the final model – in regime 3 the authors still don’t use the rule #2 even when there is a positive correlation among the present and previous tasks. Given this, I would appreciate it if the authors could focus more on the intuition of the method itself – why the scaled projection would lead to forward transfer, how their method is different from GEM/ A-GEM etc.
Please respond to the weaknesses section above.
See the weaknesses section above.
I have revised my score post-rebuttal. For detailed justification of this revision, see my response to the authors' rebuttal.
No
Q2: Claim on the first method to achieve positive backward transfer with a fixed capacity network.
A2: Thank you for pointing this claim out. We have clarified all related claims and made them consistent with the original claim in Introduction, i.e., 'To the best of our knowledge, CUBER is the first to demonstrate positive backward knowledge transfer on the considered benchmarks using a fixed capacity network without experience replay.' The modifications can be found in the first paragraph of section 5.1 in the revision.
Q3: Memory/time comparison.
A3: Per the reviewer's suggestion, we have added the memory and time comparison in Figure 4 and Table 6 in Appendix D.4 of the revision.
In particular, we compare the memory utilization and training time between CUBER and the related baseline methods in terms of the normalized value with respect to the value of GPM, following the same strategy as in Lin et al. 2022 [17]. The values of OWM, EWC, HAT, A-GEM, ER_Res are from the reported results in GPM [25]. The value of TRGP is from the reported results in [17]. It can be seen that CUBER indeed has comparable complexity with the baseline methods.
While SVD is used to extract bases in CUBER, the training time of CUBER is still less than some baseline methods due to its relatively simple and fast model update. And because we only store part of the gradient for each old task (as explained in Appendix D.4 of the revision), the required memory utilization does not increase  a lot.
Q4: Clarification of Section 3.
A4: We have the following clarifications about Section 3:
- First, we want to clarify the update rule of regime 3.
As justified by the theoretical results in Section 3, the update rule #2 is only safe for the old tasks which have strong positive correlations with the current task.
Therefore, in the algorithm CUBER, when there is a positive correlation between the current task and some old tasks, we do update the model using the rule #2. As shown in the optimization problem (3), we modify the gradient by subtracting the gradient projection on the subspace of the old tasks in regime 1 and regime 2, and update the current model using the rule #2 with the rest of the gradient, which consequently modifies the learnt model for the positively correlated old tasks in regime 3 with the rule #2. And the characterization of regime 3 is clearly based on the theoretical results in Section 3.
- Second, the purpose of Section 3 is to quantify the conditions under which modifying the learnt model of old tasks is beneficial, by characterizing the task correlations
using the introduced notions of 'sufficient projection' and 'positive correlation'.  More importantly, the task correlation is characterized based on the gradient of the old tasks w.r.t. the learnt model for that old task, instead of the gradient w.r.t. the current model. The later is clearly not accessible in the problem setup considered in this work. Leveraging the gradient w.r.t. the learnt model also enables us to characterize the task correlation before learning the current task.
Therefore, for the scenario where the old task data is not available, Section 3 first introduced two notions to formally characterize the task similarity based on the gradient w.r.t. the learnt model of old tasks, and theoretically quantified the conditions on the defined task similarity under which updating the learnt model of old tasks can be beneficial. These theoretical justifications consequently motivate the algorithm design of CUBER, where a more fine model update for the current model is proposed based on the characterization of layer-wise task correlation between the current task and all old tasks.
In particular, the learnt models for old tasks are only selectively and carefully updated to facilitate backward knowledge transfer when the conditions quantified in Section 3 are satisfied.
Thank you for your thorough reviews and constructive comments. Per your suggestions, we have made the following major revision: (1) added experimental details including the list of hyperparameters in Appendix D.1-D.3,  (2) added the memory comparison in Figure 4 in Appendix D.4, (3) added training time comparison in Table 6 in Appendix D.4, (4) added discussion of impact and limitations in Section 6, and (5) made various modifications throughout the paper based on all reviewers’ comments (the reference indices have changed). All our changes are highlighted with blue-colored texts. New comments on these changes are very welcome!
Q1: Comparison with GEM/A-GEM.
A1: We would like to clarify the important differences between our method and GEM/A-GEM as follows:
- From the perspective of problem formulation, we consider a fixed neural network and focus on the CL setup where the data of old tasks is not accessible when learning a new task, whereas the data-replay based methods like GEM/A-GEM may compromise the data privacy by storing raw data as mentioned in Saha et al. 2021 [25]. As showed in the literature of CL, addressing the forgetting issue and achieving backward knowledge transfer is more challenging without knowing raw data of the old tasks.
For example, while the CL methods demonstrate the best performance under the setup above (e.g., [25] and [17]), they still cannot address forgetting and achieve backward transfer.
- From the perspective of task correlation, we evaluate the 'positive correlation' condition based on the inner product between the gradient of old tasks with respect to (w.r.t.) the learned model corresponding to the old task and the gradient of the current task w.r.t. to the current model, whereas GEM/A-GEM calculate the inner product between the gradient of old tasks  w.r.t. the current model and the gradient of the current task w.r.t. the current model as well. Without knowing the raw data of old tasks, the inner product in GEM/A-GEM is clearly not accessible in the CL setup we considered. As the reviewer insightfully pointed out,  one may use the gradient of old tasks w.r.t. the old model as an estimation of GEM/A-GEM.
But it is not clear a priori if such an estimation will work well and what performance can be achieved.
Importantly, we provide a concrete analysis and theoretical guarantees to quantify the conditions under which modifying the learnt model of old tasks is beneficial, in terms of the sufficient projection and positive correlation introduced in this work.
- From the perspective of methodology, we propose a more fine model update for the current task by characterizing task correlations between the current task and the old tasks, whereas GEM/A-GEM seek to always project the gradient of the current task onto the direction that has minimum interference to all the old tasks.
Therefore, our model update rule is quite different from that in GEM/A-GEM.
Specifically, in GEM/A-GEM, when the gradient inner product is negative, the current model will be updated along the gradient direction that is orthogonal to the calculated gradient of the old tasks. Such a strategy is not efficient as shown in (Doan et al. 2021, A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix) and [25], because the data structure is not leveraged.
In contrast, we use the gradient correlation (based on the gradient w.r.t. the learnt model of old tasks, which is a different criteria from that in GEM) to characterize the layer-wise task correlation, which enables us to differentiate the current model updates depending on its layer-wise correlation with different old tasks.
In particular, we modify the current task gradient by subtracting its projection on the subspace of the old tasks that are less correlated with the current task, instead of the projection on the gradients of all old tasks w. r. t. the current model. For old tasks in regime 3 where modifying their learnt models is beneficial, we update the model using rule #2. As shown in eq. (3), we modify the gradient by subtracting the gradient projection on the subspace of the old tasks in regime 1 and regime 2, and update the current model using rule #2 with the rest part of gradient, which consequently updates the learnt model for the positively correlated old tasks in regime 3 with rule #2.
Intuitively, our method 'divides' the layer-wise model into three different 'parts' based on the correlation between the current task and the old tasks, and updates each 'part' differently. More importantly, GEM needs to compute the gradient for all old tasks at each iteration, whereas our method only evaluates the task correlation once before learning the current task.
Following [25], the reported results of A-GEM and ER both use a sample size of 1000, 2000, 500, 3000 for PMNIST, CIFAR-100, miniImageNet and 5-dataset, respectively. A more detailed list of hyperparameters can be found in Table 5 of Appendix D.3 in the revision.
Thank you for your response.
However, after reading the response and rereading the paper (and its updated version), my concerns about comparisons with GEM/ A-GEM are even more stronger. I will use authors’ response bullet points as headings below.
- Perspective of problem formulation: Thanks for the clarification but the problem setup was clear from the first draft as well and I didn’t ask for that clarification.
- Perspective of task correlation: As I said in my original review this difference is superficial.
In fact, GEM’s main update rule says that the update on the current task should not increase the loss of the previous tasks with previous models.
However, if you continue to satisfy that constraint at each SGD step of each task, you don’t have to compute the loss of the previous tasks on the previous models but on the most recent model.
So as opposed to what the authors of this paper claim in their rebuttal, that a priori it is not clear how the GEM update would behave if the gradients of the previous tasks are computed wrt the previous models, it is in fact quite clear that it should behave similarly. I would invite the authors to run the experiments on GEM using gradients computed with previous models and verify this for yourself.
- Perspective of methodology: I am not sure how the model update is more “fine” (quoting the authors here) in their work compared to GEM.
In fact, experimentally the authors set the \epsilon_2=0 in their main experiments, which would collapse the definition 2 to the GEM’s definition of positive correlation of different tasks (this is something I missed in my first read of the paper and I only realized now). If the claim is that GEM’s update does not leverage the data structure as the authors pointed out in their review, then this may be true, but this, I believe, is the observation that the authors borrowed from another work (and rightly quoted) and not something the authors make a big point about in their work. The other difference of computing the projection layer-wise vs for the whole network is an implementation detail which has nothing to do with the overall method (neither GEMs nor this work).
So all in all, in all three axes that the authors identified, I am not convinced that their method is particularly novel.
- Memory and time comparison: I am not sure about the validity of their Fig 4 and table 6. Let’s say anything other than the current task data, you put in the replay buffer. For GEM/A-GEM this replay buffer is populated with a small number of samples of previous tasks and for CUBER (this work) it would be gradient bases of previous tasks. My question then was, for a fixed replay buffer size in Mbytes, which of these methods would perform better. Since the authors claim that they store 1000 examples for PMNIST for A-GEM, and 300 bases (per task?) for CUBER, the replay buffer size of CUBER can’t be less than A-GEM.
Similarly, I don’t understand the time comparison either. For A-GEM, you’d simply compute one extra fwd/bwd pass followed by gradient projection.
Whereas, in CUBER, the computation of basis vectors, figuring out which tasks belong to which regions should cost extra.
While I can understand for very deep networks, the extra fwd/ bwd cost of A-GM would add up but the small models that the authors of CUBER tried, should not make CUBER more time efficient.
I would also agree with the other reviewers about running all the experiments from the same codebase and under the same setup.
Overall, I still feel that the mathematical complexity that the authors introduced is a bit distracting and their method lacks a proper experimental verification.
Therefore, I will reduce my score.
Thank you for your further comments. We have the following clarifications:
(1) Comparison with GEM:
- We agree with the reviewer that "if you continue to satisfy that constraint at each SGD step of each task, you don't have to compute the loss of the previous tasks on the previous models but on the most recent model", and this aligns well the approach used in our paper.
**We want to emphasize again that a clear difference between our work and GEM is that when the data of the old task is not accessible, there is no way to guarantee that the above constraint can be satisfied because it cannot be evaluated.**
- Our approach provides a more fine model update,  because it is able to update different components of the current model using different criteria based on the task correlation between the current task and different old tasks. The underlying rationale under this strategy is that the important weights differ among different old tasks. For old tasks that are less correlated to the current task, we protect the important weights for these tasks. For old tasks that are more correlated to the current task, we modify their important weights just using plain SGD. **A very important component of this strategy is the characterization of task correlation before learning the current task. Without characterizing the task similarity, GEM has to employ an inefficient model update by computing multiple gradient inner products using raw data for all old tasks at each step.**
- In terms of the selection of $\epsilon_2$, as shown in the ablation study, it characterizes the conservatism in selecting tasks for Regime 3 and modifying the model of selected tasks. With a larger $\epsilon_2$, we facilitate the backward knowledge transfer to the old tasks that are more correlated with the current task. As illustrated in Table 4, CUBER still achieves good performance for other selections of $\epsilon_2$.
(2) Memory comparison: We actually never claimed that we store 300 bases (per task) for CUBER and we think the reviewer has some misunderstanding about this. As listed in Table 5 for PMNIST, we use 300 data samples (not the number of bases) for each task to extract the bases of its input subspace based on SVD after learning this task, by following GPM (Saha et al. 2021). These samples are directly sampled from the dataset for this task, and we do not need to store these samples. As shown in Table 10 in GPM, the number of stored bases is much smaller than 300 and also decreases with the CL process. This is also the case in our work as we use the same bases extraction method and same threshold. The required memory is clearly reasonable compared to A-GEM which stores 1000 samples with a larger dimension.
(3) Time comparison: As mentioned in our revision, we compute the normalized time consumption w.r.t. GPM  which is much more efficient than A-GEM as shown in Saha et al. 2021. The reason why CUBER (also GPM and TRGP which both include bases extraction) is faster than A-GEM is because the model update rule here (SGD and projection on subspace) is computationally more efficient than that in A-GEM which needs to compute multiple gradient inner products at each training step. This compensates the time cost for computing the bases and task correlations. And we only compute the bases and task correlation takes once before learning each task.
More importantly, the performance of our method is much better than that of A-GEM as clearly demonstrated in Table 1.
- The difference that you are emphasizing is only a superficial one. You could store the gradients of the previous tasks computed at the previous models in the replay buffer for GEM. Accessibility of the previous data is only a superficial constraint.
- As I said in my previous response, when $\epsilon_2=0$, the setting used in the main experiments (Table 1), the task correlation (def 2) only checks the inner product (same as GEM). Again whether this check is done only at the start of the training of task 't' or at each SGD step is a detail. If you store gradients of the previous task in the replay buffer, then you can perform this check at the start of training of task 't'.
My main argument is not that there is no difference whatsoever, what I am pointing out is that the difference is not as huge or fundamental as the authors are claiming.
Memory comparison: I see. I read it wrong then. Maybe the authors need to clearly write what is stored in the replay buffer. Currently, the lines from 221-230 are very cryptic. It is not clear if $R_l^t$ is the same as $S_l^j$ used elsewhere in equations.
Time comparison: Not sure I followed. A-GEM is literally an SGD and projection onto a vector. How are there more "inner products" in A-GEM's vector projection than CUBER/ GPM/ TRGP subspace projection?
In your response to the reviewer nsyh you said that "The reviewer nMSj is clearly wrong and has some misunderstanding of our approach." and then you go on to to describe what you said above. I think it's a rather strong statement and the one not fully backed by the clear evidence.
Thank you so much for your reply. We have the following clarifications:
- As the reviewer mentioned earlier, one can store and use the gradient of the previous tasks w.r.t. the previous models to estimate the gradient of previous tasks w.r.t. the current model, so as to evaluate the constraint $\langle \frac{\partial l(f_{\theta}(x,t),y)}{\partial \theta}, \frac{\partial l(f_{\theta},M_k)}{\partial \theta} \rangle\geq 0$ for all old tasks $k<t$ in GEM, where $M_k$ is the data of old task $k$.
Observe, however, that since in general the model may change significantly during the CL process, the gradient estimation could become outdated over time and  not accurate enough such that the constraints above would not satisfied (note that the constraints are for all old tasks $k<t$ in GEM). This is what we had meant by
"a clear difference" between GEM and our study, and had thought you may have  "misunderstood" our study.
We had wanted to clarify this because the discussions impacted other reviewers as well. We would like to apologize if our words are not appropriate and may have offended  you by mistake.
- In our method, checking the task correlation before learning each task is not only an implementation detail, but also a design strategy. It is very important to characterize the task correlations in CL so as to understand both forward and backward knowledge transfer. What is proposed in this work is a comprehensive and efficient detection mechanism of the task correlation, which consequently helps  to design the model updates explicitly for better forward and backward knowledge transfer. This design strategy is clearly different with that in GEM.
Moreover, suppose we store gradients of the previous tasks in the replay buffer. If GEM is used, we need to compute the gradient projection on these previous gradients for all previous tasks at each training step, and the gradient projection is computed as $\frac{<g_{current}, g_{old}>}{<g_{old}, g_{old}>}g_{old}$. In this work, for old tasks in regime 3, we update the model just using $g_{current}$ at each training step and no gradient projection is needed, which is simpler and more efficient.
- For the memory comparison, we thank the reviewer for pointing out the unclear descriptions. $R_l^t$ is the representation matrix for each layer as defined in line 180, and $S_l^t$ is the corresponding bases extracted from the representation through SVD.
- For the time comparison, at each training step, A-GEM first needs to do two forward-backward passes on the current model, to compute the reference gradient $g_{ref}$ and the current gradient $g$.
Next, A-GEM needs to evaluate if the condition $<g, g_{ref}\geq 0$ holds. If yes, A-GEM updates the model using $g$. If not, the SGD update in A-GEM involves the computation of gradient projection onto the reference gradient, i.e., $\frac{<g, g_{ref}>}{<g_{ref}, g_{ref}>}g_{ref}$.
However, in CUBER/GPM/TRGP, at each training step, there is only one forward-back pass to calculate the current gradient. And there is no need to evaluate if the condition above holds in GPM/TRGP. CUBER does have to check the condition for degenerating old tasks from regime 3 to regime 2.
But this condition is only checked in the first several steps of training and before the degeneration.
Moreover, in CUBER, for old tasks in regime 3, we update the model only using the current gradient and no gradient projection is needed.
Once again, we appreciate the reviewer for spending previous time on reviewing our paper and providing insightful suggestions, and we believe the discussion helps to improve our paper a lot. We hope the above response can  clarify our work further.

Meta-review:
role: AC
data: metareview: This paper develops a method to improve backward transfer in continual learning without replay. In particular, upon learning a new task the proposed method also updates parts of the model responsible for the performance of the old tasks. 

The initial reviews for this paper were quite mixed. All reviewers agreed that parts of the method were novel, but reviewers also found important limitations of the initial manuscript. In the discussion, the authors were able to clarify many questions/criticisms notably with respect to related work as welll as to the exact experimental setup. The updated version of the manuscript now provides much needed details (which should help with reproducibility).

After the discussions, some questions regarding the significance of the results and the differences with respect to GEM remain. 

Regarding significance of the results, reviewer nsyh outlines that while improving backward transfer is significant, the level of improvements shown in the paper are modest and could be a result of overfitting. In particular, the method seems to outperform baselines, but mostly does not achieve positive backward transfer. The problem of backward transfer seems difficult and it is unclear if positive backward transfer is even achievable in most cases (e.g., it seems to rely on having sufficiently high similarity between tasks). In that sense, I find the results to be reasonable.

Regarding GEM, there is some disagreement about the level of similarity of this work compared to GEM and whether a simple extension of GEM could compare favourably to this method. Studying this extension of GEM would be improve this paper, but I don't see it as a requirement. This perspectivee is shared by two reviewers.

Overall, this remains a very borderline paper, one that studies an important problem, but for which reviewers disagree on significance and even to a certain extent novelty given existing work. Nonetheless, methods for improving backward transfer have not been studied much in continual learning and so this paper might spark additional work in the area. Further, the paper has no major weaknesses. Based on this I am happy to recommend this paper be accepted. 

I strongly encourage the authors to incorporate elements of the discussion you had with the reviewers into account when preparing the camera-ready version of their work. In particular, it would be useful to provide more intuitions regarding the method especially in light of related works (Saha et al.'21, Lin et al.'22, Kao et al.'21, and GEM Lopez-Paz et al.'17 + A-GEM Chaudhry et al.'19). Considering the extension of GEM would also provide an interesting baseline.
role: PC

Decision: Accept