Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Iterative Relaxing Gradient Projection (IRGP) to facilitate forward knowledge transfer within a fixed network capacity for continual learning. 
strength_and_weaknesses: Strengths
+ Interesting solution to the stability-plasticity dilemma for gradient projection continual learning methods.
+ Clear mathematical exposition of the related and proposed methods.

Weaknesses
- Although IRGP was introduced to facilitate forward knowledge transfer, the metric FWT is only reported in the appendix and compared with only one method. This is clearly unacceptable. FWT should be reported in Tables 1-3.
- The proposed method is an iterative method, yet there is no mention or analysis of computational complexity or resources are mentioned anywhere in the paper let alone compared with existing work.
- The experiments are not consistent. It is difficult to get a clear view on how the method really compares especially versus expansion-based methods because only one benchmark is reported. Why only CIFAR-100 or PMNIST?
clarity,_quality,_novelty_and_reproducibility: Clarity. The paper is well-written and easy to follow,.
Novelty. The work is novel.
Reproducibility. All details should be provided in this paper. The paper follows the experimental setup of previous work providing a citation.
summary_of_the_review: The proposed method is interesting but the empirical evaluation is not convincing. 

Update. I thank the authors for the response and addressing many of my and other reviewers' comments. I have updated the score to reflect the recent changes. The reason for not providing a higher score is current missing results due to the time constraint.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To tackle the problem of incrementally learning a sequence of tasks with a deep neural network, this paper builds upon and extends a recent line of work using gradient projection to constrain the optimization of newly encountered tasks. In particular, this paper builds upon the TRGP method proposed by Lin et al. (2022, ICLR). A disadvantage of TRGP is that, when learning new tasks, TRGP expands the model with a new trust region (and associated parameter storage) for each new task. To avoid such expansion, the IRGP method proposed in this paper instead iteratively searches for the most critical parameter subspace to maintain. This results in a method that still performs well on a specific set of benchmarks (approaching the performance of TRGP, and if also using additional parameters it can outperform it) with less rapidly increasing storage costs.
strength_and_weaknesses: I think an important limitation of this paper is that it very closely follows the experimental setup of a couple of recent papers proposing similar methods, such as Saha et al. (2021, ICLR) and Lin et al. (2022, ICLR).

One resulting issue is that without knowledge of these two previous papers, it is very challenging, if not impossible, to appreciate the contribution of the current paper. I would encourage the authors to rewrite their paper to make it possible to appreciate its contribution without knowledge of these previous papers.

Another issue resulting from this is that there is an important risk of overfitting on the specific benchmarks used. On the one hand (e.g., for reproducibility) it is great that the authors include experiments on these extensively studied benchmarks, but as for several iterations now this line of work has exclusively been evaluated on this set of benchmark, I believe there is a real risk of overfitting and I would encourage the authors to evaluate their method also on other benchmarks.

Another consequence of sticking so closely to these previous papers, is that the set of methods that is compared against is exactly the same as in those previous papers. Other recent related methods are not considered. In particular, one very related method that is not considered is Natural Continual Learning proposed by Kao et al. (2021, Neurips; https://arxiv.org/abs/2106.08085). This method also tackles continual learning by putting restrictive constraints on the optimization of new tasks, but, unlike the present paper, uses an approximation of the Fisher Matrix of old tasks for this. Intuitively, this should allow for more positive transfer than using an orthogonal projection onto the input subspaces of olds tasks. I strongly encourage the authors to compare against this paper.

In the continual learning literature, often a distinction is made between task-incremental learning, domain-incremental learning and class incremental learning (e.g., as in Kao et al., 2021, NeurIPS; or see https://arxiv.org/abs/1904.07734). The current paper does not discuss these distinctions, and it is not entirely clear to which of these three scenarios of continual learning the considered benchmarks belong. My guess would be that PMNIST follows the assumptions of domain-incremental learning, while the others follow task-incremental learning? It would help the clarity of the paper to make this clear.

Other issues:
- In section 2, why is GEM not discussed under gradient projection methods? It seems clear that this method is doing gradient projection as well.
- Whenever experimental results are copied over from another paper, this should be clearly explained in both the text and the legend of the table/figure where those results are reported.
- I think it is important to include the standard deviations in the tables in the main text. Currently they are only provided in the Appendix, but for a correct interpretation of the results they are essential.
- In Table 1, it is misleading that the BWT results of “Ours (IRGP)” are printed in bold. For none of the considered benchmarks this method scores the best on this metric.
- For AGEM and ER_Res it should be stated in the main text what memory buffer size is used.

clarity,_quality,_novelty_and_reproducibility: As mentioned above, this paper heavily builds upon two recent papers (Saha et al., 2021, ICLR; Lin et al., 2022, ICLR). For example, the experimental benchmarks used and methods that are compared against are essentially the same. This has advantages in terms of reproducibility (as code for these benchmarks was provided by these previous papers), but only in a narrow sense, as it is unclear how well the proposed method would do in settings in which it and its predecessors were not developed.
This heavy reliance on these previous papers has also negative consequences in terms of novelty, and also in terms of clarity (the paper requires familiarity with these previous papers).
summary_of_the_review: Although I believe this paper probably makes a minor contribution to the literature, the restricted experimental evaluation (in particular the heavy reliance on previous papers) causes me to currently recommend rejection. I encourage the authors to test their idea in more varied settings, compare their idea to other related methods and to improve the writing of the paper to reduce the reliance on knowledge of other papers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a method to mitigate forgetting and favor forward transfer by using a gradient projection approach with a fixed-capacity architecture. The method (Iterative Relaxing Gradient Projection (IRGP)) iteratively searches for frozen (previously trained) weights that are relevant to the current task. These weights are relaxed to consolidate knowledge of the current task, which helps with one of the problems of gradient projection methods: low flexibility when training novel tasks. In addition, the authors present a version that can be expanded (increasing the number of weights in the model), which is efficient compared to a state-of-the-art method. Results superior to various methods are presented in different scenarios of Task-Incremental Learning, in addition to some ablation studies.
strength_and_weaknesses: S:
- In order for CL to work, there must be motivation to add flexibility to the weights learned through previous tasks and that this flexibility causes minimal forgetting. It not only helps to consolidate learning, but it can also help with knowledge transfer. The motivation of the authors is interesting to the community.
- The idea of adding plasticity to the model by relaxing similar connections for both tasks (old and new) is interesting. Not only does it add plasticity to the current task, but it can also help with knowledge transfer to previous tasks. 
    - Question To the authors: Do you study how the accuracy of previous tasks is affected when weights are relaxed?
- Table 2 provides helpful insights into how the flexibility of certain connections is used. Without making the model grow, favorable results are achieved.

W:
- The definition of Forward Transfer is not the same as the one used in the paper by Lopez-Paz and Ranzato. It is not clear to me if only the formula is incorrect or if the results are as well, but the way it is presented makes me think it is not only the formula. Forward Transfer has to do with the ability of the model to acquire knowledge that may be relevant to tasks that have not yet occurred. Here it is used for the current task.
    - Focusing on the accuracy of the current task is crucial, and not many papers focus on it. However, it is not the same as Forward Transfer.
- As the proposed method takes 60% more time than GPM, and only increases accuracy by 1%, it is not a favorable trade-off. It may depend on the application and scenario, but it is a big difference. However, I do appreciate that the authors mention this result.
- In Table 4, adding more extreme values would have been helpful. In order to determine whether the assumptions mentioned in the model are met, we must observe how the model behaves.

Questions:
- Since the community is focusing more (and for a good reason) on scenarios where the task-id is unavailable in inference. Do you think this method can be adapted to work in this type of scenario?
- Is it known what percentage of weights are relaxed in each task? Can this be related to forgetting the method?
- In the fifth line of the paragraph after Eq 1. Should U^l_{t-1} just be U_{t-1} or not?
- What is defined in Eq 9?
- In Eq 10, what is the I of W^l_{t,I}?
- Algorithm 2 is in Appendix D, not C.5

clarity,_quality,_novelty_and_reproducibility: The paper shows promising research directions. However, I still have a doubt about the definition of Forward Transfer that is being used. 

I think the work has a good motivation: to take advantage of information that may be relevant to two tasks and consolidate that knowledge. Something that the authors present well, even though I have doubts as to how, how much and how many weights are moved/selected in this iterative process. It would be interesting to see how the authors' motivations to achieve a better transfer of knowledge between the different tasks are effectively fulfilled.
summary_of_the_review: As the paper is, I don't think it's good to accept it. However, I think it is details that separate it from a paper that can be accepted.

The first is the definition of Forward Transfer and the second is a more complete study of how many and how the weights change when they are made flexible. If many weights are made flexible, one would expect more forgetfulness, if there are few, how do we ensure that there is indeed knowledge transfer?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method for achieving continual learning in the presence of multiple tasks by reducing knowledge forgetting while encouraging forward transfer. The key idea is to avoid augmenting the model with new parameters by iteratively searching and relaxing the most effective trust region.  

The reviewers found the paper to be well written and also consider the main idea (especially positive transfer) to be interesting and valid. It is also notable that the method ensures that forgetting is minimized without the model having to grow. 

I would like to note that although this paper received only three reviews, there has been reasonable discussion between authors and reviewers and a lot of discussion privately among reviewers and the AC (even outside of openreview). These discussions resulted in the authors to increase their score. However, it seems that after discussing with the reviewers the consensus is that the weaknesses outweigh the strengths of the paper. 

In particular, the proposed method follows very closely [Saha et al., 2021, ICLR; Lin et al., 2022, ICLR], meaning that the work is rather incremental. At the same time, the reviewers believe that the experimental results are insufficient and at some cases weak (e.g. results for positive transfer). For example, further benchmarks or baselines should be considered. It is understood that this paper builds and therefore compares against two particular pieces of prior work, however, it would add more value to the community if the method's merits were demonstrated more independently. After all, the reader will be more concerned about how this method can be used to solve the continual learning problem in general, independently of which other methods served as inspiration to this one. 

justification_for_why_not_higher_score: Even after the rebuttal, all reviewers are still concerned around the combination of limited novelty and empirical results. All reviewers agreed that this paper needs more work.
Decision: Reject