Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a new method, Recursive Gradient Optimization (RGO), for continual learning in the task-incremental scenario. This method modifies the direction of gradients on a new task in order to minimise forgetting on previous tasks, and unlike many previous works, does not require storing past raw data to do so. The authors introduce a Feature Encoding Layer to achieve this. The authors provide experiments on 4 benchmarks of varying size (MNIST to miniImageNet), showing good performance of their algorithm, with different architectures.
main_review: Overall, I think this is a nice idea and a good application of it. It is relevant for ICLR, and the idea is novel (specifically, as far as I am aware, no one has previously used results such as Theorem 1 and Feature Encoding Layers). It is nice to be able to modify the gradient on a new task with a projection matrix, while minimising forgetting on past tasks. The experiments seem comprehensive in range (many benchmarks, many architectures). 

In general, the paper introduces the algorithm well and the order of information seems nice. I focus the rest of the review on various ways I think the paper could be improved further. 

1. The authors argue that "Single-Task Learning ... can be regarded as the strongest expansion-based method" (Introduction), and compare a lot to STL (in the abstract and experiments). I disagree that STL is the strongest baseline, as there is no potential for forward or backward transfer between tasks with STL. In fact, many algorithms can beat STL for this exact reason (STL has even sometimes been considered as a 'lower bound', see for example [1] and the Split CIFAR task they have, which is slightly different to the one here). Overall, I believe STL is a strong method, but it is not quite right to say it is the "strongest", and perhaps the authors should not emphasise comparing so much to it throughout the paper.

2. Throughout the paper, there are missing references to many Bayesian-inspired approaches to continual learning. Examples include EWC [2], online EWC [3], VCL [4,5], Online Structured Laplace [6], FROMP [1], and many many more. These are usually regularisation-based approaches ([1] also uses memory). They are based off Bayes equation for continual learning. A key benefit is that Equations 4 and 5 naturally "drop out" of applying Bayes equation (with the Laplace approximation), and I am not sure the proofs (such as Appendix A.2) are required -- people should already see that Equation 5 holds from Bayes (in particular, online EWC / online structured Laplace use the same equation). A Bayesian view also justifies the use of alpha in Section 4.2, where alpha is the prior precision (or weight-decay parameter when training the 1st task). There is of course a key difference between the Bayesian-inspired approaches and this work: the Bayesian approaches minimise the overall loss together L_k + F_k^RLL, whereas this work minimises F_k^RLL subject to \grad L_k = 0. The authors can make this clear when discussing the Bayesian approaches.

3. Modifying the direction of gradients is closely related to natural-gradient descent (Amari, 1998) and mirror-descent. See for example NCL [7] where they discuss some of this.

4. I believe a term is missing from Equation 4 (bottom line): $L_{k-1}(\theta_{k-1}^*)$

5. The authors re-use hyperparameters from previous work for baselines. Unfortunately baselines such as EWC are known to be sensitive to hyperparameter values (for EWC, this is the regularisation parameter lambda) when the benchmark and/or architecture changes. It therefore seems unfair to report results based off one value. It would be good if the authors ran a hyperparameter sweep for the baselines too for fairness, and reported in the text that additional computational effort was needed. I still expect RGO to significantly outperform baselines.

6. In Appendix B.1, what do the authors mean by "we only used the diagonal element corresponding to the ground truth label for estimation". Why did the authors not use Equation 31 itself? I thought Equation 31 was cheap to calculate and use (requiring a forward pass only through the model).

7. [Minor point] It was interesting for me to see that "deeper structures lead to more forgetting" (page 8). I would love to see further investigations / intuition as to why this is the case with RGO (or maybe in general). Perhaps this can be future work.

Typos:
- Section 2: the authors first introduce W as the space of model parameters, but for the rest of the paper use only theta.
- Section 4.1 page 4: "matrixs" -> "matrices", "applys" -> "applies"
- Algorithm 1 line 9: missing "l=1,2,...,L"
- Table 2: please say the full names of all the metrics/methods reported in this table, along with references to the papers. Perhaps this information can go in the caption.



References

[1] Pan et al., 2020, "Continual deep learning by functional regularisation of memorable past"
[2] Kirkpatrick et al., 2017
[3] Schwarz et al., 2018, "Progress & Compress..."
[4] Nguyen et al., 2018, "Variational Continual Learning"
[5] Loo et al., 2021, "Generalized Variational Continual Learning"
[6] Ritter et al., "Online structured Laplace approximations..."
[7] Kao et al., 2021, "Natural continual learning..."
summary_of_the_review: Overall, I am leaning towards accept as I like the idea and the results are very strong. I have provided potential ways to improve the paper in my main review, specifically points 2 and 5.

*****************
After rebuttal: I have increased my score to 8, thanks to the authors rephrasing certain parts / adding certain comments/references to the paper. I believe the paper could still be increased by tuning hyperparameters of baselines, however, this does not detract from the quality of the proposed method.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses a long-standing problem of deep neural networks where new training data for new tasks arrive continuously and data does not stay forever (continual learning). The network has to learn to cater for all the tasks without forgetting what it learnt for an older task as data from older tasks are no longer available. The authors came up with a formulation where learning will mean minimizing the (expectation of) forgetting and forgetting is formalized as increase of losses corresponding to the old tasks. The authors then move on to find a Taylor series expression of forgetting and an equivalent Recursive Least Loss (RLL) formulation of the loss (aka forgetting). Next the authors introduce a modification in the direction of gradient update of normal SGD that makes sure to minimize forgetting. The modification implied the introduction of a positive definite matrix and the optimization problem transformed into an optimization problem on finding the optimal positive definite matrix minimizing the RLL. As the relation between RLL and the positive definite matrix is not straightforward, an upper bound of RLL is proposed and the optimal solution is found under this upper bound. One more interesting proposal is a virtual feature encoding layer (FEL) that applies task-specific random rearrangement to the input feature maps. FEL eliminates possible interference between tasks by decorrelating features among old tasks without expanding the memory footprint. The experiments show great improvement over sota approaches on benchmark datasets and sometimes (rotated MNIST, Split ImageNet) even goes beyond supposed upperbound performance given by single task learning showing evidence of positive transfer.  Overall, I liked the approach, the presentation and the experimental results with only a few minor concerns detailed below.
main_review: Paper strengths:
* Continual learning is an important, challenging and practical problem which the authors have addressed. The problem is formulated as tackling forgetting (increment of old task losses) with the constraint of learning optimally for the new tasks.
* The theoretical analysis of the problem (forgetting as increment of old task losses, its approximation to more tractable RLL and further to its upperbound) is done very good.
* FEL with task specific random permutation of features is interesting and shown to be effective by decorrelating the features between different tasks but with strong correlation inside individual tasks.
* Being a theoretically motivated work, the authors have done a very good job in clearly motivating the different steps with derivations and explanations as and when needed.
* The experiments show the superiority of the approach very good.

Paper weaknesses:
* This is regarding eqn. (4). I think I got the major steps of this derivation. It uses the definition of $F_{k-1}(\theta^*_{k-1})$ and the fact that Hessian is symmetric. However, I wonder if there would be an extra $L_{k-1}(\theta^*_{k-1})$ term at the end. It would be helpful if a more detailed derivation is provided for.
* Though appendix B.2 provides space complexity of projection matrix, I am interested on the advantage of RLL formulation in terms of the capacity of the network. $F_k$ in (4) and $F^{RLL}_k$ in (5) both need to store all $H_j$, I see. Any comment specific to network capacity in relation to this would be good to have.
* Minor typos: First line of eqn. (4) one of the $\theta$s have the asterisk as subscript. Page 4 last paragraph and section 5 first paragraph – ‘applys’ should ‘applies’.
summary_of_the_review: In summary, the paper presents a crisp solution to a challenging and important problem with mostly clear writing and experiments. Some of the concepts introduced here (e.g., gradient direction change and FEL) can be useful not only in continual learning research but applications where forgetting can be an issue or decorrelating features can be advantageous.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a continual learning approach based on recursive gradient optimization. To this end, a projection matrix is derived for the gradient modification. This matrix, P, is computed incrementally and updated by integrating the Hessian on each task locally.

main_review: - I agree with the validity of the Taylor expansion in equation (2). However, I find the assumption in Eq(3), that \theta is in the vicinity of each of \theta_j^*, doubtful. This is as if we say that the optimal parameters for each task are in the vicinity of each other which would abolish the problem of catastrophic forgetting from the beginning.
- The approach seems to depend on two main new components (1) The new projected matrix P, and (2) the FEL layer. I am still unsure which of the two reduces forgetting. The authors are encouraged to define two new baselines each of which incorporates one of the aforementioned components.
- There is a discrepancy between Algorithm 1 and the previously derived quantities. Concretely, the square root in line 9, and I am also missing the inverse of the sum of Hessians, H_l, in Algorithm 1.
- Is there a missing sum in the first equation of (14)?

summary_of_the_review: I have two issues with (i) the assumptions made in parts of the derivations, and (ii) the missing important baselines that should be derived from your approach.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a gradient-based approach to continual learning in neural networks called Recursive Gradient Optimization (RGO). RGO modifies the gradient direction at each update by multiplying it with a projection matrix P that is designed to minimize the increase in loss on previously encountered tasks. RGO is theoretically designed to prioritise performance on the current task and, among the optimal solutions for the current task, find the one that causes least interference with previous tasks - thus it assumes that the network is overparameterised. The derivation of the method starts by approximating the continual learning loss with the “recursive least loss”, which involves a sum of the Hessians of the previous tasks and assumes that tasks are fully trained and their solutions are close by to each other. It is then shown that the recursive least loss can be upper-bounded by an expression involving the projection matrix P, via which the solution for P that minimizes this upper bound is derived. It is shown that the expectation of the step size can be preserved by guaranteeing that trace(P)=dim(P) and by introducing random task-specific permutations at each layer - this preserves the “current-task-first” principle and reduces the need for hyperparameter tuning. Experiments are run on a number of standard continual learning image classification benchmarks, demonstrating an extremely strong performance of RGO in comparison to competing methods, sometimes surpassing the performance of a baseline that trains a separate model for each task.
main_review: This paper provides an empirically very strong method for task-based continual learning, as measured by average task accuracy, alongside a solid theoretical derivation and justification for it, which will be of interest to the continual learning community and for these reasons I am recommending it for acceptance. The main weakness of the method is that it has a runtime that is quadratic in the number of parameters at each layer, as a result of the gradient projection at each update; the paper could greatly benefit from an empirical analysis of the runtime compared to competing methods in order to quantify this limitation. 

Positives:
- Very strong empirical performance on standard CL benchmarks, exhibiting essentially no forgetting across 20 tasks.
- The method is simple to implement.
- Ensuring that the expected learning rate on the current task is preserved (via setting trace(P)=dim(P) and with the feature encoding layers) reduces the need for hyperparameter tuning, in contrast to other regularization methods that often require tuning of hyperparameters that control the balance of progress on the current task and preservation of previous task performance.
- Despite the quadratic runtime in the number of parameters per layer, the projections of the gradients at each layer are independent of each other and so can be computed in parallel (once the gradients have been calculated via backprop).
- While the method requires knowledge of task ids for the random permutation layers, these only require storage of a single integer seed each. It would be interesting to investigate how this method could be extended to a task-free setting.
- Good discussion of related methods

Potential areas for improvement / questions:
- The runtime of the algorithm is quadratic in the number of parameters per layer, which could be concerning given that it is acknowledged in the paper that the method requires wide enough network layers to satisfy the overparameterisation assumption. A runtime comparison across methods should be conducted in order to quantify the differences.
- It’s a bit difficult to understand what the method does from the introduction. It could do with a diagram.
- In Table 1, shouldn’t the values in the BWT be negative as in Table 3?

Typos:
- Equation 4 typo: $\theta_{k-1^*}$ -> $\theta_{k-1}^*$
- Section 4 title: “Implemention” -> “Implementation”
- 4.1 Line 2 “matrixs” -> “matrices”
- Definition 4.1 “applys” -> “applies”

summary_of_the_review: This paper provides an empirically very strong method for task-based continual learning, as measured by average task accuracy, alongside a solid theoretical derivation and justification for it, which will be of interest to the continual learning community and for these reasons I am recommending it for acceptance. The main weakness of the method is that it has a runtime that is quadratic in the number of parameters at each layer, as a result of the gradient projection at each update; the paper could greatly benefit from an empirical analysis of the runtime compared to competing methods in order to quantify this limitation.
review_title: null

Meta-review: This paper proposes an innovative method for continual learning that modifies the direction of gradients on a new task to minimise forgetting on previous tasks without data replay. The method is mathematically rigorous with a strong theoretical analysis and excellent empirical results across multiple continual learning benchmarks. It is a clear accept. There was good discussion between the reviewers and authors that addressed a number of minor issues, including clarifying that the method has the same computational complexity as backpropagation. The authors are encouraged to make sure that these points are addressed in the final version of the paper.
Decision: Accept (Spotlight)