Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper provides a theoretical analysis on the OGD based continual learning method. The method is in fact proposed by a previous paper (Farajtabar et al. 2019) and the current paper shows a generalization bound for the regression case. The result (Thm 3) compares the generalization bounds between SGD and OGD and shows OGD leads to a tighter bound. The theorem is also based on the bound on the Rademacher Complexity (Lemma 1).  The paper also suggests OGD+, which stores some data points from past tasks. They also present some experimental results on small benchmark datasets, and show OGD+ outperforms SGD and OGD.

While the paper makes an interesting attempt on theoretical analyses of OGD based continual learning method, I feel the result is quite limited only to the OGD scheme. Also, the result is for regression, as shown in the loss function in Sec 3.2, but the experiments are on classification, so it's not clear with the connection with the theory and the experiments. The results also seem to be somewhat simple derivations from the known papers, like Jacot et al., (2018) and Liu et al, (2019). 

The experimental results are also very limited  and weak since it only compares with SGD, an obvious weak scheme that suffers from catastrophic forgetting, and does not compare with any other continual learning baselines. For example, the state-of-the-art on CIFAR-100 is around 65%, and the performance of OGD is very weak. Even though the paper aims for a theoretical contribution, it is very limited only for OGD based scheme, which is not strong in practice. So, I am not sure about the significance of the contribution of the paper. But, I haven't fully read the entire proof of the paper, and I may have missed some details regarding the proof. I would like to see other reviewers' opinion as well. 

What about comparing with more enlarged and various benchmark datasets beyond MNIST and CIFAR-100, like CUB200 or Omniglot as shown in https://arxiv.org/pdf/2003.13726.pdf ? How does OGD or OGD+ compares with other baselines like EWC or MAS?
review_title: Theoretical analysis for OGD for continual learning

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors use a Neural Tangent Kernel (NTK) approximation of wide neural nets to establish generalization bounds for continual learning (CL) using stochastic gradient descent (SGD) and orthogonal gradient descent (OGD).  In this regime, the authors prove that OGD does not suffer from catastrophic forgetting of training data.  The authors additionally introduce a modification to OGD which causes significant performance improvements in the Rotated MNIST and Permuted MNIST problems.  OGD involves storing feature maps from data points from previous tasks.  The modified OGD method (OGD+) additionally stores feature maps from the current task.  

The primary contribution of this paper is the theoretical analysis of continual learning.  Given that the CL problem does not have an extensive theoretical foundation, the generalization bound in this paper is a notable advance. The theory presented also provides a justification for the empirical observations observed by the authors that as overparameterization increases, the effect of catastrophic forgetting decreases in a variety of CL task setups.  The primary drawback of the paper is that the authors do not compare the OGD+ algorithm to other continual learning algorithms (synaptic intelligence, elastic weight consolidation, etc.).  As a result it is difficult to know how OGD+ compares to alternatives.  It is not clear to the reviewer why improving OGD to OGD+ is itself a contribution.  Given the expense occurred by OGD-type methods in storing ever increasing numbers of directions, it would be important to know the comparison of this method with others.  

Minor comments:

(1) Section 3.2: f^* is not defined as of this point in the paper.
(2) Theorem 1: The theorem needs a quantifier of lambda
(3) Line above Remark 1 k_\tau -> \kappa_\tau
(4) Theorem 2: The paper should define what "is in the memory" means when introducing OGD
s
(5) Theorem 3: Definition of R_T has incorrect dummy index in the summation

review_title: Novel Theory for Continual Learning in the context of Orthogonal Gradient Descent.

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary:

 
This paper studies the theoretical aspect of a continual learning method called orthogonal gradient descent (OGD).
In this study, authors leverage Neural Tangent Kernel and over parameterized neural networks to prove the generalization of OGD. 

##########################################################################

Reasons for score: 

 
Overall, I vote for rejection. I like the idea of the paper to analyze an exist method from different aspect and even improving it. However, my major concern is about the clarity of the paper (see cons below).

 
##########################################################################Pros: 

 
1. The paper investigate an important problem in continual learning framework which is the generalization.

 
2. This paper provides some experiments to show the effectiveness of the proposed framework. 

 
##########################################################################

Cons: 

1. Unfortunately, the paper is not clear and very difficult to follow.
a) For instance, NTK is referred without explaining it well first. 
In page 2, authors use "CL" for referring to continual learning but it has not been defined.
b) There are many typos and capital letters have been used inappropriately.
c) f_t has not been defined.
2- Although the proposed method provides several experiments, there are still many other methods and datasets that have been ignored. There has been recent studies and frameworks that have outperformed OGD, it woud be great if you include them in your baselines. For instance, SGD+Droput in https://arxiv.org/pdf/2004.11545.pdf beats OGD.

3-  There are many metrics to evaluate continual learning frameworks like backward transfer(BWT) or average accuracy over tasks.
I would suggest the authors to look at the defined metrics in GEM (gradient episodic memory), https://arxiv.org/abs/1706.08840, and compute those values. 

 
##########################################################################

Questions during rebuttal period: 

 
- Please address and clarify the cons above .
- Would you please elaborate more what could be the superior performance of OGD+ on Rotated Mnist dataset w.r.t OGD?
- What is the time complexity of OGD+?

 




review_title: Generalisation Guarantees For Continual Learning With Orthogonal Gradient Descent 

Meta-review: The reviewers were excited by the paper's theoretical contribution to continual learning, since that aspect of continual learning is underdeveloped.  However, all reviewers (including the most positive reviewer during discussions) expressed that the paper would benefit from revisions to improve the clarity and the thoroughness of comparisons in the paper.  The paper's focus on OGD is not necessarily an issue for it to be of use to the community, as mentioned as a negative point in one review that other reviewers disagreed with. The authors are encouraged to revise this paper incorporating the reviewers' suggestions.
Decision: Reject