Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a new optimization method for continual learning. The authors propose a task-aware optimizer to adapt the learning rate for each task. The proposed method is evaluated on several datasets to show its effectiveness.

main_review: I think the idea is clear, and the experiments verify the effectiveness of the proposed method. However, I found the results of A-GEM in this paper are significantly worse than the original paper. For example, the accuracy of split-CIFAR100 is 54.25% in this submission, while in A-GEM the accuracy is ~62%. Could you provide more details?

In addition, though the submission addresses the catastrophic forgetting by proposing a new optimizer, the discussion of task correlations are very similar to GEM and its variants (e.g., A-GEM) and CLAW (CONTINUAL LEARNING WITH ADAPTIVE WEIGHTS). More analysis about why the proposed method can outperform other approaches would make the paper more convincing.

Missing reference: [1] is a very relevant work. I think it might be valuable to compare and discuss with it.

Btw, the authors change the margin of the ICLR format. Please kindly revise it in the next version.

I am willing to adjust my score if the authors can address my concerns.

[1] Guo, Yunhui, Mingrui Liu, Tianbao Yang, and Tajana Rosing. "Improved schemes for episodic memory-based lifelong learning." arXiv preprint arXiv:1909.11763 (2019).


---After rebuttal --- 

The authors' response does partially address my concerns. After reading the authors' rebuttal and other reviewers' comments, I still think the contributions are not enough for publication. I will keep my score.
summary_of_the_review: The idea is clear and the results seem promising. However, the authors are supposed to provide more analysis to understand why the proposed method works.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present an approach for lifelong learning where each task is processed in a single pass. They propose to adapt the learning rate of the training algorithm depending on the current task's similarity to the previously observed tasks. The learning rate is decreased when there are many dissimilar tasks to avoid catastrophic forgetting. The paper presents an adapted RMSProp algorithm, but the procedure can be adjusted to Adagrad and Adam as well. The presented evaluation on computer vision datasets shows that the proposed modification helps to increase the final accuracy of those methods while keeping the forgetting low. When compared to baselines from literature under the conditions of the single pass setting, the proposed method achieves better final accuracy and keeps approximately the same forgetting rate. 
main_review: Pros:
* General approach that can be adapted to multiple optimization algorithms and used in other meta-algorithms
* Strong performance in a single-pass scenario
* Can achieve better learning accuracy than naive methods

Cons:
* The effect of the proposed procedure on optimization algorithms is not studied enough. For example, it’s unclear how would a few similar tasks help to learn faster when there are many distinct tasks present. Additional insights, like visualizations of trajectories, will help to build an intuition and boost the method adoption in practice. 
* Effect of task order is not studied/highlighted: the method seems to be very dependent on the task order and it is unclear how the order is chosen in experiments.

Minor comments/questions:
* what is the point of having \lambda(t,t) in eq. 3, i.e. why do you need to multiply the currently accumulated gradients by task similarity to itself?
* In appendix 3.3, Figure 5(a) there is a noticable difference between RMSProp and TAG-RMSProp on the first task. What is the reason for this? Shouldn’t they be identical/close?
summary_of_the_review: The presented method is a general procedure that helps to improve performance of many methods in lifelong learning scenario. The presented experimental results are convicing. However, the paper would benefit from more research and intuition on why the approach works. I recommend to accept the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes TAG, a method for continual learning in the task-incremental setting. This method relies on storing and using task gradients while learning a set of supervisd tasks sequentially. The influence task-based accumulated gradients is regulated through a learning rate that is adaptive according to the relatedness of the current task with the previously observed ones. The authors report results on benchmark datasets for continual learning of up to 20 disjoint tasks. Comparisons against naive non-continual optimizers such as SGD, Adam and RMSProp are reported, along with results in the continual learning setting with some state-of-the-art methods such as EWC, A-GEM and ER. The authors report performance in terms of overall accuracy, forward transfer and learning accuracy (LA).
main_review: Strenghts of this paper are: 
- The method provides a simple yet effective way of dealing with catastrophic forgetting, which is to some extent original since it proposes to use the relatedness among tasks and use this information to control gradient updates while learning new tasks.
- The method is technically sound, and most of the experimental evaluation is aligned with typical evaluations in the area, including datasets used and reported metrics.
- The paper is well-written, well-structured and easy to follow.

Weaknesses of this paper are: 
- My main concern is regarding memory/storage size requirements of the proposed method (during training). Although the proposed method avoids the need to keep examples of previous tasks, as compared to replay-based methods, and of expanding the network, compared to network expansion methods, it certainly adds memory/storage requirements of the gradients. Based on this, I would expect to see comparisons of the proposed method vs. replay-based methods and network expansion methods, in terms of memory usage.  I think that these concerns need to be fully addressed in this paper, to really demonstrate the advantages of the proposed approach. Furthermore, I would expect to see these comparisons for a reasonably large number of tasks, since memory/storage requirements increase along with the number of tasks. 
- My second biggest concern is where are the actual gains of the proposed approach. As I can infer from Table 1, most of the gain in final accuracy is due to a gain in LA. According to the definition of LA, this is actually the accuracy of each new task. Therefore, I do not agree with the claim in page 8: "The higher LA with similar Forgetting as compared to other baselines shows that while TAG exploits the adaptive nature of existing optimizers, it also ensures minimal forgetting of the gained knowledge... Hence, even if a similar (or lower) Forgetting occurs in TAG, the higher test Accuracy (with high LA) shows that TAG is capable of retaining the gained knowledge from each task.", since from Table 1 forgetting levels are actually very similar to those experienced by other methods and therefore a higher accuracy (which I agree is due to LA) comes mainly from learning new tasks better rather than encouraging "minimal forgetting" or "retaining the gained knowledge" as it is claimed. I would strongly suggest to clarify this point. 
- In the experiments, I do not see the reason for not comparing with network expansion methods. The fact that the proposed method is not making any change to the size of the model, as mentioned in page 8, does not create any limitation for comparing with these kinds of methods in the experimental setting used in the paper. 
- Finally, I see a lot of similarities of the proposed method with existing methods to control gradient updates such as OGD [1] and OWM [2]. Therefore, I would expect comparisons to these methods. 

[1] Farajtabar, M., Azizan, N., Mott, A., & Li, A. (2020, June). Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics (pp. 3762-3773). PMLR.

[2] Zeng, G., Chen, Y., Cui, B., & Yu, S. (2019). Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8), 364-372.
summary_of_the_review: Although the paper proposes a simple method for controlling catastrophic forgetting in the continual learning setting, there are major drawbacks flaws in the experimental results and the evaluation of the proposed approach. Given the nature of TAG, I think is is fundamental to measure and report memory/storage requirements in comparison to other methods that also require additional memory/storage such as replay-based methods and network expansion methods. Furthermore, claims around avoidance of catastrophic forgetting in the main results reported in Table 1 and explained in page 8 are not well-supported since it is clear that the method is strong at learning new tasks, while not necessarily much better than counter parts regarding catrastrophic forgetting.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a  task-aware adaptive learning rate method, TAG, for continual learning. The optimizer TAG is to promote the learning rate if they are similar to previous tasks, while decreasing the learning rates if they are dissimilar to previous tasks without storing previous examples. The authors combined the proposed method with existing optimizers, such as Adam, SGD, etc to demonstrate the effectiveness of the proposed method. Experimental results on several datasets show the improvements over naive optimizers.  

main_review: Strength:

*  The paper is clearly written and easy to follow
*  Experiments on different settings are considered. 


Weakness:
*  The technical novelty of the proposed method is limited. In fact, TAG is similar to LA-MAML [1] (Gupta et al.,2020). Both the motivations and derived formula (Equation 2) of TAG are very similar to [1]. From this aspect, TAG can be seen as a memory-free version of LA-MAML. 

* misinterpretation of related works.  The author claims that “Another related work (Gupta et al.,2020) also employs an adaptive learning rate while requiring a small episodic memory. But it is based on a
meta-learning setting and hence beyond the scope of our paper”. This is an incorrect statement, LA-MAML  (Gupta et al.,2020) is to solve the continual learning problems with the tools from  meta learning. They are not in meta-learning setting, but instead in the continual learning setting.

*  The proposed method seems to only work on the task-aware setting. More recent works focus on the task-free continual learning setting, where the task identity and boundary are unknown during the learning process.  It would strengthen the proposed method if  TAG can be applied in such scenarios. 

*  The experiments on more other architectures, such as MLP may demonstrate the effectiveness of the proposed method. 






summary_of_the_review: The method novelty is limited, and experiment needs to be improved.
review_title: null

Meta-review: The authors develop a memory-based method for continual learning that stores gradient information from past tasks. This memory is then used by a proposed task-aware optimizer that, based on the task relatedness, aims at preserving knowledge learned in previous tasks.

The initial reviews were reasonable but indicated that this paper was not yet ready to be published. In particular, the reviewers seemed to agree on the somewhat limited methodological novelty of the paper given prior work (such as LA-MAML and OGD in terms of method and GEM in terms of task similarity comparison).

In their response, the authors do seem to agree to a certain extent with some of the criticisms, but also point to clear differences with respect to previous work (and other distinguishing aspects such as a smaller memory footprint than OGD). The authors also carefully responded to reviewer comments and provided additional results when possible.

In the end, the main criticism from the reviewers remained (Reviewer 95tf also suggests that the authors should compare their method to others in terms of memory consumption (which the authors partly did) and compare to replay-based methods) and this paper was a borderline one. Three, out of the four, reviewers suggest that it is not ready to be published. One reviewer did give it a high score (8) but also understood the limitations raised by the other reviewers. As a result, my recommendation is that this paper falls below the acceptance threshold. 

I am sorry that for this recommendation and I strongly suggest the authors consider the reviewer's suggestions in preparing the next version of this work. In particular, it seems like providing a full study of the memory usage of your approach vs. others as well as providing more insights about the "trajectory" (see the comment from ZR5n) might go a long way toward improving the paper.
Decision: Reject