Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: I am very torn about this paper as the experiments are thorough and the results are quite significant. The results are on four popular benchmarks and they also test on Many Permutations, which is an important finding as well.  While the angle based approach presented here is interesting and intuitively appealing, I am not really sure whether it is a needed generalization beyond past approaches. I don't really have a strong intuition for where the gains are coming from. As such, I think the paper would be much stronger if it could intuitively or theoretically get at why it improves on past approaches. 

Actually, one of my biggest concerns is how good the performance of the direct approach is. The direct approach seems like a straightforward extension of experience replay with a special learning rate for current examples vs. buffer examples that is adaptive to the relative loss.  While it is potentially interesting that this approach works so well, it is not positioned as the selling point of the paper as currently written. Additionally, this is kind of an orthogonal contribution to past papers leveraging experience replay, which could potentially be modified and improved based on this approach. I wonder if the authors tried modifying any of the baseline approaches in this way as well. 

A small note on the experiments: the authors closely follow settings from past work and at times it can be unclear which results were implemented in this paper and which in past papers. I don't think that you have addressed what hyperparameters you chose for MER. It also seems to perform worse than past reported results on CIFAR and is not reported despite the best past performance on MNIST Permutations. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: ========================== Summary of paper ==========================
The paper proposes a new loss balancing approach for lifelong learning. The proposed method dynamically balances the old task loss (from an episodic memory) and the new task loss based on their magnitude. The paper outperforms state-of-the-art method (that don't use extra attribute info), and is straightforward to understand.

========================== Decision ==========================
I vote for rejecting the paper. The paper adds a very simple dynamic loss balancing to a joint loss, which has limited novelty, yet does not discuss its relationship with loss balancing in multitask learning. Although the method outperforms state-of-the-art, a very simple baseline of their method also outperforms state-of-the-art, making the contribution ineffective. The writing of the paper may also benefit from further edits.

========================== Pros/Cons ==========================
+ The paper outperforms state-of-the-art.
+ The method is simple to explain and straightforward to implement.
+ Proposes a weighted variant of forgetting measure (although the purpose is not justified -- need to discuss what use case would make one prefer this weighted variant better than the original).
- The paper is not well-placed in the literature. Not until the bottom of page 5 can readers see very closely related methods, and despite the similarity, little is discussed about the difference. To improve, relationships with existing multitask learning weight balancing methods (e.g. https://arxiv.org/abs/1810.04650, https://arxiv.org/abs/1705.07115, https://arxiv.org/abs/1711.02257) should also be discussed, and maybe compared to.
- Motivation to use the proposed loss balancing rather than that of very similar methods (e.g. GEM/A-GEM) is lacking ("These approaches cannot capture the dynamics in the lifelong learning process", but GEM's balancing is also dynamic)
- Solving for beta using optimization (eq. 10), while (eq. 9) should have an explicit, close-form solution.
- Writing sometimes is confusing, uses absolute language, or using claims unsupported by evidence.
    - (1) page 3 "In this case, the weights are only optimized for the current task while ignoring previous tasks which leads to catastrophic forgetting." ignores the existence of regularization-based methods such as EWC
    - (2) bottom of page 3 "alpha1 and alpha2 should be adjusted adaptively" does not have experiment results supporting it (especially considering GEM is also adaptive)
    - (3) "l_ref(w; ζ) = 0 implies that there is almost no catastrophic forgetting" claim is problematic. Overfitting to the episodic memory is a common problem.
    - (3) page 3 xi and zeta not clearly defined. 
    - (4) brings up NP-hardness while it is seldom of interest in this field.
    - (5) missing "∇" on the denominator in (eq. 7).
- Experiment:
    - The only ablation (the direct approach) is statistically indistinguishable from the proposed method. This also outperforms state-of-the-art, while it should not be. One can only assume that the experiment is problematic.
    - Completely uses hyperparameter from an unpublished paper.
    - Claims state-of-the-art, yet omits a state-of-the-art variant in cited A-GEM paper (with joint-embedding). This should be discussed even if the comparison is unfair.

========================== Improving the paper ==========================
- Rewrite the introduction so it is clear the paper is an improvement over A-GEM with a better loss balancing, and focus the motivation and side experiments on why this is important for lifelong learning.
- Clean up writing.
- Explain why the direct approach, while being a basic loss balancing method, outperform state-of-the-art GEM greatly. Perhaps a set of ablation studies can help.
- Replace optimizing for beta as solving for beta.
- Comparison with other loss balancing papers.
- Clarify state-of-the-art comparison.

========================== After rebuttal ==========================
I appreciate the authors addressing my concerns about the placement in the literature; it looks clearer now. However, the rebuttal did not address well the question why the simple MEGA-D variant outperforms prior state-of-the-art besides promising to publish code. The new ablation study is especially difficult to understand, since the first set of results are MEGA without using both memory and new data, but it seems to me that MEGA *needs* the memory data to work, so I have no idea what is being ablated here, and I also do not understand how this answers the question why MEGA-D performs so well. I will not be changing my rating.

Other issues:
(1) Citing A-GEM: please update the bib file so it shows the ICLR version instead of arXiv.
(2) Solving eq. 9: Here you go: https://imgur.com/a/b4QCVlv
I literally had this problem on one of my high school exams. Feel free to use this in any future version.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: UPDATE:

I thank the authors for proactively engaging with the review process and improving the paper.

After considering the other reviews and discussions with other reviewers, I also share the concern that the simple MEGA-D baseline performs very well, with little additional gain from the full MEGA approach (only on the many permutations case that was introduced in the rebuttal). Unfortunately, it doesn't look like this point has been fully addressed.

I know this is part of the contribution, and I am certainly an advocate of simple techniques that yield strong results. However, as it stands, this baseline is only mentioned briefly, with a single paragraph in the method section and a single paragraph on evaluation. Given the strength of the result, I think a lot more of the paper should be devoted to understanding the merits of this simple method and evaluating how it relates to the proposed angle-based approach.

I am also a bit confused by the new baseline; given that the memory and current batch are both used for the MEGA-C case, I think the explanation of how this differs from the full case could be clearer.

As such, I must regrettably change my score to a 3. I think this paper has potential; and with a bit more analysis and clarity on the above points, could be a good submission. I encourage the authors to address these for a future publication. 

==============================================================

This paper describes an approach to perform continual learning by maintaining an episodic memory / coreset of old examples and learning a linear weighting function between the gradients from new and old samples. A very simple direct method is proposed, as well as an angle-based approach. In the latter, projected gradient ascent is used to find an optimal rotation angle for the current data gradient such that the resulting direction also aligns well with the gradients computed from the memory buffer. This appears to generalise previous work (such as GEM and A-GEM), and a new metric of long-term remembering (LTR) is also introduced.

The experiments are comprehensive and compelling.
The paper is clearly written and easy to follow, and I think it could be quite a good contribution to the conference.

I have some questions and concerns that I think should be addressed first:
1) If I understand correctly, Algorithm 1 seems to indicate that every single batch is added to the memory buffer - I assume this is an error, as it is suggested throughout the paper that only a small buffer is used. How is the memory buffer updated?
2) It is unclear how much memory is required for this approach and whether this is consistent with previous approaches. An ablation over memory size would help with this (ideally with comparison to other episodic memory-based approaches); and a discussion on memory use of different methods is needed.
3) With the direct approach, it seems odd to specify a loss threshold of zero to determine that the current task performance is high. What loss is being used? Further, how does the direct approach relate to eg. GEM/A-GEM/MER in terms of weighting between old and new samples?
4) The related work section is quite thin, and there are several other works that could be cited; currently they seem to be focused on just "gradient-similarity based continual learning" with a few other continual learning works.
5) The paper states that progressive networks increase in memory super-linearly, but I don't believe this is the case; it would be linear or sub-linear, given that new tasks would typically benefit from forward transfer and require fewer additional units.
review_title: Official Blind Review #3

Meta-review: The submission is concerned with the catastrophic forgetting problem of continual learning, and proposes a gradient-based method which uses buffers of data seen previously to integrate the angles of the gradients and thereby mitigate forgetting. Empirical results are given on several benchmarks. 

The reviewers were impressed with the thorough validation and strong results, but noticed that the much simpler MEGA-D baseline did almost as well. Given this, they were not convinced that the proposed approach was necessary. Although the authors provided a strong rebuttal and an additional ablation, the reviewers did not feel that their concerns were met.

My recommendation is to reject the submission at this time.
Decision: Reject