Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces three main results:

- a theoretical analysis of SGD under distribution shift to highlight limitations of current OCL approaches
- a new OCL algorithm based on moving averages (AMA)
- an associated heuristic to tune the learning rate also based on moving averages (MALR)

other smaller contributions are a population-based approach to tune the EMA weights and empirical evaluation on multiple datasets
strength_and_weaknesses: The strength of the papers are mainly in the simplicity of the resulting algorithm, and in the thorough exposition of the result (attempt at theoretically grounding the approach, multiple ablations, etc...)

The main weaknesses are:

a) the algorithm analysis is only for unlimited memory SGD. The final algorithm proposed is finite-memory and employs EMA. This is an enormous disconnect, and makes Thm. 1 more of a help to exposition in the paper than as a contribution. Section A.2 only gives a very brief and informal idea of how parts of this gap can be filled. The authors should either report a theoretical result more aligned with what the algorithm is actually doing, or clarify that Thm. 1 is mostly for illustrative purposes.

b) The final algorithm seems simple but hides its complexity behind several design choice. A partial list of values to tune includes K_W, K_V, K_M, delta, and how the train/validation splits should be chosen in practice both for AMA and MALR. The conditions C1-3 also introduce extra tunable parameters, without much discussion (e.g. what happens without C3?). Only a fraction of these values are ablated in the appendix. Giving advice on how to improve this is hard, as even the obvious thing (extensive ablation study) might change greatly from dataset to dataset.

c) The only baseline is SGD+RWP. This is enough to justify that the approach is superior to SGD+RWP, but many other approaches for OCL exist in the literature with varying degrees of complexity. Even a quick search yields several off-the-shelf tools to set up some baselines (e.g. https://github.com/ContinualAI/continual-learning-baselines) and at least a few of them should be included to gauge how much AMA-MALR can learn on these datasets.
clarity,_quality,_novelty_and_reproducibility: The paper is relatively clear, with good explainations of the reasoning behind the algorithm choices. The main improvements from a clarity perspective would be a better listing of all the tunable parameters and their effect, and more explainations on the gaps between the theoretical inspiration and the final algorithm.

The final algorithm is definetly novel, mostly as a new combination of existing techniques from optimization. The theoretical analysis is a straightforward extension of (Ghadimi & Lan, 2013) and (Besbes et al., 2015).

The paper is overall good quality, but falls slightly short since the final performance is not really compared with competitive baselines, and the theoretical analysis is quite far from the algorithmic part.

Despite the large numbers of hyperparameters, the authors make a good job in taking care of reporting them in the paper to help with reproducibility.  
summary_of_the_review: Overall the paper falls a bit short of the acceptance threshold due to the lack of focus in the contribution, where the authors try to provide results from three sides (theoretical, algorithmic and experimental) but only achieve small gains in each. In particular:
- The theoretical contribution is a straightforward extension of existing results, and only act as a guideline in designing the algorithm as it cannot be applied to it. This can be fixed by a more realistic analysis of AMA+MALR.
- The algorithmic contribution is novel and based only on familiar and well established tools, but its impact is reduced by the large number of hyperparameters involved in the actual implementation.
- The experimental results are good, but not rigorously evaluated against strong baselines and might be greatly influenced by the large number of hyperparameters.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: 
In Online continual Learning scenario, where one needs to deal with non-iid data stream, information retention can be a problem, even using unlimited replay buffer. In this study, applying naive Stochastic Gradient Descent and using constant or decreasing learning rate has been partly associated to the information loss in long term. The authors put forward using heuristics based on adaptive moving average to improve the optimization process, and also propose heuristics for learning rate scheduling. 
strength_and_weaknesses: Indeed Online continual learning is a challenging scenario and the authors opted for an interesting research topic. 

In my opinion this manuscript could be reorganized in a way that it is more to the point and explain how it is differentiated to the other relevant approaches in the past.   
It would be interesting to discuss standard Adam and RMSProp and compare with the proposed heuristics in terms of time and performance. 
Although it is mentioned that the authors used a population based search to find the best adaptive point within a convex hull for \gamma_k, I think making the effort to include the proof could indeed improve the work.
The overhead of the proposed approach seems to be high, how would you justify it in an online continual learning scenario. 

clarity,_quality,_novelty_and_reproducibility: The manuscript can be improved in terms of quality. 
The proposed heuristics seem interesting but might not be enough novel to be published in this conference. 
summary_of_the_review: In my opinion this study needs more work to be organized and to get into the shape to be published.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studied the problem of improving information retention in online continual learning (OCL), the convergence of SGD in OCL are theoretical analyzed. Besides, an Adaptive Moving Average (AMA) Optimizer and a Moving-Average-based Learning Rate Schedule (MALR) to optimize the pure replay objective online are proposed. Experiment on several large scale continual learning benchmarks demonstrated the effectiveness of AMA+MALR.
strength_and_weaknesses: Strength: 
This paper considers an interesting problem, it is clearly presented and readable. The results of numerous experimental results are convincing. The proposed method is easy to implement. A wide range of related studies are discussed.

Weaknesses: 
1.The typesetting in some places is out of order, such as equations (23), (25), and (31).
2. In Page 4, "A4 bounds the degree of non-stationarity between consecutive iterations", why this assumption holds?
3. This author should add more description about the contribution of this paper.
clarity,_quality,_novelty_and_reproducibility: Clarity: Excellent
Quality: Good
Novelty: Good
Reproducibility: Good
summary_of_the_review: This paper is well written and studies an interesting problem, I think it is marginally above the acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies various adaptive training methods and schedules in online continual learning.  
strength_and_weaknesses: Strength
Studying different training methods is interesting in online continual learning. 

Weaknesses
My understanding of information retention is to avoid catastrophic forgetting. Why not use the standard terminology?

Online continual learning is a non-stop process, I don’t understand what you mean by convergence. In standard online continual learning, whenever a small batch of data from the stream is collected it is trained in one iteration. New tasks of completely different distributions (with new class labels) come constantly. In such a non-stationary data setting, it is unclear how you know when the convergency should happen.  

You wrote “We allow 80 training iterations per time step,” but that is not the standard online continual learning setting. My understanding is that you are doing online learning rather than online continual learning. In online continual learning, it is assumed that the system sees the data only once. Once a small batch of training data comes it is immediately trained in one iteration. Some classes as a task may come early and some classes may come later in a sequence of tasks. So, the distribution change at the task switch is drastic and the time of each such switch is unknown. I don't think that you are working under this setting, but I am not familiar with the datasets that you are using. In the standard online continual learning setting, it is unclear how your adaptive training can be applied. 

Again, I think you are working on online/incremental learning rather than online continual learning. Then, you should compare with methods in online learning literature. I'm not working in that field, so I cannot give you any references off the top of my head. But I did read some papers in the area a while ago. 

Since your data is not commonly used in online continual learning, it is hard for me to see how your method performs against other methods. There are many online continual learning benchmark datasets. I think you should compare with those existing online continual learning baselines using the existing benchmarks. I am surprised that you did not compare with any existing online continual learning baselines, but only various variants of training methods. If no comparison, I won’t know that your information retention is better. For example, some recent systems are (you can find a lot of other baselines in these papers): 
1. Mai et al. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning. CVPR workshop’2021.
2. Bang et al. Rainbow memory: Continual learning with a memory of diverse samples. CVPR’2021.
3. Guo et al. Online Continual Learning through Mutual Information Maximization, ICML’2022. 

If you are working under a different setting, you should have compared with the existing setting and stated the differences. The existing online continual learning systems should be easily adapted for your setting by simply learning one task of all classes. Actually, in the case, existing online learning methods may do better.

It is unclear whether you are using your adaptive training schedule at each time step or in the time span of the whole training process. My guess it is the latter. 

The figures in Figure 1 are too small to see. 

clarity,_quality,_novelty_and_reproducibility: Mostly clear
No code uploaded. 
summary_of_the_review: I think the paper is studying online learning rather than online continual learning. It didn't compare with any existing online learning or online continual learning methods. The adaptive learning methods and schedules proposed in the paper are not  new as they have been used in other areas.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Unfortunately, the reviewers were not enthusiastic enough about this paper for it to be considered for acceptance at ICLR 2023. There are just too many papers that reviewers were much more enthusiastic about this year, and ICLR has a very low acceptance rate. The authors are encouraged to take the reviewer comments very seriously, even if there are things you disagree with, and make sure that all issues are addressed, and any potential sources of confusion are completely eliminated, in the next version of the paper. Also, please ensure that these are addressed in the initial paper submission for that next conference. 
justification_for_why_not_higher_score: see above
justification_for_why_not_lower_score: see above
Decision: Reject