Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors introduce the novel OCL-PDS problem - Online Continual Learning for Progressive Distribution Shift. Authors contrast their problem with that of continual learning and domain adaptation. Authors build 4 new benchmarks and implement 12 algorithms to test these benchmarks.

strength_and_weaknesses: Strengths:
1) Paper is clearly written and the authors point to practical issues with the current approaches. 
2) Authors give very detailed justification of their choices, give details of experiments which will be helpful for other researchers to build on. 

Weaknesses: 
1) The proposed approach still has some practical issues (e.g. how does one decide the divergence threshold, how does one make sure that 3-step procedure benchmark OCL-PDS is applicable to real world problems? 
2) The final takeaway is not clear. Authors should discuss some of the real world problems and give some conclusions based on it. It is not clear what part of the discussion is helpful if a real world practitioner wants to use any of the knowledge in the paper. 

clarity,_quality,_novelty_and_reproducibility: Paper is clearly written. There is novelty but author's claim that their method is more practical is not justified. 
summary_of_the_review: 
1) From Appendix B, one can understand why divergence should be asymmetric. But from the example given, it looks like one needs structured asymmetry. Is just having asymmetry enough? 

2) Figure 1: It is not clear what distribution shift we are talking about in "FMoW-WPDS benchmark." Are most images in 2002 of prison? Are most images in 2009 of Helipads? 

3) Section 2.3: Work is also very related to domain generalization (DG) which is a harder problem compared to domain adaptation (DA) [1]. If one wants to be truly "online" then one DG might be a better setting than DA? Authors should add more discussion comparing and contrasting with DG too. 

4) In general, real world datasets do not have shift continuity. Or there could be too much non-stationarity. How would ODD check or shift continuity check work in that case?

5) How does one decide threshold on Div (D_t || D_t+1)? In the real world dataset, this could be even harder to determine. 

6) I checked the hyperparameters considered In Table 14(in appendix). But the list of hyperparameters tuned does not look complete. The training data generation (domains, dist shift and threshold on div for continual shift) should be treated as hyperparameters too. Can authors comment more on this? 




[1] Blanchard, Gilles, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. "Domain generalization by marginal transfer learning." The Journal of Machine Learning Research 22, no. 1 (2021): 46-100.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper introduces a novel “Online Continual Learning for Progressive Distribution Shift (OCL-PDS)” problem that widely exists in industrial applications from the practitioner’s perspective, aiming to close the gap between academic research and industry. They propose three modifications to the related conventional settings, build 4 new benchmarks from the Wilds dataset, and implement 12 algorithms and baselines including both supervised and semi-supervised methods. Extensive experiments on the new benchmarks also bring some observations. 
strength_and_weaknesses: Strengths:
1. This work first introduces and investigates the novel OCL-PDS problem, which more closely aligns with practitioners’ needs in the industry;
2. This work improves the setting of conventional DA and CL problems at three points, including task-free, forgetting-allowed, and infinite-storage, which is reasonable and more practical, and thus can close the gap between academic work and real industrial applications to some degree.
3. This work requires only remembering the “recent knowledge” and “important knowledge” rather than the knowledge on all tasks and designs a “regression set” to describe the important knowledge.
4. This work releases 4 new benchmarks using the 3-step procedure for the OCL-PDS setting, while adapting and implementing 12 OCL algorithms and baselines, including both supervised and semi-supervised. Their observations based on extensive experiments may help boost the development of OCL algorithms for handling PDS.

Weaknesses:
1. The “infinite storage” this work proposes is somewhat inconsistent with the actual situation, especially in scenarios where the distribution changes gradually over time, such as a recommendation system. The space cost that increases linearly with time T will bring a heavy burden when T becomes large even for the industrial scenarios. 
2. According to observation 2, the online performance depends on how the regression set is defined, and how close it is between the regression set distribution and the overall distribution. So besides the proposed “frequent data” and “critical data”, is there a more general way to construct an appropriate regression set to describe the important knowledge? 
3. It is expected to add First Batch Only (FBO) without the training regression set as a baseline, which can serve as a better lower bound of the online performance than FBO. Because the critical data may not show in the first batch.
4. The proposed i.i.d. offline may not be the upper bound of the online performance, because jointly learning multiple datasets may outperform learning each dataset independently. 

clarity,_quality,_novelty_and_reproducibility: Clarity: Good

Quality: Good

Novelty: Good

Reproducibility: Good
summary_of_the_review: The proposed setting, concepts, evaluation paradigms, and datasets are realistic and may have a large impact on industrial applications.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is intended to formally introduce the online continual learning problem with progressive distribution shift, and also presented four benchmark datasets and 12 algorithms (adapted from existing algorithms) for this setting, and finally presented empirical results of these algorithms on these datasets. 
strength_and_weaknesses: * The paper tries to be practical and bridge the gap between academic and industry, but I don’t really see anything novel. Missing citation: Cai, Sener, Koltun, “Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data”, ICCV 2021. 
* An important contribution is the benchmark datasets, and a 3-step procedure is described, but no examples are shown in the main text how these 3 steps are executed, and how the distribution is shifting with some metrics to demonstrate. I’m not very convinced these benchmark dataset are good. 
* Most of the results are presented in the format of scatter plot of “worst reg perf” vs  “avg online perf”, but it’s not well motivated why this is a good way of demonstrating the benchmarks are good.

clarity,_quality,_novelty_and_reproducibility: * Clarity: overall well written, but as a paper introducing benchmark datasets, the procedure on how the datasets are constructed is only briefly discussed, and the experiments part can be improved, for example, figure titles are not informative at all. 
* Quality: fair
* novelty: the novelty is limited, the problem is not not new, datasets are not new, algorithms are not new, missing citation of similar existing work  
* reproducibility: appears to be good 

summary_of_the_review: This paper attempts to establish some benchmark datasets and algorithms for online continual learning under progressive distribution shift. While a practitioners’ perspective is emphasized, I failed to see much novelty of this perspective, and the empirical results are not presented in a convincing enough way to show the benchmarks are good enough for wide use for the community. 

I would recommend the authors to check out this paper and explain the differences: Cai, Sener, Koltun, “Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data”, ICCV 2021.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: * The paper attempts to bridge the gap between the continual learning settings used in academic work and industrial applications. It introduces the setting of Online Continual learning for Progressive Distribution Shift (OCL-PDS) that considers subtle, gradual, and continuous distribution shifts. 
* The paper proposes three modifications to existing CL settings:
  * Online evaluation and training.
  *  Remember recent and important knowledge described by regression set instead of the complete past knowledge.
  * Infinitely large storage without the replay of all samples. 
* The evaluation is conducted on four benchmarks proposed for this setting based on the WILDS benchmark for supervised and semi-supervised settings.

strength_and_weaknesses: I am well-familiar with the literature and read the full paper in detail. Accordingly, now I will describe the strengths and weaknesses of the paper in the order of originality + significance, quality of the paper, and clarity. 

### Originality
##### Strengths
* The paper takes an important step to develop CL benchmarks based on the issues faced by the practitioners. The proposed benchmarks model the shifts in hot topics, time, and language use and would be of interest to the CL community.
* It provides various insights into the existing methods in the proposed OCL-PDS setting. In Particular, observation two, three, and five would be valuable for developing future methods in this setting.

##### Weaknesses
The paper lacks the positioning and comparison of the proposed benchmarks and observations with many prior works:
* **Proposed benchmarks:** The proposed benchmarks are a direct extension of the WILDS benchmarks. Cai et al. [1] proposed Continual LOCalization (CLOC) – 39 million images over nine years, with 712 classes. The paper refers to Lin et al. [2], which introduced CLEAR with 7.8M images based on the temporal evolution of visual concepts of Internet images in the footnote; however, it does not clarify the challenges in the creation of the regression set for CLEAR following the proposed approach. Both these benchmarks have a much larger scale than the benchmarks proposed in the paper. I would also suggest comparing the CL video benchmarks in the paper benchmarks section, providing the limitations of these already existing benchmarks and the necessity for the new benchmarks.
* **Supervised and unsupervised methods:** The paper does not compare or refer to current online CL methods [3,4]. Further, while the paper does not consider many recent unsupervised/self-supervised CL methods [5,6].

---

### Quality
The paper evaluates multiple benchmarks that reciprocate real-world distribution shifts. It focuses on both the supervised and semi-supervised settings, strengthening the overall experimental evaluation. However, I have various concerns regarding the modifications and the proposed setup that I highlight below:
* The assumption of infinite storage is impractical for most applications. Especially in the online setting, where a large number of instances are collected every second, it is not feasible to store the complete dataset. Furthermore, data storage is often restricted for many applications due to privacy restrictions.
* The choice of the time window for knowledge retention needs to be clarified. There are no guidelines to select this hyper-parameter, and it will lead to the formulation of methods that can only remember the recent knowledge, where “recent” is artificially curated for each benchmark. Further, the importance/critical data for the regression set is not well-defined. It is also unclear whether the regression set should change or update depending on the change in the criticality of the data.

---

### Clarity
The paper was overall clear and well-written. Following are a few suggestions that highlight the missing details and should help improve the overall presentation:
* ***Details about regressions set:*** The selection of the regression set should be elaborated in the paper. For instance, for FMoW-WPDS and Amazon-WPDS, how is the regression set constructed? Does it contain all the samples from America, Asia, and ten product categories for FMoW-WPDS and Amazon-WPDS, respectively?
* ***Clarity of the figures:*** The figures for the experimental results are not interpretable. For instance, in Figure 2, each point corresponds to one algorithm according to the caption. However, the figure has no labels, and it is challenging to analyze the performance of any method in the figure. The tables in the supplementary material are much easier to read than the figures; however, the notations could be more straightforward and concise.

---

### Reproducibility
The paper included the code in the supplementary, which would promote the usage of these benchmarks for future CL papers.

---

### References
[1] Cai et al. Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data. ICCV 21.  
[2] Lin et al. The CLEAR Benchmark: Continual LEArning on Real-World Imagery.  NeurIPS 21.   
[3] Yin et al. Mitigating Forgetting in Online Continual Learning with Neuron Calibration. NeurIPS 21.   
[4] Yoon et al. Online Coreset Selection for Rehearsal-based Continual Learning. ICLR 22.   
[5] Madaan et al. Representational Continuity for Unsupervised Continual Learning. ICLR 22.   
[6] Fini et al. Self-Supervised Models are Continual Learners. CVPR 22.   

clarity,_quality,_novelty_and_reproducibility: I have covered all these aspects in the above section. Therefore, I use this part of the review to clarify the remainder of the questions for the paper:
* Since the paper assumes infinite storage, what was the buffer size for replay during continual learning? 
* How was the random label feedback adopted in the evaluation? Does the paper consider the fraction of users for evaluating the model output?
* The paper experiments with pseudo-labeling semi-supervised methods; was there any specific reason to consider these methods instead of using self-supervised methods that do not require labels?

summary_of_the_review: While the paper is interesting and tackles a significant problem, it is not ready for acceptance in its current form. Remarkably, the paper lacks a comparison with existing benchmarks and CL methods, the modifications to the CL settings could be better-motivated and appropriate for many practical scenarios, and the paper's clarity (especially the results) needs to be significantly improved.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes several benchmarks for evaluating progressive distribution shift (PDS), and evaluated many baseline continual learning (CL) methods on these benchmarks.

The critical reviewers objected to the 3 step construction method being arbitrary, missing references and most importantly that the results are unclear. The authors addressed the missing reference and made further arguments on novelty. As reviewers (FxrK,  6758, UPWo) pointed out, the findings of the paper are unclear, especially on the quantitative side, and the authors did not address this important point sufficiently. For instance, in figure 2, the first two sets have positive correlations as mentioned in observation 1, the next one negative, and the last no correlation. The paper did not take a stand on which CL methods are better on the benchmark or how to better study the problem. It also misses positioning with respect to state-of-the-art CL methods and other related benchmarks (UPWo). The main empirical results are 5 hedged observations. Thus this work does not seem to "provide tools to better handle realistic PDS and design better algorithms". 

Note that there is a rating 10 review. The review repeated the paper's main arguments and but did not convey why this is an exciting paper with such a high rating. In fact it pointed out a few problems that other reviewers also pointed to, so this review was treated as a regular positive review as its text indicates.

The paper contains a large appendix ranging from an exposition of various losses to dataset statistics, making the paper difficult to read. The author is encouraged to exercise more editorial judgment to identify and focus on the main points of the paper.


justification_for_why_not_higher_score: There are important flaws identified by the negative reviewers and not addressed by the authors. The paper includes a lot of information but does not seem to have main points.
Decision: Reject