Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: 1.The authors build CLEAR from existing image collection (YFCC100M) by proposing a novel low-cost visio-linguistic dataset curation approach.
2.The authors find that state-of-the-arts CL algorithms that only utilize fully-supervised data fall short whereas unsupervised pretraining provides significant boost.
3. The authors introduce a biased reservoir-sampling algorithm that dynamically caches more recent training data, achieving the new state-of-the-arts while still leaving large room for improvement.
strengths: The major strengths of CLEAR over prior CL benchmarks include (1) smooth and realistic temporal evolution of visual concepts with real-world imagery, enabling a more practical "online" (i.e., train on past, test on future) evaluation protocol (2) high-quality labeled data along with abundant unlabeled samples per time period for continual semi-supervised and unsupervised learning.
weaknesses: The author only compare linear model with ResNet18. Is it enough for this task?
correctness: Yes
clarity: Yes, it is well written.
relation to prior work: Yes.
additional feedback: Explain the above weakness.
review_title: Good paper

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work introduces a new benchmark for continual learning. It compellingly describes many of the existing benchmarks as insufficient (due to either being artificial or not truly aligning with the premise of continual learning).

In general, the proposed method for collecting the data, filtering etc seems reasonable. The authors gave obviously given a lot of thought to making a good dataset. I am concerned by the number of "moving parts": the choices of the dynamic visual concepts, the chunking into 10 parts, the usage of CLIP and finally mturk. Each of these steps has its own set of design choices and hyper-parameters and it's not clear just how robust the whole procedure is to minor changes in each of these steps (meaning, would we get a similar kind of data?). I wonder if the reservoir sampling details in Section 5 are also a bit overfit to CLEAR, but that is a minor issue.

All in all, I like the benchmark and I think it will be useful to the community. It has a more real world nature to it so likely models evaluated on it will have more practical value.


strengths: [see review]
weaknesses: [see review]
correctness: No red flags.
clarity: Yes.
relation to prior work: Yes.
additional feedback: N/A
review_title: interesting new benchmark for continual learning

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This work proposed a new continual learning benchmark, named CLEAR, from the publicly available image collection YFCC100M. The proposed benchmark considered the temporal evolution of visual concepts over time from 2004 to 2014, and the data construction process is scalable and semi-automatic by using the pre-trained CLIP for label annotation and then human annotators for curations. Finally, several baseline methods and analysis were performed to show the necessity of the proposed benchmark.

strengths: (1) The proposed benchmark considered the realistic temporal evolution of images and their concepts in continual learning, which I think can better capture the dynamic nature in real scenarios.

(2) The combination of using CLIP for coarse-level data annotation and relying on humans for a fine-grained curation looks interesting. I agree that this can improve the scalability of data collection.

(3) An alternative online evaluation protocol has been developed for better capturing the domain shift, by always using today’s data to train and tomorrow’s data to test. 

weaknesses: (1) The gap between offline accuracy and online accuracy seems to be quite marginal for different baseline methods. For example, in Table 2, LwF [27] achieves 76.4% offline accuracy and 77.3% online accuracy. Therefore, the proposed continual learning benchmark is not more challenging than the offline evaluation protocol (i..e., on the i.i.d. data). 

(2) Unsupervised pretraining on YFCC-S0 (0.7M images of segment 0) achieves a 90%+ online accuracy. Wouldn’t it mean the proposed benchmark is too simple and there is not much room for improvement? Why do we still need the proposed benchmark for continual learning? Or does it mean unsupervised learning is the most promising solution to continual learning problems?

(3) Similar to CIFAR-10, the dataset also has 10 class labels, which dynamically evolve over time. Not sure what rationale behind this choice of labels, but it seems to me that several class labels are not exclusive to each other. For example, “cosplay” and “dress” could co-occur in the same images. Also, some labels are more closely related concepts than others, such as “dress” and “sweater” vs. “hockey” and “baseball” vs. “computer” and “camera”. 

(4) How many labeled images are left after human curation?  





================

Post-rebuttal update:

I thank the authors for providing very detailed and thoughtful feedback. Some of my concerns have been clarified, but I still feel that more efforts have to be done to make this dataset more challenging for both CL and self-supervised methods. Therefore, I raised my rating from 4 to 5.
correctness: The dataset is constructed in a sound way, though there are some unclear issues I mentioned in the above. The evaluation methods and experiment design are appropriate and performed correctly.

clarity: The paper is overall well written. Typo: In Figure 2, “visio-” => “visuo-”. 

relation to prior work: The prior continual learning benchmark and baseline methods have been clearly discussed in the Related Work section.

additional feedback: N/A
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper introduces the CLEAR recognition benchmark, consisting of 10 temporally evolving concepts across a real world time frame of roughly a decade. The data has been extracted and curated from the large YFCC100M data repository with the help of a multi-modal CLIP model and MTurk crowd-sourcing. Further, the authors propose to change continual learning evaluation scenarios and evaluation protocols and claim the introduction of a “novel” reservoir-sampling algorithm for continual learning.

----------------------- 
Post rebuttal update: 

The authors have done a great job at addressing many of the concerns and comments raised in my initial review. This includes some clarification with respect to the unsupervised procedure, sharing code and a versioned Zenodo repository, and several acknowledgements of suggestions to improve the writing. I am thus raising my rating. At the same time, I however feel that the amount of suggested changes are too many to give the paper a direct accept rating without being able to check a potentially updated version. There are also some persisting concerns, see response thread below, especially in the misleading presentation of the related works in table 1 and the ethical aspects in the dataset sheet. 


strengths: * The idea of constructing a dataset where concepts naturally evolve over time is certainly interesting. I do agree with the authors that continual learning is in dire need of data sets that are more realistic than permuted MNIST or contrived incremental scenarios. 
* The data selection and curation method seems reasonable. I can imagine the combination of CLIP + MTurk resulting in accurate labelling at a much higher throughput. 
* An “online” setting, here meant as a streaming setting where data cannot be revisited, can be argued to be intuitively closer to real-world applications. However, also see weaknesses on this part. 
* Authors provide a lengthy supplementary material with many nice additional details. The dedicated homepage is also a plus. 
weaknesses: Although the central idea and the proposed dataset is interesting, the way the paper is presently structured comes with some major weaknesses. In my opinion, the paper could be dramatically improved if it were more focused and didn’t attempt to do too many things at once. Right now, unfortunately none of the introduced aspects are very well executed. The paper claims multiple contributions, but I believe some of these are over claimed or partially conflated. Following this point, my biggest concern is the framing of the paper. Initially the paper starts of well with a motivation behind CLEAR, but then somehow diverges into assumptions, desiderata, evaluation, together with potential algorithmic novelties, with a constant connotation that this is the right way to go about and that continual learning up to date is somehow inherently flawed. The dataset itself occupies a rather minor portion of the paper, which is unfortunate. More specifically:

* The authors seem to partially conflate “continual learning”, as a general desideratum, with what is at large a question of domain adaptation. The idea behind continual learning is that there may be various forms of distribution shifts over time.  I find that the authors too readily dismiss that e.g. class incremental scenarios may be valuable. For instance, I do not think that it is unrealistic that I train a system on say distinguishing 1000 flower or insect subspecies, and then later want to add some more to this system. Most importantly, I find that the authors’ evaluation seems to be tailored to fit their narrative. They advocate to measure the system’s worth on the “next-in-line” data, whereas continual learning would care about maximizing performance over all time steps. First, if I mainly care about just the most novel (or future) task, then there is no need for “continual” learning, I can just adapt to my most recent domain unidirectionally. Second, if this is the primary criterion of evaluation, then it is not at all surprising why e.g. favoring more recent instances in the replay buffer (dynamic alpha as called in the authors’ language) would result in better accuracy. This is also evident from table 3. In my opinion, this is a very  strong assumption on how the system gets evaluated. This leads to the third point, why is this a good choice? I would argue that for some objects I no longer require to recognize very old variants (like laptops), but for the majority of real world concepts, this is not the case. 
* Extending above point, the precise nature of the distribution shift in the proposed dataset is not clear to me. For computers, I can follow the authors’ narrative that there has been a lot of changes over time that fundamentally change appearance etc. For e.g. Hockey, I am not sure that this is even on a remotely similar level (does Hockey even need continual learning to recognize the concept of a hockey player?). Intuitively, it seems like the major distribution shift here is primarily a consequence of sensor changes. 
* Unless I misunderstand something entirely, the statement “CLEAR also comes with abundant unlabeled data per time period” seems to be an attempt to make the contribution look bigger than it actually is. Unless I am mistaken, this data has not been acquired by the authors but simply randomly sampled from an already existing data set. As it is not further labelled, it also does not go through the authors’ proposed pipeline of CLIP + MTurk. The contribution then is basically just sorting + binning the data according to an already existing meta-data time label. This is hardly a contribution, but the authors should feel free to correct me if there is a severe misunderstanding. 
* The authors are using a very stochastic way to construct their replay buffer, yet table 2 shows single accuracy values. The authors should provide statistical deviations across experimental repetitions (also because training itself is stochastic anyway). 
* The used replay buffer is stated as “size of one bucket of training images”. That seems to be a lot of original data to store, so it is unsurprising that it would work well. It also potentially explains why even a naive strategy as “just uniformly sample a new element and replace an old one with a specific probability” is empirically good enough. The smaller this replay buffer size gets, the more care needs to be attributed to actually maintaining a set of exemplars that spans the observed data distribution. 
* At multiple points in the paper the argument is made that YFCC100M is too large to handle and thus a rather small subset is extracted. Although I understand that the final CLEAR dataset is smaller, due to the cost of CLIP + MTurk, YFCC100M is certainly not too large to handle in a modern time. As the authors’ themselves state, “We curated CLEAR with merely one day of engineering effort”. That is very far away from what is typically considered to be a reasonable effort for dataset construction. Even downloading a 10TB dataset is feasible, with costs of 2TB hard drives hardly superseding $100. To clarify, I am not asking the authors to necessarily extend their final CLEAR set, I believe it might already be good enough as an initial dataset, but I would advocate to change this narrative. That said, if the authors could provide an indication for how the performance of algorithms behaves with respect to how many data instances per class+bucket are considered, this would give a very good intuition for future works on whether and how the dataset should be extended. 
* The authors’ make a point about their procedure being valuable for future data extraction of different subsets or classes from YFCC100M. However, as the authors state themselves, this dataset seems to no longer be publicly accessible. When I checked the website, even the dataset browser is no longer accessible. Can the authors provide a solution here? Alternatively these statements should be removed.
correctness: * Already some initial statements in the introduction are factually incorrect, like “Dumb implies that CL may even be harmful because from-scratch classifiers tend to perform better by ignoring previous experience” (which they obviously do not if this previous experience is captured through explicitly stored data in a replay buffer). 
* The pre-training + unsupervised narrative, the way it is experimentally conducted and compared, seems misleading and unfair to me. E.g. the statement “pre-training step surpassing all popular state-of-the-art CL techniques that only make use of fully supervised data” is a direct consequence of an unfair evaluation.  I can imagine representation transfer being a very interesting aspect to look into, if done and analyzed correctly. The way I understand what the authors do, is that they pre-train a model on the same classes as provided in later evaluation, and then check how much this helps when evaluating these classes under some form of later distribution shift. (This seems less like general continual learning, and more like simple domain adaptation).  It is expected that pre-training will boost your result if the target task is actually the same throughout. But then the authors should also use semi-supervised continual learning techniques (all the generative modeling based approaches come to mind) to contrast. 
* Line 215+ states that “offline CL” is equivalent to “unlimited storage” in a sample-storage/size of replay buffer. I do not think this is true. “Offline” evaluation in continual learning just assumes that a “task” can be trained to convergence over the course of multiple epochs (revisiting data) before proceeding to the next “task” (where old task data is then no longer available, in whichever way the task is respectively defined). However, older tasks still get approximated/forgetting gets alleviated through a limited memory replay buffer. I do not see a fundamental difference here. 
clarity: * There is a major concern with the way the words “online” and “offline” are used in the paper. I acknowledge that the authors’ have put these words in quotations on purpose and even initially state that they deviate from the original definition. However,  reading further along the paper this just results in very confusing writing and it is not clear to me why this sort of re-definition needed to happen. In particular, since there is nothing obviously wrong with the initial statement on “online” being viewed as an ongoing stream of data that can only ever be seen once, resulting in immediate updates without revisits. 
* Extending above point, what the authors seem to conflate is the fact that most machine learning benchmarks are treated from a closed world perspective, i.e. the captured dataset describes all there is to see. I agree with the authors that the reality is an “open world”, that is, there can be new instances that look entirely different (the camera example is a good one here) and are unknown, and distributions generally drift (additional sensor changes etc.). Rather than making an attempt at re-defining the term online, the authors may want to take a look at open world recognition/learning. This way, the issue of notation clash can also possibly be avoided.

relation to prior work: As mentioned above in weaknesses, one of my biggest concerns is the way the paper is framed, also with respect to related work. 

* Apart from the fact that neither figure 2 nor table 1 are adequately described in the text, I believe the framing of the related work and the dataset in this context are very problematic. I find table 1 to be very misleading. For a start, it mixes datasets with methods on the y-axis. As an example, “Task-IL” references a methods paper on parameter regularization, dataset pairs one on replay with generative models. These are *algorithmic contributions* and have made *a choice* in their respective evaluation at the time. They are *neither* a dataset contribution, *nor* do they provide a critique or position on what the correct form of evaluation is. As such, their evaluation is a choice made at the time of writing the respective paper. Almost the same problem can be found on the rows of that table. The desiderata as articulated in the referenced work (A-E in the table) are treated as some form of checkmarks for CLEAR, Firehose, permuted MNIST… Desiderata (let’s leave aside whether they are actually desired or not for a moment) such as “shared output head” or “no test-time task labels” are a choice of the way an algorithm is constructed and evaluated, not a property of a dataset. Almost every “contrived” continual learning dataset evaluation on some splits of CIFAR, MNIST, ImageNet has seen methods explore settings where A-E are fulfilled or violated. Again, this is not a property of the dataset, but the way the author’s have dealt with it. The same goes for semi-supervised. I can treat any dataset as unsupervised by simply not using labels, even if they are technically annotated. I believe most of the placed checkmarks are either wrong or very misleading. In addition, neither the terms on the rows, nor the columns are explained properly in the text. 
*  A large amount of the citations are incomplete, inconsistent or wrong. To name a few: Shin et al was published at NeurIPS and not arXiv17, Lomonaco et al (reference 33), has 10 authors listed and then et al (either list all or put et al right away) and was not published in CVPR but CVPR-W, gradient episodic memory (reference 34) is not archival but has been published at NeurIPS 17… etc. I suggest the authors go through all of their references again. 
additional feedback:  * As a minor point that the authors can just improve upon easily, instead of writing “see supplementary for details”, there could just be a direct reference to section, figure, table etc. in the appendix. 
 * The authors should ideally host their dataset in a form that has a DOI and supports versioning for future changes and reproducibility. Zenodo could be one popular choice.
review_title: New data set variant/subset with interesting ideas that ultimately falls short due to presentation and writing issues

Meta-review: The paper proposes CLEAR - an image recognition benchmark for continual learning based on YFCC100M dataset. The reviewers agree that this benchmark departs significantly from existing datasets in this space in terms of task definition and scale. The scale is achieved by relying on recent high-performance vision-language models (CLIP) and crowd-source effort to ensure quality and deal with certain ethical aspects (e.g. remove inappropriate images). To have a more natural CL setting, the benchmark task is built around the temporal evolution of visual concepts (e.g. computer, dress, etc) over a decade. Since the proposed benchmark is a curated subset of YFCC100M, the authors recommend (and show the benefit of) using the remaining part of YFCC100M for semi-supervised and unsupervised learning, highlighting the shortcomings of current fully-supervised methods for CL.

Despite some initial disagreements around terminology and claimed contributions, the reviewers and the authors engaged in a very constructive review process, resulting in a significant improvement of the submission. After detailed discussion among the reviewers and the area chairs, we concluded that this paper makes a significant contribution to the CL field and has high potential to drive research in the field. There are still some concerns about some of the scores being close to saturation and clarity of describing the contributions, and we strongly encourage the authors to revisit these aspects for the camera ready version.
Decision: Accept