TiC-CLIP: Continual Training of CLIP Models
Authors: Saurabh Garg
Mehrdad Farajtabar
Hadi Pouransari
Raviteja Vemulapalli
Sachin Mehta
Oncel Tuzel
Vaishaal Shankar
Fartash Faghri
Venue: ICLR 2024 poster (2024)
Keywords: CLIP
Continual learning
distribution shift
benchmark
Primary area: datasets and benchmarks

Abstract: Keeping large foundation models up to date on latest data is inherently expensive. To avoid the prohibitive costs of constantly retraining, it is imperative to continually train these models. This problem is exacerbated by the lack of any large scale continual learning benchmarks or baselines. We introduce the first set of web-scale Time-Continual (TiC) benchmarks for training vision-language models: TiC-DataComp, TiC-YFCC, and TiC-Redcaps. TiC-DataComp, our largest dataset, contains over 12.7B timestamped image-text pairs spanning 9 years (2014-2022). We first use our benchmarks to curate various dynamic evaluations to measure temporal robustness of existing models. We show OpenAI's CLIP (trained on data up to 2020) loses $\approx 8\%$ zero-shot accuracy on our curated retrieval task from 2021-2022 compared with more recently trained models in OpenCLIP repository. We then study how to efficiently train models on time-continuous data. We demonstrate that a simple rehearsal-based approach that continues training from the last checkpoint  and replays old data reduces compute by $2.5\times$ when compared to the standard practice of retraining from scratch. Code is available at https://github.com/apple/ml-tic-clip.

Published as a conference paper at ICLR 2024
TIC-CLIP: CONTINUAL TRAINING OF CLIP MODELS
Saurabh Garg
Mehrdad Farajtabar'
Hadi Pouransari!
Raviteja Vemulapalli'
Sachin Mehta
Oncel Tuzel!
Vaishaal Shankar!
Fartash Faghri
Apple *Camneie Mellon University
sgarg2@andrew. cmu edd, fartash@apple.com
ABSTRACT
Keeping large foundation models up to date on latest data is inherently expensive. To avoid the prohibitive costs of constantly retraining, it is imperative to contin- ually train these models. This problem is exacerbated by the lack of any large scale continual learning benchmarks or baselines. We introduce the first set of web- scale Time-Come Continual (TiC) benchmarks for training vision- laiiuae models: TIC- DataComp. YFCC, and TiC-RedCaps. TiC-DataComp, our largest dataset, contains over 12.7B timestamped image-text pairs spanning 9 years (2014-2022). We first use our benchmarks to curate various dynamic evaluations to measure tem- poral robustness of existing models. We show OpenAl's CLJP (trained on data up to 2020) loses s 8% zero-shot accuracy on our curated retrieval task from 2021- 2022 compared with more recently trained models in OpenCLIP repository. We then study how to efficiently train models on time-continuous data. We demon- strate that simple rehearsal-baig based approach that continues training from the last checkpoint and replays old data reduces compute by 2.5x when compared to the standard practice of retraining from scratch
INTRODUCTION
Large multimodal foundation models (Bommasani et al. (021)) have offered unprecedented advance- ments in image-generation and zero-shot generalization, and have led to paradigm shift in multi- modal learning, e.g., CLIP (Radford et al. 2021), Flamingo (Alayrac et al. 2022). and Stable Dif- fusion (Rombach et al. 2022). These foundation models typically trained large web-scale datasets which are fixed and static in nature. For example, CLIP's training data contains 400 million image-text pairs, and Stable Diffusion was trained LAION-2B dataset (Schuhmann et al.) (022). In reality, however, these models must operate dynamic environment, where the world is in state of constant change. For instance, the internet continually evolves, with petabytes of new data being added daily (Wenzek et al. 2019 Wiener & Bronson 2014). It remains unclear how legacy models, e.g., OpenAl's CLIP models which were trained on internet-scale data up until 2020, work on future data and whether they even require any re-training to adapt to time-evolving data.
We begin by comparing robustness of OpenAl's CLIP models others in OpenCLIP repository that are trained on more recently curated web-datasets (c.g., LAION-5B. DataComp) containing data up until 2022 (Tharcoo et al., 2021). Since there is no existing benchmark to understand robustness to time- evolving vision-language data, we curate dynamic classification and retrieval tasks for years 2014- 2022 and evaluate different CLIP models (see Sec. 2.2 for our evaluation tasks). We make an intriguing observation that OpenAl models exhibit significant gap in retrieval performance data from 2021- 1- 2022 compared with 2014-2016 whereas OpenCLIP models retain their performance. In contrast, standard evaluations such accuracy ImageNet distribution shifts paint an incomplete picture that OpenAl's CLIP models are slightly more robust than OpenCLIP models (Fig. I.. Our findings not only demonstrate the critical need for models adapt and evolve alongside dynamic data distributions, but also underscores the limitations of relying solely on static benchmarks (e.g. ImageNet).
One naive but common practice for adapting to time-evolving data is train CLIP model from scratch every time we obtain a new pool of image-text data. This practice has its rationale:
Work done during an internship at Apple.
Code available at athttps:: http://tthhhbbccommaaa//l///////

---

Published as a conference paper at ICLR 2024
Imagennet dist. shift accuracy
Standard Evaluation Paradigm
Retrieaa recall on 2021 2222222
80
Our Proposed Evaluation Paradigm
Imagnett and its dist. shifts acturacy
50
TiC.Datacomp (L)
80
2
70
Performance
45
60
60
40
40
30
355
40
30
Train from scratch with new old data (standard practice)
20
Warn start new data rpplay data
25
60
80
Imageret accurasy
Retrieval recall on 2014-2016
20
0.0
0.2
0.4
1.0 Total Compute (MACs)
0.6
0.8
1.2
1.4
OpenAl models trained on data before 2020
OpenCiip models trained on data before 2022
Figure 1: (Left, Middle) OpenAl models show less zero-shot robustness on retrieval task from 2021-2022. OpenCLIP models and OpenAl models have similar robustness on standard benchmarks. However, OpenAl models show less robustness on our retrieval task when compared with recent models in OpenCLIP repository, highlighting susceptibility to time-evolving data distribution (Right) Simple continual training baseline is computationally efficient and competitive to retraining from scratch. Different points denote models trained sequentially on our TiC-DataComp (L) as data arrives over time. Warm start training with previous checkpoint and replaying all old data, performs similar to Oracle which trains from scratch every time new data arrives. by using 2.7x less compute.
initiating training from pre-existing model can make it difficult to change the model's behavior in light of new data (Ash & Adams, 2020 Achille et al. 2018: Liu et al. 2023). However, training foundation models from scratch demands significant computational resources and is often infeasible to repeat frequently. For example, ViT-g- 14 in Schuhmann et al. (2022): Cherti et al. (2022) was trained for 240K A 100 GPU hours which is approximately one month 400 GPUs The prevailing
training guidelines centered around scaling laws for CLIP training have only looked at training from scratch (Cherti et al. 2023). This leads to a pivotal question: How can we continuously update models as the data distribution evolves over time given computational constraints?
There exists a vast literature on continual learning, with focus on adapting models to dynamic environments (Parisi et al., 2019 Hadsell et al. 2020 De Lange et al., 2021). Traditionally, this field concentrated on synthetic incremental benchmarks that lack natural evolution between tasks, and hence, continual learning methods are seldom used in real-world scenarios (Cossu et al.) 2022] Lin et al. 202T). In contrast, recent works focusing on continual learning methods for CLIP models. primarily target improving performance on single or sequence of disjoint downstream tasks (Ding jet af., 2022] Zhou et al.) 2023b Zheng et al. 2023 [Tharco et al., 2022). While some recent works have started address these problems, existing benchmarks are comparatively much smaller in scale, or lack paired image-text data (Ni ct al. 2023 Lin et al. (20TT). Simply put, there is a scarcity of work focusing on continual training of CLIP models on naturally evolving data with time at web-scale.
We take the first step towards Time-Continual (TIC) training of CLIP models where data distribution evolves naturally over time (overview in Fig. 2). We introduce TiC-DataComp, new benchmark for Time-CCntinual training of CLIP models, which we create by appending "crawl time" information to existing CommonPool dataset (Gadre et al., 2023). We also repurpose other web-scale datasets gathered from diverse sources, such as Reddit and Flickr. Specifically, we curate TIC-YFCC and TIC- RedCaps by leveraging time information available in YFCC (Thomee et al. 2016) and Redcaps (Desai ct al. (02T) respectively. The primary objective of our study on this benchmark is to develop continual learning methods that operate within constrained computational budget (say C)) each time fresh batch of data becomes available. These methods compete with an Oracle, which starts training from scratch every time new data arrives, utilizing cumulative computational budget.
To assess models trained in our TIC-CLIP framework, we evaluate models on our proposed dynamic evaluation tasks that evolve with time along with 28 standard classification and retrieval tasks including ImageNet (Krizhevsky al. 2012), ImageNet distributions shifts, and Flickr (Plummer ct al. 2015), in a shro-shot manner following the work Garre et al. (2023); Radford et al. (2021).
Finally, we develop continual learning methods our benchmarks and perform over two hundred experiments with different baselines that utilize previous checkpoints (e.g., warm start, patching. and distillation), replay buffers, and learning rate schedules. Our findings highlight a key takeaway: Cumulative method that warm starts training with the latest checkpoint and replays all old data, achieves performance competitive Oracle while being 2.7x computationally more efficient.
2

---

Published as a conference paper at ICLR 2024
A. Aggregate
Continual Training
Static Dynamic Evailaaion
OR
Ciaselication
Retrieval
data
Random
Hoold ef time 1-1
Imagehes 1N.. ChypesNett (Ned, 18-5, IN-O VIAB.
Hickrtti, VIDOCCO
Data time
Standard Stalic Evalusiion Tackk
Took at
Tast at Time 1
Taskat Timm
Toak Time
Repley buffer constrants
smo
time
Train model compute budgel constraints
Boolward Transter
Exedicaioo
Forward Transter
Data at time
Proposed Dynamic Evaluation Tasks
Figure 2: Experimental protocol our proposed continual benchmarks. (A) Combine new and old data given buffer constraints. (B) Continually train a model with a compute budget (say C) either by starting with previous checkpoint from scratch. (C) Evaluate models standard datasets and our proposed dynamic datasets. Comparison with other benchmarks in Appendix
Additionally, our experiments demonstrate interesting trade-offss between buffer sizes for static and dynamic performance and provide valuable insights into learning rate schedules for sequential training. Our results span various dataset scales (from 1 11M samples to 3B) and highlight trends with different methods that are largely consistent across scales.
To make our benchmarks accessible, publicly release the code and the time information collect on top of existing datasets here Our work is just an initial step towards continual training of foundation models, and believe our research would spur more attention to this understudied area.
TIC-CLIP: BENCHMARKS AND EXPERIMENTAL PROTOCOL
In this section, introduce benchmark (Fig. focusing the training of vision-language foundation model with the Contrastive Language Image Pretraining (CLIP) (Radford et al.) (202)) objective. Notably, we train image-text data that arrives sequentially unlike the conventional image- text datasets which are static (e.g. WiT in CLIP, DataComp in Gadre et al. (203)). We curate TIC- DataComp, TIC- and TiC-RedCaps that image-text pairs sourced from the internet which we augment with auxiliary time information. We also introduce dynamic evaluation tasks to assess performance of our continually trained models on data evolving with time. The goal of a learner is to train a deployable model at each step as new data becomes available with fixed compute budget.
2.1 BENCHMARK DESIGN: HOW WE CREATE TIME- CON-ICONTINUAL DATASETS?
To instantiate continual training of CLIP, we extend existing image-text datasets with time information collected from the original source of the datasets. Our largest dataset is TIC-DataComp which contains 12.7 billion image-text pairs with "crawl-time
metadata. We create this dataset on top of the existing DataComp benchmark (Gadre et al. 2023). We also create TIC-YFCC and TeC-RedCaps on top of existing YFCC15M Thhoee et al. 2016 Radford et al., 2021) and Redcaps (Desai et al.) 2021) datasets to highlight that our findings are broadly applicable to carefully curated datasets from diverse sources such as Reddit and Flickr. While time-related metadata is absent in the DataComp benchmark, it is available in the original releases of YFCC and Redcaps. Nevertheless, to the best of our knowledge, no prior work utilizes such time information for continual training of CLIP models. We show dataset statistics for all datasets, e.g., number of examples in each year in App. C
TiC- CiC-DatCommp We collect timestamps for the CommonPool dataset introduced in DataComp which contains 12.7B image-text pairs (not including O.IB inaccessible ones). This dataset stands as the largest public image-text dataset to date. The source of DataComp Common Crawl, which periodically releases web-crawled data snapshots, typically and updated webpages. To construct TIC-DataComp. we augment each image-text pair in DataComp with their first timestamp. We followed the same construction process DataComp but retained only the image-text pair found in the earliest snapshot during the deduplication stage. This process provides timestamps at the granularity of months, spanning years 2014-2022. See App. C.7 for details on the construction process. We note that while this augmented time information may contain some noise, on average, find it to be reasonably accurate proxy for the upload time of web pages (see App.
monthly basis since 2014 with new
3

---

Published as a conference paper at ICLR 2024
A. Dynamic Retrieval Task
B. Dynamic Classification Task
Mask
Sports Car
Phone
Computer
Mount Rainier and Towers
2014
Previously unseen topics, B. COVID-19 emerge with time
image of wirus that causes arcknese coranesinus covid199
First Snow Storm the Scason
2022
Figure 3: Distribution of examples changes from 2014 2022 in our dynamic evaluation tasks. (Left) Samples for text to image retrieval. For new timestamps, images from novel concepts appear (e.g., COVID-19). (Right) Samples from our classification task for 4 categories. We observe that not only objects evolve over time but also images from recent timestamps are captured more in the wild.
Although our benchmark contains time information the granularity of months, we limit our experiments to granularity of years by consolidating data for all months in year. Similar to DataComp, our benchmark has an inclusive design. accommodating participants with varying levels of computational resources. In particular, we experiment with medium large, and xlarge sizes from CommonPool. Gadre et al. (2023) leverage different filtering strategies to select the training subset. We are concerned that filtering techniques bias the selected training data. In App. provide preliminary evidence that "Bestpool" filtering that uses off-the-shelf CLIP models, indeed biases the selected data to old time steps. Nevertheless, to highlight significance of our findings even for state-of- the filtering techniques, we experiment with both Bestpool and Basic filtering (no CLIP filtering) xlarge scale. For large and medi scales, we only experiment with Basic filtering.
we
We experiment with the 15M subset of YFCC100M (Thomee et al. 2016), namely YFCC15M, selected by OpenAI (Radford et al., (021). This filtering retains only images with natural text in captions. YFCC100M contains data from years 2008-2014 and was originally released with upload timestamps. We use this information create continual splits at the granularity of years.
TEC-
TiC- ReC-Res
RedCaps contains 12M image-caption pairs from manually curated set of subreddits across 2011-2020 (Desai al. 2021). We use the creation timestamps of the posts create splits for continual learning. Similar to the other two datasets, we experiment at the granularity of years.
2.2 EVALUATION TESTBED
Dynamic tasks We leverage the temporal information in our benchmarks to create dynamic evaluation tasks. Here, the test data comprises samples varying over years as the world evolved. For our largest dataset which is TiC-DataComp, we create dynamic tasks for both retrieval and classification described below. (examples in Figure 3 and additional examples in App.
I. Dynamic retrieval task: To create a retrieval task, we sample batch of IID image-text pairs from different timestamps and evaluate text retrieval performance given the corresponding image (similarly, image retrieval given the corresponding text). We refer to the dataset as HiC-DataComp-Retrieee
II. Dynamic classification task: We also create a classification dataset TIC-DataComp-Net with Ima- geNet classes from CommonPool and augmented with timestamps. Inspired by LAIONNet (Shirali Hardi 2023), we first filter examples where the corresponding caption contains one and only one of the synsets of ImageNet. Then we only retain examples where the similarity between ImageNet synset definition and the caption exceeds threshold of 0.5. We evaluate the similarity using an off- the-shelf sentence embedding model (Reimers & Gurevych 2019). Crucially, unlike LAIONNet, do not filter the image-text pairs with CLIP similarity scores to avoid biasing the selection process. We describe the construction process in more details in App. C.5 On TiC-DataComp-N
report average accuracy over all classes and over selected nodes (e.g., motor vehicles) at each time step.
Similarly, create retrieval tasks for TIC-YFCC and TiC-RedCapa Note that remove the extracted image-text pairs for dynamic retrieval and classification tasks from the training sets. Evaluations on dynamic tasks done in shot manner.
4

---

Published as a conference paper at ICLR 2024
Static tasks We also evaluate models numerous classification and retrieval tasks in zero- shot manner as in Radford et al. (202I). In particular, we consider 28 standard tasks: 27 image classification tasks, e.g., ImageNet and its 6 distribution shifts (e.g., ImageNetv2, ImageNet-R, ImageNet-Sketch, and Objectnet), datasets from VTAB and Flickr30k retrieval task. We refer to these as static evaluation tasks. We list all the datasets in App. C.2
Evaluation metrics We define metrics for classification tasks and retrieval tasks based on accuracy and Recall@ respectively. Let I represent the number of time steps for which we have data. For each training method, we generate total of T models, each corresponding to the end of training particular time step. For static datasets (e.g., ImageNet), we report average performance of T models. However, when dealing with dynamic evaluation datasets, we assess the performance of each of the T models on evaluation datasets collected at all time steps. Consequently, for each model and dynamic evaluation task, we obtain T performance values. We represent these values using the performance matrix E, where each entry & signifies the performance of the model obtained after observing training data at time step when evaluated on a dataset from time step j. The performance matrix & can also be succinctly summarized using three standard metrics commonly employed in continual learning evaluations (Lin et al. 2021 Diaz Rodrinodr et al. 2018):
In-domain performance: average performance at each training time step (i.e., the diagonal of E)
Backward transfer: average on time steps before each training step (i.e., the lower triangular of E)
Forward transfer: average on time steps following each training step (i.e., the upper triangular of E)
Sometimes, the metrics described above can cause the backward transfer metric be influenced by later evaluation time steps, biasing the backward transfer metric (refer to App. F for details). Therefore, in App. we present results using revised metrics that mitigate this issue.
While the static tasks capture performance on standard benchmarks, dynamic tasks capture problems due to distribution shift (for forward transfer) and forgetting (for backward transfer). The goal in our benchmark is to develop continual learning methods that maximize performance on static tasks while simultaneously optimizing for performance on dynamic tasks.
2.3 EXPERIMENTAL PROTOCOL FOR TRAINING
Streaming protocol We follow a streaming protocol, where data is progressively revealed to the learner in large batches with the objective of achieving a deployable model as early as possible after each batch arrives. We conduct experiments with data streaming at the granularity of years and our benchmark supports future research at the granularity of months. Additionally, as the amount of data from earlier time steps is limited (see App. CK we aggregate data from the earlier time steps into a single larger batch and timestamp it by the latest year in the range. After this aggregation, we have 7 time steps for TiC- -aCCaatCompp (2016-202) 2022) and 4 for both TIC-YFCC (201 1-2014) and TIC- RedCaps (2017-2020). While the number of image-text pairs revealed at each time step are of similar orders of magnitude, the exact number does vary across steps and we do not artificially alter the sizes.
Memory budget We allow methods to use the last model checkpoint at each step as the cost of keeping one checkpoint per month often negligible. In contrast, the cost of retaining old data can be high and might not be permitted due to data expiration policies. Thus, along with studying methods that retain all old data, we also explore strategies that restrict data persistence (see Sec. for details).
Compute budget To ensure a fair comparison between methods, we establish a consistent total compute budget, quantified in terms of Multiply-cccumulatc Operations (MACs), and allocate evenly for training at every time step. Unless specified otherwise. for all methods except Oracle and LwF, we use the same compute budget. For experiments on TiC-DataComp. we refer to compute configurations in DATACOMP for overall compute. For TiC-RedCaps and TIC-YFCC, we use compute of order medi um scale in TIC-DataComp. Compute budget details are in App. C.4
2.4 ANALYZING DISTRIBUTION SHIFTS IN THE CONSTRUCTED BENCHMARKS
TIC- ICC-DataComp analysis through the lens of constructed evaluation tasks First, we qualitatively analyze the examples in our retrieval and classification dataset (Fig. 3). We observe that over time, in the retrieval task, new concepts like COVID-19 emerge. Likewise, certain ImageNet classes evolve, such the shift from "masquerad" masks "surgical/prooecttve" masks in their definitions. Moreover, as time evolves, we observe that image quality improves and more images tend to appear in the wild in contrast to centered white background images. Next, we compare performance of OpenAl
5

---

Published as a conference paper at ICLR 2024
and OpenCLIP models our datasets. Here, only present the main findings, and delegate detailed discussion to App. [.6] We observe a significant performance gap between OpenAl and OpenCLIP models our dynamic retrieval task (Fig. D. This gap widens notably on retrieval queries where captions mention COVID-19. On the other hand, OpenAl and OpenCI.IP models exhibit similar robustness for retrieval data coming from Flickr highlighting that data from some domains do not exhibit shifts that cause performance drops. For our classification task, we observe a very small drop (* 1%) when averaged across all categories. However, we observe substantial gap on specific subtrees in ImageNet. For example, classes in "motor vehicle" subtree show an approximate 4% performance drop, when comparing OpenAI and OpenCLIP models. These findings highlight that while overall ImageNet classes may remain timeless, certain categories tend to evolve faster than others. Our qualitative and quantitative analysis on TiC-DataComp clearly highlights evolution of distributions and captures different properties than standard benchmarks.
Quantitative analysis on TIC-YFCC We analyze TIC-YFCC using off-the-shelf sentence and image encoders. We first embed images from different time steps with an OpenAI CLIP encoder and then compute Frechet Inception Distance (FID; Seitzer (2220). As time progresses, we observe that FID distance increases with respect to data from first time step (Fig. [8 in App. C.6p. Similarly, we use pretrained sentence transformer to extract top-5 categories from Wordnet Nouns for each caption. We observe that the TV distance over distribution of WordNet Nouns evolves over time when compared to data from the first time step. More details in App. C.6
3 TIC-CLIP: HOW TO CONTINUALLY TRAIN CLIP MODELS?
In this section, we lay out different methods specifically focus the following questions (Tab. []] (i) How to utilize/replay data from pre- vious time steps: (ii) How to leverage previously trained model checkpoints? (iii) What should be the training/optimizztionn procedure?
Table Table summarizing our methods. D; data size in each step, I' total time steps, t: current time step, C: compute budget (iterations).
Method
Each Step
Total
Train Size
Init.
Compute
Compute
Cumulative-All
tD
Last
TC TC' TC TC TC'
Cumulative Exp
2D
Last
C
Data replay methods initialized from the last checkpoint demonstrate strong performance on standard continual learning benchmarks (Sec. 5). We consider replay methods with/without initial- ization from last checkpoint(s):
Cummlative-Equal
2D
Last
Sequential
Last
Restart
tD
Rand
C
Patching
Last Putch
TC
LsF
Last
122 C
1.2 x TC
Oracle**
Rand
10
I. Oracle: Train a CLIP model from scratch (i.e., random initialization) on all image-text data received till time t using large compute budget of t C. Oracle represents a prohibitively expensive method that is the most common practice in training large-scale foundation models. The goal of other methods is to perform as close as possible the Oracle within their limited budget.
II Cumulative: Train each model initialized from last checkpoint on the union of all data up to with compute budget C. This method is analogous experience replay (Robins) 1995| Hayes et al. 2019) but with substantially larger buffers than common in the continual learning literature. Given a fixed buffer size for each past step, we observe minimal to no difference between random subsampling and other strategies. After sampling the replay data, we randomly shuffle it together with new data for training. We consider the following strategies for sampling buffer sizes per step:
-All: Replay all previous data.
-Exp: Replay buffer of size D and reduce the amount of old data by half at each step. For example, 3-rd time step, we retain D/2, D/2 of old data and at 4-th, we retain D/4, D/4, D/2 of old data. Along with D data from current step, this method trains on at most 2D data in each step.
-Equal: Replay& buffer of size D but split the buffer equally among all previous years. For example, at 4-th step, we retain D/3, D/3, D/3 of old data. Along with data from current time step, this method trains at most 2D data in each step.
III. Sequential: Train only on the new data starting from the best checkpoint of the previous time step. Sequential is similar to Cumulative but without any replay buffer.
Train each model from scratch (i.e., random initialization) on all the data till time t for compute budget C. Restart is similar to the Oracle but with compute budget C' at each time step and similar to Sequential but with random initialization. As such, Restart helps us understand the forward transfer and loss of plasticity in our benchmark (Ash & Adams, 2020 Dohare et al. 2023
IV. Restart:

---

Published as a conference paper at ICLR 2024
Table 2: Zero shot performance on our time-continual benchmarks. and ** denote methods that violate the compute budget. For static tasks, we tabulate accuracy of the models obtained on the final timestamp. For dynamic tasks, we tabulate forward/backward transfer and performance on retrieval tasks (Sec. 2.3 For TiC-DataComp (XL), we include results with Bestpool filtering (basic For all metrics, higher better.
filtering Table 5)
Benehmark
Method
Compute (MACs)
ImageNet
ImageNett dist, shift
Statie Tasks
Flickr20k
Average over 28 datasess
Backward Trinsle
Dynamic Retrieval Tasks
ID Perfor-
Forward Transfer
Resturt
10
3.6
3.0
13.2
41.4
18.6
Sequentil
10.5
Patching
3.4 10
18.9
11.3
18.5
23.3
44.7
53.4
24.5
TIC-YPCC
Comulative-Fxp
3.4 18ix
24.1
14.3
20.4
25.9
60.4
60.1
27.1
10ah
Cunalative-All
3.4 x 1826
29.3
17.6
26.5
29.0
66.4
60.2
27.5 6
Lwn
4.1 161h
16.9
9.8
14.7
21.2
36.8
56.0
24.2
CumolaziveAAl
10
17.5
6
Oracle**
1026
29.2
17.0
9
06.1
9
Restant
3.4 1628
11.7
8.5
3.7
18.1
21.3
25.4
22.
Sequemial
3.4 103h
19.3
13.7
25.8
33.0
33.6
27.5
Patching
15.2
27.8
TiC-RedCaps
Conulative-Epp
3.4 1676
27.3
19.1
10.5
20.0
44.3
42.0
92.6
Cumularive-qquu
3.4 18
27.8
19.4
10.0
30.5
44.4
42.0
32.6
Cumulative-Ae
10th
LwF
4.1 1026
21.8
14.8
27.3
35.4
36.0
28.4
Curslazive-AI
3.6 1826
32.9
23.7
14.1
32
49.0
43.4
33. 4
Oracle**
10
Sequential
19.2
16.4
26.0
25.7
26.4
14.9
Patching
3.0 10tx
19.3
16.8
18.5
26.4
26.9
25.4
14.5
TIC-DataCComp (M)
Cumulative-aag Exp
3.0 18a8
22.1
18.4
20.4
28.8
31.7
27.1
155 2
Cumalatie-All
3.0 18
24.0
20.2
20.9
90.0
333.8
26.4
15.1
LwF
3.8 16
19.2
16.3
17.7
27.0
25.6
26.6
14.9
CumsluiiveAA1
10ts
Oracle**
1.2 1020
25.5
21.2
23.3
30.8
34.9
27.8
15.0
Sequental
2.7 1829
44.7
37.4
48.4
45.7
62.6
58
41.1
Patching
10
38.9
57.5
TIC-DataCCmp
Cumulative-Fxp
30.6
Cunulativ--ang
2.7 1029
47.7
10.3
51.8
17.7
60.9
58.2
41.4
Cummlative-All
2.7 10
48.9
41.3
62.1
57.3
Cumolaaive
Oracle**
1.1 x 1020
53.6
44.0
53.9
50.4
64.3
58.6
41.8
Sequential
2.7 1029
66.5
54.2
61.2
61.0
63.1
68.9
16.88
TIC: TIC-DatuComp (XL)
CumulativeAAl
10
58.8
CumnlativeAAll
3.5 1020
72.8
60.4
68.5
68.7
71.0
68.8
57.1
Oracle"
1.1 102t
73.3
61.3
68.0
65.8
We use sequential patching from Ilharco et al. (2022). Initialize from a patched model of last step and train only the data. To obtain patched model at each time step, apply weight interpolation with the patched model (if any) trained at time step t and the model trained at time step f. We tune the mixing coefficients by optimizing average retrieval performance previous tasks.
Patching:
VI. LwF: Train only on the new data with a KL divergence penalty between the image-text similarity matrix of last checkpoint and current model each batch (Li & Hoiem [007] Ding al. (022). See App. E for results with other continual learning methods, e.g., EWC Kirkpatrick et al. 2017).
Learning rate schedule The defacto Learning Rate (LR) schedule for training CLIP models is an initial linear increase to a maximum value, i.e., warm up. followed by a cosine decay (Radford et al. 202T Gadre et al. 2023). We default using cosine LR schedule for each sequential run, resulting in cyclic schedule and observe a significant increase in training loss early in subsequent runs when the LR high. However, as training progresses, we observe that the increased loss decreases at faster rate (when compared to training from scratch) allowing to train with cyclic schedules. We discuss this more and explore an alternate learning rate schedule in App.1e. B.S
Other Training details and hy hyperparameters Unless specified otherwise, we closely follow the original CLIP training recipe (Radford et al. 2021). We train the CLIP variant with ViT-B/16 as the image encoder (Dosovitskiy et al. 2020). All training and hyperparameters can be found in App. D.2
EXPERIMENTS AND MAIN RESULTS
Our main results in Table and more detailed plots each dataset in App. B.1 Recall, goal is compete with an Oracle that re-trains from scratch every time new data observed, both on dynamic and static tasks, while being computationally efficient. Here, summarize our key findings:
7

---

Published as a conference paper at ICLR 2024
Imagenet (& dist. shilts) accuracy
Static Evaluation
Retrieaa Recal001 on 2021 2021-2022
Dynamic Evaluation
Dynamic evaluation Oracle
Similar
Gap
2016
97:45
75/00
19:00
27:0
29:07
70
2017
12.35
54.22
A7:50
41.37
35
35.15
25.55
40
00
2018
03:07
50.23
65
51.1
45.25
29
50
2019
64:22
52.03
58
57.48
85
50.27
30
-0
2000
61
6162
64
64-04
$8.07
44
2021
65:
66
05
20
30
1.0 Total Compute (MACs)
Total Compute (MACs) -12
0.5
2022
03:90
15:33
65
17.23
64355
85:42
30
2010
2017
2018
2019
20.20
2021
2022
Culllativ-All
Oracle
Evclustion time stop
Figure 4: (Left) Dynamic and static evaluations rank models differently. Models with similar per- formance on static datasets, have > 6% difference on retrieval task from 2021 1-222 TiC-DataComp (L). Different points denote models trained sequentially over time. (Right) Performance of Oracle on future time steps drops highlighting distribution shift in dataset. Each row evaluates the Ora- cle trained on TiC-DataComp (I.) at a particular time step across all dynamic retrieval tasks.
Cumulative-All saves up 4x the cost. On dynamic evaluation tasks, we observe that Cumulative- All where we replay all the past data, achieves performance close to the Oracle (within 1%) using significantly less compute (4x less on T7C-DataComp and 2.5x less on TIC-YFCC and TIC- RedCaps). On static tasks, the gap remains small at small scales but grows to 4.7% on xlarge Bestpool, and 4% on xlarge Basic (see Table 2 and Table 5). In these cases, training Cumulative models with slightly extra compute bridges the gap while remaining at least 2.7x more computationally efficient (see rows with * in Table 2). This highlights that with unconstrained access simply train sequentially and significant computational resources.
large, 1.8%
to past data,
At scale, Sequential has strong forward transfer but lacks on static tasks. On YFCC and TIC- RedCaps, which are at the smallest scale, we observe a significant gap (> 10%) between Sequential (with no data replay) and Oracle on all tasks. On the other hand, on all scales in TiC-DataComp. Sequential shows strong performance on forward transfer and ID dynamic evaluations. However, on static tasks and backward transfer evaluations, Sequential significantly underperforms the Oracle.
Patching and LwF improve over Sequential but lag behind Cumulative-All. On static tasks, LwF improves over Sequential by 2%, while on dynamic tasks, LwF improves backward transfer by 7% on TiC-DataComp (M). However, its computation cost is higher than even Cumulative-All outperforms LwF on all tasks. Patching improves over Sequential on backward transfer on all datasets (e.g.. 5% boost on TiC-DataComp I.) highlighting that Patching combines benefits of previously patched model and the new Sequential model without additional computation cost. However, such benefits do not show up on static tasks. These results hint that to continuously improve on static tasks with time, replaying old data as in Cumulative-All plays crucial role.
which
-Exp and -Equal significantly reduce replay buffer size and maintain static task performance and backward transfer. Recall, that EExp and Equual reduce the replay buffer size to a maximum 2D of old data. In particular, at the last time step, -Exp and Equal reduce the buffer size by 3.5x x for TIC- DataComp datasets. While reducing the buffer sizes, these methods still achieve performance close to Cumulative-AN (within 2%) on both static and dynamic tasks, with -Equal consistently better than -Exp strategy. As we go to large scale, e.g., from medi um to large, the gap between these methods and Cumulative-All reduces. These findings demonstrate that even a small amount of replay data from old time steps stays competitive with replaying all data and significantly improves over no replay at all.
Warm up helps training on data from first time step, but hurts on subsequent time steps. Cosine LR is commonly coupled with an initial warm-up that linearly increases the LR from zero to maximum LR. We investigate the effectiveness of warm-up in first versus subsequent time steps. Surprisingly, we observe that not using warmup for subsequent training runs is strictly more beneficial than using warm up on both static and dynamic tasks. In particular, on TTC-DataCome (L), we observe about 1.5% improvement in ImageNet accuracy and 4.3% improvement on ID dynamic retrieval when not using warmup with Cumulative (see App. B.3). Moreover, we also ablate over not using warm up for the first training run and observe a drop of approximately 4.8% accuracy in the first time step on TIC- DataComp (L). Hence, we default using warmup when training on the first time step and not using it on the subsequent time steps with all methods except for training on TIC-DataComp (XL) where we add a smaller warm up (10% of the warm up iterations used in first step) to stabilize training.
8

---

Published as a conference paper at ICLR 2024
Same maximum LR works best across all runs when using cosine schedule. We ablate on TIC- DataComp (M) to investigate how to change LR after training on data from the first time step. Unlike conventional pretraining and finetuning settings where LR is typically decreased for subsequent training, we observe that decaying maximum LR for subsequent steps in our setup hurts on static and dynamic tasks and consequently, we use same maximum L.R across runs (see App. BB3).
Filtering strategy changes the ordering of performance on static and dynamic retrieval tasks. We observe that while bestpool filtering models outperform basic filterining models on TimmDataComp (XL) by 6% on static tasks, they underperform by over 5% on dynamic retrieval task (see Fig.
Dynamic tasks provide complimentary information for model selection compared to static tasks. Choosing models solely based on static task performance may inadvertently select models that underperform on dynamic tasks. For example, Cumulative models that show relatively modest improvements on static tasks continue to improve by > 6% for retrieval on 2021-2022 (Fig. 4
Cumulative-All remains competitive Oracle even on ImageNet on up to 8 splits. CLIP models are often trained for fewer epochs and are typically not trained until they reach an "overfitting" regime. Here, we investigate how Cumulative-All performs when compared to Oracle when training is done for longer. Specifically, we assess Cumulative-Al on 2, 4 and IID splits including the full dataset (see App. D.I for details). Table summmarizes our key findings. Notably, even with up 8 splits, the difference in accuracy between Oracle and Cumulative- All remains below 0.9%. These results underscore the feasibility of continual training with Cumulative-All even on ImageNet.
Table 3: ImageNet continual training. Cumulative-All remains close to Oracle.
Number of splits
Method
(Oracle)
Cumulative-All
80.9
80.8
80.6 80.0
RELATED WORK
Benchmarks for continual learning Traditionally, the continual learning community has focused on domain, class, and task incremental benchmarks (Hsu et al., (2018] Van de Ven & Tolia 2019 Zhou et al. 2023a) with artificial task boundaries (e.g.. Split-CIFAR Perm-MNIST), These benchmarks are often task-specific and present minimal or no meaningful evolution between adjacent tasks. Consequently, continual learning methods are often confined to these benchmarks and seldom scale to practical real-world scenarios (Cossu et al.) [2022) Lin et al. (02T On the other hand, continual learning methods for CLIP models are primarily aimed at timeettning to improve performance on a single or sequence of disjoint downstream tasks (Thengame al. 2022 et al.) [2233 [[barco et al. 2022). Existing large-scale benchmarks for training CLIP models, e.g., Datacomp (Gadre et al. 2023) and LAION-5B (Schuhmann al. 2022), are curated investigate methods and scaling laws to train state-of-the-ar CLIP models in a single training run. In our work, we augment these existing datasets with temporal information create benchmarks for continual pertaining of CLIP models.
Continual learning methods Common methods can be categorized into three categories: i) regularization, ii) replay, and iii) architecture-based methods. Regularization methods add penalty to keep the fine-tuned model close to its initialization and often incur additional memory/compute costs (Kirkpatrick et al. 2017) Mirzadeh et al.) 20200a b) Farajtabar et al.) 2022)). Data replay methods retain all or a subset of the prior data for subsequent training (Lopez-Paz & Ranzato 2017 Rebuffi et al. 2017, Chaudhryy et al. 2018). Simple replay-based baselines surpass various methods on standard benchmarks (Lomonaco et al. [222 Balaji et al. 2020) Prabhu et al. 2020). Lastly, architecture-bast based methods expand the model as new tasks arrive, limiting their applicability in evolving environments without clear task boundaries (Schwarz et al. 2018 Rusu et al. (016). In this work, we compare popular continual learning methods with simple alternatives for continually pretraining of CLIP.
CONCLUSION AND FUTURE WORK
We view TTC-DataComp as the initial stride toward the continual training of large-scale vision- language foundation models. We believe that our benchmark, alongside the preliminary results obtained using simple baselines will foster future research for large-scale continual-eeariing. There are several pivotal directions for future work: (i) Compare our baselines continually streaming data at finer granularity, e.g., streaming data at the monthly level; (ii) Investigate alternate learning rate schedules (c.g., Const-Cosine in App. B.5) that forward looking, and are better suited to continual learning; (iii) Better data filtering techniques that are more inclusive of future data; (iv) Expand our problem setup encompass the training of other large-scale foundation models.

---

Published as a conference paper at ICLR 2024
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations. 2018.
Jean-Baye Saan-Bapiiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances neural information processing systems, 33:3884-3333 3894, 2020.
Yogesh Balaji, Mehrdad Farajtabar, Dong Yin, Alex Mott, and Ang Li. The effectiveness of memory replay in large scale continual learning. arXiv preprint xrxiv:00100024/6 2020.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, al. From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging, 2018. //pube:/ ncbi.. nlm. mih. /0v/3071600000
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits object recognition models. H. Wallach, H. Larochelle, Beygelzimer, d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurIPS). volume 32. Curran Associates, Inc., 2019, https:://proceete ggs neurips cc/paper/ weeelllaaafff22fe http:///aaaaaaaaaaaaaaaaaaaa1..0
Paper pdd
Sara Beery, Elijah Cole, and Arvi Gjoka. The iwildcam 2020 competition dataset, 2020, https www.xx..oraaa//////x
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.00 072507.8 2021.
Jorg Bornschein, Alexandre Galashov, Ross Hemsley, Amal Rannen-Triki, Yutian Chen, Arslan Chaudhry, Xu Owen He, Arthur Douillard, Massimo Caccia, Qixuang Feng, et al. Nevis' 22: A stream of 100 tasks sampled from 30 years of computer vision research. arXiv preprint 1rXiv:2211.11744 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative com- ponents with random forests. In European Conference on Computer Vision (ECCV), 2014.
https://linke www.inn...ring....... com/ wttpttt:e....11////. 3
29
Zhipeng Cai, Ozan Sener, and Vladlen Koltun, Online continual learning with natural distribution shifts: An empirical study with visual data. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 8281-8290, 2021.
Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barhara Caputo. Modeling the background for incremental learning semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9233-9242, 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv preprint arXiv:1812.004200 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint 19Xiv:1902. 10486, 2019.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the Instinute of Electrical and Electronics Engineers (IEEE), 2017. https: : /tts:::/eeeeeeeerreeeeee leee.
10

---

Published as a conference paper at ICLR 2024
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman -mmge learning, 2022. attps://axxxv..oc//aa///////
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVV Conference on Computer Vision and Puttern Recognition, pp. 2818-2829, 2023.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018. ttttps://arat
org/abs/ 1711.//11....... 07846
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In International Conference on Arlificial Intelligence and Statistics (AISTATS), 2011.
wgs..//eednnn.mmmmmm.........m mml...mlrr. press peress/vl5// vl5 wwwteslla.aaaa............
Andrea Cossu, Gabriele Graffieti, Lorenzo Pellegrini, Davide Maltoni, Davide Bacciu, Antonio Carta, and Vincenzo Lomonaco. Is class-incremental enough for continual learning? Frontiers in Artificial Intelligence, 5:829842, 2022.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Blass-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 9268-9277, 2019.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jía, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-3355, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference Computer Vision and Pattern Recognition (CVPR), 2009.
Leee.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.114433 2021.
Natalia Didz-Rodríguez, Vincenzo Lomonaco, David Filliat, and Davide Maltoni. Don't forget, there is more than forgetting: new metrics for continual learning, arXiv preprint arXiv:1810.13166 2018.
Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding. Don't stop learning: Towards continual learning for the clip model. arXiv preprint arXiv:2207.09244. 2022.
Shibhansh Dohare, Juan Harnandez-Garcia, Parash Rahman, Richard Sutton, and A Rupam Mahmood. Loss of plasticity in deep continual learning. 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. https:
net:
M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2007 (VOC2007) Results, 2007. http:/ // /www. www.ppaacaaeeoooooo cha l /wwnneee///////cc222000 http:///wwwwwwwwowoooooooooo nnex html
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International Conference on Artificial Intelligence and Statistics, pp. 3762-3773.
PMLR, 2020.
11

---

Published as a conference paper at ICLR 2024
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint 2xXx:23344.1106 2023.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
https://ieeexpl.cr
ieee. org/ abss:ractt/dccmmettt/ee 6248074
lan J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investi- gation of catastrophic forgetting in gradient- based neural networks. arXiv preprint arXiv: 1312.6211,
2013.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.022: 02677, 2017.
Raia Hadsell, Dushyant Rao, Andrei A Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in cognitive sciences, 24(12):1088-1040, 2020.
Tyler L Hayes, Nathan Cahill, and Christopher Kanan. Memory efficient experience replay for streaming learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 9769-9776. IEEE, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021a. https: petpps:/aariize org/ abs 2006 16241
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021b. https
//axxiv.oq//aae
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and Zsolt Kira. Re-evaluating continual learning scenarios: A categorization and case for strong baselines. arXiv preprint urXiv:/810 /2488, 201 8.
Huiyi Hu, Ang Li, Daniele Calandriello, and Dilan Gorur. One pass imagenet. arXiv preprint orXiv:2111.011 2021.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori,
Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. (ttts:/::: /ww.. ott/o/.o......0200/0// 5281/ zeerooo 5143773 If you use this software, please cite it as below.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. Advances in Neural Information Processing Systems, 35:29262-29277, 2022.
Joel Jang, Seonghyeon Ye, Sohee Yang. Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stan- ley Jungkyu Choi, and Minjoon Seo. Towards continual knowledge learning of language models. arXiv preprint arXiv:2110.03215 2021.
Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, and Minjoon Seo. Temporalwiki: lifelong benchmark for training and evaluating ever- evolving language models. arXiv preprint 12X.::2204.142111 2022.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pp. 427 431. Association for Computational Linguistics, April 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526, 2017.
12

---

Published as a conference paper at ICLR 2024
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS; A bench- mark of in-the-wild distribution shifts. In International Conference Machine Learning (ICML), h
Jonathan Krause, Michael Stark, Jía Deng, and Li Fei-Fei. 3d object representations for fine- ane-grained categorization. In International Conference on Computer Vision Workshops (/CML), 2013. https
www.ccv-ooouaaiioioooooco ion org. oww.maatt.....oooooeeeecec////ee
iccv www.wwowwwooooooeeeeeeeeoereeeee 2013shhpppp_2113
W1 9/9/ httphh///kraus____bbbebee http://uuuur._....oe.reeeprprrrr 2ttp:iiooonnnn____eeee http__221___c___ppper__ccc
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Yann LeCun. The MNIST database of handwritten digits, 1998. http: Lecun. com/ exdb/mnist?.
Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Lung-Woo Ha, and Byoung-Tal Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information process- ing systems, 30, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2947, 2017.
Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2), 2021.
Adam Liska, Tomá's Kočiský, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien de Masson d' Autume, Tim Scholtes, Manzil Zaheer, Susannah Young. Ellen Gilsenan- McMahon Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. Streamingga: bench- mark for adaptation new knowledge over time in question answering models. arXiv preprint arXiv:2205. /1388, 2022.
Xingyu Liu, Alex Leonardi, Lu Yu, Chris Gilmer-Hill, Matthew Leavitt, and Jonathan Frankle. Knowledge distillation for efficient sequences of training runs arXiv preprint arXiv:2303.06489, 2023.
Vincenzo Lomonaco and Davide Maltoni. Core50: new dataset and benchmark for continuous object recognition. In Conference on robot learning. pp. 17-26. PMLR, 2017.
Vincenzo Lomonaco, Lorenzo Pellegrini, Pau Rodriguez, Massimo Caccia, Qi She, Yu Chen, Quentin Jodelet, Ruiping Wang, Zheda Mai, David Vazquez, et al. Cvpr 2020 continual learning in computer vision competition: Approaches, results, current challenges and future directions. Artificial Intelligence, 303: 103635, 2022.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Umberto Michieli and Pietro Zanuttigh. Incremental learning techniques for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision workshops, pp. 0-0, 2019.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, and Hassan Ghasemzadeh. Dropout as an implicit gating mechanism for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pp. 232-233, 2020a.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Understand- ing the role of training regimes in continual learning. Advances in Neural Information Processing Systems, 33:7308-7777 -3200 2020b.
13

---

Published as a conference paper at ICLR 2024
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurlPS) Workshops, 2011.
www.oorraee.ogggaaaao.o.oo com/ c- popppublii cubblicaiion-dee pttp///////3/////////
gub-tools-pie
Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian. Continual vision-languae representaion learning with off-diagonal information. arXiv preprint arXiv:2305.07437, 2023,
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing, 2008.
fttts://ieeexplore.eeess. wttp:////ooooe////55555555
German Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks, 113:54-71, 2019.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In Con- ference on Computer Vision and Pattern Recognition (CVPR), 2012. hattps://eeexpl.o.
leee. Teee.org/eee.ee...... htt.../o//dccmeet/22223853
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to- -o-pnrat- correspondences for richer image-to- sentence models. In Proceedings of the IEEE international conference computer vision, pp. 2641-2649, 2015.
Ameya Prabhu, Philip HS Torr, and Puneet Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCY ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp- 524-540. Springer, 2020.
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puncet K Dokania, Philip HS Torr, Scr-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp-
1698-3707 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 7748-8763. PMLR, 2021.
Sylvestre-Alvin Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In International Conference on Machine Learning (ICML), 2019.
www.s.prees/vve intp::/prooeedingg.uu pttp:pppcceeeiiieee..e..eeeeeees wttpess/////recht1911111
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing.
Association for Computational Linguistics, 11 2019. URL http::tp//// htttt:://aaxi....t org abs 1908
10084
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorchearsal. Connection Science, 7(2): 123-146, 1995.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1XXv:1666..040 arXiv: 10.0 04671, 2016.
14

---

Published as a conference paper at ICLR 2024
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25288 2022.
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In International conference on machine learning, pp. 4528 4537. PMLR, 2018.
Yee Whye
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. httpo//tttt///gttttt..ooo////eie
pytorch- fid August 2020. Version 0.3.0.
Ali Shirali and Moritz Hardt. What makes imagenet look unlike arXiv:2306. 15769, 2023.
laion.
arXiv preprint
Tejas Srinivasan, Ting-Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for vision-and aosoon-nd-langgeee and-language tasks. Advances in Neural Information Processing Systems, 11-29440-29455 2022.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recogni- tion benchmark: a multi-class classification competition. In International Joint Conference on Neu- ral Networks (IJCNN), 2011. https: ftttp::/iieeeppooo/
ieee. org www.ddeeeeennnee330
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106. 10270, 2021.
Richard S Sutton. Two problems with backpropagation and other steepest-descent learning procedures for networks. In Proc. of Eightth Annual Conference of the Cognitive Science Society, pp. 823- 831, 1986.
Vishal Thengane, Salman Khan, Munawar Hayat, and Fahad Khan. Clip model is an efficient continual learner. arXiv preprint arXiv:2210.03/14 14, 2022.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019.
Bastiaan S Veeling. Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivari- ant CNNs for digital pathology, 2018. hatpp::/ :/tp:////aaa//////aa org/ abs 18006.80100. 03962
Tom Veniat, Ludovic Denoyer, and Marc' Aurelio Ranzato, Efficient continual learning with modular networks and task-driven priors. arXiv preprint arXiv:2012.12631,, 2020.
Eli Verwimp, Kuo Yang, Sarah Parisot, Lanqing Hong, Steven McDonagh, Eduardo Pérez-Pelliterro Matthias De Lange, and Tinne Tuytelaars. Clad: A realistic continual learning benchmark for autonomous driving. Neural Networks, 161:659-669, 2023.
Haohan Wang. songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances Neural Information Processing Systems (NeurIPS), 2019. httpss:/aarae
org/ abs /005/ 13549
Jianren Wang, Xin Wang, Yue Shang-Guan, and Abhinav Gupta. Wanderlust: Online continual object detection in the real world. In Proceedings of the IEEF/CVE international conference on compuler vision, pp. 10829-10838. 2021.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-fri free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022.
15

---

Published as a conference paper at ICLR 2024
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715, 2020.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Cenet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00333 2019.
Janet Wiener and Nathan Bronson. Facebook's 's top open data problems. htttp::/researo: http://wwww..ooooo..//////ooooo. 2014. Accessed: 2023-09-28.
10
Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database:
Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV), spr
2016. https: / http:/iiinkks.iiinnn.... com /ttp:////......a////000000/0000/
Peter Young. Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014. httpp:///acllm//oooooy.oorr
Q14-10067
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995, PMLR, 2017.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark, 2019. http://arxiv.oo ab1/1910 04867
7angwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. arXiv preprint arXiv:2303.06628, 2023.
Da-Wei Zhou, Fu- Yun Wang, Han-Jia Ye, and De-Chuan Zhan. Pycil: A python toolbox for class- incremental learning, 2023a.
Da- Wei Zhou, Yuanhan Zhang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for language models. arXiv preprint arXiv:2305. 192700, 2023b.
16

---

Published as a conference paper at ICLR 2024
A CONTINUAL LEARNING BENCHMARKS AND METHODS
We introduce a large-scale image-text benchmark with web scale streaming image text pairs specially developed for studying how efficiently one can get fresh CLIP model with new incoming batches of data. Taalee]compare that this table is not aimed to be exhaustive list of all CL datasets, but the most popular benchmarks in each domain. For language modeling tasks we report the number of examples/documents as the number of samples and for detection tasks report the number of labeled objects/bounding boxes.
the proposed benchmark with existing datasets for continual learning. Note
Table 4: Comparison with continual learning benchmarks.
Benchmark
# Samples
Years
Image-Tea Text
Task
2013 Spli MoMi-NNITT Goodfellow
60K
1998
Classification
Pemm-MNIST Goodrelloow of 2013
60K
1998
Classifiattoo
RoL-MNIST Pape & Ralnz lo
60K
1998
Classification
Split-CIFAR-UU (Zenke
50K
Classifictiion
Split Split-MINI-B Doli-MINI-Imageeee Chiull er
2019
50K
2009
Classification
Belit-ImageNet (Wen cl P0AU
1.2M
2009
Classifcation
Loiit-lmgeeee-- [Wang
30K
Classification
CORe5D (ORReSD(Lammmmn Lamonnlo & Millmi 2017
165K
2017
X
Desection
CLAD Verwimp
23K
2021
Detection
WANDERLUST
[Want (Wai
326K
Desection
Ioc-PASCAL Midinel & Ainottpan 2019
11K
2012
X
X
Segmentation
Inc-ADE20K
20K
Segmentation
StreamingQA Laka et S.. 2UP
100K
2007-2020
Question Answering
TemporalWiss Wiki Tiink Brlll 2022
32M
2021
Language Modeling
CKI. Jann
30K
2019-2021
Language Modeling
CTH.
Vental el Di 2120
300K
1998-2017
Classification
CLOC
C
1121
39M
2006-2014
Classification
CLEAR
7.8M
Classification
NEVIS Bornsciin
2022
8M
1992-2021
Classification
Mod-X
Ni
156K
2014
Retrieval
CLiMB
1022
2013-2021
Classfication
CFCC
15M
2008
2008-2014 2014
Retrieval / /ZS Classification
TiC-RedCaps
12M
1-2000
Retrieval / /ZS Classification
TIC-DataComp
100M/11/12B
2014-2022
Retrieval / /ZS Classification
A.1 EXTENDED RELATED WORK
Neural networks trained new data suffer from catastrophic forgetting of prior knowledge (Sutton 1986; Goodfellow et al. 2013). Addressing the continual learning challenge, researchers have primarily honed in on methods tailored for small-scale benchmarks, specifically focusing on domain, class, or task incremental benchmarks (Hsu et al., [0188] Van de Ven & Tolia 2019). Continual learning of foundation models would significantly reduce the costs and increase quick adaptability. While some recent works have started to introduce continual learning benchmarks, they are not
naturally time-continull and are comparatively much smaller in scale (Ni et al. 2023 Srinivasan et al. 2022). While evaluations on these benchmarks often neglect the consideration of "training time". it becomes a pivotal factor when scaling continual learning approaches to scenarios involving the training of foundation models such CLIP.
In our study, we abstain from comparing with continual learning methods that notably prolong the "training time". Methods such as GEM (Lopez- Paz & Ranzato 2017 Chaudhry al. 2018). and IMM (Lee et al.] (017). which compute gradients for two models in each training iteration. essentially double the training duration. For completeness, we include a comparison with LWF (Li & Hoiem 2017] Ding et al., 2022) and EWC (Kirkpatrick et al.) 2017). While these methods increase computation cost. over standard training due an additional forward pass, the increase in computation cost is relatively much smaller than methods that compute additional gradients. Our LWF implementation is motivated by Ding et al. (2022) which focuses on continual fine-tuning CLIP models on classification tasks by adapting LwF to CLIP models. Instead, for setups where additional compute resources are available, we run our Cumulative approach for slightly longer. Cumulative-Ale narrows the gap with Oracle (refer to Table 2 Given that data storage costs are substantially lower than computational costs at scale, we advocate for taking computational efficiency into consideration in future endeavors.
A.2 DISCUSSION AND COMPARISON WITH CLOC BENCHMARK
Cai et al. (00TT) provide interesting aaseussion/amallyis for continual learning at a large number of steps. However, our study differs from Cai et al. (202T) Th in several crucial respects: (i) Training
17

---

Published as a conference paper at ICLR 2024
Methodology: We employ noisy supervision using contrastive loss between image-texi pairs, as opposed to the cross-entropy loss used by/Cai et (202I). (ii) Scale of Experiments: Our experiments on the TiC-DataComp dataset orders of magnitude larger, scaling up by x00
These differences introduce unique challenges. The use of contrastive loss (i) necessitates a tailored approach designing evaluation studies. The significantly larger scale of our experiments (ii) poses challenges in collecting timestamped data and understanding if and how distribution shifts impact learning this scale.
18

---

Published as a conference paper at ICLR 2024
B ADDITIONAL EXPERIMENTAL RESULTS
B.
DETAILED RESULTS ON OUR BENCHMARKS
Imagenet + shifts accuracy
Static Evaluation TIC-YFCC
Forward Transfer TIC-YFCC
Backward Transfer on TiC-YFCC
30
80
Retrieval Recall
Retrieval Recall
40
60
20
40
10
Train Timestamp
Train Timestamp
Train Timestamp
Oracle**
Cumulative-All
+ Cumulative-All
Cumulative-Exp
Sequential
(a) TIC-YFCC.
Imagenet dist. shifts accuracy
Static Evaluation TiC-Redcaps
Forward Transfer Tic-Redcaps
Backward Transfer TiC-Redcaps
30
50
Retrieval Recall
Retrieval Recall
40
40
30
10
20
2017
2018
2019
2020
2017
2018
2019
2019
2020
Train Timestamp
Train Timestamp
Train Timestamp
Oracle**
Cumulative-All*
Cumulative-All
Cumulative-ENy Exp
Sequential
(b) TiC-RedCaps.
Static Evaluation TiC-Datacomp (M)
Forward Transfer TiC-Datacomp (M) Backward Transfer TiC-Datacomp
30
40
Retricual Recall
Retrieval Recall
Imagenet dist. shifts
30
20
20
10
20
10
10
2016
2018
2020
2022
2016
2018
2020
2018
2020
2022
Timestamp
Train T'imestamp
Train Timestamp
Oracle**
Cumulative All
Cumulative All
Cumulative Exp
Sequential
(c) TiC-DataComp (M).
shifts accuracy
Static Evaluation Tic. TiC.Dataommp (L)
Forward Transfer TiC- IIC-Datacomp (L)
Backward Transfer on TiC- TIC-Datacomp (L)
60
Retrieval Recall
Retrieval Recall
60
40
50
Imagenet
20
20
40
2016
2018
2020
2022
2016
2018
2020
2018
2020
2022
Train Timestamp
Train Timestamp
Train Timestamp
Oracle**
Cumulative All*
Cumulative All
Cumulative Exp
Sequential
(d) TiC-DataComp (I.).
Figure 5: Static and dynamic evaluation performance over time with selected methods in our testbed. As we get more data, all methods improve on both static and forward transfer on dynamic tasks but methods with limited replay buffer start performing slightly worse for backward transfer.
19

---

Published as a conference paper at ICLR 2024
Dysanic excertion of Seceetial
Denamic orkk wnton of Cans latee
Zer Coikoaaa on Coneistto All
Denanic and warden Clack
MX
Enkatton
(a) TIC-YFCC.
Dywenie Trrpectal
Dyywer
rode de
300
2007
200
3008
300
305
2:08
2013
2016
2013
2014
2011
2014
2011
SUN
(b) TeC-RedCaps.
Oyernic
Superiial
Cyneric
Dy
Dor unic sede delvel Cuac
2004 T1ae
2014
2014
2012
2012 3013 2013
201
307 235
20
2006
300 2012
2012
317
2044
2028
300
2012
201
2017
236
2008
2038
201 2012
(c) TIC-DataComp (M).
Zynendc ec satton of Eipeedall
Dyneni es
Do
Dynancc enliatio Candatinu
Dor wode of Dacck
2004
2004
30.4
3008
2013 X11 2015 08. 302 W010
2415 NT7 380 300 30W N0C 2010
2015 2017
735 255 3/0 NO1 W712
2015 2011
730
308 300 301
W077
(d) TIC-Dataaa CCCDDttComp (L).
Figure 6: Dynamic retrieval evaluation results on our benchmarks with Sequential, Cumulative-Exx. Cumulative-All and Oracle. These evaluations highlight the catastrophic forgetting observed with Sequential and Cumulative-Eas Moreover, by observing new data, we not only benefit on tasks from current time step but also improve performance on tasks from old time steps.
B.2 RESULTS WITH BASIC FILTERING ON TIC-DATACOMP XI.
Filtering strategy changes the ordering of performance on static and dynamic retrieval tasks. We observe that while Bestpool filtering models outperform basic filterining models on TiC-DataComp (XL) by 6% on static tasks, they underperform by over 5% on dynamic retrieval task (see Fig. 7. In the main paper (Table 2). we included TIC-DataComp (xlarge) results with Bestpool filtering. In Table we include basic filtering results. We observe that while Bestpool filtering models perform better than basic filtering models static tasks, the order is flipped on dynamic retrieval tasks. Hence, we resort to including results with Basic filtering at smaller scales, but include Bestpool results for completeness as it achieves better results static tasks.
20

---

Published as a conference paper at ICLR 2024
Table 5: Zero shot performance on our time-continual benchmarks (Basic and Bestpool filtering). and *e denote methods that violate the compute budget and use extra compute. For static tasks, we tabulate accuracy of the models obtained on the final timestamp. For dynamic tasks, we tabulate forward transfer, backward transfer and ID performance. For all metrics, higher is better. Bestpool filtering results are copied from Table
Benchmack
Method
Compute (MACs)
ImtgeNet
ImageNet dist, shift
Flickr30k
Tasks
Average over
Backoardd Thansfer
Dynamic Retrieval Tasks
Perfor- mance
Forward
28 datasets
Transfer
TiC-DataComp
Sequential
2.7 10a0
61.2
(XL; Bestpool)
Cummlative-All
2.7 10a0
71.6
58.8
65.1
61.8
70.7
68.5
57
Cumulaiive-All
3.5 16
72.8
60.4
06.3
66.77
71.0
68.6
57.1
TIC-DataComp (XL: Basic)
C CumulativeAAl All
2.7 x 10
63.5
52.0
62.8
58.7
61.6
55.5
27.6
Sequential
10
Oracle
1.1 10
68.0
51.0
63.8
39.6
Dynamic Evaluation on TiC-Datacomp (XL)
Retrieval Performance
75
70
65
Bestpool filtering
Basic filtering
60
2016
2018
2020
2022
Evaluation Timestamp
Figure 7: Comparing Oracle models trained on Bestpool and Basic filtering trained on data from all time steps. Our results clearly highlight that Basic filtering performs better than Bestpool filtering dynamic retrieval task. However, on static tasks, the order is reversed. Moreover, Bestpool filtering shows drop in retrieval performance from 2016 2022 when compared with Basic filtering.
B.3 ABLATIONS WITH LEARNING RATE WARMUP AND MAXIMUM LEARNING RATE
To continually train models more data arrives sequentially over time, multiple cycles of cosine learning rate schedule (Fig. (). There are two crucial design choices: (i) Should we warm up the learning rate for subsequent continual runs? and (ii) How should the maximum learning rate change for sequential training runs?
Table 6: Zero shot performance our time-continual benchmarks with and without initial LR wamrup for subsequent runs. Using warm up on sequential runs after training on the first time step hurts slightly when compared with not using warm up sequential runs.
Static Tasks
Dynamic Retrieval Tasks
Benchmark
ImageNet
ImageNet
Flickr20k
Average
Backward
ID Perfoe- manec
Forward
TiC-DaraComp
Cumulative-Al wammup)
CumuhttiveAll (w warmup)
23.3
20.1
20.3
17.6
35.3
26.1
14.8
TrC-DataComm (L)
Cumulative-Al All (wlo wamme))
Cumulativ All (w warmup)
48.9
47.6
60.8
41.3
50.0
50.9
36.3
35.2
60.1
62.1
67.5
53.0
38.5
41.2
21

---

Published as a conference paper at ICLR 2024
Cyclic learning schedule with warm
0.0010
0.0008
Learning Rate
0.0006
0.0004
0.0002
Warmuup
Warm up
Warm up
0.0000
0.5T
1.5T
(N-1)T (N 0.5)T NT
Iterations
(a) Multiple cycles of standard cosine learning rate schedules which involves warm-up for all subsequent training runs,
Cyclic learning schedule with no warm up in subsequent iterations
0.0010
0.0008
Learning Rate
0.0006
0.0004
0.0002
Warm
0.0000
0.5T
1.5T
2T Iterations
(N-1)T (N 0.5)T NT
(b) proposed cosine learning rate schedule without learning rate warm-up for subsequent training runs.
Figure 8: Learning rate schedule ablations. Schedules vary on how continual training is performed when the training run initialized with the best previous model. When training with cosine learning schedules for subsequent runs, we observe that keeping the same maximum learning rate the first run performs the best.
Table 7: Cumulative experiments on TIC-DataComp (M) with different maximum learning rates for subsequent runs with first fixed at LR 0.00025. Our default choice for subsequent runs is 0.00025. Performance reported on ImageNet. At maximum learning rate 0.001, the runs crashed with Nan in loss.
Method
0.00005
0.0001
0.00025
0.0005
0.001
Cumulative All
16.3
19.0
24.0
10.1
When training with large batches, linear learning rate warm-up typically employed to stabilize the start of the training when beginning from a random initialization (Goyal et al. [017] [Stenne et al.) 202T). However, when training sequentially by initializing models with checkpoints from the previous step, it remains unclear whether we should employ a learning rate warm up not. Our observations highlight that while warm up is benefits for the first time step, not using warm up on subsequent runs performs better. In particular, we observe that removing the warm up for the first training run hurts the final performance. On TIC-DataComp (large), we observe that training ViT-B/16 with warm up on the first time step (i.e., 2016) gets 29.9 zero-shot Imagenet, whereas, without warm up ViT-B/16 achieves only 24.1 1 zero-shot performance on Imagenet. shows the final performance models trained with and without warmup subsequent time steps (after training the first time step with
22

---

Published as a conference paper at ICLR 2024
warmup). In particular, on TiC-DataComp (large), we observe 1.5% accuracy gap on Imagenet and 4.3% accuracy gap on dynamic ID retrieval performance on models trained with and without warm up.
Hence, default to using warmup when training the first time step and not using it the subsequent time steps with all methods except for training on TIC-DataComp (XL) where we add a smaller warm up (10% of the warm up iterations used in first step) to stabilize training.
Next, we experiment with different maximum learning rate when training with cosine schedules. We ablate TiC-DataComp (M) to investigate how to change LR after training data from the first time step. Unlike conventional pretraining and finetuning settings where LR is typically decreased for subsequent training, observe that decaying maximum LR for subsequent steps in setup hurts on static and dynamic tasks and consequently, we use the same maximum LR across our runs (see Table
B.4 PRELIMINARY EXPERIMENTS COMPARING RANDOM SUBSAMPLING WITH OTHER STRATEGIES TO REDUCE BUFFER SIZE
In our preliminary experiments, we explored the efficacy of subsampling old data based on the alignment between text and image content from previous time steps. Specifically, when training a model at time step + 1, we used the model from the end of time step to assess this alignment. We employed two distinct subsampling methods:
1. Retaining half of the data with the lowest alignment scores, based the premise that these data points might be more challenging to learn and require additional gradient steps.
2. Retaining half of the data with the highest alignment scores, under the assumption that these represent higher quality data, as indicated by the stronger alignment between text and image pairs.
We applied these methods to the TYC- dataset and evaluated their performance against baseline of random sampling. The outcomes revealed minimal differences: less than 0.2% variation in Imagenet performance and under 0.5% in dynamic retrieval performance across different time steps. Given that these minor improvements came with a significant computational cost-requiring a full forward pass to compute alignment post each training epoch-they exceeded our compute budget constraints. As a result, we opted for random sampling in our research. We leave investigation on improved subsampling techniques for future work.
B.5 CONST-COSINE: AN ALTERNATIVE LEARNING RATE SCHEDULE
The defacto LR schedule for training CLIP models is an initial linear increase a maximum value, i... warm up, followed by cosine decay (Radford et al., [021] Gadre et al. 2023). In the main paper, we default to using cosine LR schedule for each sequential run, resulting in eyclic schedule. We observe significant increase in training loss early in subsequent runs when the LR is high. Comparing the loss on training data with Cumulative and Oracle methods, we observe that as training progresses the training loss increases every time the learning rate is increased to the maximum LR (Fig. 9).
would be ideal for continual training to employ a learning rate schedule that "forward looking", allowing to continually train from previous checkpoint without experiencing significant increase in training loss. One desirable property of such a learning rate schedule would be its ability to adapt without requiring prior knowledge of the decay period.
23

---

the same Published as a conference paper at ICLR 2024
15
Training loss
10
2.0
200
400
600
800
1000
1200
1400
1600
Iterations
Cumulative (All)
Oracle
Figure 9: Training loss increases every time the LR is reset to maximum LR for Cumulative. Loss comparison on training data with Cumulative and Oracle method. Cumulative is trained with cyclic cosine schedule without warm up for sequential training runs. For Cumulative, we plot the loss on training data, and the training progresses, samples from new time steps are added the training pool. For Oracle, the training data is the union of data from all time steps and remains the same throughout the training.
Constant + Casine learning schedule
0.0010
Spawn new run with decay
0.0008
Learning Rate
0.0006
0.0004
0.0002
Decay (0.2%N*)
Warm up
Decay
Decay
0.0000
0.5T
T
1.5T
2T
(N-1)T (N 0.5)T
Iterations
Figure 10: Const-Cosine Our proposed alternative forward-looking learning rate schedule schedule which trains one model with constant learning rate and decays the learning rate with cosine schedule only for a fraction of iterations before obtaining a deployable model. Const-Cosine schedule uses an extra compute budget than an Oracle run because an extra training run is launched for the fraction of training when learning rate is decayed.
our work, we perform preliminary experiments with the simplest alternative, Const-Cosine where after the warm up period, we train with constant learning rate and decay the learning rate only for small fraction of training towards the end when we want a deployable model (Fig. TTP. This allows us to continue training for subsequent runs from the checkpoint at the end of the constant learning rate schedule and decay the LR only in the end. For our experiments, we fix the decay period as 0.2 of the total training iterations. Due this, Const-Cosine schedule slightly increases the overall training budget of the Cumulative runs when compared with cyclic cosine schedules.
For Costne, only ablate relatively smaller scale datasets in testbed (i.e., TIC-YFCC, TiC-RedCaps, and TiC-Dataf CCC-DatCComp with the same Const-Cosine schedule. Note that for Const-Cosine experiments, maximum LR as with the cosine schedule.
For a fair comparison, we also re-run Oracle methods
We observe that training with Const-Cosine schedule significantly improves both Cumulative and Or- acle as compared to their counterparts trained with cosine learning rates Moreover, as expected, we
We also experimented with Const-Cosin schedule for Oracle training TiC-DataComp (1 (1ar and TiC- TTC-DataComp (x1 arge). We observe that with decay fraction of 0.2, Const-Cosine achieves similar results
24

---

Published as a conference paper at ICLR 2024
do not observe jumps in training loss when training Cumulative with Const-Cosine schedule. How- ever, the gap between Oracle and Cumulative with Const-Cosine doesn't 't decrease when compared with gap between Oracle and Cumulative with cosine learning rate schedules. This highlights that the jumps in the training loss observed while training with the cyclic cosine schedule might have benign effects the final performance.
Table 8: Zero shot performance on Imagenet with Const-Cosine LR schedule. We observe that Const-Cosine improves over cyclic cosine LR schedule. However, the gap between cyclic cosine LR schedule and Const-Cosing for different LR schedules remains the same. denote methods that violate the compute budget.
Method
Cosine LR Schedule
Const-Cosine LR schedule
Benchmark
Compute (MACs)
ImageNet
Compute (MACs)
ImageNet
Cumulative-All Oracke**
3.4 x 10
29.3
4.4 x 10
32.8
TIC-YFCC
10
29.2
8,5 10
33.2
TiC-RedCaps
Cumulativs-Al Oracle**
344 x 10
32.2
4.4 x 10ts
35.1
32.7
8.5
36.2
ToC-DataComp
Cumulative All Oracle
1.2 x 10
10
24.0
25.5
3.6 1048
1.2 x 10
28.2
28.9
B.6 OPENCLIP MODELS OBTAINED BY RETRAINING AFTER REMOVING ANY DUPLICATE EXAMPLES FROM THE TEST SET
OpenCLIP models (e.g., models trained Datacomp and LAION-5B) have been trained data curated from Common Crawl. Since the retrieval tasks we constructed are built on top of data curated from Common Crawl, one may argue there is possibility of train/test overlap in evaluations of OpenCLIP models. Thus, we retrain OpenCLIP models on DataComp datasets after removing the samples in test Figure IT shows that the trends observed for OpenCLIP models holds for our retrained models.
Our Proposed Evaluation Paradigm
Retrieal recall on 2022 2021-2022 20022
OpenAl mode's trained on data before 2020
OpenClip on before Our Models
40
50
60
70
80
Figure 11: We replicate OpenCLIP models by training from scratch and removing duplicates from the evaluation dataset. We observe that trends continue to hold.
B.7 RESULTS ON DYNAMIC CLASSIFICATION TASK
In the main paper, we include results on our dynamic retrieval task. For completeness, here we include results dynamic classification tasks T1C-DataComp splits (Table 9). Along with including results on all nodes of ImageNet, we also include results on classification task restricted to classes in the "motor vehicles" subtrec of ImageNet hierarchy. For the dynamic classification task, we observe trends similar to the dynamic retrieval task.
to that of the cosine learning rate schedule. In particular, Const-Cosine achieves 61.3 on large and 73.0 on x1 a arge versus Cosine schedule achieves 62.3 on arre and 73.3 x1 xlarge. This highlights the potential of training with Const-Cosine schedule in scenarios where total training duration might be unknown apriori.
25

---

Published as a conference paper at ICLR 2024
Table 9: Zero shot performance on our TiC-DataComp-Net classification task. and denote methods that violate the compute budget. We tabulate forward/backward transfer and ID performance on classification tasks (Sec. 2.3 For -Daa (XL), we include results with Bestpool filtering.
Dynamic
Tasks (All)
Dynamic Retrievul
Mooor Vehicles')
Benehmark
Method
Compute (MACs)
Backward Transfer
Perfor- mance
Forward Transfer
Backward Transfer
Perfor- mance
Forward Transfer
Sequental
3.0 x 10ls
15.9
13.3
B.9
34.5
30.0
22.6
Patching
t0
34.4
22.1
Cumplatise-Ea
3.0 x 1015
17.6
14.4
10.4
36.6
31.9
23.5
TCC-DataCommp (M)
Cumulative-
101s
17.5
36.4
23-5
Cumulative-AA
3.0 x 10ts
18.3
14.7
10.6
38.2
31.7
23.7
Luke
3.8 x Toils
16.0
13.5
9.9
35.1
30.7
23.3
Cumulative-AUr
x 10ts
40.4
32.3
23.9
Oracle*
1.2 x 1043
19:2
15.2
10.7
38.1
31.9
23.5
Sequential
10
38.3
Patching
2.7 x 10t9
38.6
36.8
33.3
58.3
51.9
19.3
Eup
2.7 x 10!9
-00.2
37.9 8
34.2
60.7
36.8
61.1
TIC-DataComp (L)
Cumulative Equal
10t
Cummlative-All
2.7 x 10
41.3
38.3
31.4
61.4
56.6
50.9
Cumulative-All Alll
101>
43.0
39.2
31.6
62.7
57.5
51.1
Oracke**
1.1 10
13.8
10.0
35.2
62.6
56.8
50.7
Saqential
27 10
55.4
55.1
53.3
67.8
66.0
63.5
TIC-DataComp (XL)
Cumulative-Ae
1020
63.8
Cumulative-Al All"
3.5 x 10
58.8
56.9
51.3
70.5
67.5
63.8
B.8 ADDRESSING DIFFERENCES BETWEEN SEQUENTIAL AND CUMULATIVE-ALL BETWEEN TIC- AND TIC-DATACOMP
In Table we observe differences in the behavior of Sequential and Cumulative- TIC-YFCC when compared with T1C-DataComp. For instance, differences between the ID performance between Sequential and Cumulative-All is larger in TIC-YFCC than in TIC-DataComp (M). Similar observa- tions hold true for backward transfer performance. In this section, explain the underlying causes for these differences.
We identify two primary reasons:
(i) the nature of the distribution shift observed in TIC-YFCC. We observe that models trained with Sequential TIC-YFCC suffer from relatively larger drops old-time steps than TIC- DataComp (M) due to catastrophic forgetting (see Fig.
(ii) compute used at each time step per data available at each time step is different for these bencmarks. Overall YFCC is 2x smaller than Tic-Datacome (M) but the compute used in both TTC- and TiC-Datacomp setup is of similar order (in fact, is slightly higher in TiC- YFCC). We re-ran the experiments for Tic- YFCC by reducing the compute. In the updated runs, we observe that the gap between ID performances Sequential and Cumulative-Al vanishes.
Table 10: Zero shot retrieval performance on TIC-YFCC with Sequential and Cumulative-All with reduced compute.
Dynamic Retrieval Tasks (original compute)
Dynamic Retrieval Tasks (reduced compute)
Benehmark
Method
Compute (MACs)
Backward Transfer
ID Perfor-
Forward Transfer
Compute (MACs)
Backward Transfer
ID Perfor- mance
Forward Transfer
TIC-YFCC
Sequential
3.4 x 10
42.2
48.4
23.7
1.5 1018
27.0
42.0
15.7
Cumulative-Al
3.1 x
66.4
60.2
27.6
10
46.3
38.7
17.3
C ADDITIONAL BENCHMARK DETAILS
C. FILTERING ABLATIONS ON TIC-DATACOMP
For Basic Filtering, Gadre et al. 20033 performs the following three steps: filter by English language (using fasttext (Joulin et al. 2017). filter by caption length over two words and 5 characters, and filter by image sizes with smallest dimensions over 200 pixels and aspect ratio above 3. We do not default other filtering techniques that use off-the-shelf CLIP models from Gadre al. (2023) to
26

---

Published as a conference paper at ICLR 2024
Dheant of Era
Dysentc enkerton Candeeios
(a) TIC-YFCC (original compute).
Eop
Cantesto
tiaing
traine
(b) (reduced compute).
Figure 12: Dynamic retrieval evaluation results with Sequential, Cumulative-All on TIC-YFCC with reduced compute.
avoid biasing dataset selection from each time step. In Fig. 13 we show that "Bestpool" filtering (which filters image-text pairs with CLIP scores and ImageNet image embeddings) biases dataset selection to preferring old time step data over new timestamp data. Moreover, we also show that models trained with Bestpool filtering is less robust when evaluated our dynamic tasks from 2021- 2022 (Fig. 13 Nevertheless, for completeness and to highlight the significance of our findings even for state-of-the filtering at x xlarge scale which is included in the main paper. In App. B.2 we include results with Basic filtering at xlarge.
filtering techniques, perform continual learning experiments with Bestpool
Retreccal Penoemance 3023 2022
OpenCiip models [Bertpook ritering):
OpenClip models miteringt
0.4
Frection retained
year
0.3
Basic
0.1
CLIP
BestPool
70
2014
2016
2018
2020
2022
Retrieval Rurforancc on 2014-220 2015
Year
Figure 13: (Left) Gap in retrieval performance for different OpenCIIP models that use different filtering techniques. (Right) Reduction in TIC-DataComp data different times with different filtering techniques. This clearly highlights that there is selection bias towards retaining more old data for CLIP/BestPool filtering. No such bias exists for basic filtering.
27

---

Published as a conference paper at ICLR 2024
C.2
STATIC DATASETS CONSIDERED FOR EVALUATION
Table 11: Evaluation tasks borrowed from Gadre et al. 2023
Tack typo
Datasel
Tusk
Test act size
Number of classes
Main metric
Food-100
Bussard clall
2014
Food recognition
25.250
101
GISRB
Stallkamp
sign recognition
accuracy
ImageNat
2005
Visual recognition
50,000
1,000
accuracy
ImageNet
Skarhh
recognition
accuracy
ImageNet
dl al.
2019
Visual recognition
10,000
1,000
accuracy
ImageNet-A
2021b
Visual recognition
7,500
200
ImageNet-O1
recognition
accuracy
ImageNettR Tlundrycks el
A1/10
Visual recogrition
30,000
200
accuracy
KITTI
Distance prediction
accuracy
LaCSm
1996
Digit recognition
10,000
accuracy
ObjectNaz Barbu
2019
Visuil recognition
18,574
113
Zipeerman
recognition
6.149
moin per class
Oxford-IIIT Pel Parkin 2017 Zin e
2019
Pei classiliction
3,669
37
men Shruune aanoee
chas
Object recognirion
PulchCamdyynn Vedling dlal
2013
Zha cla
2019
Metasiatic tissue db
32,768
Rendered SST2 ( eiirel 2n1
Sentiment clasifleation
1,821
2
RESISCA5 Cheis
Saellite imagery recogniticn
6,300
Stanfond Cara Kritise el ml 21133
Vehicle recogniion
8,041
196
ST1--10
Visual recognition
8,000
SUN-397 Xiap 25
2016
Scene recognition
108,754
397
SVHN
SVHN Never of
20T1 Phoi ot al
2019
Digit recognition
26032
accurady
on
recagnition
score
Camelyon
KKins ei 2nth
of
Metostari fissue els.
85,054
2
aceuracy
FMoW Christe
Satelliee imagery recognition
22,108
worst-region
Retrieval
Flickr3kk
Young clal.
2014
Image text restieval
31,014
N/A
R81
C.3 OUR BENCHMARK STATISTICS
In this section, we discuss statistics of our constructed benchmarks. Fig. [4 summarizes TiC-RedCaps, TIC- YFCC and TIC-DataComp dataset sizes. Fig. 15 summarizes orgiinal YFCC dataset sizes. Table 12 Table 13 and Table 14 present the exact numbers for these datasets. For TiC-DataComp. only discuss the sizes xla rge scale.
TIC-YFCC Sizes
-.94
TiC-Redcaps
TIC- TC-Daatcomp Sizes (XL.
No Filter
1.5
CLIP
Sizes
Size8
BesiPaol
1.0
0.5
2004-200 2008
2009-2010
2011-2012
2012-2014
2017
2018
2019
2020
0.0
2014
Timestamp
Timestamp
Timestamp
Figure 14: Number of examples in each year in benchmarks.
28

---

Published as a conference paper at ICLR 2024
250000
100000
50000
2002 8
2004, 10
2008 8-
2008. 3-
2008. 12
2007 10
2009. 2009.1 11
2013. 2-
2011 12
2012. 5
2015 8.
2004
2005,
2008.
2007
2008
2008
2009
2009.
2010
2010
2011.
2012
2013.
2014.
Figure 15: Number of examples in each year original YFCC 15M. X-axis the upload month and
y-axis is the number of examples in that month.
Table 12: Number of examples in TIC- TiC-RedCaps in each year.
Year
Dataset
2017
2018
2020
TIC RedCaps
4,220,262
1,660,003
2,526,575
3,115.715
Table 13: Number of examples in TIC- in each year.
Year
2004-2008
2009-2010
201 2011-2012
2012-2014
TIC-YFCC 4,337,727
4,050,166
3,976,339
2,312,753
Table 14: Number of examples in TIC- DataaCmop in each year before filtering.
Datuect
Year
2014
2015
2016
2017
2018
2019
2020
2001
2023
TIC: DuraComp (no fitte)
244 802 598
ITS 648,045
066,019.511
1.906,55 257.75
L,77,561 1715
20116011 288
7,778251 251 0066
2064,463.00 701
233.1233.121
TiC-DeuCCupp (Ieoi (iber)
52,764,775
$0,757,80
301,347,511
319,575.600
417067,014
371,745.613
Next, we tabulate the number of examples in our retrieval evaluation datasets. Since the evaluation dataset sizes are different at different time steps, we subsample the dataset to a fixed size before performing retrieval evaluations. On TIC-YFCC and TiC-RedCaps, image-text pairs from these evaluation datasets. For TiC-DataComp. we randomly sample 4000 image-text pairs. We repeat this process for 3 seeds and report the aggregated performance.
randomly sampled 1000
Table 15: Number of retrieval evaluation examples in TiC-RedCaps in each year.
Year
Dataset
2017
2018
2019
2020
TiC-RedCaps
31,316
42,539
16,738
25,565
Table 16: Number of retrieval evaluation examples in TIC-YFCC in each year.
Year
Dataset
2004-2008
2009-2010
2011-2012
2012-2014
TIC-YFCC
43,820
40,909
40,168
23,354
Table 17: Number of retrieval evaluation examples in TiC- TiC-DataComp in each year before filtering.
Year
Dataset
2016
2017
2018
2019
2020
2021
2022
TiC-DataComp
23,085
39,289
50,450
53058
42,239
49,841
38,051
29

---

Published as a conference paper at ICLR 2024
C.4 COMPUTE CONSTRAINTS FOR DIFFERENT DATASETS
We closely follow compute budget constraints from Gadre et al. (023). In particular, on TIC- DataComp, we restrict to using exactly the same amount of overall compute as fixed in Gadre et al. (023)). Below we list exact total MACs on each dataset:
YFCC: Total MACs: 3.4 10
TiC-RedCaps: Total MACs: 3.4 x 10t8
TTC-DataComp medium: Total MACs: 3.0 x 10
TIC- CC--DaCCCmmp large: Total MACs: 2.7 1029
TiC-DataComp x arag Total MACs: 2.7 1020
For a ViT-B architecure, these values correspond to 20k iterations on TIC- YFCC (batch size: 8192). TiC-RedCaps (batch size: 8192), 35k iterations on TiC-DataComp (M) (batch size: 4096), 157k iterations on TiC-DataComp (L.) (batch size: 8192), and 143.5k iterations on TiC-DataComp (XL)
(batch size: 90100). We divide these iterations equally among all time steps.
C.5 CREATION PIPELINE FOR EVALUATION DATASETS
TIC-Data C.C-DataComp-Retiti
To create a retrieval task, we sample batch of IID image-text pairs from different timestamps and evaluate text retrieval performance given the corresponding image (similarly, image retrieval given the corresponding text). Alongside general evaluations, we also construct datasets from specific domains, e.g., Covid-19 19 subset and Flickr subset. To create Covid-19, we filter the dataset to only retain pairs where the caption contains mention of "covid". This search process restricts the data to time only after 2019. For the Flickr subset, we filter the dataset to only retain pairs where the corresponding "url" contains data from Flickr.
We create our dynamic classification dataset TIC-DataComp-Ne with Ima- geNet classes from the PommonPool data augmented with temporal information. Our construction process draws inspiration from the LAIONet construction process described in infShirali & Hardi (2023). In particular, we first filter examples where the corresponding caption contains one and only the synsets of ImageNet-IK I. We also apply additional basic filtering (Gadre et al. 2023) to make sure that images of at least 200 size smallest dimension and the caption contains least 2 words and 5 characters. After filtering for examples with ImageNet synsets, we only retain examples where the similarity- as evaluated by an aff-the-shelf sentence embedding model (Reimers & Gurevych] 2019) between imagenet synset defimition and the caption exceeds threshold of 0.5. The goal of this filter- ing step is to restrict examples with "high" alignment between caption and imagenet synset definition. This last step differs from the LAIONet construction. Crucially, unlike LAIONet, we do not filter the image-text pairs with CLIP similarity scores avoid biasing the dataset selection process.
TIC-DataComp-Ne
of
C.6 DISTRIBUTION SHIFT ANALYSIS ON PROPOSED BENCHMARKS
Retreval Perfomance en Cove 2023 2023
OpenAl mede's before 2020
Filte 20220220000 000112202
70
OpenAi mede' before 2020
OpenCIip models trained on data betore
OpenCiip models traincd on data betore
50
30
80
40
30
Nutroecal Notfor
30
20
10
30
50
49
Rotrioval Parformancc on Flickr 2014-2015
Retrieval Parformance on Riike 2014-2015
Figure 16: (Left) Comparison of retrieval performance on COVID queries versus Flickr queries (construction described in App. C.5). (Right) Comparison on old Flickr versus new Flickr data. Clearly, we observe that while gap on old versus new flickr data is small, the gap is significantly larger Covid queries.
30

---

Published as a conference paper at ICLR 2024
Motor Subtree
0.75
Openo
2021-2024
2.50
Opendt
3,70
2.75
2.50
2.70
0.55
2.05
2.50
2.45
3.00
3.45
0.30
0.80
0.65
0.75
0.015 0,650
0.752 0.315 0.000
Ti--Deteoemp-beet
TR-Betetomp-Ken
Figure 17: (Left) Comparison on old versus new data from TiC-DataComp-Net. (Right) Comparison on motor vehicles node from TiC-DataComp-Ner For our classification task, we observe a very small drop (< 1%) when averaged across all categories. However, we observe substantial gap on classes in "motor vehicle" subtree, when comparing OpenAl and OpenCLIP models. These findings highlight that while overall ImageNet classes may remain timeless, certain categories tend to evolve faster than others.
TiC-DataComp analysis through the lens of constructed evaluation tasks Here, we compare performance of OpenAl and OpenCLIP models our datasets. We observe significant performance gap between OpenAl and OpenCLIP models on our dynamic retrieval task (Fig. D. This gap widens notably retrieval queries where captions mention COVID-19. On the other hand, OpenAl and OpenCLIP models exhibit similar robustness for retrieval on data coming from Flickr highlighting that data from some domains do not exhibit shifts that cause performance drops. For our classification task, we observe a very small drop ( 1%) when averaged across all categories. However, we observe substantial gap on specific subtrees in ImageNet. For example, classes in "motor vehicle" subtree show an approximate 7% performance drop, when comparing OpenAl and OpenCLIP models. These findings highlight that while overall ImageNet classes may remain timeless, certain categories tend to evolve faster than others. Our qualitative and quantitative analysis on TiC-DataCoo clearly highlights evolution of distributions and captures different properties than standard benchmarks.
Quantitative analysis on TIC-YFCC We analyze TFC-YFC using off-the-shelf sentence and image encoders. For off-the-shelf sentence embedder, we used an existing sentence transformer from Hugging Face (Reimers & Gurevych 2019). For the image encoder, we use CLIP pretrained ViT-B- 16 model (Radford et al. 2021 Tharco et al. (02I).
We first embed images from different time steps with OpenAl CLIP encoder and then compute Frechet Inception Distance (FID; Seitzer 2020)). As time progresses, we observe that FID distance increases with respect to data from first time step (Fig. I8). Similarly, transformer to extract top-5 categories from Wordnet Nouns for each caption. We then obtain a distribution these Nouns for each time step. We observe that the TV distance over the distribution of WordNet nouns evolves over time when compared to data from the first time step.
the pretrained sentence
C.7 CREATION PIPILINE FOR TIC-DATACOMP
We collect timestamps for the CommonPool dataset introduced in DataComp. We repeat the crawling process described in Gadre et al. 2023) to download WARC files from Common Crawl. In particular, we follow the same multistep process which involved: (i) parsing URLs and alt-text from Common Crawl dumps and downloading these images; (ii) tagging images with meta data and id of the common crawl batch; and (iii) conducting evaluation set duplication and safety content filtering. After downloading the WARC files, perform a join with the datacomp 12.8B examples. During this join, we lost approximately O. of examples that are no longer available online. Morcover, while performing this join, we only retain examples with their first occurrence. This is done before running any de- de-dullication on image-text pairs for exact matches as done in Gadre et al. (2023).
The source of DataComp is Common Crawl, which periodically releases web-crawled data snapshots, typically on a monthly basis since 2014 with new and updated webpages. This process provides timestamps at the granularity of months, spanning years 2022.
31

---

Published as a conference paper at ICLR 2024
03
0.10
04-
0.4
02
0013
03-
0.59
0.
Tieeslopp
Timestep
Treestep
0.00
Distance between categories from different timestep with respect to timestep
FID distance CLIP embeddings with respect to timestep 0
(a) TIC-YFCC
1.00-
1.175
5.50
4.125
5.000
0.000
1.755
0,00
0.05
11.r
You
Distance between categories from different timestep with respect timestep 2016
FID distance CLIP embeddings with respect to timestep 2016
(b) TIC- DCC-DatCComp (M)
Figure 18: Distribution shift results. Analysis on TIC-YFCC and TiC-DataComp (M) using off- the-shelf sentence and image encoders. We first embed images from different time steps with an OpenAl CLIP encoder and then compute Frechet Inception Distance (FID; Seitzer (2020)). As time progresses, we observe that FID distance increases with respect to data from first time step. Similarly TV distance over categorical distribution Wordnet Noun synsets also increases with time when compared to categorical distribution on first timestep.
TV distancc between cotegeries from exclustion timestops artt reference Emmestos
acorM bettereer images eve timestags Limeslapa
0.202
2006
6:1
0:
3
2.54
6.14
2016
6:22
0
0..00
0.5L
0:37
0.330
0.225
2018
8:22
6.31
0.00
6.3L
2019
6:17
617
0.
0:31
3.33
5.19
2019
8-2
6:27
0:51
0.00
6:21
6-21
0.81
0.000
2020
0.23
0:25
0:32
3.00
3.07
5.07
2020
8:22
8.22
0:21
6:30
0.29
0.
6.353
0:30
0.8
00,05
6:235
2017
2018
2019
2020
2021
2002
2016
2017
2018
2019
2020
2022
stop
Stop
Figure 19: Distribution shift analysis on TTC-DataComp (M) using off-the-shelf sentence and image encoders. We first embed images from different time steps with an OpenAl CLIP encoder and then compute Frechet Inception Distance (FID; Scitzer (2020)) As time progresses, distance increases with respect to data from first time step. Similarly TV distance over categorical distribution Wordnet Noun synsets also increases with time when compared to categorical distri- bution on first timestep.
observe that FID
We note that while this augmented time information may contain some noise, average, find it to be a reasonably accurate proxy for the upload time of web pages. To perform an initial check, note that data contains images from flickr which provides API query for true upload timestamp. So we extract 10k examples from our benchmark TIC-DataComp and query Flickr for their true timestamp. Fig. 20 summarizes true timestamps with timestamps extracted from CC.
32

---

Published as a conference paper at ICLR 2024
2004
0.05
0.06
40
2005 0.63
0.34
0.38
0.5
0.58
0.44
0.29
0.23
0.18
2006 2.01
1.16
1.41
1.37
1.63
2.17
0.83
0.79
0.75
35
2007
4.36
2.19
3.18
2.40
3.14
1.93
2.00
2008
5.06
4.94
5.25
5.83
3.61
2009
10.80
0.82
7:79
7.00
7:53
5.46
4.84
3.38
3:98
30
True time step
2010
12.89
6.68
8.41
7.92
5.76
4.83
3.75
4:37
2011
16.24
25
2012
19.07
11:58
10.30
13. 42
0.56
7.49
0.63
5.81
6.22
2013
12.01
8.12
10.05
10.41
8.66
7.81
6.66
6.56
6.07
20
2014
7:30
8.36
6.37
2015
0.00
41.06
9.36
8.80
6.15
7.64
6.34
6.26
4.79
2016
0.02
0.02
27.31
9.75
5.31
7.64
6.91
5.86
5:15
15
2017
0.00
13:57
8.58
7.99
2018 0.01
0.01
0.00
0.01
20.87
10.94
6.99
5.36
5.29
10
2019 0.01
0.00
0.00
0.00
0.01
18
10.75
5.90
5.49
2020 0.00
0.01
0.00
0.00
0.02
0.02
20.52
5
2021
0.00
0.02
0.02
2022 0.00
0.00
0.01
0.00
0.03
0.00
0.02
0.02
24.09
2014
2015
2016
2017
2018
2019
2020
2021
2022
CC time step
Figure 20: Comparison of Common Crawl assigned timestamp and true timestamp on a subset of 10k examples containing image-text pairs from Flickr. We observe clear trend where CC timestamps correlate with true timestamps.
D ADDITIONAL EXPERIMENTAL DETAILS
D.1 ADDITIONAL DETAILS ON IMAGENET IID SPLIT CONTINUAL LEARNING EXPERIMENT
With ImageNet data, we consider 4 and 8 splits including the full dataset. This design is inspired by Ash & Adams 2020). We consider ViT-B/16 architecture trained for 300 epochs on full data and split the iterations corresponding 300 epochs equally among k splits when training sequentially. We keep all other hyperparameters, such as learning rate, optimizer, and batch size, set to the standard values typically employed for training ViT-B/16 on the ImageNet dataset (Dosovitskiy et al. 2020). We also employ & regularization and augmentation on ImageNet training data. We evaluate the models on IID ImageNet test set.
Our Imagenet experiments were primarily inspired by the "loss of plasticity" phenomenon described As & Adams (2020). Their study demonstrates that models sequentially trained on two splits of CIFAR-IO data (initially on 50%, followed by 100% of data) exhibit poorer generalization compared to models trained from scratch on the entire dataset. Since we do not observe this behavior for continual training of CLIP, we investigated the existence of such behaviors on up 8 splits of Imagenet. Our findings reveal that the simple cumulative baseline (with no extra budget) remains competitively close to the Oracle model (that benefits from using the full compute budget on the entire pooled training data from the beginning).
Prior works (Prabhu et al. 2023 Hu et al., 2021) performed continual learning experiments on Imagenet to compare different methods and highlight the effectiveness of continual training on synthetic continual learning setups derived from ImageNet. While these papers include results with an Oracle method, differences in the settings considered in these studies limit direct comparisons.
In particular, we show the performance gap of less than 1 in the same setup used otherwise in the paper when using SOTA training procedures achieving 81 % validation performance. Comparitively the referenced Hu et al.] (202T) does not show whether the 65% to 77% performance gap in their Table 1 can be bridged by increasing the compute for their method. Instead, authors show that if they restrict the compute for Oracle in Table 2, the Oracle performance drops to 68% (with s 3% gap).
Moreover, in Prabhu et al. (202)), authors perform experiments on D1-Imagenet-2K where they start with an initial memory of Imagenet- 1k 1.2 M samples and sequentially observe data for the same classes 1k classes from Imagenet-21k pool. This makes comparing streaming accuracy (or Imagenet- 1k accuracy) for different methods incomparable with our setup (with a gap of over 7% in streaming accuracy even step 8 as compared to less than 1 1% in our setup).
33

---

Published as a conference paper at ICLR 2024
D.2 TRAINING AND HYPERPARAMETER DETAILS
We create common experimental setup by fixing the training procedure for sequential runs. Unless specified otherwise, closely follow the CLIP training recipe proposed in (Ilharco et al. [02T Radford et al., 2021) where we train models with a contrastive objective over images and captions. Given set of image-text pairs, we train an image encoder and text encoder such that the similarity between the representations of images and their corresponding text is maximized relative to unaligned pairs. Only LwF deviates from this standard training procedure. For each benchmark, we pick Vision Transformers ( (ViTs) as the image encoder, in partieular, we fix the model architecture to ViT-B/16 (Dosovitskiy et al. [202I). We fix the Adam optimizer and its hyperparameters to values suggested in (Ilharco et al. 2021).
We primarily ablate over only two things: maximum learning rate with cosine learning schedule and warm up iterations for sequential training. For choosing other hyperparameters, follow the OpenCLIP library (Ilharco et al. 2021).
D.3 REPLAY SIZES WITH EXP AND EQUAL STRATEGIES
We default to using 2D size of data where D represents incoming data size from time step. As described in the main text, for -Exp, we reduce the buffer size by half what we used at old time step and use rest of the half data from previous time step. App. C.3 lists the dataset sizes for each benchmark which dictate the exact buffer sizes.
E RESULTS WITH OTHER CONTINUAL LEARNING METHODS
E. RESULTS WITH EWC METHOD
As proposed in the original work Kirkpatrick et al. (2017). we implement EWC method where we optimize the following loss:
(ewc( (()) + Σ XEwG F.(O: 02 0
2
where (()) the standard contrastive loss on data from time step t, F; is the i-th diagonal entry of the fisher information matrix, and 0.-1 the frozen parameters from previous time step. We perform experiments with different values of XEwc E (1, 10, 4003 (see Table 18
Table 18: Zero shot performance on our time-continual benchmarks with EWC. * and ** denote methods that violate the compute budget. For static tasks, we tabulate accuracy the models obtained on the final timestamp. For dynamic tasks, we tabulate forward/backwart transfer and ID We observe that EWC performs worse than Sequential,
performance on retrieval tasks (Sec. 2.3 Patching and LwF.
Static Tasks
Dynamic Retrieval Tacks
Benchmark
Method
Compute (MACs)
ImageNet
ImageNet shift
Flickr00k
Average
Backward
Perfor
Forward
TiC-DataComp
Sequentaa Patching
10ts
19:3
19.3
16.8
14.7
26.9
25
14.5
LwE
3,8 1g4a
19.2
16.5
17.7
143
25.6
26.6
11.
EWC (kewa 1)*
3.6 1014
18.7
16.3
16.2
15.1
25.5
26.4
14S
(Aewo
101s
18.1
EWC (Aewo = 100**
3.8 x
17.6
15.4
16.3
14.8
21.4
25.4
14.3
EWC (Agre = 400"*
3.6 x 1044
17.0
15.0
16.1
143
21.1
21.9
14.0
E.2 RESULTS WITH OVERSAMPLING + COUNTING BASED SAMPLING METHOD
In this section, we perform ablation Cumulative- In particular, made the following two modifications: (i) Count based sampling: Instead of random sampling, we implemented the count- based subsampling that prioritizes not/less used examples; (ii) Oversampling: We oversampled data from old timesteps with ratio inversely proportional to the ratio of examples, i.e., the old data is of size D/2 and the new data is of size D, then upsample old data with 2:1 ratio.
However, we observe that this method doesn't improve performance over Cumulative- and in fact hurts the performance slightly (see Table T9). We hypothesize that this be due decreasing
34

---

Published as a conference paper at ICLR 2024
marginal utility of labeled data highlighted in Cui al. (2019). Their work argues that due to information overlap among data, as the number of samples incrasses, the marginal benefit a model extract from the data diminishes. As result, |Cu et al. (2019) proposed using of "effective sample size" instead of the actual number of samples to obtain the ratio used to perform re-sampling or re-weighting. In particular, the expression of "effective sample size" is given by E, = L-8" T-
where is the original sample size and ,3 is hyperparameter that Cui et al. (2019) selects from E (0.9. 0.99, 0.999, 0.9999).
For different time steps, we leverage this expression of E,, to calculate the effective number of samples. In our settings (even at small scales), our datasets contain an order of 100k image-text pairs even after subsampling data from old time step. For example, with -Equal baseline, when training on the last time step (i.e., 2022), the smallest dataset (i.e., 2016) of approximately 400k samples. Plugging in the expression for effective sample size from Cui et al. (2019). we observe that for all E (0, 0.99999). the ratio effective sample sizes for different time steps remains close to This may highlight why our naive over-sampling strategy doesn't 't improve over no-oversampling.
Table 19: Zero shot performance on our time-continual benchmarks with oversampling and counting-bass based sampling. and denote methods that violate the compute budget. For static tasks, we tabulate accuracy of the models obtained on the final timestamp. For dynamic tasks, we tabulate forward/backwart transfer and ID performance retrieval tasks (Sec. 2.3
Static Tacks
Dynamic Retrieval
Benchmark
Method
Compute (MACS)
ImageNNet
ImageNet disi, shift
Flickr301
Backward Trunsfer
ID Perfor
Forward Transfer
datasets
TiC-DataComp (M)
Cummlative Equal (Counts OS)
Soquenial
3.0 101a
3.0 101*
19.2
18.1
16.4
15.3
16.4
14.3
15.0
16.5
28.0
25.7
25.7
26.4
14.2
14,9
Cumulative
3.0 10
18.4
31.8
15.1
F RESULTS WITH NEW EVALUATION METRICS ON DYNAMIC TASKS
Recall, 7 represent the number of time steps for which we have data. For each training method, we generate total of T models, each corresponding the end of training at a particular time step. For each model and a dynamic evaluation task, we obtain T performance values. We represent these values using the performance matrix E, where each entry & signifies the performance of the model obtained after observing training data at time step i when evaluated on a dataset from time step j. Defining backward metrics as in Sec. Sec. involves averaging the entries in the upper and lower diagonal of our performance matrix ε, I.e., it was calculated as the average of time steps before Σizy Esg each training step (i.e., the lower triangular of E), i.e., (T(T-1))/2- This backward transfer metric has been used in prior works Lin et al. (2021). However, this approach inadvertently resulted in the backward transfer metric being influenced by later evaluation time steps resulting backward transfer performance numbers slightly larger than II) performance.
To address this issue, we've 'e revised our metric calculation method to metric as Díaz- Rodrigued et al. (2018). Now, we normalize the data in each row, which corresponds to evaluation time steps by subtracting the performance. This adjustment ensures more balanced and accurate representation across all training time steps. In particular, our updated forward and backward transfer metrics can be summarized as:
Backward transfer: Let B, denote the average performance on evaluation tasks before time i, then
we define backward transfer as average of B, across each training step, i.c., Σ Yiee, Eyy 1ky-6 E
T(T-1)/2
Forward transfer: Let Fi denote the average performance on evaluation tasks after time i, then define forward transfer as average of Fi across each training step, i.c., TT Li E,j Ea
7(7-1)/
35

---

Published as a conference paper at ICLR 2024
Table 20: Zero shot performance on our time-continual benchmarks. and ** denote methods that violate the compute budget. For dynamic tasks, we tabulate forward/backward transfer and ID performance on retrieval tasks with updated metrics as defined in App.
Dynamic Retrieval Tasks
Benchmark
Method
Compute (MACs)
Backward Transfer
ID Perfor-
Foeward Transfer
Relative Rackward Transfer
Relative Forward
Transfer
Restart
3.4 10
13.2
21.2
Sequentaal
34
f2.2
48.4
23.7
-9.5
-21.5
Prtching
3.4 18tt
44.7
63.4
24.3
-15.6
22.0
TIC-YFCC
Cumulative Exp
3.4 103s
60.4
60.1
27.1
-9.8
-23.0
Cumulative- E-quel
10ts
-10.3
Cumulative All
3.4
68.4
60.2
27.6
-4.1
-22.4
Luff
4.1 10ts
36.0
56.0
23.2
-27.4
-21.9
Cumulative-Al
3.6 101s
60.3
-3.9
-22.4
Oracle**
Restart
3.4 10
21.3
25.4
22.4
-4.5
2.7
Sequential
3.4 10
33.00
33.6
27.5
3.8
3.0
Panching
10ss
TiC-RedCapp
Cumulalve-Fsp
Cumulotive Equal
3.4 10
66.4
42.0
32.6
-3.0
-0.0
Cumulativo-Al All
3.4 10th
48.9
43.2
33.4
-0.6
3.5
Lup"
Cumulative All"
3.6
49.0
43.4
33
-1.0
3.5
Oracke
8.5 10
48.5
43.1
33.4
-1.0
-3.4
Sequential
3.0 101s
25.7
26.4
-4.7
Palching
10as
TIC-DataComp (M)
Cumulative Equal
Cumulative Exp
3.0 102s
3.0
31.7
31.8
27.1
26.5
15.2
15.1
0.3
0.9
7.0
7.6
Cumulativa-Al
101s
33.8
Lupa
-4.8
CumulaiiveAAI
3.9 19
36.7
28.3
15.6
3.0
7.3
Oracle*
1.2 10
34.9
27.8
15.6
2.5
7.7
Sequential
2.7 10
62.6
58.4
14.4
Paiching
27
55.2
67.5
13.9
TIC-DataComp (L)
Cumulativ- Exp
2.7 1919
60.4
58.4
41.4
-1.1
15.8
Cumulative Equall
2.7 1098
60.9
56.2
-0.3
13.8
Cumulative-All
1049
13.5
Cumulative All"
4.1
63.0
$7.8
41.2
2.1
13.5
Oracke*
1.1 1020
64.3
58
41.8
2.2
15.3
Sequential
27 1920
63.1
12.3
TrC-DataComp (XL.)
Cumulative
Cumulativ-All All"
3.5 10
71.0
68.6
57.1
2.5
11.7
36


Reviews:
[{"review_id": "nNL935WNCx", "reviewer": "Reviewer_wUu1", "summary": "The paper creates the first set of webscale Time-Continual (TiC) benchmarks for training vision-language models: TIC-DataComp, TIC-YFCC, and TIC-RedCaps with over 12.7B timestamped imagetext pairs spanning 9 years (2014–2022). And they use their benchmarks to curate various dynamic valuations to measure temporal robustness of existing models.", "questions": "The fundamental issue of continual learning is catastrophic forgetting. If we fine-tune a small number of parameters (e.g., prompt tuning) in the CLIP model, is catastrophic forgetting a major concern? On the other hand, if we fine-tune a large number of parameters, resource limitations may become a factor. Therefore, from this perspective, is it necessary to construct such benchmarks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper collects a large amount of dynamic data to study how to effectively train CLIP models continuously, ensuring the comprehensiveness of the research.\n\nIn order to ensure fairness in the evaluation, the paper has established a corresponding experimental protocol.", "weaknesses": "The dataset being solely focused on training CLIP may be somewhat limited. Can the article consider incorporating more vision-language models?\n\nThe YFCC100M dataset might be somewhat outdated in terms of the years it covers. It may be more representative to explore newer datasets for the research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9QbPMldLYj", "reviewer": "Reviewer_fC5h", "summary": "This work introduces new pretraining datasets and dynamic downstream tasks for web-scale continual learning using CLIP on natural distribution shifts.\n\nDatasets introduced by ordering metadata by time:\n- **TiC-DataComp**: 12.7 billion image-text pairs from datacomp-xlarge ordered by first seen snapshot (monthly increments available from 2014-2022), with 7 timesteps from (2014-2016) and every subsequent year.\n- **TiC-YFCC**: 15M image-text pairs on subset of YFCC100M which have captions and timestamps. Data from 2008–2014, with 4 timesteps involving (2008-2011) and every subsequent year.\n- **TiC-RedCaps**: 12M image-text curated from Reddit in 2011-2020 ordered by creation timestamps of posts, with 4 timesteps (2011-2017) and three subsequent years.\n\nEvaluation\n\n**Static Tasks**: Zero-shot evaluation on a set of 28 downstream tasks similar to (Radford et al 2021)\n\n**Dynamic Tasks**: On a small subset of samples reserved for testing at each timestep,\n- (i) T2I retrieval for samples in a given timestep\n- (ii) Classification on a LAIONNet-like 1000 class dataset (filtered by sentence embedding). Similar in nature to a scaled up CLEAR dataset. \n\n**Metric**: Use the checkpoint after every timestep and perform classification across different timesteps to evaluate In-domain Acc, Backward Transfer and Forward Transfer.\n\n**Primary Findings**:\n1) Continual training saves Cumulative* ~3x the cost for TiC-Datacomp compared to training-from-scratch (Oracle**), with a similar savings  shown for ImageNet IID-incremental.\n\n2) Comparison between Sequential and Cumulative-all on TiC-Datacomp: This highlights sequential has\n- significant catastrophic forgetting (low backward transfer performance)\n- similar forward transfer performance – indicating catastrophic forgetting does not impact generalization to new distributions\n\nThis entails:\n- poor performance static benchmarks, having poor backward transfer performance leads to a performance hit on static benchmarks which have old data (supported by other evidence: Fig 9&11)\n- Consequently adding a memory buffer seems to help static tasks.\n3) Patching helps a lot more than traditional continual learning methods like LwF when training without past data, indicating different continual learning approaches might help in these settings.\n\n4) Different models work better for static and dynamic evaluation tasks.", "questions": "Q1) What is the size of dynamic retrieval and LAIONNet test sets per chunk?\n\nMinor Comments: (Not considered for score)\n\nC1) Metric\n- The backward and forward transfer metric utilized here seems to be from (Lin et. al., 2021).\n- (Lopez-Paz & Ranzato, 2017) have a very different metric for backward and forward transfer, while (Díaz-Rodríguez et al., 2018) do not introduce any modification to this metric.\n\nOverall, I strongly agree with the motivations and findings from analysis (filtering, CLIP vs OpenCLIP) presented. I worry the continual learning experiments made poor design choices which might seriously impact most of the major findings in this work (W1, W3). If addressed, I will be very happy to increase my score. \n\nNote: I tried to detail why the ask, primarily to minimize superfluous asks which I find annoying from reviewers on my submissions (Did not mean to sound pretentious).\n\n[Edit]: Increased score from 3 to 8 and soundness from 1 to 4, as additional analysis resolve my pointed concerns.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "S1) **Tackles an important problem** [Critical]: This work correctly highlights the need to shift focus in continual learning and introduces time-evolving benchmarks for evaluating continual pretraining which turns out is quite important. I really liked the dynamic retrieval and the classification task design. Retrieval captures performance shifts in time by new concepts and distribution shifts, whereas classification task ablates the performance gap caused due to new things (e.g. covid) by choosing the same 1000 classes\n\nS2) **Insightful analysis** [Critical]: This work was a delight to read. I liked this work motivating a case where optimizing models for old datasets might lead to continuously worse performance on current-day tasks. Similarly, summary (4) indicates picking continual SSL training strategies based on downstream tasks might lead to poor design choices (best models worse than intended), shown in experiments on best-pool filtering.\n\nS3) **High-quality work** [Critical]: The contributions seem fairly clear, experiments investigate interesting related claims well and experimentation is quite extensive.", "weaknesses": "W1) **Sequential and cumulative models behave quite differently between TiC-YFCC15M and TiC-Datacomp** [Critical]\n- The paper nicely illustrates that YFCC15M has strong distribution shifts in Figure 15. \n- However, does TiC-DataComp have significant distribution shifts?\n    - The case for continually training CLIP primarily relies on Datacomp-like data having strong distribution shifts. \n    - I suspect the case there is far weaker than YFCC15M (I am worried it's too small to make this setting exciting). \n- Can the authors create a plot similar to Figure 15 for TiC-DataComp or some other mechanism to analyze distribution shift in TiC-Datacomp?\n\nW1.1) Further unexplained variations between Sequential and -All:\n- In TiC-YFCC15M and TiC-RedCaps, the performance of forward transfer is significantly affected, but that is not the case in TiC-Datacomp. \n- Similarly, the gap between backward transfer and ID retrieval is far smaller between them in TiC-Datacomp. Why is this the case?\n\nW1.2): Fig1 (mid) and Figure 10 present a different picture than Table 2.\n- Comparing backward transfer retrieval to ID performance for Cumulative exp (oracle) in Table 2\n    - Intuitively, ID performance should be higher (equal). \n- To what degree is the drop in forward transfer attributable to tasks getting harder vis-a-vis encountering new distributions.\n\nHypothesis: Past tasks are easier! Can the authors help shed some light on why the difference?\n\nW2) **Major results simply not present** [Critical]\n\nI cannot find any results for LAIONNet-like dynamic classification task. Did I miss something?\n\nOther missing ablations:\n- a) “Same maximum LR works best across all runs when using cosine schedule” – Didn't find supporting evidence.\n- b) “Warm up helps training on data from the first time step, but hurts on subsequent time steps.” – After correcting the minor shift issue, the table seems to robustly support the opposite conclusion. This seems surprising to me! I would have not expected warmup to help as the init is a very good one. Warmup mostly mitigates over-updating to high gradients when starting from a poor (random) init in my view.\n- c) “Given a fixed buffer size for each past step, we observe minimal to no difference between random subsampling and other strategies.” – Didn't find supporting evidence.\n\nW3) **Serious Design Flaws in Continual Learning Setup** [Critical]\n\nW3.1) *4-7 timesteps are too few*\n- I am worried the major findings would significantly change when tested on 20/50 timesteps but with the same computational cost per timestep, as the streams become far more non-i.i.d in that case. \n- Why? Because of memory buffers become far smaller, that it becomes very hard in my experience to bridge the gap between CL methods and -all. \n- Similarly, I am concerned there will be a far larger gap between -all and Oracle. \n\nThis critically affects some of the core findings presented in this work.\n\nW3.2) *Is memory constrained?*\n- For TiC-RedCaps and TiC-YFCC15M, all and exp/equal only differ at last timestep (Until timestep 3, they all can store all the data). This grossly undermines the comparisons as even at the last timestep, they still store 2/3rd of the data. It is very surprising that the difference in training for timestep causes such a big gap! If this is caused by D/3 data missing at the last timestep, the findings given 20-50 will probably be quite different!   \n- This issue is mitigated in TiC-DataComp as at last timestep, equal would store D/3 samples per timestep (2D/6D) which is indeed much  smaller! (Minor note: Am I correct that buffer conserved here is 2.3x smaller, not the claimed 3.5x smaller? -- The replay+current is 3D/7D). \n\nW3.3) *Replay buffers not utilized effectively, computation implicitly constrains memory.*\n- I worry most of the decline in performance is due to random sampling, and hence wanted to see supporting evidence for W2.c). I make my case below:\n- *Why?* This seems to be a case of imbalance data across timesteps due to the buffer constraints. \n   - If trained without oversampling less-represented data, I am not surprised to see a bias against replay data (i.e. poor backward transfer). \n- *Why not data loss constraining replayable sample?* I focus on TiC-Datacomp because there is a significant gap in replay buffers here\n   - In TiC-Datacomp-medium, 35k iterations with 4096 samples barely allows ~1 pass through the 128M samples.\n   - Hence, there seems to be a far stronger implicit memory constraint, as even after storing kD samples at timestep k, the network can only pass through 1/k fraction of samples from each past timestep at timestep k (training with oversampling). \n   - Given a fixed buffer size, retaining equal samples of past data still stores 2/(k-1) samples which is far higher than the limitation introduced by computational constraints. It seems possible to train on mostly unique samples, mimicking -all.\n\nHence, the gap in performance between -equal and -all should have been minimal (which seems like a serious design issue-- alleviated by far smaller buffers in the 20-50 timestep setup discussed above). However, the fact that gap still exists seems to compound the issue, and I worry it is due to poor sampling.\n\nW4) **Little discussion of past work despite similar findings/parallels** [Important]\n\nAlthough I fully admit the dominant setups in CL are quite synthetic and need improvement, as correctly pointed out by the authors.\n- (Cai et al., 2021) also has analysis covering several aspects of interest here, e.g. YFCC distribution shift plot (see their Fig 2), learning rate modulation, comparing impacts of replay buffer, web-scale training and similar compute constraints as here but on 600K timesteps. This avoids promoting CL methods which work only on a short timescale and degrade quickly when scaled beyond 5-20 timesteps (i.e. my worry about the 4-7 timesteps). \n- Finding (1) on Imagenet IID seems out-of-place here, and has detailed precisely in several works, e.g. “Computationally Budgeted Continual Learning: What Does Matter?”, CVPR23. Other works like “One Pass ImageNet” NeurIPS-W 2021 have analysis on Sequential/Cumulative variants as discussed here on Imagenet but these papers are not discussed.\n- Other works which have interesting investigations with computational limits (Bornschein et al., 2022) and (Jang et al., 2022) have nice analysis quite relevant to this work. E.g. hyperparamter optimization as a cost in the compute budget C (will be required when training in the unknown)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TjiRBlTTIy", "reviewer": "Reviewer_RTur", "summary": "This paper proposes a benchmark for continual learning using large pretrained vision-language models, e.g., CLIP. The authors construct multiple datasets with timestamp information to evaluate the performance of existing methods on time-continuous data. The experimental results show that the rehearsal-based approach can reduce computation while achieving similar performance as the oracle algorithms.", "questions": "Please refer to questions in weakness.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors construct multiple datasets with time information based on existing datasets for continual learning settings. This is an non-trivial contribution for continue learning to evaluate the effectiveness of algorithms when facing natural distribution shifts.\n2. This paper is well-written and easy-to-follow.", "weaknesses": "1. This benchmark lacks various types of continual learning methods [1]: elastic weights consolidation methods, progressive neural network methods, dynamic architecture methods, etc. Therefore, the experiments of this benchmark is relative weak and insufficient. \n2. This paper lacks some in-depth analysis of vision-language models solving continual learning. Vision-language models enable various novel model tuning paradigms, such as prompt tuning, vision prompt tuning, parameter-efficient tuning, etc. If these aspects are ignored when discussing the solving of continual learning with the CLIP model, then this benchmark is a bit over-claimed.\n\n**References:**\n\n[1] Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, De-Chuan Zhan: PyCIL: a Python toolbox for classincremental learning. Sci. China Inf. Sci. 66(9) (2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DyeQ1Ui6lQ", "reviewer": "Reviewer_R93w", "summary": "This paper presents a study on training CLIP with time-evolving data in an incremental manner. The authors timestamp the training data spanning 2014-2022, treating each timestamp as a distinct incremental learning step. Their analysis delves into the backward and forward compatibility of CLIP as it undergoes training, leading to some findings.\n\nFirstly, the research demonstrates that employing simple small replay techniques effectively mitigate forgetfulness, an insight in the context of continual learning. Secondly, the study reveals an unsurprising yet noteworthy performance gap: CLIP underperforms on future, unseen data.\n\n------------------------- After Rebuttal ----------------------------------------------------------------------------------------------\n\nDear authors, \n\nI did not imply linear probing, but more downstream task adaptation, since CLIP has been applied to almost all computer vision tasks by now. \n\nI went through all the reviews and the rebuttal. I was optimistic about this paper prior to rebuttal. My optimism remains.", "questions": "No questions so far.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1:\nThis paper addresses a critical and, to the best of my knowledge, an open issue in continual foundation model training. The challenge lies in the impracticality of re-training large foundation models like ChatGPT or CLIP, highlighting the necessity for continual learning solutions. Despite this urgency, there's a notable absence of organized datasets for the community to tackle this problem. While the CLEAR benchmark exists, it has not gained traction and lacks essential components like Web captions. This benchmark, being a captioned version of CLEAR that evolves over time, fills a gap, offering a valuable resource for researchers in the continual learning domain.\n\nS2:\nThe authors conduct insightful analyses using fundamental regularization and replay techniques. Unsurprisingly, the results show that preserving and replaying a portion of the data during incremental learning effectively mitigates forgetfulness. This empirical validation underscores the importance of such techniques in preserving model performance over time, providing a practical and valuable contribution to the field.\n\nS3:\nThe paper's central discussion on the fine-tuning cost in terms of compute MAC carries some value. This perspective equips researchers with a framework to explore low MAC solutions, emphasizing a nuanced approach rather than blindly minimizing the forget ratio.", "weaknesses": "W1: No method, only benchmarking\n\nThe paper is commendable in its focus as a benchmark study, emphasizing data and existing baselines without introducing new methodologies. The utilization of well-known and straightforward baselines is executed competently. However, the paper could have significantly bolstered its strength by incorporating benchmarks from related works such as Continual-CLIP [1] and the methods outlined in \"Robust fine-tuning of zero-shot models\" [2]. Including these comparisons would have provided a more comprehensive evaluation, highlighting the benchmark's effectiveness in contrast to existing state-of-the-art approaches.\n\nW2: No fine-tuning, only pre-training\n\nAn additional area of improvement lies in the paper's scope, particularly concerning the fine-tuning stage. While the study adeptly focuses on the continual pre-training phase, it would have been more insightful and impactful to extend the analysis to the fine-tuning stage. Specifically, investigating how well the obtained checkpoint at a given time transfers to standard vision benchmarks, beyond simple retrieval tasks, would have provided a more nuanced understanding of the model's performance. This consideration is essential for gauging the real-world applicability and adaptability of the continual learning approach outlined in the paper.\n\n\n[1] Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models\n\n[2] Robust fine-tuning of zero-shot models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)