Dynamic Historical Adaptation for Continual Image-Text Modeling
Authors: Yutian Luo
Yizhao Gao
Zhiwu Lu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Image-text modeling
continual learning
contrastive learning
cross-modal retrieval

Abstract: In realistic application scenarios, existing methods for image-text modeling have limitations in dealing with data stream: training on all data needs too much computation/storage resources, and even the full access to previous data is invalid. In this work, we thus propose a new continual image-text modeling (CITM) setting that requires a model to be trained sequentially on a number of diverse image-text datasets. Although recent continual learning methods can be directly applied to the CITM setting, most of them only consider reusing part of previous data or aligning the output distributions of previous and new models, which is a partial or indirect way to acquire the old knowledge. In contrast, we propose a novel dynamic historical adaptation (DHA) method which can holistically and directly review the old knowledge from a historical model. Concretely, the historical model transfers its total parameters to the main/current model to utilize the holistic old knowledge. In turn, the main model dynamically transfers its parameters to the historical model at every five training steps to ensure that the knowledge gap between them is not too large. Extensive experiments show that our DHA outperforms other representative/latest continual learning methods under the CITM setting.

Under review as a conference paper at ICLR 2023
DYNAMIC HISTORICAL ADAPTATION FOR CONTINUAL MMAGE-TEXT MODELING
Anonymous authors Paper under double-blind review
ABSTRACT
In realistic application scenarios, existing methods for image-text modeling have limitations in dealing with data stream: training on all data needs too much com- putation/storage resources, and even the full access to previous data invalid. this work, we thus propose new continual image-text modeling (CITM) setting that requires model to trained sequentially on number of diverse image-text datasets. A Although recent continual learning methods can be directly applied to the CITM setting, most of them only consider reusing part of previous data or aligning the output distributions of previous and new models, which is partial or indirect way to acquire the old knowledge. In contrast, we propose a novel dynamic histor- ical adaptation (DHA) method which holistically and directly review the old knowledge from a historical model. Concretely, the historical model transfers its total parameters the main/current model to utilize the holistic old knowledge. In turn, the main model dynamically transfers its parameters to the historical model at every five training steps to ensure that the knowledge gap between them is not too large. Extensive experiments show that our DHA outperforms other represen- tative/latest continual learning methods under the CTTM setting.
INTRODUCTION
In the past few years, image-text modeling has drawn much attention from both academia and indus- try with a fundamental role in various cross-modal tasks, such as image-text retrieval (Chen et al., 2020a; Lee al., 2018), image captioning (Vinyals et al., 2015; Jia al., 2015). and text-image generation (Johnson et al., 2018; Qiao et al., 2019). Although existing image-text modeling meth- ods (Lu et al., 2019; Li et al., 2020; Lei et al., 2021: Yang et al., 2021; Ging et al., 2020; Bain al., 2021; Huo et al., 2021; Jia et al., 2021) have achieved great success in these tasks, most of them assume that full (fixed) set of image-text pairs are provided for model training. which actually limits their deployment in realistic application scenarios. That is, the training data often comes in a stream way, and the current widely-used paradigm for image-text modeling faces two limitations: (1) training all data (i.e.. both previous and new data) severely increases the computational and storage overhead; (2) the full access to previous data may be invalid.
To overcome these limitations, thus propose continual image-text modeling (CITM) setting instead. Concretely, recollect four diverse image-text datasets respectively from MSCOCO (Lin et al., 2014), CC3M (Sharma et al., 2018), WTT (Srinivasan et al., 2021) and GoodNews (Biten et al., 2019), each of which is split into the training, validation, and test sets. We adopt the SimCLR-based model (Chen et al., 2020b) as the basic model which is also deployed in OpenAl CLIP (Radford et al., 2021). Under the CITM setting, the model is sequentially trained on each of the four image-text datasets, and is finally evaluated on all datasets. To demonstrate the well-known catastrophic forgetting problem, we measure the image-to-tex retrieval performance with the metric recall @ (R@) @ 1) during sequential training on the four datasets. The results in Figure clearly show that every time the model is trained on a new dataset, its performance on previous datasets has distinct degradation (i.e., catastrophic forgetting).
MecoCO test
GDmesditto Rest
CCIM lest
HAT Aest
MSCDCO-train
28.66
(13M-train
16
WIT train
8.48
7.22
8.94
Glnewss-trin 5.88
Figure 1: The results of catastrophic forgetting under the CITM setting.

---

Under review as a conference paper at ICLR 2023
Dataset 1 (2020)
Dataset (2021)
Dataset 3 (2022)
(20 Delevet2 (ACZE)
(2020)
(2027)
QTM
CIMM
Pretrained
Protramed
Prstranne
Procrinee Mocil 2000
Pratrarnl Model 2021
Medel J0z0
Model 2023
Model 2010
Model 2022
Figure 2: Schematic illustration of the realistic application of our proposed CITM setting in large- scale multi-modal pre-training (like OpenAl CLIP) with the pre-training data being updated every year. Left: The traditional setting for large-scale pre-training with annual data update. Right: Our CITM setting for large-scale pre-training with annual data update.
Among existing continual learning methods, cchearsal-based (Chaudhry et al., 2019; Buzzega et al., 2020) and regularization-bate based methods (Li & Hoiem, 2017: Rannen al., 2017; Zhang et al., 2020; Cha et al., 2021) can be easily applied to the CITM setting, while archifecture-related meth- ods (Mallya & Lazebnik, 2018; Mallya et al., 2018; Rosenfeld & Tsotsos, 2018) generally need extra task-specific modules and are unsuitable for CITM with a unified architecture. In this paper, we thus devise baseline methods for CITM mainly by deploying rehearsal-basee and regularization- based methods. Note that these two groups of continual learning methods have their own limitations. Specifically, rehearsal- -ased methods set up a memory buffer to replay previous data, and only pre- serve partial old knowledge due to the sample selection imposed on the memory buffer. Moreover, regularized-baste beaieedd-ddddddddee methods can only convey the old knowledge by aligning the output distributions of the previous and new models, which indicates that the old knowledge from the previous model can only be indirectly transferred through data-driven guidance. Such an indirect approach is thus vulnerable to large domain shifts across the previous and new tasks.
To avoid the drawbacks of the above baseline methods for CITM, we thus propose a novel dynamic historical adaptation (DHA) method which can holistically and directly review the old knowledge from a historical model. The core idea of our DHA is to directly transfer knowledge between the old and new models through parameter interaction. In our DHA, we name the model trained the current task as the main model, and the best (main) model on the last task as the historical model. During parameter interaction, we directly transfer the parameters of the historical model to the main model and then train the main model with modified parameters on the current task. Meanwhile, we dynamically update the historical model with the guidance of the main model to ensure that the knowledge gap between them is not too large. Specifically, at every five steps, the parameters of the main model are passed to the historical model for parameter modification. Overall, these two parameter transfer strategies make up our DHA method. Compared with existing methods (Li & Hoiem, 2017: Chaudhry et al., 2019; Buzzega et al., 2020; Cha et al., 2021), our DHA has two advantages: (1) DHA adopts direct parameter transfer instead of indirect model aligning (deployed by regularization based methods). and thus it is more robust to large domain shifts across the previous and new tasks. (2) DHA holistically reviews the old knowledge from the historical methods for partial data selection
model, which can overcome the drawback of rehearsal-basee (i.e., partial old knowledge is reused). To our best knowledge, we are the first to propose direct parameter transfer method to cope with the forgetting problem in the continual learning field.
As we have mentioned, we construct a benchmark dataset for the CITM setting by recollecting four diverse image-text datasets respectively from MSCOCO (Lin et al., 2014), CC3M (Sharma al., 2018), WIT (Srinivasan et al., 2021) and GoodNews (Biten et al., 2019). Under fair setting, we compare DHA with number of baseline methods (Li & Hoiem, 2017; Chaudhry et al., 2019; Buzzega al., 2020; Cha al., 2021) on this benchmark dataset. Extensive experiments prove that our DHA outperforms these baseline methods under the CITM setting.
Overall, the main contributions of this paper can be summarized as follows: (1) We propose new continual image-text modeling (CTTM) setting for image-text modeling on data stream, which has a realistic application in large-scale multi-modal pre-training (with annual data update) as shown in Figure 2. (2) We devise a novel dynamic historical adaptation (DHA) method under the CITM setting. For the first time, we identify the important role of direct parameter transfer (between the historical and main models) in continual learning. (3) We construct benchmark dataset of four di- verse sets of image-text pairs, which can facilitate the research on CITM. (4) Extensive experiments demonstrate the effectiveness of our DHA under the CTTM setting.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Image-Text Modeling, Recent image-text modeling methods can be summarized into two groups: single-stream and two-stream methods. (1) Single-stream methods aim to learn the unified repre- sentation of the image-text pair with a fusion module. Most of existing single-stream methods (Lu et al., 2019; Tan & Bansal, 2019; Zhu & Yang, 2020; Li et al., 2020; Lei et al., 2021; Yang et al., 2021) choose to concatenate the image and text embeddings as the input of the fusion module (e.g.. cross-attention transformer). Although model training is easy for single-stream methods, it requires calculating the similarities of all the possible query-candidate pairs during inference. Therefore, they suffer from heavy computation burdens. (2) Two-stream methods (Ging et al., 2020; Patrick et al., 2020; Bain et al., 2021; Huo et al., 2021; Jia et al., 2021; Radford et al., 2021) adopt indepen- dent image and text encoders to learn image and text embeddings that aligned in a joint space. Compared to single-stream methods, two-stream methods allow different depths and designs of net- work architectures for the two modalities and enjoy much more efficient inference. In this work, we follow the two-stream architecture for image-text modeling: ResNet50 (He et al., 2016) 6) is used as the image encoder, and BERT-base (Devlin et al., 2018) is used the text encoder. We adopt SimCLR (Chen et al., 2020b) as the basic contrastive learning method for model training.
Continual Learning. By reviewing recent progress in conventional continual learning, we can divide main-stream approaches into three groups: (1) Rehearsal-Baar
Methods. Early classic rehearsal-baat based method (Rebuffi et al., 2017) proposes to store part of exemplars of previous classes in order to acquire better class means. (Chaudhry et al., 2019) finds that retraining a subset of old data on new tasks can help address the forgetting problem and also provides several memory update strategies. (Aljundi et al., 2019; Chaudhry et al., 2021; Buzzega et al., 2020) further explore the approaches to selecting representative samples from old tasks. In addition, pseudo-data rehearsal generating approaches (Shin et al., 2017; Atkinson et al., 2018; Lavda et al., 2018: 88; Liu et al., 2020; Ramapuram et al., 2020) are proposed to avoid extra storage and generate more representative sam- ples for training, whereas generating pseudo-data actually increases the training time. Note that the iidearsal-basee methods suffer from the drawback that only partial historical knowledge is trans- ferred by the memory buffer. (2) Regularization-Ball -Based Methods. This group of methods mainly aim to distill the knowledge of the previous models. (Li & Hoiem, 2017; Rannen et al., 2017; Zhang al., 2020) align the output features or logits between the previous and the current models with an extra regularization penalty. Since the domain shifts exist across different tasks, such regularization penalty brings additional training difficulty (De Lange et al., 2021). Other methods (Aljundi al., 2018; Chaudhry et al., 2018; Kirkpatrick et al., 2017) constrain part of the parameters of the model. Since most of these methods are designed for classification tasks, they are hard to be directly applied to our CITM setting. (3) Architecture-Rehlle
Methods. This group of methods mitigate the dif- ference in new tasks in two ways. (Mallya & Lazebnik, 2018; Mallya et al., 2018: Serra et al., 2018) mask different parameters while training different tasks. (Aljundi et al., 2017; Rosenfeld & Tsotsos, 2018; Xu & Zhu, 2018) extend network architecture for new tasks. A potential drawback of these methods is that they generally need extra task task-specific modules and are unsuitable for CITM with unified architecture. Other than the above approaches with a single strategy, recent works (Buzzega et al., 2020; Cha et al., 2021) start design combined strategies for continual learning based on rehearsal-basee and regularization-be bbased methods. Finally, we notice that most of existing continual learning approaches have a common characteristic that the old knowledge is expressed with (partial) data, which means that the model update to mitigate forgetting may be affected by partial/indirect guidance. In contrast, our proposed DHA provides a new perspective of continual learning that the old knowledge could be holistically preserved by direct parameter transfer.
3 PROPOSED METHOD
3.1 PRELIMINARY
We first define our proposed CITM setting formally. Given a sequence of n image-text datasets D (Di, (D,,D2,.. ..., Dy.,., ) coming from n domain sources like stream, a model for CITM is supposed to be sequentially trained on D. Each dataset D, (1 < S n) is defined as D, {(+}, xz. ) where 2! and x;"" respectively denote the image and text samples in the i-th image-text pair, and N₂ denotes the number of data pairs. The image-text retrieval task (Chen et al., 2020a; Lee et al.,
2018) (018)0om each dataset D. is denoted as T2 (1 S L S ni). For each task Tr, model for image-text
3

---

Under review as a conference paper at ICLR 2023
retrieval typically learns align the image and text embeddings with contrastive loss Chen et al. (2020b). Under the CITM setting, the model only concentrates on the current task during sequential training. leading to the catastrophic forgetting of previous knowledge. For performance evaluation, the obtained final model (trained across all tasks) tested on each of the n tasks.
3.2 NETWORK ARCHITECTURE
Under the CITM setting, we propose novel dynamic historical adaptation (DHA) method which can holistically and directly review the old knowledge from a historical model. The core idea of our DHA is to directly transfer knowledge between the old and new models through parameter interaction. To this end, DHA model is devised to have two key components: the historical model and the main model, as illustrated in Figure 3. These two models share the same architecture while only the main model requires the backward update. We follow the two-stream network architecture like CLIP (Radford et al., 2021), which has achieved remarkable performance in image-text retrieval tasks. Concretely, the image encoder takes ResNet50 as the backbone and the text encoder takes BERT-Base as the backbone, which are both initialized with unimodal pre- pre-trained models.
Image and Text Encoders. Formally, the backbone ResNet50 of the image encoder denoted as fl ResNNer- Mcanwhile, the backbone BERT-Base of the text encoder is denoted as B Given an input text oi we first tokenize it into a sequence [t tk? ...,, th:"), where l; denotes the length of x². To ensure that the text and image embeddings have the same dimension, we append linear projection layers f/2 and fo to ResNet50 and BERT-Base, respectively. Given an image-text pair (1/, xff) the final image and text embeddings are given by:
of = =F(RReNN((d)))
(1)
of = T- Uh? Uk2 (..,,(kk)))) ))-
(2)
Contrastive Loss Function. Since our proposed DHA has the two-stream architecture, it can be effectively trained by the well-known contrastive learning method SimCLR (Chen et al., 2020b). Concretely, given a batch of B image-text pairs {a/.=; constructed follows. For each input image a!{ define the contrastive loss between its image embedding of and the embeddings of all positive/negative texts in the batch as InfoNCE loss:
x ) Ji=1 during training, the loss function
explet ee)
1 = B
Σ i=1
log
exple! of /T) +++Eepple .xppec--e517) s41
where T denotes the temperature hyperparametes and the vector similarity measured by dot prod- uct (-). Similarly, for each input text of. the InfoNCE loss is given by:
B
explet eT/Tr
L
Σ
log
(4)
B
exple
ef/r)+ E -
1=1
The total contrastive loss for training our DHA is thus defined as:
L = 1.02 + 12:
(5)
In this work, for fair comparison, all the competitors for CITM adopt the same network architecture and the same basic contrastive loss function as our DHA. More details can be found Sec. 4.
3.3 DYNAMIC HISTORICAL ADAPTATION
As we have mentioned, our motivation of method design is transfer the holistic knowledge con- tained in the historical model to the new model, without suffering from the drawbacks of exist- ing continual learning methods. Concretely, data rehearsal approaches (Chaudhry et al., 2019; Buzzega et al., 2020) attempt to preserve the previous data distribution, but retaining memory buffer of limited size may cause the overfitting to the partial samples the previous task. More- over, egularization-base
approaches (Li & Hoiem, 2017; Rannen et al., 2017: Zhang et al., 2020; Cha et al., 2021) attempt to store the historical knowledge by aligning the historical and main mod- els with regularization-base penalty terms, but such indirect way addressing the forgetting
4

---

Under review as a conference paper at ICLR 2023
historical model
historical model
main model
soot
BP speba
made mode
maie mosel
main model
man
Irput
f+2-2
24
steps Interval to update histonical model
stops interval update testorical model
Adactation of main Nistorical modal
Dynamic histarical
Figure 3: Overview of the proposed DHA method for the CITM setting. At the beginning of task Tú, the best main model found in task T (according the validation performance task Tz-1) is used to initialize both the historical and main models. During each training iteration, the main model first receives the transferred parameters from the historical model and then learns the dataset of task T,. Moreover, for every k training iterations, the historical model is updated with the transferred parameters of the main model.
problem is thus vulnerable large domain shifts across the previous and new tasks. In this paper, we thus propose novel dynamic historical adaptation (DHA) method which can holistically and directly review the old knowledge from a historical model. Below we introduce the details of the two update strategies applied in our DHA throughout training.
Since all the learned knowledge has been held and expressed by model parameters, we believe that directly transferring the parameters of the historical model to the main model is a direct and effective approach to preserving the historical knowledge. The direct parameter transfer process is shown in Figure 3. Formally, let 0μ, 6M. 8%r denote the parameters the historical model, the main model, and the best main model found in the last task, respectively. Moreover, let ff and 0% denote the parameters of the historical model and the main model at the end of the i-th training iteration in the current task, respectively. At the beginning of the current task, we initialize the main model and the historical model with the parameters of the best main model found in the last task (i.e., 0% 0 and 0% 01,). For each training iteration (i 2 1) before data load, we choose to update 6% with part of 0 and obtain g'pre as the new intermediate parameters of the main model. After such parameter update, the main model is trained on the input data and backward updated normally obtain 0h as the final parameters of the i-th training iteration. We define the gradient function w.r.t. 0 as:
Adaptation Main Model with Historical Model:
dL.
GL.(ON) = 00M 164=
(6)
The above adaptation strategy for the main model with the historical model can be formulated as:
Diore = 210 + 11
(7)
8/ = 0'mm L. (8'r))
(8)
where v denotes the learning rate, and A denotes the weighting coefficient. By combining Eq. (7) and Eq. (8), we have the adaptation process from 0:-1 fh as follows:
0y = 1 + - A1) GLL + +1 (1- 11)
Dynamic Update of Historical Model: Currently, the main model has received the guidance from the historical model. However, since the parameters of the historical model remain static in the current task, this may cause two concerns: (1) Since the main model always learns better on the current task the training process goes on, the knowledge gap between the historical and main
5

---

Under the CITM setting, we train our DHA model on a sequence of four datasets: MSCOCO (Task Ti), CC3M (Task T2), WIT (Task T3), and GoodNews (Task Tạ). After the main model has com- pleted its training on task T.-1, we find the best main model on the validation set T.-1- At the Under review as a conference paper at ICLR 2023
models is gradually enlarged. Therefore, the parameters transferred from the unchanged historical model tend to cause degradation to the retrieval performance of the main model on the current task. (2) Such performance degradation to the main model the current task would finally affect the performance of the final model (i.e. the best main model across all tasks) when it is evaluated on this task. To address these concerns, we choose make the parameters of the historical model gradually change by updating it with the parameters of the main model (but not so frequently). This dynamic update of the historical model at the i-th training iteration is given by:
[220 + (1 A2) ( H
if = mk. m E EN
0/ =
(10)
otherwise
where & denotes the step interval for model update, and X2 denotes the weighting coefficient.
Overall, our proposed DHA is composed of the above two update strategies, which have been shown to be effective in Sec. 4.3 and Sec. 4.4. We believe that direct parameter transfer is another promising way to handling the continual learning problem in image-text modeling. The pseudocode of the full algorithm for proposed DHA is given in Appendix A.
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Datasets. To mimic the realistic application of the CITM setting in multi-modal per-training (like OpenAl CLIP), we recollect four image-text datasets for benchmark construction from the following large diverse datasets of image-text pairs: (1) MSCOCO (Lin et al., 2014) is an image-text dataset that consists of 123, 287 images with their captions. Each image is annotated with 5 captions. Most images are related to the nature and common objects in daily life. (2) CC3M (Sharma et al., 2018) is a well-known image-capiee capeennaptioning. dataset for image-text pre- pre-training. It is composed of about 3M image-text pairs, which are collected from the Internet with weak relation between images and their textual descriptions. (3) WIT (Srinivasan et al., 2021) is a large multimodal multilingual dataset collected from the Wikipedia website. This dataset has a total of 11.5M images. Each image an- notated with the corresponding textual description or contextual information. (4) GoodNews (Biten et al., 2019) is large news amage-captioning dataset. It is collected from the New York Times. Un- like the other datasets, the captions in GoodNews are written by professional journalists and thus claimed to have implications for the style and richness of the news. In this paper, based on the afore- mentioned four image-text datasets, our benchmark dataset of four sequential tasks constructed as follows: (1) For task Ti, we randomly select 100, 000 images with corresponding captions from MSCOCO the training set, 13, 287 as the validation set, and 5, 000 as the test set. (2) For the other tasks Th - T1. we construct the three task-specific datasets from CC3M, WIT, and GoodNews, respectively. Concretely, the daining/validation/// image-text pairs uniformly for each of T2 T4.
set is formed to have 130, 000/13, 000/5, 000
Evaluation Metries. We adopt Recall @ mean (R @ mean) and Forgetting Rate (FR) as our evalu- ation metrics. @ mean indicates the mean value of Recall 1, Recall @5, and Recall @ 10, where Recall @ K (K=1,5,10) denotes the percentage of correct matching in the top-K retrieved results. The R mean on each task indicates the retrieval performance of the final model on this task. More- R
over, for the final model tested on task Te, FR is defined as FR, =
where R, denotes the
R:
R @ mean of the best main model in task T, on the test set of task Tr, and R. denotes the performance
of the final model on the test set of task T,. The average FR is FR 1
n- R R
x-
4.2 IMPLEMENTATION DETAILS
beginning of task Tú, this best main model is used to initialize both the main and history models on this new task. For fair comparison, we set the memory buffer to have 5% samples of the train set of each task for our DHA (if buffer is used) and all competitors. The details of the memory buffer updating strategy are included in Appendix A. To make comprehensive study, we implement our DHA with and without memory buffer to validate its effectiveness under the CITM setting.

---

Under review as a conference paper at ICLR 2023
Table 1: Comparative results between our DHA and other representative/latest methods. 'T2I' de- notes text-to-image retrieval and 'I2T' denotes image-to-text retrieval. All methods adopt the same network architecture. 'Mem' denotes the data rehearsal with 5% buffer.
Task Ti
Task T2
Task T3
Task T4
Average
Method
Mem
R@mean
PR
R & mean
FR
R mean
PR
R@mean
R@ mean
FR
Baseline
13.64
68.30
12.78
64,82
13.63
40.25
22.62
15.67
$7.79
LwF (Li & Hoiem, 2017)
16.81
60.94
15.69
56.31
15.33
31.41
22.68
17.63
49.55
ER (Chaudhry et al., 201.9)
16.30
62.12
16.15
55.08
14.37
36.42
21.57
17.10
51.21
T21
DER (Ruzzega et al., 2020)
20.52
52.31
20.92
41.74
16.31
24.07
21.04
19.70
39.37
CO2L (Cha et 2021)
19.64
54.35
18.95
46.14
16.13
24.94
22.95
19:42
41.81
DHAT (ours)
21.31
50.48
21.82
37.15
15.64
27.09
21.37
20.04
38.24
DHA (ours)
24.58
42.88
22,95
33.82
16.15
24.50
21.22
21.29
33.73
Baseline
17.26
66.48
11.55
68.60
13.48
42.02
23.72
16.50
59.03
LwF (Li & Hoiem, 2017)
21.59
58.07
15.36
57.58
15.42
34.69
23.29
18.92
50.11
ER (Chaudhry et al., 2019)
21.23
58.57
15.17
57.79
14.79
37.83
22.03
18.31
51.40
12T
DER (Buzzega et al., 2020)
Y
27.55
46.49
19.08
47.31
16.34
26.43
21.99
21.24
40.08
CO21, (Cha et al., 2021)
26.23
49.06
17.09
52.18
16.33
27.67
23.59
20.81
42.97
DHA (ours)
27.91
45.80
17.69
50.28
15.60
30.11
22.02
20.81
42.06
DHA (ours)
32.72
36.45
21.01
39.50
16.45
24.71
22.25
23.11
33.55
We adopt BERT-Bas (Devlin et al., 201 8)ResNetO0 (He et al., 2016) as the backbone of text/image encoder. They both use corresponding unimodal pre-trained models for initialization. The images are resized to 224x224 pixels, and the max length of the text descriptions set to 256 (tokens). We set the learning rate the beginning of each task to 5e-5 5 and multiply it by 0.1 as the validation loss does not decrease. We adopt the optimizer Adam for gradient propagation, with the weight decay 55 The batch size set to 320 for each training iteration. \1, A2, and k are empirically selected as 0.995, 0.985 and 5, respectively. The main model is trained for 15 epochs on the training set of each task. The total training time four datasets is around 12 hours with 8 Tesla V100 GPUs. The dataset and code will be released soon.
4.3 MAIN RESULTS
We compare our DHA with other representative/lates methods, including the classic regularized- based method LwF (Li & Hoiem, 2017), the classic rehearsal-based method ER (Chaudhry et al., 2019), and two fusion methods DER (Buzzega et al., 2020) and CO2L (Cha al., 2021) which combine the regularized-brat competitors are included in Appendix B). The basic method (denoted as 'Baseline") denotes training the same network sequentially on four tasks but without any continual learning strategy. The com- parative results in Table (see more results in Appendix D) show that: (1) Our DHA beats all the competitors according to average R @ mean and average FR over all tasks. The margins between our DHA and all the competitors are especially significant on average FR. This suggests that our direct parameter transfer strategy used for designing DHA is indeed effective for the CITM setting. (2) Our DHA outperforms the second best method DER by 1.59% 1.87% on average R Reman and 5.64% -6.3% average FR. This further validates the effectiveness of our direct parameter trans- fer strategy used for designing DHA. (3) Our DHA (without memory buffer) achieves better results than most of the other approaches. When the rehearsal-based strategy fused, our DHA achieves the state-of-the-ar results. That is, our DHA provides a new promising approach to continual image- text modeling. (4) On the most previous tasks (e.g., Ti and T2), our DHA performs significantly better than all the competitors in preserving much earlier knowledge. This superior ability would make a greater difference in realistic applications when there are more tasks in the data stream. (5) On the newest task T4, nearly all the methods cause a drop on @@ mean as compared to 'Baseline.. Such performance drop is mainly due to the trade-off between preserving previous knowledge and learning the current task, which is common practice in continual learning scenarios.
and rehearsal-based strategies (the implementation details of these
To show more detailed performance all methods in alleviating forgetting, we provide the results of the main model (of all methods) on task Ti during sequential training on the four tasks in Fig- ure 4 (more results on task T2 and task T3 are shown in Appendix
D). It actually shows the change tendency of R @ mean task Tj of the main model when it is being trained on the later tasks sequen- tially. Specifically, the left sub- figure show the text-to-image retrieval performance on task T, when the main model is trained from task T, to task T4, while the right sub-figure show the corresponding
7

---

Under review as a conference paper at ICLR 2023
T21: Recall@mean of the main model on Task T
12T: Recall@mean of the main model on Task T,
50
60
40
50
R-gmean
R@meaa
30
Beseline
Baseline
LWF
LwF
20
ER
FR
DER
DER
COZL
20
10
DHAT
DHA
10
THA
3
Task-ID
Task-ID
Figure 4: Illustration of the results of the main model (of all methods) on Task Ti during sequential training on the four tasks. 'T2I' denotes text-to-iman -m-age retrieval and 'I2T" denotes image-to-texs retrieval. It can be clearly seen that our DHA forgets with the lowest speed.
Table 2: Direct retrieval results on the test set Flickr30K obtained by our DHA and other rep- resentative methods. Note that all methods are trained on the sequence of four image-text datasets (i.e., (MSCOCO, CC3M, WIT, GoodNews)) under the CITM setting.
T21
12T
Method
R@1
R@5
R
Rurmean
R@
R@5
Recall 10
R&mcan
Baseline
11.04
29.70
40.42
27.05
16.30
36.60
47.00
33,30
ER (Chundhry 2019)
12.46
32.74
44.30
29.83
17.90
40.80
51.30
36.67
DER (Buzagga at. 2009)
16.52
39.28
51.46
35.93
22.70
47.60
59.40
43.23
DHA! (ours)
16.02
36,58
47.52
33.52
21.50
47.10
58.80
42.47
DHA (ours)
17.76
41.30
54.22
37.76
24.00
48.90
63.10
45.33
image-to-text performance. It can be clearly seen that: (I) Our DHA helps the main model forget with the slowest speed during sequential training among all the methods under the CITM setting. (2) Even without memory data, the forgetting speed of our DHA is still slower than that of most of the other competitors. Overall, these observations provide further evidence that direct parameter transfer strategy (used in DHA) is indeed effective in alleviating forgetting, and our DHA can be deployed
promising approach continual image-text modeling.
Additionally, we conduct direct retrieval experiments on the test set of Flickr30K (Young et al., 2014), which has overlap with the sequence of four image-text datasets (i.e., (MSCOCO, CC3M, WIT, GoodNews)) under the CITM setting. We compare our DHA with Baseline, ER, and the best competitor DER, which are all sequentially trained on the four datasets. The comparative results in Table show that our DHA achieves the best performance, i.e., our DHA has the strongest generalization ability due to the direct parameter transfer strategy used for alleviating forgetting.
4.4 ABLATION STUDY
Our proposed DHA is composed of two main strategies: (1) adaptation of the main model with the historical model (shortened 'Adapt with Hist'), i.e., the main model keeps reviewing the historical knowledge by receiving the parameters of the historical model; (2) dynamic update of the historical model (shortened as 'Dynamic Hist'), i.e., the historical model is renewed by updating its parameters with the parameters of the main model. To clearly show the influence of each strategy on the model performance and also study the effect of the memory buffer, we provide the ablation study results for our full DHA on image-to-text retrieval in Table 3. We only show the results (R @ mean) of the final model (trained across all four tasks) on each task under the CITM setting. We can observe that: (1) The most basic method with no DHA strategies and no memory buffer has the lowest performance on average @
mean. (2) Only adopting the strategy of 'Adapt with Hist" yields a 3.85% improvement on average R mean, showing that it can well retain the knowledge of the previous tasks. (3) Adopting both 'Adapt with Hist' and 'Dynamic Hist' strategies brings further improvements average R @ mean.. Particularly, such fusion yields performance gains tasks T2, Ta, and T사 out of all the four tasks. This actually validates the effectiveness of 'Dynamic Hist": by controlling the gap between the historical and main models, the found best main model of task
8

---

Under review as a conference paper at ICLR 2023
Table 3: Ablation study results @R@mean) for our full DHA under the CITM setting. 'Adapt with Hist' denotes adaptation of the main model with the historical model. 'Dynamic Hist' denotes dynamic update of the historical model. 'Mem' denotes the data rehearsal with 5% buffer. The second best results are highlighted by underline.
Adapt with Hist
Dynamic Hist
Mem
Task
Task 72
Task 73
Task
Average
17.59
12.33
12.12
22.80
16.21
32.30
17.09
12.29
18.57
20.06
27.91
17.69
15.60
22.02
20.81
21.23
15.17
14.79
22.03
18.31
36,29
20.41
13.69
18.30
22.17
32.72
21.01
16.45
22.25
23.11
Table Effect of step & (for updat- ing the historical model) on the per- formance of DHA. We only show the results (R@mean) of the final model (trained across all four tasks) on each task under the CTTM setting.
ER
24
DHNA
22.34
20-
Ti
T2
T3
T1
Avg.
1
22.62
15.21
14.20
22.39
18.60
24. 343
16
3
28.01
17.69
14.45
22.22
20.59
5
32.72
21.01
16.45
22.25
23.11
10
Butter (9)
7
33.01
21.28
16.23
21.69
23.01
Figure 5: Results of DHA with extra memory buffer.
T; suffers from much less degradation (caused by 'Adapt with Hist') on task Tz. In other words, adopting both of the two strategies can ensure a good trade-off between the previous tasks and the current task during sequential training. (4) The extra memory buffer yields improvements in most cases. Moreover, even if the memory buffer is used, the two strategies of our DHA are still effective. This means that our DHA complementary to the rehearsal-bast methods.
We further conduct experiments to explore the effect of step & on the performance of our DHA. In- tuitively, le is too small, the historical model would be updated too frequently with the parameters of the main model. Therefore, although the knowledge gap is too small to affect the performance of the main model on the current task, the historical model is hard to preserve the historical knowl- edge. On the contrary, if & is too large, the historical model would only have few updates with the parameters of the main model. As a result, a huge knowledge gap between the historical and main models would harm the performance of the main model on the current task. Overall, good trade-off can be ensured by selecting the best k. Indeed, this analysis is validated by the results in Table 4. Specifically, the performance of our DHA task Ta is the best when k 1 and gradually decreases when k increases from 1 to 7, while the performance on the previous tasks Ti Ti- grows higher the same time. We thus select
with the highest average R @ mean in this paper.
Finally, to investigate the effect of the buffer size, we make comparison among ER (Chaudhry et al., 2019), DER (Buzzega et al., 2020), and our DHA with different buffer sizes (0%, 1%, 5%, and 10% of the training data). We show the comparative results (average R @ mema)) in Figure 5. It can be seen that DHA beats ER and DER all cases. Furthermore, our DHA with 1% buffer and 0% buffer even perform better than DER and ER with up to 10% buffer, respectively. This validates the effectiveness of direct parameter transfer (used in DHA) in continual learning.
5 CONCLUSION
In this paper, we propose a continual image-text modeling (CITM) setting, under which the model required to be trained sequentially on four diverse image-text datasets and finally evaluated on all previous datasets. This new continual setting has a realistic application in large-scale image-text pre- training. We devise an effective dynamic historical adaptation (DHA) approach to coping with the forgetting problem in CITM. Different from existing continual learning methods, DHA proposes to preserve the historical knowledge with direct parameter interaction between the historical and main models. Extensive experiments show the effectiveness of our DHA under the CTTM setting.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In IEEE Conference on Computer Vision and Pattern Recognition, pp- 3366-3375, 2017.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In European Conference on Computer Vision, pp. 139-154, 2018.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in Neural Information Processing Systems, 32:11817-11826, 2019.
Craig Atkinson, Brendan McCane, Lech Szymanski, and Anthony Robins. Pseudo-recursal: Solving the catastrophic forgetting problem in deep neural networks. arXiv preprint arXiv:1802.03300 2018.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE/CVF International Conference on Computer Vision, pp. 1728-1738, 2021.
Ali Furkan Biten, Lluis Gomez, Marçal Rusinol, and Dimosthenis Karatzas. Good news, everyone! context driven entity-aware captioning for news images. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12475, 2019.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark expe- rience for general continual learning: a strong, simple bascline. Advances in Neural Information Processing Systems, 33:15920-159333 2020.
Hyuntak Cha, Jacho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In IEEE Interna- tional Conference on Computer Vision, pp. 9516-9525, 2021.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In European Confer- on Computer Vision, pp. 532-547, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint 1902. 2019.
Arslan Chaudhry, Albert Gordo, Puneet Kumar Dokania, Philip H. S. Torr, and David Lopez-Pat Paz. Using hindsight to anchor past knowledge in continual learning. In AAAI Conference on Artificial Intelligence, pp. 6993-7001, 2021.
Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Ji Liu, and Jungong Han. Imram: Iterative matching with recurrent attention memory for cross-modal image-text retrieval. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12652-12660. 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp. 1597-1607, 2020b.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 14(7):33663336) 2021.
Jacob Devlin, Ming Ming-Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, and Thomas Brox. Coot: Cooperative hierarchical transformer for video-text representation learning. Advances in Neural Information Processing Systems, 33:22605-226188 2020.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Yuqi Huo, Manli Zhang, Guangzhen Liu, Haoyu Lu, Yizhao Gao, Guoxing Yang. Jingyuan Wen, Heng Zhang, Baogui Xu, Weihao Zheng, et al. Wenlan: Bridging vision and language by large- scale multi-modal pre-training. arXiv preprini (0di::2003.06666 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916, 2021.
Xu Jia, Efstratios Gavves, Basura Fernando, and Tinne Tuytelaars. Guiding the long-short term memory model for image caption generation. In IEEE International Conference on Computer Vision, pp. 2407-2415, 2015.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1219-1222 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Grabska-Barwinska. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences. 114(13):351133526 2017.
Frantzeska Lavda, Jason Ramapuram, Magda Gregorova, and Alexandros Kalousis. Continual clas- sification learning using generative models. arXiv preprint arXiv: 1810.1012,, 2018.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. ArXiv, abs/1803..a 503.08024, 2018.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for ando--and-anguuge learning via sparse sampling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7331-7341, 2021.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. AAAI Conference on Artificial Intelli- gence, pp. 11336-11344 1344, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-2947 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, pp. 740-755, 2014.
Yaoyao Liu, Yuting Su, An-An Liu, Bert Schiele, and Qianru Sun. Mnemonics training: Multi- class incremental learning without forgetting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12245-12255 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolin- guistic representations for vision-and-language tasks. Advances in Neural Information Processing Systems, 32:13333, 2019.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In IEEE Conference on Computer Vision and Pattern Recognition. pp. 7765-7773, 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to mul- tiple tasks by learning to mask weights. In European Conference on Computer Vision, pp. 67-82, 2018.
Mandela Patrick, Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, Joao Hen- riques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. arXiv preprint wxxiv22000028222 2020.
11

---

Under review as a conference paper at ICLR 2023
Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. Mirrorgan: Learning rext-to-image generation by redescription. In /EEE/CVF Conference on Computer Vision and Pattern Recogni- fion. pp. 1505-1514, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 3748-8763. 2021.
Jason Ramapuram, Magda Gregorová, and Alexandros Kalousis. Lifelong generative modeling. ArXiv, abs//105.09955 7094477 2020.
Amal Rannen, Rahaf Aljundi, Matthew Blaschko, and Tinne Tuytelaars. Encoder based lifelong learning. In IEEE International Conference Computer Vision, pp. 1320-13328 2017.
Sylvestre-Alviss Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Amir Rosenfeld and John K Tsotsos Incremental learning through deep adaptation. IEEE Transac- fions Pattern Analysis and Machine Intelligence, 42(3):651-66) 2018.
Joan Serra, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp- 4548 4557, 2018.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemyyed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics, pp. 2556-2565, 2018.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in Neural Information Processing Systems, 00:2994-3003 2017.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In ACM SIGIR Conference on Research and Development Information Retrieval, pp. 2443-2449, 2021.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. arXiv preprint arXiv: 0774990 2019.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and D. Erhan. Show and tell: A neural image caption generator. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156- 3164, 2015.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. Advances in Neural Information Process- ing Systems, 31:907-916, 2018.
Jianwei Yang, Yonatan Bisk, and Jianfeng Gao. Taco: Token- cascade contrastive learning for video-text alignment. IEEE/CVF International Conference on Computer Vision, pp. 1 11562- 11572, 2021.
Peter Young, Alice Lai, Micah Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo. classmnncremental learning via deep model consolidation. In /EEE/CVF Winter Conference on Applications of Computer Vision, pp. 1131-11444 2020.
Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8746-8755, 2020.
12

---

Under review as a conference paper at ICLR 2023
A FULL ALGORITHM FOR DHA
In Algorithm I, we give the pseudocode of the full algorithm for DHA. In Algorithm 2, we show our detailed strategy of constructing and updating memory buffer at the beginning of the current task. The core idea of Algorithm 2 is similar ER-ring (Chaudhry al., 2019). However, since ER-ring equally allocates the buffer to each class but the concept of "class" does not exist in the image-text retrieval task, we keep the buffer evenly contains image-text pairs from all the previous datasets instead. The update strategy in Algorithm 2 is applied to all the competitors using memory buffer.
Algorithm Sequential Training with DHA
Input:
the dataset for sequential tasks {De the main model with parameters main the historical model with parameters 0hist the best model of the last task with parameters Olast max iterations bmaz each task hyperparameters ki, À1. A2
Output: the learned 6" muiin initialize Omain by training the main model on D1
initialize Blast & Omain
for Dj (DD,.... Dr) do
initialize Onist + Oiast, Omoin &- Orast for i 5- to imax do
b> Initialize the main and historical models
if 1%k- = 0) then
8f f- A2gimu hiot + (1 X2) bg)mmol
Update the historical model with main model
else
0fist +- -
D Do not update the historical model
end if
update @main according to Eq. (9) end for obtain the best faain Glast t- Omain end for
return the found best 0 Maann
Algorithm 2 Memory Buffer Update Strategy
Input:
the dataset D.-1 of task T--1
the memory buffer M used in task 7--1 the memory buffer Mr used in task T,
the samples S selected from D; (1 i < t) to compose buffer Me.
the buffer size JM
the number of tasks T
Output: Mr
if f. 2 then
S? M
D1
> Randomly select [M] samples from D1
M2 St
else if 1 < T then
for 1 < i i (t () do
S S
M > Randomly select t-1
samples from buffer M
end for
Mi
8 D
M > Randomly select --1 samples from task T Form the buffer M, used in task T
M, < (S} St St $
end if
return M,
13

---

Under review as a conference paper at ICLR 2023
B IMPLEMENTATION DETAILS OF ALL COMPETITORS
We introduce the implementation details of all competitors for continual image-text modeling:
LwF (Li & Hoiem, 2017): We adopt the best main model in the last task as the old model, maintain the old model the new task, and align the output logits between the old and current main models the distillation regularization.
ER (Chaudbry et al., 2019): As the classic data-rehearsal approach, ER mainly reduces the catas- trophic forgetting by reusing partial old data (from previous tasks) in the task. We adopt Al- gorithm (similar to ER-ring) to update the memory buffer. We set the buffer size as 5% of the training set of each task. Importantly, all the other competitors which adopt memory buffer share exactly the same buffer setting as ER.
DER (Buzzega et al., 2020): We save logits instead of inputs and old models for DER. Fol- lowing (Buzzega et al., 2020), we randomly select the previous model from 15 epochs to acquire various expressions of the buffer. Moreover, only the logits of previous sample pairs aligned.
CO2L (Cha et al., 2021): The data rehearsal and regularization strategies are both adopted by CO2L. Different from DER, CO2L applies data augmentation. Hence, adopt two augmentations for input images in each mini-batch. Following its strategy, the sample pairs in the buffer are only used negative samples.
We adopt the same temperature hyperparameter and learning rate for DHA, Baseline and all the above competitors. In this work, our DHA only adopts the same memory buffer ER and mixes with new data when training new task.
C DOMAIN SHIFT IN THE CITM SETTING
To directly demonstrate the domain shift across the four datasets used in our CITM setting, conduct cross-dataset evaluation experiments. Concretely, train the model (with the same architecture de- scribed in Sec. 3.2) independently on the train set of each dataset, and then evaluate of all four datasets to show its performance on the seen dataset and the other unseen datasets. As shown in Figure 6, the model achieves the highest perfor- mance on its seen dataset and much lower perfor- mance on unseen datasets. Therefore, we validate that there do exist domain shift, in other words, the domain gap across the four datasets. Overall, CITM setting reasonably mimics the realistic appli- cation scenarios of image-text modeling.
COCD-tesr
GDntww test
CCIM
WiT-test
R@1
cOcO-train
28.66
2.50
the sets
CC3M-train
7.92
15.98
2.78
2.48
WiT-train
4.06
3.68
9.25
3.12
GDnews-train
3.82
3.24
2.98
8.96
Figure Results cross cross-dataset evaluation after independent training.
D DETAILED EVALUATION RESULTS
More detailed experiments results are included in this section. Firstly, we show the ablative I2T sults (Average mean) of our DHA model with different values of X₁ and A2 in Table 5. Accord- ing to their definitions in Sec. 3.3, A₁ controls the update speed of the main model with the historical model, and d2 controls the update speed of the his- torical model with the main model. Thus, the bal- ance should be made between the two coefficients. We can see that the best performance could be ob- tained when λ1 0.995 and X 0.985. Moreover, most of the combination groups in Table lead to better results than DER (Buzzega et al., 2020) (21.24), which further indicates that our DHA indeed promising approach CITM.
Table 5: Effect of coefficients A and X2 on the performance of DHA.
A2
0.98
0.985
0.99
0.999
A
0.98
20.68
21.01
20.93
20.87
0.99
21.24
22.23
21.64
21.33
0.995
22.15
23.11
22.28
22.12
0.999
21.43
22.84
21.97
22.08
14

---

Under review as a conference paper at ICLR 2023
T21: R@ mean of the main model on Task 7
T21: R@mean of the main model on Task 7
45
40
35
R@ mean
30
e@mean
20
25
Baseline
8 20
LWF
15
DER
ER
14
ER
COZL
12
COZL
10
DHA
DHA
10
DHA
Task-ID
Task-ID
12T: R@mean of the main model on Task 72
12T: R@ mean of the main model on Task T,
26
40
24
35
22
eemean
30
R@mean
20
25
Baseline
Baseline
20
LwF
LKF
15
DER
CO2L
COZL
10
DHAT
DHA
5
3
Task-ID
Task-ID
Figure 7: Illustration of the results of the main model (of all methods) Task T2 and Task T3 during sequential training on the four tasks. 'T2I' denotes text-to-image retrieval and 'ITT" denotes image-to-tess retrieval. It can be clearly seen that our DHA forgets with the lowest speed.
Secondly, we present the results on tasks T2 and T3 obtained by the main model during sequential training in Figure 7. It can be clearly seen that our DHA still forgets with the lowest speed on tasks T2 and T3, similar to Sec. 4.3 (see Figure 4). Importantly, by reviewing the results on tasks Ti T3 (from Figure 4 and Figure 7), we find that DHA could address the catastrophic forgetting problem much better on the earlier seen tasks, which crucial advantage for practical applications.
Finally, we provide the detailed results in terms of Recall @ 1, Recall@5, and Recall @ 10), in addi- tion to Table We can observe from Tables 6-8 that our DHA consistently has the best performance on average recall and forgetting rate. Additionally, our DHA performs the best on all the three his- torical tasks (i.e.. Ti T3) in terms of Recall 1, which indicates that our DHA is indeed the best approach to CITM even with the most strict metric.
E FURTHER RESULTS FOR ABLATION STUDY
We compare the results between DHA-bbst-val (i.c., initializing the historical model with the best- validated model the last task) and DHA-last-iteratio (i.c., initializing the historical model with the last iteration model of the last task) under our framework in Table 9. It can be observed that these two strategies achieve almost the same performance.
Furthermore, the comparative results (R (e@m)) between training with all data (the upper bound) and training with DHA are presented Table 10. We can find that there exists 5-6% gaps between DHA and the upper bound in terms of the average performance (over all tasks).
15

---

Under review as a conference paper at ICLR 2023
Table 6: Comparative results @Recall@) @1 I and FR) between our DHA and other representative/latsst methods. 'T21' denotes eex-tt--mmage retrieval and 'I2T" denotes image-to-text retrieval. All meth- ods adopt the same network architecture. 'Mem' denotes the data rehearsal with 5% buffer.
Task Tj
Task
Task T3
Task T4
Average
Method
Mem
1
FR
R@1
PR
R@1
PR
R@1
R@
Baseline
4.61
77.73
4.92
72.82
4.62
48.09
9.72
5.97
66.21
LwF &1.& Hoiem, 2017)
N
6.04
70.82
6.00
66.85
5.22
43.38
9.82
6.77
60.35
ER (Choudhry at. 2019) 2019)
Y
5.62
72.85
5.78
67.34
4.78
45.06
9.34
6,38
61.75
T21
DER (Buzzega 2020)
Y
7.65
63.04
8.24
53.91
5.64
35.17
8.68
7.55
50.71
CO2L 202,)
Y
7.30
64.73
6.90
59.36
5.74
37.33
9.20
7.29
53.81
DHAT (ours)
8.03
61.21
8.74
49.89
5.73
35,98
7.81
49.03
DHA (ours)
9.31
55.02
9.30
43.77
5.98
25.44
8.46
8.26
41.41
Baseline
N
5.88
79.48
3.88
78.77
4.72
47.20
9.74
6,06
68.48
LwF (Li Hoosem. 2017)
N
8.64
69.85
5.42
70.29
5.50
44.11
10.04
7,40
61.42
ER (Chaudiry 2019)
Y
7.52
73.76
5.20
70.15
5.24
45.30
9.12
6.77
63.07
12T
DER (Buzzega
11.40
60.22
6.98
60.83
6.02
38.07
9.20
8.40
53.04
CO2L al.,
10.68
62.74
6.34
62.84
6.22
34.80
9.70
8.24
53.46
DHAT (ours)
11.42
60.15
6.60
59.33
5.56
36.12
8.74
8.08
51.86
DHA (ours)
14.52
49.34
8.32
48.77
6.22
34.86
9.12
9.55
44.32
Table 7: Comparative results (Recal and FR) between our DHA and other representative/latet methods. 'T2I' denotes text-to-image retrieval and 'I2T' denotes image-to-text retrieval. All meth- ods adopt the same network architecture. 'Mem' denotes the data rehearsal with 5 5% buffer.
Task T
Task T>
Task T3
Task T
Average
Method
Mem
R@5
FR
R@5
FR
R@5
FR
R@5
R@5
FR
Baseline
N
14.36
69.62
13.62
65.62
14.32
41.84
24.38
16.67
59.03
LwF (Li 2017)
N
17.88
62.17
16.88
56.54
16.32
35.49
24.20
18.82
51.40
ER (Chundiry 2019)
17.27
63.47
17.66
54.88
15.44
37.18
23.44
18.45
51.84
DER (Buzzega al. 2020)
21.93
53.61
41.93
17.52
27.36
22.86
21.28
41.71
CO2L, (Cha at, 2021)
Y
20.87
55.85
20.22
47.78
17.60
30.32
25.04
20.93
44.65
DHA (ours)
N
23.11
51.11
23.67
39.03
16.48
30.38
22.92
21.55
40.17
DHA (ours)
Y
26.65
43.62
24.78
34.65
16.80
30.12
22.50
22.68
36.13
Baseline
N
18.40
67.22
12.12
70.19
14.32
44.41
27.47
18.08
60.61
LwF
2017)
N
23.14
58.78
16.18
59.97
16.56
37.32
25.04
20.23
52.02
ER (Chondluy et 2019)
Y
23.08
58.89
16.60
58.52
15.92
38.58
24.14
19.94
52.00
12T
DER (Buzzega et 2020)
Y
30.42
46.63
19.64
48.71
16.54
31.25
24.06
22.67
42.20
CO2L (Cha 2021)
28.08
49.98
18.44
53.32
17.70
32.90
25.68
22.48
45.40
DHA (ours)
N
30.36
45.39
19.22
49.88
15.72
32.47
24.40
22.43
42,58
DHA (ours)
Y
35.24
37.23
22.76
40.67
17.84
28.66
24.02
24.97
35.52
Table Comparative results (Recall @ 10 and FR) between our DHA and other representative/lltestt methods. "T21' denotes text-to- eext-to-mmage retrieval and 'I2T' denotes image-to-te to-text retrieval. All meth- ods adopt the same network architecture. 'Mem' denotes the data rehearsal with 5% buffer.
Task T
Task T>
Task T
Task T3
Average
Method
Mem
RΦ
10
FR
R@10
FR
R@10
10
FR
R 10
R@10
FR
Buseline
N
21.96
64.06
19.80
61.37
21.96
37.11
33.76
24.37
58.18
LwF Hoiem,
N
26.50
56.64
24.20
51.81
24.46
30.47
34.02
27.30
46.31
ER (Chunddlry
Y
26.00
57.45
25.02
50.94
22.90
33.89
31.94
26.47
47.43
T2I
DER (Duzzega al., 2020)
Y
31.98
47.67
31.72
37.31
25.76
24.77
31.58
30.26
36.58
CO2L (Cha 202.)
Y
30.74
50.52
29.72
40.39
25.06
29.49
34.60
30.03
40.13
DHA (ours)
N
32.80
46.33
33.06
34.51
24.71
28.06
32.46
30.76
36.30
DHA (ours)
Y
37.78
38.18
34.76
29.89
25.26
25.02
32.70
32.63
31.03
Buseline
N
27.50
60.52
18.64
63.75
21.40
38.93
33.94
25.37
51.77
LwF (ti Hoinns 2017)
N
33.00
52.63
24.48
51.83
24.20
34.24
34,80
29.12
46.23
ER (Chandhry 2019)
Y
33.10
52.48
23.72
52.92
23.20
35.34
32.84
28.22
46.91
12T
DER (Buzzega 2020)
Y
41.30
40.71
29.60
41.41
25.34
28.54
32.42
32.14
36.89
CO2L
at.l.
Y
39.94
42.66
26.50
47.69
25.06
29.92
35.38
31.72
40.09
DHA (ours)
N
41.96
27.26
45.01
22.72
31.15
34.16
31.53
38.64
DHA (ours)
Y
48.40
30.52
31.96
35.54
25.30
29.35
33.62
34.82
31.80
16

---

Under review as a conference paper at ICLR 2023
Table 9: Comparative results between DHA with the best main model (of the last task) as the histori- cal model and DHA with the last-iteration model as the historical model. 'T2I' denotes eext-to-image retrieval and 'I2T" denotes image-to-text retrieval.
Task 71
Task 12
Task 7
Task 74 m@an
Average
Method
R@mean
FR
R@mean
FR
R @ mean
FR
R mean
FR
DHA-best-val
24.58
42.88
22.95
33.82
16.15
24.50
21.22
21.29
33.73
T2I
lattitattioo
24.47
43.13
23.06
33.51
16.01
25.15
21.40
21.24
33.93
DHA-best-val
32.72
36.45
21.01
39.50
16.45
24.71
22.25
23.11
33.55
12T
Dt--tast-iteration
32.67
36.56
21.22
38.91
16.21
25.81
22.53
23.16
33.76
Table 10: Comparative results (R@mean) between training with all data (upper bound) and training with DHA. 'T2I' denotes text-to-image retrieval and '12T" denotes image-to-text retrieval.
T2I
12T
Method
Task 1
Task 22
Task T3
Task 74 Average
Task T
Task 2g
Task T3
Task T4 Average
Upper bound
31.61
32.39
18.86
22.69
26.39
41.89
32.27
20.82
22.38
29.34
DHA (ours)
24.58
22.95
16.15
21.22
21.29
32.72
21.01
16.45
22.25
23.11
Table 11: Evaluation results for setting different values of X for the text and image modalities. 'T2I' denotes text-to-image retrieval and '1TT denotes image-to-text retrieval.
Setting
Setting Setting 3
Setting 4
A1 for the text modality
0.993
0.993
0.995
0.995
A1 for the image modality
0.993
0.995
0.993
0.995
T2I: average R @ mean 12T: average R@@ mean
20.77
21.16
21.45
21.29
21.43
22.94
23.33
23.11
Additionally, we clarify that applying the same value X for image and text modalities enables the proposed DHA to be easily deployed in other continual learning settings. However, we should point out that our proposed framework also provides convenience for discovering and making use of modality-spechee forgetting characteristics under the CITM setting. Due the generality (flex- ibility) the proposed DHA, we can easily utilize it to cope with modality-specific forgetting (or modality-wise domain gap) by setting different values of λ₁ (in Eq. (9)) for the text and image modalities. The obtained results are shown Table 11. We can see that the proposed DHA leads to further improvements when λ takes different values for the two modalities. This also suggests that coping with modality-specific forgetting is indeed necessary for our CITM setting.
17


Reviews:
[{"review_id": "y0B1ZOdogH", "reviewer": "Reviewer_xMpX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an incremental learning approach for VL understanding (in particular, cross-modal retrieval). The authors established a benchmark by combining several popular VL datasets and defining an incremental (sequential) scenario. The proposed method involves applying EMA back and forth between old (offline) and new (online) models. Experiments show improvement in the studied scenario.", "summary_of_the_review": "I had a hard time evaluating the contribution of this paper. Temporarily, I do not think the paper made important contributions that shall be published at ICLR.", "strength_and_weaknesses": "Strengths\n1. The studied problem is interesting and important.\n2. The paper is easy to follow.\n\nWeaknesses\n\nI am familiar with VL understanding but I do not work in the continual learning field (I do know too much about it). I choose to write down my opinions below and wait for other reviewers' comments and discussions with the authors.\n\nI think the proposed method is reasonable but it does not provide new insights. It is about propagating information back and forth between an online (new) model and an offline (old) model. Intuitively, by absorbing knowledge from the old model, the recognition accuracy on old datasets is expected to grow, but that on new data is expected to drop. Experiments validate the hypothesis, where we see a relatively large accuracy gain on task T1, but the gain drops rapidly and eventually becomes a deficit on task T4. Although the average accuracy is higher than the competitors, I guess it comes from a better balance among the tasks -- that said, the core issue of catastrophic forgetting is not well addressed, but the approach seems to add weights among different, historical models toward a better average accuracy. Personally, I do not praise the contribution of the proposed method.\n\nI shall admit that I did not achieve a clear thought of evaluating this paper. It seems that replaying old data (with buffer) and average old and new models are the only way to alleviate forgetting, which is trivial to me. Is this the correct direction for incremental learning?\n\nThere are some minor issues, such as the missing of diagnosis of the key hyper-parameter, k.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gw8uN1WFBG", "reviewer": "Reviewer_4FxS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on continual image-text embedding, reducing the computation and storage resources using all data in the training phase. For the first time, the paper identifies the important role of direct parameter transfer (between the historical and main models) in continual learning and the proposed DHA outperforms existing continual learning approaches, including rehearsal-based methods, regularization-based methods and hybrid models.", "summary_of_the_review": "The paper focuses on continual image-text embedding, which is an important task. The proposed DHA model outperforms its counterparts, reducing the computation and storage resources of using all data to train models. In this phase, I give it a score of 8.", "strength_and_weaknesses": "Strength:\n\nS1: This paper identifies the important role of direct parameter transfer (between the historical and main models) in continual learning, and proposed a dynamic historical adaptation model.\n\nS2: The proposed model outperforms its counterparts, including rehearsal-based methods, regularization-based methods and hybrid models.\n\nS3: Extensive experiments are conducted to show the effectiveness of the proposed model.\n\nWeaknesses:\n\nW1: It is unclear how large the gap is between using all data to train a model and the proposed DHA.\n\nW2: I am wondering whether the order of tasks matters.\n\nW3: The style of Flickr30k is similar to MSCOCO, and there should be overlaps with MSCOCO, so zero-shot may not be appropriate.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T2Dje1--XT", "reviewer": "Reviewer_4eZi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for continual learning in text-image modeling. They propose a formulation that uses a historical and main neural networks whose parameters are used interchangeably\tbetween the two networks during parameter optimization as a weighed average. In experiments, the authors compare favorably against baseline methods through thorough evaluation and they also provide a thorough ablation to highlight the contribution of the different components and hyper parameters of the proposed method.", "summary_of_the_review": "The formulation in this paper is interesting, and the authors made sure to ablate each portion of it to show the different behaviors with different pieces. In addition, the authors outperform the baselines on average. Therefore, I am leaning towards accepting this paper.", "strength_and_weaknesses": "Strengths:\n+ New formulation of continual learning with minimal forgetting\n+ Thorough evaluation and ablations\n+ Well written paper\n\n\nQuestions:\n- Performance on Task3 and Task4.\nThere seems to be a weakness of the propose model that shows in Table 1 for tasks Tasks 3 and 4. Do they authors have any analysis showing why this is happening. I would assume the performance should be best in the later Tasks than the earlier tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BA_Twu8kG92", "reviewer": "Reviewer_miUb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of continual learning under a novel Continual Image-Text Modeling setting where the model learns a sequence of multiple image-text matching tasks. The authors claim that transferring the old knowledge from the historical model parameters and periodically updating the current model to the historical model addresses the problem, since the datasets used in the proposed CITM setting have a large domain gap. The experimental results on the proposed benchmarks show the effectiveness of the proposed framework compared to the baselines.", "summary_of_the_review": "Although the paper proposes a novel and interesting problem setting, the reviewer has some concerns to be addressed, as stated in the weakness and questions section.", "strength_and_weaknesses": "- Strengths\n    - The proposed novel CITM problem has not been studied, but is a real use case, since many image-text datasets with large domain shifts have been proposed recently, and vision-language models are expensive to retrain in general.\n    - The proposed framework demonstrates better performance than other baselines for the newly constructed benchmarks, using existing image-text datasets.\n    - The idea of alternatively updating the historical model and the main model seems interesting, but the reviewer is not fully convinced yet.\n    - The ablation study well supports the claim that “Adapt with Hist” provides stability while “Dynamic Hist” addresses plasticity.\n    \n- Weaknesses\n    - It seems the proposed framework is not fully aligned with the proposed CITM setting.\n        - First, the reviewer agrees with the authors’ claim that the image-text datasets have a large domain gap, and thus the existing rehearsal-based and regularization-based methods may not be suitable to the CITM setting.\n        - Therefore, the main problem of the CITM setting (according to the paper) is domain shift.\n        - However, the domain shift problem arises from collecting multiple datasets, which is not incurred from the inheritance of multi-modal data.\n        - Thus the link between the proposed problem setting and the proposed framework is weak.\n        - For me, this kind of study should deliver some idea or solution which handles multi-modal learning via multi-modality itself or modality-specific aspects. There could be modality-specific forgetting, or modality-wise domain gap, etc.,.\n        - What makes the proposed framework special for the multi-modal task, not just for the datasets with large domain shifts?\n    - Discussion for the methodology\n        - Finding the best model for the last task using the test seems inappropriate. Although there could be some works beyond my knowledge using a test set of the previous task, continual learning usually assumes that we do not have access to the test set during incremental training steps.\n        - A more common way to choose the model of the last task is to choose the model after the last iteration.\n        - What if we just select the model after the last iteration for each task? Or, what is the mean and variance of epoch (or iteration) for each task under the current version of the framework?\n    - Discussion for the datasets\n        - It seems subsampling the same number of data from each dataset (e.g., $T_2-T_4$) diminishes the necessity of CITM setting, while this strategy addresses the dataset-imbalance problem for the experiments. In general, different image-text datasets have different magnitudes of sizes. The obvious examples are MSCOCO (123K images) and CC3M (3M images), which are used in this study.\n        - The order of the datasets is the crucial aspect that affects the performance of continual learning. The results for different dataset orders could make the authors’ claim more robust. It could be a minor point if we sample the same number of data from each dataset as in this study. However, it would be crucial if we just naively use all data in the datasets with different magnitudes of size.\n    - Minor\n        - Typo : wrok → work : last sentence in Appendix B.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject