Continual Vision-Language Representaion Learning with Off-Diagonal Information
Authors: Zixuan Ni
Longhui Wei
Siliang Tang
Yueting Zhuang
Qi Tian
Venue: Submitted to ICLR 2023 (2023)
Keywords: representation learning
continual learning

Abstract: Multimodal pre-trained methods with a contrastive learning framework (like CLIP) have recently achieved consistent advantages on various cross-model downstream tasks. However, they usually require a large amount of image-text samples and a vast computing budget for training, which makes the re-training process expensive while the training data is collected continuously (the phenomenon is widespread in real scenarios). In this paper, we discuss the feasibility of continuously training CLIP models based on discrete streaming data. We find that the multimodal retrieval performance of the CLIP in a continual training setting is significantly lower than that in a joint training setting. We name this phenomenon Cognitive Disorder(CD). By tracking the directional changes of the representation vectors in the continuously updated CLIP model, we explore and summarize the spatial variation of the modal encoders within the CLIP: Intra-modal Rotation and Inter-modal Deviation. Intra-modal Rotation means that the vision and language representation space in the CLIP is rotating greatly around the center of a high-dimensional unit sphere during continual training, accompanied by a relatively small change in the topology of the representation space. Inter-modal deviation happens when the vision and language's intra-modal rotation is unsynchronized. Moreover, we empirically and theoretically demonstrate how intra-modal rotation and inter-modal deviation lead to CD. In order to alleviate CD in continual CLIP training, we propose a new continual training framework Mod-X: Maintain off-diagonal information-matrix. By selectively aligning the off-diagonal information distribution of contrastive matrixes, the Mod-X helps the model not only better fits the newly trained data domain but also maintains the multimodal cognitive ability on the old data domain during the continual large-scale training (Section \ref{experiments}).

Under review as a conference paper at ICLR 2023
CONTINUAL VISION-LANGUAGE REPRESENTAION LEARNING WITH OFF-DIAGONAL INFORMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Multimodal pre- tre-trained methods with a contrastive learning framework (like CLIP) have recently achieved consistent advantages on various croee- downstream tasks. However, they usually require a large amount of image-text samples and a vast computing budget for training, which makes the re-training process expensive while the training data is collected continuously (the phenomenon is widespread in real scenarios). this paper, we discuss the feasibility continuously training CLIP models based discrete streaming data. We find that the multimodal retrieval performance of the CLIP in a continual training setting is significantly lower than that in joint training setting. We name this phenomenon Cognitive Disorder(CDD) By tracking the directional changes of the representation vectors in the continuously updated CLIP model, we explore and summarize the spatial variation of the modal encoders within the CLIP: Intra-modal Rotation and Inter-modal Deviation. Intra-modal Rotation means that the vision and language representation space in the CLIP is rotating greatly around the center of high- dimensional unit sphere during continual training, accompanied by a relatively small change in the topology of the representation space. Inter-modal deviation happens when the vision and language's ' intra-modal rotation is unsynchronized. Moreover, we empirically and theoretically demonstrate how intra-modal rotation and inter-modal deviation lead to CD. In order to alleviate CD in continual CLIP training, we propose new continual training framework Mod-X: Maintain off- diagonal information-matrix By selectively aligning the off-diagonal information distribution of contrastive matrixes, the Mod-X helps the model not only better fits the newly trained data domain but also maintains the multimodal cognitive ability on the old data domain during the continual large-scale training (Section 5).
INTRODUCTION
Recently, multimodal pre-trained models such CLIP al. (2021) have attracted much attention. By utilizing these pre-trained models, many works have achieved new progress in down- stream tasks such as classification Zhang et al. (2020) Wei al. (022); Lee et al. (2022), semantic segmentation Xie et al. (2021b); Wang et al. 20211). object detection Xie et al. (2021a); Wang
al. (2022a), speech recognition Baevski al. 2020)) etc. Although the CLIP model has strong paperRadford
generalization in open- data, as mentioned in CLIP paper Radford et al. (202T), the ability to
match image-text samples that are not its training data distribution is still weak. The natural idea to alleviate this problem is to scale the training data to cover different data domains. However, it is infeasible to train infinite data with limited hardware at once.
In this paper, trying to break this non-iterability, we explore the feasibility of continuously training the CLIP model through streaming data, a training paradigm that follows Continual Learning (CL. McCloskey & Cohen (1989). To simulate continual CLIP training, randomly and evenly divide into multiple sub-datasets and train the CLIP sequentially using these
the training data sub-datasets. For comparison with continual training, train CLIP additionally from scratch using the joint-dataset, which is named joint training, as the upper bound on the performance of the continuously trained model.
Traditional supervised continual learning has been proven to suffer from catastrophic forgetting Re buffi et al. (2017): Kirkpatrick al. (2017). The model's performance on old tasks drops significantly

---

Under review as a conference paper at ICLR 2023
as training phases rise. Recently, some work Ni al. 2021b); Hu et al. (2021) has validated that self-supervised models like SimCLR Chen et al. (2020) and Barlow Twins Zbontar et al. (2021) do not suffer from severe catastrophic forgetting during continual training. Some works Madaan et al. (202T): Thai et al. (2021) conjecture that the reason is that the contrastive loss is not directly affected by the supervised signal, and the self-supervised framework does not have SoftMax function to amplify the influence of labels.
However, the performance of CLIP with continual training setting overturns this hypothesis. There is a significant degradation of multimodal retrieval results with continual training compared with joint training (in Section 3 and 5). We name this phenomenon Cognitive Disorder(CD)) Due the vision and language encoders within the CLIP normalizing the representation to a unit vector through a dimension-base
122 norm, which limits the diffusion of representation vectors length, we try to analyze the representation space variation of modal extractors from spatial geometry perspective. By tracking the directional changes of the representation vectors in the continuously updated CLIP model (in Section 3). we explore and summarize the spatial variation of the modal encoders within the CLIP: the Intra-modal Rotation and Inter-modal Deviation. The intra-modal rotation refers to the representation space of the single- modal feature extractors (vision and language) within the CLIP that rotates around the center of the gigh-dimensional sphere, accompanied by a slow topology change during the continual CLIP training. The inter-modal deviation refers to the cognitive deviation of different modal extractors (vision and language) to the same entities during continuous training. Moreover, we empirically and theoretically demonstrate how intra-modal rotation and inter-modal deviation lead to cognitive disorder (in Section 3).
To alleviate this cognitive disorder in continual CLIP training, we propose a simple yet effective framework Mod-X: Maintain dff--dagonal information-matri. Unlike contrastive loss (ord et al. (2018) only focuses on the proportion of positive and negative sample pairs. The Mod-X framework pays more attention to the distribution of off-diagonal information in the contrastive matrix. The similarity distribution on the off-diagonal illustrates the model's cognition of all entities on current data. By selectively aligning the ofi-diagonal information distribution of the contrastive matrixes constructed by the current and past models based on the recent training sample, Mod-X helps the model preserve the correct cognition of various old entities while fitting the current data during continual training. The evaluations in Experiments 5 with different scale and scope datasets show that our Mod-X framework helps the model not only better fits the newly trained data domain (in Section [5.3) but also maintains the multimodal cognitive ability between the current model and old model the old data domain during the continual large-scale training (in Section (54)) More technological details and evaluations have been shown in Section F and Section our contributions are as follows:
S In summary,
We discuss the feasibility of training the CLIP model continuously through streaming data. Empirical experiments demonstrate that continual CLIP training leads to persistent performance degrades on multimodal retrieval. We name this Cognitive Disorder.
By introducing series of tools to track the directional changes of the representation vectors in the continuously updated CLIP model, we explore and summarize the spatial variation of the modal encoders within the CLIP: 1) The Intra-modal Rotation 2) The Inter-modal Deviation. Furthermore, empirically and theoretically demonstrate how intra-modal rotation and anter-modal deviation lead to cognitive disorder (in Section 3
We propose simple yet effective continual CLIP training framework Mod-X that alleviates CLIP's cognitive disorder by selectively aligning off- off-diagonal information in contrastive matrixes between the past and current models in continual training.
2 RELATED WORK
Continual Learning. Continual learning (CL) (1995), or incremental learning. is mainly focused on supervised tasks. In addition to the vision- tasks De Lange et al. 2): Kj et al. (2021) Cha et al. 2021); 3; Ahn et al. (202I). some works discussing language-based tasks Biesialska et al. 2020): Sun et al. (2019). We can summarize the existing continual learning methods into three categories: regularization Kirkpatrick et al. (2017); Ahn et al. (2019), ; al. (2021a) replay Rebuffi ct al. (2017); Rolnick et al. (2019); Wang et al. (2021a), and architecture Thai et al. (2021): Ni et al.
(202 1bb: Hu et al. (2021) x Madaan al. (2021). In unsupervised and self-supervised based
2

---

Under review as a conference paper at ICLR 2023
single modal, the latest work Thai al. (2021 Ni et al. 2021bb; Hu al. (2021) Madaan et al. (2021) has drawn some conclusions different from those of supervised. However, only few works Srimivasan et al. (2022); Fan et al. (2022) focus incremental multimodal tasks learning. Because of the cooperation between different models, continual multimodal pre-training shows different
performance and problems from single modal continual training.
Visual-Language Representational Learning. Vision-language representation learning based on contrastive loss Oord al. (2018) such as CLIP Radford et al. (2021) has attracted a lot of attention in various fields Radford et al. (202T); Li et al. 2021); Andonian et al. (002)); Mahabadi et al. (2022). And the pre-trained model perform surprisingly well on downstream tasks Shu al. (2022) Wang et al.] (2022b); Chowdhury et al. (2022). At the same time, the large-scale image-text datasets c.g..
Laion400M Schuhmann et al. (2021) and Conceptual Captions Sharma al. (2018) has played key role in multimodal pre-training. Although large-scale open-world datasets contain various samples, pre-trained model still cannot perfectly match image-text sample pairs that are not its training data domain Radford et al. (202I).
3 COGNITIVE DISORDER IN CONTINUAL CLIP TRAINING
This section discuss the performance of the CLIP&t et model, which trains CLIP continually without other operations. We name the decline in CLIP performance as Cognitive Disorder. At the same time, the spatial variation of the modal encoders within the CLIPat are explored and summarized: 1). Intra-modal Rotation 2). Inter-modal Deviation. Finally, we empirically and theoretically demonstrate how intra-modal rotation and inter-modal deviation lead to cognitive disorder.
Exploration Setup. To ensure the controllability of the exploration, we train a CLIP, model from scratch on the COCO dataset Lin et al. (2014) based the OpenAl source code OpenAl and use it as the inttial(start) of continual CLIP training. After that, we divide the Flickr30K dataset Young et al. (2014) into five sub-datasets (D,.D) uniformly and randomly to simulate streaming data. Then we train the CLIPo based on this sub-datasets sequentially, and no other data is used when training on each sub-dataset. After finishing five training phases, we obtain the model CLIP;- For comparison with CLIPs, we train CLIPj, from scratch using joint dataset COCO and Flickr30K as the upper bound of the CLIPa at The hyper-parameterr for all training phases are kept the same, and detailed settings of CLIP can be seen in Appendix 7.4
Image-Test #91 CL,P. COCO1SK)
Text-image R@1 results CLP.,, COCO/SK)
CLP:
R@B
CUP, CLPc
Phaltyy
Training Phoses
Training Phases
Text R@1 results of CLI,, Flickr30K11K)
Text- R@1 results of CUP, on Flickr30K/1K)
52 10
20
R03
16
CUP,
R81
CLP
Iibal
mooe
Phanay
Phoaee
Mhanis
Pheaay
Phone
Phooss
Training
Phases
Training Phases
Figure 1: The multimodal retrieval R@l results of CLIP, (0st< 5) on test sets COCO (5K) and Flickr30k (IK). The two sub-figures on the left show the Image- Text retrieval R@1 performance of CLIP, on the continual training phase f. The rights show the Text- R&1 results. The pentagon points (CLIP;)) show the results of the CLIP under joint training, which is an upper bound for continual CLiP training (CLIP&))
3.1 THE PERFORMANCE OF CONTINUAL TRAINING
We show the R@1 retrieval results of CLIP, (0 St 5) on the 5K test set of COCO (COCO(5K) and IK test set of Flickr30k (Flickr30K(11)) in Figure By comparing the multimodal retrieval
3

---

Under review as a conference paper at ICLR 2023
performances of the CLIPu (initial phase) and C ja on (Iic,r30K(IK), performance of CLIP,r is significantly better than that of CLIPo, which is not trained on Flickr30k. This phenomenon shows that the performance of the CLIP model is affected by the training data domain, which is consistent with the results of the paper Radford et al. (2021). Besides this, can he clearly seen that the multimodal retrieval performance of the CLIP 21 the COCO(5K) has continued to decline with the rise of training phases. The final Image-Tex Text R R@) I result of CLIP, on COCO(5K) plummeted from the initial 14.7% to 6. 1%, and the Text-Image results dropped from 10.6% 4.7%. The gap with CLIP,; je reached 10.0% and 7.0%. On the other hand, CLIP&s exhibits slow and erratic increase multimodal retrieval R01 results on the test set Flickr30K(1K) Although the results between CLIPat at and CLIPj, , on the Image-Text R@I has been narrowed from the original 13.2% to 9.5% while the Text-Image R@1 of cLIP,t has increased from 12.0% to 16.1%, 1,, the gap between CLIP, and CLI is still great. We name this phenomenon Cognitive Disorder (CD).
find that the retrieval
3.2 THE REASONS FOR COGNITIVE DISORDER
In CLIP, the vision and language encoders normalize the final representation vector a unit vector of length I using a dimension- based L2 norm. This design makes the representation space in vision and language encoders form a high-dimensional unit sphere. Based on this fact, we ignore the influence of the representation vectors' length and track their direction changes.
3.2.1 THE INTRA-MODAL ROTATION
Firstly, we analyze the directional changes of the representation vectors of vision and language extractors in continual CLIP training. Taking the visual representation space as an example, we use the visual encoder F. in CLIP; to extract the image representations of the test set COCO(5K) and obtain the vision representation vectors sets V; {V;0 V.", V/2 ...., V:N, .... V,5K), }, where i O,..., 5 stands for five different training phases. After that, we take the inner product vectors < VP, V in each vector set V; and perform arccos operation to obtain their Self-Angle relationship Matrix (S.M).. The SAMa(a.) arccos(<< Vi. V >). Any element SAM (a.b) in the SAM; matrix represents the included angle between the sample and in the vision encoder EY By counting the difference AM between the corresponding elements in two continual SAM matrix SAM, and SAM++1 as shown in Figure ((a) we get the following included angle change distribution table 2(b)
each pair of
Ex:
V
E
Vi
05AM 50) (5', 10 (10', 15
15 (159, 20
(200, 180%]
SAM
54.23%
31.64%
11.28%
2.48%
0.38%
va
SAM, 66.18%
30.21%
7.54%
0.99%
0.07%
SAM- .61.44%
30.00%
7.50%
0.98%
0.07%
t+1
SAM, 155.33%
31.75%
10.57%
2.07%
0.28%
@SAM = (V).V)) <(V Vi+1)
SAM, 50.51%
32.14%
13.17%
3.50%
0.68%
SAMo-5 42.94%
31.12%
16.73%
6.66%
2.55%
(a)
(b)
The sub-figure the left shows schematic diagram of computing 0SAM- The table on the right shows the distribution of the change of the included angle between any two samples different training phases' vision representation space. And SAM-, SSAM, SAM,I.
Figure 2:
From the table 2(b) we can find that 80% of the angle changes between any two vision representation vectors in the vision representation space are between 0 and 10 degrees in the process of continual CLIP training, while only 20% are above 10 degrees. Moreover, less than 1% of the angle changes above 20 degrees. Those angle changes between 15-20 degrees also only account for about 5% of all image pairs. Therefore, we can conclude that the topology of the visual representation of the CLIP at changes slowly during the continual CLIP training. In Appendix 7.9 we discuss this conjecture by comparing the representation quality of vision encoders.
In addition discussing the change in the included angle between sample pairs in the visual representation space, by taking the inner product of the same sample's vision representation vector from different training phases' vision encoder EV we use the arecos operation to compute the rotation angles ORAM of each sample in vision encoder F.Y and F. and get the Rotation Angle
4

---

Under review as a conference paper at ICLR 2023
arccos( V V2 >), where the a is the label of sample. The schematic diagram be seen 3(a) By counting the distribution of rotation angles, get the following rotation angle distribution table
Matrix RAM (i.))- The RAM()))
:
V
E
DRAM E
[0", 15"]
(15". 20
(20", 25"]
(25". 30
(30', 180%]
(.
0.00%
0.20%
4.62%
22.68%
72.50%
(.
0.00%
0.40%
8.30%
34.11%
57.20%
(2A.)2..))
0.00%
0.30%
8.40%
34.29%
57.01%
(.A
0.00%
0.00%
1.89%
17.11%
81.00%
@RAM VVV..v))
(.A
0.00%
0.00%
0.00%
3.20%
96.81% %
(.M
0.00%
0.00%
0.00%
0.21%
99. 80%
(a)
(h)
Figure 3: The sub-figure on the left shows a schematic diagram of computing BRAM- The table on the right shows the rotation angle distribution of the same samples in different training phases.
By observing the table 3(b) we can find that the direction of the same sample in the visual represen- tation space of different training phases has changed greatly. Only less than 0.4% samples are rotated within 20 degrees in the continual CLIP training, while the samples rotated within 20-25 degrees are at most less than 9%, and the samples of 25 degrees and above account for more than 90%. We speculate that the vision representation space of CLIP,t: has undergone large rotation around the high-ddmennional sphere center during the continual training. After analyzing the language representation space, we reach the same conclusion as the vision representation space. Detailed SAM and RAM distribution of language encoders can be viewed in Appendix 7.2
According to our analysis of the geometric changes of the single-modal encoder's representation space during continual CLIP training, conclude that: During the continual CLIP training, the representation space the CLIP significantly rotated. The topological structure of the representation space is slightly rotated compared with the rotation of the whole representation space. We name this phenomenon Intra-modal Rotation.
30-180
23
72.5
30-180
20-180
25-30
121
27.05
21-30
25-30
23-26
20-25
20-25
15:20
CLIP
CLIP1
CL/P
30-150
30-180
:Lnguage
20-25
Vision
9:15
D-2b
CIJP3-4
CLI
Figure 4: The comparison of the rotation distributions of the representation spaces of the vision and language extractors during continual CLIP training. CLIP,-; refers to the CLIP's continual training from training phase i to
3.2.2 THE INTER MODEA DEVIATION
Although the topology of the single-modal representation space changes during continual training, this slight rotation should not be the main reason for the degradation of CLIP's multimodal retrieval performance. To this end, we conduct a thought experiment: it is known that the representation spaces of vision and language encoders exhibit significant spatial rotations during continual training. Now we assume that the topology of the single-modal representation space is completely fixed during continual training. Therefore, if the CLIP 's performance multimodal retrieval does not degrade during continual training, the rotations the two encoders' representation spaces should be synchronized. However, the fact is the opposite. So think there is deviation between the rotation of the vision and language representation spaces. Based on this suppose, we compare the rotation distributions of vision encoder (Figure ((b) and language encoder (Table 7(b) and
5

---

Under review as a conference paper at ICLR 2023
draw the rotation distribution comparison diagram (Figure 4). The values under the same color represent the proportion of test samples to total samples in each rotation angle interval of the same modality. Comparing the difference in the distribution of rotation angles of the vision and language encoders, we can see that the space rotations of the two encoders are very different the continual training. The rotation of language representation space is mostly concentrated between 20-30 degrees, while the vision's 's rotations are mostly between 30-18 degrees. This shows that the rotation of the representation space of the two modal extractors within CLIP is not synchronized during the continual training, which verifies our previous inference: The unsynchronized rotation of the vision and language representation spaces leads cognitive deviations between the CLIP's modal encoders (vision and language). We name this phenomenon Inter-modal Deviation.
3.2.3 IOTRA-MODAL ROTATION AND INTER-MODAL DEVIATION LEAD TO COGNITIVE DISORDER
Based the above exploration, play key role in CLIP at cognitive disorder. However, how do they cause the model to misalign the old sample's vision and language representation? We show a schematic here illustrate this. As shown in Figure 5 the is vision representation and 3 is language representation. The a,b denote different image-text samples. For the convenience of illustration, we set the unsynchronous rotation of the two modal spaces as the visual modal's 's static and the language's relative rotation. When intra-modal rotation happens 5(a) Bn. training phase + 1 is rotated to 3% the modal similarity between a and shift from (3% 0a > (Taa,, as)) to (312
conclude that intra-modal rotation and inter-modal deviation
a aa < 38d og)). which break the alignment of the current model to old sample a. The superscript I is a transpose operation that is often used for matrix multiplication. When inter-modal deviation happens 5(b], space breaks the original modal alignment of sample a which makes the (58 ag > S/ aa). Because of this, the performance of CLIP et drops significantly during continual training. Detailed mathematical derivations can be found in Appendix
[(b] the relative rotation of the representation
Phase
Phase E+1:
Pbase
Rg 15
80
Intra-modal Rotation
Inter-modal Deviation
4o
Teaining Data Conain t: Training Data Borain
Scuining
Teaining Data Donain 5-1:
a
(b)
Figure 5: The Schematic illustration of cognitive disorder caused by intra-modal rotation and inter- modal deviation.
4 ALLEVIATING COGNITIVE DISORDER
4.1 GENERAL CONTINUAL CLIP TRAINING SETTING
Suppose have used training dataset Do got a pre-trained model CLIPp- And there is another vasion-l dataset D. We split D into sub-datasets (D,,.., Dx),
randomly and evenly, to simulate a stream of data and D, {(O, (2,44)) ..., (v/ (7)} denotes the training data in the training phase f, where & E {1, ...,, N}. enocded la normalized embeddings vision and text Vi Ei and L; (2.). When the model CLIP, is trained during the training phase using training data D,, the previous sub-datasets (Do, Di...., D are no longer available. The joint training refers to training CLIP, from scratch using all data Dj: Do D.
Then training the model CLIP, using sub-datasets sequentially. The
4.2 MOD-X: MAINTAIN OFF-DIAGONAL INFORMATION-MATRIX
To alleviate cognitive disorder of the CLIP model during continual training. We produce a simple but effective new training framework: Maintain off off-diagonal information-matee (Mod-X). It helps the current CLIP retain the cognition of past samples by aligning off-diagonal information in the similarity matrix constructed by the CLIP, at before and after continual training based on the current

---

Under review as a conference paper at ICLR 2023
training data. The entire training framework is shown in Figure The Contrastive module in Figure is traditional InfoNCE loss Baevski et al. (2020). which inherit from CLIP Radford et al. (2021). In there, mainly introduce Cognition Align module.
Data Stream
Mod-X:
Viston Encoder
Vision Encoder
SO.N
$3.0
2,00
c111
0,2 St1
Cognition Align
(-11
1
200
21
0.N
$1.0
200
Contrestive
Text Encoder
Text Encoder E
Two giraffes
group peoole are playing basebell
stand
Data Stream
together
Figure 6: The Mod-X framework mainly consists two subsections. Cognition Align helps the current model align the cognition of the old model based current data. And Contrative helps the model fit the current training data domain. The 5 means cosine similarity score of the i'th sample vision embedding and the j th sample text embedding.
4.3 COGNITION ALIGN
The diagonal elements in CLIP's contrastive matrix represent the similarity of the model's under- standing of the visual and language information the current sample. The off-diagonal elements represent the similarity between the vision and language representation of the current sample and other samples or vice versa. The distribution off-diagonal elements in the contrastive matrix represents the current model's cognition about the current training objects. So we use Cognition Align to distill the old model's "world view" of current samples to help the current model maintain the cognitive ability on past entities. Firstly, we construct contrastive matrix M,-1 and M, using the last and current model CLIP,-1 and CLIP, based on current sub-dataset D..
M CLIP- s(E (wt), E7, (4()
M = CD.P (1)), (1))
Where the s(a, b) = b the cosine similarity function. However, the last model's cognition for current data is not totally correct. For those misunderstood sample information (diagonal elements are the corresponding similarity information of the current
the largest in the current retrieval), model to replace them. Thereby removing their influence on the current model during distillation.
M
M if man mar(M #
After that, we align the information matrix M which is selected by Screening, using Kullback- Leibler Divergence Csiszár (1975).
M
(R M1-1)= Σ 1n
(4)
M
The final training loss can be joint in LMad-X and a haper-
I To - talkk
7

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTS
5.1 TRAINING DATASETS
In the experiments. we use three image-text datasets varying in scale and scope to simulate continual CLIP training. MS COCO Captions Lin et al. (2014): MS COCO Captions (COCO) is a widely used image caption dataset. It contains 80K training images, 30K validation images, and 5K testing images (COCO(5K)). Each image has 5 human-evaluated captions. Flickr30K Young et al. (2014); (F33)) contains 30K training images and IK test samples (Flickr30K(IK) collected from Flickr, together with 5 reference sentences provided by human annotators. SBU-1M Ordonez et al. (2011): The SBU-IM collects 1M images from Flickr with associated visually relevant captions which representing wider variety of content styles. After downloading and preprocessing. utilized about 0.8M image-text pairs in our experiments.
5.2 EXPERIMENTS SETTING
We apply our Mod-X framework to two experiment settings to illustrate that Mod-X helps the model not only better fits the newly trained data domain but also maintains the multimodal cognitive ability of the old data domain during the continual large-scale training. Experiment A: The Experiment A follows the setup of the exploratory experiments, pre-training the CLIPo model using the COCO dataset. And then splitting Flickr30K dataset randomly and evenly into five sub-datasets and sequen- tially training them to update CLIP, continually. Experiment B: Unlike Experiment A, which looks like fine-tume min-ttning. we try to simulate continual large-scale training in Experiment B. We train a CLIP from scratch using the joint data of COCO and Flickr30K (COCO+F30K) and continually train eight sub-datasets divided from the SBU-IM M dataset, randomly and uniformly. Each sub-dataset contains 100K unique samples, an equal scale to the initial pre-training dataset (COCO+F30K), This experiment verifies that our framework can iterate under continual large-scale training and maintain even improve the model's cognition of past entities. Evaluation Details: We use the test sets COCO(5K) and Hlicrr3O<(1]) to evaluate the performance of model in continual training. And all the results the tables and figures are the average of 3 runs with different seeds. Experiment details have been shown in Appendix 7.4
the initial model CLIP,
5.3 PERFORMANCE OF MOD-X X IN THE EXPERIMENT A
The Experiment A focuses on verifying that Mod-X can not only fit new data domains but also maintain a cognition of past data domains. This table shows the final multimodal retrieval performance of the different training strategies on COCO(5K) and Flickr30K (IK.. The CLIP, is the initial pre- trained model based on the COCO dataset. The CLIP ct means training CLIP continually without any other operation. The CLIP EWC means using the EWC method Kirkpatrick et al. (2017). 7). which is typical regularization strategy in continual supervised learning. The CL.IP x the proposed Mod-X, and the CLIP is the joint training model using the joint datasets (COCO+F30K) which an upper bound for continual CLIP training in Experiment A. The detailed performance of the different models at each training phase is shown in Appendix 7.5
Image- Text Retrieval(S)
Text-Image Retrieval(
Pretraining Dataset
Model
Flickr30K(1K)
COCO(5K)
Flickr30K(1K)
COCO(SK)
R@1
R@5
R@10
R&1
Rã5
R310
R@1
R@5
R@10
R@1 R
R@10
CLIPo
16.9
37.0
46.2
14.7
34.2
47.0
12.0
30.0
41.0
10.6
29.6
41.0
CLIIP
20.6
42.8
56.4
6.2
17.8
26.1
16.1
38.5
50.4
4.7
14.3
21.8
COCO
CLIIP
22.2
43.1
57.0
6.1
17.2
26.5
17.0
39.1
51.2
4.5
13.9
22.0
CL.PP Mod x
27.9
53.4
64.4
14.5
34.0
46.1
20.2
45.0
57.2
10.1
26.4
37.4
COCO+F30K ,L.P,,
30.1
55.9
60.1
16.1
38.1
51.9
22.5
48.5
59.6
11.2
30.9
42.7
Table 1: The final multimodal retrieval performance of the difffrrnnt continual CLIP training strategies in the Experiment A.
From the results in the Table [ it clear that our method CLIP ood-X maintains its multimodal retrieval results COCO(5K) after completing continual training on Flickr30K. The gap between
8

---

Under review as a conference paper at ICLR 2023
CLIP, and CLIP Mod-ModdX Mod-X is just 0.2% points in image-text retrieval and 0.5% points in text-image retrieval on COCO(5K). At the same time, the retrieval results of the ModP x on the test set (I)c are also affected by the training domain and have significant increase. The R@1 performance of the CLIP odx in image-text retrieval rise from 16.9% (in CLIP)) ) to 27.9%. And the R@1 results in text-image retrieval increase from 12.0% (in CLIP)) to 20.2%. The performance gap between CLIP Mod-X and CLIPje je on the Flickr30K is only at most 2.3% points. Conversely, due to the model's cognitive disorder in continual training, the performance of CLIP at on COCO(5K) drops significantly. In addition, although the performance of CLIP&t on Flickr30K(1K) has improved, it is still far from the upper bound CL The EWC, typical regularization strategy continual learning, selectively updates the model by evaluating the importance of each parameter of the model. From the above experimental results, although CLIP ELIPEWC EWC improves the accuracy of continual CLIP training on Flickr30K(1K)) K), it does not preserve the model's s understanding in past samples (COCO(SK)). According to the above comparisons, we can conclude that our Mod-X framework can not only maintain the cognitive ability past samples during continual CLIP learning but also improve the model's 's fitting ability to the current training data domain.
5.4 PERFORMANCE OF MOD-X X IN THE EXPERIMENT B
Unlike Experiment A, which looks like fine-tuning, Experiment B focuses on verifying that our framework Mod-X can maintain or even improve the model's 's understanding of past entities when iterating under continual large- scale training. Table[ssoowe the final multimodal retrieval performance of the different continual CLIP training strategies in Experiment B. The detailed R@1 results in each training phase can be seen in Appendix [7.6
Image- Text Retrieval(%)
Text-Image Retrieval %))
Pretraining Dataset
Model
Flickr30K(1K))
COCO(5K)
Flickr30K(IK)
COCO(SK)
H@1 H@5 R@10
R@1 R@5 R@10
R@1 R05 R 10 [@0 R@5
R 10
CL.PPo
33.0
56.5
67.3
17.7
40.7
52.9
23.4
48.2
59.1
12.9
32.8
44.9
CL.IPat ct
5.4
16.0
23.3
3.0
9.0
14.0
3.6
12.0
18.6
2.3
7.3
11.4
COCO+F30K
CLIP ewc
6.1
18.2
33.5
3.1
11.7
18.2
3.3
10.4
16.2
2.1
5.4
10.0
CLIP Mod
37.3
62.6
72.1
19.5
42.8
55.7
26.0
51.9
62.6
13.4
33.6
45.3
Joint Datasets CLIP,r yt
39.3
71.5
81.1
25.1
51.6
64.3
27.8
54.9
66.5
15.9
38.6
51.3
Table 2: The final multimoal retrieval performance of the different continual CLIP training strategies in the Experiment B.
ModP Compared to the initial pre-trained model CLIP,, CLIP Mod-X x not only does not show signifi- cantly drop such as CLIP& in multimodal retrieval performance on COCO(5K) and Flickr30K K) but it also shows slight improvement after continual training in SBU-1M. The image-text R@1 results on Flickr30K(1K] increase from 33.0% to 37.3% and the accuracy on COCO improved to 19.5% from 17.7%. The performance of text-image R@1 I is the same as that of image-text. The accuracy increase from 23.4% to 26.0% on Flickr30K(1K) and 12.9% to 13.4% on COCO(5K). The gap between continual CLIP training and joint training has been somewhat narrowed. Conversely, the ability of cognitive alignment in CLIP.t and CLIP FWC are lost in the large-scale data. This results in verifying that our framework Mod-X can iterate under continual large-scale training and can maintain or even improve the model's cognitive ability of past entity concepts.
6 CONCLUSION
In this paper, We explore the feasibility of training the CLIP model continuously through streaming data and name its performance decline in multimodal retrieval as Cognitive Disorder(CD). Then, By tracking the directional changes of the representation vectors in the continuously updated CLIP model, we explore and summarize the spatial variation of the modal encoders within the CLIP: Intra-modal Rotation and Inter-modal Deviation. Moreover, we mathematically demonstrate how intra-modal rotation and inter-modal deviation lead to CD. To alleviate the cognitive disorder of the continual CLIP training, propose simple yet effective continual learning framework Mod-X: X: Maintain off- diagonal information-mann The results in experiments (5.3 and 5.4) demonstrate the effectiveness of our framework.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty- -baee continual learning with adaptive regularization. Advances in neural information processing systems, 32, 2019.
Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss- il: Separated softmax for incremental learning, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 844 853, 2021.
Alex Andonian, Shixing Chen, and Raffay Hamid. Robust cross-modal representation learning with progressive self-distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16430-16441, 2022.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning speech representations. Advances in Neural Information Processing Systems, 11122244914666 2020.
Magdalena Biesialska, Katarzyna Biesialska, and Marta Costa-Jussa. Continual lifelong learning in natural language processing: A survey, arXiv preprint arXiv:2012.09823, 2020.
Sungmin Cha, YoungJoon Yoo, Taesup Moon, et al. Ssul: Semantic segmentation with unknown label for exemplar-based dass-incremental learning. Advances in Neural Information Processing Systems, 34:10919-10934 2021.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3566 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMIR, 2020.
Jishnu Ray Chowdhury, Yong Zhuang, and Shuyi Wang. Novelty controlled paraphrase generation with retrieval augmented conditional prompt tuning. arXiv preprint arXiv:2202.0053: 2022.
Imre Csiszár. I-divergence geometry of probability distributions and minimization problems. The annals of probability, pp. 146-158, 1975.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. continual learning survey: Defying forgetting classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-33 3385, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A Barge-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 leee, 2009.
Zhihao Fan, Zhongyu Wei, Jingjing Chen, Siyuan Wang, Zejun Li, Jiarong Xu, and Xuanjing Huang. A unified continuous learning framework for multi-modal knowledge discovery and pre-training. arXiv preprint 2rX.v:2206.05555, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733. 2020.
Dapeng Hu, Shipeng Yan, Qizhengqiu Lu. HONG Lanqing, Hailin Hu, Yifan Zhang, Zhenguo Li, Xinchao Wang, and Jiashi Feng. How well does self-supervised pre- trainiain perform with streaming data? In International Conference Learning Representations, 2021.
10

---

Under review as a conference paper at ICLR 2023
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinsker et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3524) 2017.
Joseph Kj. Jathushan Rajasegaran, Salman Khan, Fahad Shahbaz Khan, and Vineeth Balasubrama- nian. Incremental object detection via meta-learming IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Hyungtae Lee, Sungmin Eum, and Heesung Kwon. Exploring cross-domain pretrained model for hyperspectral image classification. IEEE Transactions Geoscience and Remote Sensing, 60: 1-12, 2022.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208, 2021.
Tsung-Yr Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:111..0500,, 2017.
Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representa- tional continuity for unsupervised continual learning. In International Conference on Learning Representations, 2021.
Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Marzieh Saeidi, Lambert Mathias, Veselin Stoyanov, and Majid Yazdani. Perfect: Prompt-free and efficient few-shot learning with language models. arXiv preprint arXiv:2204.01172 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Zixuan Ni, Haizhou Shi, Siliang Tang, Longhui Wei, Qi Tian, and Yueting Zhuang. Revisiting catastrophic forgetting in class incremental learning, 2021a.
Zixuan Ni. Siliang Tang, and Yueting Zhuang. Self-supervised class incremental learning. arXiv
preprint arXiv:2 11::2211..11 1208, 2021
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 07770000080 2018
OpenAl. Clip. bettps://gichu..ccmm/oo
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: captioned photographs. Advances in neural information processing systems, 24, 2011.
Describing images using million
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. PMLR, 2021.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-20100 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2 1..02/ 14, 2021.
11

---

Under review as a conference paper at ICLR 2023
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, 2018.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. arXiv preprint arXiv:2209.0755 1, 2022.
Tejas Srinivasan, Ting- Yun Chang, Leticia Leonor Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for vision-and-land -anguage tasks. arXiv preprint WiXiv:220.090554 2022.
Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Yi Lee. Lamol: Language modeling for lifelong language learning. arXiv preprint 0at::::9990033323 2019.
Anh Thai, Stefan Stojanov, Isaac Rehg, and James M Rehg. Does continual learning= catastrophic forgetting? arXiv preprint 1oXx::2001.07299 2021.
Sebastian Thrun. A lifelong learning perspective for mobile robot control. In Intelligent robots and systems, pp. 201-214, Elsevier, 1995.
Liyuan Wang, Kuo Yang, Chongxuan Li, Lanqing Hong, Zhenguo Li, and Jun Zhu. Ordisco: Effective and efficient usage of incremental unlabeled data for semi-supervinn continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5383-5392, 2021a
Renhao Wang, Hang Zhao, and Yang Gao. Cyborgs: Contrastively bootstrapping object representa- tions by grounding in segmentation. arXiv preprint urXiv:2203.09345 2022a.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for sulf-sude visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021b.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 139-149, 2022b.
Longhui Wei, Lingxi Xie, Wengang Zhou, Houqjang Li, and Qi Tian. Mvp: Multimodalit-guideed visual pre-training. arXiv preprint arXiv:2203. 05175, 2022.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8392-8401, 2021a.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16684-16693, 2021b.
An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, Di Zhang, Wei Lin, Lin Qu, Jingren Zhou, and Hongxia Yang. M6-T: exploring sparse expert models and beyond. CoRR, abs/2105. 1500222,5500,, 2021.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Li Zhang, Dan Xu, Anurag Arnb, and Philip HS Torr Dynamic graph message passing networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3726-3735, 2020.
12

---

Under review as a conference paper at ICLR 2023
7 APPENDIX
7.1 THE THEORETICAL DEMONSTRATE THAT INTER-MODAL DEVIATION AND INTRA-MODAL ROTATION LEAD COGNITIVE DISORDER
Inter-modal Deviation and Intra-modal Rotation can influence the CLIP's sample similarity matrix, but this does not necessarily lead to errors in multimodal retrieval results. Unless the similarity of the visual language representation of the model for the same sample is smaller than that between different samples. In there, abstract this problem and give the thcoretical conditions that the Intra-modal Rotation and Inter-modal Deviation leads to the cognitive disorder.
There has N image-text pairs
(23,),)) (.., R xW Through function M(a) and ((5), M & Q, the Euclidean space A and B of images and texts are formed.
= Mannl
(6)
B span| (() Q(8)....
The M(a,).0(3,) Rº and ||M))))] 1 1 = ij
1,2, N.
<
M(as). ((3,) > is the cosine between M(a;) and = 1,,,,,......
Suppose: 3(0.. Ba), (b) B,),,, 1,2,3, .N and & makes:
< M(aa). 2(3a)= >
arg < M(aa), 2(3.)
$,=9a
< M(as), 2(3))
arg max (Maa),,(()) 8,
7.1.1 INTER-MODAT. DEVIATION LEADS To COGNITIVE DISORDER
Prove: There a rotation matrix pair (A,B) that not only keeps the A and topology unbiased and makes the
M'(aa), arg < (M(()))
(8)
<a),,,), ('())>
arg max < M((a),); ((()) R,=3,
where the M' = A(M) and Q' = BB()), A B. And the space A and B can be written as A' and B':
A' = A(A) =
(9)
B' (B span(Q (52), ('8)
Solution: the Equ 7 can be written as:
<Maaa), Q(3.) > < M(aa), ((8)>>0088.,,,
(10)
M(as), 2(%) > < M(os), 2(3,) >< &
hence:
M(a.)) M(a.)'"Q()) 0,43, E B,i M 2(3,) (M(a))2((33)) > 0,33, b
(11)
because the rotation matrix pair (A,13 can be seen rotation matrix R(0P), where the 0 is rotation angle between AB and A'B'. Hence, when applying this rotation matrix R(O?), the Equ be written as:
13

---

Under review as a conference paper at ICLR 2023
M(a.)"R(09)00 M(a.) M < e 6 M(ar)R((0)22(33) - M(aa))R((") ((()) < 0, v3, B
(12)
Because the rotation matrix satisfies that the inner product of itself So. Equ 12 can be written as:
M(a.) (cc.) < e
=I
(13)
M(as)" (Q0)) - -(61)))))33)) <0, e 3.j b, 7((0)))
(14)
For example, when R(09) = -I, then R( (300)=((aa)) = -M((aa) the equ 15 and 16 will hold.
So, rotation matrices (A.B) that makes Equ 8 true exists.
7.1.2 INTRA-MODAL ROTATION LEADS TO COGNITIVE DISORDER
Since mntra-modal rotation just requires the length of representation vectors after rotation is 1 and does not require that the intra-modal representation space is invariant, a more general case of inter-modal deviation. This means that all rotation matrixes that satisfy [7..]] also satisfy Intra-modal Rotation. Different from intra-modal deviation, the inner product of the mapping matrix does require be 1. So, we rewrite the Equ and 16 to:
M(a.))
2(3))) < -())))))0)))()))0022220 38 EBi
(15)
- 0(3.)) (((3))
0, E
(16)
Any mapping matrix P that rotation the direction of (2(3,) ((3))) by more than 90 degrees.
7.2 DETAILED SAM AND RAM DISTRIBUTION OF LANGUAGE ENCODERS
The topological structure of the language representation space does not change significantly with the continual CLIP training. But the whole language representation space, like the vision representation space, has large rotation around the center of the high-dimensional sphere during the continual training. The angle change distribution table (a) a) and rotation angle distribution table 7 are shown below.
OSAM [0).5%
109]
(10", 13%]
(15", 20%]
(20%. 180%]
SAMo-1
64.43%
28.49%
6.23%
0.78%
0.07%
SAM1-2
71.54%
24.89%
3.35%
0.22%
0.01%
SAM3-3
71.36%
25.01%
3.40%
0.22%
0.01%
SAM3
67.30%
27.27%
4.93%
0.48%
0.03%
SAM-
58.84%
30.70%
8.77%
1.50%
0,20%
SAM
55.39%
31.60%
10.52%
2.15%
0.33%
(a) The angle difference distribution in text representation space.
DRAM E
10" 15%)
(15", 20"]
(20", 25"]
(25",30%)
(30", 180%]
(A.
0.00%
1.94%
28.38%
45.88%
23.80%
HAM((12)
0.02%
8.90%
47.76%
34.94%
8.38%
RAM22)))
0.04%
1.14%
49.86%
31.18%
7.52%
RAM((..)))
0.02%
2.84%
33.70%
43.76%
19.68%
RAM(4..))
0.00%
0.04%
3.28%
27.66%
69.02%
(AM((..)
0.00%
0.00%
0.00%
1.12%
98.88%
(b) rotation angle distribution text representation space.
Figure 7: Detailed SAM and RAM Distribution of Language Encoders.
By observing the table in Table 7 find that more than 88% of the angle change between any two language representation vectors the language representation space are between 0 and 10 degrees in the process of continual CLIP training, while only 20% above 10 degrees. Moreover,
14

---

Under review as a conference paper at ICLR 2023
less than 0.2% of the angle changes is above 20 degrees. Those angle change between 15-20 degrees also only account for about 1.5% of all images pairs. Similar to the visual representation space, the direction of the same sample in the language representation space of different training phases also has changed greatly. However, unlike most of the rotations in the vision representation space, which distributed over 30 degrees, in the language space, the rotations in the representation space mostly distributed between 20 and 30 degrees, Because of this difference, the alignment of the CLIP for different modalities of the same sample deviates during the continual training.
7.3 THE RELATIONSHIP BETWEEN CONTRASTIVE MATRIX, INTRA-MODAL ROTATION, INTER-MODAL DEVIATION AND MOD-X
From a detailed point of view, the element Ma,5 in the i.j position of the contrastive matrix M the similarity score of the i'th sample vision embedding and the j sample text embedding, Since the length of the representation vector is 1, the similarity score Mi, also refers to the angle between the i'th sample vision embedding and the j'th sample text embedding. Greater similarity means smaller angle. Therefore, the value of the diagonal elements in the contrast matrix M represents the angle between different modals of the same sample. The value of the off-diagonal elements represents the angle between the different modals of different samples in the CLIPP's representation space. Through our exploration (in section 3), the Intra-modal Rotation and the Inter-modal Deviation affect these angles or similarity scores. From an overall perspective, the similarity distribution of the contrastive matrix M is equivalent to the structure of the representation space of the model. Our Mod-X framework attempts to distill the similarity distribution of off-diagonal elements identical to distilling the model's representation space structure, which reduces the influence of intra-modal rotation and inter-modal deviation during continual CLIP training.
To better illustrate the relationship between the model's representation space and the model's 's similarity performance, we add more direct statistical analysis, inter-modal angle variation distribution. Based on the settings in section 3 in the training phase 6, we compare the change of angle distribution between modalities for the training samples retrieved correctly in the training phase t 1- A schematic diagram of inter-modal angle variation I AV is shown in Figure 8(a) where the sample a refers to the training sample that be retrieved correctly by model CLIP--1 in training phase - 1. The V is the vision representation and L is the language representation. Inter-modal angle variation distribution table can be seen in Figure 8(b)
Eg
ve
Eet1'
V++1
OIMAY E | (5".
(15), 20 (20", 180%]
TmAV
44.78% 31.54%
14.62%
8.26%
0.81% %
11
(nAV
50.37% 28.48%
16.58%
4.57%
0.00%
ImAV((2.))
49.70% 24.22%
20.53%
5.13%
0.42%
1
ImAV(3.4)
46.25% 46.25% 30.12%
18.53%
4.81%
0.29%
IIAA = -LV2, k2) <(V 12-1)
ImAV(4.5)
43.36% 32.83%
19.81%
3.82%
0.18%
ImAV
(0.5) 31.98% 33.62%
24.37%
10.01%
0.02%
(b)
Figure 8: The sub-figure on the left shows schematic diagram of computing mA The table on the right shows the distribution of the change of the included angle of the vision and language representation of the samples, which were correctly retrieved in the previous training phase.
As shown in Figure 8(b) during the continual training, the samples that were correctly retrieved in the past have apparent changes in the angle between the modalities as the training phases go up. Only less than 50% of the samples change within degrees in the continual training, and about 30% of the samples have change of 5-10 degrees. However, more than 20% of the samples change their included angle by more than 10 degrees during the training process. This shows that the inter-modal spatial alignment (similarity performance) of the CLIP et is affected by Intra-modal Rotation and Inter Inter-modal Deviation.
To illustrate our Mod-X X framework indeed alleviates the distribution shift in representation space between sample modalities during continual training, we show the inter-modal angle variation distribution of the CLIPMadd- x in Experiment A in Table
Comparing the Figure 8(b) and Table it can be found that the CLIP ooo x well maintains the inter-modal spatial alignment of the correctly retrieved samples during the continual CLIP training.
15

---

Under review as a conference paper at ICLR 2023
0IMAM
[0)
| 15
10"]
(1 20")
(20", 180%)
(m.M(0.1)
88.66%
7.81%
2.56%
0.97%
0.00%
ImAM(12))
91.79%
4.01%
3.20%
0.00%
0.00%
ImAM((233)
90.70%
9.02%
0.24%
0.04%
0.01%
ImA.,s..4
92.13%
6.20%
1.61%
0.06%
0.00%
(m.)(
91.91%
7.71%
0.38%
0.00%
0.00%
(0.5 87.81%
10.87%
1.12%
0.20%
0.00%
Table 3: The table shows the distribution of the change of the included angle of the vision and language representation of the CLIP Mod-X in Experiment A.
average, 90% of the correctly retrieved samples have an angle change of less than degrees in continual training, and the samples with angle change of more than 15 degrees account for less than 1% all samples. All of this shows that the Mod-X framework mitigates the cognitive disorder during continual CLIP training by preserving the inter-modal spatial alignment of the samples retrieved correctly in the past during the continual training.
7.3. VALIDATION OF INTER-MODAL DEVIATION ON DIFFERENT TRAINING DATASETS
In section we discuss representational space variation of CLIPat under the open- dataset Coco et al. (2014) and Flickr30K et al. (2014). In there, following the explore settings of the section 3.2.2, we compare the rotation distribution of the representation space of the vision and language extractors during the continual CLIP training dataset ECommerce-TD) Yann et al. (202T) The ECommerce-1221 contains 90k training images and 5k testing images set. Each image corresponds to a description, and the descriptions of training set and test set do not overlap. We splitting ECommerce- T21 training dataset randomly and evenly into five sub-datasets and sequentially training them update pre-trained model CLIPo with ViT-32/B OpenAl continually. By evaluating the rotation distribution of the modal's 's representation space at various training phases on the COCO(5K) testset, we drawn the rotation distribution comparison diagram in Figure
specific e-commerce text to image
36-10
30430
29.32
09.90
29.39
16:08
26.26
15:33
10:26
15:20
CLiPo11
CLIP1-2
CLIP
25-39
Language
29-25
15-29
VVision
CLIP3-
C1.1P
Figure 9: The comparison of the rotation distributions of the representation spaces of the vision and language extractors during continual CLIP training on ECommerce- T2I. CLIP:-, refers to the CLIP's continual training from training phase to j.
From Figure 9 we can find that when the CLIP trained on specific data domain, the rotation of visual representation space becomes more severe, among which more than 70% of the samples have more than 30 degrees of rotation in the visual space, which is higher than that of the open-word world dataset. Although the rotation of more than 30 degrees in the language space has also seen large proportional increase than the open-world dataset, it still significantly out of sync with the rotation in the visual space. Most samples rotated within 30 degrees in language space. Through this validation, we show that inter- mnter-modal deviation (rotational asynchrony) of the representation space of different modal encoders persists during the continual CLIP training specifie data domain.
16

---

Under review as a conference paper at ICLR 2023
7.4. DETAILED EXPERIMENT SETTING
In exploration experiments 3 and main experiments 5 we use RN50 He et al. (2016) as the vision en- coder and language encoder transformer-based architecture which follows modification proposed in CLIP |penAA] The input images are resized to 224 x 224 and the input texts are tokenized by WordPiece with a maximum length 77. We utilize AdamW W Loshchilov & Hutter (2017) optimizer and cosine annealing learning rate schedule with warmup which is consistent with OpenAl All of the experiments are conducted on 8 NVIDIA V100 GPUS. In exploration Experiment and Experiment as be shown in table ((a) Since the size of training data in our exploration experiment and experiment A is relatively small compared to large-scale pre-training, we set smaller batch size. And other hyper-parameters is consistent with CLIP OpenAl
A, we use the
(a)
(b)
Hyperaarameter
Value
Hyperparameter
Value
Batch size
280
Batch size
800
Vocabulary size
49408
Vocabulary size
49408
Training epochs
35
Training epochs
35
Initial temperature
0.07
Initial temperature
0.07
Q
20
20
Weight decay
0.2
Weight decay
0.2
Warm-up iterations (%)
20
Warm-up iterations (%)
20
Learning rate
5e
Learning rate
le
Adam 31
0.9
Adam 31
0,9
Adam B2
0.99
Adam $3
0.99
Adam e
1e
Adam e
1e"
Table 4: Table (a) is the hyperparameter in exploration experiment (Section 3) and Experiment A (Section 5.3 Table (b) is the hyperparameter in Experiment B (Section 5.4)
In Experiment B, due the size of training data reaches million, we increase the batch size to 800 and increase the initial learning rate to le 3, And other hyper-parameters is consistent with Experiment A. The detailed syper-parameteed table ((b)
7.4.1 THE SENSITIVITY OF HYPER-PARAMETE
In this section, we discuss the effect of different on the final performance of the CL.IP Mod-X based on the settings Experiment A 5.3 Table presents the final retricval results the Mod-X model with 10, 15, 20, 25, 30.
Image-Text Retrieval(%)
Text- Image Retrieval(%)
Pretraining Dataset
Model
Flickr30K(1K) R&11
COCO(SK)
Flickr3OK(IK)
COCO(5K)
R&5
R@10
R&1
R@5
R@10
R@1
R
R@10
R&1
R@5
R@10
CLIP
16.9
37.0
46.2
I1.7
34.2
47.0
12.0
30.0
41.0
10.6
29.6
41.0
CLIP ct
20.6
42.8
56.4
6.2
17.8
26.1
16.1
38.5
50.4
4.7
14.3
21.8
10
25.7
50.4
60.3
11.6
28.4
30.9
17.3
40.2
54.6
20.9
34.7
COCO
28.1
54.3
66.7
14.0
32.8
45.4
20.7
45.8
58.0
26.0
36.4
a 20
27.9
53.4
64.4
14.5
34,0
46.1
20.2
45.0
57.2
10.1
26.4
37.4
25
26.6
52.8
62.3
14.5
34.8
46.7
20.2
44.7
57.0
10.0
27.7
38.1
30
25.5
51.7
61.8
14.7
35.0
47.1
18.4
42.8
55.5
10.2
27.0
38.3
COCO+F30K CLIP, ya
30.1
55.9
60.1
16.1
38.1
51.9
22.5
48.5
59.6
11.7
30.9
42.7
Table The final multimodal retrieval performance of different a on continual CLIP training in the Experiment A.
From the table, different a does not significantly affect the effectiveness of the Mod-X framework. The perfor- mance of CLIP Mad- x is better than CLIP, under different a. As increases, the CL.P Mod-X
find that although different affects the performance of the CLIP Mod-X.
17

---

Under review as a conference paper at ICLR 2023
better maintains its retrieval abilitt on past COCO samples. The Image- @ @1 and Text-Image R@I on ( remain around 14.5% and 10.0%. However, an excessively large a also limits the model'ss ability to fit new datasets. With the value of increased from 20 to 30, the Image- @ and Text-Image R @ 1 of the Mod-x on the Flickr30k(1K) drops from 27. 27.9% and 20.2% to 25.2% and 18.4%.
7.5 THE DETAILED PERFORMANCE OF THE MODELS AT EACH TRAINING PHASE EXPERIMENT A
In figure 10. we show the effect of our framework Mod-X (CLIP x in each training phase and compare the performance with continual CLIP training strategies in Experiment A 5.3 From the
Teate-Text R results on COCO/SK)
lext- R@l results on COCO/5K)
R@l Accuracy
21
20
Clip CUPe
CLufz
CLP
Inta'
Mases
Inbba
Ahades
Training Phases
Training Phoses
Image-Text R@l results on FKick/30K/1K)
Text-image A@1 results on Flickr30K(1K)
20
R@l Accuracy
Accuracy
22
26
CLP
12
20
Intee
Phone;
Phone:
Pharey
Pranne
Phassey
Inbal
Phanes
Phane
Prans:
Phane
Training Phases
Training Phoses
Figure 10: The performance of different training strategies in each training phase in Experiment A.
results of multimodal retrieval at each training phase, we can find that our framework, Mod-X, still has good performance on the past pre-training dataset COCO(SK) during the continual training on the Flickr30K dataset. At each training phase, the R@1 results of CLIP Mod-X x on COCO(5K) did not show a significant drop, and the gap with the initial accuracy (CLIPo) remained at +1%. In addition to this, by comparing the retrieval performance of the CLIP or and CLIP Mod-X on the current training data domain (Flickr30K), it can be found that the CLIP better than CLIP ct in continual fitting the current data domain. The low performance of traditional regularization methods EWC also shows that continual multimodal training is more complex than mogle-mmdal supervised training.
Mod- x is also significantly
7.6 THE DETAILED PERFORMANCE OF THE MODELS AT EACH TRAINING PHASE EXPERIMENT B
In figure II we show the effect of our framework Mod-X (CLIP Mod-x) in each training phase in large-scale continual training settings (the Experiment B 3..4.
Comparing the R@1 results of the three continual training strategies each training phase, we can clearly see that our Mod-X framework performance in each training phase is stabilized. As the training phase continues to rise, the performance of the Mod-X improves relative to the initial pre-training results (initial). The Image-Text and Text- R@1 results on COCO(5K) had risen by 1.7% and 0.5% points, respectively. And the gap of Image- Text R@l results on the Flickr3K<IIK) between CLIP, je and CLIParod x narrowed from 6.3% to 2.0%. The performance of Text-Image R@1 on Flickr30K(1K) also improved to 25.98% from 23.44%. Conversely, the ability of cognition alignment in CLIP and CLIP are lost in the continual large-scale training. This results in verifying that our framework Mox-X can iterate under continual large-scale training and can maintain even improve the model's cognitive ability of past entity concepts.
18

---

Under review as a conference paper at ICLR 2023
Image-Text COCO(SK)
Text-lmage R@1 COCO/5K)
R@i Accuracy
Age Accuracy
CUP CUPPE
CUP CLPAN CLPa
CLiee:
Plase, Prass, Franry Phane, Franss Phann, Phanay Phose;
cthe Phaze, Phase, Prane, Phose, Phane, Phare, Marryy Phair
Training Phases
Training Phases
Image-Text R@l results Flickr30K/1K)
Text-lmage A@l results Flick330K11))
32
R@01 Accuracy
32
Accuracy
24
20
CU
CUPe
CLipe
CLP
Paare Prase Franry Phane, Frans, Phaen, Phane, Phane
otha Praze, thane, Prane, Phace, Phane, shane, Phaney Ptam Training Phases
Training Phases
Figure 11: The performance of different training strategies in each training phase in Experiment B.
7.7 THE PERFORMANCE OF THE MOD-X WHIEN TRAINING IN CC12M DATASET
In this section, show the performance of different continual training strategies in ICI2MCChap pinyo et al. 2021) training dataset. The CC12M training dataset collects about 12M images and their raw descriptions harvested from the alt-text HTML attribute associated with the webscraped images, therefore representing wider variety of content styles. Due to unavailable URLs, we utilize about 1OM examples from this dataset. Firstly, we randomly and evenly split the CC12M dataset into 10 sub-dat sub-datasets, each containing 1M image-text pairs. Then, we continuously train a CLIP based on these sub-datasets from scratch without any pre-training. The purpose of this experiment is to demonstrate that the Mod-X framework still excels in large-scale continual pre-training. In ta we show the final retrieval performance of different continual training strategies in COCO(SK) and Flickr30K(1K) test sets. The CLIP ct means continual training without any other operations. The CLIP Mod-X means continual training using our Mod-X framework. And the CLIP,, refers to training CLIP model using the joint dataset CC12M.
Image-Text Retrieval(%)
Text-Image Retrieval
Model
Flickr30(1IK)
COCO(SK)
Flickr30k(IK)
COCO(SK)
R@1
R@5 R@10
R@1 R@5 R@10
R@1 R@5 R@10 R@1 R@5 R@10
CLIP
35.50
64.80
76.10
17.38
39.24
51.68
24.54
49.96 61.44
12.10
29.60 40.26 33.68
CLIPMod-x Mod-X
40.40
67.90
77.40
22.06
46.12
58.14
27.74
53.88
64.66
14.22
45.02
CLIP,
58.00 83.90 90.40 34.38
60.30 71.50 43.02 72.34 80.92
22.63 46.44 58.35
Table 6: The final multimoal retrieval performance of the CLIPat, CLIPMor-X x and CLIP,; on COCO(5K) and FIKc
Comparing the final performance of the three training strategies, Mod-X framework (CLIP M Mod-x) still outperforms CLIPat in the large-scale pre-tnain After continual pre-training, the CLIP Mod-X obtain 40.40% Image-Text @@ 1 result and 27. 37.44% Text-Image R@1 result on Flickr30K ()) test set, which surpasses the 35.55 5,500 and 24.54% of CLIP The results on COCO(5K) are similar to those on Flickr3OK(IK). The Image-Tex Text R@1 result of CLIP x on COCO(5K) is 4.68% points higher than CLIP at and the Text-Image R@I result of ModPMod-X x COCO(5K) exceeds CLIP at 2. 12% points. The detailed R@1 performance of three training strategies at each training phase can be seen in Figure 12
19

---

Under review as a conference paper at ICLR 2023
Image-Text COCO(SK)
Text-image COCO(SK)
RdB Accuracy
R Acourary
CL
CUPaee-
CLV4
inte muse, Plase, Modey Plaacy naso, rhasyy
Plise, Hasty
Muss, madyy Maady
Training Phases
Training Phases
Image-Text R@l results Flckk30K(IK)
Text-image R@l results Flcck33K((K)
42
Accuracy
38
31
29
31
21
22
15
intt Pnase, Plane; Paatry Phase, Power Peass, Ataces Training Phases
Netal Phase, Fherr, Phane, Phank, Training
The retrieval performance of different training strategies in each training phase on COCO(5K) and Flickr30K(IK).
Figure 12:
7.8 THE PERFORMANCE OF MOD-X WHEN FINE-TUNING THE OPENAI'S CLIP
In this section, we simulate continual CLIP training based on OpenAl's pre-trained model CLIP with ViT-32/B vision encoder OpenAl
7.8.1 THE PERFORMANCE OF MOD-X WHEN FINE-TUNING THE OPENAI'S CLIP ON COCO AND FLICKR30K DATASET
We set the CLIP ma232 the initial model in the continuous training process and divide the joint- dataset (COCO and Flickr30K) into five sub-datasets uniformly and randomly to simulate streaming data. Because the pre-training datasets of CLIPP vit32 are not available, we train CLJP vir32 on the joint-dataset to get the model CLIP ft as an upper bound for the performance of continual training. We apply our framework Mod-X in this setting and compare the final multimodal retrieval results with CLIP ct which is just continual training without any other operations, in Table
Tmage-Text Retrieval (%)
Text-Image Retrieval(%) Flickr30k(1K)
Model
Flickr30k(1K)
COCO(5K)
COCO(5K)
H@1 1 R@5 R 10
R@1 R85
R
10
R81 R@5
R@
10 R@1 R@5
R 10
CLIP 15132
77.7
94.5
98.3
50.1
74.6
83.0
58.9
83.5
90.1
30.2
55.6
66.7
CLIP ot
85.6
97.3
98.8
59.7
83.2
90.2
71.2
91.5
94.9
43.5
70,9
80.6
CLIP Mod-
86.9 97.7
99.3
62.1
85.6
91.7
73.4
92.9
96.2
46.2
73.5
82.6
CLIP
86.3
97.2 99.1
63.6
86.4
92.3
72.7
92.6
96.3
46.3
73.1
82.3
Table 7: The final multimoal retrieval performance of the CLIP CLIP and CLIP based on OpenAl's CLIPpitt2 with VIT-32/B vision encoder.
The performance of our framework Mod-X is still better than CLIPa a on all of the evaluation settings. Comparing the R@1 results the test set Flickr30K(1K), we can find that CLIPMode x not only surpasses the initial results (CLIPPw332) but also 1.3% points and 2.2% points higher than CLIP at. The results COCO(5K) also illustrate that our framework not only resists the cognitive disorder of the model but also fits the new data domain better than CLIP The R@1 results of CLIP CLIPMModxX on COCO(5K) surpasses the CLIP at by 2.4% and 2.7% points, respectively.
7.8.2 THE PERFORMANCE OF MOD-X WHEN FINE-TUNING THE OPENAI'S CLIP ON ECOMMERCE- T2I DATASET
To illustrate that the Mod-X framework is not only applicable to open-world datasets, in this section, compare the performance of CLIP Mad-X x with CLIP,tt a and CLIP,,, for continual training
20

---

Under review as a conference paper at ICLR 2023
specific e-commerce data domain (ECommerce- T2I). The CLIPat is just continual training without any other operations. The CLIP/t updates CLIPest32 using joint- joint-dataset ECommerce- The ECommerce- Tericee-T2aaaggg images and 5k testing images set (EC(5K)). Fach image corresponds to a description, and the descriptions of training set and test set do not overlap. We set the CLIP wit32 the initial model in the continuous training process and divide the joint-dataset ECommerce- T2I into five sub-datasets uniformly and randomly simulate streaming data. The final multimodal retrieval results have been shown in Table g
al. (202 is text-to-image e-commerce dataset that contains 90k training
Image-Text (etrieval(S))
Text-Image Retrieval(%) (%)
Model
COCO(5K)
EC(5K)
(lickr30k(IK)
COCO(5K)
BC(5K)
R@1
R@5
|@0 R
R@1
R05
R@1
R05
|R@1
R@5
R@1 R
CLIP uxtaz
77.7
94.5
50.1
74.6
11.3
27.6
58.9
83.5
30.2
55.6
10.1
25.5
CLIPat et
63.4
87.2
36.8
61.5
16.6
40.7
44.4
71.0
20.6
42.6
15.8
40.5
CLIP Mod-
73.1
92.1
47.1
70.5
20.1
44.8
55.6
79.9
27.9
51.0
20.0
44.8
CL
64.5
88.6
39.8
64.8
23.5
50.8
46.9
73.1
22.2
44.5
23.5
50.6
Table 8: The final multimoal retrieval performance of the CLIP ct, CLIP OpenAl's ciip on specific e-commerce data domain (ECommerce-T22D)
Mod-x and CLIP,t yt based on
Text R@1 results on EC(SK)
Text-image R@] results on EC(SK)
Rig1 Accuracy
nga Accuracy
CUP
CLI2s
C1/Pwwr
CURea
Irdal
Phace
Phatry
Phote
Phose
Phane
Paan
Phase Training Phas es
Photee
Phone
Phanin
Training Phases
Image-Text R&1 results on COCO(SK)
Text-image R@1 results on COCO
R@91 Accuracy %
R.01 Accuracy
CLiPas- CUR.
CUP.
Inbal
Inb
Training Phases
Training Phases
Image: Text RG1 results FlIckr30K(1K)
Text-lmage Rg1 results Flickr30K(1K)
74
55
54
72
CLA'ss
66
CUPPac-7
CUPPaa-
CUP.
CUP.
Mussc
Phases
Ploscty
mao
Phusoy
Phasse
Phusey
Training Phases
Training Phases
Figure 13: The R@1 retrieval performance of different training strategies in each training phase on EC(5K), COCO(5K) and (IKc
Comparing the R@1 performance of the CLIP ft and CLIP vir32 on the EC(5K) test set, we once again show that the training of the CLIP model is affected by the training data domain. The R@1 results of CLIP, ft on EC(5K) 8.8% and 9.9% points higher than CLIP However, the retrieval results of CLIP,; f: on COCO(5K) and F1ickr30K(1K) have dropped by more than 10% points on average, which also means that fine-tuning (one phase continual training) the CLIP model will lose its ability to retrieve past samples. This is also verified by observations of the retrieval performance of CLIPa performs lower than CLIP ft- On the contrary, the CLIP Mod-X obtained after continual training by the Mod-X X framework only has tie drop of 3.3% points in the R@1 retrieval results on COCO(SK) and Flickr30K(1K), What's 's more, the performance of the CLIP CLIPasod- Mod- x on EC(SK) outperformed CLIP, by 3.5% points and 4.2% points Image-Text R @ 1 and Text-Image R @ 1,
21

---

Under review as a conference paper at ICLR 2023
respectively. All of this shows that Mod-X X framework not only preserves the inter-modal spatial structure old samples during the continual training but also improves the fitting ability of the CLIP in the current training data domain. Figure 13 presents the R@1 retrieval performance of this three training strategies on the COCO(5K), Flickr30K and HC(5K) at each training phase. The trend of R@1 performance of these three training strategies during continual training three test sets also illustrates that the Mod-X framework significantly alleviates the cognitive disorder during the continual CLIP training.
7.9 THE REPRESENTATION QUALITY OF VISION ENCODERS DURING CONTINUAL CLIP TRAINING
In Section 3 based on the distribution Table 2(b) we inference that the topology of the visual representation of the CLIPa changes slowly during the continual CLIP training. Due the topology of the representation space is correlated with the quality of the model's 's representation, so we use the linear evaluation method, commonly used in self- superviueri Oord et al. (2188); He et al. (2020), to detect the quality of the model's vision encoders to verify our suppose. By fixing the vision encoder, retrain single Linear layer, which is connected behind the vision encoder, based the ImageNet Deng et al. (2009) training set and evaluate its top-1 accuracy on the ImageNet test set to represent the vision encoder's representation quality.
7.9.1 THE CHANGES IN THE REPRESENTATION QUALITY OF VISUAL ENCODERS IN EXPERIMENT A.
This section discusses the representation quality of the CLIP's visual encoders in Experiment A. As shown in Figure [4 calculate the vision encoders' linear evaluation different training phases.
The Representation Quality of the Vision Encoder in Experiment
CLi
02-
30
Evaluation Eush
26
26-
22
CUP:
CUP.
CLi:)
CLP:
CLP
CUR
Phase Number
Figure 14: The changes in the representation quality of visual encoders in Experiment A.
Observing the changing trends in the linear evaluation accuracy of each training phase, that the representation quality of the vision encoder CLIPa gradually decreases as the training phase increases. The top-1 accuracy in the ImageNet test set dropped from 30.1% to 28.1% which is consistent with our conjecture 3.2.1 Compared to the decline in multimodal retrieval, the decrease in the quality of visual representations appears to be negligible. In addition, by comparing the results of CLIP CLIP Mod-X and CLIP,t- we can find that our Mod-X framework can not only help the model fit new image-text samples but also improve the representation quality of the modal encoders within the CLIP. The top-1 accuracy of the vision encoder in ModPMod-3 Mod-X improved from 30. 30.1% to 32.0%. All of this also illustrates that the quality of the extractor representation is not precisely positively correlated with the cognitive ability of the model.
find
22

---

Under review as a conference paper at ICLR 2023
7.9.2 THE CHANGES IN THE REPRESENTATION QUALITY OF VISUAL ENCODERS IN EXPERIMENT
This section discusses the representation quality of the CLIP's visual encoders in Experiment B. As shown in Figure 15 we calculate the vision encoders' linear evaluation different training phases.
The Representation Quality of the Vision Encoder in Experiment
Phazetki
10
CLP:
CUP
CUP.
Figure 15: The changes in the representation quality of visual encoders in Experiment B.
Due to the continual large-scale training, the representation quality in CLIP has fallen off cliff. The trend is similar to its performance on multimodal retrieval. Although our Mod-X X framework maintains the quality of the modal encoders and the top-1 accuracy of vision encoders increased to 36% from 34.25%, the gap between CLIP,r jr still has 4%. We think how to shorten this gap will become a question worth considering.
23


Reviews:
[{"review_id": "pftLbhVTZd0", "reviewer": "Reviewer_TamL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a continual learning problem based on the previous CLIP model. COCO and Flicker datasets are investigated to determine the representation variations when continually trained on one dataset and tested on the other.  They found that the retrieval performance on one dataset would drop significantly when continually trained. They correlate this phenomenon with representation variations. To alleviate this, they introduce a new training loss to contain the predictions between current and previous predictions. ", "summary_of_the_review": "Clarity. \n The authors investigate the image-text retrieval continual learning problem based on the CLIP model. They provide several interesting findings and modify the training objectives. But the experimental results are not sufficient to support the claims of the present paper. Therefore, I tend to reject the current version.", "strength_and_weaknesses": "Strength\n1. The continual learning of CLIP model is challenging but very interesting.  \n2. Authors correlate the degrading performance with the representation variations. \n3. The manuscript's structure is well-organized and easy to follow.\n\nWeakness\n1. The conclusion can not drawed from the provided observations and experiments in the paper. First, only two retrieval datasets are considered, the representation differences may vary in different datasets. Second, the correlations between the retrieval performance and representation variations are not evaluated.\n2. For the proposed method, the data are different during different stages, but the training objective is modified on different training iterations. Why would the new loss alleviate the forgetting issue? \n3. The experiments are not sufficient to demonstrate the claims in the introduction. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "lTmAkuwp_1", "reviewer": "Reviewer_7TxY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses continual learning in joint vision and language model. Specifically, the paper analyses the causes for task/dataset forgetting in a continuous training regime from a geometric point of view. The analysis works as follows:\n\n1. First, CLIP is trained on COCO to obtain a baseline CLIP0 model. \n2. Next, Flicker 30K is divided into 5 subsets and the CLIP0 model is further trained sequentially on each of the subsets to obtain CLIP1-CLIP5 models. These models are also denoted as CLIP_ct models, where ct stands for continual training. \n3. In addition, the authors train CLIP on the entire COCO and Flicker 30K datasets to obtain CLIP_jt, where jt stands for joint training, which serves as an upper bound to CLIP1-5 performance. \n4. The authors measure the performance of the various models on Flicker and COCO test set, demonstrating a gradual degradation in performance of the CLIP_cp models on COCO tests set and an improvement on Flicker test set, which makes sense as the models are trained (gradually) on the Flicker dataset. They name this phenomenon \"Cognitive Disorder\". \n5. Next, the paper considers the angles between all pairs of visual representations from different models, i.e.: the angle between the visual representation extracted using CLIP_t for a sample i and the visual representation extracted using CLIP_(t+1) for the same sample i. This analysis is done on the test set and denoted as Self Angle relationship Matrix (SAM). This measures to what extent the representations are rotated during continual training, with respect to themselves. They find that those angles change slowly, therefore argue to not be the reason for the performance degradation. The change in SAM is denoted as Intra-modal rotation. \n6. The paper then turns to analyse how angles between pairs of visual embeddings from different samples change during continual learning. Specifically, they consider a matrix composed of the angles of representations from different samples i,j extracted from the same CLIP_t model. This matrix is denoted as Rotation Angle Matrix (RAM). This analysis is done for the textual embeddings as well.\n7. The authors find that the RAM matrix changes more dramatically for visual embeddings than the angle between off-diagonal elements of the textual embeddings. The change in RAM is denoted as Inter-modal deviation. \n8. From this analysis, the paper argues that the inter-modal deviation (i.e.: the fact that the two modalities are not synchronised in the amount of rotation between off-diagonal elements in different training steps) is the cause for the degradation in performance during continual learning (which is denoted Cognitive Disorder as I explained above). \n\nNow, following this analysis, the paper proposes to alleviate \"cognitive disorder\" by aligning the amount of rotation of off-diagonal elements between the two modalities during training. This works as follows: first, given the trained model on sub dataset t , we consider the contrastive matrix M_t, i.e.: the matrix whose element in the i,j position is the similarity score of the i'th sample vision embedding and the j'th sample text embedding. The method add a KL divergence term between M_(t-1) to M_t to the loss. Samples (rows) for which the model in \"timestamp\" t-1 are not corrected classified are replaced in M_(t-1) by their values from M_t.\n\nThe paper presents two experiments to evaluate the performance of the proposed method. The first experiment follows the same setting as the analysis above and in the second experiment, the authors trained a CLIP0 model on COCO and Flickr datasets and gradually trained 8 additional models on 8 subsets of SBU-1M. The experiments demonstrate that using the proposed method, the resulting models only suffer from a small degradation of performance on the \"baseline\" dataset while performing well on the \"new\" gradually added sub-datasets, compared to the baseline CLIP0 model and even perform better on the \"new\" dataset than the baseline continual training model. \n \n\n\n", "summary_of_the_review": "Strong paper in my opinion. The paper identifies a continual learning performance degradation as drawback of joint vision and language embedding models. It demonstrates this drawback by training a baseline model on COCO, \"finetuning\" it on Flick and evaluating on both Flickr and COCO test set, indicating a clear degradation on COCO. The paper analyses the cause for it and finds it to be Intra modal deviation, i.e.: misalignment between the amount of rotation of diagonal elements during training between the two modalities and designs a solution via adding a term to the loss to encourage intra modal alignment. The paper provides experiments that demonstrate the affect of the solution in mitigating the drawback. I mentioned a few weaknesses and question I would appreciate if the authors could address in their rebuttal. ", "strength_and_weaknesses": "Strengths:\n1. The paper is well written and easy to follow. \n2. The paper provides a thorough analysis on the causes for performance drop in vision and language models from an original point of view.\n3. The proposed method performs very well in practice, as demonstrated in the experiments. \n4. The paper showcases a real problem, provides an analysis to its cause, proposed a solution and provides experiments supporting the claims made and demonstrating superior performance of the proposed solution. \n\n\nWeaknesses and questions:\n1. First, I strongly recommend to change the terminology of \"Cognitive Disorder\". The term has little connection to the problem it describes, it's a bit far-fetched to borrow a term originally used to describe a mental health disorder and really adds nothing to the paper. A more technical term (dataset forgetting, degradation from distribution shift,...) would be much better.\n2. The analysis on the causes of the performance degradation is done on the test set. However, the proposed solution is of course implemented during training. We obviously can't add a loss term during training and optimise it on the test set, but how much of this analysis holds when you consider the training set? I would expect the model to overfit to the training set, would there be inter-modal deviation between subset t and subset t-1 during training? \n3. In table 3 alpha is set to 20. An analysis of the effect of alpha and to what extent the method is robust to the choice of alpha would be interesting to see. Of course, as alpha is increased the effect of \"alignment\" between models is increased, but I think the paper should explain how alpha was selected to be 20 and how sensitive this value is (e.g. how dramatically the results would change if a different, less ideal value would be used). \n4. A drawback of the method is that it required to store the vision and language embeddings from model t-1 during training of model t (or re-evaluate them during step t). I'm not sure how much it would be applicable to training on huge scale datasets (e.g. LAION dataset). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "bpu-9ZbCL5", "reviewer": "Reviewer_4dCz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method based on the CLIP model for continual vision-language representation learning. The authors find that simple continual training in the CLIP model degenerates the performance on multimodal retrieval tasks, named Cognitive Disorder (CD). To tackle CD, they suggest Mod-X that selectively aligns the off-diagonal information distribution of the contrastive matrices utilizing the CLIP models from current and past timesteps.", "summary_of_the_review": "The direction of the submission is auspicious and interesting, but explanations might be required to be more kind and include the details. Also, I recommend further comparing with competitive methods for multimodal continual learning methods for solving retrieval tasks.\n", "strength_and_weaknesses": "**strength**\n- Introduce and analyze challenging issues for continual vision-language representation learning, named intra-modal rotation and inter-modal deviation\n- outperform baselines in both image-text and text-image retrieval.\n\n**weakness**\n- Missing detailed and explicit descriptions for many notations (e.g., ct, jt, the subscript (task index) i, the superscript T in Section 3.2.3., the capital letter S in figure 6, etc\n- The difference between Cognitive disorder and Catastrophic forgetting (or representational forgetting, referred to in Madaan et al. 2022) needs to be clarified. \n- Intra-modal rotation may happen to all non-iid task-based problems, including streaming learning and conventional continual learning, which is not new, and the following analyses are a bit straightforward in the continual learning field.\n- Including Appendix 7.1 for the mathematical derivation, inter-modal deviation is not explicitly related to cognitive disorder (the performance degeneration of retrieval tasks). It tends to, but in a strict sense, the manuscript doesn't provide evidence that increasing inter-modal deviation in each sample is always harmful to \"cognitive disorder\".\n- Only naive and old baseline (EWC) is used for validation, which is hard to evaluate how the proposed model achieves superior performance compared to recent strong methods for multi-modal continual learning for retrieval tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2AXcf4s02d9", "reviewer": "Reviewer_JafF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates continual learning of a multi-modal text-image model like CLIP. It investigates the angle shift of the learned representations throughout different learning sessions, and it also proposes a knowledge distillation type of loss to address the issue of catastrophic forgetting. Results showing that using knowledge distillation can significantly improve the CLIP continual learning. However, the paper’s model and analyses are disconnected and there is concern on whether the model is specific to CLIP or contrastive learning in general.", "summary_of_the_review": "The introduction of a multi-modal continual learning benchmark is interesting, and the analyses are insightful, but unfortunately the model disconnects from the analyses. I suggest the authors show a direct link why the analyses inform such a model, and also evaluate whether it is possible to run on image-only continual learning benchmarks or there are things that are specific to image+text. Overall I think the paper is slightly below the bar as it is hard to find a clear hypothesis throughout.", "strength_and_weaknesses": "The strength of the paper is that it shows an interesting phenomenon of angle shifting – the relative angle between a positive pair didn’t move as much as the absolute angle rotates. This suggests that the representation space is undergoing constant rotation during continual learning, which is an interesting insight. Also the experimental results clearly show that the proposed model is significantly better than baselines.\n\nThe main weakness of the paper is that it reads very disconnected. There is no intuition why knowledge distillation addresses angular shift, and neither are there results confirming that using knowledge distillation helps this issue by showing the angular shift before and after distillation. \n\nThe proposed model seems generic to any contrastive learning algorithm, and the whole paper could have been benchmarked on image-only continual unsupervised learning (e.g. Madaan et al. 2021), instead of image-text learning. There is nothing specific to multi-modality here, and you could treat the two vectors coming from either an image-text pair or a pair of two views of the same image.\n\nMoreover, the paper claims novelty on “cognitive alignment”, but it is basically a knowledge distillation loss (i.e. LwF Learning without Forgetting, Li & Hoiem 2016) applied directly in the context of contrastive learning (i.e. predicting image instance labels instead class labels). Since no gradient is backproped into the old model, the KL loss is equivalent to a cross entropy loss between old and new prediction. One minor difference is that the paper manually edits the old model’s prediction if the label is wrong. This seems an ad hoc operation and there is no empirical evidence presented in the paper showing that it is necessary. In summary, the paper may run into a concern of over-claiming, as the proposed method is LwF/knowledge distillation re-purposed in the context of contrastive learning, but it is instead branded with a bunch of “cognitive” terms (“cognitive disorder”, “cognitive deviation”, “cognitive align”) with no real cognitive science intuitions or references.\n\nMadaan et al. Representational continuity for unsupervised continual learning. ICLR 2022.\nLi & Hoiem. Learning without Forgetting. ECCV 2016.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject