Continual Multimodal Contrastive Learning
Authors: Xiaohao Liu
Xiaobo Xia
See-Kiong Ng
Tat-Seng Chua
Venue: NeurIPS 2025 poster ()
Keywords: Continual Learning
Multimodal Contrastive Learning
Primary area: general_machine_learning

Abstract: Multimodal Contrastive Learning (MCL) advances in aligning different modalities and generating multimodal representations in a joint space. By leveraging contrastive learning across diverse modalities, large-scale multimodal data enhances representational quality. However, a critical yet often overlooked challenge remains: multimodal data is rarely collected in a single process, and training from scratch is computationally expensive. Instead, emergent multimodal data can be used to optimize existing models gradually, \textit{i.e.}, models are trained on a sequence of modality pair data. We define this problem as Continual Multimodal Contrastive Learning (CMCL), an underexplored yet crucial research direction at the intersection of multimodal and continual learning. In this paper, we formulate CMCL through two specialized principles of stability and plasticity. We theoretically derive a novel optimization-based method, which projects updated gradients from dual sides onto subspaces where any gradient is prevented from interfering with the previously learned knowledge. Two upper bounds provide theoretical insights on both stability and plasticity in our solution. Beyond our theoretical contributions, we conduct experiments on multiple datasets by comparing our method against advanced continual learning baselines. The empirical results further support our claims and demonstrate the efficacy of our method.  Our codes are available at https://github.com/Xiaohao-Liu/CMCL.

Continual Multimodal Contrastive Learning
Xiaohao Liu
Xiaobo Xia"
See-Kiong Ng Tat-Seng Chua
National University Singapore
xiaohao... liuuu.uuuu.. nus. edu fxbx, seekiong, descts Cnus. emus..du..s... edu.sg
Abstract
Multimodal Contrastive Learning (MCL) advances in aligning different modal- ities and generating multimodal representations in a joint space. By leveraging contrastive learning across diverse modalities, large-scale multimodal data hances representational quality. However, a critical yet often overlooked chal- lenge remains: multimodal data is rarely collected in single process, and train- ing from scratch is computationally expensive, Instead, emergent multimodal data can be used to optimize existing models gradually, i.e., models trained se- quence of modality pair data. We define this problem as Continual Multimodal Contrastive Learning (CMCL), an underexplored yet crucial research direction at the intersection of multimodal and continual learning. In this paper, we for- mulate CMCL through two specialized principles of stability and plasticity. We theoretically derive a novel optimization-based method, which projects updated gradients from dual sides onto subspaces where any gradient prevented from interfering with the previously learned knowledge. Two upper bounds provide theoretical insights on both stability and plasticity our solution. Beyond our theoretical contributions, paring our method against advanced continual learning baselines. The empirical results further support our claims and demonstrate the efficacy of our method. Our codes are available at https: // //githuhtb Homm/iiahho-iia Liuu
conduct experiments on multiple datasets by com-
Introduction
Humans perceive the world in a multimodal way. They capture and process information from what they see, hear, smell, and touch. Machines are evolving towards multisensory processing to mimic this capability by leveraging specialized latent representations to interpret the world ef- fectively [1, 2, 3, 4, 5]. Multimodal contrastive learning (MCL) builds upon unimodal [6. 7, and cross-modal contrastive learning [9, 10], which demonstrates exceptional representational power and broad applicability. The core principle behind it is to bring modality pairs (e.g., vision-text or vision-audio data) from the same instance closer together while simultaneously pushing apart repre- sentations of different instances. Consequently, MCL. constructs unified representation space yet capable of accommodating diverse modalities [11, 12, 13, 14].
Extensive data support critical for ensuring representational quality. Recent studies have focused on either compensating to the existing modality data [13, 14] or introducing the emergent ones [15]. Besides learning from scratch, collected data can also be incrementally introduced to gradually enhance model's capabilities, thereby resolving two significant challenges: 1) the difficulty of collecting all prepared multimodal data single process, and 2) the high computational expense associated with training mixed datasets from initialization. With the emergent multimodal data, multimodal models can be progressively elevated via continual/ineremenaal learning. Unfortunately, recent research in MCL primarily advances in an "all in go" manner. Moreover, it is intractable
"Correponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Continual Multimodal Contrastive Learning
Steps
Vanilla
Training steps
VW.
Overridden!
VW
[vision, text)
(vision, audio)
[text, audio)
StOPT-1
VW
W
VW;
M1|
B
Stepr
Ours
Multimodal training datasets X
Training steps
VW
Projection!
WW
VW
VW. VW r+1
JiJ
Stept-1
AW.
Model parometer update
W
AW,
Figure 1: The problem of Continual Multi-
Figure 2: Our gradient-projection method (bottom) compared with the vanilla (top). Within the parameter space spanned by W, the gradient updated at the previous step V V is directly overridden in vanilla training. Instead, our method projects the new gradient VW, onto the non-effective subspaces of the old. denoting A AW, to maintain both stability and plasticity.
modal Contrastive Learning (CMCL). Mul- timodal models learn from data step by step. At each step, the two modalities are aligned through contrastive learning. Different pairs of modalities are involved at different steps. The model parameters continually updated through gradients from different training steps.
for current continual learning methods address this problem [16, 17, 18, 19], due to the inter- modal complexity and the task-agnostic constraints, where the objective of contrasting different modalities remains consistent over time?.
Aware of this, we pioneer the formulation of continual multimodal contrastive learning (CMCL) in this paper. In CMCL, sequence of multimodal data is utilized for training. At each training step, two modalities are aligned through contrastive learning, and the new modality pair is subsequently integrated into the following step, illustrated in Figure 1. We specify and plasticity in CMCL. Here stability refers to the model's capability to retain acquired knowledge previously interacted modalities, while plasticity denotes that the model can learn effectively from new modality pairs. Building upon this foundation, we theoretically derive a novel CMCL method by projecting updated parameter gradients from dual sides onto specialized subspaces, as shown in Figure 2. We project gradients based on both their own modality knowledge and their interacted ones, i.e., dual-sided projection. These inter-modality histories are utilized construct the corresponding gradient projectors. Incorporating new data at the current training step within these projected subspaces does adversely impact model performance on datasets from previ- ous steps. Furthermore, we provide theoretical guarantees by establishing two bounds addressing stability and plasticity. These bounds substantiate our method's effectiveness theoretically, further supported by empirical analyses. Besides, we showcase the flexibility of our method by extending any steps and any modality pairs thus achieving both efficacy and efficiency in practice. We conduct experiments on 7 datasets to evaluate our solution for classification and retrieval tasks. The results well justify our claims. The proposed method consistently achieves superior performance in plasticity and stability. Compared with previous state-of-the-an (SOTA) baselines of the continual learning field, it demonstrates remarkable results across multiple metrics.
definitions of stability
Before delving into details, we clearly emphasize our contributions as follows:
Conceptually, we take the initiative formulate continual multimodal contrastive learning ex- plicitly and provide specialized definitions of stability and plasticity that guide methodological development. Our work establishes a foundational framework for exploring this previously unad- dressed problem and encourages future contributions to CMCL.
Technically, we introduce a novel method that operates on the parameter gradients. These gradi- ents are projected onto the subspaces where any updated gradients have no effect on the previous knowledge. This method is built upon existing modality-binding models and can be readily ex- tended to any training steps and modality pairs in the practical training procedure.
We review the literature related to this work in Appendix D.

---

Theoretically, we rigorously derive our method based on the fundamental requirements (stability and plasticity), supported by detailed theoretical proofs. Additionally, we provide two theoretical bounds that guarantee the objectives, which can be tightened in practice to reinforce our claim.
Empirically, we conduct extensive experiments on 7 datasets to evaluate the performance in the CMCL setting. By utilizing three modality-bindiin methods as backbones, we demonstrate supe- rior performance compared to SOTA continual learning baselines, which confirms the effective- ness of our solution.
Preliminaries
Multimodal learning. Typical multimodal learning can be formulated as:
P(x, v)
P y/x (y h g(x)) (x))
(1)
where g is the multimodal encoder that generates corresponding representation z given the input x, and h is the task mapping (e.g., linear classifier [10, 11] or even large language models (LLMs) [20, 21, 22D.. In general, g is modal-agnostic. Here we use gm as the modal-specific encoder, indicating that we do not make any assumption about a shared model architecture or parameters. For different modalities, a unified representation space is desired to facilitate the development of h, so that one well-trained h serves all } MEM.
Modality bindiig (modality pairs for training). Note that g is normally implemented by modality-binding methods to obtain representations for different modalities in unified space. These methods utilize paired data mmmmn = [[x]", xT ), (x. xy )..... C. Xm x X to optimize the contrastive objective derived from CLIP [10]:
exp(z." z /T)
C = log
(2)
(xp" Z.T /T) + Ejjt exp(z!" Z /T)
where is the temperature hyperparameter. The two modalities m and m' from one instance are pulled closer while being pushed away from different instances j. Despite having only two modalities in each batch, more diverse datasets are introduced achieve multimodal contrastive learning with more paired modality data. The typical utilization of moddllty-biinding is exemplified by CLIP, which contrastive learning for bi-modality. Here only two modalities (i.e., image/vision and text) are used for training. However, several works [11, 15, 13] have recently extended this approach with more modalities on different modality pair data, e.g., image-audio or audio-text pairs.
Range-Null space. Given a matrix A, its pseudo-inverse At satisfies AA = A. A projects to the range space of A, formally, A(A Ab. (I-AAA) can be seen the null space projector that directly satisfies A(I AtA)b = o. And there are several ways resolve the pseudo-inverse At, for instance, in singular value decomposition (SVD), At VAAUT for (U = SVD(A). where A take the reciprocal the nonzero singular values in A while leaving zeros unchanged. Null space projection is used in prior works like Adam-NSCL [16, 23, 24], which constructs the null space projector based on layer-wise features. Despite the inspiration, is not capable of tackling the mnter-modality complexity in CMCL. A specialized solution is still desirable for CMCL.
3 Continual Multimodal Contrastive Learning
3.1 Problem Setup and Two Goals
Problem setup. Suppose we have sequence of datasets X' = { A. A.rmss" ...... }, where X.mema denotes the dataset containing modalities m and m' at the initial step and the sequence length is N. The parameters of each modality encoder 9m optimized with different pairs, signifying W WT. Note that W Wi when X does not include the modality m. Without the loss of generality, we consider two continuous training datasets, denoted as xmmy.mz and X mm
which share at least one common modality. The corresponding encoder Iml will be continually
3

---

optimized with different data. sequential learning paradigm:
Here we formulate multimodal learning on multiple datasets in
We = arg min
Σ
L(gm, (xt t W Imn (x ma W(-1))
(3)
Note that W 1-1 trained on X Xm Besides, if mg == m2, this formulation can be generalized to the case in CLIP [10] and the bi-modality continual training [25, 26, 27].
Two goals in CMCL. The main goal is to maintain alignment stability along with continual learning- This alignment is typically defined as the inner product between different modalities of the same
instance [10, 11, 15], signifying z!th z.' This formulation is also utilized in multimodal contrastive
learning as mentioned in Eq. (2).
Definition 1 (Stability). Given two adjacent training datasets X and X, the model satisfies stability if:
(
111-1111
model step
model previous step
where Ii- denotes the representations for modality my (t 1)-th dataset generated by the model trained after step f. 1n-1 represents the representations generated by the model immedi- ately trained on the dataset at its own t-
For the dataset used previously (i.e., x the current model parameterized by W, can perform well aligned with the previous model parameterized by W.-1- For simplicity, we denote the align- ment scores between two modalities (i.e., my and m2) as A 1-1:
mi utilizing the model at step t for
dataset - Consequently, we simplify Eq. (4) to A-1; m1.m2 = 1-1--1
Definition (Plasticity). The model hus the capability to learn from new datasets, udhering to the objective of multimodal contrastive learning: Z Z that samples the t-th dataset for two different modalities.
Pt. mm.m ,whre Pt mmm is the distribution
Another goal indicates that the model, even with constraints to memorize the old, still have the plasticity to accept the new. Specifically, given the dataset at the current step t, the parameters of the model still be optimized by multimodal contrastive learning objective fit its distribution.
Dilemmas. We consider the post-training paradigm to elevate the existing pre-trained models that already trained with various datasets. Current methods without continual learning objective face a significant challenge of forgetting: the alignment plasticity is broken when the model is trained with new datasets (i.e., A * A-1111))
3.2 Dual-sided Null Space (DNS) for CMCL
In this section, we propose projecting gradients the dual-sided null space satisfy both sta- bility and plasticity, as depicted in Figure 3.
Step f-1
Dual-sided projector
Step
VW VW
PPW
P AW
m1
Stability requirement. Recall the stability requirement in CMCL of (Z""),) T A m,, A--1--1-- That is to say, given the same modal- ity pair data from the previous step, the model trained on new datasets should maintain the alignments on modality pairs from earlier steps. This implies that new updates (i.e., gradients) should not interfere with effective parts within the parameter space.
Lon 2012
Gradient projecting
VW
VW
P' Am
Figure 3: The paradigm of the proposed method (DNS) for CMCL. At the current step t, gradients (VW and VW?") derived by contrastive learning between modalities m. and m2 are projected from dual sides to A A and A W The projectors are built upon the features from the previous steps to constrain the gradients. By substituting for the gradients, model param- eters are optimized from new modality pair data while retraining the prior knowledge.
Plasticity requirement. According to the con- trastive learning objective in Eq. (3), uncon- strained gradients naturally encourage param- eters optimized from previous datasets to adapt

---

to new datasets. Therefore, plasticity emerges directly from the inherent learning process. In vanilla CMCL, where the model sequentially trains on datasets without any explicit constraints, plasticity is inherently maintained. Introducing additional regularization strategies inherently increases risks to plasticity.
Theoretical motivation. Building upon the above analyses, we explore gradient updates in detail to establish theoretical motivations. Let us consider the left-hand side of Eq. (4), ie.. (Z]m):)) -
We introduce the prior encoders of 9m (cf., plasticity in Definition 2) and linear operator that trans- fers the priors 1-x and 1=:1; with (-1; W Z[-1;- and W E Raxd - E Rdx=
where d is the feature dimension and n denotes the batch size. Besides, we consider the gradient update rule as W
W 77 VW"" where 7) is the learning rate. Therefore, we have:
(Z,-1;)) 1i
-(Z,-..)
WT
z (Z (W ((WT (WW"))"(W""" - WW 1
global pormeeters at stop!
(Z)-1:))
W
1-11
global parametee of step
(W
VW," (VW:") w - (V..
(5)
updated global parameters W
Projecting W (globally). We consider the parameter update globally, where W is used to update (WZ"1 T W via (W?")) w ous datasets, we need to find new parameter update W that satisfies (Z WZ, 0. Therefore, Am, Am 0.
= -WTT1 ( TW w/W. To keep stability for the previ-
Theorem 3. Let the global parameter update follow (WT") ) Tw = (WT)) W - W, and the original parameter update be denoted as:
W ((W VW." + (VW?") TW (VW""" T VW"""
(6)
Then, the stability holds if == P'WPP where P and P are the space projectors for Z, and 1 respectively. See proof in Appendix B.I.
Remark. Once obtaining the global gradient update W, we are able to find the optimal solution to project it such that (Z.-1) Wz, O. W is the closest point to W but lies in the null space. In other words, the projected global gradient does not cause any effect on previous modality pairs at step t Here we call the space spanned by W PPWPP null space as well".
Discussion on W. W derived from the original W while removing its part relevant the previous training. Such relevance is probed and extracted via pre-curated projectors corresponding the dual-side modality data Z,"1, and Z,n,, (see Section Range-Null space for constructing projectors). This projection reflects the modality interaction in multimodal contrastive learning,
where the parameters are optimized from both modalities corresponding to different sides of the parameter matrix W (i.e., the row space for my and column space for m2).
"The formal definition of null space is the solution space for Ax 0, differing from our problem.
Projecting VW (locally). Following this, hope to update the local gradients in practice, i.e., optimizing the model parameters corresponding to each modality, to satisfy the stability. Here we find the deviation of these gradients, thus approaching the W P'WP lies in the null space.
Theorem 4. Let the local parameter update follow W W W.". the stability holds if
A AW. VW PVW AW! = VW P'VW."'
where P' and P are the space projectors for 2 and m2 - respectively. See proof in Appendix B.2.
5

---

Remark. We find the local gradient transformation for V VW Vm E {mi, m2) to achieve the global gradient update W - PPWP. In this case, we can transform the gradient from independent models (i.e., different encoders), yet still keep the stability.
Discussion on A A. We introduce an additional projector of the previously generated features, denoted by - 1: A W is similarly derived from the original gradient VW while removing the parts determined by both the own input modality data -1= (ef., Theorem 3) and the generated features of another modality ZT- -- P and P' project vectors onto modal-specific oulput sub- spaces, which reside within a unified representation space due to the contrastive learning paradigm. Despite sharing unified space, the subspaces of different modality features are not exactly iden- tical. Therefore, we employ two separate projectors for a more precise projection. This projection reflects the effective mapping from the modality input the aligned output space.
3.3 The Theoretical Guarantee: Bounds and Capabilities
After inferring the method to achieve CMCL, we examine the performance guarantees theoretically. Besides, we introduce the extensions for any steps and any modality pairs during the practical train- ing procedure. Note that also provide empirical support for theoretical results in Section 4.2.
Theorem (The upper bound of loss of stability). In CMCL, the upper bound of loss of stability is:
(-it
A m2
S of -1 1 F(VW),
(8)
where FF represents the interactions between different updated gradients V VWm. Specifically, F(VW) = v V ||2 + VV V/V See proof Ap- pendix B.3.
tt and V VW!
Remark. The loss of stability is primarily bounded by the square of the learning rate 17² and the norm of the input features 1.. The input features are typically normalized to stabilize the prior encoders' training. Therefore, their norm does not become so large as to interfere with the bound. Furthermore, low-dimensional sparse features have been observed to better preserve basic objectives for instance discrimination [28], potentially tightening the bound. A smaller learn- ing rate results in tighter bound as well. This is reasonable when considering the extreme case (1) = 0), where the parameters are not updated during training without any loss of stability.
Theorem 6 (The upper bound of plasticity). By transforming the gradients, the loss La update at step t from the previous loss f is following:
-
to keep the plasticity when o(n) < 0 at the time t. See proof in Appendix B.4.
Remark. The loss decreases, indicating learning from the new datasets. L. = L,-1 1 if and only if the row space of W lies in the column space of Z,-,, and the column space of W lies in the column space of Z, In this case, W can only make effects to the features within the space of previous datasets. However, W is calculated via current datasets Z,",, which makes the equality rare.
Extension for any steps (A m.m }<t. In general, on continual learning, we hope to maintain stabil- ity for any seen datasets. Therefore, we have:
A mu.m
Vi < t.
(10)
That means need to project W onto the null space of all previous datasets, via P <t We utilize the uncentered feature covariance [16] to obtain the null space projector via SVD approximation 2 T
(Section 2). Let Z = A: - + 2i Rt-1 ZZ" where 2,n و ايبي m (Z, Z (the equal- ity is proved in Appendix B.5). Similarly, for global gradient updates, W == W P' WP for extension. For the local gradient update, we are required to maintain an additional feature co- variance constructed from the previous models. We can achieve this in efficiency. For instance, A A. := VW P W where P'z, is the space projector for (Z hict- Note that, the guarantee for stability and plasticity can be easily extended to any step, as we only modify the projectors that are able to project the features onto the space that covers all previous datasets.

---

Table 1: The statisties of datasets for modality pair training, which involve modalities of V++ Vision, AttAu T VIdeo, THe-THermal, TA+TAccil,,, and De+Depth.
Dataset
Modalities
Evaluation
Tasks
Modes
#Examples
#Classes
TH TA
Retrioval
Classificatiin
UCF101
/
/
Ace
Trin, Test
8,080
101
ESC50
V
Rocall, Ace Rocall
Test
2,000
NYUDx2
V
Train. Test
48,238
VGGSound-S
1
Rocall. Ace
V
12,000
309
Clotho
1
Rocall
Train, Test
4,885
TVI.
Recall
Tmin, Test
43,741
LLVIP
Rocall
Train. Test
15,488
Extension for any modality pairs (m, m' In particular, CMCL, diverse modality pairs are involved to optimize corresponding encoders. In this case, we can simply extend the previous gradient update on {m1, m2) in step t t- 1 and t any different adjacent modality pairs. Suppose training on modality pairs of {m,, m2 at step & - and {m,, at step t. If mg ;= m2, it follows Furthermore, the corresponding feature projector
Theorem 4. If mg j m2, we set W update can be omitted as well to save computational cost. Therefore, only features at relevant steps are required to be updated rather than considering all steps. The algorithm flow and pseudo-code of training and inference procedures of our method are provided in Appendix A.I.
4 Experiments
4.1 Experimental Setups
Tasks & datasets. We evaluate on 7 multimodal datasets, including UCF101 [29], ESC50 [30], NYUDv2 [31], VGGSound-S [32]², Clotho [33], TVL [34], and LLVIP [35]. For performance comparison, we categorize the evaluation tasks as retrieval and classification. The important statis- tics are provided in Table 1. More details are provided in Appendix A.2. For datasets with more than two modalities, we split them into bi-modal ones to cater to the modality pair training. For instance, VGGSound-S will be reorganized into three versions (VI-A, VI-T, and T-A). As result, 11 training steps are involved our training and evaluation.
Implementation details. We utilize pre-trained moddllty-biinding methods as priors, followed by linear learners for each modality for training. Specifically, we employ ImageBind [11], Language- Bind [15], and UniBind [13], which already trained with sequence of modality pairs and are capable of generating multimodal representations in joint space. Upon these powerful priors, we construct the projection module that projects the novel knowledge onto the dual-sided null space of the previous one within the trainable parameters. We utilize PyTorch [36] for implementation. The model is optimized via AdamW optimizer [37] with learning rate of 0.0001 and weight decay of 0.001. The batch size is set to 64. We approximate the null space projection with truncated 0.01 for ImageBind and Unibind, while 0.0001
SVD, where minimum eigenvalue Amin for LanguageBind. See more implementation details in Appendix A.3, A.5, and E.
Baselines. We consider various methods from continual learning fields. Specifically, we select Vanilla (continual training without any strategy) [1]], Gradient Episodic Memory (GEM) [38], Dark Experience Replay (DER & DER++) [39], Elastic Weight Consolidation (EWC) [40]. Contrastive Continual Learning (Co*L) [17], C-Flat [41], and Contrastive Incremental Learning with Adaptive distillation (CILA) [18]. Notably, our method is replay-free, which maintains efficiency. The feature covariances are updated only at the end of each training step. We adapt these methods for the CMCL setting. More details of the basclines are provided in Appendix A.4 and E.
Evaluation metrics. We consider the evaluation metrics from both plasticity (learnability) and sta- bility (utt-forgetting) perspectives. For plasticity, we employ Recall@k for cross-modal retrieval tasks, where & is set in the range of {1. 5, 10). Besides, we use average accuracy (Acc) for classifi- cation tasks. For stability, we employ backward transfer (BWT), which computes the average drop of performance at the previous step after learning the current dataset. In this case, higher appearance
We downsample the original VGGSound dataset into small version, denoted as VGGSound-S.
7

---

Table Average performance (st.) on classification and retrieval tasks. Here "R@ @R@"" the abbreviation of "Recall @k". The results are calculated over ten independent trials. The best result each case is marked as bold.
Classification
Retrieval
Method
Acc
BWTA
R@1
R@5
R@
BWTRI0
Vanilla
47.32 10.27
-5.72
=0.4
9.93
10.09
87.86
10.18
38.56 10.14
-3.34 20.09
GEM
40.98 11.65
-9.75 1164
9.53 10.14
27 77.00-1
+0.28
37.38 10.31
-2.91 10.26
ImageBind
DER
$7.34.0.44
-5.70
=0.44
9,91 10.08
27.88
10.18
38.55 40020
40.08
DER+
47.40 +0.15
4,71 -0.88
9.94 +0.12
27.95 +0.18
38.72 +0.13
-3.09 +0.11
EWC
48.34 1:1.33
4.82 1118
10.07 10.24
28.35
10.55
39.22 40.69
-2.55 10.76
Co
50.13 +0.52
3.74=0.51
9.79 +0.04
27.99
+0.21
38.66
-1.77 +0.19
C-FLAT
50,96 +0.34
4.17 00.88
9,92 +0.09
28.90
+0.14
39.97
10.19
-1.41 #0.19
CILA
50.73 40.44 +0.48
-3,20
00.74
9.68 #0.11
27.76
+0.20
38.25
+0.20
-1.28 #0.21
DNS (ours)
52.52 +0.23
-0.02
=0.30
10,38
20.09
29.53
10.17
40,89
10.10
-1.07
10.12
Vanilla
51.86 +11
115.71
1.22
7.51 10.10
25.36
+0.32
36.02
+0.49
-10.19
+0.53
GEM
60.15 1:063
4,98
00.22
8.32 10.13
28.92
10.14
41.24
10.22
-.112
10.18
LanguageBing
DER
52.55 +1.08
-14.6
+0.88
7.55 10.16
25.47
10.36
36.08 40.46
-10.11 10.46
DER+
55.86 1:.64
-10.55
+1.58
7.33 1.020
25 25.66
+0.53
36.50 +.061
-9.58
+0.62
EWC
54,03 11.94
-13.45
+2.34
8.00 10.26
27.11 +0.92
38.14 11.26
-7.71 11.44
Co
58.08 +0.64
-5.79
-1.22
8.33 +0.12
28.33 +0.24
40.15 +0.31
-3.92 +0.37
C-FLAT
57.84 10.75
-7.57
=0.92
8.23 10.10
28.02
+0.17
39.72 +0.25
-5.00 =0.32
CILA
59.30 :0.62
-2.63
=0.91
8.32
+0.14
28.31
+0.26
40.48
+0.27
-1.09 =0.40
DNS (ours)
64.07 +0037
-0.09
=0.59
8.71 #0.06
29.81
+0.23
42.44 +0020
-3.00 #0.21
Vanilla
47.48 10.56
-5.46
=0.61
10.09 10.10
28.16
10.26
38.93 40.21
-3.52
=0.20
GEM
41.37 11.52
-9.29
=1.20
9.61 10.14
27.32
10.31
37.94
1.034
-2.97
+0.39
DER
100.7
-5.35
=0.55
10.10 10.09
28.16 +0.25
38.95
10026
-3,51
10.20
UniBind
DER+
47.75 1:0.23
4.69
0.71
10.06 10.08
28.30 10.21
39.10 10.15
-3.39
10.12
EWC
48.21 11.48
-5.06 11.19
-5.06
10.24 20.25
28.60 10.54
39.47
40.66
-2.89 10.67
Co
50.45 +0.59
9.87
28.30 +0.16
39.21
+0.20
-1.87 +0.21
C-FLAT
51.25 +0.42
-4.36 =0.49
10.00 =0.06
29.19 #0.17
40.51
+0.12
-1.48
#0.18
CILA
51.03 +:041
-2,93 =0.75
9.76 +0.08
28.07 +0.14
38.80
+0.17
-1.40
+0.20
DNS (ours)
52.86 +:0.29
0.31 #0.46
10.52 #0.06
29.97 10.10
41.44 +0.11
-1.19 #0.12
scores these metrics indicate higher consistency and determinism. More details of the metrics can be checked in Appendix A.5.
4.2 Results
Overall evaluation. We compare our method (DNS) with selected basclines on both classification and retrieval tasks with results being reported in Table 2. This result demonstrates that DNS outper- forms all baselines on two types of performance metries (i.e., Acc and Recall), Particularly in the classification task, DNS (ours) consistently achieves the highest accuracy and exhibits minimal or even positive BWTA, whereas other methods generally suffer from higher negative transfer. In the retrieval task, DNS outperforms other methods in all Recall @ k (R @ R @@5, and R @@ 10) metries, and it also demonstrates good backward transfer ability. These results highlight the effectiveness of DNS in mitigating catastrophic forgetting (stability) and achieving superior performance (plasticity) across both classification and retrieval tasks.
Stability analyses. We calculate the deviations between alignment scores to provide practical sup- port for the theoretical guarantee of stability in Theorem 5. At each step, we average the scores of modality pairs, which will be compared at subsequent steps, with the results shown in Figure (a). We can observe that the deviations are small and even reach nearly zero at some steps, further sustaining the stability maintaining of DNS.
Plasticity analyses. We provide empirical support corresponding to the theoretical insights on plas-
ticity in Theorem 6. Here we compute the high-order term and verify whether it satisfies e(x) " < 0.
We report the results in Figure (b) and have the following observations. At each step > 1.

---

0.05
100
0.003
6,000
0.5
Divations
0.00
0000
200
o(n)
0.0
050
00203
40,00
6.033
-0.05
-0.5
0072
-0.10
-1.0
Step.
Step r
-0.15
-1.5
10
10
(a) Stability analyses.
(b) Plasticity analyses.
Figure 4: The illustrations on stability and plasticity: the left (a) showcases deviations in the align-
ment score from A 1:t-13 the right (b) represents the average of high-order terms for step > 1.
UCF101 (T-V)) Step
ESC50 (T-A)
NYUDv2 (VD)
Clotho (T-A) step
VGCSound IT-A stop
5GGSound_s 5 (TAII stop 6
Snop
step
VGGSound_5 (A-I) step
TVL (V-TA) step
TVL (T-TA) step 9
TVL (V-7) step 10
LVVI (V-TH) step 11
ImageSind LanguageBind UniBind
Figure 5: The curves plot batch-wise training losses w.r.f. different modality- methods.
the averaged o(1) is negative and close to O, which confirms claim in Theorem 6 in practical scenario. Furthermore, the training loss converging supports the plasticity as well, in Section 4.3.
4.3 More Analyses and Justifications
Training loss curves. We evaluate the stability and plasticity through training losses batches. The batch-wise training loss illustrates the learning status at the own step (plasticity). The results showcased in Figure 5. From the observation, after the model is optimized by new datasets, and the batch-wise loss also successfully converges. These findings demonstrate that DNS reaches balance between stability and plasticity.
method can achieve stable training loss even
Imbalanced modality data. We construct a specialized dataset with modality pairs of text-video (tn) and text- audio (t,, 12, (3), to assess performance in imbalanced sce- narios. We test text-video at step (with less overlap) and text-audio at step 2 (with the most overlap), where greater overlap indicates worse forgetting and larger performance drops. As shown in the Table 3, DNS consistently outperforms Vanilla and CILA, demonstrating superior robustness.
Best
Vanilla
CILA
DNS
(T-VI) ti (T-A)
87.72
85,34
85,54
87.92 47.50
49.25
41.00
43.75
Table 3: Performance comparison under imbalanced modality data scenarios.
Efficiency analyses. DNS maintains efficiency for CMCL, which differs from rapeay-based methods that require additional buffer memory and further in-
Imagettedd Linguagebird Ulvitlan
S415
troduce more computational costs. Here we evalu-
Training time
ate the efficiency of our methods by recording the time training, as shown in Figure The replay-
40
20
based method requires more training time compared both the vanilla method and DNS. DNS introduces a slight increase in training time due to the gradient projection and SVD approximation (less than Is), while still preserving overall training efficiency.
Vanilla
Replay- bassd
DNS (ours)
Figure 6: The time recording (8) for vanilla, replay (DER), and DNS training for one step.
Misaligned & noisy pairs. tios of (0.01, 0.05, 0.1, 0.2, 0.3, 0.4, 0.5). The results are shown in Table 4. As misalignment in- creases, performance decreases across methods. However, DNS consistently outperforms CILA, demonstrating stronger robustness. We also add Gaussian noise to the training dataset at scales of
create misaligned modality pairs by exchanging features ra-

---

Misaligumont
ratio
Ace
B
Recall
BWTR
Ace
BWT
Recall
BWT
CILA
DNS
CILA
DNS
CILA
DNS
CILA
DNS
CILA
DNS
CILA
DNS
CILA
DNS
CILA
DNS
0.01
49.39
52.04
-3.52
0.11
38.03
40.32
-1.72
-1.35
50.37
52.18
-3.12
-0.40
38.46
40.65
-1.25
-1.11
0.05
46.45
46.98
-1.81
2,93
37.46
39.41
-1.32
-1.22
47.42
50.74
-5.31
-1.79
34.59
39,79
-1.11
-2.66
-7.94
0.2
33.33
35.18
1,26
1.39
33.91
36.63
-1.49
-1.85
39.60
43.82
-9,54
-6.32
32.5%
37.29
-5.09
-3.56
0.3
28.73
29.27
-0.80
1,46
32.12
34.09
-1.9
2.32
34.45
39.48
-13.80
-11.15
30.62
35.77
-7.41
.4.30
19.91
24.97
-5.59
0.17
29 29.43
32.84
-2.00
31.92
39
-14.36
-11.63
29.30
-853
-5.26
0.5
17.32
19.78
-6.48
-4300
27.58
30.55
-3.86
4,03
28.52
37.31
-18.62
-11.29
28.70
33.35
-7.94
.6.72
Table 4: Performance comparison under misaligned modality pairs (left) and noisy pairs (right).
0.45
Traning Loss
Recalligs
0.85
0.40
0.80
VTA
VTH
VIT
Modality pair sequence
Modality palr sequence
Modality pair sequence
vanilla
-
-
5e-1
16-1
-0-
5e-2
-
16.2
56.3
-0-
10-3
-
5e-4
-
10
6.5
Traning Loss
Reralliss
0.85
5.5
0.80
0.375
0.75
4.5
T4
WTT
Modality pair saquence
Modality pair Soquence
Madality pair sequence
Se-2
5e-3
Figure 7: Detailed training loss (left), recall@5 (middle), and accuracy (right) results on the data at the first step. The top figure indicates the results for different Amin. while the bottom is for different weight decays, with ImageBind as the backbone. The red line indicates the performance of the vanilla method in CMCL.
{0.01, 0.05, 0.1, 0,2, 0.4, 0.5). Performance decreases with increasing noise, but DNS outper- forms CILA across all metrics, showcasing greater robustness.
Hyperparameter analysis. We provide detialed results the step-wise training loss and perfor- mance changes on the data at the first step shown in Figure 7 with ImageBind as the backbone. Compared to the vanilla method, DNS maintains a stable performance with continual data training. We also provide empirical results on other models in Figures 8, and 9, under different hyperparam- eters settings (cf., Appendix F). More than that, report the detailed variations at each step in Figures 10, 11, and 12 (cf., Appendix F).
5 Conclusion
In this paper, we introduce and formally define continual multimodal contrastive learning (CMCL), research question on how to incrementally integrate multimodal data through contrastive learn- ing. We specify the definitions of stability and plasticity in CMCL. Theoretically, we derive a novel method, termed DNS, which employs dual-sided sradient-projectie strategy balance the incor- poration of new modality pairs while preserving existing multimodal knowledge effectively. The parameter gradients are projected from one side its own modality and another side of the inter- acted modalities onto subspaces. Any newly-projected gradient within these subspaces makes no effect on the prior effective parameter spaces. Furthermore, two upper bounds are provided with theoretical insights on both stability and plasticity, and support the effectiveness of our gradient- projection method. Empirical results across seven datasets demonstrate our method's superiority over current advanced continual learning baselines, which confirm its robustness and efficiency.
10

---

Acknowledgements
This research/project is supported by the National Research Foundation, Singapore under its Na- tional Large Language Models Funding Initiative (AISG Award No: AISG-NMLLP222-0002)) Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do reflect the views of National Research Foundation, Singapore.
Xiaobo Xia also supported by MoE. Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China (Grant No. 2421002).
References
[1] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
[2] Dianzhi Yu, Xinni Zhang, Yankai Chen, Aiwei Liu, Yifei Zhang, Philip S Yu, and Irwin King. Recent advances of multimodal continual learning: A comprehensive survey. arXiv preprint arXiv:2410.05352, 2024.
[3] Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(10):12113- 12132, 2023.
[4] Xiaohao Liu, Xiaobo Xia, Zhuo Huang, and Tat-Seng Chua. Towards modality generaliza- tion: A benchmark and prospective analysis. arXiv preprint arXiv:2412.18277, 2024.
[5] Xiaohao Liu, Xiaobo Xia, See-Kiong Ng, and Tat-Seng Chua. Principled multimodal repre- sentation learning. arXiv preprint WrXiv:2507.17343 2025.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple frame- work for contrastive learning of visual representations. In ICML,, pages 1597-1607, 2020.
[7] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pages 9729-9738, 2020.
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momen- contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[9] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang, Cross-modal contrastive learning for text-to-image generation. In CVPR, pages 833-842, 2021.
[10] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, pages 1148-8763, 2021.
[1I] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In CVPR, pages 15180-15190, 2023.
[12] Shaohan Huang, Li Dong. Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models. In NeurlPS, pages 72096-72109, 2023.
[13] Yuanhuiyi Lyu, Xu Zheng, Jiazhou Zhou, and Lin Wang. Unibind: Lim-augmented unified and balanced representation space to bind them all. In CVPR, pages 26752-26762, 2024.
[14] Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, al. Freebind: Free lunch in unified multimodal space via knowledge fusion. arXiv preprint 248.:22405.0488,, 2024.
11

---

[15] Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretrain- ing to n-modality by language-based based semantic alignment. arXiv preprini arXiv:2310.01852, 2023.
[16] Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. CVPR, pages 184-193, 2021.
[17] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In ICCV, pages 9516-95122 9525, 2021.
[18] Yichen Wen, Zhiquan Tan, Kaipeng Zheng, Chuanlong Xie, and Weiran Huang. Provable contrastive continual learning. In ICML, pages 52819-52838, 2024.
[19] Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance with self- supervupev for incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[20] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. In ICML, 2024.
[21] Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint 2.Xiv:3309.03000 2023.
[22] Lei Zhang, Yunshui Li, Jiaming Li, Xiaobo Xia, Jiaxi Yang, Run Luo, Minzheng Wang, Longze Chen, Junhao Liu, and Min Yang. Hierarchical context pruning: Optimizing real-world code completion with repository-level pretrained code llms. arXiv preprint arXiv:2406. 18294, 2024.
[23] Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355, 2024.
[24] Leheng Sheng, Changshuo Shen, Weixiang Zhao, Junfeng Fang, Xiaohao Liu, Zhenkai Liang, Xiang Wang, An Zhang, and Tat-Seng Chua. Alphasteer: Learning refusal steering with principled null-space constraint. arXiv preprint arXiv:2506 070222, 2025.
[25] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learn- ing audio concepts from natural language supervision. In ICASSP, pages 1-5, 2023.
[26] Hu Xu, Gargi Ghosh, Po- Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding. arXiv preprint arXiv:2109. 1004444, 2021.
[27] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904-4916, 2021.
[28] Shuo Chen, Chen Gong, Jun Li, Jian Yang, Gang Niu, and Masashi Sugiyama. Learning contrastive embedding in wiw-dimesiional space. In NewrIPS, pages
2022.
[29] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[30] Karol J Piczak. Esc: Dataset for environmental sound classification. In ACM MM, pages 1015-1018, 2015.
[31] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rghd images. In ECCV, 2012.
[32] Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: large-scale audio-visual dataset. In ICASSP, pages 721-725, 2020.
12

---

[33] Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP, pages 736-740, 2020.
[34] Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, and Ken Goldberg. A touch, vision, and language dataset for multimodal alignment. arXiv preprint arXiv:2402.13232, 2024.
[35] Xinyu Jia, Chuang Zhu, Minzhen Li, Wenqi Tang, and Wenli Zhou. Livip: A visible-infrared paired dataset for low-light vision. In ICCV, pages 3496-3504, 2021.
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. In NeurlPS, 2019.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[38] David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learn- ing. In NeurlPS, 2017.
[39] Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual leamning a strong, simple baseline. In NeurlPS, pages 15920-15930, 2020.
[40] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwing et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):352133526 2017.
[41] Ang Bian, Wei Li, Hangjie Yuan, Mang Wang, Zixiang Zhao, Aojum Lu, Pengliang Ji, Tao Feng, et al. Make continual learning stronger via c-flat. In NeurlPS, pages (608-7630, 2024.
[42] Wofk, Diana and Ma, Fangchang and Yang, Tien-Ju and Karaman, Sertac and Sze, Vivienne. FastDepth: Fast Monocular Depth Estimation on Embedded Systems. In ICRA, 2019.
[43] Justin Kerr, Huang Huang, Albert Wilcox, Ryan Joque,J Ichnowski. Roberto Calandra, and Ken Goldberg. Self-supervised visuo-tactile pretraining to locate and follow garment features. arXiv preprint arXiv: 13042, 2022.
[44] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877: 2023.
[45] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Chan- ning Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, pages 776-780, 2017.
[46] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene under- standing benchmark suite. In CVPR, pages 567-576, 2015.
[47] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Ro- hit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, pages 18995-19012, 2022.
[48] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakr- ishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint 1609:: 2016.
[49] Jun Xu. Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In CVPR, pages 5288-5296, 2016.
[50] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
13

---

[51] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sud- heendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, pages 6047- 6056, 2018.
[52] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In ICCV, pages 2556-2563, 2011.
[53] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In CVPR, pages 248 255, 2009.
[54] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. 2023.
[55] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML., pages 19730-19742, 2023.
[56] Renrui Zhang. Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 2023.
[57] Roger Penrose. A generalized inverse for matrices. In Mathematical proceedings of the Cambridge philosophical society, volume 51, pages 406-413, 1955.
[58] JK Baksalary and R Kala. The matrix equation ax- yb= c. Linear Algebra and its Applications, 25:41-43, 1979.
[59] Xiaohao Liu, Xiaobo Xia, Jiaheng Wei, Shuo Yang, Xiu Su, See- KionKiong Ng, and Tat-Seng Chua. Calibrated multimodal representation learning with missing modalities. arXiv preprint arXiv:2511.12054 2025.
[60] Run Luo, Yunshui Li, Longze Chen, Wanwei He, Ting-En Lin, Ziqiang Liu, Lei Zhang, Zikai Song, Xiaobo Xia, Tongliang Liu, al. Deem: Diffusion models serve the eyes of large language models for image perception. arXiv preprint arXiv:2405. 15232, 2024.
[61] Zhuo Huang, Chang Liu, Yinpeng Dong, Hang Su, Shibao Zheng, and Tongliang Liu. Ma- chine vision therapy: Multimodal large language models can enhance visual robustness via denoising in-context learning. In ICML, 2024.
[62] Shuo Yang, Zhaopan Xu, Kai Wang, Yang You, Hongxun Yao, Tongliang Liu, and Min Xu. Bicro: Noisy correspondence rectification for multi-modality data via bi-directional cross- modal similarity consistency. In CVPR, pages 19883-19892, 2023.
[63] Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. Gui-rl: A generalist rl-style vision- language action model for gui agents. arXiv preprint urXiv:2504.10455 2025.
[64] Yuchen Zhou, Jiayu Tang, Shuo Yang, Xiaoyan Xiao, Yuqin Dai, Wenhao Yang, Chao Gou, Xiaobo Xia, and Tat-Seng Chua. Logic unseen: Revealing the logical blindspots of vision- language models. arXiv preprint arXiv:2508. 1117, 2025.
[65] Shuo Yang, Peize Sun, Yi Jiang, Xiaobo Xia, Ruiheng Zhang, Zehuan Yuan, Changhu Wang, Ping Luo, and Min Xu. Objects in semantic topology. In ICLR, 2022.
[66] Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. Crossclr: Cross-modal contrastive learning for multi-modal video representations. In ICCV, pages 1450-1459, 2021.
[67] Yiwei Zhou, Xiaobo Xia, Zhiwei Lin, Bo Han, and Tongliang Liu. Few-shot adversarial prompt learning on vision-language models. In NeurlPS, pages 3122-3156, 2024.
14

---

[68] Yingzhi He, Xiaohao Liu, An Zhang, Yunshan Ma, and Tat-Seng Chua. Llm2rec: Large language models are powerful embedding models for sequential recommendation. In Pro- ceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining 2025.
2. pages
[69] Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In CVPR, pages 316-325, 2022.
[70] Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip image, text and audio. In ICASSP, pages 976-980, 2022.
[71] Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learn- ing robust audio representations from clip. In ICASSP, pages 4563 2022.
[72] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint 230.::333900611.. 2023.
[73] Zehan Wang, Ziang Zhang, Hang Zhang, Luping Liu, Rongjie Huang, Xize Cheng, Heng- shuang Zhao, and Zhou Zhao. Omnibind: Large-scale omni multimodal representation via binding spaces. arXiv preprint arXiv:2407.11899 2024.
[74] Oriol Barbany, Michael Huang, Xinliang Zhu, and Arna Dhua. Leveraging large language models for multimodal search. In CVPR, pages 1201-1210, 2024.
[75]
Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. Universal multimodal retrieval with multimodal llms. In ICLR, 2025.
[76] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209. 15352, 2022.
[77] Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation. In NeurlPS, pages 47704- 47720, 2023.
[78] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[79] Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In CAICE, pages 405-409, 2024.
[80] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi. Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal compre- hension and creation. In ICLR, 2024.
[81] Yiping Wang, Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin G Jamieson, and Simon Du. Cliploss and norm-based data selection methods for multimodal contrastive learning. In NeurIPS, pages 15028-15069, 2024.
[82] Simon Schrodi, David T Hoffmann, Max Argus, Volker Fischer, and Thomas Brox. Two effects, one trigger: on the modality gap, object bias, and information imbalance contrastive vision-language representation learning. arXiv preprint wxXiv:240400998 2024.
[83] Zhuo Huang, Gang Niu, Bo Han, Masashi Sugiyama, and Tongliang Liu. Towards out-of- modal generalization without instance-level modal correspondence. In ICLR, 2025.
[84] Sebastian Thrun and Tom M Mitchell. Lifelong robot learning. Robotics and Autonomous Systems, (5(1-2):25-46 1995.
[85] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of contin- ual learning: Theory, method and application. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 2024.
15

---

[86] Weixiang Zhao, Zhuojun Li, Shilong Wang, Yang Wang, Yulin Hu, Yanyan Zhao, Chen Wei, and Bing Qin. Both matter: Enhancing the emotional intelligence of large language models without compromising the general intelligence. In Findings of the Association for Computa- tional Linguistics: ACI. 2024, pages 11157-11176, 2024.
[87] Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Sapt: A shared attention framework for parameter- efficient continual learning of large language models. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 11641-11661, 2024.
[88] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In ECCV, pages 524-540, 2020.
[89] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv: 19100 2018.
[90] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40((2):2935-25)) 2947, 2017.
[91] Xialei Liu, Mare Masana, Luis Herranz, Joost Van de Weijer, Antonio M Lopez, and An- drew Bagdanov. Rotate your networks: Better weight consolidation and less catastrophic forgetting. In ICPR, pages 2262-2268, 2018.
[92] Kibok Lee, Kimin Lee, Jinwoo Shin, and Honglak Lee. Overcoming catastrophic forgetting with unlabeled data in the wild. In ICCV, pages 312-3211 2019.
[93] Itay Evron, Edward Moroshko, Rachel Ward, Nathan Srebro, and Daniel Soudry. How catas- trophic can catastrophic forgetting be in linear regression? In COLT, pages 4028-4079, 2022.
[94] Xuyang Zhao, Huiyuan Wang. Weiran Huang, and Wei Lin. A statistical theory of regularization baasei continual learning. In ICML, 2024.
[95] Jachong Yoon, Sachoon Kim, Eunho Yang, and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In ICLR, 2020.
[96] Ching-Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In NeurIPS, 2019.
[97] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In ICLR, 2018.
[98] Gido M Van de Ven, Tinne Tuytelaars, and Andreas S Tolias Three types of incremental learning. Nature Machine Intelligence, ((12):1185-111) 2022.
[99] Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-increment learning: survey and performance evaluation on image classification. IEEE Transactions on Patlern Analysis and Machine Intelligence, 2022.
[100] Zifeng Wang, Zizhao Zhang. Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pages 631-648, 2022.
[101] Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, and Wenwu Zhu. Decouple before interact: Multi-modal prompt learning for continual visual question answering. In ICCV, pages 2953-2962, 2023.
[102] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans- formers: An occam's ''s razor for domain incremental learning. In NeurlPS, pages 5682-5695, 2022.
[103] Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In /CML, pages 41677-41693, 2023.
16

---

Appendix
A Implementation Details
18
A.1 Training and Inference Procedures
18
A.2 Datasets
19
A.3 Modality-Binding Methods
20
A.4 Baselines
21
A.5 Evaluation Settings
22
B Proofs of Theoretical Results
23
B.1 Proof of Theorem 3
23
B.2 Proof of Theorem 4
25
B.3 The Upper Bound of Loss Stability
26
B.4 The Upper Bound of Plasticity
27
B.5 Extension to Any Steps
28
Technical Lemmas
28
D Related Literature
29
D.1 Multimodal Contrastive Learning..
30
D.2 Continual Learning
30
Supplementary Experimental Settings
F Additional Experimental Results
31
Fl hyperparameter Analysis
31
F2 Performance variations across Steps
31
G Broader Impact Statement
31
H Reproducibility
33
I Limitations
33
J Potential in Scientific Fields
33
17

---

A Implementation Details
A.1 Training and Inference Procedures
Here we provide the training algorithm flow in Algorithm 1 and the corresponding pseudo-code. The inference algorithm flow is demonstrated Algorithm 2. Algorithm describes the Continual Multimodal Contrastive Learning training process with our method DNS, which enables model to progressively learn from new multimodal data while maintaining stability in previously learned modality alignments.
Algorithm DNS for Continual Multimodal Contrastive Learning (Training)
Require: Inputs:
Pre-trained encoders {9m MEE and follow-up learner (W
WMMM MEE for each modality m,
learning rate 7).
A sequence of training datasets {x, J't1, each containing modality pairs.
Feature covariance matrix {Z J mEEM and (ZZt:t)mme MEM (initially t= 1).
Ensure: Updated model parameters (WT)) (i.e. (0% that preserve alignment stability on previ- ously seen data.
1: Initialize features covariance storages 12 and (Z to zero.
2: Initialize local parameters from (W") MEM.
3: for f = 1 to N do
4: Prior to the new step {m, m'):
5: ① Retrieve feature variance { mme(m.m' and (Z (Z
6:
Compute the projectors:
7:
P&t t Z
t <iss
Pet + Z 2
P'att t Tae. P'et t Zm' 2
8:
Train on the new dataset x.m.m'
9:
Repeat until convergence:
① Sample mini-batch {(x"", (x""")""))) )) from x.m.m
10:
11:
Compute representations and the multimodal contrastive loss:
Lm.m' (xmm(x"; W Iod (x" "" w.
3 Compute the raw gradients:
Ommmn
3Lmmm
VW
V
7Wm
+ Project the gradients so that old alignments are preserved (Theorem 4): P'zu
A Wm
VW.
P&r
VW.
A AWm
VW
P VW.
14:
Update the parameters:
W Wm AWW, WT + W A....
15:
End mini-batch.
Update the feature covariances (step-wise or batch-wise):
Z 94-11 m +
(Z
Zzg..
7m 2
7m A: L1:st
+
(Z
Tzm'
n,
n1
18:
7m 2t+11
Re-1 mm
<to (Z
Zmd
mm
Re-1 7m
R, 21:c
n. (Zm)
Z
19:
20:
end for
21: Return: (WT) preserving stability for all previously learned pairs (yn, m').
18

---

DNS provides convenient way to modify gradients before the optimizer step. By applying struc- tured transformations using precomputed projection matrices, it ensures better optimization while keeping the implementation modular and flexible. The pseudo-code for gradient updates is shown below.
Pseudo-code for gradient updates
def peanteegradiettssssstee retrieve gradients
gradienti self.P_1_ gradienti self Pel P_1 gradient2 - self.P_2. gradient2 self P_2
overwrite the gradients
invoke the function before backward propagation model dadaategraddenta () optimizer (tep
Algorithm 2 describes the inference process for Continual Multimodal Contrastive Learning. The goal is to perform inference on a given test sample from a specific modality and generate aligned representations.
Algorithm DNS for Continual Multimodal Contrastive Learning (Inference)
Require: Inputs:
Pre-trained encoders {9th JEEM and trained learner [W") JMEM for each modality m, A test sample (or query) in modality m, denoted x"
Ensure: Inference outputs (e.g., aligned representations, retrieval scores).
1:
Obtain representation:
W "imm(""))
zm -W Im (x"").
2: Perform downstream task:
3: For example, retrieve the best match in another modality m' via
max (z T m
(or apply other alignment criteria).
4: Output the alignment scores, predicted labels, retrieved results,
A.2 Datasets
Below, we introduce the used datasets in more detail.
UCF101 [29]* is widely used dataset for action recognition across 101 action categories, sourced from YouTube. It extends the UCF50 dataset, introducing greater diversity and real-world complexity with significant variations in camera motion, object appearance, viewpoint, background clutter, and illumination. The 101 action categories span 5 broad types: 1) Sluman-Obbeck Interaction (e.g., cutting in kitchen); 2) Body-Motion Only (e.g., jump rope and Tai Chi); 3) Human-Human Interaction (e.g., boxing and head massage); 4) Playing Musical Instruments (e.g., playing violin and playing tabla); 5) Sports (e.g. baskethall dunk and pole vault). In our case, we select examples for each category for training, and 10 examples for each category for testing to balance the distribution.
hhttps://wwe wttv/ ucf. edu/data/UCF101. php
19

---

ESC50 [30]* is a collection of 2,000 five-second environmental audio recordings, designed for benchmarking methods in environmental sound classification. It con- sists of 50 semantically distinct classes, each containing 40 examples, grouped into 5 major categories: 1) Animals (e.g, dog, cat, and frog); 2) Natural Sound- scapes & Water Sounds (e.g., rain, sea waves, and thunderstorm); 3) Human Non- Speech Sounds (e.g. laughing, sneezing, and footsteps); 4) Interior/Domestic Sounds (e.g., door knock, washing machine, and clock alarm); 5) Exterior/Urbai Noises (e.g. car horn, airplane, and fireworks). All clips have been manually extracted from public field recordings available on Freesound. org. To evaluate the performance, randomly split the data with a ratio of 4:1 for training (1,600) and testing (400).
NYUD>2 [31, 42] is dataset originally for depth estimation and semantic segmentation, which features video sequences of indoor scenes captured with both RGB and depth cam- eras from Microsoft Kinect. We use the preprocessed version of the original NYU Depth V2 dataset linked above. It is also used the TensorFlow dataset?. In this version, 47,584 examples are used for training, and 654 examples are for testing.
VGGSound-S [32]* is extracted from content-sharing platform, YouTube, consisting of short clips of audio sounds. It consists of 10-second video segments collected "in the wild", which ensures that the sound source visually evident in each clip. The dataset spans a wide range of acoustic environments and diverse noise conditions. making it highly rep- resentative of real-world applications. To facilitate the training and evaluation, we down- sample a small version of VGGSound, namely VGGSound-S, which consists of 10,000 examples for training and 2,000 examples for testing.
Clotho [33]² is an audio captioning dataset, with all audio examples being from the Freesound platform. The duration of the audio is in the range of 15 to 30 seconds. Each audio has captions up to 20 words in length, collected by AMT or specific protocol for crowd-soureing audio annotations. We only keep one caption per audio in our case and use its development set (3,840) for training and evaluation set (1, (1,045) for testing.
TVL [34]'0 is large-scale image-touch dataset designed for multimodal learning, combin- ing two datasets: 1) SSVTP [43], a robot-collected dataset with 4,587 image-touch pairs, manually labeled with image annotations; and 2) HCT, a large-scale dataset with 39,154 synchronously captured in-the-wild image-touch pairs, labeled using GPT-4V [44] based visual inputs. We use its provided split for training and testing.
LLVIP [35]" is a visible-infrared paired dataset designed for low-light vision applications, which contains 30,976 images (15,488 pairs) captured in extremely dark scenes. All images are strictly aligned in time and space. We also use its default split for training and testing.
A.3 Modality-Binding Methods
Table 5: Details of different modality-binding methods, which include the supported modalities VV+Visioon, T+*Txxt, VIdeo, Autii, D+-eptth, THemalllll, IMUeiarrial Measure- ment Unit, EV+EEent, and P++Point cloud), feature dimension, and the number of datasets for pre-training,
Modalities
Feature dimension
#Datasets
ImageBind LanguageBind UniBind
V, T, VI, A, D, TH, IMU 1,024
4
V, T, VI, A. D, TH
768
6
V, T. VI, A, TH, P
1,024
13
httpp://cc...ulllol/rrrrr//tttts html
"l httppp:/bopptaaaa...gtt.tboi/i/
20

---

Here we introduce three Modallty-bindiigg methods used in this work (also check the details in Table 5).
ImageBind [11] aligns six modalities, i.e., images, text, audio, depth, thermal, and IMU, into a joint embedding space, with vision-relevant modality pair data. utilizes 4 datasets, containing video-audio (Audioset [45]), image-depth (SUN RGB-D [46]), image-thermal (LLVIP [35]), and Video-IMU (Ego4D [47]), for pre-training.
LanguageBind [15]] takes language (i.e., text) as the bind across different modalities, including video, infrared, depth, audio and text. Languagebind is pre-trained on the col- lected dataset VIDAL-10M [15], which is extracted from 6 datasets: YouTube-8M [48], MSR-VTT [49], COCO [50], AVA [51], HMDB-51 [52], and ImageNet [53].
UniBind [13] learns unified and balanced representation space with large language models (LL.M)) for data augmentation. All modality embeddings are aligned with the em- bedding centers yielded from LLM-augmented data. 1,000 descriptions for each category are generated via LLMs (GPT-4 [44] and LLaMA [54]), and multimodal data descriptions are obtained by multimodal LLMs like BLIP2 [55] and LLaMA-Adapter [56].
A.4 Baselines
Below, we discuss the used baselines in more detail.
GEM [38] utilizes an episodic memory to store observed examples for each task. GEM computes gradients the current task and compares them to gradients the stored episodic memory. The inner product between these loss gradient vectors is assumed to be positive ((V,, V&) 0, vk < t) so that the proposed parameter update V, is unlikely to increase the loss at the previous tasks.
DER & DER++ [39] maintain the network's logits from the past tasks with being compared to the logits at the current task. In particular, DER++ further introduces the ground-truth labels serving
additional regularization to stabilize the optimization. In our case, since we do not have explicit labels, we directly use a binary class to determine whether the modality pairs are from the same instance.
EWC [40] utilizes the Fisher information matrix to estimate the importance of each pa- rameter from previous tasks. Besides, EWC penalizes significant deviations from previous optimal values. As a result, adds a quadratic constraint term to the loss function. The posterior is approximated mation.
Gaussian distribution and resolved via Laplace approxi-
L [17] is replay-based continual contrastive learning method. In more detail, Co modifies the supervised contrastive loss by viewing the examples within current tasks as the anchor, while examples past tasks the negatives. The instance- wise relations from the previous model are distilled to preserve the learned knowledge. Therefore, two objectives are combined with controllable hyperparameter for the optimization of model parameters.
C-FLAT [41] introduces an optimization method to enhance the existing continual learn- ing method by promoting loss landscape flatness. It incorporates zeroth-order and first- order sharpness- optimization, balancing both local minima selection and landscape smoothness. C-FLAT is also encapsulated "plug-and and-play" optimizer. In our case, implement this method built upon Co?L..
CILA [18] decomposes the continual learning objective into two components: supervised contrastive loss and a distillation loss, following Co²L. In particular, CILA introduces an adaptive distillation induced by theoretical analyses. The distillation coefficients are dy- namically updated based on the ratio past distillation and contrastive losses.
http:///ghttttaaoaototoaaaaaaaaa "nttps://githan httpe/th...co///--L///////bB//-/
21

---

We implement these methods following public repositories like mammoth 15, C. and Gradien- IEpisodicMemory""
A.5
Evaluation Settings
Evaluation metrics. We use @ecall@k, Acc, and BWT for retrieval and classification tasks. BWT indicates the anti-forgetting capability. Recall@kk measures the ability of a model to retrieve the correct item among the top-k candidates.
Formally, for dataset with queries, Recall@k is defined as:
n Σ 1=1
Recall @
I(r(1) S 1),
(11)
where II(-) is the indicator function that returns the condition is true and otherwise; r(.) denotes the position of the ground truth item for the i-th query in the ranked retrieval list. For classification tasks, average accuracy (Acc) is employed to measure the proportion correctly predicted labels. Specifically, given the ground-truth label ys and the predicted label for each instance in datasets with size of n, accuracy is given by:
a I(y: = 9:).
Acc:
(12)
Here ii is calculated by finding the closest categories vectors with the query vector, i.e., (z.") zz where z represents the textual category vector. BWT measures the effect of learning from new datasets on the performance previously learned datasets. Let T be the total number of training steps. Different steps involve different datasets. That Re is the performance at step t immediately after the model learned on the t-th dataset; ReT the performance on i-th dataset after leaning at the final step T. BWT can be formulated as:
T-1 Σ
BWT:
(ReT Reg)).
(13)
Note that for different tasks, we compute corresponding backward transfer metrics, i.e., BWTR for the retrieval task and BWTA A for the classification task.
Hyperparameter settings. We elaborate on the hyperparameter settings to facilitate reproducibil- ity. We employ linear layers for each modality built upon well-trained modality-binding models (see Appendix A.3). This architecture is also similar to UniBind [13]. In practice, we set the same dimension between the input and output features, W Ra and initialized them unit matri- ces. On the one hand, it can directly maintain the original performance from the prior encoders at the initial step. On the other hand, the full-rank property indicates Col(ZT") C Col WT"), which further weakens our suppose.
During the training. employ the Adam optimizer. All models are trained for 5 epochs at every step. The order is consistent for 10 independent trials. In general, we set the learning rate to 0.0001 and the batch size to 64. The weight decay in (0, 0.5, 0.1, 0.05, 0.01, 0.005, 0.001). As for the SVD approximation, we set the minimal eigenvalue in the range of {0, 0.1, 0.01, 0.001). We provide the hyperparameter analysis with empirical results in Appendix F.
22

---

B Proofs of Theoretical Results
B.1 Proof of Theorem 3
Recall:
Threorem 3. Let the global parameter update follow (W?")) "NTT:TTT""" ") n/W, and the original parameter update denoted as:
W = (W?-)) W
W = ((W?)) VW. + (VW?")
W - - )VW"))
(14)
Then, the stability holds if
W = WWP'WP.
(15)
where P' and P are the space projectors for Zi= and 1, respectively.
Proof.
Expand (Z,-1;,) ) 2, with the weight updating:
(Z,-1 Z
(16)
(21-1;+) (WTL) ) (W
Z 1,+
(17)
(z (-1)
WT 1- (Z.
VW. 4-1
(18)
A la
( -, ) (VWT
W
1 2,-1: + 7 (Z, (WW"") V.. 4 1.
(19)
A
M2
(,, ((WI9) TW? + (VW?")) W - 7 (VW"") VW,"
Z,=1
1it-1
(20)
For the stability to hold, the perturbations introduced by the weight updates must not interfere with the alignment. Thus, the following terms must vanish:
(WW)) TWW"" + (VW.") W T (WW"))
VW!?
0.
(21)
Z' Z'ERXX
WeRdxd
ZEROxN
Question. How to find the solution that satisfies Z' WZ 0 with minimal deviation from the original weight W, i.e., solving:
min
TT WTWZ- 0. W WIl. s.
(22)
W
① Find the general solution for ZT WZ = O.
According to the Lemma solve the equation Z' WZ 0 and obtain its general solution:
W
Z T YzZ Y (Z'z"t) T Yzz Y (P') TP') -PYPP
trivial
non-trivial
(23)
where Y is arbitrary, P' and P are the col-space projectors of Z and Z, respectively.
Any element W set (Y P'YP E (dddd satisies ZT Wz
2 Find the optimal solution for Z' Wz = 0. Now given the set:
s= Y - P'YP Y ddd]]
(24)
23

---

where P E Rdxd and P' E Raxd are projection matrices, (i.e., P P and p = P\) and their L2 norms are either 1 or 0 (see Lemma 4), Given the matrix W E Raxd we hope to find the closest element in set S, in other words, we need to solve the following equation:
min Y P-YP- WIP.
(25)
Here, verify the convexity of the above objective function.
The expression for S(Y) ca written as:
- J(Y) I P'YP- WIl (Y (Y
(26)
|| || denote the Frobenius norm. The gradient of f(Y) is given by:
Vyf(Y) - - P'YP W - --(P - P'YP - -w)p- =2 - -PYP-PP-W- W
(27)
We then vectorize the expression using the properties of the Kronecker product:
vec(P'YP) (P P'))ee(Y).
(28)
The gradient can be rewritten as:
vec Vyf(()) = 2I PII-P& P') vec -W)
(29)
The second-order derivative (Hessian matrix) is defined the derivative of the gradient with respect to YY). Therefore, we have:
Övec (V)))) 2I 2 ovec(Y))
(30)
where I is the identity matrix of appropriate dimension matching Y, and @ represents the Kronecker product.
Due the the eigenvalues of the projection matrices lie in the set {0, 1}, the eigenvalues of P & P either 0 or The corresponding eigenvalue ); (H) is:
2 0) =2 if & P') 0, 22 - =. if & P') =1
(H) -1 P))
(31)
(P & P') E {0, 1}. Since the eigenvalues H are non-negative (0 or 2). is positive semidefi-
nite. Therefore, the function f(Y = || - PPYP W is convex.
Set the gradient VxY(Y) 0, which gives the extreme value condition:
Y P'YP P P w)P.
(32)
Substitute
W into the extreme value condition:
Y-P'YP W = W P'WP- = -P'WP (left-hand side)
(33)
P'(Y-P'YY W)P PPPWP -PWWP W)P P'(-P'WP)J
(34)
= -P''PWPP = -P'WP (right-hand side)
(35)
Therefore, the left-hand side equals the right-hand side, confirming that Y = w satisfies the ex-
treme value condition, implying Y Wi is an extreme point, and further the optimal solution (Y( is convex). Thus, the closest element in the set S to the matrix W is: W w PPWP. D

---

B.2
Proof of Theorem 4
Recall:
Theorem 4. Let the local parameter update follow W W nAW? the stability holds if
A A..". VW. PVW."I''
(36)
AW.
VW? T2 - P'VW,"'P
where P' and P the space projectors for Z (--1;111 and Z t
1-11-1 respectively.
Proof. We restate the problem of how to optimize the model or update the gradients:
Question.
How can we update the gradients VW and VW. to approximate + (VW?")) WT 77 I VW,")) to W P'WP.
S T/C - when y + 0. where denotes the product of the maximum gradient norms, omit this high-order term and analyze the remaining terms. Therefore, our goal is to update the gradients through AW = Ti (VW t. ) and AWr
VW Assumption 7. Since [l W.r
To (WW))) ), and achieve ((WT))) (AT")TTWTT AW? + (AW?") W closed to W - P'WPP T and T2 are the transformation functions
functiton for gradients.
According to Lemma 9, we have the general solution for solving for AX YB = C:
x A+C A + (I AtA)M,
(37)
Y = (I- AA CB N (I AAII
For simplification, let A (W T B WT X VWTT. Yo (VWT")T and the right-hand side C being W P'WP18
Before obtaining the solution, we consider whether C satisfies the condition. The first term W ==
AX, + YBB satisfies (I AA!)((I ( 0, as:
(I - AAJ)AA, YI B'B) = -I-
+ (I BAT B = 0.
A-A=0
(38)
We then analyze the second term:
- P'P(LIBBBBBB (I- AAI)P'AX(I B²B) ABP (39)
Suppose to satisfy (I - AAT)P'WP(I BBB) = 0, the columns space of Z should be a subset of the columns space the parameter W (i.e., A). Similarly, (Z.T;) C col col). That means, at step - 1, W,-1 should be trained well to capture the key features from the input data Z,-1. According to the Lemmas 10 and 11, the condition satisfies for the general solution.
Afterward, we can update the solutions:
Y (I -I- AAUJCB (I-AA)NBB
(40)
Here omit the high-order term for simplicity in W.
25

---

-I- - -I
(41)
-I- (I (I- A - N AAIINBB
(42)
(I-AAM -I--AA A (AX+ + YoBBB' - (1 P''PP N -
(replace W)
(43)
=I AA II AA PA AAT J
(44)
=II- AA_ (I AA-A -I- P'WPB'
(45)
P'Y + +(I (A-AA PPYY, BPB BB
(set N =Y P'Y.BPBI)
(46)
-Yo-P'Y,P P (VW?")) -
(47)
X + -
(48)
A'(W P'WP) A + (I AJAJM
(49)
-AWW A A + A'AM
(50)
-AX + Y,B)
- +(I (I-ATA)M
(51)
=
=A'XX, A Y A'P'WP + +I-
(52)
A'AX, + AY A'W'PP A - BPB
(set M Xx)
(53)
X - PXXP- VW. P'
(54)
Therefore, can update the gradients:
W.: = VW.T PVW " P'
(55)
AW. = VW. P'VW."T "p.
B.3
The Upper Bound of Loss of Stability
Recall:
Theorem 5. In CMCL, the upper bound of loss stability is:
A am -m < 1²² |Z 2|a| L. F(VW),
(56)
where F represents the interactions between different updated gradients VWm. and
VWm Specifically, F(VW) := V VW "/2 [/ + VV
2
2
Proof. Here we define the measuring of the loss of stability, which implemented by the difference between A m2. and A. mi m222 mi.
1-1t
1
m2,22 | 1-11
A
A
mm
([ ((WT)) AW (AW.") WiT (AW" A AW?
4 112-
(57)
We first analyze each term:
(WT)) T AW. = -WW") VW. (W P'VW.
(58)
(AWT") WT = p m.
(59)
26

---

) (VW. PVW."''') VW. P'VW.''P)
(60)
-VWW") V (V (V'W"""''""""""" P
(61)
-P'(VW.') + P(VV""" PPPW""
PP'
(62)
Thus, update the deviation computation:
|
12 12
(63)
|1 P'WP)Z
m2
(" AW," -,,1, 2l2
(64)
11122 (1 (AW")) AW.") Zi
(65)
$ Z 1 ll
(sub-muliplicatiive property of matrix norms)
(66)
(I(VW.")) VW:" I| + I( p'vw.''P Plla
(67)
+ +P((VW
Il | T
PVW:"
"P'VW."' P
(triangle inequality)
(68)
$ < 12,
(69)
(1 12 VWT VW:""" + Vw.""> + Vw," [el"""""""""""""
(|lll2 1)
(70)
Here we use to denote the interactions among different gradient norms VV
F(WW) 2 V |22| VW," ||2
// 1/2 + VW."
(71)
Therefore, obtain the upper bound of loss of stability:
A m
1-L
A ma
1-1;-111
lll $ x²²
| 11 2 ll2 F(VW).
(72)
B.4 The Upper Bound of Plasticity
Recall:
Theorem 6. By transforming the gradients, the loss Le update step I.rrmm the previous loss Li-1 is following:
C, 1 5
(73)
keep the plasticity when o(n) < 0 at the time t.
Proof. Here we consider the capability to learn from new modality paired data (plasticity), and provide the upper bound from the loss optimization perspective. We know that
L(W!h: W!
L(W W. WT 77 (VW.") (AW"). (VW.", AWT?)) + o(n).
Ce-1
(74)
Analyze the first-order term:
W AW?" (WW AW?)
(75)
27

---

(VWT' VW. PVW, P + ( VW?" P'V
(76)
-|
VV 1 (VW,") PVW. P' (VWT" P'
(77)
IVW
F (VW," TM PV P') + VW,
(VW?" P'VW," "PPP P)
(78)
2
(VW?" PVW." P + V
(VWT' P'VW,"'P
(79)
F
2 VW.
(VW,", VW."") + /W. (WW.' VW?")
(80)
VWW
V P + | 1| |V
(81)
(82)
Since lo(nil
when yf - O, there exists a v such that [o(n)] < (VV +
(VWT" ) Therefore, Lu - - 0.
B.5 Extension to Any Steps
Theorem 8. Ler Zet = Zei-1 + Zi-1, where Z = ni-1 21-12,-1- Then, the column space of Zez is equal to the column space of [Z, Zz...., Zr-1].
Proof. First consider the base case 2). The column space of Z<a is span(Zi) }. which equal the column space of [Z,]. Z<2 Zi Zí Z Thus, the statement holds for i 2.
Assume that for some &: 2 2, Col(Z<x) = Col( [Z,, Zz..... Zx-1)). Then, consider the inductive
Step (1 = + 1). By definition, Z<k+1 = 2cck + Zg. From the inductive hypothesis, Col (Z<)) = span span/Z,, Zz...., -E Since Zμ 1 Zi, R its column space is [Za]] The column space
ni
of Z<c+1 is the sum of the column spaces of Zekk and ZR. which is:
Col(Z<x+1) = Col(Zex) + Col(2,) = span Z2, Ze...., Zu- 1} + span [Z,] span Zz, (83)
ZR}
The column space of [Z,, Zz...., Ze] is also span [Z, Zz...., Zz}. Therefore, Col(Z<<+1) Col( Zz...., ZR]), and the statement holds for t = k 1.
Due to the same column space, we can use the feature covariance to compute the projection matrix to save the computational cost.
C Technical Lemmas
Lemma (Four properties of pseudo-inverse matrix). If At is the pseudo- matrix of A, then satisfies:
AA'A=A A, MAAt=AL (AA1) AAT, (ATA)T A'A.
(84)
Lemma (Idempotent property). For the matrix P =I--UU UU where (U,A, U) SVD(A), satisfies p2_ P.
Proof.
(I- (I - -I u - - UU = =I -U =P (85)
Lemma 3 (Symmetry property). For the matrix P = ==I- -- UUT where (U,A, U} SVD(A),
satisfies P = P.
28

---

Proof. - (UU")) = =I- U0 P
Lemma (Eigenvalues of idempotent matrix). For an idempotent matrix P, satisfies P P, its eigenvalues À E {0,
Proof. If Pv Au for the nonzero vector o, then apply P again to both sides of the equation:
pdu= P(u)) = P(ee) AP A²v = Av.
(86)
(87)
Thus, we have 2 = Au, which indicates E {1,0) since v +
Lemma 5. A matrix A is positive semi-definite and only if all its eigenvalues non-negative.
Lemma 6. If the Hessian matrix H of f(-) is positive semidefinite (PSD) for all x in the domain,
then f(-) is fonextinnntton
Lemma 7 (Solving AXB = C [57]. A necessary and sufficient condition for the equation
AXB C have solution
AA =
(88)
in which case the general solution is
x=
ABCB
Y - A'AYBB'
(89)
particular
homogeneous
where Y is arbitrury.
Lemma (Solving AX YB [[88).. Let A Rmak B g2 and C Rmxn For
AX YB
(90)
the equation has a solution E R E mmxi if and only if
- B'B)
(91)
If this is the case, the general solution of (90) has the form
x = AAC+ (I - A A)M, Y = -I (I- CA + +N - -I- ABB
(92)
with M E kx and N E mR being arbitrary:
Lemma (Solving AX YB C). AX + YB C has a solution if and only if
(I (I-AA -
(93)
If this is the case, the general solution is:
x = A + A'AMM,
(94)
== (I- - AAI)NBBI.
Lemma 10. (I AAU)PA 0 where P' is projection matrix, then AAPPA
Lemma 11. If AA'P'A P'A. where P is orthogonal projection matrix, then AiP'A is projection matrix as well.
Proof. If P' is an orthogonal projection matrix, it satisfies P P', since (A'P'A)2 ATP' APP'AAP''A = A'P'P'A A'P'A, ATP'A is projection matrix.
D Related Literature
We review the topies of multimodal contrastive learning and continual learning, which are highly related to this work.
29

---

D.1 Multimodal Contrastive Learning
Multimodal learning endows the model with the capability to understand and generate multiple modalities [59, 60, 61, 62, 63, 64, 65]. Multimodal contrastive learning builds upon the princi- ples of cross-modal contrastive learning and has demonstrated remarkable success across various tasks [66, 9, 67, 68). The fundamental philosophy of contrastive learning is to align different views of the same instance while distinguishing between distinct instances. In unimodal contrastive learn- ing, instance is augmented into different views and brought closer in the feature space, following the paradigm of self-supervised learning [6. 69]. This concept extends naturally to cross- modal scenarios, exemplified by models such as CLIP [10], ALIGN [27]. VideoCLIP [26], and CLAP [25]. Recently. there has been growing interest in multimodal contrastive learning, which incorporates a broader range of modalities, e.g. AudioCLIP [70] and WAV2CLIP [71]. Besides, ImageBind [11] introduces a vision-centric approach by leveraging large-scale multimodal data. Similarly, LanguageBind [15] and PointBind [72] shift the focus toward text and point cloud modal- ities, respectively. Other methods either enhance these frameworks (e.g., FreeBind [14]) or explore alternative center-free strategies (e.g., OmniBind [73] and UniBind [13]).
Multimodal contrastive learning empowers models to learn rich and informative latent representa- tions across different modalities. This enhances Al's ability to comprehend inter-modal relation- ships, which is necessitated for many real-world tasks (e.g.. multimodal search 75], multimodal generation [76, 77, 78], and recent multimodal Al-assistant [79, 20, 12. 80J. Furthermore, the ex- ploration of modality uniqueness and commonness (i.e. invariance) contributes to advancing the foundational research on machine learning and deep learning [81, 82, 83).
Despite the significance, the innate data data-intensivvness raises questions on how to learn it efficiently. Large-scale multimodal datasets are difficult to acquire single collection process, and training models on such diverse data from scratch incurs substantial computational costs. A more practical and scalable way is to incrementally integrate emerging data into existing modality-aligne models. Motivated by this, we systematically investigate this research problem and propose novel method with rigorous theoretical analysis and guarantees.
D.2 Continual Learning
Continual learning, also referred to as incremental learning, aims to enable models to learn from new data effectively in a streaming way while retaining previously acquired knowledge [84, 85, 86, 87]. The primary challenge lies in integrating new information without causing catastrophic forgetring. Among existing approaches, replay-basee methods have showcased superior performance through buffer memory [88]. This buffer typically stores the previous samples, which are then leveraged in the subsequent training steps under additional constraints (e.g., meta-experience replay (MER) [89], gradient episodic memory (GEM) [38], and dark-experience replay (DER) [39])- Another promi- methods, mainly focuses stabilizing network updates. These
nent category, regularization-base methods either constrain the variance of network parameters (weight regularization [90, 40, 91, or regulate the model's output space (function regularization [93, 94]) to mitigate forgetting. In contrast, parameter isolation methods dynamically allocate distinct subsets of model parameters to different tasks, thereby preventing interference and reducing forgetting [95, 96, 97]. Existing continual learning works mainly focus on two settings of fask-incremental learning (TIL) [98] and class-incremente learning (CIL) [99]. Recent advancements have extended continual lear- ing to multimodal scenarios, giving rise to multimodal continual learning (MMCL) [2]. Beyond conventional CL methods, MMCL incorporates prompt-based approaches as a specialized adapta- tion [100, 101, 102]. These methods introduce learnable prompt parameters as part of the input, enabling the model to differentiate tasks based on the learned prompts.
Research in continual learning has advanced significantly while leaving a blank in continual mul- timodal contrastive learning (CMCL). Some solutions from continual uni-modal or cross-modal contrastive learning show potential while overlooking the modality complexity. Existing MMCL approaches primarily focus on task-specific continual learning scenarios, neglecting fundamental representation-leve challenges, which are task-agnostic. The absence of task boundaries and intro- duced modality complexity make extending these methods to CMCL highly challenging. To fill this blank, we take the initiative to investigate CMCL and propose solutions with theoretical analyses.
30

---

We believe our contributions provide strong foundation for future research, which inspires further advancements in both academia and industry.
E
Supplementary Experimental Settings
In this section, we elaborate the experimental settings, especially concerning the baseline im- plementation. For all buffer-required baselines, like GEM and DER, we set the number of buffer memory 256. In the following, we list the specialized settings for each baseline. We implement the Vanilla training with the same settings with DNS, while omitting the specialized one of Amin- GEM equipped with margin of 0.5. For DER, we set the penalty weight 0.1. Note that + is built upon DER. The additional hyperparameter of DER++, i.e., a, which controls the strength of ground -tuut prediction, is set 0.1 for the case where LanguageBing is the backbone and 0.001 for ImageBind and UniBind. For EWC, the regulation for the quadratic constraint term is set to 0.5. For Lo the distillation power is set to 1. and the temperature in CL is set to C-FLAT uses Adam as the base optimizer which is the default optimizer in our evaluation, and the gradient reduction method is the mean. The radius p 0.2 for the gradient norm, and the balance hyperparameter of loss terms is also set to 0.2. For CILA, the balancing distillation coefficient is set to 5.0. All the experiments are conducted with 4XNVIDIA RTX A5000 GPUs and 256GB memory.
Additional Experimental Results
We report the detailed experimental results in this section.
F.1 hyperparameter Analysis
We provide the training loss and performance (recall@5 and accuracy) changes in terms of different hyperparameter settings, including weight decays and minimum eigenvalues in truncated SVD. The results are illustrated in Figures 7, 8, and 9, corresponding to different modality-binen -iindin methods. demonstrates that our method, DNS, achieve superior performance compared to vanilla training and maintain the performance over time.
F.2 Performance variations across Steps
We also track the training loss and recall variations at each step, illustrated in Figures 10, 11, and 12. In most cases, DNS exhibits decreasing training loss and improving performance, particularly when using ImageBind and UniBind as backbones. Moreover, DNS maintains stable training loss and performance after completing designated training step.
G Broader Impact Statement
This research aims to contribute positively to the machine learning and multimodal learning fields by advancing continual multimodal contrastive learning, which enables models to dynamically in- tegrate and retain knowledge from diverse multimodal data sources over time. Although we believe our work is unlikely to have direct negative societal impacts, we acknowledge the importance of considering potential misuse scenarios, such the exploitation of continuously adapting systems in unethical surveillance or misinformation campaigns. The broader implication of our study is that it elevates multimodal models with the capability to seamlessly adapt to evolving multimodal environ- ments with minimal retraining, which makes them particularly suitable for real-time applications in adaptive human-computer interaction and autonomous systems. Such advancements could lead to more versatile, resilient, and context-aware AI applications across various real-world domains.
31

---

Modality-Bindding: LanguageBind
Traningg LoSs
6.5
0.00
Recaligs
Acc
5.5
0.50
5.0
0.7
14
AVV
WTT
TM
V.
TM
Modality
Modality
Modality
16.1
-0-
5e3
0.6
Traning Lass
Recalless
0.4
0.4
TV0
T-A
10
TTA
1V7
AVV
WITH
Modality pair sequence
Modality pai Sequence
Modality pair sequence
vanilla
Se-1
1e-1
-0-
Se-2
1e.2
-0-
Se-3
1e3
Figure 8: Detailed training loss (left), recall@5 (middle), and accuracy (right) results on the data at the first step. The top figure indicates the results about different Amin. while the bottom is for different weight decays, with LanguageBind as the backbone. The red line indicates the performance of the vanilla method in CMCL.
Modality-Binding: UniBind
Tariin Loss
6.5
6.0
Recallis
0.450
0.85
5.5
0.400
5.0
0.75
V.0
T.
VTR
VTT
VTT
Modality sequence
Modality pail sequence
Modality pair
Traning LOS5
8.0
Recal@e
0.85
5.5
0.900
0.80
5.0
0.375
0.75
4.5
V-T
WT
V-TH
Modality pair sequence
Modality palr sequence
Modelity pair sequence
Vanilla
- Se:1
1el1
-0- Se.2
-
1
-0-
Se:3
-0- 163
Figure 9: Detailed training loss (left), recall@55 (middle), and accuracy (right) results on the data at the first step. The top figure indicates the results about different Amin- while the bottom is for different weight decays, with UniBind the backbone. The red line indicates the performance of the vanilla method in CMCL.
32

---

Modality-Bindding: ImageBind
UCF101 (T-V)
NYUDv2
Cletho (T-A)
Traim na Loss
TWI
V
T.A
TW
AX0
WCTH
TA4
T-A
V.D
T4
TA
TN
VST
WITH
TA4
TA
V-0
T-
TA
TW
AVI
VCA
VCT
VCTH
TAI
T-
T4
T-A
NN
V-TA
V-2
WITH
VGGSound (TA))
VGGSound (TV))
VGGSound (AVVI)
TVL (V-TA)
Traiiing Loss
TM
V0
AV
WITH
T-A
VO
TM
TM
A-
17
WT
WITH
V40
TM
Modality pair
TVL(TTA)
TVL (MT)
LLVIP (V
Training LOSS
Dwn step (point)
Modal-relevant stop (paint)
Modal-relevant
V1
12
V40
Modality palr sequence
Modality pair sequence
Modality sequence
MYUDV2
Clotho
VGGSound
VGCSound
Reca Mggs
(AV0
(VTA)
TVL(W-)
(V-TH)
Rocallgs
Modality sequence
Modality
Modality
Modality
Modality sequence
stop
Modai-rdevant stop
Recali@5
Recallg10
Figure 10: Detailed experimental results of the training loss (the top three rows) and recall (the last two rows) over a sequence of modality pair data, with ImageBind the modality-binding backbone. Notably, the y-values are normalized for better visualization.
H Reproducibility
We provide comprehensive implementation details, including illustrative algorithm flows and pseudo-code. Additionally, source codes are publicly released.
I Limitations
This paper fills the blank of continual multimodal contrastive learning in the development of both multimodal learning and continual learning research. Specifically, this paper introduces method that projects the gradient onto specialized subspaces, where the incorporation of new data does not interfere with previously acquired knowledge. Although our work is supported by systematic theoretical analyses and empirical studies, does not yet examine the generalization errors [103] of the proposed method in downstream tasks. We regard addressing the limitation direction.
future
J Potential in Scientific Fields
The CMCL framework holds significant potential for applications complex fields like medicine and neuroscience, where modalities such as MRI, CT scans, and clectrophysiological signals
33

---

Modality Modality-Bindiing LanguageBind
UCF101 (T-VI)
NYUDv2 (V-D)
Cletho (T-A)
Traim Trainin Loss
TWI
V.0
T.
T-
TN
AX0
V1
WCTH
TA0
T-A
V.0
T4
TA
TA0
VSTA
WITH
TA
V-0
T.
TA
TW
AVI
VCA
VCT
VSTH
TAI
T-K
V.D
T4
T-A
TN
V-3
WITH
VGGSound (TAA)
VGGSound (TV))
VGGSound (AVI)
TVL (V-TA)
Training Loss
TW
V0
TM
VT
WITH
1-A
VO
TM
W
WSTH
V1 T.
124
WITH
Modaliyy pair
(TL(TTA)
TVL (VT)
LLVIP (V-TT)
Training LOSS
Dwn step (point)
Modal-relevant stop (polnt)
Modal-relevant
130
T2
14
PM
14
12
Modality palr sequence
Modality pair sequence
Modality sequence
Clotho
VGGSound
VGGSound
Recc Migs
(A-0
(VIA)
TVL(W-)
Rocaligs
RPSPP P4gE
Modality sequence
Modality
Modality
Modality
Modality
stop
Modai-rdevant stop
Recali@5
Recall@10
Detailed experimental results of the training loss (the top three rows) and recall (the last two rows) over a sequence of modality pair data, with LanguageBind as the modality- backbone. Notably, the y-values are normalized for better visualization.
Figure 11:
quire incremental integration. In medical diagnostics, CMCL can incrementally incorporate diverse imaging modalities. In neuroscience research, framework can facilitate the progressive under- standing of neural correlates by continually integrating modalities like functional MRI, EEG, and MEG, thereby providing richer insights into brain functions and disorders. CMCL. presents excit- ing avenue for future interdisciplinary research to enhance both interpretability and performance.
34

---

Bindlity UniBind
UCF101 (T-1)
NYUDv2
Clotho
Tra ming Los5
TM
V
T.A
TAN
VCTH
TA0
T-A
V.0
T-4
T
VTT
WITH
V-0
T-A
TAN
TAA
T-A
1-
VTTA
V-1
WITH
VGGSound (T-A)
VGGSound (TVV)
VGGSound (A-VI)
TVL (V-TA
Training Loss
TM1
T.
TW
T2TR
T-A
TAV
AN
V-TA
WITH
TV
V-0
TAV
V-TA
T-TA
WITH
ty pair sequence
(TL(TTA)
TVL (T)
LLVIP (VTTT)
Training LoSS
Own step (poin0)
Modal-relevant slep (paint)
Modal-rolevant
VO
WITH
TA
12
WITH
WTTH
Modality pair sequence
Modality pair sequence
Modality sequence
NYUDv2
Clotho
VGGSound
VGGSound
Roccal MBS
(A-0)
(V-)
Recallgss
ASEPE
Modality pair sequence
Modality pair
Modality
Modality pair sequence
Modality sequence
stop
Modal-reievantooo step
Recali@5
Recalig10
Figure 12: Detailed experimental results of the training loss (the top three rows) and recall (the last two rows) over sequence of modality pair data, with UniBind as the uodality-bindiig backbone. Notably, the y-values are normalized for better visualization.
35

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification: We elaborate the contributions and scope in both the abstract and intro- duction, supported by our theoretical and empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We state the examination of the generalization errors, as discussed in Ap- pendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "LLmitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a apeech- speech-to-text system might not be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
36

---

Answer: [Yes]
Justification: We provide a complete proof in Appendix B, with assumptions required for our theoretical results and theoretical lemmas (in Appendix C).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: We provide extensive experimental details in the main content (see Sec- tion 4.1) and appendix (see Appendix and Appendix E).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
37

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide the full instructions to access the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
crriiissioooocccyiiy olicy) for more details.
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips. cttpcc/ppp/c. pwwppc/llppcgeppp /taliiigggies////eee Caaes//oodeuuuuia iss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Algorithm flows and pseudo-code are provided in Appendix A well. Data statistic is also detailed in Section 4.1 and Appendix A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the experimental results with average performance (mean in ten independent trials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
38

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative еπτог rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the implementation details in Appendix E, including the hardware used for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attps:///net //teurip//eurip.... htce/pulli///Etissiiiieeeee
Answer:
[Yes]
Justification:
We adhere the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide the impact statement in Appendix G and highlight the potential in scientific fields (see Appendix J).
Guidelines:
The answer NA means that there is no societal impact of the work performed.
39

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper involves only publicly available datasets and models. No specific safeguards were necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
of the model, for example by
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the relevant papers and provide links to existing assets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
40

---

If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, saperswithoode com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: This paper does not introduce new assets. Therefore, documentation is applicable.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
This paper does not involve crowdsourcing research with human subjects. Therefore, the inclusion of participant instructions, screenshots, and compensation details is not applicable.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
41

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This paper does not involve LLMs for any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: cattps://neuripge //////uupp..... cc/Conffrencess/22555 for what should or should not be described.
42


Reviews:
[{"review_id": "aQfQzx5otF", "reviewer": "Reviewer_82Az", "summary": "This paper defines Continual Multimodal Contrastive Learning, a new problem of learning from a stream of multimodal data pairs. The authors propose a novel gradient projection method, Dual-sided Null Space, to prevent catastrophic forgetting by ensuring new updates do not interfere with prior knowledge. The method is supported by strong theoretical guarantees and empirical results showing it significantly outperforms continual learning baselines in balancing stability and plasticity.", "questions": "As the number of continual learning steps (t) increases, how does the memory and computational cost required to store and compute the cumulative feature covariance matrix (Z<t) for the projectors scale?\nThe paper defines stability as maintaining the alignment score for old data pairs. While this is a clear metric, is it sufficient to guarantee that the overall semantic structure of the embedding space is preserved?  For example, will it affect downstream tasks?", "limitations": "Although the DNS method is more efficient than replay-based approaches, it requires constructing and maintaining a feature covariance matrix from all prior learning steps and computing projections via SVD approximation. Meanwhile, the paper mainly focus on retrieval and classification tasks and gains marginal improvement. Furthermore, the method's generalization and effectiveness on more complex downstream tasks, such as multimodal reasoning, have not been validated.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "It introduces and formally defines Continual Multimodal Contrastive Learning (CMCL), establishing a new and important research direction.  It proposes a novel method, DNS, that is both theoretically justified and empirically proven to be effective at learning continually without forgetting.\n\nHowever, the method itself relies on learning from pairs of data, which isn't really how people learn—we take in everything at once. So the big question is whether this pair-by-pair approach can ever truly grasp the complex connections you only get when combining three or more senses[1].  Therefore, not updating existing knowledge does not seem to be a  long-term strategy. \n\n[1] Cicchetti, Giordano, et al. \"Gramian Multimodal Representation Learning and Alignment.\" arXiv preprint arXiv:2412.11959 (2024).", "was_revised": true, "final_justification": "The authors’ response has to some extent addressed my concerns, but no additional experimental data were provided to substantiate it; therefore I am maintaining my score.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> W1: However, the method itself relies on learning from pairs of data, which isn't really how people learn—we take in everything at once. So the big question is whether this pair-by-pair approach can ever truly grasp the complex connections you only get when combining three or more senses[1]. Therefore, not updating existing knowledge does not seem to be a long-term strategy.\n\n**Response**\n\nThanks for your insightful feedback! Preserving alignment across modality pairs is challenging due to the complex interactions of modal gradients, as shown in Equation 5. Extending this to multiple modalities within a single training step further complicates theoretical and practical guarantees. However, our method, DNS, potentially handles alignment beyond pairwise data. Below is our analysis.\n\nThere are two primary methods to align multiple modalities simultaneously. The first method aggregates features from all modalities (e.g., via concatenation and projection) into an omni-modality feature, aligned with an anchor modality (e.g., text) [1]. In this case, DNS can be easily adapted by substituting one modality with the omni-modality feature. The second method, exemplified by GRAM [2], directly aligns multiple modalities using a Gram matrix, where each entry represents the inner product of two modalities, aligning with our alignment score definition. DNS can be conceptually adapted to GRAM as follows.\nGiven the Gram matrix $\\mathbf{G} = \\mathbf{Z}^\\top \\mathbf{Z}$, where $\\mathbf{Z}$ is the concatenated multimodal features, GRAM optimizes the parameters of one modality ($m_1$) by aligning it with other modalities ($m_2, m_3, \\ldots$). The alignment score is defined as $\\mathbf{Z}^{m_1\\top} \\mathbf{Z}^{\\perp m_1} \\in \\mathbb{R}^{n \\times (|\\mathcal{M}|-1)n}$, where $\\mathbf{Z}^{m_1} \\in \\mathbb{R}^{d \\times n}$ represents modality $m_1$, and $\\mathbf{Z}^{\\perp m_1} \\in \\mathbb{R}^{d \\times (|\\mathcal{M}|-1)n}$ includes features of all other modalities. This can be expressed as:\n\n$(\\mathbf{Z}^{m_1})^\\top \\mathbf{Z}^{\\perp m_1} = (\\mathbf{Z}^{m_1}_{\\ast})^\\top (\\mathbf{W}^{m_1})^\\top\\mathbf{W}^c \\mathbf{Z}^c\\_{*}$,\n\n $\\mathbf{W}^c =[\\mathbf{W}^{m_2}, \\mathbf{W}^{m_3}, \\ldots, \\mathbf{W}^{m_{|\\mathcal{M}|}}]$,\n\n$\\mathbf{Z}^c\\_{\\ast} = \\mathrm{diag}(\\mathbf{Z}^{m_2}\\_{\\ast}, \\mathbf{Z}^{m_3}\\_{\\ast}, \\ldots, \\mathbf{Z}^{m_{|\\mathcal{M}|}}\\_{\\ast})$.\n\nUsing this formulation, DNS can adapt by replacing  the modal projector $\\tilde{\\mathbf{P}}$ with a new one built upon $\\mathbf{Z}^c_*$, aggregated data from all other modalities, while maintaining the self-modal projector $\\mathbf{P}'$. We hope that this possible solution is helpful and can address your concerns.\nThis capability demonstrates that DNS can scale to multiple modalities, supporting its long-term viability. We will expand on this in Section 3.3 to clarify our method’s broad applicability.\n\n[1] Chen, Sihan, et al. \"Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.\" NeurIPS, 2023.  \n[2] Cicchetti, Giordano, et al. \"Gramian multimodal representation learning and alignment.\" ICLR, 2025.\n\n> Q1: As the number of continual learning steps (t) increases, how does the memory and computational cost required to store and compute the cumulative feature covariance matrix (Z<t) for the projectors scale? \n\n**Response**\n\nThank you for your thorough review and interest! Our method does not incur accumulating costs when updating the feature covariance matrix across learning steps. As detailed in Section 3.3, \"Extension for Any Steps\", we update the feature covariance matrix ($\\mathbf{Z}_{<t}$) at each step using only the current features, without needing to store or recompute past features. This matrix, in $\\mathbb{R}^{d \\times d}$, maintains a fixed size, ensuring no increase in storage requirements. Therefore, our approach remains computationally efficient with minimal overhead. We will add this explanation in the final version of the paper.\n\n> Q2: The paper defines stability as maintaining the alignment score for old data pairs. While this is a clear metric, is it sufficient to guarantee that the overall semantic structure of the embedding space is preserved? For example, will it affect downstream tasks?\n\n**Response**\n\nThank you for your thoughtful question! The semantic structure in multimodal alignment is not explicitly defined but is implicitly captured through alignment scores, as seen in models like CLIP and ImageBind, consistent with CMCL. From this perspective, our method can preserve the semantic structure of previous modality data because our goal is to keep the previous alignment scores.  However, we also emphasize the learning plasticity. The embedding space is refined with continual learning; a rough semantic structure also becomes more fine-grained along with the training. For example, a factor initially representing \"dog\" may be refined to distinguish \"chihuahua\" after further training, with additional factors like \"poodle\" emerging as the null space is updated. Therefore, such refinement impacts downstream tasks, likely improving performance when tasks depend on precise multimodal alignment.\n\n> L1: Although the DNS method is more efficient than replay-based approaches, it requires constructing and maintaining a feature covariance matrix from all prior learning steps and computing projections via SVD approximation. Meanwhile, the paper mainly focuses on retrieval and classification tasks and gains marginal improvement. Furthermore, the method's generalization and effectiveness on more complex downstream tasks, such as multimodal reasoning, have not been validated.\n\n**Response**\n\nThanks! Regarding your first concern, we clarify the efficiency of updating the feature covariance matrix, which is independent of the number of learning steps (see Q1 and its related response). The efficient analyses (Section 4.3) show that maintaining projectors incurs minimal computational cost. In terms of performance, our method surpasses baselines by up to 3.98% in Acc, and 3.18% in BWT. These improvements are also noted by other reviewers (Reviewer gDQw, 2wop, and yJSf). For more complex downstream tasks, we agree that more tasks would further strengthen our method. However, adapting refined representations for advanced models (e.g., MLLMs) is resource-intensive, and due to time constraints, we could not include more results. We also clarify this in Q2 that enhanced alignment is expected to improve performance in downstream tasks like multimodal understanding that rely on precise alignment. Your suggested investigation will be reflected in the final version. \n\nWe authors sincerely thank you for your professional review attitude and comments. If you have other questions, we are happy to address them to polish this work."}, {"review_id": "6pCAdaGzM8", "reviewer": "Reviewer_gDQw", "summary": "The paper proposes a novel approach to Continual Multimodal Contrastive Learning (CMCL), a problem at the intersection of multimodal learning and continual learning. In CMCL, a model is trained incrementally on emerging multimodal data, where modality pairs (such as vision-text, vision-audio, and text-audio) are introduced sequentially. The core challenge addressed is the trade-off between stability (retaining knowledge from previously learned modalities) and plasticity (learning new modality pairs without forgetting previous ones). The authors introduce a new optimization method that projects parameter gradients onto subspaces to maintain both stability and plasticity during incremental learning. They also provide theoretical guarantees and two bounds that address these aspects. The empirical evaluation of their method across multiple datasets shows superior performance compared to other continual learning methods.", "questions": "Please check the above section.", "limitations": "Please check the above section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1.  The topic of continual multimodal contrastive learning is meaningful and with great practical value.  The proposed method is grounded in real-world issues such as the cost of training models from scratch with large multimodal datasets and the need to incorporate emerging data.\n2. The authors provide a detailed theoretical analysis, with clear definitions of stability and plasticity, and detailed proofs to support their gradient-projection method.\n3. The authors conduct extensive experiments on seven datasets and compare their approach with existing baselines, showing strong empirical results.\n\nWeakness:\n\n1. The proposed gradient-projection method is complex, and while the theory is well-supported, it remains unclear how easy it would be to implement and integrate into existing models or systems. It would be to provide discussion and experiments about the training efficiency, like FLOPs.\n2. It would be more convincing to provide hyper-parameter sensitivity analysis for \\lambda_{min}.", "was_revised": true, "final_justification": "After the rebuttal, the authors solved most of my concerns, and I will keep my positive score.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> W1: The proposed gradient-projection method is complex, and while the theory is well-supported, it remains unclear how easy it would be to implement and integrate into existing models or systems. It would be to provide discussion and experiments about the training efficiency, like FLOPs.\n\n**Response**\n\nThanks! We provide implementation details, including algorithm flow and pseudocode, in Appendix A. Notably, it is easy to integrate into existing models with minor code (only two lines of code for gradient projection). To confirm this, we also adapt our method for three existing encoders, i.e., ImageBind, LanguageBind, and UniBind, in our evaluation. We also provide the efficiency analysis in Section 4.3, indicating its efficiency with negligible time cost. As you suggested, we also provide FLOPs analysis. \n||Complete|Additional |\n|-|-|-|\n|FLOPs|23.73 T|268.44 M|\n\nAs can be seen, by using ImageBind as the backbone with a batch size of 64, the complete procedure requires 23.73T FLOPs (one forward & backward), while feature matrix maintenance and projection consume only 268.44M FLOPs, confirming our method’s efficiency. We are happy to include further evidence as needed.\n\n> W2: It would be more convincing to provide hyper-parameter sensitivity analysis for $\\lambda_{min}$.\n\n**Response**\n\nYes, we agree with you and we have already included the hyper-parameter analysis in Appendix F, examining various $\\lambda_{\\min}$ as suggested. If you have further concerns or questions, kindly let us know. We can provide clarifications. \n\nWe sincerely appreciate your professional and constructive feedback. Your insightful comments have greatly helped improve our manuscript. Should you have any further questions or suggestions, we would be more than happy to address them to further refine and enhance this work."}, {"review_id": "uWGeLjJrMq", "reviewer": "Reviewer_2wop", "summary": "This paper introduces and formulates the problem of Continual Multimodal Contrastive Learning (CMCL), where models must learn from a sequence of datasets with different modality pairs without forgetting previously learned knowledge. To address this, the authors define two principles for CMCL: stability (retaining old knowledge) and plasticity (learning new knowledge). The primary contribution is a novel, theoretically-grounded method called Dual-sided Null Space (DNS) projection. DNS works by projecting the parameter gradients onto a null space where they are prevented from interfering with previously learned modality alignments. The method is evaluated on seven datasets using three different modality-binding models as backbones. The results demonstrate superior performance over several continual learning baselines in both stability and plasticity.", "questions": "1. The construction of null space projectors relies on accumulating feature covariance matrices from past steps. As the number of continual learning steps increases, could this accumulation process lead to information loss or degradation, thereby affecting the precision of the projectors over long learning sequences?\n\n2. The visualization in Figure 2 is difficult to interpret, as the distinction between the top and bottom plots is unclear. Could you please revise the figure or add a more detailed explanation to clarify its key message?\n\n3. Could you provide a more detailed analysis of your method's memory complexity? Specifically, how does the storage cost for the feature covariance matrices scale with the number of tasks and feature dimensions, and how does this compare to the memory requirements of a typical replay-based method?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1.\tThe paper clearly defines the Continual Multimodal Contrastive Learning (CMCL) problem and proposes a theoretically-grounded solution (DNS) to address catastrophic forgetting.\n\n2.\tThe proposed method is rigorously derived and supported by theoretical proofs for stability and plasticity.\n\n3.\tThe method is extensively evaluated on seven datasets against multiple baselines, showing strong performance.\n\nWeaknesses\n\n1.\tWhile the application to CMCL is novel, the core idea of using null space projection on gradients for continual learning has been explored before. The main novelty lies in the sophisticated \"dual-sided\" adaptation for the multimodal setting, but the underlying principle is not entirely new.\n\n2. The DNS method avoids the memory buffer of replay-based methods but requires storing and updating feature covariance matrices to construct the projectors. The paper does not provide a detailed analysis of how the memory footprint of these matrices scales with a very large number of continual learning steps and high feature dimensions.\n\n3. The authors rightly point out in their limitations (Appendix I) that the paper does not examine the generalization errors of the proposed method in downstream tasks. It is plausible that strictly constraining gradient updates to preserve past alignments might limit the model's ability to learn more generalizable features that could be beneficial for an unseen downstream task.", "was_revised": true, "final_justification": "My concerns, especially regarding the method's novelty and its explanation, have been addressed. I maintain my positive review.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n**Response**\n\n> W1: While the application to CMCL is novel, the core idea of using null space projection on gradients for continual learning has been explored before. The main novelty lies in the sophisticated \"dual-sided\" adaptation for the multimodal setting, but the underlying principle is not entirely new.\n\nThank you for recognizing our contribution to CMCL and for the opportunity to clarify our method compared to conventional null space projection! At a high level, our method shares some conceptual similarities with null space projection. However, the conventional null space ($\\mathbf{AX=0}$) is insufficient for CMCL ($\\mathbf{AXB=0}$), as CMCL requires alignment across multimodalities from dual sides ($\\mathbf{A}$ and $\\mathbf{B}$). The conventional null space only cares `one side`, i.e., one modality ($\\mathbf{A}$).  Deriving a rigorous solution for CMCL and implementing it effectively are highly non-trivial, far beyond a straightforward adaptation of null space projection, also reflected in our detailed theoretical derivation. That indicates the significance of our \"dual-sided\" projection, which breaks the constraints of null space in theory and advances the practical implementation of CMCL. We call our method `dual-sided null space` (DNS) to highlight its distinct contribution, rather than framing it as a mere adaptation. In fact, we point out this distinction from null space in Line 177 and will further clarify it in the revised paper.\n\n> W2: The DNS method avoids the memory buffer of replay-based methods but requires storing and updating feature covariance matrices to construct the projectors. The paper does not provide a detailed analysis of how the memory footprint of these matrices scales with a very large number of continual learning steps and high feature dimensions.\n\n**Response**\n\nThanks! In Section 3.3 (\"Extension to Any Steps\"), we explain that our method maintains a global feature covariance matrix, updated solely with data from the current learning step, without storing historical data. This update process is independent of the number of steps, ensuring high efficiency and preventing computational cost accumulation as learning progresses. Regarding feature dimensions, the covariance matrix is defined in $\\mathbb{R}^{d \\times d}$, where $d$ is the fixed and typically small feature dimension compared to the dataset size. Once training begins, $d$ remains constant. Thus, with a fixed feature dimension and step-independent updates, the computational cost remains low and stable throughout training.\n\n> W3: The authors rightly point out in their limitations (Appendix I) that the paper does not examine the generalization errors of the proposed method in downstream tasks. It is plausible that strictly constraining gradient updates to preserve past alignments might limit the model's ability to learn more generalizable features that could be beneficial for an unseen downstream task.\n\n**Response**\n\nThanks for the comment! We understand your concern. In fact, in the research community of machine/deep learning theory, rigorously examining the generalization errors/bounds about contrastive learning is very challenging [1]. Recent studies often focus on bimodal scenarios (e.g., image and text) and linearized contrastive loss [2, 3], whereas the widely adopted non-linear contrastive loss still remains intractable in the current research and beyond our current scope. Actually, we empower DNS with certain flexibility if you want to learning the new more. The projector can be adjusted via $\\lambda_{min}$. A large $\\lambda_{min}$ will provide more plasticity for learning more generalizable features. We also provide a comprehensive analysis on hyperparameters in Appendix F. Hope these could address your concern.\n\n[1] HaoChen, Jeff Z., et al. \"Provable guarantees for self-supervised deep learning with spectral contrastive loss.\" NeurIPS, 2021.  \n[2] Zhang, Qi, et al. \"On the generalization of multi-modal contrastive learning.\" ICML, 2023.  \n[3] Xue, Yihao, et al. \"Understanding the robustness of multi-modal contrastive learning to distribution shift.\" ICLR, 2024.\n\n> Q1: The construction of null space projectors relies on accumulating feature covariance matrices from past steps. As the number of continual learning steps increases, could this accumulation process lead to information loss or degradation, thereby affecting the precision of the projectors over long learning sequences?\n\n**Response**\n\nNo, our method is efficient and independent of the number of training steps. The update of feature covariance matrices is equivalent to computing directly on data from all past steps. Therefore, there is also no information loss in our matrix update. We clarify this in Section 3.3 (\"Extension to Any Steps\") with a proof in Apendix B.5.\n\n> Q2: The visualization in Figure 2 is difficult to interpret, as the distinction between the top and bottom plots is unclear. Could you please revise the figure or add a more detailed explanation to clarify its key message?\n\n**Response**\n\nSure! We are willing to improve Figure 2 for better clarity and understanding of our method. Due to image update policies, we describe the revisions in text. Figure 2 depicts the model parameter update process for vanilla training versus DNS training. At step $t$, the gradient $\\nabla \\mathbf{W}\\_t$ is computed from modality pair data (e.g., image and audio). In vanilla training, this gradient directly updates the parameters, overwriting parts overlapping with the previously learned gradient $\\nabla \\mathbf{W}_{t-1}$, particularly around the image modality, which leads to forgetting in continual learning. In contrast, DNS avoids this overlap by projecting the new gradient $\\nabla \\mathbf{W}_t$ onto the non-overlapping subspace, preserving prior knowledge while incorporating the new. We will further clarify these distinctions in the revised paper.\n\n> Q3: Could you provide a more detailed analysis of your method's memory complexity? Specifically, how does the storage cost for the feature covariance matrices scale with the number of tasks and feature dimensions, and how does this compare to the memory requirements of a typical replay-based method?\n\n**Response**\n\nThanks for your interest! We are willing to explain this (also refer to W2). The memory required for feature covariance matrices is fixed at the start of training, depending only on the number of modalities ($|\\mathcal{M}|$) and the feature dimension ($d$), which is significantly smaller than the dataset size ($B$). Thus, our method’s memory cost can be represented as $\\mathcal{O}(|\\mathcal{M}|\\cdot d^2)$. In contrast, typical replay-based methods have memory costs of $\\mathcal{O}(|\\mathcal{M}|\\cdot B \\cdot d)$, that scale with data size, growing substantially as data increases.\n\nWe authors sincerely thank you for your professional review attitude and comments. If you have other questions, we are happy to address them to polish this work."}, {"review_id": "XAmFDaWoG4", "reviewer": "Reviewer_yJSf", "summary": "This paper introduces a novel framework for Continual Multimodal Contrastive Learning (CMCL), where modality-paired data arrives sequentially and integrated without catastrophic forgetting. To address this challenge, the authors propose Dual-sided Null Space (DNS) projection, which modifies gradient updates to lie in null spaces derived from previous modalities’ representations. The authors mainly discussed the stability (retaining knowledge) and plasticity (adapting to new modality information) of CMCL. Theoretically, the authors derive two upper bounds characterizing the loss in stability and plasticity, and empirically validate DNS on extensive multimodal datasets using three modality-binding backbones, achieving strong performance across different tasks with minimal training overhead.", "questions": "Most of the concerns have been raised in Strength and Weakness sections. It would be great if the authors could specifically address the following questions:\n\n1. Can DNS be extended to settings where 3+ modalities are introduced jointly at a step, rather than only pairs?\n2. Have you considered using explicit factorizations to separate shared and exclusive subspaces to account for the modality gap?\n3. What is the assumption on the encoders? How would a poorly pretrained encoders effect the representation quality throughout the time as the update depends on previous steps?\n4. Could the authors provide some visualization analysis of how the representation space evolves over training steps?\n5. How robust is the proposed method under scenarios where the modality pairs could be severely imbalanced or noisy? Can the authors briefly discuss this scenario? \n\nOverall, I enjoyed reading the paper and look forward to authors’ responses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "This paper addresses an interesting and important problem of continual learning for multimodal contrastive models beyond the standard bimodal setups. The proposed method (DNS) is theoretically justified and emprically effective, showing improved stability and plasticity with notable gaps compared to existing continuous learning frameworks. Its ability to plug into different modality-binding backbones and operate in a replay-free manner enhances its practicality and generaliazbility. The contributions are promising and the findings in the paper can lead to interesting discussions and extensions for multimodal community. The pros and cons are discussed below: \n\n**Pros**\n\n- **Strong motivation and interesting problem**: The paper makes a valuable conceptual contribution by formally defining CMCL and distinguishing it from traditional continual learning and multimodal learning. The clear definitions of stability and plasticity is interesting in  evaluating and understanding the setting of continual multimodal learning.\n- **Theoretical justification:** The derivation of dual-sided gradient projection is carefully grounded in linear algebraic principles, leveraging null space projectors to preserve past knowledge.\n- **Strong evaluation:** The experiments span 7 diverse datasets across vision, audio, text, thermal, depth, and tactile modalities. DNS consistently outperforms continual learning baselines across different tasks. The proposed method demonstrates superior performance in terms of information retainment and adaptation.\n- **Flexibility with existing frameworks:** The proposed method is shown to work with ImageBind, LanguageBind, and UniBind, indicating its compatibility with a wide class of pretrained modality-binding architectures. This makes the method flexible in integration with many existing or future works.\n- **Efficiency:** The authors have also demonstrated the efficiency of the proposed method with minimum overhead compared to the vanilla method, in contrast to the replay based methods.\n\n**Cons**\n\n- **Generalization beyond 2-modality pairs**: Although the method supports sequential training on modality pairs, the extension to scenarios where more than two modalities are jointly introduced (have a sequence of three or more modalities) is not addressed. This could have additional bottlenecks to more complex CMCL settings (e.g., sensor systems) where we could obtain >2 continuous modalities at the same time.\n- **Pretrained encoders**: The method assumes modality encoders can be reused or extended without explicitly retraining from scratch. It’s unclear whether the modality-specific encoders (e.g., gm3) are pretrained or jointly trained. It is unclear how does the effectiveness of the pretrained encoders impact the performance.\n- **Modality gaps or factorized latent spaces**: A key challenge in multimodal learning is modality gap, i.e., the representational divergence between modalities. The proposed method operates under the assumption of a unified embedding space, but the subspace relationships (shared vs. exclusive latent dimensions) are not examined. A discussion or experiment here would be great.\n- **Visualizations**: The paper lacks visualizations (e.g., t-SNE of modality representations across steps, heatmaps of alignment matrices) that would help diagnose how the method enhances or preserves the representation space during the updates.\n- **Imbalanced or noisy multimodal scenario**: There’s limited analysis or discussion on when or why DNS might fail, such as in highly imbalanced modality data scenarios, noisy data, or misaligned priors.\n\nI enjoyed reading the paper overall. I have summarized some questions in the later section.", "was_revised": true, "final_justification": "I appreciate the authors’ response and I’m keeping my current positive score.", "rebuttal": "Thanks for your professional and careful review. We respond to your concerns or questions as follows.\n\n> W1: Generalization beyond 2-modality pairs: Although...same time.\n\n**Response**\n\nThanks for the insightful comment! Our method potentially can extend to >2 modalities at one training step. The core alignment is defined as the inner product of two modalities, yet can be extended for more modalities. Recent methods, like VAST [1], employ the onmi-modality, which aggregates all modalities, to align with an anchor modality (e.g., text) for pair-wise contrastive learning. DNS inherently can be extended to these methods. GRAM [2], a seminar work, proposes to align more than two modalities simultaneously by minimizing the volume of a Gram matrix $\\mathbf{G} = \\mathbf{Z}^\\top \\mathbf{Z}$. Every entry in this matrix is exactly the inner product of two modalities, aligning with the definition of our alignment score in Equation 5. The difference is that, one modality (e.g., $m_1$) should be aligned with other modalities together. Therefore, the alignment can be represented as:\n\n$(\\mathbf{Z}^{m_1})^\\top \\mathbf{Z}^{\\perp m_1} = (\\mathbf{Z}^{m_1}_{\\ast})^\\top (\\mathbf{W}^{m_1})^\\top\\mathbf{W}^c \\mathbf{Z}^c\\_{*}$,\n\n$\\mathbf{W}^c =[\\mathbf{W}^{m_2}, \\mathbf{W}^{m_3}, \\ldots, \\mathbf{W}^{m_{|\\mathcal{M}|}}]$,\n\n$\\mathbf{Z}^c\\_{\\ast} = \\mathrm{diag}(\\mathbf{Z}^{m_2}\\_{\\ast}, \\mathbf{Z}^{m_3}\\_{\\ast}, \\ldots, \\mathbf{Z}^{m_{|\\mathcal{M}|}}\\_{\\ast})$.\n\n$\\mathbf{Z}^{m_1} \\in \\mathbb{R}^{d \\times n}$ represents modality $m_1$, and $\\mathbf{Z}^{\\perp m_1} \\in \\mathbb{R}^{d \\times (|\\mathcal{M}|-1)n}$ includes features of all other modalities. With this formulation, conceptually, we can replace the modal projector $\\tilde{\\mathbf{P}}$ with the one constructed from $\\mathbf{Z}^c_*$, aggregated data from all other modalities, while maintaining the self-modal projector $\\mathbf{P}'$. We hope that this possible solution is helpful and can address your concerns.\n\n[1] Chen, et al. \"Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.\" NeurIPS 2023.  \n[2] Cicchetti, et al. \"Gramian multimodal representation learning and alignment.\" ICLR 2025.\n\n> W2: Pretrained encoders: The...performance.\n\n**Response**\n\nThanks! Using a pretrained encoder is a practical implementation for our method, leveraging well-developed open-source model weights. In our implementation, modality-specific encoders are jointly trained, thus possessing certain alignment. However, our analysis does not assume this, allowing for separately trained encoders. We evaluate our method on different types of encoders, like ImageBind and LanguageBind. We can observe that the base model (the pretrained model) with better joint training would lead to a better performance (here LanguageBind beats ImageBind at most cases). Therefore, the pretrained encoders can affect the performance. A better pretrained encoder would lead to better performance. Otherwise, if we use the isolated trained encoders, the performance would also decrease.  \n\n> W3: Modality gaps or factorized latent spaces: A...would be great.\n\n**Response**\n\nThanks! The modality gap phenomenon can be preserved by contrastive learning [3]. Within a unified space, different modalities might lie in different regions [4], i.e., the modality gap. The unified space indicates that we can use a simple inner product to estimate the alignment. Our research starts from and focuses on multimodal alignment scores, using a conventional contrastive learning objective. Therefore, according to [3], our method can also preserve the modality gap. We also visualize the representations and  conduct some experiments. We compute the modality centroid by averaging different modality representations, and estimate their similarity. We have the following table: \n|Centroid similarity|Vanilla|CILA|DNS|\n|-|-|-|-|\n|t=0|9.35|9.35|9.35|\n|t=T|4.14|6.67|9.09|\n\nWe can observe that the modality gap is preserved or even worse for vanilla and cila methods (lower similarity). However, DNS exhibits a slight similarity drop. We will add this analysis in the revised version of the paper.\n\n[3] Liang, et al. \"Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.\" NeurIPS 2022.  \n[4] Schrodi, et al. \"Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models.\" ICLR 2025.\n\n> W4: Visualizations: The...during the updates.\n\n**Response**\n\nThanks!  Because most of the data are pairwise, we do not depict the heatmaps.  We visualize the representations with t-SNE across steps to observe their distribution changes as suggested. We are willing to share the insights with you: the modality gap is still preserved while is slightly changing when continually learning. This is reasonable and aligns with our experiments on W3 and its response. DNS does not influence the learning objective, so it will preserve the modality gap from contrastive learning [3]. However, DNS showcases better robustness, alleviating the enlarging modality gap of other methods. We will add this analysis in the revised version. \n\n> W5: Imbalanced or noisy multimodal scenario: There...priors\n\n**Response**\n\nThanks! It is interesting to test our method under imbalanced, noisy, or misaligned multimodal scenarios. We conduct experiments to evaluate performance across varying levels of noise and misalignment, providing a comprehensive analysis of DNS’s robustness.\n\n**Imbalanced Modality Data Scenarios**: We construct a specialized dataset with modality pairs of text-video (t0) and text-audio (t1, t2, t3), to assess performance in imbalanced scenarios. We test text-video at step 1 (less overlap) and text-audio at step 2 (most overlap), where greater overlap indicates worse forgetting and larger performance drops. As shown in the table below, DNS consistently outperforms Vanilla and CILA, demonstrating superior robustness:\n\n|Imbalanced|Best (at it own step)|Vanilla|CILA|DNS|\n|-|-|-|-|-|\n|t0(T-VI)|87.72|85.34|85.54|87.92|\n|t1(T-A)|49.25|41.00|43.75|47.50|\n\n**Misaligned Pairs**: We create misaligned modality pairs by exchanging features at ratios of 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, and 0.5. As misalignment increases, performance decreases across methods. However, DNS consistently outperforms CILA, demonstrating stronger robustness:\n\n|Misalign|acc(CILA)|acc(DNS)|BWT_acc(CILA)|BWT_acc(DNS)|Recall(CILA)|Recall(DNS)|BWT_R(CILA)|BWT_R(DNS)|\n|-|-|-|-|-|-|-|-|-|\n|0.01|49.39|52.04|-3.52|0.11|38.03|40.32|-1.72|-1.35|\n|0.05|46.45|46.98|-1.81|2.93|37.46|39.41|-1.32|-1.22|\n|0.1|41.46|43.54|0.36|2.00|36.21|38.50|-0.9|-0.90|\n|0.2|33.33|35.18|1.26|1.39|33.91|36.63|-1.49|-1.85|\n|0.3|28.73|29.27|-0.8|1.46|32.12|34.09|-1.94|-2.32|\n|0.4|19.91|24.97|-5.59|0.17|29.43|32.84|-2.6|-2.64|\n|0.5|17.32|19.78|-6.48|-4.30|27.58|30.55|-3.86|-4.03|\n\n**Noisy Data**: We add Gaussian noise to the training dataset at scales of 0.01, 0.05, 0.1, 0.2, 0.3, 0.4, and 0.5. Performance decreases with increasing noise, but DNS outperforms CILA across all metrics, showcasing greater robustness:\n\n|Noise|acc(CILA)|acc(DNS)|BWT_A(CILA)|BWT_A(DNS)|Recall(CILA)|Recall(DNS)|BWT_R(CILA)|BWT_R(DNS)|\n|-|-|-|-|-|-|-|-|-|\n|0.01|50.37|52.18|-3.12|-0.40|38.46|40.65|-1.25|-1.11|\n|0.05|47.42|50.74|-5.31|-1.79|34.59|39.79|-4.41|-2.66|\n|0.1|43.89|50.44|-7.94|-0.69|34.19|40.21|-4.07|-0.66|\n|0.2|39.6|43.82|-9.54|-6.32|32.58|37.29|-5.09|-3.56|\n|0.3|34.45|39.48|-13.8|-11.15|30.62|35.77|-7.41|-4.30|\n|0.4|31.92|39.22|-14.36|-11.63|29.3|34.89|-8.53|-5.26|\n|0.5|28.52|37.31|-18.62|-11.29|28.7|33.35|-7.94|-6.72|\n\nThese results confirm that while performance degrades in extreme scenarios for all methods, DNS consistently demonstrates greater robustness and effectiveness.\n\n> Q1: Can DNS be extended to settings where 3+ modalities are introduced jointly at a step, rather than only pairs?\n\n**Response**\n\nYes. DNS is capable of handling more than 2 modalities if we extract the gradients and conduct the projection as explained in W1 and the corresponding response. \n\n> Q2: Have you considered using explicit factorizations to separate shared and exclusive subspaces to account for the modality gap?\n\n**Response**\n\nThis is an interesting question! We did not consider explicitly factoring the spaces. We attribute the explicit factorizations to the learning objectives, while our method mainly focuses on how to project the gradient to keep the alignment scores. These alignment scores can be maintained or manipulated by different learning strategies. We also conduct experiment for clarification, as shown in the response to W3. A possible way for explicit factorizations can be implemented in the hyperbolic space [5] that preserves the hierarchies.\n\n[5] Ramasinghe, et al. \"Accept the modality gap: An exploration in the hyperbolic space.\" CVPR 2024.\n\n> Q3: What is the assumption on the encoders? How would a poorly pretrained encoders effect the representation quality throughout the time as the update depends on previous steps?\n\n**Response**\n\nGenerally, we have no assumption about the encoders. However, a better pretrained encoder indeed leads to a better performance. We detail this in W2 and the corresponding response. \n\n> Q4: Could the authors provide some visualization analysis of how the representation space evolves over training steps? \n\n**Response**\n\nSure! Due to the policy, we cannot send you the image. We share the insights with you in W4 and the corresponding response. \n\n> Q5: How robust is the proposed method under scenarios where the modality pairs could be severely imbalanced or noisy? Can the authors briefly discuss this scenario?\n\n**Response**\n\nYes! From our experiments, DNS is more robust to imbalanced/noisy/misaligned scenarios. A more detailed analysis can be found in W5 and the corresponding response.\n\nWe authors sincerely thank you for your professional review attitude and comments. If you have other problems, we are happy to address them to polish this work."}]

Author rebuttal:

Meta-review: This paper introduces Continual Multimodal Contrastive Learning (CMCL), a new problem setting where multimodal contrastive models are optimized over sequentially arriving modality pairs rather than trained from scratch. To address this, the authors propose a theoretically grounded gradient projection method that balances stability and plasticity.

This paper received overall positive reviews from the reviewers, with all final scores in the borderline accept to accept range. The authors made a sincere effort during the rebuttal to address key concerns, and reviewers generally acknowledged the responses as satisfactory. No major objections remained after the discussion phase.

While additional experimental validation could further strengthen the work, the current version presents a meaningful contribution. Overall, the AC finds the paper suitable for acceptance based on the positive consensus and the effective resolution of reviewer concerns, though some points may benefit from refinement in the final revision.
Decision: Accept (poster)