Title: Continual Multimodal Contrastive Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Continual Learning

Multimodal Contrastive Learning
Abstract: Multimodal Contrastive Learning (MCL) advances in aligning different modalities and generating multimodal representations in a joint space. By leveraging contrastive learning across diverse modalities, large-scale multimodal data enhances representational quality. However, a critical yet often overlooked challenge remains: multimodal data is rarely collected in a single process, and training from scratch is computationally expensive. Instead, emergent multimodal data can be used to optimize existing models gradually, \textit{i.e.}, models are trained on a sequence of modality pair data. We define this problem as Continual Multimodal Contrastive Learning (CMCL), an underexplored yet crucial research direction at the intersection of multimodal and continual learning. In this paper, we formulate CMCL through two specialized principles of stability and plasticity. We theoretically derive a novel optimization-based method, which projects updated gradients from dual sides onto subspaces where any gradient is prevented from interfering with the previously learned knowledge. Two upper bounds provide theoretical insights on both stability and plasticity in our solution. Beyond our theoretical contributions, we conduct experiments on multiple datasets by comparing our method against advanced continual learning baselines. The empirical results further support our claims and demonstrate the efficacy of our method.  Our codes are available at https://github.com/Xiaohao-Liu/CMCL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for Continual Multimodal Contrastive Learning (CMCL), where modality-paired data arrives sequentially and integrated without catastrophic forgetting. To address this challenge, the authors propose Dual-sided Null Space (DNS) projection, which modifies gradient updates to lie in null spaces derived from previous modalities’ representations. The authors mainly discussed the stability (retaining knowledge) and plasticity (adapting to new modality information) of CMCL. Theoretically, the authors derive two upper bounds characterizing the loss in stability and plasticity, and empirically validate DNS on extensive multimodal datasets using three modality-binding backbones, achieving strong performance across different tasks with minimal training overhead.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper addresses an interesting and important problem of continual learning for multimodal contrastive models beyond the standard bimodal setups. The proposed method (DNS) is theoretically justified and emprically effective, showing improved stability and plasticity with notable gaps compared to existing continuous learning frameworks. Its ability to plug into different modality-binding backbones and operate in a replay-free manner enhances its practicality and generaliazbility. The contributions are promising and the findings in the paper can lead to interesting discussions and extensions for multimodal community. The pros and cons are discussed below: \n\n**Pros**\n\n- **Strong motivation and interesting problem**: The paper makes a valuable conceptual contribution by formally defining CMCL and distinguishing it from traditional continual learning and multimodal learning. The clear definitions of stability and plasticity is interesting in  evaluating and understanding the setting of continual multimodal learning.\n- **Theoretical justification:** The derivation of dual-sided gradient projection is carefully grounded in linear algebraic principles, leveraging null space projectors to preserve past knowledge.\n- **Strong evaluation:** The experiments span 7 diverse datasets across vision, audio, text, thermal, depth, and tactile modalities. DNS consistently outperforms continual learning baselines across different tasks. The proposed method demonstrates superior performance in terms of information retainment and adaptation.\n- **Flexibility with existing frameworks:** The proposed method is shown to work with ImageBind, LanguageBind, and UniBind, indicating its compatibility with a wide class of pretrained modality-binding architectures. This makes the method flexible in integration with many existing or future works.\n- **Efficiency:** The authors have also demonstrated the efficiency of the proposed method with minimum overhead compared to the vanilla method, in contrast to the replay based methods.\n\n**Cons**\n\n- **Generalization beyond 2-modality pairs**: Although the method supports sequential training on modality pairs, the extension to scenarios where more than two modalities are jointly introduced (have a sequence of three or more modalities) is not addressed. This could have additional bottlenecks to more complex CMCL settings (e.g., sensor systems) where we could obtain >2 continuous modalities at the same time.\n- **Pretrained encoders**: The method assumes modality encoders can be reused or extended without explicitly retraining from scratch. It’s unclear whether the modality-specific encoders (e.g., gm3) are pretrained or jointly trained. It is unclear how does the effectiveness of the pretrained encoders impact the performance.\n- **Modality gaps or factorized latent spaces**: A key challenge in multimodal learning is modality gap, i.e., the representational divergence between modalities. The proposed method operates under the assumption of a unified embedding space, but the subspace relationships (shared vs. exclusive latent dimensions) are not examined. A discussion or experiment here would be great.\n- **Visualizations**: The paper lacks visualizations (e.g., t-SNE of modality representations across steps, heatmaps of alignment matrices) that would help diagnose how the method enhances or preserves the representation space during the updates.\n- **Imbalanced or noisy multimodal scenario**: There’s limited analysis or discussion on when or why DNS might fail, such as in highly imbalanced modality data scenarios, noisy data, or misaligned priors.\n\nI enjoyed reading the paper overall. I have summarized some questions in the later section.\n\nQUESTIONS:\nMost of the concerns have been raised in Strength and Weakness sections. It would be great if the authors could specifically address the following questions:\n\n1. Can DNS be extended to settings where 3+ modalities are introduced jointly at a step, rather than only pairs?\n2. Have you considered using explicit factorizations to separate shared and exclusive subspaces to account for the modality gap?\n3. What is the assumption on the encoders? How would a poorly pretrained encoders effect the representation quality throughout the time as the update depends on previous steps?\n4. Could the authors provide some visualization analysis of how the representation space evolves over training steps?\n5. How robust is the proposed method under scenarios where the modality pairs could be severely imbalanced or noisy? Can the authors briefly discuss this scenario? \n\nOverall, I enjoyed reading the paper and look forward to authors’ responses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces and formulates the problem of Continual Multimodal Contrastive Learning (CMCL), where models must learn from a sequence of datasets with different modality pairs without forgetting previously learned knowledge. To address this, the authors define two principles for CMCL: stability (retaining old knowledge) and plasticity (learning new knowledge). The primary contribution is a novel, theoretically-grounded method called Dual-sided Null Space (DNS) projection. DNS works by projecting the parameter gradients onto a null space where they are prevented from interfering with previously learned modality alignments. The method is evaluated on seven datasets using three different modality-binding models as backbones. The results demonstrate superior performance over several continual learning baselines in both stability and plasticity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.\tThe paper clearly defines the Continual Multimodal Contrastive Learning (CMCL) problem and proposes a theoretically-grounded solution (DNS) to address catastrophic forgetting.\n\n2.\tThe proposed method is rigorously derived and supported by theoretical proofs for stability and plasticity.\n\n3.\tThe method is extensively evaluated on seven datasets against multiple baselines, showing strong performance.\n\nWeaknesses\n\n1.\tWhile the application to CMCL is novel, the core idea of using null space projection on gradients for continual learning has been explored before. The main novelty lies in the sophisticated \"dual-sided\" adaptation for the multimodal setting, but the underlying principle is not entirely new.\n\n2. The DNS method avoids the memory buffer of replay-based methods but requires storing and updating feature covariance matrices to construct the projectors. The paper does not provide a detailed analysis of how the memory footprint of these matrices scales with a very large number of continual learning steps and high feature dimensions.\n\n3. The authors rightly point out in their limitations (Appendix I) that the paper does not examine the generalization errors of the proposed method in downstream tasks. It is plausible that strictly constraining gradient updates to preserve past alignments might limit the model's ability to learn more generalizable features that could be beneficial for an unseen downstream task.\n\nQUESTIONS:\n1. The construction of null space projectors relies on accumulating feature covariance matrices from past steps. As the number of continual learning steps increases, could this accumulation process lead to information loss or degradation, thereby affecting the precision of the projectors over long learning sequences?\n\n2. The visualization in Figure 2 is difficult to interpret, as the distinction between the top and bottom plots is unclear. Could you please revise the figure or add a more detailed explanation to clarify its key message?\n\n3. Could you provide a more detailed analysis of your method's memory complexity? Specifically, how does the storage cost for the feature covariance matrices scale with the number of tasks and feature dimensions, and how does this compare to the memory requirements of a typical replay-based method?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach to Continual Multimodal Contrastive Learning (CMCL), a problem at the intersection of multimodal learning and continual learning. In CMCL, a model is trained incrementally on emerging multimodal data, where modality pairs (such as vision-text, vision-audio, and text-audio) are introduced sequentially. The core challenge addressed is the trade-off between stability (retaining knowledge from previously learned modalities) and plasticity (learning new modality pairs without forgetting previous ones). The authors introduce a new optimization method that projects parameter gradients onto subspaces to maintain both stability and plasticity during incremental learning. They also provide theoretical guarantees and two bounds that address these aspects. The empirical evaluation of their method across multiple datasets shows superior performance compared to other continual learning methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1.  The topic of continual multimodal contrastive learning is meaningful and with great practical value.  The proposed method is grounded in real-world issues such as the cost of training models from scratch with large multimodal datasets and the need to incorporate emerging data.\n2. The authors provide a detailed theoretical analysis, with clear definitions of stability and plasticity, and detailed proofs to support their gradient-projection method.\n3. The authors conduct extensive experiments on seven datasets and compare their approach with existing baselines, showing strong empirical results.\n\nWeakness:\n\n1. The proposed gradient-projection method is complex, and while the theory is well-supported, it remains unclear how easy it would be to implement and integrate into existing models or systems. It would be to provide discussion and experiments about the training efficiency, like FLOPs.\n2. It would be more convincing to provide hyper-parameter sensitivity analysis for \\lambda_{min}.\n\nQUESTIONS:\nPlease check the above section.\n\nLIMITATIONS:\nPlease check the above section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper defines Continual Multimodal Contrastive Learning, a new problem of learning from a stream of multimodal data pairs. The authors propose a novel gradient projection method, Dual-sided Null Space, to prevent catastrophic forgetting by ensuring new updates do not interfere with prior knowledge. The method is supported by strong theoretical guarantees and empirical results showing it significantly outperforms continual learning baselines in balancing stability and plasticity.\n\nSTRENGTHS AND WEAKNESSES:\nIt introduces and formally defines Continual Multimodal Contrastive Learning (CMCL), establishing a new and important research direction.  It proposes a novel method, DNS, that is both theoretically justified and empirically proven to be effective at learning continually without forgetting.\n\nHowever, the method itself relies on learning from pairs of data, which isn't really how people learn—we take in everything at once. So the big question is whether this pair-by-pair approach can ever truly grasp the complex connections you only get when combining three or more senses[1].  Therefore, not updating existing knowledge does not seem to be a  long-term strategy. \n\n[1] Cicchetti, Giordano, et al. \"Gramian Multimodal Representation Learning and Alignment.\" arXiv preprint arXiv:2412.11959 (2024).\n\nQUESTIONS:\nAs the number of continual learning steps (t) increases, how does the memory and computational cost required to store and compute the cumulative feature covariance matrix (Z<t) for the projectors scale?\nThe paper defines stability as maintaining the alignment score for old data pairs. While this is a clear metric, is it sufficient to guarantee that the overall semantic structure of the embedding space is preserved?  For example, will it affect downstream tasks?\n\nLIMITATIONS:\nAlthough the DNS method is more efficient than replay-based approaches, it requires constructing and maintaining a feature covariance matrix from all prior learning steps and computing projections via SVD approximation. Meanwhile, the paper mainly focus on retrieval and classification tasks and gains marginal improvement. Furthermore, the method's generalization and effectiveness on more complex downstream tasks, such as multimodal reasoning, have not been validated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}