Title: Bisecle: Binding and Separation in Continual Learning for Video Language Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: continual learning

vision-language model

video understanding

multimodal large language model
Abstract: Frontier vision-language models (VLMs) have made remarkable improvements in video understanding tasks. However, real-world videos typically exist as continuously evolving data streams (e.g., dynamic scenes captured by wearable glasses), necessitating models to continually adapt to shifting data distributions and novel scenarios. Considering the prohibitive computational costs of fine-tuning models on new tasks, usually, a small subset of parameters is updated while the bulk of the model remains frozen. This poses new challenges to existing continual learning frameworks in the context of large multimodal foundation models, i.e., catastrophic forgetting and update conflict. While the foundation models struggle with parameter-efficient continual learning, the hippocampus in the human brain has evolved highly efficient mechanisms for memory formation and consolidation. Inspired by the rapid **Bi**nding and pattern **se**paration mechanisms in the hippocampus, in this work, we propose **Bisecle** for video-language **c**ontinual **le**arning, where a multi-directional supervision module is used to capture more cross-modal relationships and a contrastive prompt learning scheme is designed to isolate task-specific knowledge to facilitate efficient memory storage. Binding and separation processes further strengthen the ability of VLMs to retain complex experiences, enabling robust and efficient continual learning in video understanding tasks. We perform a thorough evaluation of the proposed Bisecle, demonstrating its ability to mitigate forgetting and enhance cross-task generalization on several VideoQA benchmarks.


Peer reviews:
comments: SUMMARY:
Inspired by the neurological principles of how the hippocampus processes complex and dynamic information streams, the authors propose a video continual learning framework for mLLMs. The authors present a cross-modal auxiliary reconstruction task for fast binding across modalities and a task-aware comparative prompt learning to avoid update conflicts between tasks. Experiments demonstrate that the proposed method can effectively improve the generalization and mitigate catastrophic forgetting.

STRENGTHS AND WEAKNESSES:
**Strengths:**
* The paper is well-motivated, and the components design can be reasonably explained by the principle of the hippocampus.
* The writing is clear, and the paper is well-organized.
* Experiments are comprehensive, validating the effectiveness of the proposed components.

**Weaknesses:**
* Does the training of the Multi-Directional Supervision module need three forward passes? It would be great to show the actual training time or computational overhead compared to the baseline.
* How to compute the contrastive loss when learning the first task? Does this mean that all task settings are pre-defined? It seems to be not reasonable in real-world applications.

QUESTIONS:
* How is task type embedding initialized?

LIMITATIONS:
Limitations are well discussed in the supplementary material.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

comments: SUMMARY:
This paper is addressing a growing need to extend VLMs to continual video-language tasks.

Its neurobiological framing (binding/separation) is refreshing and clearly influences the algorithm design. The methodology is sound and clearly described, and the experiments are thorough. 


However, the novelty is incremental in some respects (prompt tuning + auxiliary objectives), though the grounding in biological metaphors adds conceptual depth.

STRENGTHS AND WEAKNESSES:
Strengths

1. Very easy to follow
2. Outperforms baselines on NExT-QA, DramaQA, and STAR datasets, improving accuracy by +15.79% and reducing forgetting by 8.49% on NExT-QA
3. The biological analogy with hippocampal mechanisms is more than illustrative—it clearly informs the system design and is empirically validated.

Weaknesses

1. The contrastive learning formulation (Equation 4) assumes well-separated task embeddings, but its sensitivity or failure modes are not discussed.

2. Bisecle still requires prompt injection at up to 32 layers which is super high.

QUESTIONS:
1. Can we add more points to Table 4 to see the relationship between #layers and acc?

2. What is the latency, compared to the original model? Need a table comparison

3. Can this be applied to MoE LLMs?

4. Could you provide experiments with QWEN, which will make the proposed method more general?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the problems of catastrophic forgetting and update conflicts in continual learning for large Vision-Language Models (VLMs). Inspired by neuroscience concepts, the authors propose a framework named Bisecle. This framework strengthens associations within multimodal information via a \"Multi-directional Supervision\" module and isolates task knowledge through a \"Contrastive Prompt Learning\" mechanism. Experimental results on multiple Video Question Answering (VideoQA) benchmarks demonstrate the effectiveness of the proposed method. The paper presents clear reasoning, and the experimental section is relatively detailed. The main contribution lies in the innovative proposal to formalize the concepts of \"Binding\" and \"Separating\" into two specific learning objectives and apply them to continual learning tasks for VLMs. Nevertheless, there remain some noteworthy issues regarding the depth of analysis of the core mechanisms and the rigor of the experiments, which impact the reliability of the conclusions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The writing is clear and easy to follow. The motivation is clearly presented.\n- The proposed method demonstrates superiority on three videoQA datasets.\n\nWeaknesses\n- I don’t think the Neurobiology-Inspired Problem Reframing can be listed as a core contribution. In the continual learning literature, there are many neuro-inspired works [1-4]. Moreover, there is no obvious evidence that the proposed multi-directional supervision module can mimic the hippocampus’s rapid Binding and pattern separation mechanisms. I acknowledge that the proposed method is interesting, but the neuro-inspired contribution is somewhat not very convincing.\n- As another LLM-based video understanding framework with continual learning, the ColPro should be carefully discussed and compared. However, it is unclear how is the proposed method superior to ColPro in terms of methodology. The authors only claim that ‘Different from ColPro, our proposed method Bisecle provides a simpler strategy …’. I suggest the authors elaborate their comparison.\n- Ablation studies (Table 2) show that the L_Q loss alone yields a significant performance boost on NExT-QA (accuracy rising from 46.58% to 61.13%). The paper attributes this to \"strengthening causal understanding,\" an explanation which is rather vague. Considering that the experimental tasks were divided by question type (e.g., \"why\", \"how\"), the L_Q task (predicting the question itself) might unintentionally teach the model to recognize task meta-information (i.e., identify which category the current question belongs to). This learned \"task recognition\" capability could be the key to the performance improvement, rather than a more generalized \"causal understanding.\" This raises the question: would it still be effective if tasks were partitioned using other criteria, such as video domain (e.g., sports, cooking)? The lack of discussion on this point is a significant limitation of the paper.\n- All experimental results report only the mean value from a single run. The standard deviations or error bars from multiple independent runs (e.g., using different random seeds) is missing. This prevents an assessment of the result's stability and potential for randomness. \n- On the DramaQA dataset, Bisecle shows a very marginal accuracy improvement compared to ColPro (71.49% vs. 71.24%). This indicates that the advantages of the method are not equally significant across all scenarios.\n\n[1] Incorporating neuro-inspired adaptability for continual learning in artificial intelligence, Nature Machine Intelligence volume 5, pages1356–1368 (2023)\n\n[2] Brain-inspired replay for continual learning with artificial neural networks, Nature Communications volume 11, Article number: 4069 (2020) \n\n[3] Continual Learning Through Synaptic Intelligence, ICML 2017.\n\n[4] Interactive Continual Learning: Fast and Slow Thinking, CVPR 2024. \n\n[5] NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks, ICML2022\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes,  the authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Bisecle, a framework for continual learning in video-language understanding, inspired by neurobiological mechanisms, specifically the hippocampus's binding and pattern separation functions. The authors aim to address key challenges in continual learning: catastrophic forgetting and update conflicts, which arise when models try to adapt to new tasks without overwriting previous knowledge. Bisecle integrates a multi-directional supervision module and contrastive prompt learning to mitigate these issues while maintaining parameter efficiency. The paper demonstrates the effectiveness of Bisecle through extensive experiments on several VideoQA benchmarks, showing improvements in task retention and cross-task generalization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Novelty: This paper offers a fresh approach inspired by neurobiology, specifically the hippocampus's memory mechanisms, which is uncommon in video-language models. The use of multi-directional supervision and contrastive prompt learning creatively tackles catastrophic forgetting in large multimodal models, providing a unique solution to memory retention in dynamic data streams.\n\n2. Significance: The method addresses real-world challenges in video-language understanding, particularly for continuous data like dynamic scene analysis from wearable devices. It bridges cognitive neuroscience and machine learning, with strong empirical results showing practical impact on VideoQA benchmarks.\n\n3. Technical Rigor: The methodology is clear and well-justified, with robust experiments comparing Bisecle to existing approaches. The results demonstrate notable improvements in reducing forgetting and enhancing cross-task generalization.\n\nWeaknesses\nComplexity of Mechanisms: While the neurobiological inspiration is compelling, the methods, particularly the contrastive prompt learning, may be seen as complex. Although the paper claims computational efficiency, a deeper discussion of the trade-offs and overhead introduced by the task-specific prompts would be valuable.\n\nQUESTIONS:\n1 Error Bars and Statistical Significance: The results would be much more convincing if the paper included error bars or some statistical analysis to quantify the uncertainty in the reported performance. Would the authors consider providing these or explaining the rationale for omitting them?\n2 Clarification of Mechanisms: The contrastive prompt learning and multi-directional supervision mechanisms are innovative, but the paper could benefit from further clarity on how these mechanisms scale with different model sizes or task complexities. Could the authors provide more insights into the scalability of Bisecle for larger or more diverse tasks?\n3 Computational Efficiency: While the paper emphasizes parameter efficiency, can the authors provide more specific benchmarks on how Bisecle compares in terms of computational cost, especially when scaled up to larger models or more complex datasets? It would be helpful to understand the trade-offs between computational cost and performance improvements.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}