Bisecle: Binding and Separation in Continual Learning for Video Language Understanding
Authors: Yue Tan
Xiaoqian Hu
Hao Xue
Celso M de Melo
Flora D. Salim
Venue: NeurIPS 2025 poster ()
Keywords: continual learning
vision-language model
video understanding
multimodal large language model
Primary area: general_machine_learning

Abstract: Frontier vision-language models (VLMs) have made remarkable improvements in video understanding tasks. However, real-world videos typically exist as continuously evolving data streams (e.g., dynamic scenes captured by wearable glasses), necessitating models to continually adapt to shifting data distributions and novel scenarios. Considering the prohibitive computational costs of fine-tuning models on new tasks, usually, a small subset of parameters is updated while the bulk of the model remains frozen. This poses new challenges to existing continual learning frameworks in the context of large multimodal foundation models, i.e., catastrophic forgetting and update conflict. While the foundation models struggle with parameter-efficient continual learning, the hippocampus in the human brain has evolved highly efficient mechanisms for memory formation and consolidation. Inspired by the rapid **Bi**nding and pattern **se**paration mechanisms in the hippocampus, in this work, we propose **Bisecle** for video-language **c**ontinual **le**arning, where a multi-directional supervision module is used to capture more cross-modal relationships and a contrastive prompt learning scheme is designed to isolate task-specific knowledge to facilitate efficient memory storage. Binding and separation processes further strengthen the ability of VLMs to retain complex experiences, enabling robust and efficient continual learning in video understanding tasks. We perform a thorough evaluation of the proposed Bisecle, demonstrating its ability to mitigate forgetting and enhance cross-task generalization on several VideoQA benchmarks.

Bisecle: Binding and Separation in Continual Learning for Video Language Understanding
Yue Tan School of Computer Science University of New South Wales Sydney, Australia yue. tan@unsw. edu.au
Xiaoqian Hu
School of Computer Science University New South Wales Sydney, Australia xiaoqian.hu@studden hu@student.
xiaoqian
unsw edu.au au
Hao Xue
Celso De Melo
School of Computer Science and Engineering University of New South Wales Sydney, Australia hao. xue aenueuunwweee. edu.
DEVCOM Army Research Laboratory USA
celso miguel de www.mmemmmmlm..om
Flora D. Salim
School of Computer Science and Engineering
University of New South Wales Sydney, Australia www.enuuswwwuu..u.eee edu. au
flora. sal im@unsw.........m..m
Abstract
Frontier vision-language models (VLMs) have made remarkable improvements video understanding tasks. However, real-world videos typically exist as continu- ously evolving data streams (e.g., dynamic scenes captured by wearable glasses), necessitating models to continually adapt to shifting data distributions and novel scenarios. Considering the prohibitive computational costs of fine-tuning models on new tasks, usually, small subset of parameters is updated while the bulk the model remains frozen. This poses new challenges existing continual learning frameworks in the context of large multimodal foundation models, i.e., catastrophic forgetting and update conflict. While the foundation models strug- gle with parameter-efficies continual learning, the hippocampus in the human brain has evolved highly efficient mechanisms for memory formation and con- solidation. Inspired by the rapid Binding and pattern separation mechanisms in the hippocampus, in this work, we propose Bisecle for video-language continual learning, where multi-directional supervision module is used to capture more cross-modal relationships and a contrastive prompt learning scheme is designed to isolate task-specific knowledge to facilitate efficient memory storage. Binding and separation processes further strengthen the ability of VLMs to retain complex experiences, enabling robust and efficient continual learning in video understanding tasks. We perform thorough evaluation of the proposed Bisecle, demonstrating its ability mitigate forgetting and enhance cross-task generalization several VideoQA benchmarks.
"Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a)
Task 1: Counting
Frozen
Trainable
(b)
Improved
Designs
Hippocampus
Inspire
Continual Learning Sequential Tasks
Answer: Watching a game.
User: How Mony men toe wrestlligg?
Projection Layer
V
Assistant: Two.
Pre-trained LLM
A
Ranid Binding
Task 2: Localization
Prompt
Model
V
Memory
User: Where does the man walk?
Multi- Directional
Assistant: Field.
Projection Layer
Supervision
Pattern Saparation
Task 3: Activity
Token Embedder
Task-Aware
Prompt
Visual Encoder
Memory
people travelling?
User: Now
Video:
Question: What are the people doing?
02
Assistant: Driingg
PAPER
Contrastive
Prompt Learning
Memory
Figure 1: (a) The backbone of our continual learning framework for video understanding; (b) The improved designs proposed in this paper; (c) The motivations from a neurobiological perspective.
1 Introduction
Recent advances in large multimodal foundation models, such as vision-language models (VLMs) combining vision encoders with large language models (LLMs), have demonstrated remarkable capabilities in video understanding tasks like visual question answering (VideoQA) and video captioning [I 2 3] However, in real-world applications, these models often face continuously evolving data streams, where the domain and data distribution exhibit inevitable shifts over time [4]. To address this challenge, continual learning has emerged as critical research direction, enabling models to adapt to new tasks while retaining knowledge from previous ones [5] 6 While traditional continual learning methods typically optimize the entire model, this approach becomes impractical in the context of large multimodal foundation models due to their massive scale and the prohibitive computational costs of fine-tuning the whole framework, including both the visual encoder and LLM 7 8. A more feasible alternative is to update only small subset of pivot parameters while keeping the bulk of the model frozen IO Figure a provides a sketch map of parameter-effiiien continual learning for video understanding, where only the lightweight projection layers are learnable during the continual learning process to adapt to sequential tasks (9
Although updating only a small fraction of parameters can lead to great feasibility and training efficiency, this strategy also introduces significant challenges. Challenge 1: catastrophic forgetting. As the model is continuously trained on data from new tasks, the multimodal associations learned from previous tasks are gradually overwritten lost. Due the limited number of trainable parameters, the learnable modules struggle to maintain the knowledge for old tasks, making the forgetting problem more severe compared full-model fine-tuning scenarios. Challenge 2: update conflict. Given the diversity and heterogeneity of video understanding tasks, they usually require the model parameters induce incompatible gradient signals. When the trainable parameters are limited, these divergent adaptation needs inevitably lead to conflicts, as the scarce parameters are hard to simultaneously optimize for multiple, often incompatible objectives.
While large multimodal foundation models struggle with parameter-effiieent continual learning, our brains have evolved highly efficient mechanisms to tackle similar challenges in real-world scenarios. Responsible for memory formation and consolidation, the hippocampus in the human brain employs several unique mechanisms to handle complex, dynamic information streams [3] To efficiently encode episodic memories, the hippocampus utilizes a rapid binding mechanism to link multimodal information dynamically into cohesive memory 14 15]. As shown in the upper part of Figure [c). the visual, auditory, and contextual cues are integrated through synchronized neural activity, enabling the hippocampus to form multimodal memory traces. Meanwhile, to minimize interference between similar memories, the hippocampus employs pattern separation mechanism to generate distinct neural representations for overlapping inputs [16 17 As demonstrated in the lower part of Figure Tc), this process ensures that memories of different events are stored in different

---

neural subspaces through sparse coding in the dentate gyrus. With efficient mechanisms like rapid binding and pattern separation, the hippocampus can achieve robust memory formation and retrieval using only 0.5% of total brain neurons. Considering the powerful memorizing capability of the hippocampus, a natural question arises: Can these neurobiological principles inspire the design of parameter-effiieet continual learning frameworks for video understanding?
To answer the above question, in this paper, we propose simple yet effective method, Bisecle, inspired by rapid Binding and pattern separation mechanisms, for ladoo-lnng continual learning (see Figure 2 for the icon). To be more specific, to address Challenge 1, we design a multi-directional supervision module (as shown the upper part of Figure (T]b) to mimic the hippocampus 's rapid binding capability. Specifically, apart from the task specificc training objective that takes videos and user questions as inputs and expected answers as outputs, we introduce auxiliary reconstruction tasks, in- cluding predicting questions from videos and answers and generating videos from questions and answers, to enforce bidirectional binding between modalities. This approach improves the semantic alignment between cross- moodll task elements, thereby mitigating catastrophic forgetting. Moreover, to handle Challenge 2, we propose a contrastive prompt learning module (as shown in the lower part of Figure (T)) inspired by the pattern separation mechanism. Concretely, we introduce task-specific task type embeddings to store information for different tasks and leverage them to optimize the learnable prompts using a contrastive loss. This loss enhances the agreement between reweighted prompts obtained by attending to task-specific questions and their corresponding learnable task-specific knowledge, which alleviates the update conflict in shared parameters. In this way, Bisecle effectively isolates task-specific knowledge and reduces the potential update conflict across different tasks during the continual learning process. To sum up, the contributions of this paper three-fold:
Binding
Separation
Continual Learning
Figure 2: A "Bisecle" with two wheels (binding and sepa- ration) running on the road (a sequence of tasks).
Neurobiology-Inspieee Problem Reframing, We uniquely reframe this challenge through the lens of neurobiological mechanisms (i.e., rapid binding and pattern separation) to address catastrophic forgetting and update conflicts under strict parameter efficiency constraints.
Novel Methodology. We propose novel method, Bisecle, that integrates multi-directional supervision and contrastive prompt learning to enable robust and efficient continual learning in video understanding tasks with multi-modal LLMs.
Extensive Experiments. We conduct extensive experiments to validate the effectiveness of Bisecle, demonstrating significant improvements in mitigating forgetting and enhancing cross-task generalization across three VideoQA benchmarks.
2 Related Works
Continual Learning for Large Language Models. Continual learning is promising technique
to extend large language models (LLMs) to evolving tasks and applications [18] 19 20). Existing
methods to achieve continual learning LLMs can be broadly categorized into three classes:
continual pre-training 21
22
23
24 continual instruction tuning 25 26 27], and continual
alignment [28] 29 30 each focusing
a specific stage in the deployment of LLMs [31]. Among them, continual instruction tuning most closely aligns with the objective of extending LLMS to video understanding tasks, since it enables the model adapt new tasks while retaining previously learned knowledge through task-specific instructions
32 33, 26). To address the catastrophic
forgetting problem. Contunual-To (32 utilizes a memory buffer-based rehearsal mechanism to organize and replay the data of previous tasks. DynaInst (34) introduces dynamic instruction replay and local minima-inducing regularize to enhance the generalizability of models while preserving low computational cost. To enable parameter- parameter-efficiency continual learning with LLMs, Progressive Prompts (35) aims to freeze most parameters and only tunes the task-specific prompts for each task in the continual learning process. Jang et al. [36] propose to learn small expert adapter on top of the L.M for each task and allocate a corresponding expert for each new-coming task. Despite their efficiency in natural language processing tasks, it is non-trivial to apply them to video understanding due to the unique challenges posed by multimodal data, such as the need for robust cross-modal alignment and the dynamic nature of video content.
3

---

Continual Learning for Video Understanding. To handle different video understanding tasks in continual learning manner, recent studies explore existing continual learning techniques for video As one of the representation methods, CLOVE [37 utilizes a scene graph-based prompt mechanism for data replay in continual learning for videoQA. CI.-
understanding models 4
37
38
39
40
41
42
VQA (38] models the intricate relationships between different modalities via multimodal decoupled prompts. DAM (43) introduces a dynamic merging mechanism for the parameters of adapters, which aims to handle the domain-incremente continual learning problem. In [44], continual predictive learning approach is developed to learn a mixture world model via predictive experience replay. Tang et al. [6] propose a benchmark that systematically formulates the learning paradigm and provides a comprehensive evaluation of the existing methods. Nevertheless, the above methods do not leverage LLMs for video understanding, thereby falling short in the comprehension and interpretation of semantic information. To harness the powerful capability of LLMSs for video understanding with evolving tasks, very recently, Cai et al. [9] propose the first LLM-based video understanding framework with continual learning. The proposed method, ColPro, incorporates various prompting techniques (i.e., question constraint, knowledge acquisition, and visual temporal awareness) to deal with the catastrophic forgetting issue. Different from ColPro, our proposed method Bisecle provides simpler strategy that leverages multi-directional supervision module and a contrastive prompt learning mechanism to strengthen the memorization process and reduce update conflicts, significantly enhancing the performance continual video understanding.
3 Methodology
3.1 Problem Definition and Backbone Architecture
Problem Definition. In this paper, focus continual learning for video question answering (VideoQA), one of the most representative and essential tasks in video understanding. Consider a sequence of tasks indexed from 1 to T. Given the t-th task, we have its dataset D' {di. d% J. where n, is the number of data samples and each data sample de =< V V/ 24,44 > consists of three elements: video V," question Q and answer A² The objective is to train a video-language model f(V, Q3) = A on the datasets of the sequence of tasks, and the model can achieve strong performance on both current (e.g.. the T-th task) and previous (e.g., the 1.... T 1-th tasks) tasks. In continual learning, different tasks should exhibit certain level of diversity, reflected in differences in their data distributions
9 37 45 A brief example of task evolving is demonstrated in the left part
of Figure aa).
LLM-based Backbone Model. In this work, we consider a LLM--bseed video-language model serving as f(V', Q1) = A for video understanding tasks. Following Cai et al. [9]. we employ the LLaMA-Adapter framework [TT] with a ViT [46] visual encoder as our backbone model. As shown in the right part of Figure 1 a). the visual encoder takes the video V, its input, with a following
projection layer to transfer it into sequence of visual tokens V Vi.... VNg.. E N xD
where N is the number of frames in V,S and D) denotes the channel dimension for the extracted frame feature. Correspondingly, the question and answer tokens output by the pre-trained fixed tokenizer are denoted as Q: (4.--..,Nv::
Ni, xD and A a1. aNt
TR Ni, XD
respectively, where N1,, and Ni,, denote the number of question tokens and answer tokens. Taking V and Q as the initial input, a frozen LLM with a trainable projection layer is expected to predict the answer tokens A in an autoregressive fashion. To fine-tune the learnable parameters of model f. a cross-entropy loss is employed, which can be written as:
N 1
C log P(A) V7 (+) -
Σ
log P (a) (af,et1 V Q Af
(I)
where A = ......... a includes the first tokens of the answer sequence. Note that the modules with large number of parameters (i.e., the LLM and visual encoder) are frozen, leaving only a small number of adaptive modules as learnable. This design ensures parameter-efficien training and adaptation throughout the continual learning process with sequential tasks.

---

Multi-Directonal Supervision
Hippocampus
Mapell
pepoyo
Rapid Binding over Multiple Modalities
P(QIV,A)
P(VIQ.A)
Causal
Cross-modal
binding
binding
Pattemn Separation over Different Tasks
Sequentiel Tasks
Video:
Question:
Answer:
Contrastive Prompt Learning
What are
Watching
Baak-Awwae Prompt
Video:
Question:
Answer:
Weighted
saupppaqua REAL tatk
Where did
Field.
Prompt P
:
Ques Diestions
Video:
Question:
Answer:
Qi
Why was
Playing
Figure 3: The sketch map to explain the two modules in Bisecle, multi-directional supervision and contrastive prompt learning.
3.2 Rapid Binding- Inspired Multi-Directional Supervision
Although the aforementioned backbone model can leverage the reasoning capabiliities of LLMs while enabling efficient continual fine-tuning. like other continual learning systems, its performance is still impacted by the issue of catastrophic forgetting 9 47 48 Specifically, the model may lose previously learned knowledge, such as the ability to focus on key frames and perform question- specific semantic reasoning in older video understanding tasks, when trained to performance degradation. In the context of frozen LLM and visual encoder, this forgetting phenomenon becomes even more severe. This is because the neurons responsible for visual and language understanding are fixed, making it difficult for the model to learn knowledge for scenario- specific understanding required for new tasks. Due to the single training mode where the model is only trained to predict answers given videos and questions, the learnable adapters struggle to capture deeper task understanding, such the underlying semantic reasoning required for complex video understanding tasks, as well as the intricate cross-modal relationships between visual and textual information. This limitation hinders their ability to adapt effectively across different new tasks, particularly those requiring nuanced multimodal interactions.
tasks, leading
Neurobiological Motivation Rapid Binding. Given the similar challenge of memory and knowl- edge establishment, how does the human brain address this issue? The hippocampus, critical region for memory formation, employs rapid binding mechanism to dynamically link multimodal information, such as visual, auditory, and contextual cues, into cohesive episodic memories. This process is facilitated by synchronized neural activity, particularly theta- gettagaamma oscillations, which enable efficient encoding and retrieval of complex associations. Importantly, this binding is multi- directional: given partial cue (e.g., a visual scene), the hippocampus can reconstruct the associated context (e.g., the corresponding event or emotion), and vice versa, ensuring robust memory retention even under partial or noisy inputs. By deepening the understanding of multimodal information through these dynamic associations, the hippocampus strengthens memory traces and enhances the ability of brains to retain and recall complex experiences.
Multi-Directiin Directional Supervision. Inspired by the rapid binding mechanism of the hippocampus [15]
49 50 51, we propose a multi-directional supervision module for continual learning of L.LM-based video-language models. Considering the VideoQA task, an essential data sample, corresponding to "memory" for vidco-language models, is composed of video VAA, question Q and answer Af. The conventional learning objective is to maximize log P(A V Q?). i.c., the likelihood of generating the correct answer A given the video V and question Ø Despite its alignment with the actual downstream task, the current objective is Lingledirecttonal. That is to say, the model only learns the unidirectional connection from videos and questions answers, while ignoring the potential for bidirectional or collii-iirectonaal associations, such as the connections from V A to Q; and which from Q A to V.+. Such single-directional supervision may limit the ability the model to capture deeper causal and cross-modal relationships, thereby preventing it from forming a comprehensive understanding of the current training task. As result, with insufficient learnable parameters, the model becomes more vulnerable to suffering from catastrophic forgetting.
5

---

To address the limitations of single-direction supervision, in Bisecle, we produce calti--irectional supervision module that incorporates multiple directions of learning objectives among video V/r question Q and answer A/. Concretely, to explicitly bind the causal relationship between question Qf and corresponding video V and answer Ai. we introduce a question prediction task as auxiliary supervision. Formally, the learning objective function can be written as:
N
L log P(Q V7 A1) =- log P (q
Qi,sk)
where Q i q includes the first h tokens of the question sequence. Through the question prediction loss Lg. the model can capture more causal knowledge about the specific VideoQA task. For example, when given video of a car accident and the answer "brake failure", the model can infer the question "what caused the accident?", which enables the model to establish a deeper understanding of the causal relationship between the question and the answer.
Apart from the question prediction task, Bisecle also incorporates a video prediction task to provide extra supervision signals. Through predicting the video sequence based the question and answer, the video-language model can enhance its ability to capture cross-modal relationships as well as temporal dynamics, leading to a more robust understanding of the task content. However, it is challenging to directly predict the visual tokens as they are out of the discrete token space of language models. Inspired by [52], adopt an alternative strategy that maximizes the mutual information between the input visual tokens of frames and the output feature of LLMs. Specifically, the learning objective function of our video prediction task can be written as:
N
Cy = - log P(V,' (,,A)= =
Σ
log (Vix+1 Q Vl,sh)
k=
(3)
No.
exp (vi (i, hiie)
log
j=1 exp (vis 1i,))
where h is the output token representation of LLMs before the start of visual tokens. The video prediction loss encourages the model to predict the sequence of video frames based on the preceding frames, thereby enhancing its ability to capture temporal dependencies and improve cross-modal video understanding within the current training task in the continual learning process.
3.3 Pattern Ipparation-Inspiree Contrastive Prompt Learning
Apart from the above challenge, another critical issue is the update conflict of the learnable param- eters. To be concrete, when multiple tasks compete for updates to the same set of parameters, the model may prioritize new tasks at the expense of previously learned knowledge. Especially when the learnable parameters are scarce, this conflict becomes more pronounced, leading to significant inter- ference between tasks. While existing approaches employ task-specific prompts [9] (33 mitigate this issue, they require precise matching between test-time questions and training tasks to select the corresponding prompts. This matching process proves particularly challenging in open-world QA scenarios, where additional knowledge is often needed to identify the correct task association. More- over, the substantial divergence between task-specific prompts may lead to overfitting to individual tasks, consequently degrading model performance. Therefore, how to alleviate the update conflict issue without introducing task-specific parameters remains a significant challenge.
Neurobiological Motivation Pattern Separation. From the perspective of neurobiology, the memory encoding mechanism of our brains can provide a promising solution deal with the update conflict issue. The dentate gyrus, critical region in the hippocampus for higher-order cognitive functions, employs a pattern separation mechanism to encode distinct representations for overlapping inputs. This process is facilitated by sparse coding and lateral inhibition, which ensure that similar but non-identical memories are stored in non-overlapping neural subspaces. Notably, this mechanism allows the brain to differentiate between similar experiences while preserving the unique details of

---

each knowledge. By isolating task-specific knowledge through pattern separation, the hippocampus can efficiently retain and recall complex experiences without interference.
Contrastive Prompt Learning. To enable the model to learn task-specific knowledge while main- taining model generalization ability and parameter efficiency, we introduce task-aware learnable prompts that shared across all training tasks and then introduce contrastive prompt learning strategy optimize the prompts with task-specific restrictions. Formally, the task-aware learnable prompts are attached to multiple transformer layers in the LLM as adaptive task adapters, which can be denoted as a prompt matrix P E R(NxxL,) xDD, where Np is the number of learnable prompt tokens at each layer, Tpp is the number of transformer layers where prompts are injected, and D is the feature dimension of L.Ms. Here, Np and Tp are hyper-parameter to define the size of prompts. At the corresponding layer, the N, learnable prompt embeddings are concatenated with the question embeddings.
While shared prompts help preserve generalization ability across diverse and open-set downstream tasks during testing, they inevitably suffer from knowledge interference among different training tasks during prompt parameter updates. To mitigate this issue, we draw inspiration from the pattern separation mechanism and propose contrastive prompt learning strategy. This approach regularizes learnable prompts with task-specific embeddings, explicitly strengthening their associations with task-specific knowledge. Specifically, allocate task type embedding et IR for each task i, serving as the non-overlapping neural subspace to store the knowledge for the specific task. We dynamically reweight the learnable prompts based on specific training questions and employ contrastive loss to enforce mutual agreement between each reweighted prompt and its corresponding task type embedding. This approach enables task-aware prompts adapt both the specific tasks and questions through explicit alignment with task-related knowledge, thereby mitigating potential conflicts in these bottleneck parameters.
In formal, given a question token matrix Q and the task-aware prompt matrix P. the reweighted prompt vector can be calculated by p E RDD (q' PT) P, where of TED the averaged question representation obtained by mean mrean-polling the token embeddings of Q along the sequence dimension. In this way, O effectively encodes task-specific knowledge with the awareness of the current input context. Notably, more complicated mechanisms (e.g., attention across multi-layer outputs) can be applied to calculating P/. but empirically found that our lightweight reweighting achieves comparable performance with reduced computational overhead. Once we obtain p/, the contrastive loss across pi and e' can be calculated by:
exp (p: e'/T) ErEA( exp (p) c² /T)
Lr log
(4)
where A(t) = (0....,tt) is the set denoting the index of previous and current tasks from 0 to t, and r the temperature parameter that can adjust the tolerance for feature difference. Through this contrastive mechanism, learnable prompts are discriminately aligned with the task-specific embed- dings, mitigating the problem of inter-task interference and further alleviating update conflicts during continual learning. In this mechanism, the task type embedding matrix e' functions analogously to the dentate gyrus in human memory systems, strategically partitioning different task representations into distinct latent regions. With trade-off seper-paraeeter y, the overall training loss of Bisecle can be written as C LA - LQ + Lv + TLp.
Experiments
4.1 Experimental Setup
Datasets. We conduct experiments on three VideoQA datasets, i.e., NEXT-QA 54 DramaQA 55
and STAR (56). For NEXT-QA, we split questions into eight task types (e.g., causal why/how, temporal what/when, and descriptive where/how many). Following prior work 9 we adopt the task order <TP, CW, DC, TC, DL. DO, TN, CH> For DramaQA, we partition questions into five types and use the order with maximum forgetting, i.e., <What, Who, Where, How, Whys. For STAR, we follow its reasoning tasks <Interaction, Sequence, Prediction, Feasibility>> to evaluate situational understanding in the continual learning scenarios. More details are in Appendix A.I.I
7

---

Table 1: Comparison with state-of state-of-the-as -htt methods on NEXT-QA, DramaQA, and STAR datasets.
Bold and underline represent the best and runner-up results in each column.
NEXT-QA
DramaQA
STAR
Method
Acc (1)
Fog ())
Ace (t)
Fog (L)
Ace (1)
Fog ())
Backhone (LLaMA-Adapter)
46.58
13.83
60.99
24.39
46.89
11.54
L2P
53
48.82
12.25
62.50
20.67
48.25
10.82
DrapPrompt
50.62
11.74
65.89
17.93
49.73
10.15
LAE 58
49.38
11.47
65.82
17.35
49.15
9.87
DAM 4
53.88
9.99
67.37
15.19
50.64
8.92
ProgPrompt 35
53.95
10.69
67.92
14.95
51.07
8.75
ColPro 9
55,14
7.43
71.24
12.64
48.67
8.13
Bisecle (ours)
62.37
5.34
71.49
10.37
52.16
7.60
Baselines. We compare Bisecle to 6 representative methods for visual/video continual learning,
including I.2P [3 DualPrompt [57], LAAE (58), DAM [43,, ProgPrompt [35], and ColPro []] Among them, ColPro also employs adapter II as its backbone, making it direct counterpart to method for a fair comparison. More details for baselines can be found in Appendix A.1.2
Evaluation Metries. We evaluate the performance of baselines and Bisecle with two commonly-used metrics 33 To evaluate video understanding performance, we adopt the standard metric of average final accuracy (Avg. Acc) over T tasks for multiple-chi choice question answering. To evaluate their continual learning capability, we employ average forgetting (Avg. Fog) to quantify catastrophic forgetting, where a smaller value indicates better preservation of previously learned knowledge.
Implementation Details. We use LI LLaMA-Adapter 11 as backbone model, following (9] We use the pre-trained I.I. L.aMA-2-78 (59 as the L.M and ViT-L/14 [60 61] as the visual encoder, both of which are fixed during the continual learning process. All models are trained for five epochs with a batch size of 32 on all datasets. The number of adapter layers is set to 32, the adapter length is 10, and the weight decay is 0.14. We conduct all experiments on NVIDIA H100 GPUs. Detailed experimental settings can be found in Appendix
4.2 Experimental Results
Performance Comparison. The performance comparison on three benchmark datasets is demon- strated in Table I from which we have the following observations. Our method establishes new state-of-the-art results across all datasets, surpassing the backbone model in both accuracy (195.79% on NEXT-QA) and forgetting reduction (8.49% lower Fog on NEXT-QA). The superior performance demonstrates the powerful capability handle continual learning challenges and tackle catastrophic forgetting issues. The poor performance of the backbone (adapter) highlights its vulnerability to sequential learning, with forgetting rates up to 24.39% on DramaQA, underscoring the need for dedicated continual learning mechanisms for LLMM-based video understanding models. @ While ColPro employs more complicated prompt learning mechanism based the same backbone, our approach achieves superior performance (12.35% Acc, 42.27% Fog on DramaQA) through lightweight multi-directional supervision and contrastive prompt learning mechanisms.
Ablation Studies. To investigate the contribution of each component our method, we compare Bisecle with different variants with different removed loss terms. According to the results in Table [ we have the below findings. 1 The complete version of Bisecle utilizing all loss components demonstrates the best performance across all three datasets. This indicates synergistic effects between the different mechanisms (i.e., multi-directional supervision and contrastive prompt learning), where joint optimization maximizes model capability while minimizing catastrophic forgetting and alleviating update conflict. @ In most cases, all loss terms bring positive effects to the performance, indicating the effectiveness of the proposed mechanisms. O Among the three loss terms, Lo provides the most significant performance improvement, demonstrating that strengthening the causal relationship between questions and answers critical for mitigating catastrophic forgetting.
Data Efficiency. To verify the performance of Bisecle under data-scarce scenarios, we conduct this experiment under varying training data sizes, shown in Figure The results demonstrate three key
8

---

Table Ablation studies on the effects of different loss components across three datasets.
Components
NExT-QA
DramaQA
STAR
LQ
L.v
Cr
Acc (1)
Fog (4)
Acc (t)
Fog (4)
Acc (1)
Fog (4)
X
X
X
46.58
13.83
60.99
24.39
46,89
11.54
X
61.13
6.63
71.40
10.90
49.71
9.43
X
53.94
9.61
67.39
17.15
49.48
8.86
X
X
V
55.82
8.93
64.74
19.01
47.93
9.19
V
X
59.78
6,58
70.25
12.65
48.50
9.24
V
/
51.95
11.73
67.84
14.10
49.68
7.97
61.38
7.20
68.47
14.63
51.96
8.43
62.37
5.34
71.49
10.37
52.16
7.60
70
20
L.aMA-A LLMMA-A
Table 3: Performance on various sizes of LLMs (i.e.,
(eesle(No))
(Fog)
LLaMA-3.2-IB. LLaMA-2-7B, and LLaMA-2-13B).
60
Accuracy (%)
Method
LLaMA
#f L. Par.
Ace (1)
Fog (1)
50
IB
1,673,890
20.39
0.10
10
Backbone
7B
4,499,456
52.81
4.69
40
13B
6,034,560
57.98
4.52
1B
1,777,920
21.17
0.00
30
80
100
Bisecle
4,540,416
53.45
2.60
Training Data (%))
13B
6,085,760
59.43
3.03
Figure 4: Performance on different
sizes of training datasets.
findings: Our method Bisecle consistently outperforms the backbone (i.e., Adapter) across two metrics, indicating its strong robustness when training data is limited. @ Bisecle exhibits remarkable resistance forgetting when sufficient training data available, indicating the effectiveness of its key mechanisms, i.e., multi-directional supervision and contrastive prompt learning. 2 Bisecle can achieve superior performance even in low-resource settings, indicating data efficiency.
Robustness with Different LLMs. To validate the adaptation capability of Bisecle with different LLMs, we conduct experiments with LLaMA variants (IB/7B/13B parameters), and the results are shown in Table 3 For the sake of time, we only run experiments on three typical question types of NExT-QA dataset. We have these key observations emerge: Bisecle achieves accuracy improvements across all model sizes, demonstrating its robustness to LLM backbones. @ The forgetting rate is significantly reduced by Bisecle, especially for mid-scale models. This suggests our neurobiologiiaall-isspre
designs effectively mitigate catastrophic forgetting regardless of model scale. O Despite adding only few extra learnable parameters (104k-51k) Bisecle delivers disproportionate benefits.
Table 4: Performance with different numbers of prompt layers.
Layer
Acc (1)
Fog (L)
53.86
10.13
58.18
7.52
24
57.30
7.99
32
62.37
5.34
Sensitivity the Numbers of Layers with Prompt Injection. To study the impact of different numbers of learnable prompts, we vary the numbers of layers with prompt injection (# P. Layer) from to 32, with the results shown Table 4 The results demonstrate clear scaling trend: increasing the number of learnable prompt layers usually improves both accuracy and forgetting resistance. This indicates that task-specific parameters play dual critical roles in continual learning with
models.

---

Table Performance different contrastive prompt learning manners.
Method
Acc (1)
Fog (1)
Variant 1
58.96
6,56
Variant 2
59.96
7.58
Bisecle (ours)
62.37
5.34
Performance of Varying Contrastive Prompt Learning Manners. Table 5 of using different contrastive prompt learning manners NExT-QQ dataset. In variant 1, task type embeddings are computed as the mean of question tokens within the same class rather than learnable embeddings. In variant task type embeddings components without the prompt reweighting procedure. It can be observed that by defining the task type embedding as learnable variables, VLMs better enhance performance on various tasks.
shows the performance
directly used as the contrastive
Backbone:
Question:
Moves the heed
Why does lady baby before getting down the alide?
Bisecle (ours): Adjust his position
Figure 5: Answers by Backbone and Bisecle.
Case Study. We further evaluate the effectiveness of Bisecle by examining failure cases (See Figure more can be found in Appendix (B3) of the backbone model where our approach succeeds. While the backbone model fails to establish causal relationships between questions and answers, Bisecle successfully resolves the case, thanks to causal understanding capability acquired from multi-directional supervision.
5 Conclusion
In this paper, we present Bisecle, a neurobiologically inspired framework for langu-la continual learning that addresses critical challenges in adapting large VLMs to dynamic real-world scenarios. By emulating the binding and pattern separation mechanisms in human brain through multi-directional supervision and contrastive prompt learning, Bisecle effectively mitigates catastrophic forgetting while maintaining parameter efficiency. It not only advances the state of continual learning for multimodal systems but also provides new insights into biologically inspired AI design. Extensive evaluations on VideoQA benchmarks demonstrate the superior ability of our method to preserve past knowledge and generalize to novel tasks compared to existing approaches.
10

---

References
[1] Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, al. Video understanding with large language models: A 2023.
survey. arXiv preprint arXiv:2312.
[2] Rui Qian, Xiaoyi Dong, Pan Zhang. Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. Advances Neural Information Processing Systems, 37:119336-11966 2023.
[3] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12585-12602, 2024.
[4] Keval Doshi and Yasin Yilmaz. Rethinking video anomaly detection-a continual learning approach. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3961-3970, 2022.
[5] Andrés Villa, Juan León Alcázar, Motasem Alfarra, Kumail Alhamoud, Julio Hurtado, Fabian Caba Heilbron, Alvaro Soto, and Bernard Ghanem. Pivot: Prompting for video contin- ual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24214 24223,, 2023.
[6] Tianqi Tang, Shohreh Deldari, Hao Xue, Celso De Melo, and Flora Salim. Vilco-bench: Video language continual learning benchmark. Advances in Neural Information Processing Systems, 37:70213-70229, 2024.
[7] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 1-23776-237366 2022.
[8] Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730 19742. PMLR, 2023.
[9] Chen Cai, Zheng Wang, Jianjun Gao, Wenyang Liu, Ye Lu, Runzhong Zhang, and Kim-Hui Yap. Empowering large language model for continual video question answering with collaborative prompting. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3921-3932, 2024.
[10] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint arXiv:2308.0 08747, 2023.
[11] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
[12] Larry R Squire, Heike Schmolck, and Shauma M Stark. Impaired auditory recognition memory in amnesic patients with medial temporal lobe lesions. Learning & Memory, 8(5):252-256, 2001.
[13] Christian Schultz. and Maren Engelhardt. Anatomy of the hippocampal formation. Frontiers of neurology and neuroscience, 34:6-17, 2014.
[14] Rosanna K Olsen, Sandra N Moses, Lily Riggs, and Jennifer D Ryan. The hippocampus supports multiple cognitive processes through relational binding and comparison. Frontiers in human neuroscience, 6:146 2012.
[15] Daniel M Cer and Randall C O'Reilly. Neural mechanisms of binding in the hippocampus and neocortex: insights from computational models., 2006.

---

[16] Michael A Yassa and Craig EL Stark. Pattern separation in the hippocampus. Trends in neurosciences. 34(10):515-52 2011.
[17] Jill K Leutgeb, Stefan Leutgeb, May-Britt Moser, and Edvard I Moser. Pattern separation in the dentate gyrus and ca3 of the hippocampus. science, 315(58 14):1111)66666 2007.
[18] Xiao Wang. Yuansen Zhang, Tianze Chen, Songyang Gao. Senjie Jin, Xianjun Yang, Zhiheng Xi. Rui Zheng, Yicheng Zou, Tao Gui, et al. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.0676. 2023.
[19] Junhao Zheng, Shengjie Qiu, Chengming Shi, and Qianli Ma. Towards lifelong learning of large language models: A survey. ACM Computing Surveys, 57(8):1-35, 2025.
[20] Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models-base based continual learning: A survey, ACM Computing Surveys, 57(5):1-38, 2025.
[21] Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 3968-8975, 2020.
[22] Andrea Cossu, Antonio Carta, Lucia Passaro, Vincenzo Lomonaco, Tinne Tuytelaars, and Davide Bacciu. Continual pre-training mitigates forgetting in language and vision. Neural Networks, 179:106492, 2024.
[23] Kuan- Kung-Ying Lee, Yuanyi Zhong, and Yu-Xiong Wang. Do pre-trained models benefit equally in continual learning? In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision. pages 6485-6493, 2023.
[24] Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre-training of language models. arXiv preprint arXiv:2302.03244 2023.
[25] Jinghan He, Haiyun Guo, Ming Tang, and Jinqiao Wang. Continual instruction tuning for large 2023.
multimodal models. arXiv preprint
[26] Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Jingkuan Song, and Lianli Gao. Coin: A benchmark of continual instruction tuning for multimodel large language models. Advances in Neural Information Processing Systems, 37:57817-57840, 2024.
[27] Meng Cao, Yuyang Liu, Yingfei Liu, Tiancai Wang, Jiahua Dong, Henghui Ding, Xiangyu Zhang, Ian Reid, and Xiaodan Liang. Continual llava: Continual instruction tuning in large vision-language models. arXiv preprint 2.X1:22111.02564 2024.
[28] Han Zhang, Lin Gui, Yuanzhao Zhai, Hui Wang, Yu Lei, and Ruifeng Xu. Copf: Continual learning human preference through optimal policy fitting. CoRR, 2023.
[29] Alane Suhr and Yoav Artzi. Continual learning for instruction following from realtime feedback. Advances in Neural Information Processing Systems, 00332340-323555 2023.
[30] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Slea: Slow learner with classifier alignment for continual learning a pre-trained model. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 19148-19158, 2023.
[31] Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.01.36 2024.
[32] Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6107-6122, 2022.
[33] Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, and Yujiu Yang. Insel: A data-efficient continual learning paradigm for fine-tuning large language models with instructions. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 663-6677 2024.
12

---

[34] Jisoo Mok, Jaeyoung Do. Sungjin Lee, Tara Taghavi, Seunghak Yu, and Sungroh Yoon. Large- scale lifelong learning of in-context instructions and how tackle it. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 12573-12589, 2023.
[35] Anastasia Ruzdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. In The Eleventh International Conference on Learning Representations, 2023.
[36] Joel Jang. Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. In International Conference on Machine Learning, pages 14702-14723 PMLR, 2023.
[37] Stan Weixian Lei, Difei Gao, Jay Zhangjic Wu, Yuxuan Wang. Wei Liu, Mengmi Zhang, and Mike Zheng Shou. Symbolic replay: Scene graph as prompt for continual learning on vqa task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1250-1259, 2023.
[38] Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, and Wenwu Zhu Decouple before interact: Multi-modal prompt learning for continual visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9962, 2023.
[39] Lama Alssum, Juan Leon Alcazar, Merey Ramazanova, Chen Zhao, and Bernard Ghanem. Just a glimpse: Rethinking temporal information for video continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2474 2483, 2023.
[40] Hemang Chawla, Arnav Varma, Elahe Arani, and Bahram Zonooz. Continual learning of unsupervised monocular depth from videos. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8419-8429, 2024.
[41] Giulia Castagnolo, Concetto Spampinato, Francesco Rundo, Daniela Giordano, and Simone Palazzo. baseline on continual learning methods for video action recognition. In 2023 IEEE International Conference on Image Processing (ICIP), pages 3240-3244. IEEE, 2023.
[42] Jaeyoo Park, Minsoo Kang, and Bohyung Han. Class-incremental learning for action recognition in videos. In Proceedings of the IEÉE/CVF international conference on computer vision, pages -37077 2021.
[43] Feng Cheng, Ziyang Wang, Yi-Lin Sung, Yan- Bo Lin, Mohit Bansal, and Gedas Bertasius. Dam: Dynamic adapter merging for continual video qa learning. arXiv preprint arXiv:2403.08755 2024.
[44] Geng Chen, Wendong Zhang, Han Lu, Siyu Gao, Yunbo Wang, Mingsheng Long, and Xiaokang Yang. Continual predictive learning from videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10728-10737, 2022.
[45] Xi Zhang, Feifei Zhang, and Changsheng Xu. Vqael: A novel visual question answering continual learning setting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19102-19112, 2023.
[46] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang. An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87-110, 2022.
[47] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In International conference on machine learning, pages 3925-3934 PMLR, 2019.
[48] Wenpeng Hu, Zhou Lin, Bing Liu, Chongyang Tao, Zhengwei Tao Tao, Dongyan Zhao, Jin- wen Ma, and Rui Yan. Overcoming catastrophie forgetting for continual learning via model adaptation. In International conference learning representations, 2019.
13

---

[49] Ivilin Stoianov, Domenico Maisto, and Giovanni Pezzulo. The hippocampal formation as hierarchical generative model supporting generative replay and continual learning. Progress in Neurobiology, 217:102329, 2022.
[50] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[51] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Expe- rience replay for continual learning. Advances in neural information processing systems, 32, 2019.
[52] Dohwan Ko, Ji Lee, Woo- Young Kang, Byungseok Roh, and Hyunwoo Kim. Large language models are temporal and causal reasoners for video question answering. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4300 4316, 2023.
[53] Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139-149, 2022.
[54] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-9786, 2021.
[55] Seongho Choi, Kyoung- Woon On, Yu-Jung Heo, Ahjeong Sco, Youwon Jang, Minsu Lee, and Byoung-Tae Tak Zhang. Dramaqa: Character- video story understanding with hierarchical qa. In Proceedings of the aaai conference on artificial intelligence, volume 35, pages 1166-1174, 2021.
Kyoung-Woon
(56) Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning real-world videos. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[57] Zifeng Wang. Zizhao Zhang. Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su. Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal- continual learning. In European conference on computer vision, pages 631-648. Springer, 2022.
[58] Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang, unified continual learning framework with general parameter-efficien tuning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 14833-1149, 2023.
[59] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 230:i::307700288 2023.
[60] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, G Heigold, S Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2020.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 2021.
pages
[62] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)), 2008.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction (Section 1) accurately summarize our contribu- tions: (1) neurobiology-insping problem reframing, (2) a novel methodology for video understanding in continual learning, and (3) empirical validation multiple benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are explicitly stated in Appendix C.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Full experimental details (datasets, hyperparameters, random seeds) are in Appendix A. Code/data are provided in supplementary documents.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Anonymized code and preprocessed data are publicly available. Original datasets are cited in Section
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ips.cc/ppblli... wwtppbblcggguaiieecceeeeeeooeeee http://ddese/eee/e sondeeSubmsssiooPooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Appendix C, we list hyperparameters (learning rates, batch sizes), and describe optimization method.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Results report the mean of test accuracy and forgetting rate over all historical tasks, which is a typical measurement in continual learning paradigm. We do not report error bars due to the large computational costs required and limited computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Experiments used 2*H100 GPUs (500 GPU-hours total). Details are provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics / http://wwwpp..ccuuuuu/ii/sctccaa
Answer: [Yes]
Justification: We confirm compliance with the NeurlPS Code of Ethics. No identifiable data or harmful applications are involved.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impact in Appendix C.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets are public datasets and well cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Experimental Details
A.1 Detailed Setting Table
A.1.1 Task and Data Setting
For NEXT-QA and DramaQA datasets, we follow the continual learning setting in ColPro 9 For NEXT-QA, there eight tasks including causal questions (CW, CH), temporal questions (TC, TN, TP), descriptive questions (DC, DL), and other types of questions (DO). For DramaQA, there are five tasks including what, who, where, how, and why. For STAR, there are four tasks corresponding to interaction, sequence, prediction, and feasibility. Table 6 Table 7 and Table give several question examples for each task in NExT-QA, Drama QA, and STAR, respectively.
Table 6: Question examples of different tasks in NEXT-QA.
Task Type
Question Examples
What did the man in grey do before the plane took off?
What did the man the stage do before sitting?
Why does the man have to throw the plane first in the middle the video?
CW
Why did the man wear hat while riding the horse?
How many children are in the video? How many people cycling?
What did the lady do when they reached the bush?
What does the man in white do when he walks onto the stage?
Where is the boy projecting his photos on?
DL
Where the cat lying?
What are the different colors of the balls floating in the pool?
DO
What is the possible relationship between the girl and the boy?
How did the two women react after the woman in stripes released their hand?
TN
How did the dog react after the lady caress it?
How did the boy in stripped open the book see its contents?
CH
How did the child his hands to show his excitement the start?
Table 7: Question examples of different tasks in DramaQA.
Task Type
Question Examples
What is being cooked in the pot?
TW
What kind of thing is eat on the plate?
Who enters the house with wearing bag?
DO
Who is yelling?
Where is Deogi looking while talking?
DL
Where did Sukyung put her hand?
How does Jiya think will happen if Jiya doesn't come back with the lost money?
CH
How did Taejin
play the game when Taejin is with chairman?
Why was Deogi in the kitchen?
CW
Why did Hacyoungl light fire under the iron pot?
22

---

Table 8: Question examples of different tasks in STAR.
Task Type
Question Examples
Which object was eaten by the person? Which object was put down by the person?
Interaction
Which object did the person eat after they put down the book?
Sequence
Which object did the person open after they the table?
What will the person do next?
Prediction
Which object would the person put down next after they take the bag?
What else is the person able to do with the dish?
Feasibility
Which object the person able throw after walking through the doorway?
A.1.2 Details Baseline Methods
L.P (53] reframes continual learning a prompt-selection problem. It maintains small pool of learnable prompts, each paired with key, and keeps a large pre-trained transformer frozen. For each incoming sample, L2P computes a query from the input, retrieves the top-N matching prompts, prepends them to the input embeddings, and then updates only the prompt pool and lightweight classifier via a combined cross-entropy and key-matching loss. This design decouples task-specift from shared knowledge, requires no rehearsal buffer or task IDs at test time, and consistently outperforms sate--o-thhe-eee methods across dlass-incremental, task-agnostic benchmarks.
and
rompt 57 is rehearsal-free continual learning framework that enables frozen, pretrained vision transformer to learn a sequence of class-incremental tasks without storing any past data. does so by introducing two small, complementary sets of learnable prompts (G-Prompts for capturing task-invariant "general" instructions and E-Prompts for encoding task-specific "expert" instructions), which are attached to selected multi-head self-attention layers. A simple matching mechanism retrieves the appropriate E-Prompt at test time, and the model is trained end-to-end with a combination of classification and prompt-matchine losses.
LAE [58] introduces a unified approach for continual learning by leveraging parameter-efficient tuning methods, such as Adapter, LoRA, and Prefix, adapt pre-trained models new tasks efficiently. consists of three key components: 1) learning with calibrated adaptation speeds to align different tuning methods, accumulation of task-specific knowledge into offline fine-tuning module via momentum updates, and 3) ensemble of online and offline modules during inference to balance performance across old and new tasks. This design ensures robust continual learning performance while minimizing catastrophic forgetting and computational overhead.
DAM 4 is parameter-effiiient method designed for continual video question answering. It mitigates catastrophic forgetting by training dataset-specific adapters while dynamically merging them during inference to handle unknown domains and enable knowledge sharing. DAM has demonstrated the effectiveness across diverse video and image tasks.
ProgPrompl 35 introduces a novel continual learning approach for language models by progressively learning and concatenating soft prompts for each new task while keeping the base model frozen. This method leverages task-specific prompts to prevent catastrophic forgetting and enables forward transfer by reusing knowledge from previous prompts. Additionally, it incorporates residual MLP-basee reparameterization technique to stabilize training and improve performance, achieving significant gains over existing methods in both sho- and full-data settings.
ColPro 9 is a rehearsal-fre free continual learning framework that leverages frozen large language model and three complementary prompting strategies: task-specific question constraint prompting, knowledge acquisition prompting, and visual temporal awareness prompting. These strategies work together to encode question context, multimodal knowledge, and temporal dynamies into prompts that steer the model to learn new tasks without overwriting prior knowledge.
23

---

A.1.3 Model Architecture
The whole model involves one LLM backbone which is the LLaMA-Adapter (TT] one visual encoder which is ViT-L/14 [60] 6T] projection layer aligning the output latent representations of different modalities, and another projection layer as the final linear layer that maps the latent representations to logits over the vocabulary space.
For the LLM part, have chosen LLaMA-7B as the backbone for most experiments except for the implementation to verify the robustness with different sizes of LLMs. We adopt LLaMA-Adapter as the parameter-effiiien fine-tuning method. Concretely, instead of updating the full 7B parameters, LLaMA-Adapter freezes the pre-trained LLaMA and only learns the adaptation prompts with 1.2M parameters on top.
For the visual encoder, the input resolution is 224 x 224. For each input image, the encoder splits into 14x14 non-overlapping patches, and each patch is flattened and linearly projected to D = 1024 dimensions. The visual encoder consists of 24 transformer encoder layers, each employing multi-head self-attention (16 heads) and feedforward MLP with hidden dimension of 4096 (GELU activation), followed by Layer Normalization (Pre-Norm) for stability.
A.1.4 Hyperparameters and Training Details
We use dataset-spectne speciifc batch sizes together with AdamW across all tasks. In particular, for NEXT-QA we set the batch size to 32, for DramaQA to 4, and for STAR to 16. All experiments employ the AdamW optimizer with base learning rate of 0.09. Weight decay is 0. 14 for NExT-QA and 0. 0010 for both DramaQA and STAR. Video inputs consist of 10 frames resized to 224 x 224, and token sequences are truncated padded 128 tokens for NEXT--Q,, 280 for DramaQA, and 150 for STAR. We train each model for 5 epochs (with warm-up epochs) and fix the random seed to 0 for all tasks, Experiments are conducted on two NVIDIA H100 GPUs (94GB of memory per GPU). The GPU-hours for training is around 500 in total.
Table shows the parameters of our contrastive learning setup, governing how question type repre- sentations are learned and how negative examples are weighted in the contrastive loss.
Table 9: Training Details of Contrastive Learning.
NexIQA
DramaQA
STAR
Task Types
8
5
4
Task Type Embedding Size
[8, 4096]
[5, (0.40
[4, 4096]
Negative Temperature
1.28
1.25
1.25
Contrastive Loss Weight
0.15
0.10
0.10
B Additional Results
B.1 Visualization of Prompts across Tasks.
To learn about how the task type embeddings and weighted prompts of different tasks evolve along the continual learning process, we visualize the samples in NEXT-QA test set by t-SNE [62]. In Figure 6 the points in different colors refer to representations after fine-tuning one epoch or four epochs. It suggests that, compared with weighted prompts, task type embeddings are more distinguishable in latent space, indicating that they retain more task-specific knowledge. Also, the distribution of weighted prompts is evolving along the task sequence in a more dynamic way, i.e., the positions of different clusters are changing across epochs, because new tasks generally refresh the understanding of the model on old tasks via task-aware prompt updates. This process subsequently affects the weighted prompts through contrastive learning mechanisms.
specific task for
24

---

(a) Epoch
(b) Epoch 4
(c) Epoch 1
(d) Epoch 4
Figure 6: (a) and (b): t-SNE visualization of learnable task type embeddings of different tasks; (c)
and (d): t-SNE visualization of weighted prompts of different tasks.
B.2 Performance on Varying Tasks Orders
To investigate how the task order affects the continual learning performance, we evaluate the perfor- mance of LLaMA-Adapter and our method across different task orders. Tujggess that the task order can influence both the test accuracy and forgetting rate, due to the various difficulty degrees of the tasks and their intrinsic correlation. also shows that our method outperforms L.aMA-Adapter in both accuracy and forgetting rate in most cases, demonstrating the stability and robustness of our method to diverse continual learning task settings.
Table 10: Performance of LLaMA- and Bisecle (ours) across different task orders.
Avg. Acc (f)
Avg. Fog (1)
Task Order
LLaMA-A
Biseele (ours)
LLaMA-A
Bisecle (ours)
<CH, DL, TP, IC, DC, DO, TN. CW>
56.79
63.09
5.55
2.87
<TP TN, CH. TC, DL, DO, CW, DC>
57.68
57.98
5.89
7.16
<DO, CW. DC, CH, TP, TC, TN, DL>
55.63
57.95
7.15
8.93
<CW, DO, TN DL, TC, TP, DC, CH>
52.65
62.25
12.85
5.70
25

---

B.3 Case Study
We present more cases on NExT-QA and STAR dataset in Figure and Figure respectively, where the backbone fails to give the right answer while our method succeeds. Each subfigure corresponds specific task in the learning sequence, helping the readers better learn about the video QA tasks we are working on.
Question:
Why does the lady hold the baby before getting down the alide?
Backbone:
Choices:
A. Moves the head
Moves the head
Swing arms
Lay still
Bisecle (ours): Adjust his position
D. Adjust his position
E. Fidgeting
(a)
Question:
How many dogs video?
Backbone: One
Choices:
A. One
Four
Three
Bisecle (ours):
Six
C. Three
Five
(b)
Figure 7: More cases NEXT-QA.
Question:
Backbone: A. The bag
Which object was taken by the person?
Choices:
A. The bag
Bisecle (ours): D. The sandwich
B. The sandwich
(a)
Question:
Backbone: Tidy up
What will the person do next with
the table?
Choices:
A. Tidy up
Bisecle (ours): Sit
B. Sit
(b)
Figure More cases STAR.
26

---

B.4 Performance of Gemini and GPT
To investigate how frontier multimodal LLMs perform on the tasks in our continual learning sequence, we conduct a series of experiments based on Gemini and GPT family, i.e., Gemini 2.0 Flash, GPT- 4o-mini, and GPT-4o. It worthwhile note that these frontier multimodal LLMs do not support parameter updates, and we can only use APIs to test their performance. Hence, the problems emphasized by this paper; i.e., catastrophic forgetting and update conflict, do not obviously exist for these frontier models and traditional continual learning metrics, e.g., the forgetting rate, cannot be measured appropriately. Although it unfair for our method that experiences severe forgetting problems during the learning process to compare with these frontier models, this analysis still provides valuable insights into the capabilities of cutting-edge methods on such tasks, highlights their limitations, sheds light on open challenges, and guides subsequent improvements.
Performance Gemini and GPT on the video-language tasks. As shown Figure we report the test accuracy of each task along the task sequence in NExT-QA dataset. It can be observed that Gemini 2.0 Flash achieves the highest accuracy in almost tasks. Moreover, although our method experiences the negative impacts brought by continual learning setting, it achieves comparable performance in some tasks (CW, DC, TC, TN, CH) with GPT-4o-mini.
Gemini 2.0 Flash
GPT-4o-mini
LLaMA-Adapter
80
Bisecle (ours)
Test Accuracy
60
40
20
+TC
+D0
Task Sequence
Figure 9: The performance Gemini 2.0 flash, GPT-4o-mini, LLaMA-Adapee. and our method along the task sequence in the continual learning setting.
100
frames
frames
80
Test Accuracy
60
40
TP
CW
DC
TC
DL
CH
Task Sequence
Figure 10: The performance of Gemini 2.0 Flash on different tasks with various numbers of input video frames. The experiments are conducted on NEXT-QA dataset.
Weaknesses of Gemini and GPT. Despite the strengths of Gemini and GPT, the weaknesses of these frontier MILLMs still exist. Firstly, the zero-shot ability the models highly depends on the number of input video frames. As shown in Figure 10, the number of video frames decreases,
27

---

the performance on all tasks keeps dropping, demonstrating the dependency on sufficient or even redundant visual inputs. Secondly, the models have exhibited limited temporal reasoning ability, struggling with tasks that involve delayed causality and action/location sequencing capability. In Figure Ti can be found that our method outperforms GPT-4o and GPT-4o-mini task CW and DI. corresponding to "why do" and "where is", respectively. The former task requires the model to capture cause-and-effect relationships where the effect occurs at time lag after the cause, rather than instantaneously, while the latter task requires the model to be equipped with sequencing ability, that to acquire the temporal relationship different actions and locations. Figure gives a case study of the aforementioned issues faced by frontier models. Specifically, Figure 22 shows two failed cases related to delayed causality and Figure 12(b) shows them related to location sequencing.
100
(ours)
GPT-4o-mini
80
GPT-4o
Test Accuracy
60
40
TP
CW
DC
TC
DL
CH
Task Sequence
Figure 11: The performance of GPT-4o and GPT-4o-mini on the learning tasks NEXT-QA in our continual learning setting.
28

---

Question:
Bisecle (ours): A. hold tight
Why the baby put other hand on
after awhile of putting the fruit in his mouth?
Choices:
GPT-4o-mini:
A. hold tight
B. chocolate stuck on his fingers
B. chocolate stuck his fingers
C. to be more comfortable
D. pull fruit out
GPT-4o:
E. to make it less messy
C. to be more comfortable
Question:
Bisecle (ours): play with
Why did the girl take the piece of card at start of the video?
Choices:
A. keep toys
GPT-4o-mini:
play with
D. curious
C. lose interest in lady
D. curious
GPT-4o: curious
handing to boy
(a) Delayed causality
Question:
Bisecle (ours):
Where is this place?
A. forest
Choices:
A. forest
GPT-4o-mini: outdoors
B. outdoors
C. near staircase
D. living room
GPT-4o: B. outdoors
E. their house
Question:
Bisecle (ours): D. living room
Where are the people hanging out?
Choices:
classroom
GPT-4o-mini: kitchen
B. bus
C. kitchen
D. living room
GPT-4o: C. kitchen
E. snow mountain
(b) Action/lccation sequencing
Figure 12: The weakness of frontier VLMs in dealing with continual learning tasks can be found as limited temporal reasoning ability, which makes them struggle with tasks that involve (a) delayed causality and (b) action/location sequencing.
29

---

B.5 Experiments on Varying LLM Backbones
To examine the generalizability of Bisecle, we further conducted extra experiments on more open- source backbone LLMs, including QWEN-7B and Gemma-7B, with the experimental results shown Table [T] According to the results, we can see that Bisecle consistently leads to the optimal results compared to the baseline model (i.e., I.LaMA-Adapter), This observation illustrates the flexibility and generalizability of our proposed method.
Table 11: The results of LLaMA-A and Bisecle with various LLM backbones.
LLaMA-A
Bisccle
Backbones
Acc
Fog
Acc
Fog
-LaMA-2-7B
46.58
13.83
62.37
5,34
Qwen-7B
60.77
6.54
63.97
3.57
Gemma-7B
58.44
8.48
61.26
5.88
B.6 Latency Comparison
We compared the training time of Bisecle and two variants (including the original model), which is shown Table From the results, we can see that the multi-directional supervision mechanism can lead to longer training time, while contrastive prompt learning only brings minor computational cost. is reasonable because multi-directional supervision requires end-to-end model training on extra data, leading to increased computational overhead. Compared to the original model, the additional training time acceptable, while the performance gain is substantial, demonstrating the efficiency of Bisecle.
Table 12: The training time required by Bisecle and two variants.
LQ
Lv
Cr
Time
Acc (t)
Fog (1)
X
x
x
93 min
46.58
13.83
x
171 min
59.78
6.58
176 min
62.37
5.34
30

---

C Limitations and Broader Impacts
While Bisecle demonstrates significant potential in advancing continual learning for vision-language models through popooampus-insppiee mechanisms, our work primarily focuses on video understand- ing tasks. Future studies could extend this paradigm to other multimodal domains (e.g.. audio-visual learning or embodied Al) and explore its scalability to diverse foundation models. Additionally, the current framework assumes task boundaries are known during training. Relaxing this assumption to enable true task-free continual learning remains an open challenge.
It should be noted that method is not designed to outperform existing MILMS, but rather explore their potential in handling continually evolving data. This work provides preliminary insights for enabling LLMs personalization in dynamic environments. Moreover, given the promising results of binding and separation principles in mitigating forgetting, we believe this work opens new avenues for biologically-insee learning systems in AL.
31


Reviews:
[{"review_id": "XPgj6I5Mdd", "reviewer": "Reviewer_EdJu", "summary": "This paper presents Bisecle, a framework for continual learning in video-language understanding, inspired by neurobiological mechanisms, specifically the hippocampus's binding and pattern separation functions. The authors aim to address key challenges in continual learning: catastrophic forgetting and update conflicts, which arise when models try to adapt to new tasks without overwriting previous knowledge. Bisecle integrates a multi-directional supervision module and contrastive prompt learning to mitigate these issues while maintaining parameter efficiency. The paper demonstrates the effectiveness of Bisecle through extensive experiments on several VideoQA benchmarks, showing improvements in task retention and cross-task generalization.", "questions": "1 Error Bars and Statistical Significance: The results would be much more convincing if the paper included error bars or some statistical analysis to quantify the uncertainty in the reported performance. Would the authors consider providing these or explaining the rationale for omitting them?\n2 Clarification of Mechanisms: The contrastive prompt learning and multi-directional supervision mechanisms are innovative, but the paper could benefit from further clarity on how these mechanisms scale with different model sizes or task complexities. Could the authors provide more insights into the scalability of Bisecle for larger or more diverse tasks?\n3 Computational Efficiency: While the paper emphasizes parameter efficiency, can the authors provide more specific benchmarks on how Bisecle compares in terms of computational cost, especially when scaled up to larger models or more complex datasets? It would be helpful to understand the trade-offs between computational cost and performance improvements.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Novelty: This paper offers a fresh approach inspired by neurobiology, specifically the hippocampus's memory mechanisms, which is uncommon in video-language models. The use of multi-directional supervision and contrastive prompt learning creatively tackles catastrophic forgetting in large multimodal models, providing a unique solution to memory retention in dynamic data streams.\n\n2. Significance: The method addresses real-world challenges in video-language understanding, particularly for continuous data like dynamic scene analysis from wearable devices. It bridges cognitive neuroscience and machine learning, with strong empirical results showing practical impact on VideoQA benchmarks.\n\n3. Technical Rigor: The methodology is clear and well-justified, with robust experiments comparing Bisecle to existing approaches. The results demonstrate notable improvements in reducing forgetting and enhancing cross-task generalization.\n\nWeaknesses\nComplexity of Mechanisms: While the neurobiological inspiration is compelling, the methods, particularly the contrastive prompt learning, may be seen as complex. Although the paper claims computational efficiency, a deeper discussion of the trade-offs and overhead introduced by the task-specific prompts would be valuable.", "was_revised": true, "final_justification": "The clarification on computational efficiency and the inclusion of error bars are helpful. It’s good to see the explanation regarding the scalability of Bisecle and the additional time for multi-directional supervision. The benchmarks on performance and computational cost are also appreciated. All my questions are clarified, so I decide to maintain my positive score.", "rebuttal": "We appreciate Reviewer EdJu for the perception of our contributions and thank the reviewer for the insightful feedback. The detailed responses are provided below.\n\n**W1. Complexity of Mechanisms**\n\n**Answer:** We appreciate your point regarding the overhead of Bisecle, especially contrastive prompt learning. We would like to verify its computational efficiency from two aspects. 1) Complexity analysis. Given the contrastive module, the time complexity for each sample is $O(NDT + DT^2)$ (detailed analysis will be provided in our future version), where $N$ is the token number, $D$ is the latent dimension, and $T$ is the task number. Since $T$ is usually far less than $N$ and $D$, while the cost is linear in the sequence length, the overall computation cost of the contrastive module is relatively low and scalable in practice. 2) As shown in the table in the answer to Q1, the contrastive learning module only brings 2.9% (5 min of 171 min) extra time, which confirms its practical efficiency and minimal cost. \n\n**Q1. Error Bars and Statistical Significance**\n\n**Answer:** Thanks for your question. We now include error bars (standard deviation over 3 runs with different random seeds) and report the accuracy and forgetting rate in the following table. It can be observed that Bisecle has a relatively low standard deviation in both accuracy and forgetting rate, illustrating its stability and robustness across different runs and low sensitivity to initialization. \n\nWe did not report error bars on most experiments for the following two reasons: 1) The reported test accuracy and forgetting rate are already averaged over all previous tasks, which partially reflects the model's stability; 2) Generating error bars would require significant additional computation, which was not feasible due to limited computational resources. Considering its significance, we will report more statistical error bars in our future revision.\n\n|Method|Acc(std)|Fog(std)|\n|-|-|-|\n|Bisecle|61.59(0.68)|5.64(0.30)|\n\n\n**Q2: Clarification of Mechanisms**\n\n**Answer:** Thank you for the insightful question. In terms of model size, the results in Table 3 show that Bisecle is compatible with LLMs ranging from 1B to 13B, introducing only a small number of additional parameters and computational cost. In terms of task complexity, it is usually correlated with the length of videos and the length of texts, which are both approximately proportional to the number of tokens. As the complexity of all modules in Bisecle is typically either independent of or linear with respect to this number, the additional computational cost introduced by task complexity is aligned with that of the transformer-based LLM itself. To sum up, Bisecle enjoys comparable scalability to the LLM backbone with respect to both model sizes and task complexities.\n\n**Q3: Computational Efficiency**\n\n**Answer:** We appreciate the reviewer for the suggestion. According to your advice, we compared the training time cost of Bisecle and two variants (including the original model), which is shown in the table below. From the results, we can see that the multi-directional supervision mechanism can lead to longer training time, while contrastive prompt learning only brings minor computational cost. It is reasonable because multi-directional supervision requires end-to-end model training on extra data, leading to increased computational overhead. Compared to the original model, the additional training time is acceptable (<2x), while the performance gain is substantial (33.8%), demonstrating the efficiency of Bisecle. Due to the limitations of time and space, we did not report the running time for more baselines and datasets, and these will be provided in our future revision.\n\n|L_Q|L_V|L_P|Time|Acc|Fog|\n|-|-|-|-|-|-|\n|✗|✗|✗|93min|46.58|13.83|\n|✓|✓|✗|171min|59.78|6.58|\n|✓|✓|✓|176min|62.37|5.34|\n\n**C1: Consistent Use of Bold and Italics**\n\n**Answer:** Thank you for pointing this out. We will correct and standardize all bold/italic usages in the final version."}, {"review_id": "Ah26mPDuY1", "reviewer": "Reviewer_tR7C", "summary": "This paper investigates the problems of catastrophic forgetting and update conflicts in continual learning for large Vision-Language Models (VLMs). Inspired by neuroscience concepts, the authors propose a framework named Bisecle. This framework strengthens associations within multimodal information via a \"Multi-directional Supervision\" module and isolates task knowledge through a \"Contrastive Prompt Learning\" mechanism. Experimental results on multiple Video Question Answering (VideoQA) benchmarks demonstrate the effectiveness of the proposed method. The paper presents clear reasoning, and the experimental section is relatively detailed. The main contribution lies in the innovative proposal to formalize the concepts of \"Binding\" and \"Separating\" into two specific learning objectives and apply them to continual learning tasks for VLMs. Nevertheless, there remain some noteworthy issues regarding the depth of analysis of the core mechanisms and the rigor of the experiments, which impact the reliability of the conclusions.", "questions": "See above.", "limitations": "Yes,  the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The writing is clear and easy to follow. The motivation is clearly presented.\n- The proposed method demonstrates superiority on three videoQA datasets.\n\nWeaknesses\n- I don’t think the Neurobiology-Inspired Problem Reframing can be listed as a core contribution. In the continual learning literature, there are many neuro-inspired works [1-4]. Moreover, there is no obvious evidence that the proposed multi-directional supervision module can mimic the hippocampus’s rapid Binding and pattern separation mechanisms. I acknowledge that the proposed method is interesting, but the neuro-inspired contribution is somewhat not very convincing.\n- As another LLM-based video understanding framework with continual learning, the ColPro should be carefully discussed and compared. However, it is unclear how is the proposed method superior to ColPro in terms of methodology. The authors only claim that ‘Different from ColPro, our proposed method Bisecle provides a simpler strategy …’. I suggest the authors elaborate their comparison.\n- Ablation studies (Table 2) show that the L_Q loss alone yields a significant performance boost on NExT-QA (accuracy rising from 46.58% to 61.13%). The paper attributes this to \"strengthening causal understanding,\" an explanation which is rather vague. Considering that the experimental tasks were divided by question type (e.g., \"why\", \"how\"), the L_Q task (predicting the question itself) might unintentionally teach the model to recognize task meta-information (i.e., identify which category the current question belongs to). This learned \"task recognition\" capability could be the key to the performance improvement, rather than a more generalized \"causal understanding.\" This raises the question: would it still be effective if tasks were partitioned using other criteria, such as video domain (e.g., sports, cooking)? The lack of discussion on this point is a significant limitation of the paper.\n- All experimental results report only the mean value from a single run. The standard deviations or error bars from multiple independent runs (e.g., using different random seeds) is missing. This prevents an assessment of the result's stability and potential for randomness. \n- On the DramaQA dataset, Bisecle shows a very marginal accuracy improvement compared to ColPro (71.49% vs. 71.24%). This indicates that the advantages of the method are not equally significant across all scenarios.\n\n[1] Incorporating neuro-inspired adaptability for continual learning in artificial intelligence, Nature Machine Intelligence volume 5, pages1356–1368 (2023)\n\n[2] Brain-inspired replay for continual learning with artificial neural networks, Nature Communications volume 11, Article number: 4069 (2020) \n\n[3] Continual Learning Through Synaptic Intelligence, ICML 2017.\n\n[4] Interactive Continual Learning: Fast and Slow Thinking, CVPR 2024. \n\n[5] NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks, ICML2022", "was_revised": true, "final_justification": "I am grateful for the authors' detailed rebuttal. However, my major concern is not fully addressed, that is, the novelty of this paper, especially compared to ColPro. The authors do not demonstrate a sufficient and significant contribution beyond ColPro. Therefore, I choose to keep my score.", "rebuttal": "We are grateful to Reviewer tR7C for providing insightful feedback. The detailed responses are provided below.\n\n**W1: Concern of Neurobiological Problem Reframing**\n\n**Answer:** Thank you for the thoughtful feedback. We agree that many prior works are inspired by neuroscience, and our goal is not to position the neurobiological analogy itself as a novel contribution. Rather, our contribution lies in the structured reframing of continual video-language learning through two key hippocampal mechanisms, i.e., rapid binding and pattern separation, which directly motivate the design of our multi-directional supervision and contrastive prompt learning modules. **We take the first step to reframe the MLLM-based continual learning from the perspective of how human's hippocampus works.** While we do not claim a precise biological simulation, these modules embody the functional principles of association formation and representation disentanglement observed in the hippocampus. Similar to how neural networks abstractly draw from biological neurons (despite the significant difference between neural networks and real biological neuron systems), our approach uses these neurobiological insights as motivational analogies to guide architectural choices and learning dynamics. \n\nWe will add a discussion to further clarify how our neurobiologically-inspired method makes its unique contribution in the current stage of multimodal LLMs and also include an explanation on the contribution compared with other neuroscience-inspired works listed by the reviewer. \n\n**W2: Difference between Ours and ColPro**\n\n**Answer:** We appreciate this insightful suggestion. The difference between Bisecle (our method) and ColPro is three-fold: 1) Bisecle incorporates a multi-directional supervision module to handle catastrophic forgetting via additional supervision signals. In contrast, ColPro does not have any design to augment the supervision signals. 2) ColPro introduces a series of prompt learning techniques in their architecture design, but none of them is designed to handle the update conflict issue. Differently, we introduce a contrastive prompt learning mechanism to address the update conflict problem, using different motivations and designs compared to ColPro. 3) From the motivation perspective, the key designs in our method are inspired by neurobiological mechanisms, with carefully discussed motivations, biological mimicking, and tailored architectural designs. In contrast, ColPro is mainly based on other heuristic motivations, such as question constraint, knowledge acquisition, and visual temporal awareness. \n\nWe will include this analysis and comparison in our future revision to better demonstrate the difference between our method and the key baseline.\n\n\n**W3: Reason for Improvement by $L_Q$**\n\n**Answer:** We appreciate this valuable insight. We would like to clarify that the $L_Q$ loss is not limited to predicting the question type (e.g., \"why\" or \"how\"). Instead, it involves reconstructing the full question content, including the subject, predicate, and object, and these elements that are often implicitly or explicitly grounded in both the video and the answer. This requires the model to learn structured correspondences across modalities and to recover event semantics, such as causality or temporal dependencies, rather than relying solely on surface-level task cues. Therefore, the observed performance gain reflects enhanced multimodal reasoning, particularly in aligning causal factors across video and textual modalities.\n\n**W4-1: Single-Run Experiments**\n\n**Answer:** Thanks for your question. We now include error bars (standard deviation over 3 runs with different random seeds) and report the accuracy and forgetting rate in the following table. It can be observed that Bisecle has a relatively low standard deviation in both accuracy and forgetting rate, illustrating its stability and robustness across different runs and low sensitivity to initialization. \n\nWe did not report error bars on most experiments for the following two reasons: 1) The reported test accuracy and forgetting rate are already averaged over all previous tasks, which partially reflects the model's stability; 2) Generating error bars would require significant additional computation, which was not feasible due to limited computational resources. Considering its significance, we will report more statistical error bars in our future revision. \n\n|Method|Acc(std)|Fog(std)|\n|-|-|-|\n|Bisecle|61.59(0.68)|5.64(0.30)|\n\n**W4-2: Performance Improvement on DramaQA**\n\n**Answer:** We appreciate the review for pointing out the problem. DramaQA is a relatively mature and saturated dataset, where many recent methods already reach a high performance ceiling. In such scenarios, even a small improvement can be meaningful. Also, the forgetting rate of our method is significantly lower than the baselines on this dataset. Moreover, Bisecle demonstrates gains over strong baselines like ColPro on all datasets. This supports the generalizability and robustness of our method across different video-language reasoning tasks."}, {"review_id": "yTuPyyFcFw", "reviewer": "Reviewer_9h35", "summary": "This paper is addressing a growing need to extend VLMs to continual video-language tasks.\n\nIts neurobiological framing (binding/separation) is refreshing and clearly influences the algorithm design. The methodology is sound and clearly described, and the experiments are thorough. \n\n\nHowever, the novelty is incremental in some respects (prompt tuning + auxiliary objectives), though the grounding in biological metaphors adds conceptual depth.", "questions": "1. Can we add more points to Table 4 to see the relationship between #layers and acc?\n\n2. What is the latency, compared to the original model? Need a table comparison\n\n3. Can this be applied to MoE LLMs?\n\n4. Could you provide experiments with QWEN, which will make the proposed method more general?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Very easy to follow\n2. Outperforms baselines on NExT-QA, DramaQA, and STAR datasets, improving accuracy by +15.79% and reducing forgetting by 8.49% on NExT-QA\n3. The biological analogy with hippocampal mechanisms is more than illustrative—it clearly informs the system design and is empirically validated.\n\nWeaknesses\n\n1. The contrastive learning formulation (Equation 4) assumes well-separated task embeddings, but its sensitivity or failure modes are not discussed.\n\n2. Bisecle still requires prompt injection at up to 32 layers which is super high.", "was_revised": true, "final_justification": "Based on my suggestions, the authors conducted more experiments, which seems very promising.", "rebuttal": "We appreciate Reviewer 9h35 for the positive review and constructive comments. We provide our responses as follows.\n\n**W1: Sensitivity or Failure Modes of Contrastive Learning**\n\n**Answer:** Thanks for your valuable question. Since the task embeddings are fully learnable, the inner mechanism of contrastive loss (i.e., closing the positive pairs and separating the negative pairs) can regularize each task embedding to be close to the corresponding task-related query while pushing it away from unrelated task queries, thus promoting better discrimination between different task embeddings. Fig.6 in the supplemental material can prove such separation. This mechanism can mitigate potential failure modes of this loss, such as collapsed or entangled task embeddings. Moreover, Eq. (4) is an additional regularization loss in Bisecle and will not directly control the final prediction of the model. In this case, even if the contrastive loss degenerates into failure modes, the worst-case scenario is that the contrastive mechanism becomes ineffective, without causing a significant negative impact on the final model performance (e.g., worse than the original model).\n\n**W2: Requirements of Prompt Injection Layers**\n\n**Answer:** We appreciate the reviewer for the comment. As discussed in our sensitivity experiments (Line 341-349), there is a trade-off between injected layers and final performance. As shown in Table 4 and Answer to Q1, 12&16 injected layers can also result in competitive performance that exceeds all baselines. Meanwhile, even with 32 injected layers, the number of overall learnable parameters is still far less than the LLM backbone (see Table 3), which demonstrates the parameter efficiency of Bisecle from a marco perspective. We hope our explanation can address your concern.\n\n**Q1: More Points to Table 4**\n\n**Answer:** We thank the reviewer for the valuable comment. As a response, we add more points (12, 20, and 28 layers) to Table 4, and the added results are shown as follows. As we can see in the results, a general trend is that more injected layers can lead to higher accuracy and lower forgetting rates. When the layer number is larger than or equal to 12, the performance becomes competitive, which demonstrates the parameter efficiency of Bisecle.\n\n| Layer | Acc   | Fog   |\n|-------|-------|-------|\n| 8     | 53.86 | 10.13 |\n| 12    | 58.94 | 7.84  |\n| 16    | 58.18 | 7.52  |\n| 20    | 59.23 | 6.96  |\n| 24    | 57.30 | 7.99  |\n| 28    | 58.64 | 6.27  |\n| 32    | 62.37 | 5.34  |\n\n\n\n**Q2: Latency Comparsion**\n\n**Answer:** We appreciate the reviewer for the suggestion. According to your advice, we compared the training time of Bisecle and two variants (including the original model), which is shown in the table below. From the results, we can see that the multi-directional supervision mechanism can lead to longer training time, while contrastive prompt learning only brings minor computational cost. It is reasonable because multi-directional supervision requires end-to-end model training on extra data, leading to increased computational overhead. Compared to the original model, the additional training time is acceptable, while the performance gain is substantial, demonstrating the efficiency of Bisecle.\n\n|L_Q|L_V|L_P|Time|Acc|Fog|\n|-|-|-|-|-|-|\n|✗|✗|✗|93min|46.58|13.83|\n|✓|✓|✗|171min|59.78|6.58|\n|✓|✓|✓|176min|62.37|5.34|\n\n**Q3: Application to MoE LLMs**\n\n**Answer:** Thanks for your inspiring question. While Bisecle can be a plug-and-play module for normal LLMs, it may require several special designs to be adapted to MoE LLMs. The difficulties can lie in how to insert adapter modules into sparsely activated experts without breaking the gating mechanism, and how to ensure efficiency and stability under expert parallelism. Fortunately, several novel approaches, such as MoRA and DynAdapter, have shed light on the application of adapter-like techniques in sparse and expert-based architectures. In this case, we believe that Bisecle can also work with MoE LLMs with minor modification, which can be one of our future directions.\n\n**Q4: Experiments on more LLM Backbones**\n\n**Answer:** Thanks for your comments. To examine the generalizability of Bisecle, we further conducted extra experiments on more open-source backbone LLMs, including QWEN-7B and Gemma-7B, with the experimental results shown below. According to the results, we can see that Bisecle consistently leads to the optimal results compared to the baseline model (i.e., LLaMA-Adapter). This observation illustrates the flexibility and generalizability of our proposed method.\n\n|Method|Acc@LLaMA-2-7B|Fog@LLaMA-2-7B|Acc@Qwen-7B|Fog@Qwen-7B|Acc@Gemma-7B|Fog@Gemma-7B|\n|-|-|-|-|-|-|-|\n|LLaMA-A|46.58|13.83|60.77|6.54|58.44|8.48|\n|Bisecle|62.37|5.34|63.97|3.57|61.26|5.88|"}, {"review_id": "WHRx2x5sCN", "reviewer": "Reviewer_a9TX", "summary": "Inspired by the neurological principles of how the hippocampus processes complex and dynamic information streams, the authors propose a video continual learning framework for mLLMs. The authors present a cross-modal auxiliary reconstruction task for fast binding across modalities and a task-aware comparative prompt learning to avoid update conflicts between tasks. Experiments demonstrate that the proposed method can effectively improve the generalization and mitigate catastrophic forgetting.", "questions": "* How is task type embedding initialized?", "limitations": "Limitations are well discussed in the supplementary material.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n* The paper is well-motivated, and the components design can be reasonably explained by the principle of the hippocampus.\n* The writing is clear, and the paper is well-organized.\n* Experiments are comprehensive, validating the effectiveness of the proposed components.\n\n**Weaknesses:**\n* Does the training of the Multi-Directional Supervision module need three forward passes? It would be great to show the actual training time or computational overhead compared to the baseline.\n* How to compute the contrastive loss when learning the first task? Does this mean that all task settings are pre-defined? It seems to be not reasonable in real-world applications.", "was_revised": true, "final_justification": "I have carefully read the author's rebuttal and other reviewers' comments. I appreciate the authors for providing more implementation details and computational overhead, which makes the article clearer. Overall, my concerns are basically addressed, and I think this paper can provide some inspiration for related areas.", "rebuttal": "We appreciate Reviewer a9TX for the valuable feedback and acknowledge our technical contributions and the effectiveness of the proposed method. We address the concerns raised by the reviewer as follows.\n\n**W1:Computational Overhead**\n\n**Answer:** Thanks for your comments. It is true that the multi-directional supervision module requires three forward passes, resulting in a slightly higher training time costs and computational requirements. We provide an additional quantitative experiment on NExT-QA dataset to analyze the time cost in the Table below. It can be observed that compared to the backward propagation, the forward pass incurs much lower computational cost, making it well-suited for multiple executions to enhance model performance. \n\n|L_Q|L_V|L_P|Time|Acc|Fog|\n|-|-|-|-|-|-|\n|✗|✗|✗|93min|46.58|13.83|\n|✓|✓|✗|171min|59.78|6.58|\n|✓|✓|✓|176min|62.37|5.34|\n\n**W2:Calculation of Contrastive loss**\n\n**Answer:** We appreciate your comments. The contrastive loss is mainly discussed in the last paragraph of Sec. 3.3. As shown in Eq. (4), for a reweighted prompt vector computed based on the question token matrix and task-aware prompt matrix, its positive sample is the current task type embedding, while its negative samples include both the previous and current task type embeddings. For the first task, the contrastive loss value equals zero because there are no actual negative samples. By defining the contrastive loss in this way, 1) the loss computation is compatible with the first task, 2) it supports dynamically evolving task streams without the need to pre-define the task sequence. \n\nWe are sorry for not making it clear in our current version. We will add related discussion to Sec. 3.3. \n\n\n**Q1:How is task type embedding initialized**\n\n**Answer:** Thanks for your comments. Since the task type embeddings themselves are learnable parameters (line 243), we directly use random initialization to initialize them, similar to other parameters in neural networks. We added the implementation details about how the task type embeddings are initialized in Sec. 3.3 in our future revision."}]

Author rebuttal:

Meta-review: This paper presents Bisecle, a neuro‑inspired continual learning framework for video‑language understanding that combines multi‑directional supervision and contrastive prompt learning to address catastrophic forgetting and update conflicts, achieving improved task retention and cross‑task generalization on multiple VideoQA benchmarks with parameter‑efficient adaptation.

Strengths:
- Clear motivation and presentation: Multiple reviewers highlight the clarity, organization, and ease of following the paper’s narrative and problem framing.
- Neurobiology‑inspired design informing the method: The work offers a fresh approach inspired by hippocampal memory mechanisms, which is uncommon in video‑language models. The integration of multi‑directional supervision and contrastive prompt learning creatively addresses catastrophic forgetting in large multimodal models, providing a distinctive solution to memory retention in dynamic data streams.
- Methodological soundness with targeted objectives: The combination of multi‑directional supervision (knowledge preservation) and contrastive prompt learning (reducing update conflicts) is well‑justified and effective under parameter‑efficient adaptation.
- Strong empirical results across datasets: Consistent gains over baselines on NExT‑QA, DramaQA, and STAR, including notable improvements in accuracy and reductions in forgetting (e.g., on NExT‑QA: +15.79% accuracy, −8.49% forgetting).

Weaknesses:
- The key unresolved concern is that the contribution beyond ColPro is not sufficiently demonstrated; one reviewer maintains a novelty objection despite the rebuttal.
-  The proposed multi‑directional supervision still results in nearly double the training cost, which could be further optimized in future work.

Final Recommendation:

The authors provided additional experiments (e.g., error bars, latency comparison, Qwen/Gemma backbones) and clarifications in rebuttal that addressed the majority of reviewer questions. Three of four reviewers now lean positive, citing strong empirical results and methodological soundness. I recommend acceptance, but note that the authors should include in the final version the additional experimental results they committed to (e.g., running time for more baselines and datasets).
Decision: Accept (poster)