COLT: Enhancing Video Large Language Models with Continual Tool Usage
Authors: Yuyang Liu
Meng Cao
Xinyuan Shi
Xiaodan Liang
Venue: Accepted by TMLR ()

Abstract: The success of Large Language Models (LLMs) has significantly propelled the research of video understanding. To harvest the benefits of well-trained expert models (i.e., tool), video LLMs prioritize the exploration of tool usage capabilities. Existing methods either prompt closed-source LLMs or employ the instruction tuning paradigm for tool-use finetuning. These methods, however, assume an established repository of fixed tools and struggle to generalize to real-world environments where tool data is perpetually evolving and streaming in. To this end, we propose to enhance open-source video LLMs with COntinuaL Tool usage (termed COLT), which automatically acquires tool-use ability in a successive tool stream without suffering "catastrophic forgetting" of the past learned tools. Specifically, our COLT incorporates a learnable tool codebook as a tool-specific memory system. Then, relevant tools are dynamically selected based on the similarity between user instructions and tool features within the codebook. To unleash the tool usage potential of video LLMs, we collect a video-centric tool-use instruction tuning dataset VideoToolBench. Extensive experiments on both previous video LLM benchmarks and the tool-use-specific VideoToolBench dataset demonstrate the state-of-the-art performance of our proposed COLT.

Published in Transactions on Machine Learning Research (01/2026)
COLT: Enhancing Video Large Language Models with Continual Tool Usage
Yuyang Liu"
yuyang wnpong.Hu@@mzzati.ccoo
Department of Computer Vision University of Mohamed bin Zayed University of Artificial Intelligence
Mong Cao"
mengrraphuu@mmail.cco
Department Computer Vision
University of Mohamed bin Zayed University of Artificial Intelligence
Xinyuan Shi
sirouurr.shi@mmbbuue
Department of Computer Vision University of Mohamed bin Zayed University of Artificial Intelligence
Xiaodan Liang
rinodan, ionooann.iang@mmaaudde ac..o
Department of Computer Vision
Universily of Mohamed bin Zaged University of Artificial Intelligence
Reviewed on OpenReview: https:
// openreview. net/ forum? id=
Abstract
The success of Large Language Models (LLMs) has significantly propelled the research of video understanding. To leverage the strengths of specialist models (í.e., tool) for specific video tasks, recent video LLMs have focused on integrating specialist models as tool usage capabilities into their architectures. Existing methods either prompt closed-source LLMs or employ the instruction tuning paradigm for tool-use finetuning. These methods, how- ever, assume an established repository of fized tools and struggle generalize to real-world environments where tool data is perpetually evolving and streaming in. To this end, propose to enhance open-source video LLMs with COntinuaL Tool usage (termed COLT), which automatically acquires tool-use ability in a successive tool stream without suffering "catastrophic forgetting" of the past learned tools. Specifically, our COLT incorporates a learnable tool codebook as a tool-specific memory system. Then, relevant tools are dynami- cally selected based on the similarity between user instructions and tool features within the codebook. To unleash the tool usage potential of video LLMs, we collect video-centric tool- instruction tuning dataset Vool Extensive experiments on both previous vidco LLM benchmarks and the tool-use-specific VideoTool test split demonstrate the state-of- the-art performance of our proposed COLT.
1 Introduction
Large Language Models (LLMs), e.g., GPT (Achiam et al. 2023 Brown el al. 2020), PaLM (Anil el al. 2023), and LLaMA TTovroo al.) 2023a b). have exhibited remarkable success in understanding user instructions, aligning with human intents, and generating trustworthy responsos. Trained largo-scale cor- pora and equipped with billions of parameters, these models demonstrate impressive generalization abilities across wide range of natural language tasks such reasoning, summarization, and question answering. They have fundamentally reshaped the paradigm of natural language understanding by enabling unified instruction- follooiiinn interfaces that support various downstream applications, from conversational agents
* "Equal contribution.

---

Published in Transactions on Machine Learning Research (01/2026)
Instruction data video recognition tool-use g
Inttrctioo-flllwwnng data video object segmentation tookuse
(b)
Please Kentffy the action the video.
Please segment the objects within the video.
As language model don't have ability segment objects videos.
To objects within video, Object Dssection algorithmd
The viceo shows person using a computer mouse and keyboard on tiled floor. The person mowes the mouse around and dicts on the kryboard.
Juse the video objecct segmentation to fulfil request reuults are returned.
action shown video is the person picking up mouse.
will employ the act on recogrition model, Acconding the reragrithon resutts, the the video scen moving a computer mouse closer to computer keyboard.
8 User Instruction
Video LLavA
VideoChat2
COLT (Ours)
Figure 1: Our proposed COLT contimually learns to invoke tools from tool-stream data without catastrophic forgetting. Benefiting from tool usage, COLT (a) is adept at dynamic content understanding and (b) supports flexible generation compared to existing methods Lin et al. 2023a Li et al. (002cc). The incorrect parts of responses are marked in red.
to coding assistants. Recently, the research focus has been gradually shifting from pure text-based LLMs to multi-modal LLMs such as BLIP-2 (Li et al. 2023a), Flamingo (Alayrac et al. 2022), and MiniGPT- 4 (Zhu et al., 2023b),, which augment language models with the capability to perceive and reason over multiple modalities such as vision, audio, and video. By integrating visual encoders with pretrained LLMs, these models can interpret images or videos and produce context-aware, grounded responses, thus bridging perception and language understanding. Such a shift extends the role of LLMs from passive text genera- tors to more general perceptual agents, which has aroused significant. research interest within the academic community and beyond.
Despite the progress in image-based LLMs, advancing LLMs capacity to comprehend video data remains a demanding pursuit. Recent representative efforts such as Video-LLaMA (Zhang et al. 2023a and VideoChat (Li et al. 2023b) illustrate the potential video LLMs, yet the field still in its early stage.
The overwhelming majority of video LLM methods follow the instruction-tuning paradigm. Building upon
open-source LLMs such as Vicuna (CChiang et al., 2023) and LLaMA [Tovron et al., 2023a b). they entail end-to-end training on instruction-tuning datasets generated by GPT models (Achiam et al. 2023 Brown et al. 2020). These methods, however, demonstrrat limited ability to comprehend video actions and present responses with hallucinations (e.g., non-existent "keyboard click" actions). Besides, most of them lack the capability of any-to-ary any generation, e.g., video object segmentation in Figure (b).
To alleviate this, agent-based methods such as Assist- GPT (Gao al.) 2023), ToolL.LM (Qin al. DoraemonGPT Yang et al. 2024b), VideoAgent (Fan et al. 2024), and VideoAgeet-22 (Wang al. 2024b
prompr geteraation
2023),
Mechod 41 COLT (Ourg
Instruccions
close- source LLMs
0.8
evenaee accurary
are int rodueed to advocate the tool-use capabilitain of LLMs. More recently, VITAL (Zhang et al.
(h)
ieperviied
finctime
0.7
Malhod #2 Seguential Training
2025 ex-
066
fixed tuning dataser
open-source LLMs
plores reinforcing tool-augmented video reasoning via multimodal reinforcement learning on long videos. As shown in Figure 2 (a), they mostly bootstrap closed-
0.5
courinsed
finenine
0.4
WJ
tuning dataset
open-source L.Ms
number learned tools
source LLMs (Achiam et al.) 2023 Brown et al.) 2020)) to decompose the user instructions into more manage- able sub-tasks and incorporate external tools to as- sist. These training-fre free methods require delicately designed prompls and lengthy context windows for in- structions (Liu et al. 2023c). Instead, recent attempts like VA-plus (Liu al. 2023c) and TooILLM (Qin) et al.) 2023) explore developing tool-specific instruc- tion tuning data and elicit. the tool-use capabilities of
Figure 2: (a) Agent-based methods bootstrap closed-source LLMs via delicately designed prompts; (b) Instruction tuning with fixed tool-use dataset; (c) Instruction tuning with stream tool-use dalaset (Ours); (d) Average tool calling accuracy on VidcoTool us. learned tools. Sequential train- ing denotes training on a sequence of lasks indepen- dently.
2

---

Published in Transactions on Machine Learning Research (01/2026)
open-source LLMs (c.f. Figure 2 (b)). However, these methods typically assume a pre-set repository con- taining fixed tools, which is difficult. to generalize into the real-world scenario where tool data arrives in a never-ending stream. In such cases, the conventional sequential training, i.e., training on new data with the pre-trained weights on the previous tool data as initialization, lcads to significant catastrophic forgetting of prior tool-usage knowledge. For example in Figure 2 (d); the tool-call accuracy of sequential training rapidly
decreases as the number of tools learned increases.
In this paper, we propose to enhance open-source video LLMs with COntinuaL Tool usage (dubbed COLT). By "continual", mean that video LLMs possess the inherent capacity to learn and automatically activate tool functionality in successive tool-stream format. To achieve this, two critical issues arisc: 1) how to build a tool-specific memory to mitigate catastrophic forgetting of past knowledge? One intuitive idea to store a few past tool data and replay them with the new tool data for continual training. However, the requisite memory expands proportionally with the increase in the number of tools, and previous data may be unavailable due to privacy constraints. Instead, we set up tool codebook consisting of learnable prompis to store wool-specifie information in more concise manner; 2) how lo exploil relevant. tools based imput user instructions? For tool activation, we compute the cosine similarity between the embedded user instructions and cach prompt within the tool codebook. The highest-response prompts are then selected to "instruct" the model to invoke appropriate tools.
Although several video LLM benchmarks (Maaz et al. 2023 Li et al., 2023c have been proposed, the community still lacks a benchmark for video-centric tool-use instruction tuning. To this end, we collect VideoTool to facilitate tool-use capabilities for open- video LLMs. Specifically, we collect a set of video specialist models and prompt GPT (Achiam et al. 2023) to generate diverse instructions for tool calls. Then the tool execution results are used to form the instruction-following dataset. Under continual learning setup, COLT can incrementally incorporate newly added tools from a expanding repository, while mitigating forgetting of previously learned ones.
In summary, our contributions are threefold:
We proposed COLT, video LLM with continual tool usage. By maintaining a tool codebook, COLT incrementally learns new tools without catastrophic forgetting.
A tool-using instruction-follovii dataset VideoTool is introduced to unlock the potential for tool usage within video LLMs.
Experiments on both existing video LLM benchmarks and the Video'Tool test split have manifested the state-of-the-art performance of COLT. For example, on MSRVTT-QA (Chen & Dolan 2011), our COLT boosts the previous SOTA method (Li et al. 2023c by 8.2% on the accuracy of zero-shot video-question answering.
2
Related Work
Video LLMs. Recent successes of LLMs AAchiam et al. 2023 Anil al. 2023 have shed light
the burgeoning proliferation of video LLMs (Lin et al. 2023a Maaz et al. 2023 Gao el al.
2023). The
mainstream video LLMs follow the instruction-tuning paradigm. Building upon open-source LLMs (Chiang
et al.
2023 Touvron et al. 2023a b), this kind of method adapts the pre-trained video features into LLM
understandable representations via multi-layer perception projector (Lin et al. 2023a Maaz et al. 2023
ct al.
2023d
Munasinghe et al. 2023 Liu et al. 2024b; Jin et al.) 2023 Liu et al. 2023b), Q-former (Zhang
ett al.
2023a
Li et al. 2023b), or discretization tokenizer (Jin al. (024).. These methods, however, fail to
generalize to broader video understanding tasks, which may require flexible input and output formats (Wu
et al.
2023 Zhan et al. 2024). Several methods attempt to achieve this by employing additional functional
modules (Jin et al. 2024 Munasinghe et al. (023) (e.g., grounding (Munasinghe et al. 2023) or diffusion modules (Jin et al.
2024)) ), which are not flexible enough to adapt to diverse video underslanding needs.
To this end, another stream of agent-based methods (Qin et al., 2023 Yang et al. 2024b Fan et al. 2024
proposed, where multi-modal agents mostly bootstrap closod-source LLM (Achiam) et al. 2023 Brown et al. (020) decompose solution paths and call external tools. This kind of method relies heavily on delicately designed prompts and may fail to acquire the usage of a novel tool. Therefore,
Wang et al. 2024b)
3

---

Published in Transactions on Machine Learning Research (01/2026)
Table Tool repository of the proposed Video'ooo including single and compositional tools.
Tool
Specialist Model
Action Recognition (AR)
VideoMAE Tong al.
2022
Dense Video Caption (DVC)
PDVC
Wang et al.
2021
single tool
Temporal Action Localization (TAL)
IntervVides
Wang et al. 2022a
Optical Character Recognition (OCR)
EasyOCH
[JaidedA]]
2023)
Automatic Speech Recognition (ASR)
Whisper Radford et al
(2023
Video Relation Detection (VRD)
VidVRD
SSangg et al.
2017
Video Object Segmentation (VOS)
VisTR Wang et al.
(2021)
Text-to- Video Generation (T2V)
T2V Khachatryan al. (0023
muitik
AR + ASR
VideoMAE + Whisper
AR VOS
VideoMAE + VisTR
how to implement automatic invocation of related tools in video LLMs under non-stationary tool stream remains to be solved. This paper combines the strengths of the above two methodologies to empower video LLMs with automatic and continual tool-use ability.
Continual Learning. Continual learning (Wang et al. 2024a Lee et al., 2017 McCloskey & Cohen 1989
refers to the ability to incrementally acquire, update, and accumulate knowledge throughout the model lifetime without catastrophically forgetting previously learned information. Conceptually, four varieties of al.,
methodologies are posited. Regularization- basaaab-izatio-bbbbeeeeee approaches (Kirkpatrick
2017 Li & Hoiem 2017
Feng et al. 2022 Yang al., 2024a] strike the balance between the old and new tasks by adding explicit
regularization terms. Architecturo-basee approaches Yoon et al. 2018 Li et al. 2019
Ke al.
20200 isolate
model parameters for different tasks. Rehearsal- methods (Bonicelli et al.
2022
Chen & Chang
2023
Lin et al. 2023b) typically use memory buffer to store several training samples from previous classes, which
are used to approximate and recover old data distributions. Prompt-based mothods (Wang et al. 2022c
Smith et al. 2023 Wang et al. 2022b Li et al., 2024) usually construct task-adaptive prompts and select appropriate prompts during inference. This kind of method rehearsal-free and thus more computationally efficient. L2P (Wang et al., 2022c) introduces the concept of a prompt pool and selects prompts by query- key mechanism. To overcome the separate optimization issue of L2P (Wang al.) 2022c), CODA-Prompt (Smith et al. 2023) assembles learnable prompts with input-conditioned weights. DualPrompt (Wang et al. 2022b) and KC-Prompt [Li ct al. 2024) set up prompt pools to respectively encode task-invariant and lask-specific knowledge.
Borrowing the favorable rehearsal-free merit, our COLT inventively devises a video LLM with the continual tool-use lcarning capability. We differ from current prompt-based methods (Wang al. 2022c Smith et al.
Wang et al.] 2022b Li al.; 2024) in the following two aspects. Firstly, our locus is on the more complex task of multi-mode modal language generation opposed to the basic image classification task. Secondly, our COLT employs traigghtthhoough (Van Den Oord et al. 2017 Bengio et al. 2013) gradient estimation to
2023
mitigate the optimization challenges encountered in previous prompt-based approaches (Wang et al., 2022c
Smith et al. 2023: Wang et al. 2022b Li et al. 2024).
3 Dataset Construction
Dataset Structure. Each instance in VideoTool contains two rounds of conversations between human and gpt. The first round includes human instructions related to the video content and the LLM responses on choosing appropriate tools; The second round contains the execution results the corresponding tools and
the final responses from LLMs. Specifically, we follow Yaao et al., 2022 Yang et al.] 2023 Liu et al. 2023c) to unify the response format of GPT into three fields, including thought, action and value to mimic a human-like task-solving procedure.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Response X
Response Xa
Large Language Model (Ø)
Large Language Model (+)
Hl
Hw
H
Hw
Prt
Projector fs
Projector fa
Tool Codebook P
Visual Encoder g
Instruction Xw
Visual Encodor g
Hg Query Encoder fß
Image/ Image/Video X,
Stage
Image/Video X,,
Instruction Xw
Stage 2 - Stage 3
An overview of COLT. Stage 1 aligns the visual and textual modalities through the individual training of the linear projector fs; In stage 2 and stage the prompt within tool codebook P is adaptively selected according to the cosine similarity with the query feature Hq.
Figure 3:
This datasot is constructed using GPT-3.5-turie (Achiam et al. 2023 with self- instruction. The involved tools incorporate both video understanding and generation tasks. The prompt and in-context learning cases are available in supplementary materials. As shown in Table [ collect ten tools including eight single tool and two compositional tool. The tool list can be easily extended using similar dataset construction manner. We initially generate 5,000 instruction-followig samples for cach tool. We conduct both data format checks and manual verification of semantic meanings to filter out error data. After a thorough examination, the dataset is partitioned into distinct training and testing splits, adhering to proportionality ratio of 9:1. The full statistics of the final VodeoTool are summarized in supplementary materials.
Dataset Construction.
4 Method
Our COLT learns the continual tool usage from the stream instruction-tuningg dataset {D*) each D containing triplets of the visual data X user instructions X and the response sequence X For clarity, we claborate on the architecture and training strategy for one single datasct D' and the superscripts of X WT and X are omitted.
X
4.1 Architecture
Visual & Textual Embedding. Following LLn et al. 2023a), we adopt LanguageBind vision encoder (hhu et al. 2023a) g()) to extract visual features. Then, a linear projection layer fa(.) parameterized by ā is applied to compress the visual information into an LLM understandable feature space.
H. = fx())
(1)
where Hy PxC denotes the visual embeddings, with P and C respectively representing the patch number and the feature dimension. For input user instructions Xw, we adopt the widely used BPE tokenizer (Sennrich 2016 to obtain the textual embeddings Hw E SxC where S is the textual token nunber.
et al.
Tool Codebook Query Encoder. As shown in Figure 3 we set up a tool codebook as the tool-specific memory for continual tool usage. The codebook consists of N learnable tool prompts, f.e., P (P,, 1 with P. Cx1, For each input user instruction Xq. we aim to retrieve relevant tool prompts from the codebook. To achieve this, query encoder ja with parameters is firstly employed to encode user instructions X, into Hg E gxxi Then, we compute the cosine similarities between the query embedding Hę and each tool prompt Pn, E [1,2,- 2,,...,000 NJ. The tool prompts with the top- highest similarity scores are selected.
n" arg topk (P.. Hq).
(2)
In this paper, use tool represent the general skills and specialist model denote the specific model.
5

---

Published in Transactions on Machine Learning Research (01/2026)
where n" is the selected tool prompt index. Then each xx- is concatenated with the video feature Hy and user instruction features Hw and fed into LLMs. The hint contained in the codebook helps alleviate the catastrophic forgetting when facing the tool stream data.
4.2 Training
The proposed COLT adopts a three-stage training methodology, with each stage featuring unique trainable parameters and loss functions.
Stage 1: Vidco-to-text Alignment. This stage aims to train the projection layer fa parameterized by 5, which acts as visual tokenizer to align visual signals with pre-trained LLM word embedding. As shown in we freeze all the weights except the projection layer in this stage, i.e., trainable parameters 0 = 8. Given the predicted response sequence Xa = (Xa X2 x } of length L, we use the vanilla auto-regressive language modeling (LM) loss to supervise this stage of training:
Figure 3
= CLM = 1 log
Hy, Hwwxx,'
(3)
where <i is the response tokens before the i-th token.
this stage, we pre-train the tool codebook P and the query encoder fa while keeping the other parts frozen, i.e., the trainable parameters are = {P. We firstly select the appropriate tool prompt P,, via Eq.equation 2 Then P,, serves additional condition for language modeling follows:
Stage Tool Codebook Pre-training.
CiM 1 Σ log (Xa Hw, Pa,, Pa.,
(4)
where X and X is as defined in Eq. equation However, directly optimizing LiM will truncate the gradients w.r.t the query encoder fa due the den-derivable arg topk operation in Eq. equation 0 Thercfore,
we resort to the straight-through estimator (Van Den Oord et al. 2017 Bengio et al. 2013) to approximate
the gradient computation and define the overall loss function follows:
L = LiM + X1 ||sg [Hw] - P.- 112 + [Hw sg P..
(5)
E
where sg|:] stands for the stop-gradient operator that acts as an identity in the forward process and has zero partial derivatives during backward propagation. Lg is the quantisation loss for codebook update by forcing P towards the user instruction embeddings Hw, Le is the commitment loss to prevent. unrestricted update of codebook embeddings, X and x are the balancing weights.
Stage 3: End-to-end Fine-tuning. We keep the vision encoder 9(-) frozen and finetune remaining parts including the projection layer fs, tool codebook P, textual encoder fa and LLM parameterized by Φ, i..., the trainable parameters are f = {6, P.B. The training loss is the same as Eq. equation
5
Experiments
5.1
Experimental Settings
Training Datascts. In the first stage, we follow Video LLaVA Lin et al. (2023a) to use LAION-CC-SBU image subset. Schuhmann al. (2021) and the filtered CC3M video dataset Changpinyo al. (2021) for video-to-text alignment. For the second and third stage, use the combined dataset of 665K image-text instruction data from LLaVA v1.5 Liu et al. (2023)) and 100K video-text instruction data from Vidco- ChatGPT Maaz et al. (2023). The tool-use instruction data of our collected Video'Tool is introduced into continual training sequentially. To ensure conformity in data formatting, we reformat all the instruction pattern. Refer to supplementary materials for more details.
tuning datasets to the dhouugh--ction-Vie
Implementation Details. For vision encoder g()), we choose pre-trained LanguageBind Zhu et al. 2023a with VIT-I,/14 Dosovitskiy et al. (2020). The text. tokenizer is derived from LLLMM Touvron al. 2023a
6

---

Published in Transactions on Machine Learning Research (01/2026)
Table 2: Comparisons with state-of-the-art methods on zero-shot video-question answering.
"Acc." denotes accuracy (%) and "Score" denotes the relative score from to assigned by GPT Brown
et al. (2020). The best performance bold and the second best is underlined. The backend LLMs include
Vicuna-P 7B Chiang al. (2023) and LLaMA-7B Touvvon al. (2023a).
MSVD-QA
MSRVTT-QA
ActivityNet-Q2 QA
Method
LLM
Acc.
Score
Acc.
Score
Acc.
Score
Video-LLaMA
Zhang al.
2023a
Vicuna-7B
51.6
29.6
1.8
12.4
VideoChat
Li al.
2023b
Vicuna-77
56.3
45.0
2.5
26.5
Video-ChatGPP Maaz al.
(2023
Vicuna-7B
64.9
3.3
49.3
2.8
35.2
2.7
BT-Adapter
Liu et al.
(2023b
Vicuna-7B
67.5
3.7
57.0
3.2
45.7
3.2
LLMA-VID
Li et al.
2023d
Vicuma-7h 7B
69.7
3.7
57.7
3.2
47.4
3.3
LLMAAVVD
Li et al. 2023d
Vicuma-1 13B
70.0
3.7
58.9
3.3
47.5
3.3
Video-LLaVa.
Lin et al. [2023a
Vicuna- 7B
70.7
3.9
59.2
3.5
45.3
3.3
Chat-UniVi Jin et al.
2023)
Vicuna-7h 7B
65.0
3.6
54.6
3.1
45.8
3.2
LLMMA-Aapper Zhang et al.
2023b
LLaMA-7B
54.9
3.1
43.8
2.7
34.2
2.7
VideoChat2 Li et (2023c)
Vicuna-TT 7B
70.0
3.9
54.1
3.3
49.1
3.3
ST-LLM Liu et al.
2024b)
Vicuna-71
74.6
3.4
COLT joint (Ours)
Vicuna- 7B
78.2
4.2
3.6
54.7
3.8
COLT 5x2 (Ours)
Vicuma-7B
75.5
3.9
63.9
3.5
52.5
3.5
COLT 10x1 (Ours)
Vicuma-7B
74.7
3.9
63.0
3.4
51.2
3.4
with a vocabulary size 32,000, and Vicuna-7B v1.5 Chiang et al. 2023) is employed as the large language model. We uniformly sample frames from each video, and each frame resized to 224 224. We the batch size to 256 for the first stage and 128 for the second and third stages. AdamW optimizer is used with cosine decay schedule. We set learning rate to 1 10-4 1 10-4, and 1 10 for three stage training, respectively. The balancing weight Å and X2 in Eq. equation are set to 1 and 0.25. The codebook size N and selected prompt number K are set to 50 and 3, respectively. COLT is trained on NVIDIA A100 GPUs (80 GB memory each), and the full training process lakes approximately 10 hours.
Evaluation Metrics of Tool Continual Learning. We detail the metrics of average accuracy and average forgetting used in Sec. 5.3
Let Olky (0, 1] denote the tool call accuracy of j-th tool after incrementally training on the k tool data (j s k). The average accuracy of specific tool denotes the overall call accuracy during the incremental learning process. Then the average accuracy at task & is defined as follows.
1
AAE
okj-
(6)
Since average accuracy does not provide any information about the forgetting profile of the continual learning process, average forgetting is introduced to bridge the gap. For a particular tool, the forgetting measure is defined as the difference between the maximum tool call accuracy throughout the past learning process and the current tool call accuracy. In particular, the forgetting for the j-th tool after incrementally training up to k tool is as follows.
8
max atjj azjz Vj <.
The average forgetting of k-thh tool computed by normalizing against the number of tools seen previously:
k- AF
b: Σ
(8)
We report the average accuracy and average forgetting after the last tool learning.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Comparisons (%) with state-of-the-art methods on MVBench. The tasks include Action Sequence (AS), Action Prediction (AP), Action Antonym (AA), Fine-grained Action (FA), Unexpected Action (UA), Object Existence (OE), Object Interaction (OI), Object Shuffle (OS), Moving Direction (MD), Action Localization (AL); Scene Transition (ST), Action Count (AC), Moving Count (MC), Moving Attribute (MA), State Change (SC), Fine-grained Pose (FP); Character Order (CO), Egocentric Navigation (EN), Episodic Reasoning (ER), Counterfactual Inference (CI), and the average of all 20 metrics (AVG). The best performance is in bold and the second best is underlined.
Model
AVG
AS
AP
AA
FA
UA
OE
OI
OS
MD
AL
InstructBLIP Dai
20024
32.5
20.0
16.5
46.0
24.5
46.0
51.0
26.0
37.5
22.0
23.0
LLaVA Liu et al.
(2024a
36.0
28.0
39.5
63.0
30.5
39.0
53.0
41.0
41.5
23.0
20.5
VideoChatGPT Maaz et
2023
32.7
23.5
26.0
62.0
22.5
26.5
28.0
40.0
23.0
VideoLLaMA Z hang et al.
2023a
34.1
27.5
25.5
51.0
29.0
39.0
48.0
40.5
38.0
22.5
22.5
VideoChat Li cl
(2023b
35.5
33.5
26.5
56.0
33.5
40.5
53.0
40.5
30.0
25.5
27.0
VideoChat2
2023c
34.7
24.5
27.0
27.0
38.0
53.0
28.0
40.0
25.5
27.0
VideoChat2 Li et al.
2023c
51.1
66.0
47.5
83.5
49.5
60.0
58.0
71.5
42.5
23.0
23.0
GPT-4V OpenAl 2023
43.5
55.5
63.5
72.0
46.5
73.5
18.5
59.0
29.5
12.0
40.5
COLT joint (Ours)
53.4
64.0
65.0
77.0
44.5
54.5
75.5
70.0
34.0
36.5
33.0
COLT bx2 (Our)
51.8
62.0
61.0
75.0
42.5
52.5
74.5
69.0
33.5
35.0
32.0
COLT 10x1 (Ours)
50.6
60.0
60.5
73.5
42.0
51.5
73.5
68.5
32.5
33.0
30.0
Model
AVG
ST
AC
MC
MA
SC
FP
CO
EN
ER
InstructBLIP Dai et al. 2024
32.5
46.5
42.5
26.5
40.5
32.0
25.5
30.0
25.5
30.5
38.0
LLaVA Linn
2024a
36.0
45.0
34.0
20.5
38.5
47.0
25.0
36.0
27.0
26.5
42.0
VideoChatGPT Maaz et al. 2023
32.7
31.0
30.5
25.5
39.5
48.5
29.0
33.0
29.5
26.0
35.5
VidcoLLaMA hhng ct al. 2023a
34.1
43.0
34.0
22.5
32.5
45.5
32.5
40.0
30.0
21.0
37.0
VideoChat Li et b0233
35.5
48.5
35.0
20.5
42.5
46.0
26.5
41.0
23.5
23.5
36.0
VideoChat text
al.
2023c
34.7
38.5
41.5
27.5
32.5
46.5
26.5
36.0
33.0
32.0
40.0
VideoChat2 Li al. 2023c
51.1
88.5
39.0
42.0
58.5
44.0
49.0
35.0
40.5
65.5
GPT-4V OpenAl
2023
43.5
83.5
39.0
12.0
22.5
45.0
47.5
52.0
31.0
59.0
11.0
COLT (Ours)
53.4
88.0
37.5
53.5
75.5
35.5
42.5
48.0
34.5
42.0
57.5
COLT 6x2 (Ours)
51.8
87.0
37.0
52.5
72.0
33.0
48.0
33.5
40.5
COLT 10x1 (Ours)
50.6
86.0
36.0
51.5
71.5
31.5
39.5
47.5
32.0
38.0
54.0
Table 4: Ablations studies on MVBench. (a) training strategics; (b) training losses; (c) prompt se- lection mechanisms: T-based and V-based denote tool prompt selection based on text and visual features, respectively; (d) prompt positions.
Stage 2
Stage 3
AVG
Lag
Le
AVG
T-based
V-based
AVG
Position
AVG
50.4
37.7
45.3 tool-vision-teat
51.7
49.2
40.3
51.8 vision-tool-teat
51.7
V
v
51.8
51.8
V
46.4 nision-tezt-tool
51.8
(a)
(b)
(c)
(d)
5.2 Comparisons with Video LLMs
We set up three model variants: 1) COLTsx2 receives five successive groups of data and each group contains data of two tools; 2) COLT 10x1 is defined similarly: 3) COLT receives all tool data once and is trained with all the data collectively. The performance of COLT joint is regarded the upper-bound of the continual learning counterpart.
Evaluation Benchmarks. Our experiments are carried out on both established video LLM benchmarks and bulf- tool-using datascts: 1) zero-shot vidco-qucstion answering. We experiment on commonly used open-ended question-answer datascts: UIVDDQQA Chen & Dolan (001)), MSRVTT-QA Xu et (2016).
and ActivityYet--QA Yu et al. (2019). Following Li et. al. (003bb) Lin et al. (202aa)
GPT GPT-assisted
8

---

Published in Transactions on Machine Learning Research (01/2026)
(a) Comparisons with SOTA methods on test split of VodeoTool. AR, VRD, and ASR denote ac- tion recognition, video relation detection and aulomatic speech recognition, respectively. "Acc.' denoles accuracy (%) and 'Score' denotes the GPT-evaluated score.
(b) Comparisons (%) with continual learning methods on VideoTool. AA and AF denote aver- age accuracy and average forgetting, respectively.
Five Groups
Ten Groups
VRI)
ASR
Method
Method
Acc.
Score
Acc.
Score
Acc.
Score
AAT
AFL
AA
AF1
VideoLLaVA
47.7
2.89
6.0
Sequential
48.6
35.3
42.3
39.7
VideoChat2
51.4
3.13
14.0
1.36
3.60
Rehearsal (10/Tool)
52.9
32.1
48.2
36.3
VhaenChatt
29.0
2.31
4.5
0.90
2.80
1.28
Rehearsel (30/ TToo]
30.2
49.3
Videnagent
68.13
3.16
16.7
1.29
1.79
0.83
Rehearsal
Tool)
59.3
27.4
50.2
32.7
VITAL
52.23
3.09
15.28
1.24
1.98
0.85
T.2P Wang et al.
2022c
72.4
8.3
65.7
12.4
COLT Joint
77.9
3.84
24.0
1.75
24.2
1.95
Dual Wang et al.
20226
75.1
6.9
68.6
7.5
COLT ax2
3.60
CODA Smith et
2023
76.3
7.0
71.0
7.2
COLT 1081
73.5
3.48
19.6
1.27
21.9
COLT (urs)
5.8
74.7
evaluation to assess the model's capabilities by reporting the accuracy and relative score; 2) MVBench Li
let al. (2023c). This benchmark consists of 20 demanding video lasks, each comprising 200 samples presented as multiple-choice questions. These tasks offer a thorough and unbiased evaluation of a model's capacity to comprehend videos. We report the choice accuracy as the metric; 3) VideoTool. We built this dataset to probe the abilities enabled by tool proficiency. Since most existing video LLMs only support textual outputs, select three tools (i.e., action recognition, video relation detection, and automatic speech recognition) and
compare our COLT to state-of-the-art video LLMs Lin et al. 2023a Li et al. 2023c bb. The GPT-evaluated
accuracy and scores are reported.
Performance Analysis. The comparison results on zero-shot video-question answering, MVBench, and VidcoTool test split are summarized Table 2 Table 3 and Table 5a respectively. We can conclude with the following findings. 1) Both COLT joint and COLT 5x2 demonstrate superior performance compared to preceding state-of-the-art methodologies. For example on MSRVTT-QA (c.f. Table 2 COLT joint remark- ably surpasses the previous best performing method VidcoChat2 Li et al. (202cc) by 8.2% on the metric of accuracy; 2) On the test set of Toolenoole (c.f. Table 5a our COLT consistently outperforms prior approaches by clear margin, including both existing video- VQA models and representative tool-based agents, demonstrating its effectiveness in tool-intensive scenarios; 3) Even with a rather lengthy learning curve, COLT 10x1 still achieves comparable performance with previous SOTA methods; 4) The performance of COLT 10x1 slightly worse than COLT 5x2, which consistent with the intuition that tool learning becomes increasingly challenging when facing lengthening tool-chain.
5.3 Comparisons with Continual Learning Methods
We additionally compare the proposed COLT with popular continual learning methods on Video'Tool test split to demonstrate the life-long tool-usage learning capability of our method.
Evaluation Metrics. To evaluate how the system retains tool-using knowledge over time, we adapt the conventional continual learning metric Wang et al. (2024a); Chaudhry et al. (2018) to our tool-using scenario. We up two metrics: 1) average accuracy of specific tool denotes the overall call accuracy during the incremental learning process; 2) average forgetting is defined as the mean difference between the maximum tool call accuracy throughout the past learning process and the current tool call accuracy. A reduction in the value of average forgetting is indicative of enhanced continual learning capability. Refer to the supplementary materials for the formula and detailed explanations.
Compared Methods. We set comparison experiments as follows: 1) Sequential training: training on new data with the pre-trained weights on previous tool data as initialization; 2) Rehearsal training: replay past tool data (ie. "buffer") and combine them with new data. Buffer size is respectively set to 10/30/50 for
9

---

Published in Transactions on Machine Learning Research (01/2026)
each tool in experiments: 3) Popular continual learning methods including L2P Wang et al. (202b)). and CODA Smith et al. (2023)
Wang et al. (022c). Dual
Performance Analysis. We conduct experiments in two settings, i.e., five/ten groups with two/one tools per group. As shown in Table 5b COLT consistently exhibits superior performance in both average ac- curacy and forgetting across both settings. For example, under the five-group setting, COLT outperforms CODA Smith et al. (2023) by 3.5% in average accuracy, highlighting its effectiveness in mitigating catas- trophic forgetting. Notably, despite being given rehearsal buffers of size 10/30/50 per tool, rehearsal-based methods remain significantly weaker (Table 5b). as they depend on storing and replaying past data and still struggle with interference across heterogencous tools. In contrast, COLT achieves stronger retention withoul accessing old samples, demonstrating that its codebook-based tool memory provides a more stable and efficient continual-learingg mechanism than rehearsal.
5.4 Ablation Studies
We conduct extensive ablation studies to provide more insights into our proposed COLT. The experiments are conducted on MVBench Li et al. 2023ch with the contimual learning model variant COLT 5x2-
A Ablation on Training Strategies. Our COLT adopts a threo-stage training pipeline. We conduct abla- tion studies on the training process by respectively skipping the second and third stages of training. The comparison results on MVBench Li et al. (2023c) in Table 1a demonstrate that both stages are crucial to the final performance, e.g., skipping the second stage leads to 1.4% drop in average scores.
Ablation on Training Loss. We use the straight-through estimator Van Den Oord et al. (2017 Bengio et al. (2033) for the training of stage 2 and stage 3 (c.f. Fq. equation 5 which contains the quantisation loss Lg and commitment loss L. to enables the mutual updates between the selected tool prompts and query features. The ablation results in Table 4b underscore the significance of both quantisation loss Cg and commitment loss Le.
Ablation on Prompt Selection. Recall that select the most matched tool prompts based the similarity between the textual instruction features and codebook (i.e., text- cart-codebook). Here we ablate on the prompt selection based on the vision-codebook similarity or the average similarities of both. The results are listed in Table 4c which showcases that the tezt-codebook similarity is more reliable. That could be altributed to the fact it is easier for video LLMs to decide whether to invoke or which specific tool to invoke from the user instructions instead of the input visual data.
Ablation on Tool Prompt Position. We ablate the insertion position of the tool prompt. We've listed three options, including lool-vision-lezl,, vision-toolllear, and vision-tert-Lool, demonstrating placing the tool feature before/between//atee /after the vision and text features, respectively. The findings in Table 4d indicate that the performance is insensitive with regard to the tool prompt position.
A blation on hyper-parameters. We conduct ablation studies on the codebook size N and the number of selected prompts K. The comparison results are listed in Figure 4il As shown, the average performance is positively correlated with the codebook size N and reaches saturation at T 50. The optimal results are achieved when setting K 3.
Visualizations of performance us. continually introduced tasks. To intuitively show the impact of incrementally introducing tools, we show the phased zero-shot video-question answering performance under different training strategies. Specifically, we report the accuracy score on MSVD-QA dataset in Figure [ As shown, equipping video L.LSs with dense video caption tools leads to the significant performance boost for all four training strategics. Besides, our proposed COLT shows better learning stability, i.e., less forgetfulness when faced with new tool data.
6 Limitations
While COLT demonstrates strong continual tool-use capabilities, several limitations remain. First, VidcoTool is primarily GPT-generated, which may introduce distribution biases and limit the dataset's real-world
10

---

Published in Transactions on Machine Learning Research (01/2026)
76
N
20
30
40
50
60
COLT (Ours)
AVG
48.4
50.9
51.4
51.8
51.8
Accuracy
CODA
74
Rehearsal Training (50/ Tool)
K
1
3
4
5
Sequential Training
AVG
50.8
51.6
51.8
51.3
51.3
71
AR DVC TÀL.
OCR ASR vos r2V AAR-OCR ASR-DVC
Continually Introduced Tools
(i) Accuracy of zero-shot video-question ing on MSVD us. continually introduced tasks. The specific tool for each abbreviation is available in Table 1
(ii) Ablations nypee-parameters including the codebook size and the number of selected prompt K.
diversity compared with fully human-curated corpora. Second, the current dataset focuses mainly on single- tool and simple multi-tool compositions, which do not fully capture the complexity, interdependence, and noise present in real-world tool ecosystems, where tool chains may be substantially longer and more intricate. Future work includes extending VideoTool to more diverse and human-in-Che-lloy scenarios, scaling COLT to larger and more complex tool vocabularics, and exploring reinforcement learning other adaptive strategics to support dynamic and automatic tool composition.
7 Conclusions
In this work, we present COLT, which continually learns new tool-using knowledge in a data-stream scenario for general-purpose video understanding. To achieve this, we propose a learnable tool codebook where the specific tool is retrieved and activated according to the similarities with the user instructions. Due to the absence of video-centric tool-using eatruetion-tuningg dataset within the community, we devised VideoTool address this deficiency and foster the exploration of tool-using capacities of video LLMs. Experimental results indicate that proposed COLT adeptly invokes the necessary tools with precision, thereby achieving state-of-the-arl performance on widely used benchmarks and the proposed Video Tool test split.
Broader Impact Statement
The ability of our COLT to continually learn new tool usage without catastrophic forgetting may support the development of personalized video LLMs. Data within specific domains or pertaining to particular users can be incrementally fin-tuned, leading to an ever-evolving and personalized intelligent assistant. However, there also exists the risk that the model could exploited for malicious purposes. Besides, it may raise privacy concerns, especially is deployed in surveillance systems or social modia platforms.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. anXiv preprint arXim:2303.082 08774, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Mioch, Iain Barr, Yana Hasson, Karel Lone, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 627366, 2022.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. orXiv preprinl arXiv:2355.14403, 2023.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1rXi::198834392 2013.
Lorenzo Bonicelli, Matteo Boschini, Angelo Porrello, Concetto Spampinato, and Simone Calderara. On the effectiveness of lipschitz-driven rehearsal in continual learning. Advances in Neural Information Processing Systems, 35:31886 -19011, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1900 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Concept 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 3558-3568, 2021.
Arslan Chaudhry, Puncet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference computer vision (ECCV), pp. 332-547, 2018.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguisties: humon language technologies, pp. 190-200, 2011.
Xiuwei Chen and Xiaobin Chang. Dynamic residual classifier for class incremental learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 18743 18752, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL http://///nnn.oooggg//////////// vicuna/
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Newral Information Processing Systems, 36, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Rep- resentations, 2020.
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding. arXiv preprint arXin:2403. 1 11481, 2024.
Tao Feng, Mang Wang, and Hangjie Yuan. Overcoming rophst forgetting in incremental object detec- tion via elastic response distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9427-9436, 2022.
Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint urXiv:2306.88640 2023.
JaidedAI. Easyoer. http:://gttabb.cm/aaacddaa/////t
2023.
Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. ar preprint arXiv:231.000 10000000 2023.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Yang Jin, Zhicheng Sun, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, et al. Video-lavit: Unified video-language pre-training with decoupled visual- motional tokenization. arxiv preprint 200:2402.00161,, 2024.
Zixuan Ke, Bing Liu, and Xingchang Huang. Continual learning of mixed sequence of similar and dissimilar tasks. In Advances in newral information processing systems, pp. 18493 18504, 2020.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-lo-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15954- 15964, 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Baanskaa-Briiskka,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, :5521 35521-3526,,11 2017.
Sang- Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung- Woo Ha, and Byoung- Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems, 30, 2017.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, arXiv preprint 0lb.2305.06335. 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.1 1005, 2023c.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. In International Conference on Machine Learning, pp. 3925-33030 3934,,34,440 2019.
Yanwei Li, Chengyao Wang, and Jiaya Jia Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint ar arXto:2311.17443 2023d.
Yaowei Li, Yating Liu, Xuxin Cheng, Zhihong Zhu, HongXiang Li, Bang Yang, and Zhiqi Huang, Ke-prompt: End-to-end wwarreedge-complmeentte prompting for rehearsal-free continual learning. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5, IEEE, 2024.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pottern analysis and machine intelligence, 40(12):2935 2947, 2017.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by aligument before projection. aix preprint urXiv:2311. Xit::2311.00122 10122, 2023a.
Huiwei Lin, Baoquan Zhang, Shanshan Feng, Xutao Li, and Yumming Ye. Per: Proxy-based contrastive replay for online class-increnental continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24246 24255, 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In NewrIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
Ruyang Liu, Chen Li, Yixiao Ge, Ying Shan, Thomas H Li, and Ge Li. One for all: Video conversation is feasible without video instruction tuning. arXio preprint ar arXi:22009.5577 15785, 2023b.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. Xiv preprint or 0000022444000308 2024b.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05433 2023c.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiu:2506.05424, 2023.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and molivation, volume 24, pp. 169-165. Elsevier, 1989.
Shehan Munasinghe, Rusiru Thushara, Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, Mubarak Shah, and Fahad Khan. Pg-vidoo-llava: Pixel grounding large vidco-language models. arXiv preprint arXiv:2311.3343. 2023.
OpenAL. Gpt-Av(ision) system card. httpss//ggp..ennatiiccooo. htt.://a......rooouuuuoa02222210 2023.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models master 16000+ real-world apis. arXiv preprint 2rXf:22377117888 2023.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Nya. Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference Machine Learning, pp. 28492-28518. PMLR, 2023.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint xrv:22211 11.00114, 2021.
Rico Semnrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1715-1725, 2016.
Xindi Shang, Tougwei Ren, Jingfan Guo, Hanwang Zhang, and Tat-Seng Chua. Video visual relation detec- tion. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1300-1308, 2017.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Mascante-Bonille Donghyun Kim, Assaf Ar- belle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention- based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11909-11919, 2023.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders data-efficient Icarners for self-supervieed vidco pre-training. Advances in neural information processing systems, 35: 10078 10093, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprint Xrxi:22302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arxie preprint arXiv:2307.09988 092888 2023b.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2024a.
Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understand- ing with large language model as agent. arXiv preprint (0f::2403.100111 2024b.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint Xrx:::22 039191, 2022a.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF conference on compuler vision and poltern recognition, pp. 8741 8750, 2021.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision pp. 631-648. Springer, 2022b.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning prompt for contimual learning. In Proceedings of the IEEE/CVP Conference on Computer Vision and Paltern Recognition, pp. 139 149, 2022c.
Shenggiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm.
arXiv preprint XrX:2:2309. 05519, 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Mss-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEFE conference on computer vision and pallern recognilion, pp. 5288-5296, 2016.
Bang Yang, Yong Dai, Xuxin Cheng, Yaowei Li, Asif Raza, and Yuexian Zou, Embracing language inclusivity and diversity in clip through continual language learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6458 6466, 2024a.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 1381, 2023.
Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. Doracmongpt: Toward understanding dynamic scenes with large language models. arXie preprint or Xrxix:2401.0088 088392, 2024b.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expand- able networks In International Conference on Learning Representations, 2018.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yucting Zhuang, and Dacheng Tao. Activitynct-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.
2019.
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling, anXiv preprint Xix::2402. 1 12226, 2024.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 543 553, 2023a.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Haoji Zhang, Xin Gu, Jiawen Li, Chixiang Ma, Sule Bai, Chubin Zhang, Bowen Zhang, Zhichao Zhou, Dongliang He, and Yansong Tang. Thinking with videos: Multimodal tool-angmented reinforcement learning for long video reasoning, 2025. URL
Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arv preprint XrX:::2303. 16199, 2023b.
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, WANG Hongl', Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending rideo-langual pretraining to n-modality by language- based semantic alignment. In The Twelfth International Conference. on Learning Representations, 2023a.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing vision- language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2023b.
A Appendix
The outline of the appendix is as follows:
Dataset Details
Dataset Structure of VidcoTool
Prompt for Curating VodeoTTol
Instruction Tuning Dataset Reformat
Quality Control of VideoTool
Illustrations of Tool Selection
Experimental Results
Qualitative Results
A.1 Dataset Details
Dataset Structure of Vool Each instance in VideoTool contains two rounds of conversation between human and gpt. The response of gpt includes three fields of thought, action and value. One example is available in Table
Prompt for Curating VidcoTool. We collect VidcoTool by self-instruct GPT-3.5-turbo. The prompt and in-context learning cases are shown in Table 7
Instruction Tuning Dataset Reformat. In the second and third stage of COLT training, we reformat the instruction tuning dataset from LLaVA v1.5 (Liu et al., 2023ab and CTT (Maaz al. 2023 into the chought-action-e by GPT indicating that the question can be answered without invoking any tools:
pattern for the data format conformity. Specifically, the thought is generated
The questions can be answered by the information in the context, without need any external tools.
The action is an empty list while value the original response.
Quality Control of VideoTool: To ensure the reliability and integrity of the proposed dataset, conduct quality control from two primary aspects: data format checks and of semantic meaning checks. The format check entails verifying whether the generated data adheres to a predefined structure, specifically confirming that each entry corresponds lo two-round conversation and that the responses from the GPT model contain essential components such as thought/ heatt//ccio////eee consistency of tool names used throughout the datasct. Manual checks on semantic meanings are conducted to validate the contextual relevance and accuracy of the generated content, ensuring that it aligns with the intended purpose of the dataset and maintains coherence within the conversations.
keys. Furthermore, this phase involves confirming the
16

---

Published in Transactions on Machine Learning Research (01/2026)
Table 6: Dataset structure VideoTool.
An example of VideoTool dataset
First Round of Conversation
Human
Please segment the objects in the video.
GPT
<thoughts>
Request process video object segmentation based on the user's prompt.
<actions>
[[' name': video--ohjettsegmeetttiooo 'API params': (H)
<values>
Sure thing! I'll initiate object segmentation model to fulfill your request. Please wait while the processing takes place.
Second Round of Conversation
Human
You have successfully completed the process of video captioning and object segmentation.
GPT
<thoughts>
have successfully completed the process of video object segmentation. Now, it is time lo update the user.
<actions>>
<value>>
The objects have been segmented. You can now review the results.
Table System prompts and in-context learning examples for the generation of VodeoTool..
Prompts for GPr-b.5-turbo to generate instruction-tuming dataset:
You are an Al assistant and you receive a of conversations json string format, the content of which is used as data for instruction finetune. Your task is to rephrase the dialogue to generate new example. Note to give in standard json format. DO NOT modify the "from" and "actions' parts. Only modify the "value" and "thoughts' part.
In-context learning examples:
"Transcribe the audio content from the provided video.". "actions": [}, ( ('from': 'gpt', "thoughts": 'I need to find an ASR model and initiate the transcription process as per the user's request.', 'actions': I('AP ((AAl name": "asr", "API params': {)), "value": 'Certainly! I'll employ matic Speech Recognition (ASR) model. Please wait while the ASR model processes the video"), } { ('from': 'human', 'value': 'asr output: You have used ASR model to transcribe the audio content from the provided video based on my original request.", "actions": [}, ("from': 'gpt", 'thoughts': "Now that the ASR model has processed the video, I can update the 'value": "The ASR transcription you requested has been completed. Here is the transcribed text for your reference.") }}
('from': "human", 'value":
Auto-
the transcription results.", "actions":
A.2 Illustrations of Tool Selection
Figure 5 shows the full pipeline of our tool selection mechanism.
A.3
Experimental Results
employ additional CLIP-initialized encoder fo to extract query feature Hq. Then in Eq. equation 2 the appropriate tool prompt is selected based on the cosine similarity with H.. Here we conduct the ablation study which directly uses the encoded LLM features Hw for similarity computation. The comparison results in Table demonstrate that the introduced CLIP feature Hg leads to better performance. In addition, we further analyze how tool-use training and the codebook mechanism affect general vidco understanding beyond tool-centric benchmarks. As reported in Table B removing VideoToolBench data results in performance close to standard video- VQA models, while discarding the codebook causes substantial
In Sec. 4.2
17

---

Published in Transactions on Machine Learning Research (01/2026)
Query: Detect visual relations in the provided video, representing them relation triplets...
User Instruction
Tool Codebook
A
Query Encoder f
Compute Cosine
Similarity
Query Embedding Hg Rc
Large Language Model
Topk Prompt Selection
Figure 5: Illustration of the tool selection mechanism based cosine similarity. Given the query embedding h and a tool codebook {Pi..... Pv J we first compute the cosine similaritics between h, and each tool prompt p.. The most relevant tools are then selected (e.g., via top-K) and concatenated with the visual/text embeddings before being fed into the large language model.
MSVD-QA
MSRVTT-QU QA
ActivityNet-QU
Method
Acc.
Score
Acc.
Score f
Acc. f
Score
COLT (w/o VideoToolBench)
70.9
3.8
58.9
3.4
45.4
3.2
COLT (w/o Codebook)
52.3
3.3
42.7
3.0
33.7
2.9
COLT-joint
78.2
4.2
65.1
3.6
54.7
3.8
Table 8: Controlled ablations on standard VQA benchmarks to disentangle the effects of tool-use training (Vodeo"TollBench) and the codebook mechanism.
degradation across all benchmarks. These results suggest that the codebook plays critical role in preserving and reusing tool-related knowledge, enabling COLT to transfer tool-use behaviors to standard VQA tasks rather than merely memorizing tool traces.
Table 9: Ablations of the query feature source for tool prompt selection.
Query Feature Source
AVG
LLM feature Hw
47.4
CLIP feature Hg
51.8
A.4 Qualitative Results
We present additional qualitative results on MVBench (Li et al. 2023ch. zero-shot video-question answering Chen Dolan 2011 Xu al.) 2016 Yu et al. 2019), and VideoToo test split Figure 0 Figure 7 and
Figure respectively. Notably, our COLT preciscly captures dynamic vidco information and generate more reasonable responses.
18

---

Published in Transactions on Machine Learning Research (01/2026)
What was the unusual scene in the video that seemed impossible?
A man is swimming in coffee cup.
A man swimming in coffee cup.
man has his lower body inside coffee cup.
What makes the video counter-intuitive?
Invisible barriers are preventing movement.
Interaction with mythical creatures.
Humans display supernatural powers.
What happened before the person put down the book?
Opened the bag.
Opened the bag.
Took the bag.
What happened before the person put down the book?
Opened the door.
Put down the sandwich.
Closed the refrigerator.
User Instruction
Video LLavA
VideoChat2
COLT (Ours)
Figure Qualitative results on MVBench.
19

---

Published in Transactions on Machine Learning Research (01/2026)
8
Please create captions summarizing the content the video.
The video shows young girl playing with dog yard.
Overall, the video shows young girl playing in field with a dog.
Based on the video, can be observed that man playing with dog field.
The video shows dog pulling a girl down hill, and the girl unable stop herself.
Please caption the provided video and summarize the overall content.
The video shows standing outside, holding bottle of green liquid and glass. He then proceeds to pour the liquid into the glass and drinks it.
The video shows man a blue shirt tie standing front black box. He is holding sword and swinging it around in circular mation.
The video shows man using a sword to break bottle of water. He then pours the water into bottle and drinks it.
The video shows man attempting to cut bottle with a sword. He swings the sword at the bottle, but doesn't break. He then tries again, and this time the bottle breaks,
Please caption the provided video and summarize the overall content.
The video showcases a man playing the guitar in a serene garden setting. The man is standing under pergola, surrounded lush greenery and brick path.
The video shows squirrel walking on path in park. The squirrel is carrying cup of coffee in its mouth and walking towards bush.
The video shows squirrel playing with cup and then running away.
The video shows squirrel with yogurt stuck in its head, which is the main focus video. The squirrel seen walking around area, and the camera captures its movements.
User Instruction
Video LLaVA
VideoChat
VideoChat2
COLT (Ours)
Figure 7: Qualitative results zero-shot video-question answering.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Detect visual relations in the provided video, representing them relation triplets.
The video shows two red pandas playing with each other. The first panda seen eating bamboo tree while second panda playing with
I'm sorry, but cannot detect visual relations in the given video as it is not clear what the question is asking
The answer triplet of visual relations.
The video shows panda walking past another red panda, which be represented as relation triplet of "red panda, walk past, red panda"
want to identify the movements captured in this video. Can analyze and provide me with the results?
Sure, can analyze the video and provide you with the movements captured in it.
Based on observations, the video shows group of people gathered in a room for meeting.
Based the video, it can be observed that the is speaking and gesturing with his hands. He also seen standing and walking around.
Sure, can analyze video and provide you with the results. The video shows group of people playing rock scissors paper. The players are standing in circle and taking turns play the game.
Please transcribe the spoken content from the provided video into text
The video shows a diagram with arrows and text, and person speaking to the camera. The person is explaining the diagram.
I'm sorry, but the provided video does not contain any spoken content.
sorry, but cannot provide transcription of spoken content the video as is not available in the given text.
The video shows network trained using examples resulting from a strong attack can be expected to be also robust against weaker attacks.
8 User Instruction
Video
VideoChat
VideoChat2
COLT (Ours)
Figure 8: Qualitative results the test split of VideoTool.
21


Reviews:
[{"review_id": "aNYVY6LN8L", "reviewer": "Reviewer_hEDo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes COLT, a video LLM that learns to use tools continually via a learnable tool codebook and a query encoder that maps user instructions to relevant tool prompts. The selected prompts are concatenated with visual and textual features and fed into a Video-LLaVA–style architecture. Training is done in three stages (video–text alignment, codebook pre‑training, and end‑to‑end fine‑tuning). The authors also build VideoToolBench, a GPT‑generated video‑centric tool‑use dataset over 10 tools (8 single‑tool, 2 compositional). Experiments show strong performance on zero‑shot video QA benchmarks, MVBench, and VideoToolBench compared to existing VLMs, and improved average accuracy/forgetting compared to standard continual‑learning baselines.\n\nKey strengths:\n1. Simple but effective mechanism (tool codebook + query encoder) for rehearsal-free continual tool use.\n2. New video-centric tool-use dataset, filling a gap in current benchmarks (despite limitations).\n3. Strong empirical results on continual-learning metrics.\n4. Ablations are thorough and informative (training stages, losses, query features, codebook size, K, etc.).\n\nKey weaknesses:\n1. VideoToolBench is fully LLM-generated, with relatively shallow and low-order tool compositions; realism is unclear.\n2. Baselines in VQA and MVBench experiments are not tool-use models (as far as I can tell); the benefit of “tool usage” vs the proposed COLT framework is not clearly isolated.\n3. No comparison to training-free or training-based tool-using agents, so the advantage over competing tool-use frameworks is unclear.\n4. Some writing/presentation issues, especially citation formatting and a few awkward phrases.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The core empirical claims are generally supported: COLT clearly outperforms prior video LLMs on zero‑shot video QA, MVBench, and the selected VideoToolBench tasks, and shows better average accuracy and lower forgetting than sequential training, rehearsal, and prompt‑based CL methods. \n\nHowever, broader claims about real‑world continual tool usage and tool‑use–driven improvements on VQA are less directly supported. The tool data is synthetic and relatively simple, and tools do not seem to be invoked in the standard VQA and MVBench evaluations. Moreover, tool‑use baselines that actually call tools (e.g., training‑free agent methods or training‑based tool‑use models) are missing.", "audience_yn": "Yes", "audience_explanation": "The tool-use VLM task and the continual learning challenge will be interesting for many of the TMLR audiences.", "requested_changes": "1. Clearly state, for each benchmark and model, whether any tools are actually invoked at inference time, and adjust the discussion so that tool-free evaluations are not over-interpreted as direct evidence of tool usage.\n\n2. Explain the motivation for including standard VQA/MVBench experiments in a tool-use paper (e.g., to show general video understanding) and clarify how, if at all, the tool-use training and codebook contribute to these gains (for example, via an ablation without VideoToolBench or without the codebook).\n\n3. Clarify the status of baselines: explicitly state that the main VQA/MVBench baselines and the VideoToolBench video-LLM baselines do or do not use tools, and make it clear that comparisons there measure general video-LLM strength rather than superiority over other tool-use mechanisms.\n\n4. Add, where feasible, comparisons to training-free tool-using agents (e.g., prompting-based and retrieval-based tool planners) and/or training-based tool-use models.\n\n5. (Suggestion) Discuss the limitations of VideoToolBench more prominently in the main text: emphasize that it is GPT-generated, that tool compositions are mostly single-tool or very simple, and that real-world tool ecosystems may involve more complex and noisy tool chains.\n\n6. (Suggestion) Describe more precisely how tool-use and continual-learning metrics are computed: define what counts as a correct tool call, and summarize the formulas for average accuracy and average forgetting in the main paper rather than only in the appendix if space allows. \n\n7. (Suggestion) Add a bit more qualitative or quantitative analysis of forgetting and continual learning (e.g., a small number of examples where sequential training forgets earlier tools while COLT succeeds, beyond the existing plots).\n\n8. Improve writing and formatting, in particular standardizing in-text citations (e.g., “we adopt the LanguageBind vision encoder (Zhu et al., 2023a)” instead of “we adopt LanguageBind Zhu et al. (2023a) vision encoder”).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o6H3INRMmE", "reviewer": "Reviewer_agNa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an open-source video LLM framework that allows LLMs to use specialist models (tools) via a learnable tool codebook, enabling dynamic selection and continual learning of new tools without catastrophic forgetting. To complement the method, VideoToolBench is proposed as both a video-centric instruction-tuning training set and a evaluation set, supporting diverse tasks and sequential introduction of new tools. Results report COLT shows resistance to forgetting when incrementing new tools and surpassing prior methods in accuracy.\n\nStrengths and Weaknesses:\nS1) Uses a learnable tool codebook to enable video LLMs to integrate new specialist models (tools) sequentially without catastrophic forgetting is a novel continual learning approach, and it targets the underexplored problem of continual tool addition, stronger than prior fixed-tool assumption.\n\nS2) Provides a multi-task, instruction-based video dataset for both training and benchmarking continual tool-use. This is valuable asset as  currently there are no other publicly available datasets or benchmarks with similar nature.\n\nW1) The whole paper writing needs to be improved. See Requested Changes. There are issues with claims, terms definitions, and formatting.\n\nW2) The empirical results demonstrate continual learning for a preset sequence of tools, but it is not showing integration of truly arbitrary tools, nor scalability to very large tool vocabularies. The claims might need to be revised.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The overall claims made in the submission supported by accurate and clear evidence. A key claim is that COLT retains performance on previously learned tools when new tools are introduced, well supported by the experiments. However claiming COLT \"a lifelong learning video LLM\" might be overclaiming. The generalization claims on novel tasks and new tool scenarios are also weak because the method is evaluated using controlled sequential protocols, not open-ended or unseen tasks.", "audience_yn": "Yes", "audience_explanation": "I think this paper will be of significant interest to researchers working on continual learning and agentic tool use VLMs, as it presents a novel framework for dynamically integrating new specialist models (tools) into vision language models / video llms, along with a new multi-task dataset for training and benchmarking such capabilities.", "requested_changes": "C1) The paper writing at abstract and introduction lacks clarity and makes the paper somewhat difficult to comprehend at the first glance. I would recommend applying the following changes to improve the writing:\n\nC1a)  The abstract needs to be polished, especially on \"To harvest the benefits of well-trained expert models (i.e., tool), video LLMs prioritize the exploration of tool usage capabilities.\" change to \"To leverage the strengths of specialist models (i.e., tools) for specific video tasks, recent video LLMs have focused on integrating specialist models as tool usage capabilities into their architectures.\" The word choices in current version does not provide clear message e.g. on \"tool usage\".\n\nC1b) Consider re-formating the in-text citations in the entire write up. \"(Achiam et al., 2023; Brown et al., 2020)\" is easier to read than \"Achiam et al. (2023); Brown et al. (2020)\".\n\nC1c) When citing prior works, Its recommended to clearly point out the work for more clear view e.g. In introduction \"... recent attempts  Liu et al. (2023c); Qin et al. (2023) explore developing tool-specific instruction tuning data and elicit...\" change to \"... recent attempts such as LLAVA-plus (Liu et al., 2023c) and ToolLLM (Qin et al., 2023) explore developing tool-specific instruction tuning data and elicit...\".\n\nC2) \"VideoToolBench\", as a name of benchmark is used for both a dataset and a benchmark. This is confusing and should be clarified in the paper.\n\nC3) Please revise the claims that we discussed.\n\nC4) The quote formatting looks weird. “\" should be ``'' on latex? Please fix the notations.", "broader_impact_concerns": "/", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yFCzeZqbRJ", "reviewer": "Reviewer_3fH7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces COLT (COntinuaL Tool usage), a novel approach designed to enhance video Large Language Models (LLMs) by enabling them to continually learn and invoke new tools without suffering from \"catastrophic forgetting.\" The main contribution is the development of a tool codebook system, which acts as a memory store for tool-specific knowledge. This system allows the model to dynamically select and apply relevant tools based on user instructions.\nThe authors also propose VideoToolBench, a new dataset designed to train and evaluate tool-use capabilities in video LLMs. The paper demonstrates that COLT outperforms existing video LLM methods on standard benchmarks, achieving state-of-the-art results, particularly in tasks requiring the invocation of multiple tools.\n\n**Strengths:**\n\n1. Innovative Continual Learning Approach: COLT addresses the challenge of catastrophic forgetting in continual learning environments, making it highly adaptable for real-world scenarios where tool data evolves over time.\n\n2. State-of-the-Art Performance: The proposed system achieves superior results on both existing video LLM benchmarks and the newly introduced VideoToolBench dataset, showcasing its effectiveness.\n\n3. Practical Relevance: The use of a continually updated tool codebook and dynamic tool selection makes COLT more practical and flexible for video understanding tasks that require diverse tool functionalities.\n\n**Weaknesses:**\n\n1. Still relies on frozen specialist models; if those models drift or are updated, COLT must relearn.\n\n2. Tool codebook size and number of retrieved prompts are hand-tuned hyper-parameters; no principled way to grow the codebook automatically.\n\n3. Like all retrieval-based prompt methods, performance depends on the quality of the similarity metric; poor retrieval leads to wrong tool calls.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, the claims made in the submission are supported by accurate, convincing, and clear evidence. \n\n1. Experimental Validation: The paper provides thorough experimental results on well-established video LLM benchmarks, such as MSVD-QA, MSRVTT-QA, and ActivityNet-QA, where the proposed COLT model outperforms previous state-of-the-art methods. These results are presented in clear tables, showing percentage improvements in accuracy and scores over previous methods. This empirical evidence directly supports the claim that COLT improves tool usage and performance in video understanding tasks.\n\n2. Tool-Use Specific Dataset: The authors introduce the VideoToolBench dataset, specifically designed to evaluate tool-usage capabilities in video LLMs. This novel dataset and the associated tool-use instruction-following tasks provide additional evidence that COLT is capable of handling real-world, evolving tool streams. The dataset and its construction are detailed, showing careful thought and consideration in its design, further strengthening the validity of their claims.\n\n3. Comparison with Continual Learning Approaches: The paper also compares COLT with other continual learning methods on the VideoToolBench test split. The evidence provided through these comparisons highlights COLT's superior performance in mitigating catastrophic forgetting and its higher average accuracy compared to methods like rehearsal and sequential training. These comparisons are robust and show that COLT outperforms or matches state-of-the-art methods, even in difficult scenarios of continual learning.\n\n4. Ablation Studies: The paper includes extensive ablation studies that help clarify the importance of each component in COLT's architecture (e.g., training strategies, loss functions, prompt selection mechanisms). These experiments confirm the contributions of the various design choices in COLT, such as the use of a learnable tool codebook and the straight-through gradient estimator, providing convincing evidence that each part of the system contributes to its success.\n\n5. Clear Presentation: The paper is well-structured and clearly explains the methodology, experimental setup, and results. The figures and tables are easy to follow, and the authors provide comprehensive explanations of the technical details, making the evidence accessible and understandable.", "audience_yn": "Yes", "audience_explanation": "Yes, some individuals in TMLR’s audience would likely be interested in the findings of this paper. The paper addresses key challenges in video LLMs and continual learning, which are areas of growing interest in the AI research community. It introduces COLT, a framework for enabling continual tool usage in video models, and the VideoToolBench dataset for evaluating tool-use capabilities. These contributions are relevant to researchers working in video understanding, multi-modal models, and lifelong learning. Additionally, the paper demonstrates state-of-the-art performance in these areas, which could attract attention from those interested in improving or benchmarking video LLMs.", "requested_changes": "**Critical Adjustments:**\n\nClarify Generalization to Other Domains: Discuss how COLT might generalize to other domains, such as text-based or image LLMs, to broaden its applicability.\n\nAcknowledge Limitations and Future Work: Address potential limitations (e.g., scalability issues) and mention future research directions to provide a balanced view.\n\nExplain Hyperparameter Choices: Clarify the rationale behind selecting hyperparameters like the codebook size (N) and the number of selected prompts (K), to aid in replication.\n\n**Strengthening Adjustments:**\n\nProvide More Details on VideoToolBench Construction: Expand on the dataset’s diversity and impact on COLT’s learning.\n\nMore Discussion on Comparison to Rehearsal-Based Methods: Dive deeper into why rehearsal-based methods are less effective in this scenario.\n\nProvide Visual Illustrations of Tool Selection: Include a diagram to better explain the tool selection mechanism based on cosine similarity.\n\nMention Computational Resources Required: Add details on the computational resources needed to train COLT.\n\nBroader Impact Statement: Expand on potential ethical concerns, such as privacy risks in surveillance or social media applications.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)