SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM
Authors: Ming Nie
Dan Ding
Chunwei Wang
Yuanfan Guo
Jianhua Han
Hang Xu
Li Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: large language model
video understanding
Primary area: machine_vision
TL;DR: We introduce SlowFocus, a straightforward yet effective approach, which, mirroring human cognition, identifies the relevant segments pertinent to the query and subsequently completes fine-grained temporal tasks based on the temporal details located.

Abstract: Large language models (LLMs) have demonstrated exceptional capabilities in text understanding, which has paved the way for their expansion into video LLMs (Vid-LLMs) to analyze video data. However, current Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information (i.e., a sufficient number of tokens per frame) and comprehensive video-level temporal information (i.e., an adequate number of sampled frames per video). This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding. To address this issue, we introduce the SlowFocus mechanism, which significantly enhances the equivalent sampling frequency without compromising the quality of frame-level visual tokens. SlowFocus begins by identifying the query-related temporal segment based on the posed question, then performs dense sampling on this segment to extract local high-frequency features. A multi-frequency mixing attention module is further leveraged to aggregate these local high-frequency details with global low-frequency contexts for enhanced temporal comprehension. Additionally, to tailor Vid-LLMs to this innovative mechanism, we introduce a set of training strategies aimed at bolstering both temporal grounding and detailed temporal reasoning capabilities. Furthermore, we establish FineAction-CGR, a benchmark specifically devised to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks. Comprehensive experiments demonstrate the superiority of our mechanism across both existing public video understanding benchmarks and our proposed FineAction-CGR.

SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM
Ming Nie!
Dan Ding'
Chunwei Wang²
Yuanfan Guo
Jianhua Han?
Hang Xu²
Li Zhang"
School of Data Science, Fudan University
Noah's s Ark Lab, Huawei
https://githhl: htwpe//hhh.cccouuuuaaoooaaaaaa Swwrg//lowooussss
Abstract
Large language models (LLMs) have demonstrated exceptional capabilities in text understanding, which has paved the way for their expansion into video LLMs (Vid-LLMs) to analyze video data. However, current Vid-LLMS struggle to simul- tancously retain high-quality frame-level semantic information (i.e. a sufficient number of tokens per frame) and comprehensive video-level temporal information (i.e., an adequate number of sampled frames per video). This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding. To address this issue, we introduce the SlowFocus mechanism, which significantly enhances the equivalent sampling frequency without compromising the quality of frame-level visual tokens. SlowFocus begins by identifying the query-related temporal segment based on the posed question, then performs dense sampling on this segment extract local high-frequency features. A multi-frequency mixing attention module is further leveraged to aggregate these local high- gigh-freeuency details with global low-frequency contexts for enhanced temporal comprehension. Additionally, to tailor Vid-LLMs to this innovative mechanism, we introduce set of training strate- gies aimed at bolstering both temporal grounding and detailed temporal reasoning capabilities. Furthermore, we establish FineAction-CGR, a benchmark specifically devised to assess the ability of Vid-LLMs to process fine-gre ine-grained temporal under- standing tasks. Comprehensive experiments demonstrate the superiority of our mechanism across both existing public video understanding benchmarks and our proposed FcteA-ci....GRR
1 Introduction
Large language models (LLMs) have garnered significant attention due to their exceptional text understanding capabilities. Building the strengths of LLMs, video large language models (Vid- LLMs) [15, 18, 42] adapt them to the video modality, extending their reasoning and interactive skills to video data. By training video-level tasks like captioning and question answering, they establish coarse-grained video-language correspondence and acquire the capabilities of vidco understanding.
While Vid-LLMs have demonstrated the promising performance in video understanding, they still face a significant challenge. To embed video features into LLMss under computing resource constraints, Vid-LLMs typically need to sparsely sample the original video (e.g., retaining one frame every second) into collection low-frequency frames. Subsequently, the token number of each frame are also compressed through visual adapter like average pooling [16] Q-former [14]. This leads to a dilemma: Vid-LLMs have to choose between compromised frame-level features and video-level features, each resulting in reduction of video information. As illustrated in Figure 1 (a), under the premise a constant total token number, we investigate the performance Vid-LLMs by
Li Zhang (lizhangfd@ (ddan.edu.en) the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

35
Total token numbers per video 256
1024
4096
Accurscy FineAAtion CGR
What does the worrer do ofter introducing product?
20
2-
(Irput frames 10,20 62)
Query-related Frames Missing
The women is applying lpetick
15
(Urput fromes: 10, 20, 60)
The relevent tengenal segment from to
(Additional input fromes:
Temporal frequency Frome token num.
wemon opons tokes nakeup.
(a)
(b)
Figure I: (a) Trade-off between video sampling frequency and frame token number. The horizontal axis represents the ratio (og-traasformme) of these two factors. Each curve corresponds fixed total number of tokens (e.g.. 256 for I-minute video). (b) Deficiency of existing Vid-LLMs, such as LLaMA-VID, when facing fine-graine gna-ggri video understanding, and the efficacy of our approach.
dynamically adjusting the sampling frequeney and frame token number. The results clearly show that both low sampling frequency and low frame token number lead to performance degradation. Low-frequency sampling results in sparse image collection input, omitting crucial temporal details. On the other hand, excessive frame-feature compression degrades the semantic and spatial contexts of each frame. When confronted with fine-graine video understanding tasks, this limitation becomes significantly evident, as depicted in Figure (b). Existing models (e.g., LLaMA- VVD [17]) often overlook crucial details early in the input due to low-frequency sampling, leading inaccuracy.
To address this challenge, we introduce SlowFocus, which is designed to pinpoint relevant temporal segments in response to questions, and subsequently maintains high-quality temporal details to enrich fine-grained video comprehension. For video understanding tasks, we assume that the relevant details concentrated within one several clips. SlowFocus initially identifies these segments based on the provided questions. It then densely samples the segmented temporal clips at a high-frequency to extract local temporal features highly pertinent the questions. To effectively model the temporal relationships between frames and capture inter-frame contexts, we propose a specialized temporal encoder and multi-frequency mixing attention module for enhanced temporal comprehension.
To enhance Vid-LLMs' ability to perform SlowFocus based on mixed frequencies, we propose a set of training and inference strategies to improve their temporal localization and fine-grained temporal reasoning capability. Following VTimeLLMM [10], we fine-tune our Vid-LLM the second stage on dense video captioning and temporal grounding tasks. This process enhances the model's ability to predict discrete bins that define the relevant temporal segments. In the third stage, we adapt the Vid-LLM to the SlowFocus mechanism for high-quality, fine-grained temporal-related tasks, which enables our model to reason precisely based on high-frequency temporal details.
In addition to the scarcity of methods for fine mee-grained video understanding, existing benchmarks [37, 40] also fall short in providing adequate challenges for specific temporal-related tasks. To bridge this gap and evaluate our proposed framework, we introduce FineAction-CGR, newly dedicated benchmark that focuses on fine-grained video understanding, especially reasoning tasks based on temporal details. Our method demonstrates superior performance on the proposed benchmark, offering a promising solution to high-quality video understanding.
The contributions of this paper are summarized as follows: (i) We introduce SlowFocus, straight- forward yet effective framework designed resolve the prevalent trade-off in existing Vid-LLMs between capturing limited frame-leve details and overarching video-level contexts. SlowFocus adeptly maintains high-free high-frequency local details alongside lo-frequency global contexts, facilitating the identification of pertinent temporal segments and precise reasoning on video contents. (ii) We present a novel training strategy specifically designed to enhance the temporal localization abilities of Vid-LLMs, and seamlessly adapt them to our newly proposed SlowFocus approach. (iii) We establish a comprehensive new benchmark and carry out extensive experiments to rigorously assess the fine- fine-grained video understanding capabilities of Vid-LLMs. The empirical evidence strongly

---

indicates that our SlowFocus approach significantly outperforms existing models, particularly in tasks requiring detailed temporal understanding and reasoning within videos.
2
Related works
Vision large language models. Researchers have made significant efforts to enable Large Language MLdsl)(LLMMs) to comprehend visual information. BLIP-2 [14] aligns vision-language representation with lightweight Querying Transformer by concept of Q-Former. MiniGPT-4 [45] aligns detailed image descriptions with advanced LLM which significandly enhances its multi-modal abilities. Exploring diverse multi-modal Instruction-followin data, LLaVA [20] has demonstrated impressive multi-model chat abilities. Recent LLMs, like Kosmos-2 [26] and VisionLLM [33], probed into
more specific aspects of image comprehension, including referring and grounding [27], remarkably enhancing the capability to describe intricate image details.
Video large language models. The exploration of LLMs' potential has been extended from images videos, which contributes to emergence of Video LLMs. VideoChat [15] combines fine-tuning and LLM-based video agents and is fine-tuned using specially designed video-centric instruction dataset. Video-ChatGPT [23] proposes novel human assisted and semi-automatic annotation framework for generation high quality instruction data for videos. Video-LLaMA [42], trained on vision-language branch and audio-language branch separately with same visual data and process, has demonstrated impressive abilities in understanding both visual and auditory content in vidcos. Video-LLaVA [18] learns united visual representation by alignment before projection and conducts joint training on images and videos simultaneously. With efforts, Video LLMs have exhibited powerful task-handling capabilities in downstream tasks, like text-vides retrieval and video captioning. However, these models remain limited in ability to comprehend fine- gee-irrinnd content. To solve this problem, we introduce a model with powerful capability of fine-grained video understanding.
Fine-grained video understanding. Comprehending videos in fine-grained aspects requires precisely locating and understanding specific events within video. It is roughly divided by previous works [10] into temporal grounding [2. 7] and dense video captioning [13, 31] tasks. Temporal grounding demands the model preciscly identify start and end timestamps of video segment according to a given text query. Dense video captioning requires both temporal localization and captioning for all events within an untrimmed video. Both tasks are constrained by the iobto-inteesive nature of annotation, resulting in relatively small datasets. Moreover, fine-grained video understanding demands a deep comprehension of how objects change and interact over time, particularly a detailed level. This complexity necessitates temporal reasoning tasks to effectively analyze these dynamics. However, current research in this field falls short adequately addressing these requirements. To solve this problem, our proposed large-scale dataset provides mass temporal annotations and captions in different dimensions, supplying a wealth of information for training in temporal video grounding and reasoning tasks.
3 Method
In this section, we begin with a preliminary overview of video LLMs (Vid-LLMs) in Section 3.1. Following that, we offer a comprehensive explanation of our proposed SlowFocus in Section 3.2, followed by detailed introduction to our innovative training strategy in Section 3.3.
3.1 Preliminary
Contemporary Vid-LLMs typically feature a modular architecture, which includes a visual encoder Ey, series of visual adapters Q, and a large language model L. For given video V (V(t) x3 = 0, ..., T} that consists of T frames, along with its associated question 9. Vid-LLMs generally perform downsampling on the original video V at fixed interval Mr. This process results in sparsely distributed frames Vz.
V = (V(Mtt) E := 0,..,
where M, >1 1. Consequently, only a very limited number of frames (T/M, & T) are selected as the actual input for the Vid-LLM, technique we refer to as low- oow-freqynccy sampling in this study.
3

---

User Query
Stage Relevont segment grounding
Stage 2: Mixed -Frequency Sampling
What doer intreducing
Masse lecode the tempersi
Additionl compona!
noter query.
from 035
Video
Temporel Encoder
Temporal Encoder
High- Freeoee H Somplng
High-fro freq
The womon opess the box
From 023
035
makeep
Viaual Encoder
Visual Encoder
MMA
Tempord soken
tukin.
Wilud
Low- Freq
Figure 2: The framework of SlowFocus. We initially identify the relevant temporal segments based on the given query. Subsequently the seg-frreuencc sampling is performed on these segmented clips. Combined with low-frequency sampling across the entire video, our SlowFocus mechanism maintains freeuunny visual tokens accurately answer the query.
Subsequently, the visual encoder processes the downsampled frames V2 and encodes them into series of visual tokens denoted as 2g = Ev (V2). These visual tokens are then transformed to align with the embedding space of the language model through the visual adapter Q. resulting in aligned visual tokens hı = ((z)). Concurrently, the input text query q is encoded into linguistic tokens hy by the textual encoder. These visual and text tokens concatenated into unified sequence [hr, hg]. which then serves as the input for the decoder component of the large language model The model leverages this combined representation infer the appropriate answer showcasing its ability to perform cross-modal reasoning and respond to human queries.
= ((rr,,y))
Although this paradigm is well developed in the field of video understanding, it encounters significant limitations in tasks requiring fine-grained temporal reasoning. By retaining only small, discontinu- subset of frames (t, where t << T). substantial information loss occur, depicted in Figure I (b). Fine-grained temporal video understanding demands that the model concentrate on one or more specific temporal intervals, which could be potentially brief. However, the low-frequency sampling method predominantly captures overarching information while neglecting crucial local details, lead- ing inaccuracies such incorrect responses hallucinations. To address these shortcomings, develop the SlowFocus strategy.
3.2 SlowFocus
To improve Vid-LLMs with the SlowFocus mechanism, we first expand the traditional training and inference paradigm by introducing Mixed-Frequents Sampling (MFS). Subsequently, we explicitly model the temporal relationships among the sampled frames. Finally, we enhance the visual tokens with our newly proposed multiple-frequency mixing attention, designed capture long-term contexts. now provide a detailed elaboration on each of these components.
Relevant segment grounding. To better mimic human cognition in our Vid-LLM, we transform its inference paradigm into a multi-round dialogue format that integrates both the query and temporal awareness elements. The methodology of this transformation is illustrated in Figure 2. Initially, we sample the entire video at a fixed interval to capture the low-frequency frames Vr,, which are then encoded into visual embeddings hr. Following that, we reformat the original question into temporal grounding questions qI, such as "<video> nPllease provide the temporal segment help to reason the question: <question>", where <question> refers to the original question

---

Stage 1: Modolity Alignment
Stage 2: Boundary Enhancement
Stage 3: SlowFocus Adaption
Dote
Data
Data
b68k
Assistant: <captiory
User: <image>
9BK
Userr ovideo hnprompph Assistont: ronswer>
widee=<kegramge Additiona temperal
User:
User: vvidkor Assistont: coaptiore
User: evideos
140
csegments vandwars
Assistont: <coptiore
User: evideos owideo/\mppong Imprompts Aspistant: esegment>
Model
Model
Temporal Encoder
Temporal Encoder
Trependl
Visual Encoder
Visual Encoder
Visual Encoder
Excalie
MWA
Projectors
Projectors
Projectora
LLM
LLM
Loke
LLM
Figure 3: The training strategy of SlowFocus, including data distribution and parameter updating in each stage, <image> and <video> denote the tokens for image and video, respectively.
With the augmented question 91, direct the Vid-LLM to identify the relevant temporal segments within the video that are pertinent to the original question This is accomplished by providing the model with fow-freuuecyy frames containing global information:
[( 1n
where by, represents the textual embedding of question qi and [0, The identified segment is designed to encompass query-related details that are crucial for addressing the question
Mixed Mixed-frequency sampling. Subsequently, perform dense sampling the identified segment to obtain feguufrey frames VH:
| 1).
VH (M E "xxx33
x3 1 () maz(
We dynamically adjust the sampling interval MH based the temporal length of segment |r| and to ensure an adequate number of samples NH are taken from the relevant segment. The densely sampled frames VH are then encoded into frequenyy visual tokens hH by visual encoder.
With the inclusion of local details from high-frequency frames, we augment the initial question g with prompts, such as "Additional temporal clues to focus on: ...", to form g2, as illustrated in Figure 2. We then combine the mixed-frees freuueerrenncc visual tokens to predict the final answer:
ans L(h,, x (hr,, hyp))
where и is the
mixing attention, which will be detailed subsequently.
Multiple-frequency mixing attention. Because of the roughness of the low-frequency and the detailed specificity of the figh-freeuency visual features, simply concatenating these two directly into the language model may not yield optimal results. Fine-grained temporal understanding often requires modeling relationships between multiple events, necessitating the integration of global contexts into high-frequency local features. From this perspective, we introduce the Multiple-frequency Mixing Attention (MMA):
huhT VLL
x Th,, hz) = softmaz
(5)
Vd
where, for simplicity, we omit the process of applying FFN to h2 and hn The resulting output is then fed into LLM to predict the textual response, described in Equation 4.
Temporal relationship modeling. While LLM can implicitly capture temporal relationships among input visual embeddings based on their sequential positioning, it faces difficulties when the visual tokens are not uniformly distributed along timeline. To address this challenge, we propose temporal encoder that encodes the relative positions of frames into set of discretized temporal
5

---

Step Video Preprocessing
Shep II: Annotation Construction
Step III: Formation of Downstream Task
specific strategy
build comprehensiive annotations
generate with specific prompts
Coptioning
clip
doing
Instructional Video An individual
inatrnciionalodded
applies heal eyeliner to enothee person is seyes in close-up shat against soffly in backdrap.
Clip Captions
The person a epplying madoupp eyebroos
Terporai Gournding
Q: Whehe the person the vidco aist eyetense?
from 40s
Video Recaptioner
New Annotations 000hoosicepp
Tempord Reassning
split
duration
G: haw meny times dise the person paint eye shadow from 8ih to 72:50em video"
Ground Truth
summery perdon spo syeliter precis arather person's face.
GPT-4
fílenam:
duration
action Jun pounse, meto
QA
avathotioe::
Ox Can you polutrllllyy desol the conte' wdee which makeup application
conne
(ongment (oo,, xx]]
(ekpee 4,
sysbrowx))
'spely
dip_ clpphio indrvicual esples eyer.
epell eyebrm maisep,, epeoterinn undilpuzic on another personts fooz with procsion. the indivduall dunng from Ao During this time, the persmn appp aboant, epeccetthing maaleus face that painting oye shodow.
appless
dipuactionn lopely eydiner)
conuistions:
Time segments
(negmentt (ox, sx1 labet unq
Video
Capp Stgments
Figure 4: Pipeline of instruction-follovinng data generation. Split the filtered Fction videos into clips and extract time segments. Use the fine-tuned Video Recaptioner Model and GPT-4V to generate captions for both video clips and full videos. Integrate the ground truth data, new captions, and time segments to create comprehensive annotations. Finally, generate QA pairs for various tasks using different prompts via GPT-4.
tokens& NCC, where N represents the discrete temporal space. For frames sampled at multiple frequencies (t). the corresponding temporal token is e;, where LNV t/TJ.. These temporal embeddings are then incorporated into the visual tokens by directly adding them to the frame features.
3.3 Training strategy
Considering training efficiency, we introduce novel training procedure which is structured into three distinct phases in this work: (i) pre-training for modality alignment, (ii) fine-tuning for enhancing temporal grounding, and (iii) SlowFocus adaption, as illustrated in Figure 3. We now provide detailed elaborations on the training strategies and datasets employed for each of these stages.
Modality alignment. In the pre- pre-training stage, our primary focus is to optimize the visual adapters and temporal encoders, while the visual encoder and language model are frozen to ensure that the visual features align effectively with the language space. Following the approach used in LLaMA-VID [16], we utilize the image-text LCSS-58K dataset from LLaVA [20], along with 232K video-caption samples from the Web 2.5M [3].
Boundary enhancement. After the pre-training stage, the Vid-LLM gains proficiency in processing visual information. During the fine-tuning stage, we focus on enhancing the model's ability to comprehend sequences of video frames, thereby improving its temporal localization capabilities. In alignment with practices from VTimeLLMM [10], we employ the Intern Vid- 10M-FLT dataset [34], which specifically designed for temporal-awarrnes, awareness training. The tasks within this dataset in- clude: (i) dense video captioning, which requires detailed descriptions of all events along with their corresponding timestamps, and (ii) segment captioning and temporal video grounding, which in- volve generating descriptions based on timestamps or determining timestamps based on descriptions. Throughout this stage, we continue to train the visual adapters and temporal encoders. Additionally, the LLM is further trained using LoRA [9] to refine its capabilities.
SlowFocus adaption. Following the fine-tuning stage, our model has demonstrated the ability to comprehensively understand activities within the video and accurately align them with their respective timestamps. In the final stage, to further enhance
comprehension and integration with the MMF mechanism, we construct an instruction- tumnnn dataset using samples from ActivityNet Captions [13] and FineAction [21]. We transform the annotations into over 100K high-quality QA dialogues, which will be detailed in the subsequent section. In alignment with our SlowFocus mechanism, during training with fine- nreigraided captioning and reasoning tasks, the model is provided with the ground-truth temporal segments Tor and subjected to mixed frexednnr sampling. In this

---

Temporal grounding
Temporal captioning
Tempural reasoning
Mothod
LLM
LoRA
mloU
R@0.3
R@0.5
R@0.7
B
M
R
Acc
Score
VideoLLaMA [42]
Vicuna-7B
x 0.17
0.25
0.11
0.00
0.06
0.09
0.08
0.13
8.75
0.53
Video-ChatiPT [23]
Vicuna-7B
x
0.06
0.10
0.01
0.00
0.15
0.12
0.10
0.21
13,93
0.79
LLaMA-VID [16]
Vicuna-7B
x
0.35
0.52
0.17
0.03
0.16
0.12
0.11
0.23
15.65
0.87
VTimeLLM [10]
Vicuna-7B
27.69
32.83
24.26
21.87
0.05
0.08
0.12
9.96
0.54
LLaMA-VIDf
Vicuna-7B
22.38
26.17
19:47
17.53
0.23
0.20
0.37
1.03
24.81
1.26
Ours
Vicuna-7B
66.68
85,80
73.01
56.25
0.66
0.41
0.70
3.27
53.10
2.78
Table 1: Main results on FineAction- Aeeeiiiioo-GRRR benchmark. The column LoRA represents whether the LLM is fine-tuned fully or using LoRA. t: Model is re-trained on the stage 3's data. B: B @4. M: METEOR. R: ROUGE. C: CIDEr.
MSVD-QA
MSRVIT-QA
NetivilyNe-QQA
Video-hasol genentive perforrance
Method
LLM
LoRA
Acc
Score
Anc
Score
Ace
Score
Cornectness
Detail
Content
Temporal
Consistency
FrozenBLLM
DeBERIa Y2
32.2
16.8
24.7
VileoLaMM [42]
Viauns78
51.6
2.5
1.8
2.16
1.82
Adapter [[]]
LL
3.1
2.30
1.98
VideoClut [15]
Vicuns-BB
56.3
2.8
2,5
2.2
2.23
2.50
2.53
1.94
Video-CharGiP [23]
Vicunn-7B
64.9
3.3
49.3
2.8
33.2
2.7
2.40
2.52
2.02
1998
2.37
LLMM-VVD MIAA-VIDIII [16]
Vicuna-B8 7B
69.7
3,7
57.7
3.2
47.4
3,3
2.96
3.00
3.53
2.46
2,51
MMA--ID
Viauss78
60.2
3,4
57.1
2.9
45.6
3.3
2,87
2,89
3.78
2.13
2,47
Vicuna
70.1
3,9
58.3
3.5
3.6
2.95
3,03
361
2.54
Table 2: Comparison with existing methods on coarse-grained video understanding benchmarks. Our method achieve on par performance with state-of-the-art models.
stage, we freeze all the visual adapters and attention modules, only training the language model as well as the MMA module. The parameters of visual encoder are frozen all over the stages.
4 FineAction-CCe CGR benchmark
In addition the scarcity of previous methods for fine- fine-grained video understanding, existing bench- marks [37, 40] do not adequately challenge specific temporal-relatee tasks. To address this gap and evaluate our proposed framework, introduce FineAAcion-CCOn
comprehensive and high- quality instruction- following benchmark designed for evaluating the capabilities of Video LLMs in fine-grained video understanding. FineAction [21] is large-scale and fine-grained video dataset en- compassing diverse action categories with detailed annotations of action instances and time segments. For the purpose of SlowFocus adaption and model evluation, we divide the FineAction dataset into training and testing sets based on videos, allocating 75% to the training set and 25% to the test set, ensuring these is no overlap between them. Building on this foundation, we probe time information and content information from FineAction and expand it into multi-tasks. This section outlines the data construction procedures, which consist of three main steps: (i) video preprocessing, (ii) annotation construction, and (iii) formation of downstream tasks. More details are included in the appendix.
Video preprocessing. As depicted in Step I of Figure 4, employ modified two-stage splitting algorithm from Panda70M [5] to segment videos into clips, thereby providing raw materials for fine-grained information at the clip level. This preprocessing step yields 62,912 video clips.
Annotation construction. Step in Figure 4 outlines our approach to creating new annotations, which include time segments, captions, and action labels. Given the high cost of using GPT-4V [1] for generating large-scale, clip-level captions, we generate detailed captions for entire videos with GPT- 4V and then use fine-tuned Video Recaptioner Model to produce large-scale, clip-level captions. By integrating action labels and time segments from the ground truth with clip segments and the generated captions, we create comprehensive new annotations.
Formation of downstream tasks. Utilizing the new annotations, we design four types of video catruction-following tasks, as illustrated in Step III of Figure 4: (i) segmented captioning, (ii) temporal video grounding, (iii) temporal video reasoning and (iv) multi-turn QA. Detailed information on construction procedures, task formats, prompts, and case studies can be found in Appendix C- E.
7

---

Global mode Score
Breakpoint mode
Method
LLM
LaRA
Ase
Method
LLM
LoRA
Acc
Acc
Score
FrozenBiL [99]
DeBERTa [8]
VideoChat [15]
Vicuna-7B
K
57.8
3.00
46.1
2.29
VIOLET [6]
X
19.9
Lade [42]
Vicuna-7B
51.7
39.1
2.04
IntenViikeo [35]
32.1
L. [41]
Lama2-78 7B [29]
34.0
Video-ChaGGPT [23]
Vicuna-7B
K
47.6
2.55
48.0
2.45
Vamos [3]]
GPT4
36.7
MovieChut [28]
Vicuna-7B
x
62.3
3.23
48.3
2.57
LangRepo- [12]
Mixtral [11]
Ours
Vicuna-7B
58.6
3.14
48.1
2.53
Ours
Vicuna-78
39.7
Table Comparison with existing methods on MovieChat-IK.Taa 4: Comparison with existing methods on EgoSchema.
Temporal grounding
Temporal reasoming
Sampling strategy
Temporal encoder
MMA
mloU
R@0.3
R@0.5
R@0.7
Acc
Score
Ve
x
X
32.54
41.67
34,52
25.84
30.25
1,57
Vn, NH
x
34.81
44.65
36.49
39.12
V2 + VH,NH 10
/
x
57.26
73.47
62.28
47.16
46.37
2.37
VL. VH,NH
x 64.92
83.25
70.19
55.13
51.68
2.66
V. VH,Nv 40 /
x 65.03
82,96
70.23
56.01
51.59
2.62
V2 VH,
66.68
85.80
73.01
56.25
53.10
2.78
Table 5: Components analysis. Vy means only low-frequency frames are sampled. Vz VV represents performing mixed -exeedffreuency sampling and Nи denotes the number of high-frequency frames.
Evaluation protocol. To assess the capability of video language model in comprehending fine- fine-grained video understanding tasks, we integrate multiple evaluation metrics in comprehensively evaluate the model's performance against our proposed benchmark. For temporal grounding task, we calculate the Intersection over Union (IoU) between the temporal segments generated by the model and the corresponding ground truth, and we report meanloU (mloU) metric. For segmented captioning, employ commonly accepted captioning bassed metric, including BLEU [25], METEOR [4], ROUGE [19] and CIDEr [30]. For temporal reasoning task, we utilize GPT-4 [1] to evaluate the accuracy and score of the generated answers.
Experiments
5.1 Experiment setup
In our experiments, we implement LLaMA-VID [17] as baseline and utilize Vicuna-7B v1.5 [44] foundational LLM. More implementation details provided in the appendix. We adjust the resolution of input videos to 224 x 224, and each frame is condensed into 64 tokens. The low-frequency sampling interval Mr. is set to match the original video fps, ensuring
frame is sampled every second. We define the dense sampling number N₂ as 20 and the size temporal token space N 1000. The AdamW [22] optimizer is applied with cosine learning and decay and a warm-up period. We train our Vid-LLM for three stages. During the initial pre-training stage, the learning rate set to 10-3, For the subsequent fine-tuning stages, the learning rate is adjusted to 2 x 10 Additionally, the LoRA parameters are configured with r = 64 and alpha 128. All experiments are conducted V100 GPUs.
5.2 Main results
Results on fine-grained video understanding. We evaluate the performance of existing Vid- LLMs on our proposed fine-grained video understanding benchmark FineAAcion-CCol CGR, as detailed in Table 1. Further specifics are available in the appendix. Our method significantly outperforms other counterparts, achieving a mloU of 66.68 for temporal grounding reasoning task. Notably, most other models, with the exception of VTimeL.LM [10], exhibit subpar performance in both the temporal grounding and reasoning tasks on the benchmark. We propose several possible explanations for this observation: (i) These models lack sensitivity precise time
an accuracy of 53. 53.10% for

---

the shat from 100 to the video scored not?
Look at this video, If person storts with playing the keyboord. conyytu prodict
noat action
(Eepat fromes: 0, 25, 75, 125)
(Erput Fromes: O 20, 40. 100)
nelevant temparal asgment
nelevant hemparal augment 100
(Adeit and impur framesi 100, S0L. 110
(Additional input fromes 45, 46,
Oar
No, the shot is Net scured.
Yes. After play the keyboard, people seltch to playing the druns.
Figure 5: Qualitative examples. Our proposed SlowFocus can effectively leverages the segmented temporal clues to accurately answer the posed question.
boundaries, making it challenging for them to accurately predict temporal segments. (ii) Due to their reliance on low-frequency sampling, these models struggle to capture fine-grained temporal details, which adversely affects their performance on tasks requiring fine-graine grained temporal reasoning.
We also fine-tune the baseline using stage 3. To ensure fairness, the implementation details for baseline fine-tuning are kept consistent with those of our method. The baseline's performance improves because stage 3 includes tasks focused on temporal grounding and is specifically designed for fine-go tne-grained analysis. Furthermore, the remaining performance gap further supports our explanations.
Temporal grounding R@0.3
Temporal reasaning Score
Temporal grounding
Temporal reasoning
Stages
mloU
R@0.5
Re0.7
Acc
Fps
Ny
mloU
R@0.3
R@005
R@0.7
Acc
Score
1
0.11
0.29
0.00
0.00
7.13
0.51
64
1
24.15
33.05
24.84
25.71
34.58
1.80
142
51.67
63.29
57.84
45.81
20.34
1.04
16
4
45.19
58.67
50.17
38.18
45,36
2.35
36.72
32.27
1.63
16
65.12
2.67
14243
66.68
85.80
73.01
56.25
53.10
2.78
64
66.68
85.80
73.01
56.25
53,10
2.78
Table 6: Ablation of training stages. The first stage Table 7: The results of dynamically adjusting sam- itself yields poor result. Integrating these stages pling frequency (fps) and frame token number Ny: together results in the optimal performance.
when maintaining constant total token number.
Results coarse-grained video-based benchmarks. In Table 2, present comparative evaluation of our method against various state-of-the-ar methods across three zero-shot video- QA benchmarks: MSVD-QA [36], MSRVTT-QA [37], and ActivityNet-QAA [40]. We also conduct experiments on the video-based generative performance benchmark [23]. The results demonstrate that the proposed SlowFocus mechanism not only enhances fne-grained video understanding but also delivers competitive performance in coarse-grand -aariee video understanding tasks, achieving results on par with state-of-the-art models.
Results on long video benchmarks. To further investigate the effectiveness the proposed method on more challenging scenarios, we provide evaluations on long video benchmarks, includ- ing MovieChat- 1K [28] and EgoSchema [24]. As shown in Table 3, we evaluate our method on MovieChat- IK. The results show that although our method is not specifically trained on long video benchmarks (in contrast, MovieChat [28] has undergone targeted training for long videos), it still achieves competitive results (58.6% accuracy in global mode and 48.1% % in breakpoint mode).
Additionally, we also conduct experiments on EgoSchema [24] benchmark, as detailed in Table The results further demonstrate that, despite not being specifically trained on long video datasets, our method still achieves competitive performance.
Qualitative results. Figure 5 illustrates the qualitative results of our method on different videos. Our SlowFocus comprehensively analyzes the entire video, accurately identifies relevant temporal details based on the posed question, and provides precise answer within the context of videos.
5.3 Ablations
In this section, we provide detailed ablation studies of our method conducted utilizing Vicuna-7B as the foundation model.
Component-wis wise analysis. We first investigate the impact of each proposed component in Table 5. Importantly, we observe that the baseline model, which relies solely on low-frequency frames (Vr,), achieves limited performance, with an mloU of 32.54 and an accuracy of 30.25. This highlights the

---

Temporal grounding
Temporal captioning
Temporal reasoning
N
mloU
R@0.3
R@0.5
@@0.7
B
M
R
C
Acc
Score
O.1K
43.81
62.26
51.64
33.18
0.47
0.33
0.49
2.15
39.41
2.20
1K
66.68
85.80
73.01
56.25
0.66
0.41
0.70
3.27
53.10
2.78
10K
63.74
86.19
71.05
55.17
0.66
0.40
0.71
3.24
52.59
2.71
Table 8: Ablation study on temporal token space N.
challenges faced by current Vid-LLMs in addressing fine-grained temporal tasks using only low- frequency sampling. Additionally, the results in the second row indicate that the inclusion of MFS significantly improves reasoning capabilities, resulting in an accuracy increase of +8.87. Furthermore, the temporal encoder, which models temporal relationships effectively. boosts both grounding and reasoning capabilities, with an increase of +22.45 in mloU and +7.25 in accuracy. Increasing the high-freynency sampling number N₂ from 10 20 leads to noticeable performance gain (+7.66 in mloU and +5.31 in accuracy), although further increasing Nu to 40 offers only marginal benefits. Lastly, the MMA module contribute an accuracy enhancement of +1.42.
Necessity of fine-tuning strategy. We also analyze the impact of each training stage, as reported in Table 6. Directly evaluating the pre-trained model after stage I yields poor performance. Based on necessary pre-training stage 1, when only utilizing stage 2 for boundary enhancement, there is significant improvement by 51.56 in mloU in temporal grounding ability. Furthermore, implementing only stage 3 for SlowFocus adaptation boosts the performance in temporal reasoning by 25.14 in accuracy. Integrating these stages results in the highest performance both mIoU and accuracy metrics, highlighting the irreplaceability of our proposed comprehensive training strategy.
What contributes more fine-grained video understanding? In Table 7, conduct further experiments to explore how changes in sampling frequency and the number of frame tokens influence our method. For meaningful comparison, we maintain constant total token numbers while gradually decreasing the sampling frequency. The results indicate that within proposed SlowFocus, performance improves with an increase in frame tokens and remains relatively unaffected by decrease in global sampling frequency. This demonstrates the efficacy of our approach in high-frequency sampling and effectively alleviates the trade-off dilemma between sampling frequency and frame tokens.
Ablation on discretized temporal space. We investigate the influence temporal token space in Table 8. The results demonstrate that when the token space increase from 0. 1K to 1K, significant improvement (+22.87 in mloU and + +13.69 in accuracy) occurs. While further increasing N from 1K to 10K, the performance drops instead.
6 Conclusion and limitations
Conclusion. We introduce SlowFocus, a straightforward yet potent mechanism that significantly enhances fine-grained video understanding. Our approach improves the temporal localization capa- bilities Vid-LLMs, enabling them to precisely identify relevant temporal segments based on the given query. Moreover, with newly developed temporal encoder and multi-free multi-frequency mixing attention module, our method effectively models the temporal relationships among frames and cap- tures frame context. Demonstrating superior performance our newly established fine-grained video understanding benchmarks, we hope that our work can propel the development of Vid-LLMs in achieving advanced video understanding.
Limitations. Despite significant advancements made this work to enhance Vid-LLM's access to high-frequency temporal details and its temporal reasoning capabilities, challenges persist due to the limited existing research on maintaining high resolution in video. Consequently, our method may still encounter inaccurate predictions stemming from the ambiguity of spatial details.
10

---

Acknowledgements
This work was supported in part by National Natural Science Foundation of China (Grant No. 62106050 and 62376060) and Natural Science Foundation of Shanghai (Grant No. 22 1407500).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint, 2023. 7,8
[2] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. ICCV, 2017.
[3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021. 6
[4] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop, 2005. 8
[5] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, YiingYYingaa Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross- mosslttodllty teachers. arXiv preprint, 2024. 7, 14
[6] Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint, 2021.
[7] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV, 2017. 3
[8] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.0366 2020. 8
[9] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint, 2021. 6
[10] Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower 1lm to grasp video moments. arXiv preprint, 2023. 2, 3, 6, 7,8, 8, 14
[11] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint, 2024. 8
[12] Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding. arXiv preprint, 2024. 8
[13] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. captioning events in videos. In ICCV, 2017. 3,6 6
Dense-
[14] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML, 2023. 1,
[15] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, arXiv preprint, 2023, 1, 3, 8
[16] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint, 2023. 1,
[17] Yanwei Li, Chengyao Wang, and Jiaya Jia, Llama-vid: An image is worth tokens in large language models. arXiv preprint, 2023. 2,

---

[18] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprinz. 2023. 1,3
[19] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004. 8
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neurul information processing systems, 2024. 3, 6,
[21] Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal action localization. TIP, 2022. 6, 7
[22] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint, 2017.
8
[23] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint, 2023. 3, 7, 8,
[24] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 2023. 9
[25] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. ACL, 2002. 8
[26] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint, 2023. 3
[27] Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed, Michael Ryoo, and Tsung- Yu Lin. Learning to localize objects improves spatial reasoning in visual-Ilms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12977-12987, 2024. 3
[28] Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. CVPR, 2024. 8, 9
[29] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint, 2023. 8
[30] Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basse image description evaluation. In CVPR, 2015. 8
[31] Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, and Yong Xu. Bidirectional attentive fusion with context gating for dense video captioning. In CVPR, 2018.
[32] Shijie Wang, Qi Zhao, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Vamos: Versatile action models for video understanding. arXiv preprint, 2023. 8
[33] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, al. Visionllm: Large language model is also an open-ended decoder for vision-centrie tasks, Advances in Neural Information Processing Systems, 2024. 3
[34] Yi Wang, Yinan He. Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma. Xinhao Li. Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint, 2023.
[35] Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang. Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint, 2022. 8
12

---

[36] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. ACM MM, 2017. 9
[37] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Mst-vtt: A large video description dataset for bridging video and language. In CVPR, 2016. 2, 7.9
[38] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 2022. 7
[39] Antoine Yang. Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 2022. 8
[40] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019. 2, 7.9
[41] Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple Ilm framework for long-range video preprint, 2023.
arXiv
[42] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint, 2023. 1, 7,8
[43] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llapa-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint, 2023.
[44] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 8
[45] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models. arXiv preprint, 2023.
3
13

---

A Overview
In this appendix we present:
More implementation details (Section B).
Data construction details (Section C).
Data statistics (Section D).
Prompts used for instruction generation (Section E).
Discussion on broader impacts (Section F).
B More implementation details
In our work, we employ CLIP-ViT-L- 14 for the frozen visual encoder, which accepts image resolutions 224 224 as input. For visual adapters, we follow LLaVA [20] to train a projector layer, aligning visual features with the pre-trained LLM word embedding.
We also elaborate on how enable LLM predict temporal segments during stages 2 and fine-tuning. Following VT [10], we utilize the textual format "from s to e" to denote a video clip, where and represent the starting and ending points, respectively. These time points are normalized and range from 000 to 999, corresponding to the discretized temporal token space N. During training, Vid-LLM is exposed supervisory signals described above, which enable it to develop the capability to accurately locate temporal segments.
C Data construction
Step I: video preprocessing. We split videos into video clips to provide materials for fine-grained information in video clip level.
A proper splitting and stitching strategy is necessary. For one thing, the content of each clip should be semantically consistent. Without this consistency, actions within the same video clip obtained by erroneous segmentation can't be inferred reasonably and in chronological order, which negatively impacts the performance of downstream tasks such as action recognition and temporal reasoning. For another, the video clips with insufficient information that are too short subsequent tasks, such as temporal grounding. Therefore, we adopt a modified two-stage splitting algorithm in Panda70M [5] obtain semantically consistent video clips.
suitable for
In first stage, We collect time boundaries of events in the video. Instead of detecting abrupt changes in pixels of adjacent frames, detect lens switching that actions in the same video clip can be inferred reasonably. In second stage, we stitch adjacent events based on time boundaries in first stage. Our stitching strategy could be depicted merging short video clips and that are semantically similar. To be specific, we merge current event with the previous, if one of the following two conditions is met: t)e video clip duration is less than five seconds, 2) the feature distance between the start frame of the current event and the end frame of the previous event does not exceed 0.1. Finally, we split videos into clips according to time boundaries from second stage.
Our adjusted splitting strategy has two advantages: 1) preserves the information density of each segment, ensuring that subsequently generated captions are reasonable; 2) it maintains logical temporal consistency of actions within each segment, which facilitates the generation question- answer pairs in downstream tasks. Following the preprocessing step, obtained 62,912 video clips.
Step II: annotation construction. As Figure 4 illustrates, new annotations we construct consist of time segments, captions and actions. (1) Action labels are from FineAction ground truth. (2) Time segments cover clip's time segments from above segmentation step and the action time segments from ground truth. (3) Captions cover video captions and clip captions. We first utilzie GPT-4V to generate detailed captions for entire videos. However, due to the cost of GPT-4V for generating large-scale, clip-level video captions, we fine tune a Video Recaptioner Model based on the captions generated by GPT-4V, which is then utilized to generate large-scale, clip-level captions. With actions
14

---

and time segments from ground truth, we integrate clip segments from splitting stage and captions generated by GPT-4V and Video Recaptioner Model to get new, comprehensive annotations.
Step III: formation of downstream tasks. We design various types of instruction-follovii tasks with different prompts to comprehensively train and evaluate Vid-LLMs. FineAction-CGit encompasses 4 video tasks: (1) captioning, (2) temporal video grounding, (3) temporal video reasoning and (4) multi-turn QA. A few examples from different tasks are shown in Figure 8-9,
Captioning task needs model to detects what actions occur given period.
Action recognition: recognize action in level of clip which may contain kind of action several kinds of actions.
Temporal grounding tasks aims predict the boundary in the video given instruction in which the start and end time of clip are mentioned to attract model's 's attention to target segment. We design two types of question achieve fine-grained temporal localization.
First/last time grounding: identify the fist/last time a certain action appears.
Temporal reasoning tasks requires model understand relationship between actions.
Action sequence reasoning: infer possible action before/after an action over a period of time.
Count of times: counts times a certain action appears a clip.
Multi-turn QA task contains multiple rounds of dialogues involving the above three tasks. The problems are progressive from the entire video to the sub-segment.
Action Types
Task Distribution
4.75% 0.61%
000%
447%
14%
32%
5%
90.12%
27%
Figure 6: Video Statistics in FineAction-CGR. It contains diverse distribution of action types and tasks
clips
Figure 7: Distribution of clips number in FineAction-Cos
D Data statisties
After filtering, we obtained 11,188 videos with different number of action types. The distribution is illustrated in Figure 6 left. After splitting, videos are segmented into clips. The amount of videos with different clip number is presented in Figure 7, exhibiting toilet distribution. The largest
15

---

Actions Recognítion
Review the video from 347s to 147.647s Can you tell me what the basketball player in white jersey doing during this time section?
dunk basketball
First/Last Time Grounding
Identify the first time the athletes dribble basketball this clip.
From 36.003s to 37.104s
Action Sequence Reasoning
Based on segment 36.003s 42,342s, likely happen after player dribbles and casts the basketball?
The player is likely to dunk the basketball.
Count of Times
How many times do the athletic basketball players dunk the basketball from 150.684s to 162 6299 in video?
Multi-turn QA
Can you briefly describe video's content?
The video captures a high-energy basketball game featuring players in blue and white uniforms. There are also segments showcasing abstract shapes letters, a dynomic basketball practice session, group activity invalving athletes in dark blue uniforms.
What are the players doing from 36. to 145.913s?
From 36.003s to 146.912 16.9133, players in team uniforms are diligently involved basketball game. They are seen dribbling, passing, and shooting the basketball including performing dynamic slam dunks.
Could you tell me when the first slam dunk was performed from 36.003s to 146. 913s?
O
The first slam dunk was performed from 41,842 to 42,342s
player performed from 214,447s 236.669s?
last oction player performed casting basketball
Between the periods from 36.003s to 146.913s and from 150.684s 182.649s, which had more slam dunks?
period from 36.003s 146.913s had more slam dunks
Figure Example of different tasks in FineAction-CGR
16

---

Actions Recognition
What were the performers doing from Os to 35.33s in the video?
They playing the keyboard
First/Last Time Grounding
When was the first time of the people played the keyboard in the performance?
From Os 35.33s
Action Sequence Reasoning
Look this video from 50.33s to 103.66s, if person starts with playing the keyboard, con you predict what he will do next?
After playing the keyboord, the individual switches playing the drums.
Count of Times
How many times does individual play the keyboard from Os 142.933s the video?
Multi-turn QA
Can you describe the event showed in this video?
Two people are performing music on stage, One is playing the keyboard and other playing the drums.
During which period does the keyboard playing occur from Os to 142,955 142,933s?
The keyboard playing occurs from Os to 35.33s, from 50.33s to 57.66s, and from 104.33s to 141.66s
When does the first scene of playing the drums take place from Os to 142.933s?
The first scene of playing the drums takes place from 57 66s to 103,665
What first
occurs the period from 142,933s7
The first action that occurs is playing the keyboord
What typically happens after the keyboard playing ends during Os to 142,933. 12.9333?
Typically, playing drums follows after keyboard performance
Figure 9: Example of different tasks in FineAction-CGR
17

---

clip number is more than 68. The horizontal coordinate in the figure is only displayed to 35. After generation step, we obtain 131,984 pairs of QA. The distribution in different tasks are illustrated in Figure 6 right. Explanation of the legend 1) 'ar': Captioning task where the model needs to recognize a single action clip, 2) 'asr': Captioning task where the model needs to recognize multiple actions in clip, 3) ssst:: Temporal grounding task where the model needs to localize the first/last occurrence of given action clip, 4) 'sequence": Temporal reasoning task where the model needs to identify what action happens before or after given action in clip, 5) 'times": Temporal reasoning task where the model needs to count the occurrences given action a clip, 6) 'miqqa: Multi-turn QA task where the model engages in multi-turn question and answer dialogue about the entire video.
E
Prompt
Prompts used for generation of different kinds of instruction data are shown in Figure 10-13. Due to page length constraints, have omitted some in-context examples in certain tasks.
F
Broader impacts
The proposed SlowFocus has the potential to greatly enhance the capability of Video LLMs for video content analysis. Therefore could be beneficial for applications such video analyties, surveillance and automated content moderation. It could also be used in educational settings to analyze educational videos to provide a better understanding of complex contents in videos for educational purposes.
As for the potential negative impacts, there is a risk that the technology could be misused to analyze private videos and spreading disinformation.
18

---

Prompts for ction Recognition
Generate a concise dialogue with questions and answers based on the following video caption, clip captions, action label and time segments with sub segments. All the sub-segments in segment represent all the time periods in which an action occurs. Below are few examples:
<Bxample 1>
[Video Caption]
{A spirited basketball game unfolds an indoor court, where teams in contrasting uniforms compete, witnessed by an excited audience under the glare of bright lights.}
[Time]
("0:00:00.000", "0:00:20.520"]
[Clip Caption]
(Players in yellow and jerseys engage in an intense baskethall game, running, passing, and dribbling on a well- lit court, captured in wide-ange angle static shot.)
ction Label]
dribble basketball
[Time Segments with Sub-segments]
["0:00:00.000", 00:00:).,520";: {[5.014, 5.983], [6.953, 7.989], [9.025, 9.794], [10.997, 11.966), [15.008, 16.245), [17.382, 19.454))
IQ & A]
[("from":" "rromm"""umnn,,
"value":" "oatte":"Lookk" at from 15.008 to 16.245 of the video. What the player is doing?" },
("from": : "gpt",
"value"." dribble basketball")]
Now given the following video caption, clip captions, action label and time segments with sub segments, please generate a question to test whether the deep leamning model can give an action label based on the information provided by asking question like "W hat does the person do from to b in this video?" or " hat do the people do from c d this video?".
{video caption)
{time segment}
{clip_ captianttinn
(action label}
(time segment: : sub segments)
The people mentioned must be from [Video Caption] and [Clip Caption]. And the time mentioned should refer to [Time]. The answer must be [A ctioo Label].
Just return the list as your response, like following format:
[[ ("fomm":" "human"""""" "value"."... from 0 to 20 ...").
("from":" "opo", "value"."... Just return the list.
Figure 10: Prompt for Action Recognition
19

---

Prompts for Temporal Grounding
Generate concise dialogue with questions, answers based on the following clip caption, action labels and their time sgements. The action labels are in chronological order. Below few examples:
<Example 1>
[Action Label]
dribble basketball, cast basketball, dunk basketball
[Clip Caption and Time Segment]
("0:00:00.000 "000000.000,,,,,. "0:00:20.524 " Players in yellow and purple jerseys engage in intense basketball game, running, passing, and dribbling on a well-lit court, captured in wide wide-angle static shot.
[Action Labels and Time Segments]
[0.33, 1.301] dribble basketball,
[1.668, 2.636] cast basketball,
[4.33, 4.638) dribble basketball,
[5.0, 5.639] cast basketball,
[11.011, 12.679) dribble basketball,
[13.013, 13.981] dunk basketball
[Q & A]
"value":" "Identify the first time the player dribbles baskethall in this clip. }. ("from" "gpt", "value":"from 0.33 ") ("from"" "hhuman"""" "value": :" "deetify the last time a player casts baskethall in this clip.
("from": : gpt', "value":"from 5 to 5.639")]
Now given the following segment caption, action labels and their time sgements, please generate questions and their answers, please generate three questions related to time and action to test whether the deep learning model recognition fine- grained action based on the information provided by asking question like "identify the first/last time the person/people does/do a certain action during this clip."
(action label)
fclip_cappione caption]: (time_ segment3 (actions and segments list)
Don't mention time segments in questions. The people/person in questions must refer [Clip Caption]. The actions mentioned in the questions are selected from [A ction Label]. The answers should be directly inferred from [A ction Labels and Time Segments). Find the time period of the first/last occurrence of an action based on the time given in the question. Keep the answers brief, with no more than words each.
Just return the list as your response, the format must be like following example: "value".".." ").
("from": "grt", "value":"fron <s3> to <o3>") ", ["from": ""hmma":""umnn" "value":" ""..u)"".....), ("from": ""rtt, "value":"fro <s4> to <o4>", (, "fhuma""" "value":"... ".. ("from": 'gpt", "value":"fron <s5> to <o>>)}]
The value from 'human' is question. The value from 'ppt' answer.
Juse Just return the list.
Figure 11: Prompt for Temporal Grounding
20

---

Prompts for Temporal Reasoning
Generate concise dialogue with questions and answers based the following clip caption, action labels and time sgements. The action labels are in chronological order. Below is an example:
<Example 1>
[Clip Caption and Time Segment]
("0:00:00.000 ...00.000","..... "0:00:20.522 ] Play ers in yellow and purple jerseys engage in an intense basketball game, running, passing, and dribbling on well-lit court, captured in wide-angle static shot.
[Action Label]
dribble basketball, cast basketball, dunk basketball
[Action Labels and Time Segments]
[0.33, 1.301] dribble basketball,
[13.013, 13.981] dunk basketball [Q & A]
[["from":" hhuman,,, "value": :infer from 0.33 to 2.636, what the player would most possibly do after dribble basketball?" }. f""foom:" :gpp", "value":"Go "Cast baskethall") }}
("from": "human", "value": "Look at this clip from 0 to 20, what players usually do in basketball game?"), ), ("from":"gpt) "value": "Playerss a basketball game usually dribble basketball and cast basketball.")
("from" "humann, "value": 'Look at this clip from 11.011 to 13.981, player dribbles basketball. Guess what would he do after that?" }, I"frrmm: ""ppt,""p" "value": ""hhe player may dunk basketball. "},
["answer_segment" ([apmntt([(16600
Now given the following clip caption, action labels and time segments, please generate questions and their answers, please generate three questions related to time and action to test whether the deep leaming model can recognition fine- gre-ggrained action based on the information provided by asking question like 'Based on sub segment of this segment and infer what will most possibly happen after an action." or "Look this video from a to b, W hat's the most possible action before/after this step?"
time segment: clip, caption
action list
actions segments list
The actions mentioned in the questions are selected from [Action Label]. The should be inferred from the provided time and action sequence in ction Labels and Time Segments].
Just return the list as your response, please follow below format:
[["from": "humann, "value":" ...") ".."), ["from"m: '''pt,, "value":".."" "..."),""nnssn ("answer segment": <<3>> <+3>|}
The value from huuman' is question. The value from 'gpt' is answer. The time in every question must be in format from ... ...' and be the maximum period containing question's and answer's actions' but not others. "answer segment" is a list of time segments pointing to minimum time priod of the answer's action and each is subset of time in corrresponingg question. Dont't write what you imagine, Just return the list.
Figure 12: Prompt for Temporal Reasoning
21

---

Prompts for M Multi- tum QA
You are an excellent ent assistant for generating multi-round conversations. I need you to generate a dialogue with multi- turn questions and answers based on the information of video. The information is complete annotation of video which contains one or more clips in chronological order. It mainly has captions, time segments and actions. Below is an example:
<Example 1>
<Information>
"ummmary": "A fast- paced basketball game ... "action list": ["cast basketball",
"meta": [t "clipid" 0, "clip caption": "An intense basketball game indoor court..",
"clip action": [...]. "time segment": [...].
"annotations": : "segment"; [...], "label"; "... "...B
IQ A]
[[ "rrommm""uue ""human", "value":" you briefly describe the video's content?"), ("from":"gptt "value":"Ti well-lit indoor arena hosts an energetic basketball game.."
("from": "fromm""humman, hhuman", "value": Wat the players is doing from 0 to 11.878?"), ("from":"gpt" "value":"Tha players are competing. They dribble and cast basketball on the court."),
("from":"hume "hummn", "value": :aatuee"""Plaa"" "llease capture the first time player is dribbling basketball from 0 11.878."), ",, ("from": "ittom"="gpt" "gpt", "value":" from 0.033 to 1.969.") ".
"frrom""human" "value": "What is the last action before dunking basketball from 52.686 to 52.986?" ), ( "fromm""gpt" "value": nAn player casts basketball."" }},
{"from "'rom"""human" "human", "value":"Please" infer what players ers usually do a baskethall game." }, ("from":"gpt", "value"."TT usually dribble basketball or cast basketball." }, "task":[ '''p',''', 'ap', 'grd', 'sn', rsn')]
Now given the following <Information>, please generate dialogue with five pairs of questions and answers to test whether the deep learning model has capability of captioning, temporal grounding and especially temporal reasoning based on the information provided.
Captioning task contains video captioning, clip captioning and action recognition during a time peiord. Temporal grounding task needs to find out target time period according to requirements whose answer a time period. Temporal reasoning task needs to logically infer behavior motive from video, such as time sequence between actions, action sequence in an activity or reasons why people in video do such an action. Questions must designed according to information provided and must contain all three tasks. You can design temporal reasoning questions different from examples I give you, but don't write what you imagine.
(all information)
Just return the list as your response, following is format example:
[f"from":"humm "value": "Can you briefly describe ...?"), ('from" "gppt,, "value"....."" ("from":" "humann, "value":" from ... ...?",, "tfffomm""gnm "value":".."""
("from": "human",, "value":" ... from ... ....".. ), ("from": :: "gpp", "value" "from to .... .).....
("from":" "rrom"""humann" "human", "value":" from ... ...?"), }, "'fmmmm : gptt", "value"."...
("from": humann" "value".".."" ("from":" gpt", "value":" ".."", "task" "".
The value from 'human' is question. The value from 'gpt' answer. 'task' list of string in which 'op' means captioning, 'grd' means grounding, 'isn' means reasoning. The length of 'task' list are equal to the number questions. Just return the list.
Figure 13: Prompt for Multi-turn QA
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We make the main claims clearly in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of this paper in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
23

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We discuss the experimental setup in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [No]
Justification: Code and benchmark would be released after submission and review. does include a new open-source benchmark.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ public/guides/ https//oooeSuuusiinnnnnn
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCodeSubmiis
httpps.ccppubiic//eecc S ssi www.inssssooiiiiiiio sesionnyocy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details of training and testing are specified in Section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments in this paper are conducted in multiple iterations, yielding stable results. We reported the average of these results for consistency and reliability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational cost and resources are specified in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: pettpa:/nneurigee http/puulic/Bthccaiieeeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in this paper conforms in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The Broader Impacts are discussed Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We follow the safeguards of base LLM Vicuna-7B v1.5.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models and datasets used in this paper have been properly credited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Benchmark would be released after submission and review. The paper does provide well-structured details of the dataset.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "mVFqPnsJtr", "reviewer": "Reviewer_et83", "summary": "This paper designs a SlowFocus mechanism to allow Vid-LLM's input signals to combine both high frame rate and low frame rate inputs simultaneously. This addresses the issue of maintaining the effectiveness of input information within a limited context window in LLMs. Low frame rate inputs contain global information, while high frame rate inputs contain local details. Additionally, this paper proposes a Multiple-Frequency Mixing Attention mechanism to better integrate these inputs and a Temporal Relationship Modeling mechanism to preserve temporal relationships in the temporal dimension.", "questions": "1. Why is the visual encoder turned off in all stages? Some recent work also start open the gradient of CLIP encoder, would the results improve in your model if it were turned on?\n\n2. Is the use of LoRA for pre-training due to resource constraints? Would full-rank training result in better performance?", "limitations": "It is recommended to include comparisons with closed-source models such as Gemini Pro, GPT-4V, etc., to provide reference scores for  FineAction-CGR benchmark in Table 1.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The SlowFocus mechanism, which combines high frame rate and low frame rate inputs, is highly reasonable and efficient. \n\n2. The Multiple-Frequency Mixing Attention mechanism and Temporal Relationship Modeling are appropriate technical designs.\n\n3. The proposed three-stage training and corresponding dataset construction are rational. \n   - The use of temporal grounding data for the second stage of training followed by video level instruction fine-tuning data for the third stage is highly instructive.\n\n4. The FineAction-CGR benchmark fills the gap in the community for fine-grained video understanding evaluation.\n\n5. The experiments in this paper are comprehensive, effectively demonstrating the validity of each design point.\n   - The writing is clear and easy to understand.", "weaknesses": "1. Line 126: Using the same letter \"L\" to represent both \"Low\" and \"LLM\" might appear confusing to readers. \n\n2. Table 5: There is some confusion regarding whether this pertains to the low frame rate parts or the high frame rate parts.\n\n3. Line 449: \n   - What is the data ratio processed by GPT-4V and the Video Recaptioner Model? \n   - Do their results show significant distribution differences? \n   - Are you using GPT-4V to generate frame-by-frame videos or video captions?\n\n4. The analysis on the effectiveness and rationality of the FineAction-CGR benchmark evaluation is insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Line 126: Using the same letter \"L\" to represent both \"Low\" and \"LLM\" might appear confusing to readers\n\n**A**: Thanks, we will make a clearer representation in the revision.\n\n**Q2**: Table 5: There is some confusion regarding whether this pertains to the low frame rate parts or the high frame rate parts.\n\n**A**: Thanks, the ``fps`` term mentioned in Table 5 of the main paper refers to the low-frequency part.\nThe high-frequency part maintains a fixed sampling number $N_H$.\nWe will provide a clearer representation in the revised version.\n\n**Q3**: Line 449: 1) What is the data ratio processed by GPT-4V and the Video Recaptioner Model? 2) Do their results show significant distribution differences? 3) Are you using GPT-4V to generate frame-by-frame videos or video captions?\n\n**A**: \n1) GPT-4V and the Video Recaptioner Model are both utilized during the process of generating captions for each video.\nGPT-4V is utilized to generate captions for the entire video by sampling 10 frames uniformly from each video.\nThe Video Recaptioner Model is utilized to generate captions for the segmented video clips by uniformly sampling 8 frames from each clip.\n2) The caption of the entire video generated by GPT-4V tends to summarize coarse-grained and general content.\nWhile the Video Recaptioner Model generates captions for segmented video clips, containing more details and action descriptions.\n3) The video in FineAction may contain multiple objects and it's better not to generate frame-by-frame captions.\nTake the InternVID data set as an example, the frame-by-frame captions are not applicable when preceding objects are inconsistent with following objects.\nTherefore, GPT-4V is utilized to generate a caption by sampling 10 frames uniformly from a video to recognize different objects. This method not only saves cost but also provides global information for the QA generation of downstream tasks.\n\n**Q4**: The analysis on the effectiveness and rationality of the FineAction-CGR benchmark evaluation is insufficient.\n\n**A**: The evaluation protocol and metrics for FineAction-CGR align with widely-accepted benchmarks, such as ActivityNet-QA, ensuring the effectiveness of the benchmark evaluation.\nFor other methods, we adhere to the official implementation guidelines.\nWe will add more discussions and case studies on the benchmark evaluation in the revision.\n\n**Q5**: Why is the visual encoder turned off in all stages? Some recent work also start open the gradient of CLIP encoder, would the results improve in your model if it were turned on?\n\n**A**: Turning off the visual encoder and fine-tuning the multi-modal projector is a common practice among the baselines we compare, such as VideoLLaMA, Video-ChatGPT, and LLaMA-VID.\nTo ensure a fair comparison, we adhere to this approach.\n\nHowever, we acknowledge that turning on the visual encoder could be beneficial, and it would be a promising direction to explore methods for fine-tuning the visual encoder to enhance the unified representation of images and videos.\nNonetheless, this aspect is not the primary focus of our work.\n\n**Q6**: Is the use of LoRA for pre-training due to resource constraints? Would full-rank training result in better performance?\n\n**A**: Yes, we use LoRA to fine-tune the LLM primarily due to resource constraints.\nWe have conducted a comparison with full-rank training, as shown in Table 2 of the main paper, which demonstrates that the performance degradation when using the LoRA approach is minimal.\n\n**Q7**: It is recommended to include comparisons with closed-source models such as Gemini Pro, GPT-4V, etc., to provide reference scores for FineAction-CGR benchmark in Table 1.\n\n**A**: Thanks for the suggestion, we employ GPT-4V to evaluate on the benchmark, and the evaluation of Gemini Pro will be added in the revision.\nThe results are presented in the table below.\nDue to token limitations, each video is sampled at 10 frames, and GPT-4V is tasked with answering questions based on these sampled frames.\nWe observe that GPT-4V performs well on captioning tasks but exhibits suboptimal performance on tasks such as temporal grounding and reasoning, which require fine-grained temporal cues.\n| Method | Temporal grounding (mIoU) | B    | M    | R    | C    | Acc   | Score |\n|--------|---------------------------|------|------|------|------|-------|-------|\n| GPT-4V | 9.28                      | 0.59 | **0.53** | 0.65 | 2.74 | 19.39 | 1.1   |\n| Ours   | **66.68**                     | **0.66** | 0.41 | **0.7**  | **3.27** | **53.1**  | **2.78**  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1gkrKAqq91", "reviewer": "Reviewer_uGD4", "summary": "This works proposes “SlowFocus” to improve the balance between *tokens per frame* and *frames per video* used in Video LLMs. SlowFocus identifies video segments relevant to a given query and samples selected segments at high frequencies. The high frequency tokens are mixed with low frequency global video tokens. Authors propose suitable training strategies to learn the newly introduced layers. A new evaluation benchmark is also proposed by the authors.", "questions": "N/A", "limitations": "Addressed", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Motivation for work is well established\n2. Clear explanation of proposed methodology \n3. Evaluation benchmark contribution\n\nNOTE: Is the proposed dataset released publicly?", "weaknesses": "1. **Motivation not justified:** The idea is to improve video LLM performance at a fixed compute. However, there are no results to evaluate the inference speed of proposed SlowFocus with prior works. Hence, the benefit of this approach over uniform sampling at a higher resolution is unclear.  \n\n2. **Table 1 Unfair:** It appears that prior works are zero-shot while SlowFocus is trained on this FineActionCGR dataset. This makes the comparison unfair.\n\n3. **Table 2 Missing Details:** Is SlowFocus re-trained on this data? How many frames do the prior works use? What is their inference compute requirement in comparison? Also, can a simple VLM + LLM baseline like LLoVi [1] be added for comparison? This will give a better idea on usefulness of proposed method.  \n\n4. **Missing Ablations:** How does this compare against simply feeding the selected high-res frames and the global low-res frames (arranged in temporal order) into an existing Vid-LLM (e.g. LLoVi)? \n\n5. [Minor] Related work [2] using spatial coordinates for same video tasks \n\n\n\n[1] LLoVi: https://arxiv.org/pdf/2312.17235\n\n[2] Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs, CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: Motivation not justified: 1) Evaluate the inference speed. 2) The benefit of this approach over uniform sampling at a higher resolution is unclear.\n\n**A**: The additional cost of inference speed is actually minimal, as the low-frequency visual tokens only need to be encoded once.\nMoreover, the relevant segment grounding is not a time-consuming task, as it only involves processing short sequence lengths.\n\nWe report our inference speed compared to the baselines in the table below, demonstrating that our method results in only a slight increase in inference cost while significantly enhancing fine-grained video understanding.\n| Method    | Inference time (s) | mIoU  | B    | M    | R    | C    | Acc   | Score |\n|-----------|--------------------|-------|------|------|------|------|-------|-------|\n| VTimeLLM  | 1.27               | 27.69 | 0.05 | 0.09 | 0.08 | 0.12 | 9.96  | 0.54  |\n| LLaMA-VID | 1.25               | 0.35  | 0.16 | 0.12 | 0.11 | 0.23 | 15.65 | 0.87  |\n| Ours      | 1.51               | 66.68 | 0.66 | 0.41 | 0.7  | 3.27 | 53.1  | 2.78  |\n\nThe phrase ``at a higher resolution`` may be confusing.\nWe hypothesize that it refers to more tokens per frame, and we compare our approach with uniform sampling that adopts more token numbers per frame, as shown in the table below.\n| Method    | Fps | Tokens per frame | mIoU  | B    | M    | R    | C    | Acc   | Score |\n|-----------|-----|------------------|-------|------|------|------|------|-------|-------|\n| LLaMA-VID | 1   | 64               | 0.35  | 0.16 | 0.12 | 0.11 | 0.23 | 15.65 | 0.87  |\n| LLaMA-VID | 1   | 256              | 0.29  | 0.23 | 0.16 | 0.23 | 0.77 | 18.91 | 1.13  |\n| Ours      | 1   | 64               | 66.68 | 0.66 | 0.41 | 0.7  | 3.27 | 53.1  | 2.78  |\n\nAfter increasing the token numbers per frame $N_V$ to 256, the performance gets a minimal improvement, indicating the limitation of simply increasing frame resolutions.\n\n**Q2**: Table 1 Unfair: SlowFocus is trained on this FineActionCGR dataset.\n\n**A**: FineActionCGR is employed in stage 3's fine-tuning to align the LLM with our mixed-frequency search method.\nWe carefully split the dataset to ensure no overlap in scenes or activities between the training and testing sets.\nDetailed ablation studies, as shown in Table 3 of the main paper (lines 280-284), demonstrate that the observed performance gap is primarily due to proposed MFS approach.\n\nAdditionally, to address the reviewer's concern, we also fine-tune the prior work (LLaMA-VID) using stage 3.\nThe results presented in the table below prove that the improvement achieved by introducing the stage 3 dataset is limited, with the primary gains being attributed to the proposed algorithm.\n| Method    | Stage 3 | mIoU  | B    | M    | R    | C    | Acc   | Score |\n|-----------|---------|-------|------|------|------|------|-------|-------|\n| LLaMA-VID | ×       | 0.35  | 0.16 | 0.12 | 0.11 | 0.23 | 15.65 | 0.87  |\n| LLaMA-VID | √       | 22.38 | 0.23 | 0.2  | 0.37 | 1.03 | 24.81 | 1.26  |\n| Ours      | √       | 66.68 | 0.66 | 0.41 | 0.7  | 3.27 | 53.1  | 2.78  |\n\n**Q3**: Table 2 Missing Details: 1) Is SlowFocus re-trained on this data? 2) How many frames do the prior works use? What is their inference compute requirement in comparison? 3) Also, can a simple VLM + LLM baseline like LLoVi be added for comparison?\n\n**A**:\n1) No, MSVD-QA, MSRVTT-QA, and video-based generative performance benchmarks are zero-shot benchmarks.\nOur method is not re-trained on these datasets.\n\n2) To address the reviewer's concern, we have updated Table 2 of the main paper to include more implementation details, such as the number of frames and the computational requirements for inference, as detailed in the table below.\n\n3) For a fair comparison, we also implement LLoVi using LLaVA-1.5 as the video captioner and Vicuna-7B as the LLM, also shown in the table below.\n| Method        | Sampling strategy | Tokens per frame | ActivityNet-QA (Acc) |\n|---------------|-------------------|------------------|----------------------|\n| FrozenBiLM    | N_L=10            | 256              | 24.7                 |\n| VideoLLaMA    | N_L=8             | 256              | 12.4                 |\n| VideoChat     | Fps=1             | 256              | 26.5                 |\n| Video-ChatGPT | N_L=100           | 576              | 35.2                 |\n| LLaMA-VID     | Fps=1             | 64               | 47.4                 |\n| LLoVi         | Fps=1             | 256              | 45.2                 |\n| Ours          | Fps=1             | 64               | 48.4                 |\n\n**Q4**: Missing Ablations: Compare against simply feeding the selected high-res frames and the global low-res frames into an existing Vid-LLM (e.g. LLoVi).\n\n**A**: First, we clarify that LLoVi is not a multi-modal Video-LLM, but rather a language-based combination of a visual captioner and a LLM.\n\nSecond, our paper does not introduce the concept of high-res and low-res frames. We hypothesis these refer to high and low frequency.\nIt would be unfair to directly compare our method to one that feeds selected high-frequency frames into LLoVi, as this selection process inherently reveals the ground-truth.\n\nTo ensure a fair comparison under this setting, we feed LLoVi (same implementation as **Q3**) with the video frames sampled at the same frequency, as detailed in the table below.\nThe results demonstrate that although the video captioner provides a substantial amount of language descriptions to the LLM (this process is quite time-consuming and less practical), the video captioner may still miss important temporal details, leading to potential errors in subsequent LLM inference.\n| Method | mIoU  | B    | M    | R    | C    | Acc   | Score |\n|--------|-------|------|------|------|------|-------|-------|\n| LLoVi  | 10.83 | 0.4  | 0.27 | 0.43 | 0.94 | 32.27 | 1.61  |\n| Ours   | 66.68 | 0.66 | 0.41 | 0.7  | 3.27 | 53.1  | 2.78  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QtOQxdnIts", "reviewer": "Reviewer_6ATY", "summary": "This paper focuses on fine-grained video understanding with large language models. Current works suffer from the dilemma between the per-frame token number and temporal sampling frequency to maintain an acceptable sequence length into the language model. The authors propose to sample a global view with low sampling frequency to perceive the temporal positions that are related to the question, then sample the frames within the predicted temporal window with high frequency for fine-grained video perception. The proposed dataset FineAction-CGR can well reflect the ability in temporal grounding and fine-grained question answering.", "questions": "How do you condense each frame into $N_v$ tokens?", "limitations": "The authors have discussed the limitations on the resolution.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The exploration in the per-frame token number and temporal sampling frequency for video understanding is important and fundamental.\n2. The authors propose to first perceive the related temporal window and then densely sample the frames within the temporal span. The architecture makes sense and is suitable for fine-grained video captioning and question answering while maintaining computation efficiency.\n3. The dataset can evaluate the temporal grounding and fine-grained video understanding ability and could make contribution to the progress in video understanding with large language models.", "weaknesses": "1. How is the duration of the videos for temporal grounding evaluation? Will the low frequency sampling result in the $h_L$ to lose much information in excessively long videos?\n2. In Fig.2, the temporal grounding derives from the low frequency sampled video features. Why the high frequency sampling also has impact on the temporal grounding performance?\n3. The architecture of the temporal encoder is not illustrated.\n4. In Table 5, what does Fps=64 mean? The video dataset usually contains videos with around 30 fps, how does 64 fps work?\n5. The proposed method presents overwhelming advantage on the proposed FineAction-CGR dataset while the improvement on other benchmarks are marginal. I understand it is because the rest are comparatively easy and of coarse-grained. However, it is better to provide the evaluation on other challenging video benchmarks like long video datasets, e.g., EgoSchema [1], MovieChat [2] to validate the effectiveness of the temporal grounding in long sequences for more comprehensive comparison.\n\n[1] Mangalam, Karttikeya, Raiymbek Akshulakov, and Jitendra Malik. \"Egoschema: A diagnostic benchmark for very long-form video language understanding.\" Advances in Neural Information Processing Systems 36 (2024).\n[2] Song, Enxin, et al. \"Moviechat: From dense token to sparse memory for long video understanding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: How is the duration of the videos for temporal grounding evaluation? Will the low frequency sampling result in the to lose much information in excessively long videos?\n\n**A**: The duration of temporal grounding tasks typically ranges from 1 to 10 seconds.\nTo explore this question, we carry out ablations on low frequency sampling (fps) in the table below.\nExperimental results demonstrate that fps value at 1 is sufficient to perform temporal grounding task.\n| Low frequency | mIoU  | R@0.3 | R@0.5 | R@0.7 | Acc   | Score |\n|---------------|-------|-------|-------|-------|-------|-------|\n| Fps=0.5       | 63.47 | 81.71 | 69.44 | 53.27 | 50.13 | 2.61  |\n| Fps=1         | 66.68 | 85.8  | 73.01 | 56.25 | 53.1  | 2.78  |\n| Fps=2         | 67.2  | 86.28 | 73.29 | 56.71 | 54.19 | 2.83  |\n\n**Q2**: In Fig.2, the temporal grounding derives from the low frequency sampled video features. Why the high frequency sampling also has impact on the temporal grounding performance?\n\n**A**: Actually, when evaluated on temporal grounding tasks, the mixed-frequency sampling algorithm is also applied.\nThis is why high-frequency sampling impacts the temporal grounding performance.\nWe have also conducted an ablation study on the effect of low and high-frequency features when mixed-frequency sampling is turned off.\nThe results presented in the table below demonstrate that under this setting, the high-frequency sampling does not influence the performance.\n| Low frequency | High frequency | mIoU  | R@0.3 | R@0.5 | R@0.7 | Acc   | Score |\n|---------------|----------------|-------|-------|-------|-------|-------|-------|\n| Fps=1         | 10             | 41.27 | 52.88 | 43.71 | 31.89 | 33.61 | 1.69  |\n| Fps=1         | 20             | 41.52 | 52.06 | 44.18 | 31.64 | 33.55 | 1.68  |\n| Fps=0.5       | 10             | 38.05 | 48.25 | 40.65 | 29.11 | 31.23 | 1.55  |\n| Fps=2         | 10             | 42.46 | 55.13 | 46.61 | 31.74 | 34.07 | 1.69  |\n\n**Q3**: The architecture of the temporal encoder is not illustrated.\n\n**A**: We have described the architecture of the temporal encoder in the main paper (line 171-177), which consists of several learnable queries corresponding to different relative sequential positions.\nWe will illustrate this in Figure 2 in the revised version.\n\n**Q4**: In Table 5, what does Fps=64 mean? The video dataset usually contains videos with around 30 fps, how does 64 fps work?\n\n**A**: FineAction is an action-centric dataset that includes some videos recorded at high fps (over 64).\nFor videos with a fps of 64 or lower, we dynamically adjust the sampling frequency to match the original fps, which means all frames are used as input.\nAdditionally, we will clarify the term ``fps`` by referring to it as ``max fps`` in the revised version for better clarity.\n\n**Q5**: The proposed method presents overwhelming advantage on the proposed FineAction-CGR dataset while the improvement on other benchmarks are marginal. I understand it is because the rest are comparatively easy and of coarse-grained. However, it is better to provide the evaluation on other challenging video benchmarks like long video datasets, e.g., EgoSchema [1], MovieChat [2] to validate the effectiveness of the temporal grounding in long sequences for more comprehensive comparison.\n\n**A**: To address the reviewer's concern, we evaluate our method on MovieChat-1K, as detailed in the table below.\nThe results show that although our method is not specifically trained on long video benchmarks (in contrast, MovieChat has undergone targeted training for long videos), it still achieves competitive results.\n| Method        | LLM       | LoRA | Global mode (Acc) | Global mode (Score) | Breakpoint mode (Acc) | Breakpoint mode (Score) |\n|---------------|-----------|------|-------------------|---------------------|-----------------------|-------------------------|\n| VideoChat     | Vicuna-7B | ×    | 57.8              | 3                   | 46.1                  | 2.29                    |\n| VideoLLaMA    | Vicuna-7B | ×    | 51.7              | 2.67                | 39.1                  | 2.04                    |\n| Video-ChatGPT | Vicuna-7B | ×    | 47.6              | 2.55                | 48                    | 2.45                    |\n| MovieChat     | Vicuna-7B | ×    | **62.3**              | **3.23**                | **48.3**                  | **2.57**                    |\n| Ours          | Vicuna-7B | √    | *58.6*              | *3.14*                | *48.1*                  | *2.53*                    |\n\n**Q6**: How do you condense each frame into Nv tokens?\n\n**A**: We condense the features of each frame using spatial grid pooling.\nSpecifically, after passing through the visual encoder, each frame is represented by $16 \\times 16 = 256$ tokens.\nThese tokens are then further condensed into a $n \\times n$ grid, resulting in $n^2$ tokens, depending on the experimental setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B7sPMnTXlm", "reviewer": "Reviewer_bGgU", "summary": "This paper focuses on the video LLMs (Vid-LLMs) area and attempts to improve the fine-grained video understanding. The authors design modules and training data to identify relevant temporal segments based on the given query and then generate outputs based on the identified segments.", "questions": "See weaknesses above.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the writing is well-organized and easy to follow. The strengths can be summarized as follows:\n1. This work introduces a two-stage inference strategy into the Vid-LLMs area, following the coarse-to-fine mode. Such a strategy provides a new perspective for enhancing video understanding.\n2. This work proposes a new benchmark FineAction-CGR to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks, which provides a new perspective to evaluate Vid-LLMs' capabilities.", "weaknesses": "There are some weaknesses listed as follows:\n\n1. Although the proposed coarse-to-fine reasoning approach is relatively novel in Vid-LLMs, I have concerns about its practicality. Firstly, this reasoning method nearly doubles the inference cost for the same question compared to other methods. Secondly, the accuracy of the final response heavily relies on the accuracy of event localization in the first stage. According to the benchmark validation provided by the authors, even when trained on proprietary data, SlowFocus only barely meets the passing mark in temporal grounding mIoU. This significantly impacts the accuracy of temporal reasoning in the second stage.\n\n2. The authors chose relatively weak baselines and attempted to claim the superiority of their results. I suggest comparing their method with MovieChat [1] and VideoChat2 [2] to substantiate their claims of advancement. Moreover, despite using more data than LLaMA-VID, their performance on various benchmarks is not significantly better than LLaMA-VID, raising questions about the effectiveness of this two-stage approach.\n\n3. The authors need to validate the effectiveness of their two-stage model on benchmarks involving long videos, which is crucial to substantiate their claims. For instance, they could use benchmarks like EgoSchema and MovieChat-1K for this purpose.\n\n\n[1] MovieChat: From Dense Token to Sparse Memory for Long Video Understanding. CVPR 2024\n\n[2] MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: I have concerns about its practicality. Firstly, this reasoning method nearly doubles the inference cost for the same question compared to other methods. Secondly, the accuracy of the final response heavily relies on the accuracy of event localization in the first stage. According to the benchmark validation provided by the authors, even when trained on proprietary data, SlowFocus only barely meets the passing mark in temporal grounding mIoU. This significantly impacts the accuracy of temporal reasoning in the second stage.\n\n**A**: \n1) The additional cost is actually minimal, as the low-frequency visual tokens only need to be encoded once.\nMoreover, the relevant segment grounding is not a time-consuming task, as it only involves processing short sequence lengths.\nWe also report our inference speed compared with baselines in the table below, showing that our method results in only a small increase in inference cost while significantly improving fine-grained video understanding.\n| Method    | LoRA | Inference time (s) | mIoU  | B    | M    | R    | C    | Acc   | Score |\n|-----------|------|--------------------|-------|------|------|------|------|-------|-------|\n| VTimeLLM  | √    | 1.27               | 27.69 | 0.05 | 0.09 | 0.08 | 0.12 | 9.96  | 0.54  |\n| LLaMA-VID | ×    | 1.25               | 0.35  | 0.16 | 0.12 | 0.11 | 0.23 | 15.65 | 0.87  |\n| Ours      | √    | 1.51               | 66.68 | 0.66 | 0.41 | 0.7  | 3.27 | 53.1  | 2.78  |\n\n2) The temporal grounding task on FineActionCGR is challenging, as the ground truth often occurs within a very short time.\nTherefore, the current mIoU of 66.68 is actually sufficiently practical.\nWe have also conducted experiments demonstrating that the current temporal grounding capability has reached saturation and is not the main limiting factor for further performance improvement, as detailed in the table below.\n| Stage 2 | Temporal grounding (mIoU) | B    | M    | R    | C    | Acc   | Score |\n|---------|---------------------------|------|------|------|------|-------|-------|\n| 0.1     | 25.17                     | 0.24 | 0.21 | 0.19 | 0.86 | 24.88 | 1.33  |\n| 0.25    | 47.83                     | 0.49 | 0.38 | 0.42 | 2.12 | 44.73 | 2.28  |\n| 0.5     | 61.29                     | 0.64 | 0.43 | 0.65 | 3.05 | 52.59 | 2.72  |\n| 1       | 66.68                     | 0.66 | 0.41 | 0.7  | 3.27 | 53.1  | 2.78  |\nThe column ``stage 2`` means the number of epochs for stage 2's fine-tuning.\n\n**Q2**: The authors chose relatively weak baselines and attempted to claim the superiority of their results. I suggest comparing their method with MovieChat and VideoChat2 to substantiate their claims of advancement. Moreover, despite using more data than LLaMA-VID, their performance on various benchmarks is not significantly better than LLaMA-VID, raising questions about the effectiveness of this approach.\n\n**A**: Thank you for the feedback.\nWe have added the comparisons with MovieChat and VideoChat2 in the table below.\n| Method     | LLM       | LoRA | Temporal grounding (mIoU) | B    | M    | R    | C    | Acc   | Score |\n|------------|-----------|------|---------------------------|------|------|------|------|-------|-------|\n| MovieChat  | Vicuna-7B | ×    | 1.87                      | 0.23 | 0.11 | 0.15 | 0.42 | 20.81 | 1.21  |\n| VideoChat2 | Vicuna-7B | ×    | 0.28                      | 0.25 | 0.16 | 0.21 | 0.37 | 18.26 | 1.1   |\n| Ours       | Vicuna-7B | √    | 66.68                     | 0.66 | 0.41 | 0.7  | 3.27 | 53.1  | 2.78  |\n\nWe have also fine-tuned prior work using the same dataset, as detailed in the table below.\nThe results prove that the improvement achieved by introducing the stage 3 dataset is limited, with the primary gains being attributed to the proposed algorithm.\nCompared to prior work, our approach requires only a limited amount of additional data to adapt to the MFS algorithm and achieve overwhelming advantages on fine-grained benchmarks.\n| Method    | LLM       | LoRA | Stage 3 | Temporal grounding (mIoU) | B    | M    | R    | C    | Acc   | Score |\n|-----------|-----------|------|---------|---------------------------|------|------|------|------|-------|-------|\n| LLaMA-VID | Vicuna-7B | ×    | ×       | 0.35                      | 0.16 | 0.12 | 0.11 | 0.23 | 15.65 | 0.87  |\n| LLaMA-VID | Vicuna-7B | √    | √       | 22.38                     | 0.23 | 0.2  | 0.37 | 1.03 | 24.81 | 1.26  |\n| Ours      | Vicuna-7B | √    | √       | 66.68                     | 0.66 | 0.41 | 0.7  | 3.27 | 53.1  | 2.78  |\n\n**Q3**: Need to validate the effectiveness of their two-stage model on benchmarks involving long videos, which is crucial to substantiate their claims. For instance, they could use benchmarks like EgoSchema and MovieChat-1K.\n\n**A**: To address the reviewer's concern, we evaluate our method on MovieChat-1K, as detailed in the table below.\nThe results show that although our method is not specifically trained on long video benchmarks (in contrast, MovieChat has undergone targeted training for long videos), it still achieves competitive results.\n| Method        | LLM       | LoRA | Global mode (Acc) | Global mode (Score) | Breakpoint mode (Acc) | Breakpoint mode (Score) |\n|---------------|-----------|------|-------------------|---------------------|-----------------------|-------------------------|\n| VideoChat     | Vicuna-7B | ×    | 57.8              | 3                   | 46.1                  | 2.29                    |\n| VideoLLaMA    | Vicuna-7B | ×    | 51.7              | 2.67                | 39.1                  | 2.04                    |\n| Video-ChatGPT | Vicuna-7B | ×    | 47.6              | 2.55                | 48                    | 2.45                    |\n| MovieChat     | Vicuna-7B | ×    | 62.3              | 3.23                | 48.3                  | 2.57                    |\n| Ours          | Vicuna-7B | √    | 58.6              | 3.14                | 48.1                  | 2.53                    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for the valuable feedback, with consistent recognition for the motivation and innovation.
We are pleased that the reviewers recognized the strengths of our paper:

* This paper is well motivated (**et83**, **uGD4**, **6ATY**, **bGgU**).
* The designed architecture makes sense and is suitable for fine-grained video understanding (**et83**, **6ATY**).
* Evaluation benchmark contribution (**et83**, **uGD4**, **6ATY**, **bGgU**).
* The writing is well-organized and easy to follow (**et83**, **uGD4**, **bGgU**).

The following is our response to the major concerns raised by the reviewers, supported by further empirical investigations with each respective review:

* **Inference time details**: Both reviewers **uGD4** and **bGgU** express concerns about the inference time of our method.
To address these concerns, we evaluate the inference latency of our approach and observe only a modest increase in inference cost (20\%), while achieving significant improvements in fine-grained video understanding.
The primary reason for this efficiency is that low-frequency visual tokens only need to be encoded once, and the relevant segment grounding is not a time-consuming task, as it only involves processing short sequence lengths.

* **Additional comparisons with more advanced counterparts**: In response to the suggestions from reviewers **uGD4** and **bGgU**, we conduct further experiments comparing our method with the suggested methods (LLoVi, MovieChat, and VideoChat2).
Our results indicate that our method continues to achieve superior performance in fine-grained video understanding.

* **Additional experiments on long video benchmark**: Both reviewers **6ATY** and **bGgU** highlight the necessity of evaluating on long video benchmarks.
In response, we evaluate our method on the MovieChat-1K dataset.
The results demonstrate that, even though our method is not specifically trained on long video data, it still achieves competitive results.

We then address the specific concerns raised by the reviewers in our individual rebuttal responses.

Overall, we look forward to engaging in further fruitful discussions in the coming weeks to enhance our work.

Meta-review: For video-language modeling, this work aims to address the challenges of retaining high-quality frame-level semantic information (i.e., a sufficient number of tokens per frame) and comprehensive video-level temporal information (i.e., an adequate number of sampled frames per video) for video representation.  SlowFocus is designed by identifying the query-related temporal segment based on the posed question, then performing dense sampling on this segment to extract local high-frequency features. A fine-grained video benchmark was provided. 

Reviewers have appreciation for the motivation, reasonable designed architecture, contributed evaluation benchmark, and writing, while raising concerns on lack of evaluation on long video benchmarks, inference time comparision, weak baselines, etc. The rebuttal had addressed the concerns from Reviewer et83, uGD4, 6ATY, where the reviewer uGD4 had raised the score from 3 to 6. Reviewer bGgU expected result comparisons on long video benchmarks EgoSchema and Video-MME to see how SlowFocus performs. The authors provided results on EgoSchema which shows superior performance with the compared methods. 

Considering the positive feedback from most reviewers, AC recommends acceptance but agrees with the reviewers and strongly suggests the authors to include the comparison on long video benchmarks and incorporate the suggestions of reviewers for comprehensive and clarity.
Decision: Accept (poster)