Title: SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM
Venue: NeurIPS 2024
Accepted: yes
Keywords: large language model

video understanding
Abstract: Large language models (LLMs) have demonstrated exceptional capabilities in text understanding, which has paved the way for their expansion into video LLMs (Vid-LLMs) to analyze video data. However, current Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information (i.e., a sufficient number of tokens per frame) and comprehensive video-level temporal information (i.e., an adequate number of sampled frames per video). This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding. To address this issue, we introduce the SlowFocus mechanism, which significantly enhances the equivalent sampling frequency without compromising the quality of frame-level visual tokens. SlowFocus begins by identifying the query-related temporal segment based on the posed question, then performs dense sampling on this segment to extract local high-frequency features. A multi-frequency mixing attention module is further leveraged to aggregate these local high-frequency details with global low-frequency contexts for enhanced temporal comprehension. Additionally, to tailor Vid-LLMs to this innovative mechanism, we introduce a set of training strategies aimed at bolstering both temporal grounding and detailed temporal reasoning capabilities. Furthermore, we establish FineAction-CGR, a benchmark specifically devised to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks. Comprehensive experiments demonstrate the superiority of our mechanism across both existing public video understanding benchmarks and our proposed FineAction-CGR.

Ming Nie

Abstract

Large language models (LLMs) have demonstrated exceptional capabilities in text understanding, which has paved the way for their expansion into video LLMs (Vid-LLMs) to analyze video data. However, current Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information ( i.e. , a sufficient number of tokens per frame) and comprehensive video-level temporal information ( i.e. , an adequate number of sampled frames per video). This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding. To address this issue, we introduce the SlowFocus mechanism, which significantly enhances the equivalent sampling frequency without compromising the quality of frame-level visual tokens. SlowFocus begins by identifying the query-related temporal segment based on the posed question, then performs dense sampling on this segment to extract local high-frequency features. A multi-frequency mixing attention module is further leveraged to aggregate these local high-frequency details with global low-frequency contexts for enhanced temporal comprehension. Additionally, to tailor Vid-LLMs to this innovative mechanism, we introduce a set of training strategies aimed at bolstering both temporal grounding and detailed temporal reasoning capabilities. Furthermore, we establish FineAction-CGR, a benchmark specifically devised to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks. Comprehensive experiments demonstrate the superiority of our mechanism across both existing public video understanding benchmarks and our proposed FineAction-CGR.

1 Introduction

While Vid-LLMs have demonstrated the promising performance in video understanding, they still face a significant challenge. To embed video features into LLMs under computing resource constraints, Vid-LLMs typically need to sparsely sample the original video ( e.g. , retaining one frame every second) into a collection of low-frequency frames. Subsequently, the token number of each frame are also compressed through a visual adapter like average pooling [16] or Q-former [14]. This leads to a dilemma: Vid-LLMs have to choose between compromised frame-level features and video-level features, each resulting in a reduction of video information. As illustrated in Figure 1 (a), under the premise of a constant total token number, we investigate the performance of Vid-LLMs by

What does the women do after introducing product?

The woman is applying lipstick.

The relevant temporal segment

is from 22 to 28:

The woman opens the box and takes some makeup.

(b)

5

10

15

20

25

30

35

Temporal frequency

/

Frame token num.

Accuracy on FineAction-CGR

-3

-2

-1

0

1

2

3

4

5

256

4096

1024

(a)

t

0

10

20

30

40

50

60

(Input frames: 0, 10, 20, …, 60)

(Input frames: 0, 10, 20, …, 60)

Ours

(Additional input frames: 22, 23, …, 28)

×

√

Total token numbers per video

Query-related Frames Missing !

LLaMA-VID

dynamically adjusting the sampling frequency and frame token number. The results clearly show that both low sampling frequency and low frame token number lead to performance degradation. Low-frequency sampling results in a sparse image collection as input, omitting crucial temporal details. On the other hand, excessive frame-feature compression degrades the semantic and spatial contexts of each frame. When confronted with fine-grained video understanding tasks, this limitation becomes significantly evident, as depicted in Figure 1 (b). Existing models ( e.g. , LLaMA-VID [17]) often overlook crucial details early in the input due to low-frequency sampling, leading to inaccuracy.

To address this challenge, we introduce SlowFocus, which is designed to pinpoint relevant temporal segments in response to questions, and subsequently maintains high-quality temporal details to enrich fine-grained video comprehension. For video understanding tasks, we assume that the relevant details are concentrated within one or several clips. SlowFocus initially identifies these segments based on the provided questions. It then densely samples the segmented temporal clips at a high-frequency to extract local temporal features highly pertinent to the questions. To effectively model the temporal relationships between frames and capture inter-frame contexts, we propose a specialized temporal encoder and a multi-frequency mixing attention module for enhanced temporal comprehension.

To enhance Vid-LLMs' ability to perform SlowFocus based on mixed frequencies, we propose a set of training and inference strategies to improve their temporal localization and fine-grained temporal reasoning capability. Following VTimeLLM [10], we fine-tune our Vid-LLM in the second stage on dense video captioning and temporal grounding tasks. This process enhances the model's ability to predict discrete bins that define the relevant temporal segments. In the third stage, we adapt the Vid-LLM to the SlowFocus mechanism for high-quality, fine-grained temporal-related tasks, which enables our model to reason precisely based on high-frequency temporal details.

In addition to the scarcity of methods for fine-grained video understanding, existing benchmarks [37, 40] also fall short in providing adequate challenges for specific temporal-related tasks. To bridge this gap and evaluate our proposed framework, we introduce FineAction-CGR, a newly dedicated benchmark that focuses on fine-grained video understanding, especially reasoning tasks based on temporal details. Our method demonstrates superior performance on the proposed benchmark, offering a promising solution to high-quality video understanding.

The contributions of this paper are summarized as follows: (i) We introduce SlowFocus, a straightforward yet effective framework designed to resolve the prevalent trade-off in existing Vid-LLMs between capturing limited frame-level details and overarching video-level contexts. SlowFocus adeptly maintains high-frequency local details alongside low-frequency global contexts, facilitating the identification of pertinent temporal segments and precise reasoning on video contents. (ii) We present a novel training strategy specifically designed to enhance the temporal localization abilities of Vid-LLMs, and seamlessly adapt them to our newly proposed SlowFocus approach. (iii) We establish a comprehensive new benchmark and carry out extensive experiments to rigorously assess the fine-grained video understanding capabilities of Vid-LLMs. The empirical evidence strongly

indicates that our SlowFocus approach significantly outperforms existing models, particularly in tasks requiring detailed temporal understanding and reasoning within videos.

2 Related works

Vision large language models. Researchers have made significant efforts to enable Large Language Models(LLMs) to comprehend visual information. BLIP-2 [14] aligns vision-language representation with a lightweight Querying Transformer by concept of Q-Former. MiniGPT-4 [45] aligns detailed image descriptions with advanced LLM which significantly enhances its multi-modal abilities. Exploring diverse multi-modal instruction-following data, LLaVA [20] has demonstrated impressive multi-model chat abilities. Recent LLMs, like Kosmos-2 [26] and VisionLLM [33], probed into more specific aspects of image comprehension, including referring and grounding [27], remarkably enhancing the capability to describe intricate image details.

Video large language models. The exploration of LLMs' potential has been extended from images to videos, which contributes to emergence of Video LLMs. VideoChat [15] combines fine-tuning and LLM-based video agents and is fine-tuned using a specially designed video-centric instruction dataset. Video-ChatGPT [23] proposes a novel human assisted and semi-automatic annotation framework for generation high quality instruction data for videos. Video-LLaMA [42], trained on vision-language branch and audio-language branch separately with same visual data and process, has demonstrated impressive abilities in understanding both visual and auditory content in videos. Video-LLaV A [18] learns united visual representation by alignment before projection and conducts joint training on images and videos simultaneously. With efforts, Video LLMs have exhibited powerful task-handling capabilities in downstream tasks, like text-video retrieval and video captioning. However, these models remain limited in ability to comprehend fine-grained content. To solve this problem, we introduce a model with powerful capability of fine-grained video understanding.

Fine-grained video understanding. Comprehending videos in fine-grained aspects requires precisely locating and understanding specific events within a video. It is roughly divided by previous works [10] into temporal grounding [2, 7] and dense video captioning [13, 31] tasks. Temporal grounding demands the model to precisely identify start and end timestamps of video segment according to a given text query. Dense video captioning requires both temporal localization and captioning for all events within an untrimmed video. Both tasks are constrained by the labor-intensive nature of annotation, resulting in relatively small datasets. Moreover, fine-grained video understanding demands a deep comprehension of how objects change and interact over time, particularly at a detailed level. This complexity necessitates temporal reasoning tasks to effectively analyze these dynamics. However, current research in this field falls short of adequately addressing these requirements. To solve this problem, our proposed large-scale dataset provides mass temporal annotations and captions in different dimensions, supplying a wealth of information for training in temporal video grounding and reasoning tasks.

3 Method

In this section, we begin with a preliminary overview of video LLMs (Vid-LLMs) in Section 3.1. Following that, we offer a comprehensive explanation of our proposed SlowFocus in Section 3.2, followed by a detailed introduction to our innovative training strategy in Section 3.3.

3.1 Preliminary

Contemporary Vid-LLMs typically feature a modular architecture, which includes a visual encoder E V , a series of visual adapters Q , and a large language model L . For a given video V = { V ( t ) ∈ R H × W × 3 | t = 0 , ..., T } that consists of T frames, along with its associated question q , Vid-LLMs generally perform downsampling on the original video V at a fixed interval M L . This process results in sparsely distributed frames V L :

where M L ≫ 1 . Consequently, only a very limited number of frames ( T/M L ≪ T ) are selected as the actual input for the Vid-LLM, a technique we refer to as low-frequency sampling in this study.

High-Frequency

Sampling

Please locate the

temporal segment

help to reason the

<user query>.

What does the

woman do after

introducing product?

Temporal Encoder

Visual Encoder

LLM

From

023

to

035

.

<

user query

>.

Additional temporal

clues to focus on:

from 023 to 035.

Temporal Encoder

Visual Encoder

The woman

opens the

box and

takes some

makeup.

LLM

MMA

User Query

Video

t

1

10

20

30

40

50

60

Stage 1: Relevant segment grounding

t

23

25

27

29

31

33

35

Stage 2: Mixed-Frequency Sampling

Low-freq

High-freq

Low-freq

Temporal token

:

Low-freq

visual token

:

:

High-freq

visual token

Subsequently, the visual encoder processes the downsampled frames V L and encodes them into a series of visual tokens denoted as z L = E V ( V L ) . These visual tokens are then transformed to align with the embedding space of the language model through the visual adapter Q , resulting in aligned visual tokens h L = Q ( z L ) . Concurrently, the input text query q is encoded into linguistic tokens h q by the textual encoder. These visual and text tokens are concatenated into a unified sequence [ h L , h q ] , which then serves as the input for the decoder component of the large language model L . The model leverages this combined representation to infer the appropriate answer ans = L ([ h L , h q ]) , showcasing its ability to perform cross-modal reasoning and respond to human queries.

Although this paradigm is well developed in the field of video understanding, it encounters significant limitations in tasks requiring fine-grained temporal reasoning. By retaining only a small, discontinuous subset of frames ( t , where t ≪ T ), substantial information loss can occur, as depicted in Figure 1 (b). Fine-grained temporal video understanding demands that the model concentrate on one or more specific temporal intervals, which could be potentially brief. However, the low-frequency sampling method predominantly captures overarching information while neglecting crucial local details, leading to inaccuracies such as incorrect responses or hallucinations. To address these shortcomings, we develop the SlowFocus strategy.

3.2 SlowFocus

To improve Vid-LLMs with the SlowFocus mechanism, we first expand the traditional training and inference paradigm by introducing Mixed-Frequency Sampling (MFS). Subsequently, we explicitly model the temporal relationships among the sampled frames. Finally, we enhance the visual tokens with our newly proposed multiple-frequency mixing attention, designed to capture long-term contexts. We now provide a detailed elaboration on each of these components.

Relevant segment grounding. To better mimic human cognition in our Vid-LLM, we transform its inference paradigm into a multi-round dialogue format that integrates both the query and temporal awareness elements. The methodology of this transformation is illustrated in Figure 2. Initially, we sample the entire video at a fixed interval to capture the low-frequency frames V L , which are then encoded into visual embeddings h L . Following that, we reformat the original question q into temporal grounding questions q 1 , such as ' <video> \ nPlease provide the temporal segment help to reason the question: <question> ', where <question> refers to the original question q .

LLM

Visual

Encoder

Temporal

Encoder

Projectors

LLM

Visual

Encoder

Temporal

Encoder

Projectors

LLM

Visual

Encoder

Temporal

Encoder

Projectors

MMA

User: <image>

Assistant: <caption>

User: <video>

Assistant: <caption>

User: <video>\n<prompt>

Assistant: <answer>

User: <video>\n<prompt>

Assistant: <segment>

User: <video>\n<time>

Assistant: <caption>

User:

<video>\n<prompt>.

Additional temporal

clues to focus on:

<segment>

Assistant: <answer>

Stage 1: Modality Alignment

Stage 2: Boundary Enhancement

Stage 3: SlowFocus Adaption

232K

558K

98K

140K

100K

Model

Data

Model

Data

Model

Data

With the augmented question q 1 , we direct the Vid-LLM to identify the relevant temporal segments τ within the video that are pertinent to the original question q . This is accomplished by providing the model with low-frequency frames containing global information:

where h q 1 represents the textual embedding of question q 1 and τ ⊂ [0 , T ] . The identified segment τ is designed to encompass query-related details that are crucial for addressing the question q .

Mixed-frequency sampling. Subsequently, we perform dense sampling on the identified segment τ to obtain high-frequency frames V H :

We dynamically adjust the sampling interval M H based on the temporal length of segment | τ | and to ensure an adequate number of samples N H are taken from the relevant segment. The densely sampled frames V H are then encoded into high-frequency visual tokens h H by visual encoder.

With the inclusion of local details from high-frequency frames, we augment the initial question q with prompts, such as ' Additional temporal clues to focus on: ... ', to form q 2 , as illustrated in Figure 2. We then combine the mixed-frequency visual tokens to predict the final answer:

where π is the multiple-frequency mixing attention, which will be detailed subsequently.

Multiple-frequency mixing attention. Because of the roughness of the low-frequency and the detailed specificity of the high-frequency visual features, simply concatenating these two directly into the language model may not yield optimal results. Fine-grained temporal understanding often requires modeling relationships between multiple events, necessitating the integration of global contexts into high-frequency local features. From this perspective, we introduce the Multiple-frequency Mixing Attention (MMA):

where, for simplicity, we omit the process of applying FFN to h L and h H . The resulting output is then fed into LLM to predict the textual response, as described in Equation 4.

Temporal relationship modeling. While LLM can implicitly capture temporal relationships among input visual embeddings based on their sequential positioning, it faces difficulties when the visual tokens are not uniformly distributed along the timeline. To address this challenge, we propose a temporal encoder that encodes the relative positions of frames into a set of discretized temporal

Step I: Video Preprocessing

Video

Clips & segments

split

specific strategy

Step II: Annotation Construction

Step III: Formation of Downstream Task

Video Recaptioner

Instructional Video. An individual

applies teal eyeliner to another

person's eyes in a close-up shot

against a softly lit backdrop.

Ground Truth

New Annotations

Time

segments

Q

: Can you tell me what the person is doing from

40

s to

45

sin this instructional video?

A

: The person is applying makeup on eyebrows.

Q

: When does the person in the video first apply

eyebrows?

A

: from

40

s to

50

s

Q

: How many times does the person paint eye

shadow from

63

s to

72.33

sin the video?

A

: 3

Captioning

Temporal Gournding

Temporal Reasoning

Multi-turn QA

Q1

: Can you briefly describe the video's content?

A1

: The video showcases a detailed makeup

application process, in which a person applies

eyeliner, eyebrow makeup, eyeshadow, mascara,

and lipstick on another person's face with precision.

Q2

: What is the individual doing from

125

s to

134

s?

A2

: During this time, the person is applying vibrant,

eye-catching makeup to the face that involves

painting eye shadow.

…

…

clip4

video

: v_

0000

xxxx.mp4

duration

: xxx

summary

: A person applies eyeliner

with precision on another person's

face ..

action_list

: [xxxxx, ..]

meta

:

[..

{clipid: 4,

clip_caption

: An individual applies

teal eyeliner to another person's

eyes ..

clip_action

: [apply eyeliner]

time_segment

: [xx,xx]

annotations: [

{segment: [xx, xx],  label: xx} ..]}

..]

filename

:

v_

0000

xxxx.mp4

duration

: xxx,

annotations

: [

{segment: [xx, xx]

label: "apply

eyebrows"} .. ]

Clip

Captions

generate with specific prompts

build comprehensive annotations

…

1.84 - 50.08

50.08 - 60.36

60.36 - 66.33

66.33 - 79.4

147.08 - 156.52

…

tokens E ∈ R N × C , where N represents the discrete temporal space. For frames sampled at multiple frequencies V ( t ) , the corresponding temporal token is ϵ i , where i = ⌊ N ∗ t/T ⌋ . These temporal embeddings are then incorporated into the visual tokens by directly adding them to the frame features.

3.3 Training strategy

Considering training efficiency, we introduce a novel training procedure which is structured into three distinct phases in this work: (i) pre-training for modality alignment, (ii) fine-tuning for enhancing temporal grounding, and (iii) SlowFocus adaption, as illustrated in Figure 3. We now provide detailed elaborations on the training strategies and datasets employed for each of these stages.

Modality alignment. In the pre-training stage, our primary focus is to optimize the visual adapters and temporal encoders, while the visual encoder and language model are frozen to ensure that the visual features align effectively with the language space. Following the approach used in LLaMA-VID [16], we utilize the image-text LCS-558K dataset from LLaVA [20], along with 232K video-caption samples from the WebVid 2.5M [3].

Boundary enhancement. After the pre-training stage, the Vid-LLM gains proficiency in processing visual information. During the fine-tuning stage, we focus on enhancing the model's ability to comprehend sequences of video frames, thereby improving its temporal localization capabilities. In alignment with practices from VTimeLLM [10], we employ the InternVid-10M-FLT dataset [34], which is specifically designed for temporal-awareness training. The tasks within this dataset include: (i) dense video captioning, which requires detailed descriptions of all events along with their corresponding timestamps, and (ii) segment captioning and temporal video grounding, which involve generating descriptions based on timestamps or determining timestamps based on descriptions. Throughout this stage, we continue to train the visual adapters and temporal encoders. Additionally, the LLM is further trained using LoRA [9] to refine its capabilities.

SlowFocus adaption. Following the fine-tuning stage, our model has demonstrated the ability to comprehensively understand activities within the video and accurately align them with their respective timestamps. In the final stage, to further enhance multi-modality comprehension and integration with the MMF mechanism, we construct an instruction-tuning dataset using samples from ActivityNet Captions [13] and FineAction [21]. We transform the annotations into over 100K high-quality QA dialogues, which will be detailed in the subsequent section. In alignment with our SlowFocus mechanism, during training with fine-grained captioning and reasoning tasks, the model is provided with the ground-truth temporal segments τ GT and subjected to mixed-frequency sampling. In this

stage, we freeze all the visual adapters and attention modules, only training the language model as well as the MMA module. The parameters of visual encoder are frozen all over the stages.

4 FineAction-CGR benchmark

In addition to the scarcity of previous methods for fine-grained video understanding, existing benchmarks [37, 40] do not adequately challenge specific temporal-related tasks. To address this gap and evaluate our proposed framework, we introduce FineAction-CGR, a comprehensive and highquality instruction-following benchmark designed for evaluating the capabilities of Video LLMs in fine-grained video understanding. FineAction [21] is a large-scale and fine-grained video dataset encompassing diverse action categories with detailed annotations of action instances and time segments. For the purpose of SlowFocus adaption and model evluation, we divide the FineAction dataset into training and testing sets based on videos, allocating 75% to the training set and 25% to the test set, ensuring these is no overlap between them. Building on this foundation, we probe time information and content information from FineAction and expand it into multi-tasks. This section outlines the data construction procedures, which consist of three main steps: (i) video preprocessing, (ii) annotation construction, and (iii) formation of downstream tasks. More details are included in the appendix.

Video preprocessing. As depicted in Step I of Figure 4, we employ a modified two-stage splitting algorithm from Panda70M [5] to segment videos into clips, thereby providing raw materials for fine-grained information at the clip level. This preprocessing step yields 62,912 video clips.

Annotation construction. Step II in Figure 4 outlines our approach to creating new annotations, which include time segments, captions, and action labels. Given the high cost of using GPT-4V [1] for generating large-scale, clip-level captions, we generate detailed captions for entire videos with GPT4V and then use a fine-tuned Video Recaptioner Model to produce large-scale, clip-level captions. By integrating action labels and time segments from the ground truth with clip segments and the generated captions, we create comprehensive new annotations.

Formation of downstream tasks. Utilizing the new annotations, we design four types of video instruction-following tasks, as illustrated in Step III of Figure 4: (i) segmented captioning, (ii) temporal video grounding, (iii) temporal video reasoning and (iv) multi-turn QA. Detailed information on construction procedures, task formats, prompts, and case studies can be found in Appendix C- E.

Evaluation protocol. To assess the capability of video language model in comprehending fine-grained video understanding tasks, we integrate multiple evaluation metrics in comprehensively evaluate the model's performance against our proposed benchmark. For temporal grounding task, we calculate the Intersection over Union (IoU) between the temporal segments generated by the model and the corresponding ground truth, and we report meanIoU (mIoU) metric. For segmented captioning, we employ commonly accepted captioning-based metric, including BLEU [25], METEOR [4], ROUGE [19] and CIDEr [30]. For temporal reasoning task, we utilize GPT-4 [1] to evaluate the accuracy and score of the generated answers.

5 Experiments

5.1 Experiment setup

In our experiments, we implement LLaMA-VID [17] as baseline and utilize Vicuna-7B v1.5 [44] as our foundational LLM. More implementation details are provided in the appendix. We adjust the resolution of input videos to 224 × 224, and each frame is condensed into 64 tokens. The low-frequency sampling interval M L is set to match the original video fps, ensuring one frame is sampled every second. We define the dense sampling number N H as 20 and the size of temporal token space N as 1000. The AdamW [22] optimizer is applied with a cosine learning rate and decay and a warm-up period. We train our Vid-LLM for three stages. During the initial pre-training stage, the learning rate is set to 1 × 10 -3 . For the subsequent fine-tuning stages, the learning rate is adjusted to 2 × 10 -4 . Additionally, the LoRA parameters are configured with r = 64 and alpha = 128 . All experiments are conducted on 8 V100 GPUs.

5.2 Main results

Results on fine-grained video understanding. We evaluate the performance of existing VidLLMs on our proposed fine-grained video understanding benchmark FineAction-CGR, as detailed in Table 1. Further specifics are available in the appendix. Our method significantly outperforms other counterparts, achieving a mIoU of 66.68 for temporal grounding and an accuracy of 53.10% for the reasoning task. Notably, most other models, with the exception of VTimeLLM [10], exhibit subpar performance in both the temporal grounding and reasoning tasks on the benchmark. We propose several possible explanations for this observation: (i) These models lack sensitivity to precise time

Is the shot from 100.5s in the video scored or not?

The relevant temporal segment

is from 100 to 110:

No, the shot is not scored.

0

25

50

75

100

125

(Input frames: 0, 25, 75, …, 125)

Ours

(Additional input frames: 100, 101, …, 110)

Look at this video, if a person starts with playing the keyboard,

can you predict what the next action will be?

The relevant temporal segment

is from 100 to 110:

Yes. After play the keyboard, people switch to playing the drums.

0

20

40

60

80

100

(Input frames: 0, 20, 40, …, 100)

Ours

(Additional input frames: 45, 46, …, 55)

boundaries, making it challenging for them to accurately predict temporal segments. (ii) Due to their reliance on low-frequency sampling, these models struggle to capture fine-grained temporal details, which adversely affects their performance on tasks requiring fine-grained temporal reasoning.

Wealso fine-tune the baseline using stage 3. To ensure fairness, the implementation details for baseline fine-tuning are kept consistent with those of our method. The baseline's performance improves because stage 3 includes tasks focused on temporal grounding and is specifically designed for fine-grained analysis. Furthermore, the remaining performance gap further supports our explanations.

Results on coarse-grained video-based benchmarks. In Table 2, we present a comparative evaluation of our method against various state-of-the-art methods across three zero-shot videoQA benchmarks: MSVD-QA [36], MSRVTT-QA [37], and ActivityNet-QA [40]. We also conduct experiments on the video-based generative performance benchmark [23]. The results demonstrate that the proposed SlowFocus mechanism not only enhances fine-grained video understanding but also delivers competitive performance in coarse-grained video understanding tasks, achieving results on par with state-of-the-art models.

Results on long video benchmarks. To further investigate the effectiveness of the proposed method on more challenging scenarios, we provide evaluations on long video benchmarks, including MovieChat-1K [28] and EgoSchema [24]. As shown in Table 3, we evaluate our method on MovieChat-1K. The results show that although our method is not specifically trained on long video benchmarks (in contrast, MovieChat [28] has undergone targeted training for long videos), it still achieves competitive results (58.6% accuracy in global mode and 48.1% in breakpoint mode).

Additionally, we also conduct experiments on EgoSchema [24] benchmark, as detailed in Table 4. The results further demonstrate that, despite not being specifically trained on long video datasets, our method still achieves competitive performance.

Qualitative results. Figure 5 illustrates the qualitative results of our method on different videos. Our SlowFocus comprehensively analyzes the entire video, accurately identifies relevant temporal details based on the posed question, and provides precise answer within the context of videos.

5.3 Ablations

In this section, we provide detailed ablation studies of our method conducted utilizing Vicuna-7B as the foundation model.

Component-wise analysis. We first investigate the impact of each proposed component in Table 5. Importantly, we observe that the baseline model, which relies solely on low-frequency frames ( V L ), achieves limited performance, with an mIoU of 32.54 and an accuracy of 30.25. This highlights the

challenges faced by current Vid-LLMs in addressing fine-grained temporal tasks using only lowfrequency sampling. Additionally, the results in the second row indicate that the inclusion of MFS significantly improves reasoning capabilities, resulting in an accuracy increase of +8.87. Furthermore, the temporal encoder, which models temporal relationships effectively, boosts both grounding and reasoning capabilities, with an increase of +22.45 in mIoU and +7.25 in accuracy. Increasing the high-frequency sampling number N H from 10 to 20 leads to a noticeable performance gain (+7.66 in mIoU and +5.31 in accuracy), although further increasing N H to 40 offers only marginal benefits. Lastly, the MMA module contribute an accuracy enhancement of +1.42.

Necessity of fine-tuning strategy. We also analyze the impact of each training stage, as reported in Table 6. Directly evaluating the pre-trained model after stage 1 yields poor performance. Based on necessary pre-training stage 1, when only utilizing stage 2 for boundary enhancement, there is a significant improvement by 51.56 in mIoU in temporal grounding ability. Furthermore, implementing only stage 3 for SlowFocus adaptation boosts the performance in temporal reasoning by 25.14 in accuracy. Integrating these stages results in the highest performance in both mIoU and accuracy metrics, highlighting the irreplaceability of our proposed comprehensive training strategy.

What contributes more to fine-grained video understanding? In Table 7, we conduct further experiments to explore how changes in sampling frequency and the number of frame tokens influence our method. For meaningful comparison, we maintain constant total token numbers while gradually decreasing the sampling frequency. The results indicate that within proposed SlowFocus, performance improves with an increase in frame tokens and remains relatively unaffected by a decrease in global sampling frequency. This demonstrates the efficacy of our approach in high-frequency sampling and effectively alleviates the trade-off dilemma between sampling frequency and frame tokens.

Ablation on discretized temporal space. We investigate the influence of temporal token space N in Table 8. The results demonstrate that when the token space increase from 0.1K to 1K, a significant improvement (+22.87 in mIoU and +13.69 in accuracy) occurs. While further increasing N from 1K to 10K, the performance drops instead.

6 Conclusion and limitations

Conclusion. We introduce SlowFocus, a straightforward yet potent mechanism that significantly enhances fine-grained video understanding. Our approach improves the temporal localization capabilities of Vid-LLMs, enabling them to precisely identify relevant temporal segments based on the given query. Moreover, with our newly developed temporal encoder and multi-frequency mixing attention module, our method effectively models the temporal relationships among frames and captures inter-frame context. Demonstrating superior performance on our newly established fine-grained video understanding benchmarks, we hope that our work can propel the development of Vid-LLMs in achieving advanced video understanding.

Limitations. Despite significant advancements made in this work to enhance Vid-LLM's access to high-frequency temporal details and its temporal reasoning capabilities, challenges persist due to the limited existing research on maintaining high resolution in video. Consequently, our method may still encounter inaccurate predictions stemming from the ambiguity of spatial details.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint , 2023. 7, 8

Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV , 2017. 3

Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV , 2021. 6

Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop , 2005. 8

Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint , 2024. 7, 14

Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token modeling. arXiv preprint , 2021. 8

Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In ICCV , 2017. 3

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 , 2020. 8

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint , 2021. 6

Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. Vtimellm: Empower llm to grasp video moments. arXiv preprint , 2023. 2, 3, 6, 7, 8, 14

Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint , 2024. 8

Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding. arXiv preprint , 2024. 8

Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Densecaptioning events in videos. In ICCV , 2017. 3, 6

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML , 2023. 1, 3

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint , 2023. 1, 3, 7, 8

Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint , 2023. 1, 6, 7

Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint , 2023. 2, 8

Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint , 2023. 1, 3

Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , 2004. 8

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 2024. 3, 6, 14

Yi Liu, Limin Wang, Yali Wang, Xiao Ma, and Yu Qiao. Fineaction: A fine-grained video dataset for temporal action localization. TIP , 2022. 6, 7

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint , 2017. 8

Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint , 2023. 3, 7, 8, 9

Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems , 2023. 9

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL , 2002. 8

Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint , 2023. 3

Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed, Michael S Ryoo, and Tsung-Yu Lin. Learning to localize objects improves spatial reasoning in visual-llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12977-12987, 2024. 3

Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In CVPR , 2024. 8, 9

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint , 2023. 8

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR , 2015. 8

Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, and Yong Xu. Bidirectional attentive fusion with context gating for dense video captioning. In CVPR , 2018. 3

Shijie Wang, Qi Zhao, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Vamos: Versatile action models for video understanding. arXiv preprint , 2023. 8

Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems , 2024. 3

Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint , 2023. 6

Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint , 2022. 8

Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM MM , 2017. 9

Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR , 2016. 2, 7, 9

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems , 2022. 7

Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems , 2022. 8

Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI , 2019. 2, 7, 9

Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering. arXiv preprint , 2023. 8

Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint , 2023. 1, 3, 7, 8

Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint , 2023. 7

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 8

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint , 2023. 3

A Overview

In this appendix we present:

More implementation details (Section B).

Data construction details (Section C).

Data statistics (Section D).

Prompts used for instruction generation (Section E).

Discussion on broader impacts (Section F).

B More implementation details

In our work, we employ CLIP-ViT-L-14 for the frozen visual encoder, which accepts image resolutions at 224 × 224 as input. For visual adapters, we follow LLaVA [20] to train a projector layer, aligning visual features with the pre-trained LLM word embedding.

We also elaborate on how we enable LLM to predict temporal segments during stages 2 and 3 fine-tuning. Following VTimeLLM [10], we utilize the textual format 'from s to e' to denote a video clip, where s and e represent the starting and ending points, respectively. These time points are normalized and range from 000 to 999, corresponding to the discretized temporal token space N . During training, Vid-LLM is exposed to supervisory signals as described above, which enable it to develop the capability to accurately locate temporal segments.

C Data construction

Step I: video preprocessing. Wesplit videos into video clips to provide raw materials for fine-grained information in video clip level.

A proper splitting and stitching strategy is necessary. For one thing, the content of each clip should be semantically consistent. Without this consistency, actions within the same video clip obtained by erroneous segmentation can't be inferred reasonably and in chronological order, which negatively impacts the performance of downstream tasks such as action recognition and temporal reasoning. For another, the video clips with insufficient information or that are too short are not suitable for subsequent tasks, such as temporal grounding. Therefore, we adopt a modified two-stage splitting algorithm in Panda70M [5] to obtain semantically consistent video clips.

In first stage, We collect time boundaries of events in the video. Instead of detecting abrupt changes in pixels of adjacent frames, we detect lens switching so that actions in the same video clip can be inferred reasonably. In second stage, we stitch adjacent events based on time boundaries in first stage. Our stitching strategy could be depicted as merging short video clips and that are semantically similar. To be specific, we merge current event with the previous, if one of the following two conditions is met: 1)the video clip duration is less than five seconds, 2) the feature distance between the start frame of the current event and the end frame of the previous event does not exceed 0.1. Finally, we split videos into clips according to time boundaries from second stage.

Our adjusted splitting strategy has two advantages: 1) it preserves the information density of each segment, ensuring that subsequently generated captions are reasonable; 2) it maintains logical temporal consistency of actions within each segment, which facilitates the generation of questionanswer pairs in downstream tasks. Following the preprocessing step, we obtained 62,912 video clips.

Step II: annotation construction. As Figure 4 illustrates, new annotations we construct consist of time segments, captions and actions. (1) Action labels are from FineAction ground truth. (2) Time segments cover clip's time segments from above segmentation step and the action time segments from ground truth. (3) Captions cover video captions and clip captions. We first utilzie GPT-4V to generate detailed captions for entire videos. However, due to the cost of GPT-4V for generating large-scale, clip-level video captions, we fine tune a Video Recaptioner Model based on the captions generated by GPT-4V, which is then utilized to generate large-scale, clip-level captions. With actions

and time segments from ground truth, we integrate clip segments from splitting stage and captions generated by GPT-4V and Video Recaptioner Model to get new, comprehensive annotations.

Step III: formation of downstream tasks. We design various types of instruction-following tasks with different prompts to comprehensively train and evaluate Vid-LLMs. FineAction-CGR encompasses 4 video tasks: (1) captioning, (2) temporal video grounding, (3) temporal video reasoning and (4) multi-turn QA. A few examples from different tasks are shown in Figure 8-9.

Captioning task needs model to detects what actions occur in given period.

Action recognition: recognize action in level of clip which may contain a kind of action or several kinds of actions.

Temporal grounding tasks aims to predict the boundary in the video given an instruction in which the start and end time of clip are mentioned to attract model's attention to target segment. We design two types of question to achieve fine-grained temporal localization.

First/last time grounding: identify the fist/last time a certain action appears.

Temporal reasoning tasks requires model to understand relationship between actions.

Action sequence reasoning: infer possible action before/after an action over a period of time.

Count of times: counts times a certain action appears in a clip.

Multi-turn QA task contains multiple rounds of dialogues involving the above three tasks. The problems are progressive from the entire video to the sub-segment.

90.12%

4.47%

4.75%

0.61%

0.04%

Action Types

1

2

3

4

5

32%

14%

27%

9%

14%

4%

Task Distribution

ar

asr

1st

sequence

times

0

500

1000

1500

2000

2500

3000

3500

4000

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

Amount of videos

Number of video clips

D Data statistics

After filtering, we obtained 11,188 videos with different number of action types. The distribution is illustrated in Figure 6 left. After splitting, videos are segmented into clips. The amount of videos with different clip number is presented in Figure 7, exhibiting a long-tailed distribution. The largest

Actions Recognition

First/Last Time Grounding

? Can you briefly describe the video's content?

?

? Could you tell me when the first slam dunk was performed from 36.003s  to 146.913s?

? What is the last action a player performed from 214.447s to 236.669s?

?

?

?

? Identify the first time the athletes dribble basketball in this clip.

? Review the video from 147.347s to 147.647s Can you tell me what the basketball player in white jersey is doing during this time section?

dunk basketball

Action Sequence Reasoning

Count of Times

Multi-turn QA

From 36.003s to 37.104s

Based on the segment from 36.003s to 42.342s, what is likely to happen after a player

dribbles and casts the basketball?

The player is likely to dunk the basketball.

How many times do the athletic basketball players dunk the basketball from 150.684s to

162.629s in the video?

4

The video captures a high-energy basketball game featuring players in blue and white uniforms. There are also segments showcasing abstract shapes and letters, a dynamic basketball practice session, and a group activity involving athletes in dark blue uniforms.

What are the players doing from 36.003s to 146.913s?

From 36.003s  to 146.913s, the players in team uniforms are diligently involved in a basketball game. They are seen dribbling, passing, and shooting the basketball including performing dynamic slam dunks.

The first slam dunk was performed from 41.842 s to 42.342s

The last action a player performed is casting a basketball

Between the periods from 36.003s to 146.913s and from 150.684s to 182.649s, which had

more slam dunks?

The period from 36.003s to 146.913s had more slam dunks

24

What were the performers doing from 0s to 35.33s in the video? ?

They were playing the keyboard

When was the first time one of the people played the keyboard in the performance? ?

From 0s to 35.33s

Look at this video from 50.33s to 103.66s, if a person starts with playing the keyboard, can you predict what he will do next? ?

After playing the keyboard, the individual switches to playing the drums.

How many times does one individual play the keyboard from 0s to 142.933s in the video? ?

3

Can you describe the event showed in this video? ?

Two people are performing music on a stage. One is playing the keyboard and

the other one is playing the drums.

During which period does the keyboard playing occur from 0s to 142.933s? ?

The keyboard playing occurs from 0s to 35.33s, from 50.33s to 57.66s, and from 104.33s to 141.66s

When does the first scene of playing the drums take place from 0s to 142.933s? ?

The first scene of playing the drums takes place from 57.66s to 103.66s

What is the first action that occurs in the time period from 0s to 142.933s? ?

The first action that occurs is playing the keyboard

What typically happens after the keyboard playing ends during 0s to 142.933s? ?

Typically, playing the drums follows after the keyboard performance

Actions Recognition

First/Last Time Grounding

Action Sequence Reasoning

Count of Times

Multi-turn QA

口

clip number is more than 68. The horizontal coordinate in the figure is only displayed to 35. After generation step, we obtain 131,984 pairs of QA. The distribution in different tasks are illustrated in Figure 6 right. Explanation of the legend 1) 'ar': Captioning task where the model needs to recognize a single action in a clip, 2) 'asr': Captioning task where the model needs to recognize multiple actions in a clip, 3) '1st': Temporal grounding task where the model needs to localize the first/last occurrence of a given action in a clip, 4) 'sequence': Temporal reasoning task where the model needs to identify what action happens before or after a given action in a clip, 5) 'times': Temporal reasoning task where the model needs to count the occurrences of a given action in a clip, 6) 'mtqa': Multi-turn QA task where the model engages in a multi-turn question and answer dialogue about the entire video.

E Prompt

Prompts used for generation of different kinds of instruction data are shown in Figure 10-13. Due to page length constraints, we have omitted some in-context examples in certain tasks.

F Broader impacts

The proposed SlowFocus has the potential to greatly enhance the capability of Video LLMs for video content analysis. Therefore it could be beneficial for applications such as video analytics, surveillance and automated content moderation. It could also be used in educational settings to analyze educational videos to provide a better understanding of complex contents in videos for educational purposes.

As for the potential negative impacts, there is a risk that the technology could be misused to analyze private videos and spreading disinformation.

Prompts for Action Recognition

Prompts for Temporal Grounding

Prompts for Temporal Reasoning

Generate a concise dialogue  with questions and answers based on the following clip caption, action labels and time sgements. The action labels are in chronological order. Below is an example:

<Example 1>

[Clip Caption and Time Segment]

["0:00:00.000", "0:00:20.520"] Players in yellow and purple jerseys engage in an intense basketball game, running, passing, and dribbling on a well-lit court, captured in a wide-angle static shot.

[Action Label]

dribble basketball, cast basketball, dunk basketball

[Action Labels and Time Segments]

[0.33, 1.301] dribble basketball,

…

[13.013, 13.981] dunk basketball

[Q & A]

[{"from":"human", "value":"infer from 0.33 to 2.636, what the player would most possibly do after dribble basketball?"},{"from":"gpt", "value":"Cast basketball"},

{"from":"human", "value":"Look at this clip from 0 to 20, what players usually do in a basketball game?"},{"from":"gpt", "value":"Players in a basketball game usually dribble basketball and cast basketball."},

{"from":"human", "value":"Look at this clip from 11.011 to 13.981, a player dribbles basketball. Guess what would he do after that?"},{"from":"gpt", "value":"The player may dunk basketball."},

{"answer_segment":[[1.668, 2.636],[0,20],[13.013.13.981]]}]

Now given the following clip caption, action labels and time segments, please generate questions and their answers, please generate three questions related to time and action to test whether the deep learning model can recognition fine-grained action  based on the information provided by asking question like " Based on one sub-segment of this segment and infer what will most possibly happen after an action ." or " Look this video from a to b, What's the most possible action before/after this step? "

time_segment: clip_caption

actions_segments_list

The actions mentioned in the questions are selected from [Action Label]. The answers should be inferred from the provided time and action sequence in [Action Labels and Time Segments].

Just return the list as your response, please follow below format:

[{"from":"human", "value":" ..."},{"from":"gpt", "value":"..."},{"answer_segment":[<s3>, <e3>]}]

The value from 'human' is question. The value from 'gpt' is answer. The time in every question must be in format 'from ... to ...' and be the maximum period containing question's and answer's actions' but not others. "answer_segment" is a list of time segments pointing to minimum time priod of the answer's action and each is a subset of time in its corrresponding question. Dont't write what you imagine. Just return the list.

Prompts for Multi-turn QA

You are an excellent assistant for generating multi-round conversations. I need you to generate a dialogue with multi-turn questions and answers based on the information of a video. The information is a complete annotation of a video which contains one or more clips in chronological order. It mainly has captions, time segments and actions. Below is an example:

<Example 1>

<Information>

[Q & A]

[{"from":"human", "value":"Can you briefly describe the video's content?"}, {"from":"gpt", "value":"The well-lit indoor arena hosts an energetic basketball game..."},

{"from":"human", "value":"What the players is doing from 0 to 11.878?"}, {"from":"gpt", "value":"The players are competing. They dribble and cast basketball on the court."},

{"from":"human", "value":"Please capture the first time a player is dribbling basketball from 0 to 11.878."}, {"from":"gpt", "value":"from 0.033 to 1.969."},

{"from":"human", "value":"What is the last action before dunking basketball from 52.686 to 52.986?"}, {"from":"gpt", "value":"An player casts basketball."},

{"from":"human", "value":"Please infer what players usually do in a basketball game."},{"from":"gpt", "value":"They usually dribble basketball or cast basketball."}, "task":['cp','cp','grd','rsn','rsn']]

Now given the following <Information>, please generate a dialogue with five pairs of questions and answers to test whether the deep learning model has capability of captioning, temporal grounding and especially temporal reasoning based on the information provided .

Captioning task contains video captioning, clip captioning and action recognition during a time peiord. Temporal grounding task needs to find out target time period according to requirements whose answer is a time period . Temporal reasoning task needs to logically infer behavior motive from video, such as time sequence between actions, action sequence in an activity or reasons why people in video do such an action. Questions must be designed according to information provided and must contain all three tasks. You can design temporal reasoning questions different from examples I give you, but don't write what you imagine.

{all_information}

Just return the list as your response, following is format example:

The value from 'human' is question. The value from 'gpt' is answer. 'task' is a list of string in which 'cp' means captioning, 'grd' means grounding, 'rsn' means reasoning. The length of 'task' list are equal to the number of questions. Just return the list.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We make the main claims clearly in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of this paper in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We discuss the experimental setup in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Code and benchmark would be released after submission and review. It does include a new open-source benchmark.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details of training and testing are specified in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Experiments in this paper are conducted in multiple iterations, yielding stable results. We reported the average of these results for consistency and reliability.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational cost and resources are specified in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conforms in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The Broader Impacts are discussed in Appendix F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: We follow the safeguards of base LLM Vicuna-7B v1.5.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All models and datasets used in this paper have been properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Benchmark would be released after submission and review. The paper does provide well-structured details of the dataset.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the video LLMs (Vid-LLMs) area and attempts to improve the fine-grained video understanding. The authors design modules and training data to identify relevant temporal segments based on the given query and then generate outputs based on the identified segments.\n\nSTRENGTHS:\nOverall, the writing is well-organized and easy to follow. The strengths can be summarized as follows:\n1. This work introduces a two-stage inference strategy into the Vid-LLMs area, following the coarse-to-fine mode. Such a strategy provides a new perspective for enhancing video understanding.\n2. This work proposes a new benchmark FineAction-CGR to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks, which provides a new perspective to evaluate Vid-LLMs' capabilities.\n\nWEAKNESSES:\nThere are some weaknesses listed as follows:\n\n1. Although the proposed coarse-to-fine reasoning approach is relatively novel in Vid-LLMs, I have concerns about its practicality. Firstly, this reasoning method nearly doubles the inference cost for the same question compared to other methods. Secondly, the accuracy of the final response heavily relies on the accuracy of event localization in the first stage. According to the benchmark validation provided by the authors, even when trained on proprietary data, SlowFocus only barely meets the passing mark in temporal grounding mIoU. This significantly impacts the accuracy of temporal reasoning in the second stage.\n\n2. The authors chose relatively weak baselines and attempted to claim the superiority of their results. I suggest comparing their method with MovieChat [1] and VideoChat2 [2] to substantiate their claims of advancement. Moreover, despite using more data than LLaMA-VID, their performance on various benchmarks is not significantly better than LLaMA-VID, raising questions about the effectiveness of this two-stage approach.\n\n3. The authors need to validate the effectiveness of their two-stage model on benchmarks involving long videos, which is crucial to substantiate their claims. For instance, they could use benchmarks like EgoSchema and MovieChat-1K for this purpose.\n\n\n[1] MovieChat: From Dense Token to Sparse Memory for Long Video Understanding. CVPR 2024\n\n[2] MVBench: A Comprehensive Multi-modal Video Understanding Benchmark. CVPR 2024\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on fine-grained video understanding with large language models. Current works suffer from the dilemma between the per-frame token number and temporal sampling frequency to maintain an acceptable sequence length into the language model. The authors propose to sample a global view with low sampling frequency to perceive the temporal positions that are related to the question, then sample the frames within the predicted temporal window with high frequency for fine-grained video perception. The proposed dataset FineAction-CGR can well reflect the ability in temporal grounding and fine-grained question answering.\n\nSTRENGTHS:\n1. The exploration in the per-frame token number and temporal sampling frequency for video understanding is important and fundamental.\n2. The authors propose to first perceive the related temporal window and then densely sample the frames within the temporal span. The architecture makes sense and is suitable for fine-grained video captioning and question answering while maintaining computation efficiency.\n3. The dataset can evaluate the temporal grounding and fine-grained video understanding ability and could make contribution to the progress in video understanding with large language models.\n\nWEAKNESSES:\n1. How is the duration of the videos for temporal grounding evaluation? Will the low frequency sampling result in the $h_L$ to lose much information in excessively long videos?\n2. In Fig.2, the temporal grounding derives from the low frequency sampled video features. Why the high frequency sampling also has impact on the temporal grounding performance?\n3. The architecture of the temporal encoder is not illustrated.\n4. In Table 5, what does Fps=64 mean? The video dataset usually contains videos with around 30 fps, how does 64 fps work?\n5. The proposed method presents overwhelming advantage on the proposed FineAction-CGR dataset while the improvement on other benchmarks are marginal. I understand it is because the rest are comparatively easy and of coarse-grained. However, it is better to provide the evaluation on other challenging video benchmarks like long video datasets, e.g., EgoSchema [1], MovieChat [2] to validate the effectiveness of the temporal grounding in long sequences for more comprehensive comparison.\n\n[1] Mangalam, Karttikeya, Raiymbek Akshulakov, and Jitendra Malik. \"Egoschema: A diagnostic benchmark for very long-form video language understanding.\" Advances in Neural Information Processing Systems 36 (2024).\n[2] Song, Enxin, et al. \"Moviechat: From dense token to sparse memory for long video understanding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\nHow do you condense each frame into $N_v$ tokens?\n\nLIMITATIONS:\nThe authors have discussed the limitations on the resolution.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis works proposes “SlowFocus” to improve the balance between *tokens per frame* and *frames per video* used in Video LLMs. SlowFocus identifies video segments relevant to a given query and samples selected segments at high frequencies. The high frequency tokens are mixed with low frequency global video tokens. Authors propose suitable training strategies to learn the newly introduced layers. A new evaluation benchmark is also proposed by the authors.\n\nSTRENGTHS:\n1. Motivation for work is well established\n2. Clear explanation of proposed methodology \n3. Evaluation benchmark contribution\n\nNOTE: Is the proposed dataset released publicly?\n\nWEAKNESSES:\n1. **Motivation not justified:** The idea is to improve video LLM performance at a fixed compute. However, there are no results to evaluate the inference speed of proposed SlowFocus with prior works. Hence, the benefit of this approach over uniform sampling at a higher resolution is unclear.  \n\n2. **Table 1 Unfair:** It appears that prior works are zero-shot while SlowFocus is trained on this FineActionCGR dataset. This makes the comparison unfair.\n\n3. **Table 2 Missing Details:** Is SlowFocus re-trained on this data? How many frames do the prior works use? What is their inference compute requirement in comparison? Also, can a simple VLM + LLM baseline like LLoVi [1] be added for comparison? This will give a better idea on usefulness of proposed method.  \n\n4. **Missing Ablations:** How does this compare against simply feeding the selected high-res frames and the global low-res frames (arranged in temporal order) into an existing Vid-LLM (e.g. LLoVi)? \n\n5. [Minor] Related work [2] using spatial coordinates for same video tasks \n\n\n\n[1] LLoVi: https://arxiv.org/pdf/2312.17235\n\n[2] Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs, CVPR 2024\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nAddressed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper designs a SlowFocus mechanism to allow Vid-LLM's input signals to combine both high frame rate and low frame rate inputs simultaneously. This addresses the issue of maintaining the effectiveness of input information within a limited context window in LLMs. Low frame rate inputs contain global information, while high frame rate inputs contain local details. Additionally, this paper proposes a Multiple-Frequency Mixing Attention mechanism to better integrate these inputs and a Temporal Relationship Modeling mechanism to preserve temporal relationships in the temporal dimension.\n\nSTRENGTHS:\n1. The SlowFocus mechanism, which combines high frame rate and low frame rate inputs, is highly reasonable and efficient. \n\n2. The Multiple-Frequency Mixing Attention mechanism and Temporal Relationship Modeling are appropriate technical designs.\n\n3. The proposed three-stage training and corresponding dataset construction are rational. \n   - The use of temporal grounding data for the second stage of training followed by video level instruction fine-tuning data for the third stage is highly instructive.\n\n4. The FineAction-CGR benchmark fills the gap in the community for fine-grained video understanding evaluation.\n\n5. The experiments in this paper are comprehensive, effectively demonstrating the validity of each design point.\n   - The writing is clear and easy to understand.\n\nWEAKNESSES:\n1. Line 126: Using the same letter \"L\" to represent both \"Low\" and \"LLM\" might appear confusing to readers. \n\n2. Table 5: There is some confusion regarding whether this pertains to the low frame rate parts or the high frame rate parts.\n\n3. Line 449: \n   - What is the data ratio processed by GPT-4V and the Video Recaptioner Model? \n   - Do their results show significant distribution differences? \n   - Are you using GPT-4V to generate frame-by-frame videos or video captions?\n\n4. The analysis on the effectiveness and rationality of the FineAction-CGR benchmark evaluation is insufficient.\n\nQUESTIONS:\n1. Why is the visual encoder turned off in all stages? Some recent work also start open the gradient of CLIP encoder, would the results improve in your model if it were turned on?\n\n2. Is the use of LoRA for pre-training due to resource constraints? Would full-rank training result in better performance?\n\nLIMITATIONS:\nIt is recommended to include comparisons with closed-source models such as Gemini Pro, GPT-4V, etc., to provide reference scores for  FineAction-CGR benchmark in Table 1.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}