VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos

## Abstract

Abstract Video-language understanding tasks have historically focused on short video clips, often struggling with the complexities of long-form video understanding.
Recently, many long video-language understanding approaches have taken advantage of the reasoning capabilities of Large Language Models (LLMs) to perform long video question answering, transforming videos into densely sampled frame captions, and asking LLMs to respond to text queries over captions.
However, the frames used for captioning are often redundant and contain irrelevant information, making dense sampling inefficient, and ignoring the fact that video question-answering requires varying levels of granularity, with some video segments being highly relevant to the question (and hence needing more fine-grained detail) while others being less relevant.
Thus, these LLM-based approaches are prone to missing information and operate on large numbers of irrelevant captions, lowering both performance and efficiency.
To address these shortcomings, we introduce VideoTree , a query-adaptive and hierarchical framework for long-video understanding with LLMs. Specifically, VideoTree dynamically extracts query-related information from the input video and builds a tree-based video representation for LLM reasoning.
First, VideoTree adaptively selects frames for captioning by clustering frames based on their visual features and scoring clusters based on their relevance to the query. We iterate this process until enough query-related keyframes are extracted.
Second, it organizes visual clusters into a query-adaptive and hierarchical tree structure; the structure of the tree encodes varying levels of granularity, with higher (deeper) resolution on relevant segments. Finally, VideoTree produces an answer to each question by traversing the tree’s keyframes and passing their captions to an LLM answering model, which answers the query.
Our experiments show that our training-free adaptive method improves both reasoning accuracy and efficiency compared to existing methods: VideoTree achieves a 7.0 % percent 7.0 7.0\% , 2.2 % percent 2.2 2.2\% , and 2.7 % percent 2.7 2.7\% improvement in accuracy over existing methods on the popular EgoSchema, NExT-QA, and IntentQA benchmarks, respectively, while
reducing inference time by 40 % percent 40 40\% .

## 1 Introduction

While recent developments in video-language understanding have led to major advances in answering questions about videos [, , , , , ], past work has largely focused on short video tasks, with videos typically ranging from 5 to 15 seconds [, , ].
Given the surge in accessible minutes-long video content and the importance of applications like long-form human behavior analysis and movie analysis, developing models that can reason over and answer questions about long videos is increasingly crucial.
However, most contemporary models [, , , , ] developed for short videos struggle with long-form understanding tasks.
Recently, several approaches [, ] have emerged that leverage the long-sequence reasoning capabilities of Large Language Models (LLMs) to tackle the challenge of long-range modeling in long video-language understanding.
Typically, these approaches leverage vision-language models (VLM) to caption densely sampled frames, thus representing the video in text format.
This text representation is then subsequently fed into an LLM, which reasons over the video and responds to the provided query.
Although this strategy has demonstrated great potentials on video understanding tasks, it still faces three major limitations that become particularly relevant when applied to long-form video understanding tasks:

Figure: Figure 1: Overview of VideoTree for LLM reasoning on long videos. VideoTree helps to extract query-related information in long videos in a coarse-to-fine style. We first cluster videos via their visual embeddings, and then build a tree structure in a query-adaptive manner. Lastly, the LLM takes selected keyframe captions in the tree to conduct long video reasoning.
Refer to caption: /html/2405.19209/assets/x1.png

1) Informational Overload:
Long videos inherently contain high levels of information redundancy, and current long video understanding approaches [, ] lack a principled method to effectively address this challenge. A deluge of redundant information can overwhelm the LLM, leading to mistakes in video reasoning. As shown in [Fig. 3](#S5.F3), performance on video tasks actually decreases as the number of frames sampled increases.

2) Lack of Query Adaptability: Existing methods [, ] transform video inputs into textual descriptions without considering the query, resulting in irrelevant information being fed to the LLM. This is sub-optimal, and makes it harder for the LLM to accurately answer specific queries.

3) Inability to Capture the Coarse-to-Fine Video Structure: Existing approaches [, ] often simplify video content into a list of captions without any structure, failing to account for the hierarchical nature of video information.
Especially in long videos, some video regions are information-dense – requiring fine-grained temporal understanding – while others are irrelevant to the query, or information-sparse.
Because of this, dense sampling approaches not only suffer from the information overload problem (2) mentioned above, but also omit key information from the captions.

These limitations underscore the pressing need for a new long video understanding method.
Specifically, the method should be *query-adaptive*, dynamically selecting parts of the video that are relevant to the question and ignoring those that are not, and *hierarchical*, i.e. able to capture different granularities of information, from coarse summaries to fine-grained actions.
In other words, we would like a method that adaptively identifies key video segments and allocates more computational resources to these areas. To facilitate this, we represent the video as a tree structure.
Specifically, we introduce VideoTree, an adaptive tree-based method for long-video understanding. VideoTree dynamically extracts query-related keyframes from the video input in a coarse-to-fine manner and organizes them within a tree structure, with child nodes representing more fine-grained information.
Our sampling is *adaptive*, meaning that VideoTree allocates more frames to relevant video regions and fewer samples to irrelevant ones.
VideoTree relies on three crucial steps: adaptive breadth expansion ([Fig. 1](#S1.F1)a), relevance-guided depth expansion ([Fig. 1](#S1.F1)b), and LLM-based reasoning ([Fig. 1](#S1.F1)c).
A more in-depth look at our method is given in [Fig. 2](#S3.F2);
initially, we extract clusters of query-related keyframes to form the first level of tree nodes for the video representation (see [Figure 2](#S3.F2), Step 1).
We adaptively increase the number of keyframes through an iterative process of visual clustering, keyframe captioning, and relevance scoring until sufficient query-related information is gathered for LLM reasoning.
After the initial clustering, for each first-level cluster, we extract its keyframe and determine the relevance of the cluster based on the keyframe’s relevance to the query.
We then expand deeper into the most relevant clusters to extract more detailed information (see [Fig. 2](#S3.F2), Step 2).
For each cluster, we obtain a single keyframe and pass it to a VLM captioner.
Finally, the keyframe captions from our tree representation are passed to an LLM, which reasons over them to answer the question (see [Fig. 2](#S3.F2), Step 3).
This addresses the key problems with uniform dense sampling: VideoTree is coarse-to-fine and conditioned on the query, making it not only more performant but also more computationally efficient, needing fewer frames to achieve stronger performance than densely-sampled frame baselines.
Crucially, we demonstrate that the tree in VideoTree provides a more relevant set of keyframes, and hierarchically represents coarse-to-fine information in an adaptive fashion.
This significantly enhances the LLM’s ability to reason effectively over long videos.

We demonstrate the effectiveness of VideoTree by evaluating it on three standard long video question answering (LVQA) datasets: EgoSchema [], which focuses on egocentric long-form video-language understanding; NExT-QA [], a widely-used video question answering benchmark featuring videos that average 44 seconds in length; IntentQA [], an LVQA dataset focused on reasoning about people’s intent in long videos.
Across these tasks, VideoTree substantially improves accuracy compared to state-of-the-art LLM-based approaches, achieving an absolute accuracy improvement of $7.0\%$ on EgoSchema benchmark, $2.2\%$ on NExT-QA, and $2.7\%$ on IntentQA.
In our analysis, we show that our method results in the best of both worlds: better performance than strong baselines (e.g. []) *and* better efficiency.
In summary, we present three major contributions:

- •
We propose VideoTree, an adaptive tree-based representation for LLM reasoning over long-videos, which intelligently allocates a computational budget by dynamically extracting query-related keyframes from the video input in a coarse-to-fine manner.
- •
We show that using VideoTree, we can extract frames that allow the LLM to understand and reason over the long videos better, resulting in substantial improvements over strong baselines across three standard long video question-answering datasets (EgoSchema, NExT-QA, and IntentQA).
- •
In our analysis, we find VideoTree is not only more effective but also more efficient than uniform sampling baseline, achieving a $5.6\%$ accuracy improvement using only 35% of the frame captions.
Qualitatively, we find that VideoTree selects query-relevant portions of the video to expand into, while glossing over irrelevant portions.

## 2 Related Work

Structural Video Representation.
Video understanding [, , , , , , , , , , , , , , , , , ] has shown impressive advancement in both views of comprehension and efficiency. Recently, several video-language methods [, , , , , , , , , ] have further introduced a structured understanding of video frames to allow compact and efficient recognition of scene contexts. For example, HierVL [] proposes a *bottom-up* hierarchical video-language embedding that capture video representations across short and long time periods.
Specifically, HierVL performs contrastive learning between short clips and their captions, then aggregating representations for long video understanding tasks.
VideoReCap [] introduces a progressive video captioning approach that generates short clip-level captions and summarizes them into longer segments.
These methods process long videos by progressively building high-level knowledge from local temporal information, i.e. in a bottom-up fashion that first captures all low-level details and then aggregates.
This results in significant computational and time overhead.
In contrast, VideoTree employs a top-down approach with dynamic depth, enabling efficient and effective long video understanding by dynamically extracting query-related keyframes for LLM reasoning .

Video Understanding with LLMs.
Inspired by the powerful reasoning capabilities of LLMs, recent works have explored using LLMs to address complex video-related tasks. Since LLMs primarily process text, various methods [, , , , , , , , , , , , , , , ] have been developed to efficiently train multimodal projectors to connect the visual encoder and LLMs or leverage caption-centric information.
More specifically related to video reasoning,
CREMA [] integrates information from various modalities to address video reasoning problems.
CREMA introduces a multimodal Q-former to project different modality features into the LLM embedding space, enabling the LLM for the compositional understanding of videos to predict answers.
While CREMA trains their Q-former, other past work [, , , , , , , , ] has investigated training-free combinations of captioners and LLMs for video understanding.
Specifically, LLoVi [] proposes a simple language-guided video understanding method. First, it extracts short-term video descriptions with a captioning model, and then an LLM summarizes these dense captions and responds to the given prompt. VideoAgent [] introduces a multi-round frame search strategy using an LLM.
This approach performs answer prediction and self-reflection based on the captions from previously collected frames and the input prompt, and explores the next frame until the LLM is satisfied.
However, these methods struggle with high computational and monetary costs during long-video comprehension, as they require processing various types of input, generating dense captions for video frames or clips, or frequent LLM calls to perform self-evaluation and iterative frame selection. Instead, we propose to extract the key information from long videos by building an adaptive tree representation and sparsely selecting the keyframes for further LLM reasoning.

## 3 VideoTree : Adaptive Tree-based Representation for Long Video-Language Understanding with LLMs

We present VideoTree, an efficient and effective long video understanding framework that constructs a query-aware, query-adaptive hierarchical video representation for LLM reasoning over long videos.
As illustrated in [Fig. 2](#S3.F2), the VideoTree framework consists of three main steps: adaptive breadth expansion, relevance-guided depth expansion, and LLM video reasoning.

### 3.1 Adaptive Breadth Expansion

We build the first layer of the tree representation by finding key semantic information within the video. To extract key information, existing approaches [, ] select a fixed number of keyframes as the video representation.
However, as mentioned in [Sec. 1](#S1), this kind of uniform keyframe selection is sub-optimal for a general long video-language understanding framework, as it can miss information, includes redundant frames, and does not intelligently allocate the frame budget according to the video and query.
To this end, we introduce an adaptive breadth expansion method that builds the first level of the tree representation.
This method dynamically finds clusters of frames that are relevant to the query.
Specifically, as shown in the left of [Fig. 2](#S3.F2) (Step 1), given the video and a query about it, we build the first level of the tree by iterating three operations: visual clustering, cluster captioning, and relevance scoring.
These operations first group similar frames together, then assign captions to each cluster and determine how relevant each cluster is to the query.
By iterating the process, VideoTree *adaptively* allocates more computation to the most relevant regions of the video.
In the following, we motivate and introduce each operation in detail.

Figure: Figure 2: A detailed view of VideoTree. To construct the tree structure, we begin with *Adaptive Breadth Expansion* (Step 1), which dynamically extracts query-related key information by considering both video and question inputs. Then, starting from the highly relevant root nodes, we explore deeper into the tree branches with *Relevance-guided Depth Expansion* (Step 2), re-clustering at each level to capture finer visual cues. Finally, we gather the selected nodes (keyframes), caption them, and arrange them in temporal order for *LLM reasoning* (Step 3).
Refer to caption: /html/2405.19209/assets/x2.png

Visual Clustering. To reduce frame redundancy, we first propose a visual clustering operation that groups similar frames together before extracting information from them.
Specifically, given a video sequence $V=(F_{1},F_{2}...,F_{n})$, where $F_{i}$ is the frame at the time step $i$ and $n$ is the length of the video, we extract visual features for each frame with the pre-trained visual encoder [] $E$, such that $f_{i}=E(F_{i})$, where $f_{i}\in\mathbb{R}^{d}$ is the visual representation extracted by the frame $F_{i}$.
Note that this visual feature extraction is less expensive than extracting textual features like captions using a VLM since visual feature extraction does not involve autoregressive decoding, making it about 2 times faster.
These features serve as a compact representation of each frame’s visual content, capturing diverse semantics of each frame such as scenes and objects.
Then we use K-Means clustering [], to group frame features into $k$ distinct clusters, which we denote as:

$$ $(C_{1},C_{2},...C_{k}),(c_{1},c_{2},...c_{k})=\text{K-Means}((f_{1},f_{2},...,f_{n}),k)$ (1) $$

where, $C_{i}$ is the $i$th cluster that groups multiple frames, $c_{i}$ is the centroid vector for the $i$th cluster and $k$ is the number of clusters, which can be thought of as the granularity of the tree’s first level.
This clustering process allows us to convert the video input into $k$ clusters of similar frames.

Keyframe Captioning.
To convert the cluster’s visual features into textual information that can be processed by an LLM, we caption a single keyframe from each cluster.
Specifically, for the cluster $C_{i}$, we find the keyframe $F_{i}$ that is closest to the centroid vector $c_{i}$ and consider it as the representative frame of the $i$th cluster.
We then feed all extracted keyframes into the VLM-based captioner $Cap(\cdot)$ [] and obtain a text caption $t_{i}=Cap(F_{i})$ for each cluster.

Relevance Scoring.
After obtaining the keyframe captions $t_{i}$, we use the LLM to decide whether the extracted key information from each cluster is sufficient for the LLM to answer the given query.
To this end, we feed all cluster captions $\{t_{i}\>\>\forall i\in[1,\ldots,k]\}$ from the last operation and the video query $Q$ into the LLM and output a set of relevance scores $\{r_{i}\>\>\forall i\in[1,\ldots,k]\}$ for each cluster, where $r_{i}$ is the relevance of the $i_{th}$ cluster.
To obtain each $r_{i}$, we prompt an LLM with the VLM captions and the query and ask it to assign each caption a relevance score of 1 to $max_relevance$, with 1 being least relevant.
See [Tab. 8](#A3.T8) for all prompts.

Then, we set a maximum value for the number of clusters ($max_breadth$) and a requirement of the high-relevance cluster number (selected based on validation set) for adaptively extract the keyframe information within each cluster.
If the number of high-relevance clusters is below the requirement, that indicates the information extracted from the current set of keyframes is insufficient for the LLM to answer the video query.
In that case, we increase the number of clusters $k$ and repeat the clustering, captioning, and relevance scoring operations.
If the number of high-relevance clusters meets the requirements or the number of clusters reaches $max_breadth$, we append the extracted clusters with their keyframes to the first layer of the tree and continue to the next step (see more details in lines 21-26 in [Algorithm 1](#alg1)).

### 3.2 Relevance-Guided Depth Expansion

Existing keyframe selection approaches [, , ] typically treat the selected frame as an unstructured list, neglecting potential internal structures within the video information.
Specifically, these methods overlook that certain video regions are information-rich and require detailed sampling, whereas other areas, irrelevant to the query, may only need coarse or minimal sampling.
The uniform or random sampling approach results in a long, redundant, and disorganized caption list that can confuse the LLM, ultimately limiting its ability to reason effectively and failing to provide a clear depiction of the video’s structure.
Thus, as shown in Step 2 of [Fig. 2](#S3.F2), we build a query-adaptive hierarchical video representation on top of the clusters from the first breadth expansion step.
Specifically, we expand the depth of the tree according to relevance score of each cluster from the first step.
The intuition is that for high-relevance clusters, the LLM requires more detailed, granular information, while for low-relevance clusters, more information could actually lead to irrelevant details and could overwhelm the LLM, leading to incorrect reasoning.

The relevance of a top-level node (cluster) informs how many levels of more granular information we will extract from it, i.e. how many children/grandchildren the node will have.
Since the relevance score $r$ is one of $[1,2,...,max_relevance]$, we expand the tree based on the remaining relevance of the parent node. Thus, the value of $max_relevance$ is equal to the max depth $max_depth$ of the tree-based video representation.
Specifically, we do not expand nodes with low relevance ($r_{i}=1$) further.
For nodes with medium or high relevance (greater than 1), we re-cluster their constituent frames into $w$ sub-clusters, where $w$ denotes the branch-width of the tree, on the premise that these more relevant clusters should have more keyframes allocated to them.
We then add those sub-clusters as children in the tree, giving them a relevance that is one less than their parent’s.
We recursively repeat this process until all nodes are leaf nodes with relevance 1 (see lines 2-12 in [Algorithm 1](#alg1)).
After the breadth and depth expansion on the tree, we obtain the adaptive hierarchical video representation for LLM reasoning over the long video.

### 3.3 LLM Video Reasoning

Finally, in order to use the LLM’s ability on video reasoning, we need to present the LLM with a text-based video description.
To this end, we traverse the nodes of the tree starting at the roots and expanding to the leaves, extracting keyframes from the tree’s clusters at all levels and passing them into the VLM captioner to obtain keyframe captions.
We then sort these keyframe captions in temporal order and concatenate them into a textual description of the video.
Finally, we pass this description and the input query to the LLM and output the final answer (see line 38-40 in [Algorithm 1](#alg1))
Our prompt is in [Tab. 9](#A3.T9).

## 4 Experimental Setup

Tasks & Datasets. We test our VideoTree framework on three diverse long-form video question-answering benchmarks:
(1) EgoSchema [], a long-range video question-answering benchmark consisting of 5K multiple choice question-answer pairs spanning 250 hours of video and covering a wide range of human activities.
Our ablation studies are conducted on the official validation set of EgoSchema which contains 500 questions (referred to as the EgoSchema Subset). The videos are 180s long on average.
(2) NExT-QA [], a video question-answering benchmark for causal and temporal reasoning. It contains 5440 videos with an average length of 44s and approximately 52K questions. NExT-QA contains 3 different question types: Temporal (Tem.), Causal (Cau.), and Descriptive (Des.).
(3) IntentQA [] contains 4,303 videos and 16K multiple-choice question-answer pairs focused on reasoning about people’s intent in the video. We perform a zero-shot evaluation on the test set containing 2K questions. The videos are more than 44s in average length.

Implementation Details.
We adopt GPT-4(^1^11version 1106) [] as our LLM.
Following VideoAgent [], we leverage EVACLIP-8B [] as our visual encoder and CogAgent [] as the captioner for NExT-QA benchmark. Following LLoVi [], we use LaViLa [] as our captioner for the EgoSchema benchmark. Similar to [], we use LLaVA1.6-7B [] as our captioner for the IntentQA dataset.
For clustering, we use kmeans_pytorch Library (^2^22[https://github.com/subhadarship/kmeans_pytorch](https://github.com/subhadarship/kmeans_pytorch)).
The best setting for $max_breadth$, $max_depth$, and $branch_width$ on the EgoSchema validation set is $32$, $3$, and $4$ and for NExT-QA and IntentQA, we set the hyper-parameter as $8$, $3$, and $2$.
We ablate the hyper-parameter choices in detail in [Sec. A](#A1). We also provide additional implementation details in Appendix [Sec. C](#A3), including detailed prompting information for VideoTree.

Baselines.
We compare VideoTree with relevant baselines from previous work, including models based on video transformers [, , , ], open-source LLMs [, ], and those utilizing proprietary models, including the methods using GPT-3.5 [], GPT4 [, , ], PaLM-2 [] and GPT-4V [].

Evaluation Metrics.
We evaluate VideoTree on all datasets under the multiple-choice QA setting. We utilize standard accuracy metrics for all experiments.

## 5 Experiments

### 5.1 Comparison with State-of-the-art Approaches

**Table 1: Comparison with other methods on EgoSchema, NExT-QA, and IntentQA datasets. We compare our VideoTree framework with three types of existing works, including video transformer models, open-source LLM-based models, and proprietary LLM-based models.**
| Model | LLM | EgoSchema |  | NExT-QA |  | IntentQA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Sub. | Full |  | Tem. | Cau. | Des. | Avg. |  | Full |  |  |
| Video Transformer Models |  |  |  |  |  |  |  |  |  |  |
| LongViViT [] | - | 56.8 | 33.3 |  | - | - | - | - |  | - |
| MC-ViT-L [] | - | 62.6 | 44.4 |  | - | - | - | 65.0 |  | - |
| InternVideo [] | - | 32.1 | - |  | 43.4 | 48.0 | 65.1 | 49.1 |  | - |
| InternVideo2 [] | - | 41.1 | - |  | - | - | - | - |  | - |
| Based on open-source LLMs |  |  |  |  |  |  |  |  |  |  |
| SeViLA [] | FlanT5-3B | 25.7 | 22.7 |  | 61.3 | 61.5 | 75.6 | 63.6 |  | 60.9 |
| MVU [] | Mistral-13B | 60.3 | 37.6 |  | 55.4 | 48.1 | 64.1 | 55.2 |  | - |
| Based on proprietary LLMs |  |  |  |  |  |  |  |  |  |  |
| ProViQ [] | GPT-3.5 | 57.1 | - |  | - | - | - | 64.6 |  | - |
| LLoVi [] | GPT-4 | 57.6 | 50.3 |  | 61.0 | 69.5 | 75.6 | 67.7 |  | 64.0 |
| VideoAgent [] | GPT-4 | 60.2 | 54.1 |  | 64.5 | 72.7 | 81.1 | 71.3 |  | - |
| VideoAgent [] | GPT-4 | 62.8 | - |  | - | - | - | - |  | - |
| MoReVQA [] | PaLM-2 | - | 51.7 |  | 64.6 | 70.2 | - | 69.2 |  | - |
| IG-VLM [] | GPT-4V | 59.8 | - |  | 63.6 | 69.8 | 74.7 | 68.6 |  | 64.2 |
| VideoTree (Ours) | GPT-4 | 66.2 | 61.1 |  | 67.0 | 75.2 | 81.3 | 73.5 |  | 66.9 |

LABEL:tab:main_results shows a comparison of the existing works and VideoTree on three diverse video question-answering benchmarks, including EgoSchema, NExT-QA, and IntentQA.

EgoSchema. On the long-form video question-answering dataset EgoSchema [], our VideoTree framework substantially outperforms the existing GPT-4-based approaches, including LLoVi [] and VideoAgent [, ]. Specifically, we outperform the state-of-the-art VideoAgent methods [, ] with $3.4\%$ and $7.0\%$ improvements on the subset and full test set, verifying the effectiveness of the proposed adaptive tree-based representation on long video understanding.
Moreover, VideoTree outperforms the GPT-4V-based method IG-VLM [], and surpasses strong multimodal LLM without using an expensive end-to-end multimodal model.

NExT-QA. On the NExT-QA benchmark, VideoTree achieves $73.5\%$ zero-shot accuracy on the validation set, outperforming existing state-of-the-art method VideoAgent [] by $2.2\%$ on average accuracy using the same captioner and LLM. Furthermore, VideoTree again outperforms the GPT-4V-based method IG-VLM []. Notably, NExT-QA contains various video queries, including temporal modeling (Tem.), causal reasoning (Cas.), and descriptive questions (Des.). We show that the VideoTree framework surpasses the existing approaches [, ] on all query types, indicating VideoTree improves the LLM reasoning ability on long videos under different scenarios.

IntentQA.
On IntentQA, our VideoTree framework achieves $66.9\%$ zero-shot accuracy on the test set, surpassing the existing approaches with $2.7\%$ improvements.
This result shows that VideoTree improves performance in answering questions about intent, which is challenging since intent understanding [] requires the model to understand the various video contexts, including the immediate communicative context, the shared experience, and the commonsense.

### 5.2 Analysis

Below, we provide a detailed analysis of our VideoTree framework.
All quantitative analyses are conducted on the validation subset of the EgoSchema dataset.
First, we analyze the tradeoff between efficiency and effectiveness, comparing VideoTree to the LLoVi baseline [].
Here, we show that our method has better efficiency *and* performance across all settings.
We then verify the effectiveness of the query-adaptive hierarchical video representation by comparing against different alternative representations.
Finally, we visualize the output trees from VideoTree and show the clusters VideoTree chooses to expand, qualitatively supporting its quantitative gains.
We also provide an extensive ablation study (including hyper-parameter analysis and the design choices of VLM/LLM) in Appendix [Sec. A](#A1).

#### 5.2.1 Efficiency-Effectiveness Analysis

In [Fig. 3](#S5.F3), we plot the Pareto curve between efficiency and performance, i.e. we analyze the relationship between the efficiency (caption numbers) and effectiveness (EgoSchema performance) of the VideoTree framework and compare with the baseline approach [].
Specifically, using the same captioner and LLM, we compare VideoTree, which features an adaptive tree-based video representation, against a uniform baseline.
Specifically, we use the uniformly-sampled frame caption list from the LLoVi baseline [].
Firstly, across all frame settings, VideoTree outperforms LLoVi by $2.5\%$ on average.
This indicates that our framework substantially improves the LLM reasoning ability over long videos under different frame budgets.
Furthermore, we also see that VideoTree achieves comparable performance with the best baseline (32 frames) using only $15.6$ frames on average, indicating the efficiency of the proposed framework.
In other words, VideoTree can do more with less, needing only half as many frames to achieve comparable performance.

Figure: Figure 3: Analysis of the efficiency and effectiveness relationship for VideoTree. We compare our VideoTree framework with LLoVi [] under different frame number settings.
Refer to caption: /html/2405.19209/assets/x3.png

#### 5.2.2 Effectiveness of the Adaptive Tree Representation

In [Tab. 2](#S5.T2), we verify the effectiveness of the proposed adaptive tree-based representation on a long video understanding task, comparing with the uniform sampling baseline [] and other tree representations.
First, the results show that both tree representations (adaptive tree and static tree) outperform the uniform sampling baseline, indicating the importance of the hierarchical nature of the video representation.
Furthermore, we compare our adaptive tree representation with a static tree variant, which is a tree representation obtained using the same visual clustering process as our adaptive tree-based representation but without any query adaptation.
Instead of dynamically extracting query-related information using a relevance score, the static tree builds always use the maximum value of the width and depth for all branches, resulting in a full tree.
This representation is hierarchical, but not adaptive.
The results show that both tree representations (static/adaptive) outperform the uniformly sampled caption list, which confirms the importance of having a structural representation for long video understanding using LLM.
We show the adaptive tree representation we use in VideoTree outperforms the static tree in both efficiency (63.2 frames vs 112.8 frames) and effectiveness (66.2% vs 63.8%), highlighting the importance of having an efficient and query-related representation for LLM reasoning over long videos.

#### 5.2.3 Qualitative Analysis

Figure: Figure 4: Qualitative examples of VideoTree keyframes and captions selection. Red options are answered wrongly with uniformly sampled 32 frames. Green options are answered correctly with VideoTree. Best viewed in color. We include additional visualization in Appendix [Sec. D](#A4).
Refer to caption: /html/2405.19209/assets/x4.png

In [Figure 4](#S5.F4), we visualize qualitative results from VideoTree.
Specifically, we show the keyframes and their captions extracted by our adaptive tree representation given a video query.
This example is drawn from EgoSchema, and shows the query format, which consists of a query and multiple-choice answers.
With the proposed VideoTree strategy, we can split a complex multi-scene video (e.g.cleaning house across rooms) into several key scenes via visual clustering and determine the most query-relevant scene via the relevance score.
We then can obtain more fine-grained visual cues by descending into each relevant cluster (Levels 2 and 3 in [Figure 4](#S5.F4)).
For example *“C opens a washing machine”* is deemed highly relevant to the question, which asks about the sequence of events.
At the same time, frames like *“C moves around”* are deemed irrelevant to the query and not expanded.
In the end, VideoTree shows a dynamic ability to select relevant segments and can answer the given question correctly with only 50% of the baseline’s 32 input captions.
The baseline (fixed uniformly sampling) fails to correctly answer the question, sampling a large number of redundant and irrelevant frames. We also provide additional qualitative results in Appendix [Sec. D](#A4).

## 6 Conclusion

After noting three key problems with dense sampling, we proposed VideoTree, an adaptive and hierarchical strategy for sampling frames, captioning them, and reasoning with an LLM to answer questions about long videos.
VideoTree resulted in substantial performance increases on three popular datasets (EgoSchema, NExT-QA, and IntentQA), while also improving efficiency by captioning fewer frames than uniform sampling.
We analyzed the role of the adaptive cluster selection we implement in VideoTree, finding that it is crucial to strong performance.
In our qualitative analysis, we showed that given a complex multi-scene video and its query, our VideoTree framework is capable of extracting key scenes and zooming into more detailed information that is highly related to the query.

## Limitations and Broader Impact

##### Limitations.

Like all LLM-based video-reasoning systems (including dense sampling) our method is limited by the ability of the captioner to accurately capture the contents of sampled frames.
However, our method’s modular nature means that as captioners improve, we can easily include them into the VideoTree framework; similarly, we can use increasingly strong LLMs as the reasoning backbone of VideoTree.
While VideoTree is training-free, it includes a small number of hyperparameters.
In [Sec. A](#A1), we ablate these hyperparameters, showing that VideoTree outperforms the uniform-sampling baseline regardless of the choice of max depth and branch width.
Thus, while better hyperparameters can benefit the method, even with suboptimal settings VideoTree outperforms the uniform baseline.

##### Broader Impact.

Our results indicate that we can have the best of both worlds: improved accuracy *and* improved efficiency.
Given the importance of long video reasoning tasks, improving accuracy has obvious broader implications for building more usable video reasoning systems, which could contribute to a wide variety of positive applications.
Efficiency improvements also contribute to the applicability of long video systems, as reducing latency and computational cost can speed up adoption.
Furthermore, since both VLM captioners and LLM reasoners generally improve with increased scale, reducing the number of calls to them will become increasingly important; we expect the efficiency benefits coming from our method to play an even larger role in the future.
Our work does not have any particularly relevant potential for negative applications or misuse beyond the standard caveats that apply to all machine learning systems.

## Acknowledgements

We thank Ce Zhang, David Wan, and Jialu Li for their helpful discussions.
This work was supported by DARPA ECOLE Program No. HR00112390060, NSF-AI Engage Institute DRL-2112635, DARPA Machine Commonsense (MCS) Grant N66001-19-2-4031, ARO Award W911NF2110220, ONR Grant N00014-23-1-2356, Sony Faculty Innovation award, Laboratory for Analytic Sciences via NC State University, and Accelerate Foundation Models Research program.
The views contained in this article are those of the authors and not of the funding agency.

## Appendix

In this Appendix, we present the following:

- •
Additional ablation study for our VideoTree framework ([Sec. A](#A1)).
- •
The detailed algorithm for VideoTree ([Sec. B](#A2)).
- •
Additional implementation details ([Sec. C](#A3)).
- •
Additional qualitative analysis ([Sec. D](#A4)).
- •
License information ([Sec. E](#A5)).

## Appendix A Additional Ablation Study

In this section, we report additional ablation studies for our VideoTree framework. First, we ablate the effect of the different tree settings for VideoTree. Then, we analyze the effect of different VLM/LLM designs for VideoTree.

Hyperparameter Analysis.
In [Tab. 4](#A1.T4), we study the effect of the max depth of the tree-based representation for the VideoTree.
The results show that as depth increases, the performance increases, showcasing the importance of depth expansion.
The max performance is reached when the max depth is set to 3, VideoTree obtains the best performance.
However, having a larger depth hurts VideoTree’s performance; note that all settings still outperform uniform sampling.

**Table 3: The effect of different settings for max depth of VideoTree. We show that when the max depth is set to 3, VideoTree obtains the best performance on the subset of the EgoSchema dataset. We also show that decreasing the max depth can make the model more efficient while retaining strong performance, outperforming all existing approaches.**
| Max Depth | ES Acc$\uparrow$ | #Frame$\downarrow$ |
| --- | --- | --- |
| 1 | 63.4 | 32 |
| 2 | 64.6 | 46.5 |
| 3 | 66.2 | 63.2 |
| 4 | 62.6 | 89.1 |
| Uniform Baseline | 61.2 | 180 |

Next, in [Tab. 4](#A1.T4), we study the effect of the branch width of the tree-based representation for the VideoTree.
The best performance is obtained when the branch width is set to $4$.
As with depth, excessive branch width reduces the VideoTree performance due to the information overwhelming to the LLM; however, even with the worst branch width setting, VideoTree still outperforms the baseline.

In [Tab. 6](#A1.T6), we study the effect of the max breadth of the adaptive tree-based representation. The results indicate that even with a smaller max tree breadth, VideoTree achieves good performance while using much fewer frames. Increasing the breadth generally increases performance, with the best performance when the max breadth is set to $32$.
However, having an excessive max breadth leads to worse results, suggesting that incorporating too much information in the adaptive tree-based representation limits the LLM reasoning ability.
This links back to the intuition of having an efficient representation for the LLM reasoning over long videos.

**Table 5: The effect of different settings for the max breadth of the first level of the tree. Results show that when the max breadth is set to 32, VideoTree obtains the best performance. Reducing the max breadth can improve efficiency while retaining performance.**
| Max Breadth | ES Acc | #Frame |
| --- | --- | --- |
| 8 | 62.0 | 36.9 |
| 16 | 64.4 | 49.0 |
| 32 | 66.2 | 63.2 |
| 64 | 62.4 | 94.6 |

VLM/LLM Choice.
We ablate the design choice of captioner and LLM for the VideoTree framework in [Tab. 6](#A1.T6). The results show that with a better captioner and LLM, VideoTree can achieve better performance on long video understanding tasks, indicating the potential VideoTree to improve as its modules become more advanced.
Notably, our GPT-3.5 variant substantially outperforms existing methods with the same LLM (VideoAgent [] $48.8\%$, LLoVi [] $51.8\%$), achieving $57.6\%$ accuracy on EgoSchema subset.

Error Bar Analysis.
We provide the error bar analysis for our main results in LABEL:tab:error_bar.

**Table 7: Comparison with other methods on EgoSchema, NExT-QA, and IntentQA datasets with error bar analysis.**
| Model | LLM | NExT-QA |  | EgoSchema |  | IntentQA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Tem. | Cau. | Des. | Avg. |  | Sub. | Full |  | Full |  |  |
| Video Transformer Models |  |  |  |  |  |  |  |  |  |  |
| LongViViT [] | - | - | - | - | - |  | 56.8 | 33.3 |  | - |
| MC-ViT-L [] | - | - | - | - | 65.0 |  | 62.6 | 44.4 |  | - |
| InternVideo [] | - | 43.4 | 48.0 | 65.1 | 49.1 |  | 32.1 | - |  | - |
| InternVideo2 [] | - | - | - | - | - |  | 41.1 | - |  | - |
| Based on open-source LLMs |  |  |  |  |  |  |  |  |  |  |
| SeViLA [] | FlanT5-3B | 61.3 | 61.5 | 75.6 | 63.6 |  | 25.7 | 22.7 |  | 60.9 |
| MVU [] | Mistral-13B | 55.4 | 48.1 | 64.1 | 55.2 |  | 60.3 | 37.6 |  | - |
| Based on proprietary LLMs |  |  |  |  |  |  |  |  |  |  |
| ProViQ [] | GPT-3.5 | - | - | - | 64.6 |  | 57.1 | - |  | - |
| LLoVi [] | GPT-4 | 61.0 | 69.5 | 75.6 | 67.7 |  | 57.6 | 50.3 |  | 64.0 |
| VideoAgent [] | GPT-4 | 64.5 | 72.7 | 81.1 | 71.3 |  | 60.2 | 54.1 |  | - |
| VideoAgent [] | GPT-4 | - | - | - | - |  | 62.8 | - |  | - |
| MoReVQA [] | PaLM-2 | 64.6 | 70.2 | - | 69.2 |  | - | 51.7 |  | - |
| IG-VLM [] | GPT-4V | 63.6 | 69.8 | 74.7 | 68.6 |  | 59.8 | - |  | 64.2 |
| VideoTree (Ours) | GPT-4 | 67.0 (±0.5) | 75.2 (±0.4) | 81.3 (±0.8) | 73.5 (±0.4) |  | 66.2 (±1.2) | 61.1 |  | 66.9(±0.6) |

## Appendix B Detailed Algorithm

In [Algorithm 1](#alg1), we present the algorithm behind VideoTree.

Figure: Algorithm 1 VideoTree

## Appendix C Additional Implementation Details

Prompt Details.
We provide detailed prompts for the relevance scoring prompt in [Tab. 8](#A3.T8) and LLM reasoning prompt in [Tab. 9](#A3.T9) on the EgoSchema benchmark.

**Table 8: VideoTree with relevance scoring prompt on EgoSchema.**
| User<br>You are presented with a textual description of a first view video clip, it consists of about caption_number frame captions sparsely sampled from the video (#C means the first person view, and #O indicates another). The ultimate goal is to answer a question related to this video, choosing the correct option out of five possible answers.<br>It is crucial that you imagine the visual scene as vividly as possible to enhance the accuracy of your response. After selecting your answer, rate your confidence level in this choice on a scale from 1 to 100, where 1 indicates low confidence and 100 signifies high confidence. Please provide a concise one-sentence explanation for your chosen answer. If you are uncertain about the correct option, select the one that seems closest to being correct. Meanwhile, could you provide a relevance score for each frame caption to evaluate their relevance with the query-answering process. The score is between 1,2,3, where 1 indicates low relevance and 3 signifies high relevance. Please return the relevance score in the format of a list of caption_number scores.<br>Examples: Examples<br>Description: Captions<br>Question: Question<br>Options:<br>A: Option-A. B: Option-B. C: Option-C. D: Option-D. E: Option-E.<br>The prediction, explanation, confidence and frame relevance are (please response in the format of ’prediction:, explanation:, confidence:, frame relevance:’)<br>Assistant<br>prediction, explanation, confidence, frame relevance |
| --- |

**Table 9: VideoTree with LLM reasoning prompt on EgoSchema.**
| User<br>You are presented with a textual description of a first view video clip, it consists of frame captions sparsely sampled from the video (#C means the first person view, and #O indicates another). The ultimate goal is to answer a question related to this video, choosing the correct option out of five possible answers.<br>It is crucial that you imagine the visual scene as vividly as possible to enhance the accuracy of your response. After selecting your answer, rate your confidence level in this choice on a scale from 1 to 100, where 1 indicates low confidence and 100 signifies high confidence. Please provide a concise one-sentence explanation for your chosen answer. If you are uncertain about the correct option, select the one that seems closest to being correct.<br>Examples: Examples<br>Description: Captions<br>Question: Question<br>Options:<br>A: Option-A. B: Option-B. C: Option-C. D: Option-D. E: Option-E.<br>The prediction, explanation, and confidence is (please response in the format of ’prediction:, explanation: ,confidence:’)<br>Assistant<br>prediction, explanation, confidence |
| --- |

Figure: Figure 5: Qualitative examples of VideoTree keyframes and captions selection. Red options are answered wrongly with uniformly sampled frames. Green options are answered correctly by VideoTree. Best viewed in color.
Refer to caption: /html/2405.19209/assets/x5.png

Experiments Compute Resources. All experiments are conducted on 4 (or less) NVIDIA-A6000 GPU and Azure Cloud APIs (for OpenAI models). The minimal GPU memory requirement is 24GB.

## Appendix D Additional Qualitative Analysis

In [Fig. 5](#A3.F5) we show another visualization from VideoTree.
Here, VideoTree localizes a single key activity (embroidering a cloth) taking place in the video and dynamically expands its constituent frames to answer the question correctly using a minimal number of frames.

## Appendix E License

We will make our code and models publicly accessible.
We use standard licenses from the community and provide the following links to the licenses for the datasets, codes, and models that we used in this paper.

LLoVi: [MIT](https://github.com/CeeZh/LLoVi/blob/main/LICENSE)

LifelongMemory: [MIT](https://github.com/Agentic-Learning-AI-Lab/lifelong-memory/blob/main/LICENSE)

NExT-QA: [MIT](https://github.com/doc-doc/NExT-QA/blob/main/LICENSE)

IntentQA: [IntentQA](https://github.com/JoseponLee/IntentQA?tab=readme-ov-file)

EgoSchema: [Ego4D license](https://ego4ddataset.com/ego4d-license/)

Kmeans-pytorch: [MIT](https://github.com/subhadarship/kmeans_pytorch/blob/master/LICENSE)

PyTorch: [BSD-style](https://github.com/pytorch/pytorch/blob/master/LICENSE)

Huggingface Transformers: [Apache](https://github.com/huggingface/transformers/blob/master/LICENSE)

Torchvision: [BSD 3-Clause](https://github.com/pytorch/vision/blob/master/LICENSE)

SKLearn: [BSD 3-Clause](https://github.com/scikit-learn/scikit-learn/blob/main/COPYING)

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 4
questions: Please see weaknesses.
soundness: 3
strengths: (1) This paper presents an interesting idea by building a query-adaptive and hierarchical video representation to identify key frames.

(2) This paper is well-organized and the writing is clear.
summary: This paper proposes VideoTree, a training-free framework that builds a query-adaptive and hierarchical video representation for LLM reasoning over long-form videos. Specifically, VideoTree first extracts query-relevant information from the input video through an iterative process, progressively refining the selection of keyframes based on their relevance to the query. Then VideoTree incorporates multi-granularity information into a tree-based representation, allowing the model to extract query-relevant details from long videos in a coarse-to-fine manner. Finally, VideoTree aggregates the hierarchical query-relevant information within the tree structure and feeds it into an LLM reasoning model to get the final answer.
weaknesses: (1) From a technical standpoint, the innovation is limited and can be categorized as incremental. Specifically, the query-adaptive visual cluster and the coarse-to-fine strategy for identifying keyframes have been explored in previous work.

(2) In the Introduction Section, the authors mention one of the motivations for VideoTree as the Inability to Capture the Coarse-to-Fine Video Structure. However, this motivation is unconvincing because capturing the coarse-to-fine video structure is merely a method for identifying key frames, rather than a true challenge faced in the field of long video understanding. It seems that the authors are somewhat justifying their proposed approach rather than addressing a broader, established challenge.

(3) In Section 3.1, the authors write that for a cluster $C_i$, they identify the keyframe $F_i$ closest to the centroid vector $\mathbf{c}_i$ and consider it as the keyframe of the $i$th cluster. However, this straightforward method of converting the image to a caption can result in a significant loss of information relevant to the query, leading to potential error propagation. Additionally, this image-level captioning operation overlooks substantial motion information, making it inadequate for addressing queries related to temporal dynamics. 

(4) The experimental results on Video-MME are not inspiring and insufficient, and do not convincingly demonstrate the effectiveness of VideoTree. For instance, why not comparisons with similar methods (e.g., LLoVi, VideoAgent) ?  Additionally, it would be beneficial to see results that incorporate subtitles, as this could provide further insight into how VideoTree performs relative to existing approaches and under different input conditions.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: How does the hierarchical structure impact the LLM’s ability to reason across segments? Does the model see improvements primarily in short-term dependencies or in understanding the overall narrative?

Could VIDEOTREE be adapted for real-time video processing, or is it primarily suited to post-processing of pre-recorded content?

How well does VIDEOTREE generalize to different domains, such as instructional videos, movies, or surveillance footage? Are certain types of video content more compatible with the model's tree-based representation?
soundness: 3
strengths: This paper, VIDEOTREE, introduces a tree-based structure that efficiently organizes video content, making it well-suited for handling long-form videos with complex temporal dependencies. 

The tree-based model can be scaled to accommodate a wide variety of video lengths and complexities, making it versatile and adaptable across different domains.
summary: This paper introduces VIDEOTREE, a training-free framework that enables large language models (LLMs) to perform efficient reasoning on long-form videos. VIDEOTREE builds a hierarchical, query-adaptive video representation by selectively extracting relevant keyframes and organizing them in a tree structure, where coarse-to-fine details are progressively refined based on query relevance. This approach reduces redundancy and informational overload, allowing the model to focus on pertinent video segments, thus improving reasoning accuracy and efficiency. Experiments show that VIDEOTREE outperforms existing training-free methods on long video benchmarks, achieving higher accuracy and faster inference times without additional video-specific training.
weaknesses: The baseline methods evaluated in Video-MME are selected deliberately, the current version does not provide a comprehensive evaluation.

The performance of VIDEOTREE heavily relies on how well the video segments are defined. Inaccurate or suboptimal segmentation could impact the overall representation quality and LLM understanding.

The method assumes compatibility with existing LLMs for video processing, which may limit its effectiveness depending on the specific architecture and capacity of the LLMs being used.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see the Weaknesses.
soundness: 3
strengths: 1.	This paper is well written and easy to understand.
2.	The proposed training-free VIDEOTREE achieves better performance than the strong proprietary GPT-4V model and other MLLMs that were trained on video data on the long split of Video-MME benchmark.
summary: This paper introduces the VIDEOTREE, an adaptive and hierarchical framework for LLM reasoning over long-form videos. VIDEOTREE adaptively extracts query-relevant keyframes from the video input in a coarse-to-fine manner and organizes them into a hierarchical representation, enabling the LLM to effectively handle complex queries. Abundant experiments on three popular datasets EgoSchema, NExT-QA, and Video-MME shows the excellent performance of the VIDEOTREE.
weaknesses: 1. The applicability of the proposed method is limited, as its effectiveness has only been verified on tasks such as multiple-choice questions. How about its performance on other video understanding-related tasks (open-ended VideoQA or text generation), such as action recognition, text-video localization, temporal reasoning tasks, and prediction-related tasks?
2. Although the authors claim to use coarse-to-fine hierarchical feature extraction, essentially, it still involves aggregation at the video frame level. This will prevent the model from effectively extracting fine-grained information within video frames, thereby limiting its performance on finer-grained video understanding tasks.
3. This algorithm requires multiple uses of LLM or VLM. Given the limitations of LLMs or VLMs, such as severe hallucination issues, how do the authors ensure the accuracy of the results obtained each time? For example, in Relevance Scoring, on one hand, Cap(.) is used to obtain captions for keyframes. How can we ensure that critical information is not lost? Additionally, using an LLM to judge relevance to the query, how can we ensure the accuracy of this relevance judgment? Furthermore, is it appropriate to filter and aggregate all video content based solely on the query? For instance, can a simple question like "Please describe the video content" be answered accurately?
4. This method involves a large number of hyperparameters.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In line 450, the authors write "...VideoAgent baseline, which suffers from performance degradation after 11 frames..." However, the x-axis in Figure 3 represents the number of captions. Did the authors mean "captions" instead of "frames"? Are they referring to the number of captioned frames?

2. Line 450: "our method continues improving, generalizing to 62.4 frames". What is the meaning of this fractional frame count? Assuming that they are referring to the number of captioned frames, the question still stands. Was this value averaged over video samples? Did this situation arise because some videos are shorter?

3. The caption of Table 8 states that VideoAgent's avg. LLM calls are estimated. Why weren't real values used? How were they estimated?

4. The prompts in Table 14 and 15 include queries about confidence. How are these confidence values used?

5. What is the reasoning behind the FPS choice (1 FPS for EgoSchema and NExT-QA, 0.125 for Video-MME)? How would the performance change if the FPS were changed?
soundness: 3
strengths: Although hierarchical video representation existed before this paper, they operated in a bottom-up manner. In contrast, the proposed method improves efficiency and effectiveness by employing a top-down approach with dynamic depth. Thus, the method exhibits sufficient novelty. By surpassing the performance of previous training-free methods, the method attempts to demonstrate its significance (However, it falls short against state-of-the-art MLLMs, both proprietary and open-source.) 

The paper explains the method clearly, with diagrams visualizing the information flow. The main text provides a concise overview, which is supplemented by the appendix that provides further details.
summary: This paper introduces VideoTree, a framework that offers a dynamic hierarchical video representation enabling LLMs to reason over long videos.

The method has 3 stages:

- The first stage—adaptive breath expansion—utilizes K-means to cluster visual features extracted from video frames, a captioner to obtain text descriptions, and an LLM to assign a relevance score for each cluster.
- These clusters are expanded according to their relevance scores in the next stage, with the most relevant clusters expanding into two-level trees.
- Finally, in the last stage, this tree is traversed top-to-bottom in temporal order to obtain a textual description of the video, which is then fed into the LLM alongside the query. Intuitively, tree representation aims to reduce the high redundancy of information in video data while preserving the fine-grained details that are relevant to the query.

The authors claim that VideoTree outperforms non-hierarchical methods such as VideoAgent and LLoVi on EgoSchema and NExT-QA benchmarks, obtaining a result comparable to LVNet, which capitalizes on the stronger GPT-4o backbone.

On Video-MME benchmark, which features long videos up to 1 hour in length, VideoTree slightly outperforms the proprietary GPT-4V model but comes up short against GPT-4o and Gemini 1.5 Pro.

Against 6 open-source MLLMs that were extensively trained on video data, VideoTree defeats 4 of them (Table 2) despite being a training-free approach.
weaknesses: Throughout the paper, the authors claim that both redundancy and irrelevance of the information is harmful (e.g., in line 10, "Long-form video understanding has been a challenging task due to the high redundancy in video data and the abundance of query-irrelevant information.").  Although it looks easy to intuitively recognize the detrimental effect of irrelevant information, the claim about redundancy is not well-substantiated, especially in terms of empirical proof. In fact, one could argue that some of the results constitute empirical proof against this claim. Specifically, Figure 3 demonstrates that increasing the number of captions improves performance for both LLoVi and VideoTree. 

Due to temporal redundancy in videos, more captions translate to more redundancy, which should be detrimental according to the authors, yet this seems to improve the accuracy. Moreover, the proposed method—despite filtering out irrelevant content—introduces more redundancy in a different form because of the tree structure. Given that reducing redundancy is a major motivation behind the proposed method, authors should justify their claim about redundancy, either by citing relevant work (if available) or providing empirical proof.

In line 30, the authors write "VIDEOTREE framework achieves better performance than the strong proprietary GPT-4V model and other MLLMs that were extensively trained on video data." I think this sentence seems to suggest VideoTree outperforms all tested MLLMs, which does not seem to be the case, as shown in Table 2. The authors should clarify this point by replacing "other MLLMs" with "many other MLLMs", for example. Overall, the authors should make it clear that the proposed method does not outperform the best proprietary and the best open-source MLLMs. 

Although the fact that it can outperform numerous MLLMs despite being a training-free method is a technical feat, the paper doesn't explain the advantages of this method over an off-the-shelf MLLM. In other words, given that the training cost of MLLM has already been paid, why should we use a training-free approach? Therefore, more justification as to why this training-free method is preferable to using a pretrained MLLM would help emphasize the importance of the method. For instance, the authors could show that MLLMs require more computation and longer inference times.

Finally, the paper does not consider the possibilities for future work. The authors can address this by briefly mentioning some ideas in the conclusion section. This consideration can also improve the paper's position within the literature, thereby highlighting its significance.

Minor writing mistakes:
- Space before dot at line 128.
- Line 285: "recent-proposed" should be "recently proposed".

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: See weaknesses
soundness: 3
strengths: - The paper is well-structured and the flow of idea is easy to follow.
- By dynamically selecting only the most relevant keyframes through Adaptive Breadth Expansion and Relevance-guided Depth Expansion, the method reduces the noise caused by irrelevant or redundant frames. This leads to more accurate reasoning over long videos and avoids overwhelming the model with unnecessary data.
- The method is computationally efficient due to its sparse selection of keyframes and hierarchical tree structure, which reduces the need for dense frame processing.
summary: The paper introduces a training-free framework for long-form video understanding that improves reasoning efficiency and accuracy by constructing a hierarchical query-relevant video representation. Its key contributions are:

1.  *Query-Adaptive Frame Selection*: It dynamically selects keyframes based on their relevance to the query, minimizing the inclusion of redundant or irrelevant information, improving both speed and accuracy. This is done via the following steps:
    - Perform an **Adaptive Breadth Expansion** on the video frames which clusters them based on their similarities, captions each clusters and then score them based on their relevance to the query.

    - Deepen the tree via a **Relevance-guided Depth Expansion** which sub-clusters the most relevant clusters allowing more fine-grained representations of the query-relevant parts of the video.
2. *Training-Free Efficiency*: Unlike many methods that require extensive video-specific training, this method performs well without training. It outperforms existing training-free models on EgoSchema, NExT-QA, and Video-MME, achieving superior performance and faster inference times.
weaknesses: - *Dependence on the initial keyframe selection and the value of k*: During the Adaptive Breadth Expansion, the initial clustering might miscluster some frames, and the error might propagate through the whole tree. How is this accounted for in the process? Is VIDEOTREE's performance highly dependent on the number of clusters (k)? Could you include ablation experiments to test for that?
- *Many moving parts*: The paper thoughtfully aggregates several off-the-shelf models. Would the performance be bottlenecked by a suboptimal captioner or LLM reasoner?
- *Limited Novelty*: The main contribution of the paper is a new way to assemble a tree for long-form video understanding, which is very limited in scope given that the other important parts of the framework are off-the-shelf. Is it possible to extend it to other tasks?