Language Repository for Long Video Understanding

## Abstract

Abstract Language has become a prominent modality in computer vision with the rise of multi-modal LLMs. Despite supporting long context-lengths, their effectiveness in handling long-term information gradually declines with input length. This becomes critical, especially in applications such as long-form video understanding. In this paper, we introduce a Language Repository ( LangRepo ) for LLMs, that maintains concise and structured information as an interpretable (i.e., all-textual) representation. Our repository is updated iteratively based on multi-scale video chunks. We introduce write and read operations that focus on pruning redundancies in text, and extracting information at various temporal scales. The proposed framework is evaluated on zero-shot visual question-answering benchmarks including EgoSchema, NExT-QA, IntentQA and NExT-GQA, showing state-of-the-art performance at its scale. Our code is available at github.com/kkahatapitiya/LangRepo .

###### Keywords:

## 1 Introduction

Video data is central to learning systems that can interact and reason about the world. Yet, they also associate with significant challenges such as increased compute requirements and redundant information, to name a few. This is especially critical in long-term videos. Even so, recent literature on video understanding have progressed so far, enabling reasoning capabilities in hours-long video streams [, ], in contrast to very-limited temporal spans (*e.g*. seconds or minutes) just a few years ago. Such progress is intriguing considering how complex the semantics become when temporal span is increased [, ]. Work on efficient spatio-temporal attention mechanisms [, ], memory management [, ], and large-language-models (LLMs) [, , ] have been key ingredients for such recent improvements.

LLMs, or more-specifically, vision-large-language-models (VLLMs) have been outperforming pure vision models in recent years in all facets, including image-based reasoning [, , ], grounding [, ], video understanding [, , ], and even robotics [, , ]. The sheer model scale and the vast pretraining data have enabled such frameworks to capture world knowledge and semantics, beyond what is possible with visual data only. Besides, the ability to process long context-lengths is also key, as it helps modeling long-term dependencies that are crucial for more-complex reasoning and interactions. However, recent studies show that despite the availability of such context-lengths, the effectiveness of models declines with longer input sequences []. This promotes the search for alternate representations that can compress input language data without losing meaningful information, essentially managing the context utilization of LLMs.

Moreover, the use of text (i.e., language) in modeling has shown numerous benefits such as rich semantics [, , ], ease of information sharing between different specialized-models [] or modalities [, ], and interpretability [, ]. Among such, interpretability has a huge societal impact in the age of LLMs, to manage adversities such as bias [, ] and hallucinations [, ]. Simply put, it enables human observers to understand and monitor what really happens within models. Hence, interpretable representations have also been of interest to the community, in place of latent representations [, ].

Figure: Figure 1: Overview of our Language Repository (LangRepo): We propose an all-textual repository of visual information that updates iteratively, creating a multi-scale and interpretable representation. It extracts information from captions (generated by a VLLM [, ]) corresponding to video chunks. In write-to-repo, we prune and rephrase input descriptions, creating concise entries in the repository. In read-from-repo, such language descriptions (together with any optional metadata, *e.g*., timestamps) at multiple semantic-scales are summarized to generate outputs suited for video VQA. Here, rephrase and summarize are LLM-calls. We also compare LangRepo against state-of-the-art methods, showing strong performance at its scale.
Refer to caption: x1.png

Motivated by the above, we introduce Language Repository (LangRepo), an interpretable representation for LLMs that updates iteratively. It consumes input captions corresponding to video chunks
, as shown in Fig. 1 (left).
As LangRepo is all-textual, we rely on text-based operations to write and read information. The write operation (write-to-repo) prunes redundant text, creating concise descriptions that keep the context-utilization of LLMs in-check. Its iterative application with increasingly-longer chunks enables it to learn high-level semantics (*e.g*. long temporal dependencies). The read operation (read-from-repo) extracts such stored language information at various temporal scales, together with other optional metadata within the repository entries (*e.g*. timestamps). Altogether, our proposed framework is applied to long-term video reasoning tasks including visual question-answering (VQA) on EgoSchema [], NExT-QA [] and IntentQA [], and visually-grounded VQA on NExT-GQA [], showing strong performance at its scale, as given in Fig. 1 (right). Finally, we ablate our design decisions, providing insights on key components.

## 2 Related work

#### 2.0.0.1 Long-video understanding:

Video models have progressed over the years, going from primitive recognition tasks [, ] to complex and fine-grained reasoning tasks [, , , , ] over long horizons. Both convolutional baselines [, , ] and transformer architectures [, , ] have explored research directions such as multi-scale representations [, , ], efficiency concerns associated with heavy spatio-temporal computations [, ], and handling redundant information within video inputs [, ]. More recently, long-video understanding has made a leap forward thanks to benchmark datasets [, , ] and model improvements [, , ], validating the importance of modeling complex interactions that happen over long periods of time. Still, the sub-par performance of SOTA models on such benchmarks suggests the room for improvement.

#### 2.0.0.2 Long-context models:

Even before the age of LLMs, models based on convolutions [, , , ], recurrent blocks [, , ] or transformers [, , ] have exploited long-term dependencies, especially in the context of video understanding [, ] and reinforcement learning [, ]. With the rise of LLMs, scaling laws have revealed the importance of longer contexts even more [, ], and, thanks to the breakthroughs such as sparse processing [, ], caching [, , ], model-sharding [, , ], and efficient attention [, ], such long-context LLMs have become a reality. Even with very large context lengths, maintaining the effectiveness of reasoning over longer inputs is challenging [, , ]. This motivates us to think about concise representations that can better-utilize LLM context.

#### 2.0.0.3 Compressing representations:

When handling heavy inputs, deep learning models have relied on compressed representations. It may come in the form of pruning [, ], latent memory [, , ], or external feature banks [], to name a few. Despite the intuitive novelties and efficiency gains of such techniques, it is challenging to realize which information get preserved, and how semantically-meaningful they are post-compression. An interpretable representation that supports compression, if available, may shed light on such details.

#### 2.0.0.4 Language as an interpretable modality:

More-recently, language has emerged as a dominant modality in computer vision due to its strong generalization capabilities [, ]. It can also act as a bridge between various domain-specific models [], other modalities [, ], and even human instructions [, ], showing intriguing applications in domains such as chat agents (*e.g*. ChatGPT, Bard) and robot-learning [, ]. Since language is interpretable, it enables humans to interact with models more-naturally and make sense of model predictions.

Motivated by the above, we introduce an interpretable language representation that can (1) prune redundant information, and (2) extract multi-scale (or, high-level) semantics, enabling better context-utilization within LLMs.

## 3 Observations on long-range inputs

In this section, we investigate how LLMs perform with increasing inputs lengths (i.e., #tokens). Recent LLMs with very-large context lengths such as Gemini-Pro-1.5 [] (1M tokens) or Claude-2.1 (200k tokens), can support extremely long input sequences. Yet, when feeding longer inputs, the reasoning capabilities (especially, long-term reasoning) of such models diminish. This behavior is also observed in concurrent work [], and evident in benchmark results of state-of-the-art models [, ] (i.e., better performance with shorter inputs, or fewer video frames). To better investigate this in our setup, we evaluate VQA performance on standard long-term video understanding benchmarks while varying the input length (see Table 1). We consider frame/short-clip captions extracted using a VLLM at a baseline framerate (1$\times$) as inputs (introduced in []). We either subsample (0.5$\times$) or replicate (2$\times$) the captions, decreasing/increasing the input lengths of a question-answering LLM, namely, Mistral-7B [] with 8k (or, theoretical 128k) context length. All inputs fit within the context, without any overflow. The observation from this study is consistent: even though the context length of the LLM is sufficient to process given inputs, the effectiveness of its predictions (shown by VQA performance) drops with longer inputs. This motivates us to introduces a concise language representation that preserves important details of long-range inputs, while pruning any redundant information.

**Table 1: Observations on increasing input length: We evaluate the performance of a question-answering LLM (Mistral-7B []) at different input lengths, on multiple long-video reasoning benchmarks: EgoSchema [], NExT-QA [] and IntentQA []. Here, $1\times$ corresponds to captions extracted by a VLLM at a default frame-rate (and, $0.5\times$/$2\times$ corresponds to a compression/expansion of them by a factor of 2). We see that even though the context length of the LLM can fit all inputs without any overflow, the effectiveness of predictions (i.e., VQA performance) decreases with input length.**
| Dataset | Captions per-video |  |  |
| --- | --- | --- | --- |
| 0.5$\times$ | 1$\times$ | 2$\times$ |  |
| EgoSchema (subset) [] | 49.8 | 48.8 | 46.8 |
| NExT-QA [] | 48.2 | 48.2 | 46.9 |
| IntentQA [] | 47.1 | 46.9 | 45.2 |

## 4 Language Repository

Figure: Figure 2: Detailed view of our Language Repository (LangRepo): Here we present the write and read operations within LangRepo. Given short-captions corresponding to video chunks, write-to-repo first prunes redundant captions within each chunk. The same process is iteratively applied on increasingly longer (or, higher-level) chunks— that are already within the repository— to generate multi-scale repository entries. Pruning consists of two stages: (1) grouping most similar captions based on embedding (eg: CLIP []) similarities between two subsets, and (2) rephrasing grouped captions with an LLM-call. The resulting LangRepo will include rephrased-captions and any optional metadata (*e.g*. #occurrences, timestamps). Next, read-from-repo generates concise descriptions for different semantic levels by summarizing the multi-scale language representation, which is also an LLM-call.
Refer to caption: x2.png

We present a Language Repository (LangRepo) that iteratively updates with multi-scale descriptions from video chunks. In contrast to external feature banks [] or learnable latent memory representations [, , ], our proposal has a few key advantages: (1) it requires no training (i.e., zero-shot), and (2) it is compatible with both LLM-based processing and human interpretation, as it is fully-textual, i.e., it exists in language-space instead of a latent-space. LangRepo consists of two main operations: (1) information writing (write-to-repo), which prunes redundancies and iteratively updates language descriptions based on increasingly-longer video chunks, and (2) information reading (read-from-repo), which extracts preserved descriptions (with any optional metadata) in multiple temporal scales. We show a detailed view of these operations in Fig. 2, and further elaborate in the following subsections.

Consider a long video that is split in to $n$ non-overlapping chunks, denoted as $V=\{v_{i}\;|\;i=1,\cdots,n\}$. Assume that we already have frame or short-clip captions extracted by a VLLM (*e.g*. LLaVA []) corresponding to such chunks, denoted by $C=\{c_{i}\;|\;i=1,\cdots,n\}$. Here, each chunk may consist of $p$ such captions as in $c_{i}=\{c_{ij}\;|\;j=1,\cdots,p\}$. Altogether, $V$ is represented by $n\times p$ captions which we consider as inputs to our framework.

### 4.1 Writing to repository

We intend to create a concise, all-textual representation with multiple scales (or, semantic-levels) of information. Hence, our writing operation is text-based, and applied iteratively on different scales of input. In the first iteration, it consumes low-level details in each chunk $i$, in the form of captions $c_{i}$, generating initial entries to the repository repo${}^{1}(i)$, or $r^{1}_{i}$.

$$ $\displaystyle r^{1}_{i}$ $\displaystyle=\texttt{write-to-repo}^{1}(c_{i}).$ (1) $$

In each subsequent iteration $k$, previous repo entries of iteration $k-1$ are re-combined into longer chunks and processed in the same way, generating information for higher semantic-levels.

$$ $\displaystyle r^{k}_{i^{\prime}}$ $\displaystyle=\texttt{write-to-repo}^{k}(\texttt{re-chunk}([\cdots,\;r^{k-1}_{ i-1},\;r^{k-1}_{i},\;r^{k-1}_{i+1},\;\cdots])_{i^{\prime}}).$ (2) $$

Here, re-chunk($\cdot$) denotes the creation of longer chunks within the repository. More specifically, we simply concatenate (denoted by $[\cdot]$) all entries from previous iteration, and split them again into fewer number of chunks (hence, longer chunk size). Note that $i^{\prime}$ in the above equation is not the same as the previous chunk indexing $i$, as we may have different (usually, fewer) number of chunks in each subsequent iteration.

Each write operation involves two stages: (1) Grouping redundant text, and (2) Rephrasing, which are detailed below.

#### 4.1.0.1 Grouping redundant text:

Given textual descriptions of a video chunk (i.e., captions in the first write iteration, or previous repo descriptions in subsequent iterations), we plan to identify most-similar ones and merge them as a single description. Without loss of generality, let us consider the first write iteration, for which the input is in the form of $c_{i}=\{c_{ij}\;|\;j=1,\cdots,p\}$. Inspired by [], we first split the captions of each chunk into two sets, namely, source (src) captions $c_{\text{src},i}$ and destination (dst) captions $c_{\text{dst},i}$. Let us drop the chunk index $i$ for brevity. Here, dst captions $c_{\text{dst}}$ are sampled uniformly distributed across the temporal span of a chunk, while all the rest are considered as src captions $c_{\text{src}}$ (see Fig. 2).

$$ $\displaystyle c_{\text{dst}},\;c_{\text{src}}$ $\displaystyle=\texttt{split}(c).$ (3) $$

Here, we usually have fewer dst captions (i.e., $|c_{\text{dst}}|<|c_{\text{src}}|$). Next, we embed all captions using a text-encoder (*e.g*. CLIP []), and compute the cosine similarity of each pair between src-dst sets to find most-similar matches.

$$ $\displaystyle\text{sim}_{\text{src-dst}}$ $\displaystyle=\texttt{sim}(\texttt{encode}(c_{\text{src}}),\;\texttt{encode}(c _{\text{dst}})).$ (4) $$

Based on the similarity matrix above ($\text{sim}_{\text{src-dst}}$), we then prune the highest $x\%$ similarities by grouping such source captions with their corresponding destination matches, forming a set of grouped descriptions $c_{\text{grp}}$ for the given chunk. Refer to the color-coded captions after ‘Group’ in Fig. 2.

$$ $\displaystyle c_{\text{grp}}$ $\displaystyle=\texttt{group}(c_{\text{dst}},\;c_{\text{src}},\;\text{sim}_{ \text{src-dst}},\;x).$ (5) $$

Here $x\%$ is a pre-defined hyperparameter (*e.g*. 50%). Finally, such grouped descriptions go through a rephrasing operation prior to entering the repository.

Figure: Figure 3: LLM prompt templates in LangRepo: Here, we show the zero-shot prompt templates used for rephrasing ($\texttt{template}_{\texttt{reph}}$) and summarizing ($\texttt{template}_{\texttt{sum}}$) operations. Rephrase prompt needs a list of grouped captions as input, while its output adheres to more-strict requirements (*e.g*. same order, same number of list items) needed for correct parsing. Summarize prompt takes in each repository entry and generates a more-flexible (i.e., open-ended) output, while also conditioning on the question.
Refer to caption: x3.png

#### 4.1.0.2 Rephrasing:

Grouped captions $c_{\text{grp}}$ of each chunk are rephrased via an LLM-call. This allows redundant information within each group to be dropped, while generating a concise and coherent description. We first form a list of grouped captions, where each list item corresponds to a single group (i.e., a dst caption and any one or more src captions matched to it), and feed it to the LLM, wrapped in a rephrasing-template ($\texttt{template}_{\texttt{reph}}$) as shown in Fig. 3 (top).

$$ $\displaystyle{c_{\text{reph}}}$ $\displaystyle=\texttt{rephrase}(\texttt{template}_{\texttt{reph}}(c_{\text{grp }},\;l)).$ (6) $$

Here, $l$ denotes the number of grouped captions in the given chunk. The LLM output (${c_{\text{reph}}}$) is restricted to be a list in the same order with the same number of items, where each item is a single concise sentence. Finally, such rephrased descriptions together with other metadata such as timestamps ($t$) and number of occurrences ($o$) are written in the repository.

$$ $\displaystyle r^{1}$ $\displaystyle=\{(c_{\text{reph},j},\;t_{j},\;o_{j})\;|\;j=1,\;\cdots,\;p^{ \prime}\}.$ (7) $$

Note that here $p^{\prime}<p$ as we have grouped and rephrased $x\%$ of most-similar captions. Alongside each description in a repository entry, $t$ maintains a growing list of timestamps corresponding to its founding captions, whereas the occurrences counter ($o$) keeps track of the number of such captions. A qualitative example of a repository entry is given in Fig. 4.

In subsequent iterations, the same operations apply when writing multi-scale entries. The only difference is the change in input, which now constitutes of previous repo entries re-combined into high-level chunks (i.e., $c\rightarrow\texttt{re-chunk}([\cdots,\;r^{k-1},\;\cdots])$ in Eq. 3). Each new iteration generates information corresponding to a higher semantic-level (i.e., going from short-range to long-range dependencies), forming our multi-scale language representation.

Figure: Figure 4: A qualitative example of a LangRepo entry: Given a video chunk, redundant captions are first grouped together during pruning operation. During rephrasing, such groups are more-concisely written to the repository, along with additional metadata. Other non-redundant captions are written directly. This process is continued iteratively with increasingly-longer chunks, creating multi-scale repository entries (refer Fig. A.1 for a more-detailed view). Finally, such descriptions from various temporal scales are read to generate the output.
Refer to caption: x4.png

### 4.2 Reading from repository

As we make a single VQA prediction for a given long video— instead of making predictions every chunk— our read operation (read-from-repo) is applied only after fully-forming the multi-scale repository (i.e., after all write iterations). The repo entries after $K$ write iterations can be denoted as
$\{r^{k}\;|\;k=1,\cdots,K\}$ where each scale ($r^{k}$) may consist of multiple entries $\{\cdots,\;r^{k}_{i-1},\;r^{k}_{i},\;r^{k}_{i+1},\;\cdots\}$. When reading, we generate summaries for each entry in the repo separately, allowing it to focus on varying temporal spans. More specifically, each entry goes through a summarizing-template ($\texttt{template}_{\texttt{sum}}$) as shown in Fig. 3 (bottom), and the resulting prompt is fed to the LLM.

$$ $\displaystyle{d^{k}_{i}}$ $\displaystyle=\texttt{read-from-repo}(r^{k}_{i})=\texttt{summarize}(\texttt{ template}_{\texttt{sum}}(r^{k}_{i})).$ (8) $$

Here, $d^{k}_{i}$ corresponds to the output description of each entry $i$ in the repository, at the respective scale $k$. Optionally, we can make use of additional metadata such as timestamps and #occurrences, by prompting the read operation with descriptions of repo entries formatted as “[timestamps] description ($\times$#occurrences)” (see Fig. 4). Finally, we concatenate all output descriptions and prompt the LLM again to generate the answer prediction.

$$ ans $\displaystyle=\texttt{vqa}([\cdots,d^{k}_{i},\cdots]).$ (9) $$

### 4.3 Design decisions

#### 4.3.0.1 Similarity-based pruning:

We notice that the short captions generated by the VLLM can be highly-redundant, as it has a limited temporal span. Such excess details can adversely affect the performance (see Table 1), while also wasting the LLM context. This motivates us to prune redundancies. We consider prompting the LLM directly to identify and rephrase redundant information. However, the outputs in this setup can be noisy and lack of any structure that is useful for parsing. In other words, although redundancies get pruned, there is limited controllability and inability of identifying what gets pruned. Hence, we decide to delegate the function of identifying redundancies to a separate module: a similarity-based grouping with the help of text embeddings. This gives more control on what to prune and how much to prune, while generating outputs that can be parsed to extract other useful metadata (*e.g*. timestamps).

#### 4.3.0.2 Processing videos as chunks:

Our decision to consume longer videos as chunks is motivated by prior work [, ]. It allows us to not lose short-term details, while also keeping track of long-term dependencies via multi-scale processing.
Additionally, although not explored in the scope of this paper, such a setup integrates well with temporally-fine-grained prediction tasks, where an LLM needs to make multiple predictions over time.

#### 4.3.0.3 Choice of metadata:

To avoid the loss of important details during pruning, we maintain additional metadata in our LangRepo. Since captions across time can be grouped together in a single repo description, we save their timestamps as a separate field. This can help with temporal reasoning questions. We also update an occurrence counter, which shows the number of captions grouped within a single description. This can act as a weight, to help in cases such as counting or identifying repetitive events.

#### 4.3.0.4 All-textual repository:

Instead of being a latent representation [, , ], our LangRepo is all-textual. This promotes interpretability for human observers, while also being a more-natural form of structure for LLM-based processing. Additionally, our implementation can be formulated to be zero-shot, without requiring any training or finetuning.

#### 4.3.0.5 Classifier for close-ended VQA:

The standard multiple-choice question-answering setup considers a generative classifier. Meaning, an LLM is prompted to generate the correct answer option among multiple-choices, directly as next-token prediction. Another approach used in NLP literature is the log-likelihood based classification. Here, the LLM is prompted separately for each of the multiple choices with a template such as “Question: Answer-option”. The choice that maximises the log-likelihood of predicted tokens (i.e., tokens corresponding to Answer-option) is selected as the correct answer. This is a more-natural setup for close-ended VQA since it avoids hallucination. Among these classifiers, we find the latter to be better-performing. Yet, it is more-sensitive to the prompt template. We direct the reader to supplementary A.1 for more details.

## 5 Experiments

### 5.1 Implementation details

In our experiments, we rely on frame or short-clip captions pre-extracted using VLLMs, as given in []. For EgoSchema [], LaViLa []
is used to generate captions at 1fps. For NExT-QA [] data used in other benchmarks, LLaVA-1.5 [] generates captions at 0.5fps. By default, we use either Mistral-7B [] (w/ 7B parameters) or Mixtral-8$\times$7B [] (w/ 12B active parameters) as our LLMs. As the text encoder in similarity-based pruning, we use CLIP-L/14 []. Note that all the models used in our framework are open-source and within a reasonable model-scale, promoting further research in academic settings. We do zero-shot inference on all datasets without any finetuning, evaluating the performance on long-term video VQA benchmarks.

### 5.2 Dataset details

#### 5.2.0.1 EgoSchema:

EgoSchema [] is a long-video VQA dataset derived from Ego4D [], by semi-automatically generating and verifying mutliple-choice questions. Each video is 3 minutes long, and the questions are filtered to have a long temporal certificate (i.e., at least 100 seconds of a video is required to answer the question). The public validation subset consists of 500 videos, each with 5 answer-choices and the correct ground-truth answer. The fullset of 5K videos is hosted as a Kaggle challenge. There is no training split for this dataset, motivating zero-shot evaluation. In our experiments, we report the performance for both subset and fullset.

#### 5.2.0.2 NExT-QA:

NExT-QA [] is a popular VQA dataset with videos up to 2 minutes long, at an average of 44 seconds. It consists of 52k open-ended questions and 48k close-ended questions (i.e., multiple-choice with 5 answer options) in total. The questions are further classified into temporal, causal, or descriptive categories, to enable evaluation of different reasoning capabilities of models. We consider zero-shot evaluation on validation set of 5k multiple-choice questions, and report the performance on each category.

#### 5.2.0.3 IntentQA:

IntentQA [] is based on the same NExT-QA videos (specifically, the ones for temporal and causal reasoning), yet re-purposed focusing on intent-related questions. A new set of 16k multiple-choice questions are provided for 4.3k videos, each with 5 answer choices. These questions are further classified as why?, how? or before/after, based on the appearance of the same keywords in the question. In our evaluation, we focus on zero-shot setting on test set of 2k questions, reporting performance for each category.

#### 5.2.0.4 NExT-GQA:

NExT-GQA [] is a visually-grounded VQA dataset, which provides additional 10.5K temporal grounding annotations on top of original
NExT-QA [] data. The test split contains 5.5k question-answer pairs, together with grounding labels that convey the temporal span needed to answer the question. Here, we focus on zero-shot inference similar to [], and report multiple metrics including Intersection-over-Prediction (IoP) that measures the overlap w.r.t. the predicted window, Intersection-over-Union (IoU) that measures the overlap w.r.t. the union of ground-truth and predicted windows, and Acc@GQA that measures the accuracy of correctly-grounded predictions.

### 5.3 Main results

**Table 2: Results on EgoSchema []: We present comparisons with state-of-the-art models on EgoSchema subset (500-videos) and fullset (5000-videos). We focus on the zero-shot setting. LangRepo shows a strong performance at its scale. Models with significantly higher #parameters or video-caption pretraining${}^{\dagger}$ are de-emphasized.**
| Model | Params | Subset (%) | Fullset (%) |
| --- | --- | --- | --- |
| finetuned |  |  |  |
| MC-ViT-L [] | 424M | 62.6 | 44.4 |
| ImageViT [] | 1B | 40.8 | 30.9 |
| ShortViViT [] | 1B | 47.9 | 31.0 |
| LongViViT [] | 1B | 56.8 | 33.3 |
| zero-shot |  |  |  |
| VIOLET[] | 198M | - | 19.9 |
| InternVideo [] | 478M | - | 32.1 |
| FrozenBiLM [] | 890M | - | 26.9 |
| \rowfont SeViLA${}^{\dagger}$ [] | 4B | 25.7 | 22.7 |
| mPLUG-Owl [] | 7B | - | 31.1 |
| LLoVi [] | 7B | 50.8 | 33.5 |
| Vamos [] | 13B | - | 36.7 |
| \rowfont Vamos [] (GPT-3.5) | 175B | - | 41.2 |
| \rowfont LLoVi [] (GPT-3.5) | 175B | 57.6 | 50.3 |
| \rowfont Vamos [] (GPT-4) | 1.8T | - | 48.3 |
| LangRepo (ours) | 7B | 60.8 | 38.9 |
| LangRepo (ours) | 12B | 66.2 | 41.2 |

#### 5.3.0.1 EgoSchema:

In Table 2, we present the VQA performance of LangRepo on standard EgoSchema [] splits, comparing with other state-of-the-art frameworks. Here, we focus on zero-shot evaluation, yet also report finetuned setting (i.e., any downstream-data-specific training) for completeness. We consider Mistral-7B [] and Mixtral-8$\times$7B [] as the choice of LLMs in our setup, both with reasonable model scales (7B and 12B active parameters, respectively). We de-emphasize models with significantly-higher #parameters (*e.g*. 175B GPT-3.5, or 1.8T GPT-4 variants), and the ones that use video-caption pretraining. LangRepo shows significantly-better performance compared to other methods at a similar scale, validating its effectiveness in long-video reasoning. We achieve $+10.1\%$ on fullset compared to mPLUG-Owl [], $+15.4\%$ on subset and $+7.7\%$ on fullset compared to LLoVi [] (w/ Mistral-7B []), and $+4.5\%$ on fullset compared to Vamos [] (w/ Llama2-13B []).

**Table 3: Results on NExT-QA []: We compare LangRepo against state-of-the-art methods on NExT-QA validation set, highlighting standard splits: causal, temporal and descriptive. We focus on the zero-shot setting. Our method shows strong performance at its scale. Models with significantly higher #parameters or video-caption pretraining${}^{\dagger}$ are de-emphasized.**
| Model | Params | Causal (%) | Temporal (%) | Descriptive (%) | All (%) |
| --- | --- | --- | --- | --- | --- |
| finetuned |  |  |  |  |  |
| CoVGT [] | 149M | 58.8 | 57.4 | 69.3 | 60.0 |
| SeViTFiD [] | 215M | - | - | - | 60.6 |
| HiTeA [] | 297M | 62.4 | 58.3 | 75.6 | 63.1 |
| MC-ViT-L [] | 424M | - | - | - | 65.0 |
| InternVideo [] | 478M | 62.5 | 58.5 | 75.8 | 63.2 |
| BLIP-2 [] | 4B | 70.1 | 65.2 | 80.1 | 70.1 |
| \rowfont SeViLA${}^{\dagger}$ [] | 4B | 74.2 | 69.4 | 81.3 | 73.8 |
| LLama-VQA [] | 7B | 72.7 | 69.2 | 75.8 | 72.0 |
| Vamos [] | 7B | 72.6 | 69.6 | 78.0 | 72.5 |
| zero-shot |  |  |  |  |  |
| VFC [] | 164M | 45.4 | 51.6 | 64.1 | 51.5 |
| InternVideo [] | 478M | 43.4 | 48.0 | 65.1 | 49.1 |
| \rowfont SeViLA${}^{\dagger}$ [] | 4B | 61.3 | 61.5 | 75.6 | 63.6 |
| Mistral [] | 7B | 51.0 | 48.1 | 57.4 | 51.1 |
| LLoVi [] | 7B | 55.6 | 47.9 | 63.2 | 54.3 |
| \rowfont ViperGPT [] | 175B | - | - | - | 60.0 |
| \rowfont LLoVi [] (GPT-4) | 1.8T | 69.5 | 61.0 | 75.6 | 67.7 |
| LangRepo (ours) | 7B | 57.8 | 45.7 | 61.9 | 54.6 |
| LangRepo (ours) | 12B | 64.4 | 51.4 | 69.1 | 60.9 |

#### 5.3.0.2 NExT-QA:

In Table 3, we report the performance of LangRepo on standard NExT-QA [] validation splits (Causal, Temporal and Descriptive) and the full validation set. On zero-shot evaluation, our framework outperforms other methods consistently. Compared to smaller models, we gain $+11.8\%$ over InternVideo [] and $+9.4\%$ over VFC []. Compared to models of similar scale, we gain $+9.8\%$ over baseline Mistral-7B [] and $+6.6\%$ over LLoVi (w/ Mistral-7B) []. We de-emphasize LLoVi variant with GPT-4 due to its whopping scale of 1.8T parameters. We also de-emphasize SeViLA [] which is pretrained with video-captions, whereas we rely on LLaVA-1.5 [] captions that has not seen any video pretraining. Finally, we observe that LangRepo outperforms competition on semantic splits showing the generalization of our language representation.

**Table 4: Results on IntentQA []: We compare LangRepo against state-of-the-art methods on IntentQA test set, highlighting standard splits: why?, how? and before/after. We focus on the zero-shot setting. Our method shows strong performance at its scale. Models with significantly higher #parameters or video-caption pretraining${}^{\dagger}$ are de-emphasized.**
| Model | Params | Why? (%) | How? (%) | Before/After (%) | All (%) |
| --- | --- | --- | --- | --- | --- |
| finetuned |  |  |  |  |  |
| HQGA [] | 46M | 48.2 | 54.3 | 41.7 | 47.7 |
| VGT [] | 511M | 51.4 | 56.0 | 47.6 | 51.3 |
| Vamos [] | 7B | 69.5 | 70.2 | 65.0 | 68.5 |
| \rowfont BlindGPT [] | 175B | 52.2 | 61.3 | 43.4 | 51.6 |
| \rowfont CaVIR [] | 175B | 58.4 | 65.5 | 50.5 | 57.6 |
| zero-shot |  |  |  |  |  |
| \rowfont SeViLA${}^{\dagger}$ [] | 4B | - | - | - | 60.9 |
| Mistral[] | 7B | 52.7 | 55.4 | 41.5 | 50.4 |
| LLoVi [] | 7B | 57.9 | 55.4 | 42.3 | 53.6 |
| \rowfont LLoVi [](GPT-4) | 1.8T | 68.4 | 67.4 | 51.1 | 64.0 |
| LangRepo (ours) | 7B | 56.9 | 60.2 | 42.1 | 53.8 |
| LangRepo (ours) | 12B | 62.8 | 62.4 | 47.8 | 59.1 |

#### 5.3.0.3 Intent-QA:

In Table 4, we evaluate our zero-shot framework against other state-of-the-art models on Intent-QA [] test splits (Why?, How? and Before/After) and the full test set. LangRepo outperform comparable models with similar scale consistently, showing gains of $+8.7\%$ over baseline Mistral-7B [] and $+5.5\%$ over LLoVi [] (w/ Mistral-7B). Again, we de-emphasize significantly larger models (LLoVi w/ GPT-4) and models pretrained with video-captions (SeViLA []), whereas ours rely on LLaVA-1.5 [] frame-level captions.

**Table 5: Results on NExT-GQA []: We compare LangRepo against state-of-the-art methods on NExT-GQA test set. We focus on the zero-shot setting. Our method shows strong performance at its scale. Models with significantly higher #parameters or video-caption pretraining${}^{\dagger}$ are de-emphasized.**
| Model | Params | mIoP | IoP@0.5 | mIoU | IoU@0.5 | Acc@GQA |
| --- | --- | --- | --- | --- | --- | --- |
| weakly-supervised |  |  |  |  |  |  |
| IGV [] | 110M | 21.4 | 18.9 | 14.0 | 9.6 | 10.2 |
| Temp[CLIP] [, ] | 130M | 25.7 | 25.5 | 12.1 | 8.9 | 16.0 |
| FrozenBiLM [, ] | 1B | 24.2 | 23.7 | 9.6 | 6.1 | 17.5 |
| \rowfont SeViLA${}^{\dagger}$ [] | 4B | 29.5 | 22.9 | 21.7 | 13.8 | 16.6 |
| zero-shot |  |  |  |  |  |  |
| Mistral [] | 7B | 20.4 | 20.2 | 8.7 | 5.9 | 9.2 |
| LLoVi [] | 7B | 20.7 | 20.5 | 8.7 | 6.0 | 11.2 |
| \rowfont LLoVi [] (GPT-4) | 1.8T | 37.3 | 36.9 | 20.0 | 15.3 | 24.3 |
| LangRepo (ours) | 7B | 20.3 | 20.0 | 8.7 | 6.0 | 11.2 |
| LangRepo (ours) | 12B | 31.3 | 28.7 | 18.5 | 12.2 | 17.1 |

#### 5.3.0.4 NExT-GQA:

In Table 5, we compare the performance of LangRepo with state-of-the-art models on NExT-GQA. Our method achieves a strong performance at its scale, outperforming baseline Mistral-7B [] by $+7.9\%$ and LLoVi [] (w/ Mistral-7B) by $+5.9\%$ on Acc@GQA metric. Despite being zero-shot, it is also competitive with weakly-supervised baselines. Here, we de-emphasize significantly-larger models (LLoVi w/ GPT-4) and models pretrained with video-captions (SeViLA []).

### 5.4 Ablation study

**Table 6: Ablations on EgoSchema []: We evaluate different design decisions of our framework on EgoSchema 500-video subset for zero-shot video VQA.**
| LLM | Scale | Acc. (%) |
| --- | --- | --- |
| Llama2 [] | 13B | 43.0 |
| Mistral [] | 7B | 50.8 |

#### 5.4.0.1 Choice of backbone LLM, text encoder and classifier:

We consider different open-source LLM-backbones in our setup, namely, LLama2 [] and Mistral []. We ablate the choice of LLM within LLoVi [] framework, as shown in Table LABEL:tab:ablation:llm_choice. We observe that Mistral-7B is significantly better at video reasoning compared to LLama2-13B. Next, we consider different text encoders to embed our text descriptions prior to pruning. Namely, we use either CLIP-L/14 [] or Sentence-T5-XL []. As presented in Table LABEL:tab:ablation:text_encoder. Surprisingly, CLIP outperforms Sentence-T5 that is trained with a sentence-level objective (which is expected to better align with our caption-similarity computation). Finally, we evaluate different classifiers used for close-ended (i.e., multiple-choice question) VQA setups (see Table LABEL:tab:ablation:vqa_classifier). Even though the generative classifier is commonly-used in LLM literature, we observe that the log-likelihood classifier performs better. Such performance is also intuitive as the latter constrains predictions within the given answer choices (hence, no hallucination). Based on this observation, we adopt the log-likelihood classifier in our main experiments.

#### 5.4.0.2 Repository setup and metadata:

In the formulation of LangRepo we ablate different hyperparameter settings related to the number of repo-updates (#iterations), the number of video chunks in each iteration (#chunks), and multiple temporal-scales considered when reading data in repository. In Table LABEL:tab:ablation:memory_setup, we make two observations: (1) more update iterations with finer chunks (higher #chunks per iteration) can preserve more-useful information, and (2) reading information in multiple temporal-scales is consistently better. Moreover, we consider optional metadata that can help preserve information that may get lost when pruning (*e.g*. temporal ordering, or repetitive captions), namely, timestamps and #occurrences (i.e., the number of captions grouped within each repo description). We see in Table LABEL:tab:ablation:metadata that #occurrences help weigh each description when summarizing, resulting in better performance. However, timestamps do not provide meaningful improvement in our setup, in the context of EgoSchema VQA.

#### 5.4.0.3 Input format and length:

We also consider different ways of consuming long video data, either as a whole or as chunks (see Table LABEL:tab:ablation:streaming_setup). Among these options, processing as chunks enables preserving more fine-grained details in LLM outputs. Our repository setup provides further improvement showing its effectiveness over the baseline with the same chunk-based processing. Finally, we re-visit the experiment on how the input length affects the effectiveness of LLMs, presented in Table 1. In Table LABEL:tab:ablation:input_length, we show that LangRepo provide more-stable performance with increasing input lengths, in contrast to baselines, validating how it addresses this shortcoming.

## 6 Conclusion

In this paper, we introduced a Language Repository (LangRepo), which reads and writes textual information corresponding to video chunks, as a concise, multi-scale and interpretable language representation, together with additional metadata. Both our write-to-repo and read-from-repo operations are text-based and implemented as calls to a backbone LLM. Our empirical results show the superior performance of LangRepo on multiple long-video reasoning benchmarks at its respective scale, while also being (1) less-prone to performance drops due to increasing input lengths, and (2) interpretable, enabling easier human intervention if and when needed.

#### 6.0.0.1 Acknowledgements:

This work was supported by the SUNY Empire Innovation Program (EIP). We thank the Robotics Lab at SBU for helpful discussions, and especially Xiang Li, for managing the lab servers and promptly responding to all the infrastructure-related issues.

## A Appendix

### A.1 Prompting for VQA

As the evaluation setup, we consider multiple-choice visual question-answering (VQA) on long videos. Given the close-ended answer formulation, we can consider two different classifiers to make the prediction: (1) a Generative classifier, which directly generates the answer choice, or (2) a Log-likelihood classifier, which select the most-probable choice based on the joint-probability of tokens in each answer option given the description and the question. As we discussed in Sec. 4.3, the latter generally performs better, as it is less-prone to hallucinations (i.e., prediction is explicitly constrained to answer choices). However, it is also sensitive to the prompts we use. Hence, we include a discussion on prompting in the following subsections.

#### A.1.0.1 Generative classifier:

Here, we direcly prompt the LLM to generate the correct answer, conditioned on the descriptions generated by LangRepo, the question and the answer options (inspired by []). To make sure that the output can be parsed, we provide additional guiding instructions and any syntax specific to the LLM (Mistral []). This also discourages any hallucinations. On all benchmarks, we use the common prompt given below.

‘‘
 
 You are a helpful expert in first person view video analysis. 
 Please provide a single-letter answer (A, B, C, D, E) to the following multiple-choice question, and your answer must be one of the letters (A, B, C, D, or E). You must not provide any other response or explanation. You are given some language descriptions of a first person view video.
The video is ${duration} seconds long. Here are the descriptions: ${description}.\n You are going to answer a multiple choice question based on the descriptions, and your answer should be a single letter chosen from the choices.\n Here is the question: ${question}.\n Here are the choices.\n A: ${optionA}\n B: ${optionB}\n C: ${optionC}\n D: ${optionD}\n E: ${optionE}\n 
’’

#### A.1.0.2 Log-likelihood classifier:

In this setup, we prompt the LLM with each answer option separately, and select the highest-probable answer. The probability is computed only on the tokens of the answer option, conditioned on the input sequence. In our experiments, we notice that the effectiveness of this method is sensitive to the prompt. This is due to the question-answer formats in the dataset considered. For instance, EgoSchema [] consists of full-sentence answers, whereas NExT-QA [] consists of answer phrases. Hence, the latter benefits from additional guidance from formatting within the prompt template. More specifically, on EgoSchema [], our prompt has the following format.
‘‘${description} ${question} ${answer_option}’’
Here, the probability is computed only on ${answer_option}. However, on the benchmarks based on NExT-QA [] data, our prompt has the following format with more structure.
‘‘${description} Based on the description above, answer the following question: ${question}? Select one of these choices as the answer:\n A: ${optionA}\n B: ${optionB}\n C: ${optionC}\n D: ${optionD}\n E: ${optionE}\n The correct answer is, ${option_id}: ${answer_option}’’
Here, the probability is computed only on ${option_id}: ${answer_option}. We observe that neither prompt template works as effective when interchanged.

### A.2 Qualitative examples of repository entries

We present qualitative examples from EgoSchema [] dataset to better clarify the operations in LangRepo. In Fig. 4, we show the format of repository entries. Here, non-redundant captions from the input get directly written to the repo. In contrast, any redundant captions— grouped based on similarity— get rephrased as concise descriptions (1 per-group). Each repository description may come with additional metadata such as timestamps and #occurrences to avoid the loss of meaningful information due to pruning. In Fig. A.1, we further elaborate on multiple scales within the repository, which are generated by iteratively processing increasingly-longer chunks (created by re-chunk operation). During reading, we can decide to summarize information at various temporal scales to generate output descriptions useful for VQA.

Figure: Figure A.1: A qualitative example of iterative writing and multi-scale reading in LangRepo: Here, we present an example with 2-scales, given captions of a 180s long video. In scale-1, we consider 3 chunks of 60s each, and in scale-2, we re-chunk them into 2 chunks of 90s each. We only show the redundant captions that go through pruning, and also, omit any metadata (*e.g*. timestamps) within the repository. In each scale, captions grouped based on similarity get rephrased concisely. To generate inputs of the subsequent scale, we simply order previous repository descriptions in time, and split (i.e., re-chunk) into fewer (and, longer) chunks. When reading, each entry in each scale is summarized separately to create output descriptions of various temporal spans. In general, we always consider the last-scale descriptions to be mandatory, but any prior-scale to be optional. Yet, we observe multiple scales to be beneficial (see Table LABEL:tab:ablation:memory_setup). Best-viewed with zoom-in.
Refer to caption: x5.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 8
presentation: 3
questions: N/A
soundness: 3
strengths: 1. The newly proposed concept of using concise text as a representation for video reveals why current captioning-based long video understanding methods perform well, which is enlightening.
2. The authors conducted observations and various ablation experiments to demonstrate the effectiveness of their approach.
3. Experiments on multiple datasets show consistent improvements compared to the most similar related work, LLoVi.
summary: This paper introduces a video modeling method called LangRepo, which forms a text data organization strategy. The method performs well on four datasets, demonstrating its effectiveness.
weaknesses: 1. The usage of log-likelihood is not fair, as other models do not utilize it, especially on the IntentQA dataset.
2. The experiments on long videos are not sufficient. NeXT-QA and NeXT-GQA are not typical long video benchmarks. At least the long split in VideoMME[1] should be incorporated. Otherwise, the authors should not stress long videos in the title, since textual representation for videos is also valuable for short videos. 
3. The observation experiment in Table 1 is hard to interpret; it would be helpful if the token sequence length were provided.


[1] Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In experiment Table 1, what is the specific input of Mistral-7B, and how long is the input? It is important to show how to obtain 1X, 0.5X and 2X captions. What about prompting VLLM to generate descriptions in different lengths (complexity)?
2. How do you get the video chunks? Even sampling frames or via some semantic approach, such as shot detection. For each video clip, how to use image LLaVA to obtain a caption set for videos?
3. What if using powerful LLM as an oracle for QA, e.g., Gemini and GPT to replace the models in Table 6a.
4. The input description is highly formated and seems long and redundant (might be inevitable, since the video is split into chunks and does separate text summarization). How to ensure the LLM can parse such format and won't trigger the long-context information decay issue?
soundness: 3
strengths: 1. Using language as an intermediate modality to deal with long-term information decay in LLM makes sense and sounds interesting. This could be a popular trend and this work would play an important role. However, the authors should do a detailed literature review and thorough discussion of this trend.
2. The code and implementation of the methods is provided which will greatly ease the reproduction process.
3. While in a zero-shot setting and no video pretraining is required, the proposed method achieves promising results on all the involved long video benchmarks.
summary: The paper focuses on the problem that LLM's ability of long-term information gradually declines with input length, which is an important and interesting problem to study. The authors propose a textual-only video representation approach to deal with the information decay problem. Specifically, the overall pipeline contains read and write operations. Writing operation describes the video content with texts and stores. Reading operation read the description and summarize the content for text QA to finish VQA in an all-textual approach. Though the all-textual video question answering has been explored in recent literatures, such Language-Repo approach remains interesting. The experiments show the superiority of the method on long-video benchmarks, such as EgoSchema, NExT-QA, IntentQA and NExT-GQA.
weaknesses: 1. The idea of transforming visual content to texts and performing efficient question-answering has been explored in image [1] and video [2][3] domains. There should be a separate subsection in related work to discuss this trend and the difference, which is critical.
2. Reasonability of experiments in Table 1: the authors argue that by varying the input length into QA-LLM (such as Mistral-7B), it can derive the conclusion that longer text input produces worse performance. However, this requires several strict assumptions: first the captions in different lengths contain evenly distributed and all adequate video content information. If it is not, it could be the QA benchmark bias to derive the performance changes, that the answers are hind in the fixed part of the video (say the early part of the captions/videos); Secondly,  the input to Mistral-7B is long enough to trigger the information decay issue. I am doubting if the input is video caption only, the input context length might not be sufficient to discuss long-term information processing decay.
2. As illustrated in Figure 4, the group operation is not conducted in a strict temporal order (like t=2 is grouped with 26-28 and 37). Will there be a negative impact on temporal sensitive tasks and errors in describing action/event orders?
3. Experiments - Proved by SUM-shot[2], the all-textual approach should be performing well, as well as in short video understanding. Comparison should be included on MSRVTT-QA and ANet-QA (which is missing in current long video experiments).
4. Experiments - Extremely long videos: there have been several emerging long video benchmarks, such as Video-MME, Event-Bench and ShortFilms. It would be great and more promising to provide results on these benchmarks and do a comparison/discussion to discuss the potential of dealing with extremely long videos (which should work, given the motivation and design pipeline of LanguageRepo).
5. It would be interesting to explore how vision-language hallucination rate varies. Short video LLMs + Texts vs. Long video LLMs. Will the intermediate modality increase or decrease the hallucination? This is not mandatory but I think is interesting to explore.
6. Formulation in Eq10 should be improved. The use of $\texttt{vqa}$ here would be confusing.
7. Table 6 (g) shows the oracle captioner achieved 69.2 on EgoSchema sub-set. It means the performance upper bound of the method. It might not be a good signal for an extendable framework. Please make further discussion.
8. It is important to ensure the Eq10 process can parse the formated video content and won't lead to long-context information processing decay issues. Please make analysis and explainations.

[1] From images to textual prompts: Zero-shot visual question answering with frozen large language models
[2] Shot2Story20K: A New Benchmark for Comprehensive Understanding of Multi-shot Videos
[3] A simple llm framework for long-range video question-answering [already included in the paper]

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness above.
soundness: 3
strengths: 1. The authors propose an iterative approach with progressively longer chunks, enabling LangRepo to learn high-level semantics and extract stored language information at various temporal scales. This seems beneficial for LLMs in understanding video content.
2. The figures in the manuscript effectively illustrate the pipeline of the method and provide clear examples of entries in LangRepo, making it easy to follow.
3. Extensive experiments on various video question-answering datasets validate the impressive performance of LangRepo.
summary: This paper introduces a Language Repository (LangRepo) for LLMs, addressing the challenge of decreased effectiveness with longer inputs. LangRepo maintains concise and structured information, updating iteratively with multi-scale video chunks. It includes operations to eliminate redundancies and extract information at various temporal scales. The framework demonstrates state-of-the-art performance on zero-shot video question-answering benchmarks.
weaknesses: 1. Table 1 shows that longer input captions can reduce LLM accuracy.  Adding details like frame rate, caption length, downsampling methods, and the effects of adjusting frame rates, rather than just subsampling or replicating captions, would make the findings more convincing.
2. The authors use CLIP to group redundant text. However, as seen in the example from Figure 4, some non-redundant sentences are quite similar to grouped ones, differing only in terms like "man X" and "Person C," which are semantically very similar, raising concerns about CLIP's ability to distinguish them. Additionally, it's unclear how CLIP's similarity threshold is set for grouping, as there are no ablation experiments shown. What's more, directly using LLMs for grouping might be more effective, and a quantitative comparison could be beneficial.
3. If a more effective VLLM or LLM is used, would LangRepo achieve consistent performance improvements? Specifically, the effectiveness of LangRepo with captions extracted by different VLLMs, and whether captions extracted by LangRepo perform better across various LLMs, requires further validation.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: All questions from the weaknesses, and: 

1. The comparison in the right part of Figure 1 is concerning, since the methods like VideoAgent only uses only 8.4 frames, the authors should take the caption/frame numbers into account (also main tables). 
2. Could the authors test the model with the more advanced and larger LLMs, like qwen-72B or LLaMA-3.1-70B? 
3. Could the authors provide comparison results in main table using "generative classifier" for the proposed method as well for fair comparison?
soundness: 2
strengths: 1. The authors include enough relevant papers in the related works section. 
2. The authors provide a detailed ablation study.
summary: The paper presents a Language Repository for LLMs, which generates multi-scale summaries for the video input and applies in long video understanding. The authors evaluate the proposed LangRepo on EgoSchema, NExT-QA, IntentQA and NExT-GQA benchmarks.
weaknesses: 1. The contribution of novelty is concerning. In this paper, the authors propose a summarization-based method based on the dense caption from LLoVi [1] paper. However, the original LLoVi paper already proposes a summarization prompt which significantly improves the performance of long video QA tasks. But in this paper, the authors do not have any discussion on this point in related work or any other sections. Also, the “Grouping redundant text” section is similar to the “Caption Digest” in the LifelongMemory[2] paper. The authors should provide strong evidence of the paper’s novel contribution compared to the existing works. 

2. The need for complicated textual operation is questionable. In this paper, the authors only showcase the results on open-source LLM on a 7B/13B scale and the EgoSchema fullset results are significantly lower than the proprietary LLMs models (even comparing to the same scale open-source VLMs, like InternVideo2, is lower). The potential need of the summarization module is only because the LLM backbone is weak and could not take in the dense caption list. Thus, the authors should validate the effectiveness of the proposed framework on strong proprietary LLMs and have fair comparison with those methods. 

3. The use of “Log-likelihood classifier” is confusing and creates unfair comparison. The authors mention that the Log-likelihood classifier is only used for EgoSchema dataset and improves significantly to the subset performance. In that case, since most methods in the main tables are using the lower-performance “generative classifier”, the use of this classifier creates unfair comparisons for the most related works. Also, the results on the EgoSchema subset are significantly lower than the subset (over 20%), this is not intuitive since the subset is just a part of the full set. Could the authors elaborate more on this? 

4. The evaluation of “long video” is not sufficient. The authors provide results on NExT-QA, IntentQA and NExT-GQA benchmarks, however, the average video length of these dataset is only about 44 seconds, which is considerably short. The authors should provide results on more than 10 minutes to hour long video benchmarks to showcase the effectiveness of the proposed method on long videos. 

5. The efficiency of the proposed method is very concerning. Since the multi-round summarization process is quite slow, and in Table 6.f, compared to the baseline approach LLoVi, LangRepo consumes 70% more time per video-query pair, using 85 seconds to process a 180 seconds video from egoschema. The authors should also show the detailed split of times using on captioning/summarization/question-answering etc.. 

[1] A Simple LLM Framework for Long-Range Video Question-Answering

[2] LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos

Meta-review: This submission received two negative sores and two positive scores after rebuttal. The reviewer who assigned the score 8 still has a  potential generalization issue: paper's performance on other datasets (NExT-QA, IntentQA, and NExT-GQA) does not match its success on the EgoSchema dataset. After carefully reading the paper, the review comments, the AC can not recommend the acceptance of this submission, as the average score is under the threshold bar and the concerns about efficiency and effectiveness of proposed approach remain. The AC also recognizes the contributions confirmed by the reviewers, and encourages the authors to update the paper according to the discussion and submit it to the upcoming conference.

Decision: Reject