Realizing Video Summarization from the Path of Language-based Semantic Understanding

## Abstract

Abstract The recent development of Video-based Large Language Models (VideoLLMs), has significantly advanced video summarization by aligning video features—and, in some cases, audio features—with Large Language Models (LLMs). Each of these VideoLLMs possesses unique strengths and weaknesses. Many recent methods have required extensive fine-tuning to overcome the limitations of these models, which can be resource-intensive. In this work, we observe that the strengths of one VideoLLM can complement the weaknesses of another. Leveraging this insight, we propose a novel video summarization framework inspired by the Mixture of Experts (MoE) paradigm, which operates as an inference-time algorithm without requiring any form of fine-tuning. Our approach integrates multiple VideoLLMs to generate comprehensive and coherent textual summaries. It effectively combines visual and audio content, provides detailed background descriptions, and excels at identifying keyframes, which enables more semantically meaningful retrieval compared to traditional computer vision approaches that rely solely on visual information, all without the need for additional fine-tuning. Moreover, the resulting summaries enhance performance in downstream tasks such as summary video generation, either through keyframe selection or in combination with text-to-image models. Our language-driven approach offers a semantically rich alternative to conventional methods and provides flexibility to incorporate newer VideoLLMs, enhancing adaptability and performance in video summarization tasks. Figure 1: Visualization of two of our extended applications on HowTo100M (Miech et al., 2019 ) . The “Input Video” refers to the keyframes selected from the original video based on our textual summary, and pairing with our textual summary, we can simulate visual manual generation . The images generated by Stable Diffusion 3 (Esser et al., 2024 ) simulate privacy-preserving content generation .

## 1 Introduction

In recent day, the proliferation of video content across various platforms has led to an overwhelming amount of information, making it challenging for users to efficiently access and digest the key information. As a result, video summarization has emerged as a crucial task, enabling the efficient extraction of key segments from lengthy videos. The goal of video summarization is to condense extensive video content into textual summaries, short video clips, or a collection of representative images. Given the vast amount of video data generated daily, effective summarization not only enhances user experience by reducing the time required to access essential information but also supports efficient content management and retrieval across platforms. Additionally, video summarization has significant applications in areas such as surveillance, education, entertainment, and multimedia indexing, making it a vital tool for navigating and leveraging the vast expanse of video data available today.

The success of Visual Language Models (VLMs) (Liu et al., ; Wang et al., ; Alayrac et al., ) has paved the way for the development of Video LLMs, such as VideoLLaMA (Zhang et al., ), VideoChat (Li et al., ), and VideoLLaVA (Lin et al., ). These VideoLLMs leverage human-annotated data for instruction tuning, and they propose different methods to align video features with the LLM feature space. Each model exhibits distinct strengths: for example, PG-Video-LLaVA (Munasinghe et al., ) demonstrates pixel-grounded capabilities for capturing detailed scenes, Video-LLaMA adopts a multi-branch cross-modal framework that incorporates audio information in addition to video content, and LLaMA-VID (Li et al., ) excels in capturing background scene details. However, despite these strengths, existing VideoLLMs have inherent shortcomings and lack coherent methods to address them. For instance, Video-LLaVA and LLaMA-VID are unable to retrieve audio signals, while Video-LLaMA lacks the grounding abilities required for retrieving fine-grained details. Additionally, LLMs within these VideoLLMs often suffer from hallucination issues. To overcome these limitations, previous approaches typically resort to fine-tuning or retraining models, which can be computationally expensive. Our observation, however, suggests that the limitations of one VideoLLM can often be mitigated by the strengths of another. This leads us to ask: What if we could utilize existing VideoLLMs collaboratively, instead of resorting to costly fine-tuning or retraining of a new model?

In this work, we draw inspiration from the Mixture of Experts (MoE) (Shen et al., ; Lin et al., ) paradigm, which is designed to enhance performance in processing large and complex tasks by leveraging multiple expert sub-models. Specifically, our approach employs multiple VideoLLMs for video summarization, integrating the concept of LLM cooperation to combine the outputs from these video “experts” through our proposed inference-time algorithm. This method allows us to address the limitations of individual VideoLLMs by compensating with the strengths of other expert VideoLLMs. Furthermore, since our framework does not require fine-tuning or retraining, it can seamlessly adapt to incorporate new or updated VideoLLMs as additional expert models.

Overall, we propose a novel video summarization method that follows a unique path of language-based semantic understanding. By proposing an inference-time algorithm, we can generate comprehensive textual summaries that capture not only visual content but also audio information, providing detailed descriptions of background scenes to offer users a more holistic view of the original videos. Additionally, with our comprehensive textual summaries, we can perform various downstream video summarization tasks, such as identifying keyframes and generating images and videos, thereby surpassing the capabilities of existing VideoLLMs.

Our main contributions can be summarized as follows:

- •
We propose an inference-time algorithm that leverages the capabilities of LLMs to combine the output summaries of multiple VideoLLMs into a single, coherent, and unbiased summary. This approach provides more detailed and comprehensive information, enhancing the overall quality of video summarization.
- •
Additionally, our comprehensive and coherent summaries enhance keyframe retrieval with a simple keyframe selection algorithm, surpassing the performance of existing approaches.
- •
Our proposed method is both flexible and general. The components of our framework can be easily replaced with more powerful models. Moreover, it is general enough to support extended video applications that can leverage our intermediate outputs, such as textual summaries and keyframes.

## 2 Related Work

##### Large Language Models.

Large Language Models (LLMs) have revolutionized the field of natural language processing (NLP) and artificial general intelligence (AGI) with their exceptional capabilities in language generation, in-context learning, and reasoning. The historical evolution of these models began with foundational architectures such as BERT (Devlin, ), GPT-2 (Radford et al., ), and T5 (Raffel et al., ), which set the stage for subsequent advancements. The introduction of GPT-3 (Brown, ), with its 175 billion parameters, marked a significant breakthrough, showcasing remarkable performance across a wide spectrum of language tasks. This progress spurred the development of an array of other influential LLMs, including Megatron-Turing NLG (Smith et al., ), Chinchilla (Hoffmann et al., ), PaLM (Chowdhery et al., ), OPT (Zhang et al., ), BLOOM (Le Scao et al., ), LLaMA (Touvron et al., ), MOSS (Sun et al., ), and GLM (Zeng et al., ). These models, characterized by their scale and open-source availability, have become invaluable for both training large models and fine-tuning them for specific applications.

##### Visual Language Models.

With the emergence of LLMs, recent works (Liu et al., ; Wang et al., ; Alayrac et al., ) have increasingly explored their use in processing visual inputs, giving rise to Visual Language Models (VLMs). The central idea behind this line of work is to align visual features with the textual features of LLMs by utilizing a common framework. This framework typically involves a pretrained visual encoder to extract visual features, a projection layer to map these visual representations into the text latent space of LLMs, and the pretrained LLM to generate responses, thereby enabling the powerful capabilities of LLMs to be applied to vision tasks. Video-based Large Language Models (VideoLLMs) extend the capabilities of VLMs by incorporating temporal and/or audio features, allowing for richer video-language understanding through human-video dialogue interactions. For instance, methods such as VideoChatGPT (Maaz et al., ) and Valley (Luo et al., ) use pooling over visual tokens to obtain compact visual representations. VideoChat (Li et al., ) employs pretrained video foundation models and Q-Former (Zhang et al., ) from BLIP-2 (Li et al., ) to aggregate video representations. Video-LLaMA (Zhang et al., ) introduces a Video Q-Former and an Audio Q-Former for multimodal video comprehension. Furthermore, MovieChat (Song et al., ) proposes an advanced memory management mechanism for reasoning over extended video content.

##### LLM Evaluator.

The field of Natural Language Generation (NLG) evaluation has evolved considerably over the years, launching from traditional metrics to more advanced methodologies, particularly with the advent of LLMs. Early metrics, such as ROUGE (Lin, ) and BLEU (Papineni et al., ), have been foundational in assessing the quality of generated text by comparing it to reference texts based on n-gram overlap. However, these methods have limitations in capturing deeper semantic nuances. To address this, embedding-based metrics like BERTScore (Zhang et al., ) were introduced, measuring the semantic similarity between texts using word and sentence embeddings. With the rise of LLMs, evaluation methods have further advanced. LLM-based evaluators, such as GPTScore (Fu et al., ), G-Eval (Liu et al., ), and UniEval (Zhong et al., ), leverage the comprehensive understanding and generation capabilities of LLMs to provide deeper insights into NLG quality. Recognizing the inherent limitations of these early approaches, subsequent studies concentrated on enhancing factual accuracy (Min et al., ), ensuring interpretability (Lu et al., ), reducing position bias (Wang et al., ), and aligning evaluation more closely with human judgment standards (Liu et al., ). These efforts represent a significant shift toward more robust and human-aligned evaluation methods in NLG.

Figure: Figure 2: An overview of our framework. Our approach consists of three main modules: (1) Video Summarization, which constructs coherent textual summaries by leveraging multiple existing VideoLLMs and our proposed inference-time algorithm; (2) Keyframe Retrieval, which identifies key moments based on our textual summary using a simple keyframe selection algorithm; and (3) Extended Applications, which utilize our informative textual summaries and keyframes to address real-world tasks beyond traditional video summarization.
Refer to caption: x2.png

## 3 Methodology

Our holistic video summarization framework, illustrated in Figure 2, is composed of three key modules. In Section 3.1, we introduce two components that utilize VideoLLM “experts” to produce textual summaries within the video summarization module. In Section 3.2, we present our keyframe retrieval module, which details how video frames and textual summaries are projected into a joint embedding space to identify relevant keyframes. Finally, in Section 3.3, we explore the extended applications of our framework, demonstrating how the textual summaries and corresponding keyframes can be used for real-world applications.

### 3.1 Video Summarization

We perform inference on the given input video using multiple VideoLLMs. To fully leverage the capabilities of these models, we design and employ prompts specifically tailored to the architecture of each VideoLLM. This approach results in four unique summaries, each capturing different aspects of the input video and reflecting the strengths of each model.

#### 3.1.1 Denoise-and-Cooperate

There are two primary challenges in utilizing the generated summaries from these VideoLLMs. First, each VideoLLM exhibits varying degrees of the “hallucination” issue, which can mislead users and make us difficult to identify the inaccuracies specific to each model. Second, effectively integrating and combining the “strengths” of each model from the resulting summaries is a complex task. To address these challenges, we propose the following strategies:

Filter Outliers. We propose two outlier filtering strategies to remove the summaries that deviate from the others, that is, from the four distinct summaries generated in the previous step, we identify and exclude the summary that exhibits the lowest similarity to the other three, considering it an outlier. For the first strategy, we reference the scoring method from Open-Sora(^1^11[https://github.com/hpcaitech/Open-Sora/tree/main/tools/scoring](https://github.com/hpcaitech/Open-Sora/tree/main/tools/scoring) (last accessed: 2024/09)) to evaluate the summaries generated by each VideoLLM. By calculating the matching score between each summary and the middle frame of the video, we identify and remove the summary with the lowest score. As for the second strategy, we aim to enhance the video-text alignment between the generated summaries and the input video, our implementation is outlined in Algorithm 1. This involves calculating the average CLIP score across the summaries and discarding the one with the lowest score.

Cooperate. After filtering outliers, we leverage the capabilities of state-of-the-art LLMs, to combine the remaining summaries into a single coherent paragraph. We propose three distinct strategies for this synthesis: Merge, Find Common Ground, and Select.

- •
Merge: This strategy integrates all information from the VideoLLM summaries into a comprehensive single summary, capturing the full spectrum of details provided by each model. The resulting summary aims to be inclusive and detailed.
- •
Find Common Ground: This approach focuses on extracting and consolidating only the common elements across all VideoLLM summaries. The process produces a coherent summary that emphasizes the most consistent and reliable information, potentially reducing noise and inconsistencies.
- •
Select: This strategy chooses the summary that achieves the highest score based on our evaluation metric in the outlier filtering stage. We find this approach particularly effective for certain video types, such as instructional videos in datasets like HowTo100M (Miech et al., ).

These strategies provide flexibility in addressing various video content types, allowing for adaptability in the fusion process. The choice of strategy can be tailored to the specific needs of the task or the nature of the video content being summarized.

Figure: Algorithm 1 Calculate Average CLIP Score and Remove Minimum

### 3.2 Keyframe Retrieval

After obtaining our coherent summary from the Video Summarization module, previous methods either prompt the VideoLLM to generate short segments most relevant to the summary (Qian et al., ; Huang et al., ), which is an area where current VideoLLMs often underperform, or training a model specifically to encode visual and textual features (Lin et al., ; Moon et al., ). The latter approach often employs a sliding window technique to capture and align temporal information, enabling the accurate identification and retrieval of relevant video segments that correspond to the summary. However, this method is computationally expensive and can sometimes result in redundant information.

Given that our textual summary is highly informative, we propose an alternative approach that avoids the need for training a new model. Instead, we utilize a fixed joint embedding space, combined with a similarity metric, to guide the keyframe retrieval. Specifically, we encode the input video frames at two-second intervals, following a sampling technique inspired by Moment-DETR (Lei et al., ), alongside our textual summary. Both the text and video frames are encoded using CLIP (Radford et al., ). We then calculate the cosine similarity between the text embeddings (whole summary) and the individual frame embeddings, sorting the similarity scores in descending order to identify the top-$k$ video frames as keyframes.

### 3.3 Extended Applications

Our method extends beyond existing video summarization, offering practical real-world applications that leverage both our coherent textual summary (from Section 3.1) and retrieved keyframes (from Section 3.2). These include visual manual generation for instructional videos, aiding product manufacturers in creating efficient user guides, and privacy-preserving content generation, which produces short video clips and representative images that capture the essence of the original video without revealing sensitive content. These applications demonstrate our method’s versatility, addressing challenges in content creation, information dissemination, and privacy protection across various domains, thus surpassing the capabilities of existing VideoLLMs.

## 4 Experiments

### 4.1 Experimental Setup

##### Dataset.

We evaluate our approach on four well-established datasets: QVHighlights (Lei et al., ), TACoS (Regneri et al., ), Charades-STA (Gao et al., ) and DiDeMo (Anne Hendricks et al., ). These datasets span diverse video domains, including sports, product reviews, cooking scenarios, and household activities, etc., providing a comprehensive foundation for assessing our method’s performance. Table 1 summarizes the key characteristics of each dataset.

**Table 1: Statistics of datasets used in our evaluation.**
| Dataset | Videos | Video-Qeury Pairs | Avg. Video Len (sec) | Video Types |
| --- | --- | --- | --- | --- |
| QVHighlights | 10,148 | 10,310 | 150 | Diverse (daily, travel, news, etc.) |
| TACoS | 127 | 18,818 | 287 | Cooking |
| Charades-STA | 9,848 | 18,131 | 30 | Indoor activities |
| DiDeMo | 10,464 | 40,543 | 30 | Diverse (from Flickr) |

##### Implementation Details.

In our experiments, we employ four VideoLLM “experts”: Video-LLaMA (Zhang et al., ), Video-LLaVA (Lin et al., ), PG-Video-LLaVA (Munasinghe et al., ), and LLaMA-VID (Li et al., ), which serve as both components of our approach and individual baselines. We obtain summaries from each expert, use average CLIP scores to remove outliers, and apply our Find Common Ground strategy with Llama-3-8B-Instruct(^2^22[https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) (last accessed: 2024/09)) to synthesize the final coherent summary. For keyframe retrieval task, we encode video frames (sampled at two-second intervals) and our generated textual summary into CLIP (Radford et al., ) embedding space before calculating similarity metrics.

### 4.2 Experimental Results

##### Textual Video Summarization.

To evaluate the quality of our textual summaries and their alignment with ground truth, we employ G-Eval (Liu et al., ), which we utilize GPT-4-Turbo(^3^33[https://openai.com/index/gpt-4/](https://openai.com/index/gpt-4/) (last accessed: 2024/09)) as the LLM backbone. This method evaluates summaries across seven dimensions: aspect coverage, coherence, faithfulness, fluency, relevance, sentiment consistency, and specificity. Importantly, G-Eval not only assesses video-text alignment through the relevance score but also provides insights into potential human preferences through the remaining metric scores. The results, presented in Table 2, demonstrate that our generated summaries consistently outperform all baseline methods. Our approach achieves superior scores in both video-text alignment and across all aspects that typically correlate with human preference. This comprehensive evaluation underscores the effectiveness of our method in producing high-quality, relevant, and potentially more appealing summaries compared to existing approaches. We also present qualitative results comparing our textual summaries with those of baseline models in Figure 3 (More qualitative results on different datasets are provided in the Appendix, cf. Figure 8 and 9). While most summaries generated by our baselines capture the essential content, but our approach captures a broader spectrum of information from the given video, providing a more complete and nuanced representation of the content. Also, our method demonstrates potential as an automatic (re-)annotation tool. In cases where ground truth summaries may be inaccurate, as shown in our qualitative results, our framework can serve as a valuable means to verify and potentially correct existing annotations. This capability highlights an additional extensibility of our approach, offering a robust mechanism for enhancing the quality and reliability of video annotation datasets.

##### Visual Keyframe Retrieval.

Following the evaluation metrics in TVR (Lei et al., ) and Tall (Regneri et al., ), we compute the mean Intersection over Union (mIoU) and Recall@1 with IoU thresholds of 0.5, and 0.7. In addition to individual VideoLLMs as prompt-based baselines, we also include CG-DETR (Moon et al., ) as the query-based baseline. The results, presented in Table 3, demonstrate our approach’s effectiveness. We outperform all baselines on the Charades-STA, TACoS and DiDeMo datasets, and surpass prompt-based baselines on QVHighlights. Notably, our method, without fine-tuning, achieves superior performance on Charades-STA, TACoS and DiDeMo compared to the fine-tuned CG-DETR. While CG-DETR shows better results on QVHighlights, it’s important to consider that CG-DETR benefits from dataset-specific fine-tuning. In contrast, our method’s strong performance across datasets in a zero-shot setting underscores its robust generalization capabilities. We also provide the qualitative comparison of our keyframe retrieval results against those of our baselines in Figure 4. The visual comparison clearly demonstrates that our selected keyframes achieve a significantly higher coverage rate of the ground truth compared to prompt-based baselines. Moreover, our approach shows superior performance even when compared to CG-DETR. These results visually reinforce the quantitative findings, highlighting our method’s effectiveness in accurately identifying and retrieving key moments from videos.

##### Extended Applications.

We demonstrate two extended applications of our framework on the HowTo100M (Miech et al., ) dataset, which primarily consists of instructional videos. Figure 1 presents the qualitative results of these applications, and more results are provided in the Appendix (cf. Figure 7).
For visual manual generation, our generated summary mimic the textual instruction, and the selected keyframes are the visual instructions. This combination of textual and visual elements effectively simulates the creation of visual manuals for instructional content. In the privacy-preserving content generation, we utilize Stable Diffusion 3 (Esser et al., ) to generate images based on our textual summaries. The resulting images successfully interpret the content of the original videos without revealing sensitive information. These qualitative results illustrate the versatility of our framework in generating practical, real-world applications beyond standard video summarization tasks.

**Table 2: Quantitative evaluation of our generated textual video summary among various approaches with G-Eval (Liu et al., ). The best results are marked in bold.**
| Dimension | OURS | Video-LLaVA | PG-Video-LLaVA | LLaMA-VID | Video-LLaMA |
| --- | --- | --- | --- | --- | --- |
| aspect coverage | 2.77 | 1.31 | 1.85 | 1.97 | 1.72 |
| coherence | 3.35 | 1.56 | 2.12 | 2.76 | 1.83 |
| faithfulness | 2.14 | 1.31 | 1.63 | 1.65 | 1.49 |
| fluency | 3.31 | 1.66 | 2.29 | 2.89 | 2.01 |
| relevance | 2.59 | 1.5 | 1.66 | 1.96 | 1.42 |
| sentiment consistency | 1.92 | 1.23 | 1.38 | 1.6 | 1.31 |
| specificity | 3.22 | 1.41 | 2.12 | 2.44 | 1.97 |

Figure: Figure 3: Visualization of textual video summaries generated by individual VideoLLMs and our proposed collaboration approach. Keyframes are displayed at the top as input video. Additionally, we provide the ground truth summary for reference.
Refer to caption: x3.png

Figure: Figure 4: Visualization of prediction results comparison on QVHighlights (Lei et al., ). The ground truth keyframes are shown at the top as the input video, and the prediction unit is in seconds.
Refer to caption: x4.png

Table: Table 3: Quantitative evaluation of our keyframe retrieval prediction among prompt-based and query-based approaches. The best results are marked in bold, and the second-best results are underlined.

### 4.3 Ablation Studies

For the experiments in the following studies, the experimental setup follows our main setting in Section 4.1, and we focus on the keyframe retrieval task evaluated on three datasets: QVHighlights, Charades-STA, and TACoS, and metrics: mIoU, R@0.3, R@0.5, and R@0.7, unless otherwise specified.

##### Effect of filtering outliers.

To assess the impact of our “Filter Outliers” component, we compare our framework’s performance with and without this feature. In both scenarios, we utilize GPT-4-Turbo to synthesize summaries from individual VideoLLMs using the “Find Common Ground” strategy. The key difference lies in the input to this fusion process: with outlier filtering, we exclude the detected outlier, while without it, all four summaries are included. As demonstrated in Table 4, the inclusion of outlier filtering led to a significant improvement in performance, enhancing both mean Intersection over Union (mIoU) and Recall metrics by at least 2%. This consistent improvement across metrics underscores the effectiveness of our outlier filtering approach in refining the quality of the final summary.

##### Effect of different cooperation strategies with different LLMs.

We examine the impact of different cooperation strategies and LLMs on our framework’s performance. We compare two cooperation strategies, Merge and Find Common Ground, implemented with two distinct LLMs: the open-source Llama-3-8b-Instruct and the closed-source GPT-4-Turbo, and the prompt template we apply is provided in the Appendix (cf. A.1). We present the results in Table 5. Our analysis reveals that the choice of LLM and cooperation strategy has only marginal effects on the overall performance. However, all combinations demonstrate substantial improvements over utilizing only the individual VideoLLM summaries, as shown in Table 3. Our results strongly suggest that our method of combining and refining summaries from multiple VideoLLMs produces more comprehensive and accurate textual representations, which in turn lead to improved keyframe selection.

Table: Table 4: Ablation study of the effect of filtering outliers.

Table: Table 5: Ablation study of effects of different cooperation strategies utilizing different LLMs. In the ”LLM” column, “GPT” represents “GPT-4-Turbo”, and “LLaMA” denotes “LLaMA-3-8b-Instruct”. In the “Cooperate Strategy” column, “CG” refers to the “Find Common Ground” strategy, while “M” stands for the “Merge” strategy. The best results are marked in bold.

##### Effect of audio information.

To assess the influence of audio information, we conduct experiments with and without audio input, noting that some VideoLLMs, such as Video-LLaMA, incorporate audio information, others like Video-LLaVA and LLaMA-VID do not include this modality in their frameworks. For Video-LLaMA, we remove the audio branch to simulate scenarios without audio information. In the case of PG-Video-LLaVA, we deactivate the audio branch in our default setting. The results, presented in Table 6, demonstrate the significant contribution of audio information to the quality of video summaries. Including audio led to a 5-10% improvement in downstream keyframe retrieval performance.

**Table 6: Ablation study of the impact of audio information. We remove the audio branch of Video-LLaMA (Zhang et al., ) to simulate the case of “w/o audio”.**
|  | QVHighlights | Charades-STA |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mIoU | R@0.3 | R@0.5 | R@0.7 | mIoU | R@0.3 | R@0.5 | R@0.7 |  |
| w audio | 24.08 | 22.31 | 15.33 | 10.29 | 35.16 | 36.1 | 27.03 | 13.94 |
| w/o audio | 19.45 | 17.28 | 10.73 | 7.1 | 27.75 | 22.24 | 10.58 | 2.5 |

## 5 Conclusion

We propose a holistic video summarization framework that leverages multiple VideoLLMs to generate comprehensive textual summaries that capture the detail of the given video without fine-tuning. Our extensive experiments demonstrate the effectiveness of our method in downstream tasks like keyframe retrieval and extended applications such as visual manual generation and privacy-preserving content creation. Our framework’s adaptability allows for easy integration of more advanced models, ensuring its relevance as the field progresses. By establishing a foundation for integrating visual and linguistic information, our approach paves the way for more sophisticated multimedia analysis tools. We anticipate that this framework will catalyze advancements in video understanding and natural language processing, leading to more intuitive and powerful systems across various domains.

## Appendix A Appendix

### A.1 Merging Prompt

Figure: Figure 5: Prompt template of “Find common ground” strategy in the cooperation step.
Refer to caption: x5.png

Figure: Figure 6: Prompt template of “Merge” strategy in the cooperation step.
Refer to caption: x6.png

### A.2 HowTo100M Qualitative Result

Figure: Figure 7: HowTo100M textual summary, selected keyframes, and generative results.
Refer to caption: x7.png

### A.3 Charades-STA Qualitative Result

Figure: Figure 8: Textual summary qualitative comparison results on Charades-STA.
Refer to caption: x8.png

### A.4 TACoS Qualitative Result

Figure: Figure 9: Textual summary qualitative comparison results on TACoS.
Refer to caption: x9.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. Please address the weakness 1,2,4.
2. How were the generated image frames evaluated?
soundness: 3
strengths: * Unlike other models that require resource-intensive fine-tuning, this framework works at inference time, saving computational resources and time and showing descent performance in various downstream tasks.

* The framework is flexible and can incorporate new VideoLLMs as they are developed, allowing it to stay current with advancements in the field.
summary: * This work introduces a new framework inspired by the Mixture of Experts (MoE) approach, allowing VideoLLMs to complement each other’s strengths without fine-tuning.
* It provides detailed background descriptions and identifies keyframes, improving semantically meaningful video retrieval over traditional methods that rely solely on visual features.
* The summaries enhance performance in downstream tasks such as summary video generation, both through keyframe selection and by using text-to-image models.
weaknesses: 1. The main weakness of this work lies in the novelty part. In addition, this work is more suitable for submission to application-related conferences.

   (a). The filtering strategy is more like an ensemble method that tries to get the consensus agreement across models via similarity matching, which is a common approach. 

   (b). For the keyframe retrieval part, utilizing CLIP via similarity score across image and text is already well explored in [1,2]. 

   (c). The high-level idea of summarizing video using VLM and LLM was explored in [3], which should also be cited. 

   (d). The idea of visualizing the visual steps for summarization is interesting. However, it was also explored in [4], where it generate the instruction step by diffusion models.

2. In the experiment, the proposed model in Table 3 is significantly higher than the baseline with 94.17, which is nearly identical to ground truth. Is there any reason behind this? The result seemed seemed abnormal.

3. The paper requires proofreading polished L346 ?? presents. Table 3 is out of bound.

4. The framework relies on the availability and compatibility of multiple VideoLLMs, which may not always be feasible or efficient for every application, particularly in low-resource environments.

[1] Luo, Huaishao, et al. "Clip4clip: An empirical study of clip for end-to-end video clip retrieval and captioning." Neurocomputing 508 (2022): 293-304.

[2] Portillo-Quintero, Jesús Andrés, José Carlos Ortiz-Bayliss, and Hugo Terashima-Marín. "A straightforward framework for video retrieval using clip." Mexican Conference on Pattern Recognition. Cham: Springer International Publishing, 2021.

[3] Argaw, Dawit Mureja, et al. "Scaling Up Video Summarization Pretraining with Large Language Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Souček, Tomáš, et al. "Genhowto: Learning to generate actions and state transformations from instructional videos." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: I would like to see the responses to both of two points raised as the weaknesses.
soundness: 2
strengths: The method improves the performance. Some application-oriented textual video summarization may benefit from the paper. Also, the applications of the method are interesting.
summary: The paper presents a method for textual video summarization using multiple multimodal LLMs. The multiple multimodal LLMs individually takes video frames and generates textual summaries, which are then reduced to a single summary for the input video. The method is evaluated on several datasets, showing performance gains. Ablation studies are also provided. Some applications are also shown,
weaknesses: The idea of the paper, which is to use multiple LLMs to generate multiple video summaries and then integrate them into a single video summary, looks naïve, and people can expect that they perhaps improve the performance. The technical contribution of the paper looks weak, and no interesting insights are provided. I would like to see more specific descriptions about why these results are worth sharing with the community.

The first point of the summary of the contributions, saying the method an generate single coherent, and unbiased summary, does not make much sense for me. What does it mean by a coherent and unbiased summary? As far as I understand, the experimental results do not say anything related to coherency and unbiasedness of generated summaries.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 1
questions: Please answer my comments in the Weakness part.
soundness: 3
strengths: + The quantitative evaluation for text summary generation and keyframe prediction shows superior performance as compared to the subset of the proposed method.
+ Some possible applications are demonstrated, which might be useful.
summary: This paper presents a text summary generation method for videos by combining multiple fundamental models as a mixture of experts. The easy-to-understand flow is presented in Figure 2. A summary pool is generated by using multiple VideoLLMs. Then, the summaries are polished (outliers are filtered out) to generate a better summary. Some applications such as key frame extraction and abstracted key story generation.
weaknesses: - Technical novelty is very weak.
  - As can be seen in Section 3, this work is just a combination of existing fundamental models. The flow and how to combine them might be the main scope of this paper, but I could not see any novel technical contributions in this paper. The authors might want to clarify their technical novelty.
  - No technical details are given in the manuscript. Some key components are presented such as “filter outliers” and “cooperate,” but their technical details are all missing. The authors might want to show the equations or the pseudo code of them so that possible readers can grab the idea. It is almost impossible to reproduce the proposed method.

- Experimental comparison is not good enough
  - The proposed work is compared only with the subset of their proposed model: each LLM without MoE. It is apparent that the mixture model would outperform a single model. The author might want to design the experimental comparison better.
  - In Section 4, only the numerical comparison is discussed. Since the task is the video summarization, the authors should show the generated summaries and discuss how they are qualitatively different (I see a set of examples in Figure 3, but this is just an example, no detailed discussion is given). 
  - In addition, the authors might want to discuss the failure cases and limitations.

Here are some comments to improve the paper:
- reference error in line 346

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: The paper is well written and easy to follow. I believe it is a great engineering project that shows how off-the-shelf VideoLLMs can be leveraged with a inference-time algorithm to generate a single textual summaries. My main concern is the technical novelty of this submission. It appear to me that the paper only provides a way to combined existing models to produce a better summary. In other words, the results are benefitted mainly from the capability of pre-trained LLM and VideoLLMs. I would really like the authors to outline what is the novel technical contributions and/or insights generated from this approach that is beneficial to researcher in this field.
soundness: 3
strengths: + This paper propose a Mixture of Experts inspired paradigm that jointly utilising multiple off-the-shelf VideoLLMs via LLM cooperation. The resulting system can be easily upgraded by incorporating more VideoLLMs.
+ This paper proposes an inference-time algorithm to generate textual summaries, key frames identification, and visual summary generation.
summary: This submission focuses on the task of video summarization. Specifically, it integrates multiple VideoLLMs to generate comprehensive and cohrent textual summaries without additional fine-tuning. The proposed model also enhance the quality of downstream tasks that required rich semantic extraction and retrieval capabilities.
weaknesses: - The first strategy in filter outliers is based on the assumption that most of the LLM will output correct and similar. The second strategy compare visual and textual embedding for correctness of the summary. While these are reasonable for simpler video, it may face problems if the input video is rich and highly diverse, and exists multiple events. The definition of "good summary" is highly depend on the quality of the selected VideoLLMs.