Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Authors: Sanghyeok Chu
Seonguk Seo
Bohyung Han
Venue: ICML 2025 poster (2025)
Keywords: Long video captioning
zero-shot video captioning
scene graph
Primary area: applications->computer_vision
TL;DR: We propose a novel framework for long video captioning based on graph consolidation.

Abstract: Recent advances in vision-language models have led to impressive progress in caption generation for images and short video clips. However, these models remain constrained by their limited temporal receptive fields, making it difficult to produce
coherent and comprehensive captions for long videos. While several methods have been proposed to aggregate information across video segments, they often rely on supervised fine-tuning or incur significant computational overhead. To address these challenges, we introduce a novel framework for long video captioning based on graph consolidation. Our approach first generates segment-level captions, corresponding to individual frames or short video intervals, using off-the-shelf visual captioning models. These captions are then parsed into individual scene graphs, which are subsequently consolidated into a unified graph representation that preserves both holistic context and fine-grained details throughout the video. A lightweight graph-to-text decoder then produces the final video-level caption. This framework effectively extends the temporal understanding capabilities of existing models without requiring any additional fine-tuning on long video datasets. Experimental results show that our method significantly outperforms existing LLM-based consolidation approaches, achieving strong zero-shot performance while substantially reducing computational costs.

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Sanghyeok Chu
Seonguk Seo
Bohyung Han
Abstract
et al., 2022; Dai et al., 2023; OpenAI, 2023; Chen al., 2024b; Huang et al., 2024; Zhang al., 2025; Xu et al., 2024; Maaz et al., 2024). Notably, substantial progress has been made in generating captions for images and short video clips (Liu et al., 2024; Chai et al., 2025; Zhao et al., 2024; Wang et al., 2024; Chen et al., 2024a; Mun et al., 2019).
Recent advances in vision-language models have led to impressive progress in caption generation for images and short video clips. However, these models remain constrained by their limited tempo- receptive fields, making difficult to produce coherent and comprehensive captions for long videos. While several methods have been pro- posed aggregate information across video seg- ments, they often rely on supervised fine-tuning or incur significant computational overhead. To address these challenges, we introduce a novel framework for long video captioning based on graph consolidation. Our approach first gener- ates segment-level captions, corresponding in- dividual frames short video intervals, using off-the-shelf visual captioning models. These cap- tions are then parsed into individual scene graphs, which are subsequently consolidated into a uni- fied graph representation that preserves both holis- tic context and fine-grained details throughout the video. A lightweight graph-to-text decoder then produces the final video-level caption. This framework effectively extends the temporal un- derstanding capabilities of existing models with- out requiring any additional fine-tuning on long video datasets. Experimental results show that our method significantly outperforms existing LLM- based consolidation approaches, achieving strong zero-shot performance while substantially reduc- ing computational costs.
However, generating captions for longer videos remains a significant challenge. Most existing models are designed for short-term visual inputs, such images or short video clips, and lack effective support for holistic encoding of entire long videos. As a result, captioning videos beyond a model's 's temporal window typically requires processing and integrating information from multiple temporal segments. Several approaches, such as memory- 2024; Song et al., 2024; Balazevic et al., 2024) and recur- sive frameworks (Zhou et al., 2024; Islam et al., 2024; Qian et al., 2024; Weng et al., 2024; Kahatapitiya et al., 2024), have been proposed to consolidate information across these segments. However, these methods often rely on supervised fine-tuning with the target datasets, which limits their gen- eralizability to unseen video domains. More recently, large language models (LLMs) have been employed generate textual summaries across multiple video segments (Wang et al., 2022b; Chen et al., 2023; Zhang al., 2024a). While these LLM-based approaches eliminate the need to adapt existing models for long videos, they typically incur high inference overhead and require significant computational resources.
(Zhou et al.,
To address these limitations, propose novel framework that integrates segment-level captions into unified global description via graph-based consolidation. We first obtain segment-level captions each corresponding to either sin- gle frame or a short video clip, depending on the chosen visual captioning model-using an off-the-shelf captioning algorithm. Each caption then parsed into a scene graph, and these graphs are consolidated into a unified structure that captures the comprehensive semantics of the entire video. Finally, a lightweight graph-to-text decoder, trained solely on external text corpora, translates the consolidated graph into coherent global caption.
1. Introduction
Vision-language models (VLMs) have demonstrated im- pressive capabilities across diverse vision-language tasks, including visual question answering, visual dialogue, cross- modal retrieval, and spatiotemporal understanding (Alayrac
'ECE, Scoul National University, Korea. IIAII, Scoul National University, Korea, t: Currently at Meta. Correspondence to: Bohyung Han <bhhan @ an'.
The proposed approach enhances understanding and process- ing of long-range temporal information without requiring architectural changes or fine-tuning on long video datasets.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
In particular, our framework can be paired with any off-the- shelf VLM, effectively extending its captioning capability beyond the model's inherent temporal constraints. Unlike other LLM-based consolidation methods, it minimizes com- putational overhead by employing a lightweight graph-to- text decoder with significantly fewer parameters. Our ex- perimental results demonstrate that our approach achieves superior performance in both zero-shot video captioning and zero-shot video paragraph captioning, demonstrating its effectiveness and efficiency.
single representation. Another approach, often termed zero- shot, involves text-only training without paired video-text annotations, where text decoders are used in conjunction with image-text aligned encoders such as CLIP and Image- Bind (Girdhar et al., 2023). Methods such as DeCap (Li et al., 2023b) and C (Zhang et al., 2024b) generate captions by aligning visual and textual features in a shared embed- ding space. However, these approaches often fail to produce accurate and coherent captions, especially when applied to videos with complex events.
In summary, our key contributions are organized as follows:
Zero-shot long video captioning Generating coherent and comprehensive captions for long-context videos under zero-shot settings often relies on the consolidation of infor- mation derived from multiple temporal segments. Existing consolidation techniques, including memory-based (Zhou et al., 2024; Song et al., 2024; Balazevic et al., 2024) and recursive approaches (Islam et al., 2024; Qian al., 2024; Weng et al., 2024; Kahatapitiya et al., 2024), require su- pervised fine-tuning on the target dataset, which limits their applicability to zero-shot scenarios. Recently, LLMs have emerged as a promising tool for zero-shot consolida- tion, leveraging their general reasoning capabilities without task-specific fine-tuning. For example, VidIL (Wang et al., 2022b) constructs prompts by integrating multi-level textual information from image-language models, including objects, events, attributes, frame captions, and subtitles. Due to the complexity of these prompts, it incorporates illustrative few- shot exemplars from training dataset, to guide LLMs in inter- preting and utilizing these textual cues for video captioning Similarly, Video ChatCaptioner (Chen et al., 2023) adopts an interactive framework, where an LLM queries an im- age VLM for captions of individual frames and aggregates them to generate video caption. While these methods are powerful and flexible, they typically incur high computational costs.
We propose novel approach to generate fine-grained captions for long videos using the information across multiple temporal segments.
We introduce a graph consolidation algorithm that merges segment-level scene graphs into a unified rep- resentation to capture both holistic context and fine- grained details across the entire video.
Our method achieves strong zero-shot captioning per- formance with significantly lower computational cost compared LLM-based approaches.
2. Related Works
Video captioning Recent advances in video captioning have predominantly rely on supervised training using large- scale datasets, achieving impressive results across various benchmarks (Lei et al., 2021; Wang et al., 2022a; Yan et al., 2022; Liu et al., 2024; Zhao al., 2024; Wang al., 2024; Chen et al., 2024a). However, extending these supervised approaches to longer videos remains challenging, primar- ily due to the scarcity of annotated data covering extensive temporal contexts and the computational complexity in- volved in modeling long-range dependencies. While various methods have been proposed to tackle these challenges, the needs for supervised fine-tuning for specific target datasets hampers scalability and generalization to unseen video do- mains (Yang et al., 2023; Islam et al., 2024; Song et al., 2024; Balazevic et al., 2024; Qian et al., 2024; Weng et al., 2024; Kahatapitiya et al., 2024).
3. Scene Graph Construction for Videos
To enable effective captioning of long videos, we propose novel framework that constructs and consolidates scene graphs derived from segment-level captions, as illustrated in Figure 1. The framework comprises four main stages: (1) generating captions for individual video segments using VLMs, (2) converting these captions into scene graphs, (3) merging the scene graphs from all segments into a unified graph, and (4) generating a comprehensive description from the consolidated graph. By aggregating information across segments, the proposed method produces captions that are more coherent and contextually informative, capturing fine- grained details throughout the video. Throughout this paper, we use the term segmens to denote a temporal unit of a circo--itheraae a single frame or a short interval -nterval-dependinng on the characteristics of the employed VL...
Zero-shot video captioning Researchers have explored methods for video captioning without using paired video- text annotations. One approach involves refining language model outputs solely at test time. ZeroCap (Tewel et al., 2022) and related methods (Tewel et al., 2023) use image- text alignment score calculated by CLIP (Radford et al., 2021) in gradient updates to adjust language model fea- tures, while MAGIC (Su et al., 2022) employs a CLIP- induced decoding strategy to ensure semantic relevance. Although initially developed for images, these methods ex- tend to videos by aggregating feaee-level features into a
2

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
(s) Segment-level caption generation
(b) Scane graph parsing
Video-Sevel scene graph
Graph input output
Coutter
"An elderly woman with plarses
Farsing
is sitting in a kitchos, speoking to the
(c) Scoree grophh consolidaaiin
Soranking
"An elderly wearing red Parsing shirt cooking kitchen
Skir
Cabbeet
white cabinets."
WWrmmn
Graph-So- model
(d) video-level caption generation
"An elegant shown Pamiing
kitchen, painting to something on the counter."
kisher
red cooking kitchen white
"An
wearing glasses and
Graph-ro-teet
model
cablnets, speaking to the camero and politing something the counter."
poRtTs
pointigg
Figure 1. An overview of our zero-shot video caption generation framework. (left): The pipeline consists of (a) segment-level caption generation using off thelshelf VLMs, (b) seene graph parsing for cach caption, (c) consolidation of individual scene graphs into a unified graph representing the entire video, and (d) video caption generation through our graph-to-text model. (right): Illustration of how the scene graph is transformed into an input for the graph-to-text model to generate a caption.
3.1. Generating segment-level captions
Algorithm Scene graph consolidation
1: Input:
Given an input video, we first divide it into series of temporal segments. We then generate segment-level cap- tions using off-the-shelf VLMs, with prompts guiding the models to produce descriptive sentences suitable for scene graph construction. While we primarily utilize open-source VLMS as our captioning backbone, our framework is flexi- ble enough to incorporate any VLM, including proprictary or closed-source models, as long as APIs are accessible.
2:
G GG,, G₁ J: set of scene graphs
3:
o(-): graph encoder
4:
(-): a function returning the 7th object in a graph
5: m: a permutation function
T: a threshold
7: Output: G video-level scene graph
8: while [9] do
9: Retrieve the most similar pair (G*, G.) from G
10: G = (O*, G = E+)
3.2. Parsing captions into scene graphs
11: GT = "Om, + (Ot U
A scene graph G E) is defined by set of objects O = {or, 02, ... and set of edges between objects, E. Each object = (ca, A.) consists of an object class e; E C and its attribute set A S A. where C is set of object classes and A is a set of all possible attributes. A directed
Σ 12: x + arg max
((((G)))
()((()))
well
[NT()((*))]]
(+) ))
13:
for (p, q) E M such that Sp.g > do
14:
Set the class label of the merged object,
15:
6m + Ac At)
edge, C,,j = (o,,0) 0;) E &, has label ri. E R. specifying the relationship from one object to the other. All object
16: O + (6m) U (om \ {oz og1)
17: Update 8t Em, t- Ep,, and eamm t Co,g
classes, attributes, and relationship labels are represented as
18: end for
text strings.
19: G - (GM) U (g)
We convert the generated caption from each segment into
20: end while
a scene graph, providing a more structured understanding of each segment. A caption is parsed into a scene graph by textual scene graph parser, and FACTUAL-MR parser (Li al., 2023c) is used in our implementation. This parser first intermediate semantic representa-
21: Gviden +- extract(g)
22: return Gvideo
maps the caption tion consisting of objects, attributes, and relationships, then deterministically converts it into a scene graph. By repre- senting each segment as a graph consisting of objects and their relationships, we can apply a graph merging technique to produce holistic representation of the entire input video.
scene graphs derived from each segment into unified graph that represents the overall visual content of the video. We first describe our graph merging procedure and then intro- duce subgraph extraction technique designed to support more focused and coherent video caption generation.
3.3. I. MERGING TWO SCENE GRAPHS
3.3. Scene graph consolidation
We first describe our scene graph merging technique. Given two scene graphs, G = (O*, and G = 'O', (E), con-
The scene graph consolidation step combines all individual
3

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
structed from captions corresponding two different seg- ments, run the Hungarian algorithm to obtain an optimal matching between the two object sets, O" and O which is formally expressed as
4. Video Caption Generation
Our ultimate goal is to generate captions from a consoli- dated scene graph. To this end, we develop a graph-to-text decoding model trained on a dataset of graph-text pairs. At inference time, the model takes the consolidated scene graph representing the entire video as input and generates a caption that describes the video as whole.
((())))) (+())) [((
((((G))
n² = arg max xell
(1)
where o(-) denotes a graph encoder, ()) is a function to extract the 7th object from an embedded graph, and π E II indicates permutation of objects in a graph. Note that the object matching is based on their cosine similarity, where we introduce dummy objects to deal with different numbers of objects for matching.
4.1. Graph-to-text model
Our graph-to-text model consists of a transformer-based graph encoder and a text decoder. The encoder processes the input scene graph to produce graph embedding, which conditions the decoder to generate the final caption. To reflect the graph topology in our model, we design an at- tention mask in the graph encoder that restricts attention propagation to the edges defined in the scene graph.
After computing all matching pairs using the Hungarian algorithm, we identify a set of valid matches .M by selecting object pairs (of, of whose similarity score 8p.g exceeds a predefined threshold r. For each valid match (p. q) E .M,
To construct input tokens for the graph encoder, we con- vert the text values associated with each graph component, such as object classes G. attribute sets A,, and edge labels Fi, (e.g.. "elderly,, "woman", "cook in". "kitchen"), to se- quenees of embedding vectors. Additionally, we append a learnable embedding token that attends to all other tokens, enabling the aggregation of global context and facilitating information flow across the entire graph, including between disconnected nodes.
the merged object om E Ô is defined as
om =
(2)
where è denotes a class label of merged object and Ô represents the set of all merged objects obtained from valid matches. Note that è may differ from the original class label of og or og. This procedure results in new merged scene graph, G = (Om, ,, which combines each valid pair of matched objects, creating a new object.
4.2. Training
We train the graph-to-text model on a large-scale collection of graph-text pairs. To construct this dataset, we curated approximately 2.5 million captions from diverse image cap- tioning datasets, including MS-COCO (Chen et al., 2015), Flickr30k (Young et al., 2014), TextCaps (Sidorov al., 2020), Visual Genome (Krishna et al., 2017b), and Visual Genome Paragraph Captions (Krause al., 2017), cover broad range of visual scene contexts. To further enrich the dataset, we incorporated model-generated captions for videos in Kinetics-400 (Kay et al., 2017), where LLaVA- NeXT-7B (Liu et al., 2024) is applied to four uniformly sampled frames per video. Each caption is then parsed into scene graph using a textual scene graph parser, yielding a graph-text pair for training.
We perform graph merging by iteratively selecting and con- solidating pairs of graphs based on their embedding similar- ity. In each iteration, the two most similar graphs are merged into a single graph, which replaces the original pair in the set of graphs. This process is repeated until only one unified scene graph remains. The final scene graph provides a com- prehensive representation of the entire video that preserves detailed information from individual segments. Algorithm describes the detailed procedure of our graph consolidation strategy.
3.3.2. PRIORITIZED SUBGRAPH EXTRACTION
When concise and focused video captions are desired, we apply subgraph extraction to retain only the most contextu- ally relevant information. During the graph merging process, we track each node's merge count as a measure of its sig- nificance within the consolidated graph. We then identify the top k nodes with the highest merge counts and extract their corresponding subgraphs. This approach prioritizes objects that consistently appear across multiple frames, as they often represent key entities in the scene. By focusing on salient elements and filtering out irrelevant details, our method constructs a compact scene graph that enables more focused video captioning.
Using the graph-text pairs, we train the graph-to-text de- coder with a next-token prediction objective, aiming to gen- erate the ground-truth caption conditioned on the input scene graph, as formally defined below:
N
[(()= =
Σ log Pa (is-1,
(3)
where t; represents the 7h token in the source text, and N denotes the total number of tokens.
4

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
5. Experiment
5.1.2. EVALUATION METRICS
This section presents the effectiveness of the proposed ap- proach through performance evaluation and analysis on both video captioning and video paragraph captioning datasets.
Following standard performance evaluation protocols in video captioning, our experiments adopt n-gram- met- rics, including BLEU-4 (B @) (Papineni al., 2002), ME- TEOR (Banerjee & Lavie, 2005), and CIDEr (Vedantam et al.. 2015), which measure the overlap between generated and reference captions. Since these n-gram-basse metrics are limited in capturing semantic details and contextual accuracy beyond literal phrase matching, we additionally employ BERTScore (Zhang et al., 2020), an embedding- based evaluation metric widely used in natural language processing tasks such as machine translation and summa- rization. BERTScore measures token-level cosine similar- ities between generated and reference captions, capturing semantic similarity beyond n-gram matches as follows:
5.1. Experimental setup
We provide the detailed information about target tasks with their datasets and baselines. We also discuss a list of perfor- mance metrics used in our evaluation.
5.1.1. TARGET TASKS AND BASELINES
Our evaluation consists of two zero-shot tasks: (1) video captioning, using the standard test splits of MSR-VTT (Xu et al., 2016) and MSVD (Chen & Dolan, 2011), and (2) video paragraph captioning, using the ae-val set of Actiy- ityNet Captions (Krishna et al., 2017a), which contains longer videos with multiple events.
PRERT
1
max 2 2gz 2:22 2,62
Σ
(4)
2
We primarily compare our method against LLM-based ap- proaches. Specifically, we first establish an LLM summa- rization baseline, which directly summarizes the same set of segment-level captions used by our method. This base- line provides a direct comparison between the proposed scene graph consolidation and the simple aggregation of segment-level captions by LLMs. We use the open-source for all datasets. For the Activ- ityNet Captions dataset, we additionally employ GPT-4o mini, a more powerful proprietary model. Details of the prompt instructions used for the LLM summarization base- lines are provided Appendix B.
RBERT
1
max 2 2zz Z 2
(5)
2
2 PBEKT RBERT
FBERT
(6)
PRERT + RAERT
where 2 = {=1, 22,...) } and 1 = (2,, represent the sets of token embeddings in the reference and generated captions, respectively.
5.2. Implementation details
Our graph-to-text model consists of a graph encoder and a text decoder, with a total of 235M parameters. The BERT- base model (Devlin et al., 2019) is employed for our encoder, with attention masking as described in Section 4.1, and only the decoder part of T5-base (Raffel et al., 2020) is used as our text decoder.
We also compare our method against LLM-bbsed video un- derstanding methods, e.g., VidIL (Wang al., 2022b) and Video ChatCaptioner (Chen et al., 2023), which utilize com- mercial LLMs along with textual representations derived from VLMs. VidIL constructs rich input sequences by com- bining various textual cues such as objects, events and frame captions extracted from multiple image-based VLMs, and incorporates few-shot exemplars guide the LLM in gener- ating video captions. Similarly. Video ChatCaptioner adopts an interactive question- annserinn framework between im- age VLM and LLMs.
The graph-to-text model is trained on graph-text pairs con- structed in Section 4.2 for 1K iterations with a batch size of 512. We employ the Adam (Loshchilov, 2019) opti- mizer with weight decay of 0.05, an initial learning rate of 0.0001, and linear warm-up for the first 1 1% of training steps. For video paragraph captioning, the model is further fine-tuned for 400 iterations on the subset of the constructed graph-text pairs obtained from the Visual Genome paragraph captioning dataset (Krause et al., 2017).
Note that we primarily focus on LLM-based approaches, as other approaches typically require supervised fine-tuning, making direct zero-shot comparisons infeasible. Addi- tional comparisons with broader zero-shot video captioning approaches- for example, test-time optimization, inference optimization, and text-only training methods-od on MSR- VTT are included in the supplementary document.
Segment-level captions are generated using off-the-shelf VLMs. To demonstrate the flexibility of our approach, we employed both image- VLMs, including BLIP (Li et al., 2022) and BL.IP2 (Li et al., 2023a), and video-centric VLM, Intern (Chen et al., 2024a). For MSR-VTV and MSVD, we uniformly sample six frames per video to generate captions using image-centric models. For Activ- ityNet Captions, we select twelve frames per video when
'https://huut 'https://huggingfacc co/mi /ttaaiir Mistral-7B In
5

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Table Zero-shot video captioning results on the MSR-VTT (Xu al., 2016) and MSVD (Chen & Dolan, 2011) sets, comparing our
method (SGVC) with LLM- based video understanding methods. indicates that the method utilizes reference captions from the target dataset to construct few-shor exemplar prompts. Bold numbers indicate the highest scores among methods not using reference captions.
Dataset
Method
Backbone VLM
B@4
METEOR
CIDEr
PERRT
HRERT
FRERT
VidIL (Wang et ul., 2022b)
3.2
14.8
3.1
0.134
0.354
0.225
BLIP+CLIP
VidIL' (Wang et al., 2022b)
13.6
20.0
20.2
0.461
0.552
0.490
MSR-VTT
Video ChatCaptioner (Chen 2023)
BLIP2
13.2
22.0
16.5
0.396
0.510
0.436
BLIP
17.7
22.5
24.0
0.476
0.539
0.490
SGVC (Ours)
BLIP2
18.4
23.1
26.1
0.467
0.542
0.487
VidII. (Wang al., 2022h)
2.5
16.5
2.3
0.124
0.404
0.238
BLIP+CLIP
VidII.1 (Wang et al., 2022h)
30.7
32.0
60.3
0.656
0.726
0.674
MSVD
Video ChatC ChaiCaptioner (Chen et al., 2023)
BLIP2
22.7
31.8
35.8
0.496
0.651
0.550
BLIP
22.6
30.2
50.2
0.575
0.646
0.589
SGVC (Ours)
BLIP2
25.3
32.0
53.3
0.571
0.669
0.597
Table 2. Zero-shot video captioning results on the MSR-VTT (Xu al., 2016) and MSVD (Chen & Dolan, 2011) test sets, comparing SGVC with the LLM summarization baseline. Bold numbers indicate the highest scores.
Dataset
Method
Backbone VLM
B@4
METEOR
CIDEr
RyaRt
BLIP
9.6
21.6
10.8
0.313
0,516
0.395
Summarization Mistral-7B
BLIP2
11.5
23.1
15.4
0.308
0.528
0.397
MSR-VIT VTT
BLIP
17.7
22.5
24.0
0.476
0.539
0.490
SGVC (Ours)
BLIP2
18.4
23.1
26.1
0.467
0.542
0.487
BLIP
15.2
28.3
30,3
0,477
0.623
0.527
Summarization w/ Mistral-7B
BLIP2
22.5
31.9
41.6
0.500
0.664
0.558
MSVD
BLIP
22.6
30.2
50.2
0.575
0.646
0.589
SGVC (Ours)
BLIP2
25.3
32.0
53.3
0.571
0.669
0.597
using image-centric VLMs, while extracting twelve video clips for the video-centric model.
zero-shot performance across most metrics on both the MSR-VTT and MSVD datasets, outperforming the existing methods. VidII., although it leverages diverse textual cues from multiple sources, shows limited performance in the zero-shot setting. Notably, SGVC performs competitively even against. VidIL's few-shot setting, which heavily de- pends on dataset-specific exemplars. Video ChatCaptioner, which aggregates information through multi-turn question answering between an LLM and BLIP2, often suffers from hallucinations or overemphasis on irrelevant details, leading to failures in capturing the core content of the video (e.g.. "There are no animals present in the park scene.").
To obtain graph embeddings for Hungarian matching, the graph encoder of our graph-to-text model is used.
For generating the final video caption, we apply a beam search with five beams, maximum sequence length of 32 and length penalty of 0.6. For video captioning on MSR- VTT, we apply prioritized subgraph extraction with A = to emphasize salient visual information. Video paragraph caption, which requires more detailed descriptions, is gen- erated using a beam search with three beams, a maximum sequence length of 400, and a repetition penalty of 3.0.
Table 2 provides controlled comparison between SGVC and an LLM-based summarization method, clearly high- lighting the effectiveness of our scene graph consolida- tion approach. Both methods start from an identical set of segment-level captions and this experiments isolates the impact of the graph consolidation. Although L.LM summa- rization produces fluent and expressive captions, it some- times overlooks details of objects and events within scene. In contrast, SGVC explicitly integrates segment-level scene graphs into unified representation, which is helpful for preserving object identities and relationships consistently throughout the video.
5.3. Main results
We present quantitative results for zero-shot video caption- ing on the MSR-VTT and MSVD datasets in Tables I and 2, and for zero-shot video paragraph captioning the Ac- Net Captions ae-val set in Tables 3 and 4.
5.3.1. 1. ZERO-SHOT VIDEO CAPTIONING
Table 1 compares the proposed method, SGVC, with exist- ing LLM-based video understanding approaches, VidIL and Video ChatCaptioner. SGVC consistently achieves strong
6

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Table Zero-shot video paragraph captioning results on ActivityNet Captions (Krishna al., 2017a) ae-val set, comparing our method (SGVC) with LLM- based video understanding methods. indicates that the method utilizes reference captions from the target dataset construct fho- exemplar prompts. Bold numbers indicate the highest scores among methods not using reference captions.
Method
Backbone VLM
B@4
METEOR
CIDEr
PRERT
RBERT
FRENT
VidIL (Wang et ul., 2022b)
5.8
4.6
0.122
0.135
0.125
BLIP+CLIP
VidIL (Wang al., 2022b)
2.9
7.6
3.3
0.414
0.243
0.323
Video ChatCaptioner (Chen 2023)
BLIP2
2.4
8.9
1.6
0.207
0.202
0.200
BLIP
6.7
11.6
16.6
0.367
0.285
0.322
SGVC (Ours)
BLIP2
7.4
12.4
20.9
0.367
0.304
0.331
Table 4. Zero-shot vidco paragraph captioning results on the ActivityNct Captions (Krishna et al., 2017a) ae-val set, comparing SGVC with the LLM summarization baselines. Bold numbers indicate the highest scores.
Method
Backbone VLM
B@4
METEOR
CIDEr
PRERT
RBERT
FRERT
BLIP
3.4
7.5
0.292
0.268
0.276
Summarization w/ Mistral-7B
BLIP2
4.1
10.4
9.6
0.307
0.293
0.295
V1.2
4.5
10.8
11.6
0.333
0.318
0.319
BLIP
4.6
10.2
10.3
0.325
0.284
0.300
Summarization w/ GPT-4o mini
BLJP2
10.6
0.343
0.301
0.317
InterV
5.8
11.4
15.3
0.352
0.332
0.336
BLIP
6.7
11.6
16.6
0,367
0.285
0.322
SGVC (Ours)
BIJIP2
7.4
12.4
20.9
0.367
0.304
0.331
InterVVL.2.
8.0
13.2
24.1
0.359
0.326
0.338
Table 5. Comparison of computational costs between SGVC and L-Med methods the MSR-VTT test set.
Method
VLM Backbone
Params. (B)
GPU (GB)
Time
CIDEr
Using reference
Using GPT API
VidIL
BLIP+CLIP
0.67
3.57
1.32
20.2
Video ChatCaptioner
BLIP2
3.75
14.53
16.5
BLIP
7.50
14.50
1.27
10.8
Summarization w/ Mistral-7B
BLIP2
11.00
28.20
1.51
15.4
BLIP
0.74
5.07
1.14
24.0
SGVC (Ours)
BLIP2
4.24
18.40
1.37
26.1
5.3.2. ZERO-SHOT VIDEO PARAGRAPH CAPTIONING
mance, benefiting from its video-centric design and strong temporal modeling capabilities, despite having significantly fewer parameters than BLIP2 (938M vs. 3.74B). These results clearly demonstrate SGVC's flexibility and plug-and- play compatibility with a wide range of vision-language model architectures.
Table presents a comparison between SGVC and other LLM-based video understanding methods for zero-shot video paragraph captioning on the ActivityNet Captions ae-val set. Consistent with the results observed in zero-shot video captioning in Table SGVC clearly outperforms competing methods. The performance gap is more pro- nounced in the paragraph captioning task, where effectively modeling long-range context and maintaining coherence across multiple events is essential.
5.4. Analysis
Efficiency Table 5 presents detailed comparison of com- putational costs, in terms of average per-video inference time and peak GPU memory usage on single NVIDIA A6000 GPU, along with captioning performance (CIDEr) on the MSR-VTT test set. SGVC consistently outperforms LLM--ased summarization approaches across all compu- tational measures, regardless of the underlying backbones. Moreover, our scene graph merging algorithm, which cur- rently the CPU, could be further accelerated by
Table compares SGVC with LLM summarization tech- niques, using both Mistral-7B and stronger commercial model, GPT-4o mini. While GPT-4o mini offers signifi- cant performance gains over Mistral-7B. it still falls short of SGVC, highlighting the effectiveness of our graph con- solidation approach. Furthermore, replacing the backbone captioner with Intern VL2.5 further improves SGVC's perfor-

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
(Ground-truth) A man opening a toy egg sel.
[Ground-truth] A track runner preparing to run race.
[LLM summ.] A hand opens toy bax, revealing a gift set with a toy airplane, an orange plastic.
[LLM summ.] A group of runners, including ferales, stretch, crouch at the starting line, and.
[VidiL] person is unbawing and demonstrating toy craft kit. [Video ChatCaptioner] The video shows person holding toy car and cup of water while wearing a shirt.
[VidIL] group of athletes competing in various track and field events.
(Video ChatCaptioner] The video shows woman participating in track and field event, wearing a red shirt and shorts.
[Ours] A hand holding toy airplans in front of box with surprised expression.
[Ours] A group of runners crouching down a line track compsting in a race.
[Ground-truth] child
dancing song.
[Ground-truth]
daughter walking around receonmnnnnt
[LLM summ.] A family is dancing together in a room, featuring man, woman, and child.
[LLM summ.] A woman daughter, accompanied by other women, walking down street.
[VidIL] A father and daughter share special moment through dance.
[VidIL] group people are walking down a street Japan.
[Video ChatCaptioner] little girl striped
in a living
[Video ChatCaptioner] The video shows girl wearing a white shirt walking down street a bag. The color the bag not known.
room unknown background music.
[Ours] man and a young girl dancing living room.
[Ours] A woran and her daughter walk down a street with a bicycle in the background.
NAVY
(Ground-truth) A female soldier talks about her athlefics.
[Ground-trutie truth] People sitting table with food.
[LLM summ.] A woman in blue jacket poses outdoors, followed by man in military uniform standing.
[LLM summ.] television show scene man and woman
long purple hair, followed woman.
[VidiL] group of friends are having dinner party and trying out oifferent hairstyles.
[VidIL] female athlete competes in military-themed sports event.
[Video ChatCaptioner]
woman navy uniform standing in front a sign that says "phil"" with white wall in the background.
[Video ChatCaptioner] group of people sitting a a park, eating. There are animals present in the park scene. [Ours] woman dress is sitting table people.
[Ours] A woman unifomm stands in front of sign, holding medals and smiling.
surrounded by
Figure Example zero-shot video captioning results on MSR-VTT test set. We compare our results with LLM-based methods, listed from top bottom LLM summarization using Mistral-7B, 2) VidlL, 3) Video ChatCaptioner, and 4) SGVC (Ours).
Table 6. Analysis on the hyperparameter in the prioritized sub- graph extraction, on the MSR. VTT test set.
Table 7. Analysis on the threshold 7 used in graph consolidation, the MSVD test
k
METEOR
CIDEr
PaLRr
RBERT
FIEXT
CIDEr
FBERT
f
CIDEr
FRERT
1 23.1
26.1
0.467
0.542
0.487
50.0
0.589
0.85
49.9
0.589
3
23,8
24.9
0.454
0.554
0.486
0.90
50.2
0.589
0.80
49,9
0.589
GPU implementation. VidIL and Video ChatCaptioner ex- hibit slower inference times and lower captioning accuracy. While they consume less GPU memory, their dependence on GPT API calls introduces additional latency.
tracted subgraph, described in Section 3.3.2. As shown in Table 6, lower lt values result in more concise subgraphs that emphasize salient objects, leading to improvements in precision-oriented metrics, such as CIDEr and PHERT-
In contrast, higher ls values yield richer subgraphs that capture broader contextual information, thereby improving recell-oriented metries, METEOR and RBERT
Impact of hyperparameters We analyze the effect of the hyperparameter k. which controls the size of the ex-
8

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
[Ground-truth] Two men are at a gym to demonstrate proper form for the exercise. The man in the black shorts gets on one knee as the instructor gives Instructions on what to do. The man in black shorts lifts a bar from the kneeling position. After a few reps, the two men conclude the video.
[LLM summ.] Two men working out in gym, performing various activities such as weightlifling, martial arts, and stretching.
women are seen working out in gym, doing various exercises such flipping tires, punching bags, and using sled. [Video ChatCaptioner] The video features a man wearing a black shirt standing on a ledge in front of red wall indoors. He appears to be leaning forward and looking at the camera with nervous expression.
[VidL]] group of
[Ours] Two young men standing in a gym, practicing martial arts. One of the men is holding baseball. The other man is wearing gray shirt. The man is standing behind the man. The man is holding a weight. The man is standing with his arms raised.
(Ground-truth] A woman pours ice into glass. She adds shots alcohol to the glass, She then pours it into another glass and shakes it. She pours that into glass and sticks straw in
[LLM summarization] Multiple people are preparing and pouring drinks at a bar, including a woman in a striped shirt. [VidIL] A woman is making drink at bar.
[Video ChatCaptioner] The video features a man standing in front of a mirror holding bottle of wine, In the mirror's reflection, person and dog can seen in bathroom setting.
[Ours] A bartender is preparing drink with cocktail shaker. She is wearing a striped shirt. The woman is pouning the drink into glass.
Figure 3. Example of zero-shot vidco paragraph captioning results on the ae-val set of the ActivityNec captions dataset. We compare our results with LLM-based methods, listed from top bottom LLM summarization using Mistral-7B, 2) VidIL, 3) Video ChatCaptioner, and 4) SGVC (Ours).
We also conducted evaluation by varying the cosine similar- ity threshold 7, reported in Table 7. The results demon- strate stable performance within the range r E [0.80, 0.95], and we set T = 0.9 for all experiments.
approaches, our method significantly reduces computational demands by leveraging a lightweight graph-to-tex, model with substantially fewer parameters. Extensive experiments on both video captioning and video paragraph captioning tasks validate the effectiveness of our method. These results highlight the potential of graph-based consolidation as a foundation for future advances in long video captioning.
Qualitative results Figures 2 and 3 present qualitative ex- amples of zero-shot video captioning on the MST test set and video paragraph captioning on ActivityNet Captions ae-val set, respectively. Our method generates detailed and contextually rich captions that accurately capture events, ob- jects, and relationships across frames. While L.M summa- rization and Video ChatCaptioner produce fluent sentences, they occasionally introduce hallucinated content, such as objects or attributes that are not actually present in the video.
Acknowledgements
We thank Do Young Eun at North Carolina State Univer- sity for the valuable discussions. This work was supported in part by National Research Foundation of Korea (NRF) grants
Trustworthy Artificial Intelli- gence; RS-2024-0040861 Brain Pool program], Institute of Information & communications Technology Planning & Evaluation (ITPP grants [RS2022-11220959 (No.2022-0- 00959), (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making; No.RS-2021. I1212068, AI Innovation Hub (AI Institute, Seoul National University); No.RS-202. RRS-2021-112111 1343, Artificial Intelligence Graduate School Program (Scoul National University)] funded by the Korea government (MSIT).
6. Conclusion
We introduced a novel framework for fine-grained caption- ing of long videos by consolidating information across mul- tiple temporal segments. Our approach merges scene graphs extracted from segment-level captions to generate compre- hensive and coherent video descriptions. This framework provides a computationally efficient and training-fie free al- ternative to existing methods. In contrast to LLM--base
9

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Impact Statement
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In ACL, 2019.
The broader impact of this research lies in enabling effective captioning of long videos by leveraging existing vision- language models without any additional fine-tuning on large- scale annotated video datasets. While there is potential for societal impacts arising from this technology, we have not identified any significant negative consequences directly associated with our approach.
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, Imagebind: One embedding space bind them all. In CVPR, 2023.
Huang, B., Wang. X., Chen, H., Song, Z., and Zhu, W. Vtimellm: Empower Ilm to grasp video moments, In CVPR, 2024.
References
Islam, M. M., Ho, N., Yang, X., Nagarajan, T., Torresani, L.,, and Bertasius, G. Video recap: Recursive captioning of hour-long videos. In CVPR, 2024.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., al. Flamingo: visual language model for few-shot learning. In NeurIPS, 2022.
Kahatapitiya, K., Ranasinghe, K., Park, J., and Ryoo, M. S. Language repository for long video understanding. arXiv, 2024.
Balazevic, I,, Shi, Y., Papalampidi, P., Chaabouni, R., Kop- pula, S., and Henaff, J. Memory consolidation enables long-context video understanding. In ICML, 2024.
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F,, Green, T., Back, T., Natsev, P., et al. The kinetics human action video dataset. arXiv, 2017.
Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005.
Krause, J., Johnson, J., Krishna, R., and Fei-Fei, L. hierarchical approach for generating descriptive image paragraphs. In CVPR, 2017.
Chai, W., Song, E., Du, Y., Meng, C., Madhavan, V., Bar-Tarl Bar-Tal, O., Hwang, J..N., Xie, S., and Manning, C. D. Auroracap: Efficient, performant video detailed captioning and a new benchmark. In ICLR, 2025.
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., and Niebles, J. C. canse-captioning events in videos. In ICCV, 2017a.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J,, Shamma, D. A., et al. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. IJCV, 2017b.
Chen, D. and Dolan, W. B. Collecting highly parallel data for paraphrase evaluation. In ACL, 2011.
Chen, J., Zhu, D., Haydarov, K., Li, X., and Elhoseiny, M. Video chatcaptioner: Towards enriched spatiotemporal descriptions. arXiv, 2023.
Lei, J., Li, L.. Zhou, L., Gan, Z., Berg, T. L., Bansal, M., and Liu, J. Less is more: Cliphert for aide--and-languaag learning via sparse sampling. In CVPR, 2021.
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zimick, L. Microsoft coco captions: Data collection and evaluation server. arXiv, 2015.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-ina image pre-training for unified vision- language understanding and generation. ICMI., 2022.
Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao. Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding per- formance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv, 2024a.
Li, Li. D.., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language- pre-training with frozen image encoders and large language models. In ICML, 2023a.
Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al. How far are gpt- 4v? closing the gap to commercial multimodal models with open-source suites. arXiv, 2024b.
Li, W., Zhu, L., Wen, L., and Yang, Decap: Decoding clip latents for zero-shot captioning via text-only training, In ICLR, 2023b.
Dai, W., Li, J. Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards ceneral-purpose vision-language models with instruction tuning. In NeurlPS, 2023.
Li, Z., Chai, Y., Zhuo, T. Y., Qu, L., Haffari, G., Li, F., Ji. D., and Tran, Q. H. FACTUAL: A benchmark for faithful and consistent textual scene graph parsing. In ACL Findings, 2023c.
10

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, oer, and world knowledge. https://llava ww..ara.i..giiiit.. io/blog/2022 0aallo//220-011-----1a--------- 2024.
Tewel, Y., Shalev, Y., Schwartz, I., and Wolf, L. Zero-shot image-to-text generation for visual-semantic arithmetic. In CVPR, 2022.
Tewel, Y., Shalev, Y., Nadler, R., Schwartz, I., and Wolf, Zero-shot video captioning by evolving pseudo-tokens. In BMVC, 2023.
Loshchilov, Decoupled weight decay regularization. ICLR, 2019.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. Video- chatgpt: Towards detailed video understanding via large vision and language models. In ACL, 2024.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus-basee image description evaluation. In CVPR, 2015.
Mun, J., Seo, P. H., Jung, and Han, Marioqa: Answer- ing questions by watching gameplay videos. In ICCV, 2017.
Wang. J., Chen, D., Wu, Z., Luo, C.,, Zhou, L., Zhao, Y., Xie, Y., Liu, C., Jiang, Y.G., and Yuan, L. Omnivl: One foundation model for image-language and video- language tasks. In NeurlPS, 2022a.
Mun, J., Yang, L., Ren, Z., Xu, N., and Han, B. Streamlined dense video captioning. In CVPR, 2019.
Wang. Y., Li, K., Li, X., Yu, He, Y., Chen, G., Pei, B., Zheng, R., Xu, J., Wang, Z., et al. Internvideo2: Scaling video foundation models for multimodal video understanding. In ECCV, 2024.
Noh, H., Seo, P. H., and Han, B. Image question answer- ing using convolutional neural network with dynamic parameter prediction. In CVPR, 2016.
Wang. Z.,, Li, M., Xu, R., Zhou, I.., Lei, J., Lin, X., Wang, S., Yang, Z., Zhu, C., Hoiem, D., et al. Language mod- els with image descriptors are strong few-shot video- language learners. In NeurlPS, 2022b.
OpenAl.
Gpt-4v(ision) system card. https:
//cdn. openai.com/papeeeeeepoeeppppp httpa//a//ppppeeeee/eeeeeeeee Card.pdf. 2023.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
Weng. Y., Han, M., He, H., Chang, X., and Zhuang. B. Longyim: Efficient long video understanding via large language models. In ECCV, 2024.
Qian, R., Dong, X., Zhang, P., Zang, Y., Ding, S., Lin, D., and Wang, Streaming long video understanding with large language models. NeurIPS, 2024.
Xu, J., Mei, T., Yao, T., and Rui, Y. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, 2016.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Xu, L., Zhao, Y., Zhou. D., Lin, Z., Ng, S. K., and Feng, J. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv,, 2024.
Yan, S., Zhu, T., Wang, Z., Cao, Y., Zhang, M., Ghosh, S., Wu, Y., and Yu, J. Videococa: Video-text modeling with zero-shot transfer from contrastive captioners. arXiv, 2022.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140))1-67, 2020.
Yang, A., Nagrani, A., Seo, P. H., Miech, A., Pont-Tuset, J., Laptev, I.. Sivic, J., and Schmid, C. Vid2seq: Large-scale pretraining of visual language model for dense video captioning. In CVPR, 2023.
Sidorov, O., Hu, R., Rohrbach, M., and Singh, A. Textcaps: dataset for image captioning with reading comprehen- sion. In ECCV, 2020.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantie inference over event descriptions. ACL, 2014.
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Chi, H., Guo, X., Ye. T., Zhang, Y., al. Moviechat: From dense token sparse memory for long video un- derstanding, In CVPR, 2024.
Zhang, B., Li, K., Cheng, Z., Hu, Z., Yuan, Y., Chen, G., Leng, S., Jiang, Y., Zhang, H., Li, X., et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv, 2025.
Su. Y., Lan, T., Liu, Y., Liu, F., Yogatama, D., Wang, Y., Kong, L., and Collier, N. Language models can see: Plugging visual controls in text generation. arXiv, 2022.

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Zhang, C., Lu, T., Islam, M. M., Wang, Z., Yu, S., Bansal, M., and Bertasius, G. A simple IIm framework for long-
range video question-answering In EMNLP, 2024a.
Zhang. T., Kishore, V. Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating generation with bert. In
ICLR, 2020.
Zhang, Y., Sui, E., and Yeung-Levy, S. Connect, collapse, corrupt: Learning cross-moda tasks with uni-modal data. In ICLR, 2024b.
Zhao, L.. Gundavarapu, N. B., Yuan, L., Zhou, H., Yan, S., Sun, J. J., Friedman, L.,, Qian, R., Weyand, T., Zhao, Y., Hornung, R., Schroff, F., Yang, M.-H., Ross, D. A., Wang. H., Adam, H., Sirotenko, M., Liu, T., and Gong, B. VideoPrism: A foundational visual encoder for video understanding. In ICML, 2024.
Zhou, X., Arnab, A., Buch, S., Yan, S., Myers, A., Xiong, X. Nagrani, A., and Schmid, C. Streaming dense video captioning. CVPR, 2024.
12

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
A. Additional Experiment
B.2. LLM summarization
We provide an extended comparison against a broader set of zero-shot video captioning methods on MSR-V VTT test set in Table 8.
To construct the LLM summarization baseline in our experi- ments, we designed prompts by combining the instructions with segment-level captions, as shown in Table 10. This inputs guide the LLM to generate a concise and coherent video-level summary.
We compared our approach with several existing approaches, including: 1) test-time optimization via gradient manipu- lation with CLIP embeddings, e.g., ZeroCap (Tewel et al., 2022) and Tewel et al. (Tewel et al., 2023), 2) optimiza- tion of inference procedure in the decoder using the CLIP image-text similarity, e.g., MAGIC (Su al., 2022), and 3) text-only training methods, e.g., DeCap (Li et al., 2023b)
C. Failure Cases
We present two failure cases from our framework, arising due hallucinations in the initial segment-level captioon
Case 1: Incorrect entity counting
and C3 (Zhang et al., 2024b), which are trained solely text corpora, 4) LLM--based video understanding methods, e.g., VidIL (Wang et al., 2022b) and Video ChatCaptioner (Chen et al., 2023), which utilize proprietary, commercially avail- able LLMs along with textual representations derived from various laage-la models, and 5) LLM summariza- tion, which takes the same set of segment-level captions as method and generates video captions using a pretrained LLM. I--ntrtctttt by text summarization.
Reference captions: ["A group of people dressed in all of the colors of the rainbow sing a happy song.", "Two elderly women dancing with group of men.".
SGVC output: "Two guys in multi-colored tops dance in front of a wall."
While the caption accurately captures specific visual details such "multi-coloree" tops", "wall", and "dance", the VLM hallucinate the number of individuals ("two guys", instead of the actual group 'fve people").
Note that DeCap-MSRVTT, C and VidIL all utilize an- notations from the training dataset but differ in how these annotations are employed. Specifically, DeCap-MSRVTT and C use text annotations from the MSR-VTT training set to train their text decoders. In contrast, VidlI. constructs few-shot exemplars to serve prompts, enabling LLM to perform video captioning through in-context learning.
Case 2. Object misidentification
Reference captions: [*A man fixes a piece of machin- ery that appears to be a miniature tank.", "A guy fixing his camera equipment.".
This comprehensive comparison demonstrates that our ex- plicit scene- scene-graph- based modeling achieves superior perfor- mance over existing zero-shot video captioning methods across all evaluation metrics.
SGVC output: "A man is holding a drill in his hand while working on machinery."
The object in the person's 's hand is camera, but the initial frame-level captioner incorrectly identified it as a "drill", influenced by the surrounding context. This hallucinated detail was propagated to the final consolidated caption.
B. Prompt Instructions
We provide prompt instructions for segment-level caption generation and LLM summarization of these captions, illus- trated here using an image-centric VLM for video caption- ing.
D. Illustration of the Overall Framework
We provide illustrations of the end-to-end flow of our pro- posed framework for long video captioning, along with additional examples, in Figures 4. The framework includes generating segment-level captions using off-the-shelf VLMs, scene graph parsing for these captions, scene graph consoli- dation to produce unified representation, and graph-to-text translation for generate video generation.
B.1. Segment caption generation
Table 9 lists the instructional prompts, generated using GPT- 4, which guide VI.M in generating the segment-level cap- tions. These prompts are designed to ensure captions remain grounded in the visible content of the image, thereby avoid- ing factual errors or hallucinated details not supported by the image. A prompt was randomly selected for each seg- ment, allowing captions to reflect diverse aspects of a video.
E. Additional Qualitative Results
We provide additional qualitative results for video caption- ing on the test set of MSR-VTT (Xu et al., 2016) dataset in Figure 5 and for video paragraph captioning on the ae-val set of the ActivityNet (Krishna et al., 2017a) Captions dataset in Figure 6.
II all experiments, we lext-davinci-002 has been deprecated.
since
13

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
Table Zero-shot video captioning results the MSR-VTT test set (Xu al., 2016). v indicates that the method utilizes reference captions from the MSR-VTT dataset. indicates methods were adapted to zero-shot video captioning by Tewel et al. (Tewel et al., 2023). Bold numbers indicate the highest scores among methods using reference captions.
Method
Backbone VLM
Using reference
B@4
METEOR
CIDEr
PRERT
RBFRT
FRERT
Consolidation-uusse approuches
3.2
14.8
3.1
0.134
0.354
0.225
VidIL (Wang et al., 2022b)
BLIP+CLIP
13.6
20.0
20.2
0.461
0.552
0.490
Video ChatCaptioner (Chen et al., 2023)
BLIP2
13.2
22.0
16.5
0.396
0.510
0.436
BLIP
9.6
21.6
10.8
0.313
0.516
0.395
Summ, w/ Mistral-7B
BLIP2
11.5
23.1
15.4
0.308
0.528
0,397
-exV--Next-7B
15.3
23.8
19.5
0.338
0.535
0.414
BLIP
17.7
22.5
24.0
0.476
0.539
0.490
SGVC (Ours)
BI.IP2
18.4
23.1
26.1
0.467
0.542
0.487
ILAVA--exx--7
17.1
23.0
24.0
0.455
0.547
0.497
Other zero-shot video captioning approaches
MAGIC (Su ct al., 2022)
CLIP
5.5
13.3
7.4
ZeroCap" (Tewel et al., 2022)
2.3
12.9
5.8
CLIP
Tewel et al. (Tewel al., 2023)
3.0
14.6
11.3
0.280
0.391
0.319
Decap- Corpkouus (I.
2023h)
6.0
12.7
12.3
Decap-COCO (Li et ul., 2023b)
CLIP
14.7
20.4
18.6
0.429
0.537
0.465
Dorap-MSRVTT (Li al., 2023b)
23.1
23.6
34.8
(Zhang et al., 2024b)
ImageBind
25.3
23.4
27.8
0.518
0.550
0.519
Table 9.
The list of instructional prompts for segment- level caption generation using image- centrcc VLM.
"Please descrihe what happening the image using one simple sentence. Focus only on what is visible."
"Now, provide single sentence caption that descrihes only what is explicitly shown the image"
"In one sentence, describe what see in the image without adding any extra details."
"Provide a concise one-sentence description of the image, focusing on only the visible elements."
"Please give one-sentence caption that includes only what is clearly shown the image."
"Describe what is happening in the image in one simple sentence, without any added information."
"Please generate single sentence caption that describes only what can be seen in the image."
"Provide one-sentence description of the image, focusing solely on what is shown."
"Now, give brief, one-sentence caption based strictly on the visible content in the image."
single sentence, describe what the image shows, without including anything extra."
Table 10. Illustration of the input construction for LLM summarization, consisting of the instructional prompt and segment-level captions.
Instructional prompt:
Below captions generated from individual frames video, each describing specific moments. Please review these frame-by-frame captions and summarize them into single, compact caption.
Frame captions:
[1/ 6] A woman in blue jacket is sitting in front of sports logo.
[2 6) Woman blue jacket standing outdoors.
[3 6] A man a military uniform is standing front a navy sign.
[4 / 6] Man in military uniform standing in front of navy sign.
[5 6] The image shows three women wearing sports uniforms and holding medals, smiling and posing for the camera.
[6/ 6) Three women wearing blue and white uniforms, smiling and holding medals.
14

---

Fine-Grainee Captioning of Long Videos through Scene Graph Consolidation
(a) Sagment-lewe nement-levvllleee caption generation
(b) Scone graph parsing
Window
Thy bax
"The Image showe woman
VLM
standing in room with window and a toy box."
with
scene graph
Wermen
Towe!
Rooor
Tibe
(c Scome graph consolidaaioon
Towc!
Perron
VLM
person's reaphing
mach
Braw
hold
Hand
Fun turoom
Bothroom
reach1t
Towe
Wommn
rodent he
Pannng
Person
have
stand
w
to In
with
"A holding krown, furry asimol
Parsing
Aniiaal
in towe!""
holld
Person
Graph-to-text
Seaal
(d) Video caption generation
hrild
"A person &s holding e smoll asinal towe!."
Pevsing
VLM
Parion
"A woman is standing in a room with small, and furry rodent wropped in a towel on tiled floor."
Seall
person is holding samal)
VLM
Panson
Towe!
(a) Segment-level caption generation
(b) Scene graph parsing
"A woman standlig nost e basket of loundhy."
VLM
Wommn
Jaundry
Video-level scene graph
Road
VLM
Image shows cortoon sittie ground."
Persing
Dog
Camtaan
Ground
[d Scane graphe con
Grrl
Sheet
hove
Toked
Hair
sanddnggettts
young Passig
EirT
Woman
Eaurdry
carry
stock of folted
Warsing
Newssappr
Graph-to
wedir
VIM
corrying banket of loundry down a dirt road."
womon to block dress
Pansing
Dress
Worann
cnryy
(d) Video caption generation
"A young girl with blond hair and cartoon dog sitting on
wetr
"Woron block dress looking man holding white sheee""
Porsing
dirt
sheet."
VUM
Sheet
Man
Figure 4. Illustrations of the end-ta-end flow of the proposed framework. The pipeline consists of: (1) segment-level caption generation via VLMs, (2) scene graph parsing for each segments, (3) scene graph merging to produce unified representation, and (4) graph-to-text transformation for final caption generation.
15

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
[Ground-truth] bunch of people dance and sing on beach.
(around-truth) contestant sings in competition.
(Tewel et al.] Picture shows a caption of the song, which is sung by actor Justin Bleber and features subtitles that read 'I.
[Tewel et al.] Image shows contestant singing the song, and another with her hand in
[Decap-COCO] A guy in the middle of jump with frisbee. [C'] group of dancers dancing music video.
[Decap-COCO] A person is in front of couple of people on screen.
[C'] performing song the voice.
[LLM summ.] A group of people are surfing, dancing, and running on beach.
[LLM summ.] woman sings stage, accompanied by group of
young girls and singer in red.
[VidIL] group of friends enjoying day at the beach, surfing and
[VidIL] talent show contestant singing on stage while being judged by woman red chair:
eating hot dogs from a nearby stand.
[Video ChatCaptioner] group of people are dancing on the beach to
[Video ChatCaptioner] The video features woman singing happily into 8 microphone.
rap
[Ours] group people dancing the sand on besch with a man in shirt.
[Ours] A woman in a white top singing into microphone on stage.
[Ground-truth] music video for band showing crowd of people.
[Ground-truth] woman with dark makeup applying more.
TTwwel et al.] Picture of the ghostly fiare in a song video. [Decap-COCO] A person is in the middle of room with something. [C'] rock band performing song.
[Tewel et al.] Video showing a makeup artist using her own blood to make an image for film, 'I'm not a. [Decap-COCO] A woman that is holding phone up to her face. [C*] A woman is applying maksup her face.
[LLM summ.] A person is spotlighted in front drum set (1/6), followea by several.
[LLM summ.] person holds black object to their face, tattoos and red nail polish.
with
[VidIL] band performing dark and unconventional setting with city
skyline in the background.
[VidIL] A woman with tattoos and unique style applies makeup in creative way.
[Video ChatCaptioner] group of people are dancing on the beach to
rap
[Video ChatCaptioner] The video features woman face wearing black d'ress in dark room.
her
[Ours] group people playing musical instruments dark room with
light shining their face.
[Ours] with tattoos applying makeup her with ring.
[Ground-truth] man in jumpsuit talking on stage.
[Ground-trie truth] A woman gives a presentation on human musculature.
(Tewel et al.] Video shows the actor performing onstage at a musical in New York, which is being held on stage.
[Tewel et al.] Picture of anatomy by the author, drawn in animation. [Decap-COCO] A person that is in front of a room with phone. is giving lecture biology equation.
[Decap-COCO] A man is in the middle of a room trying to get [C'] Tom jones performs on stage.
[C*]
[LLM summ.] A women black is teaching anatomy using a skaleton, anatomical diagram.
[LLM summ.] jumpsuit.
red outfit, followed by
orange
[VidIL] A physiotherapist is giving lecture on human anatomy and demonstrating different stretches and exercises for the audience.
[VidIL] theatrical performance with fancing and breakdancing elements.
[Video ChatCaptioner] The video features man an orange jumpsuit standing front of cage prison setting.
[Video ChatCaptioner] women standing in front of a skeleton. The setting includes whiteboard with drawing of skeleton.
[Ours] man an orange jumpsuit walking through gate on stage.
[Ours] woman in e suit and black suit is teaching anstomy in front of projector drawing
Figure Additional example of zero-shot video captioning results on MSR-VT ETR-VTTTRR test set. We compare our results with other comparisons, listed from top to bottom !) Tewel et al.: test-time optimization method, 2) Decap-COCO: text-only training on COCO, 3) CS; text-only training on MSRVTT, 4) LLM summarization using Mistral-7B Bitrall-B--nnn 1B-Instrrct-v0.33 5) VidIL: LLM- based video understanding with few-shot examples, 6) Video ChatCaptioner: video understanding via multi-turn conversations between VLM and LLM, and 7) SGVC (Ours).
16

---

Fine-Grained Captioning of Long Videos through Scene Graph Consolidation
(iround-truth] We see ledy sitting in front of a keyboard. The lady moves the sheet music. We see the lady shows the keys and pretend to play. We see the lady change the sheet music. We the lady pretend to play egain.
[LLM summarizaton] person plays plano while women plays the flute; music sheet involved in multiple scenes, woman is sitting desk with keyboard.
[VidIL] woman seen playing a silver in music studio.
[Video ChatCaptioner] The wddeo features a person standing in a living room, wearing a black shirt. The room contains a couch, a chair, and a coffee table. The person facing the camera
appeers to be interacting with objects in the room.
[Ours] A woman is sitting at desk in front of a piano playing flute. The woman is holding a sheet paper with musical symbols on the stand.
[Ground-truth] A dog is seen walking outside walking runny while one person sits and the other leads the dog. The woman and dog walking around the driveway continues walking strange.
en [LLM summarization] A white dog sitting, standing. and walking on various tiled surfaces, sniffing bushes and the ground, and accompanied woman in high heels at different points.
[VidIL] woman is seen walking into house with bag in her hand.
[Video ChatCaptioner] The video features two men interacting in a narrow staircase indoors. The men are not in a hurry and appear to be engaged in physical activity.
[Ours] A white dog wafking on a tiled walkwway in front of a floor. woman is standing outside next to woman with a white door. The dog sniffing& bush near the hedge.
Live Wire ng
(Ground-truth) A man performs on stage standing and playing a dirum set while singing. The crowd waves their hands in the air.
[LLM summarization] A group of musicians are performing on stage, with a man singing into a microphone, a man playing guitar, and several men playing drum sets. The stage is illuminated by red lights.
[VidIL] A band is seen performing on a stage with red lights and a man playing drums. [Video ChatCaptioner] The video features a man outdoors hammering a nail into a board using a sledgehammer at a construction site. He is wearing shirt and shorts, without any safety equipment. The scene has red and black color scheme.
[Ours] A man is playing drum set on stage. The man is singing into a microphone on stage. There is a red light on the stage.
[Ground-truth] A baby sits on a mat on the ground. She plays with an iron. She pretands to iron clothes. She sets the clothes aside.
[LLM summarization] A young child plays with an iron while various individuals Iron clothes mats or the floor. A baby girl in green and white dress is also present.
[VidIL] A child playing with various household items, including a hair dryer, remote control, and iron.
[Video ChatCaptioner] The video features a man sitting on bench in small garden with friend. They appear relaxed in the blue and white color scheme the surronddnggs.
[Ours] A young child is sitting on the floor playing with an ironing machine. She is wearing a green drese. The ironing is white. The child is holding an orange cloth.
Figure 6. Additional example of zero-shot video paragraph captioning results on the ae-val set of the ActivityNet captions dataset. We compare our results with other comparisons, listed from top to bottom as 1) LLM summarization using Mistral-7B -int--ittttttt 2) VidIL, 3) Video ChatCaptioner, and 4) SGVC (Ours).
17


Reviews:
[{"review_id": "cwv9LhoyWJ", "reviewer": "Reviewer_SHDF", "summary": "The authors propose a zero-shot video captioning approach that combines frame-level scene graphs from a video to obtain intermediate representations for caption generation. This method generates frame-level captions using an image VLM, converts them into scene graphs, and consolidates these graphs to produce comprehensive video-level descriptions. To achieve this, the authors leverage a lightweight graph-to-text model trained solely on text corpora, eliminating the need for video captioning annotations.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Reliance on the quality of frame-level captions and vulnerability to cumulative parsing errors in scene graphs**\n\n*  Error propagation is common in other algorithms:\nThe primary objective of this work is to understand video content and generate fine-grained, video-level captions without encoding entire videos at once. This approach is especially effective for handling long videos, which require division into multiple segments or frames. However, consolidating information from these segments inevitably leads to cumulative errors—a challenge common to various algorithms despite their efforts to minimize it.\n\n*  Error quantification is difficult:\nDue to the zero-shot nature of our method, directly quantifying the cumulative error from abstracting segment-level (or frame-level) visual information into textual captions and extracting video-level representation based on a scene graph generation and consolidation is challenging. This is because existing methods employ fundamentally different algorithms, including variations in encoder and decoder choices for representation and captioning.\n\n*  An attempt to estimate the impact of cumulative parsing error:\nOne way to isolate the parsing error in scene graph construction is to compare performance between LLM summarization and our approach, SGVC. Note that LLM summarization follows the same pipeline as SGVC up to the segment-level (or frame-level) captioning stage but consolidates multiple captions directly using an LLM. As shown in Tables 1, 2, and 3 of the main paper, SGVC significantly outperforms LLM summarization, indicating that scene graph consolidation effectively integrates information without introducing substantial errors.\n\n*  Artifact from the generation of segment-level (frame-level) captions:\nAnalyzing the impact of segment-level (or frame-level) caption generation is more complex. However, converting video segments (or frames) into text is sufficiently effective for merging semantic information across multiple segments (or frames), facilitating more coherent video-level understanding. Note that the counterpart, maintaining visual representations for individual segments (or frames), is not interpretable and requires an additional module for consolidating the information from multiple segments (or frames).\n\n\n**2. Limited ability to capture subtle temporal dynamics between frames**\n\nAs discussed in our response to #1, our core contribution lies in developing a pipeline for fine-grained video captioning without encoding entire videos at once, which involves scene graph generation and its consolidation. This allows for a variety of captioning models to be used; our approach is not restricted to frame-level captioners but can also incorporate video-level ones. Notably, the combination of a video-level encoder (InternVL2.5-1B [1]) and SGVC demonstrates promising performance, outperforming LLM summarization, although comparisons with external algorithms are not available at this time. For detailed results, please refer to our response to Reviewer 3YCD for #2.\n\n\n\n\n\n\n\n\n**3. Additional reference**\n\nWe appreciate the reference to HIG [2] for video scene graph generation. This study offers a valuable technique for constructing more comprehensive scene graphs. However, our work is not a competitor to HIG; rather, it has significant potential to benefit from integrating HIG’s scene graph generation techniques to enhance performance. Additionally, HIG requires extensive training to learn a scene graph generation model, but since pretrained models are not publicly available, immediate integration into our approach may not be feasible. \n\n\n[1] Chen, Z. et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. In arXiv, 2024.\n\n[2] Nguyen, T. et al. HIG: Hierarchical interlacement graph approach to scene graph generation in video understanding. In CVPR 2024.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Need more comparisons with work [1].\n\n\n[1] Nguyen, Trong-Thuan, Pha Nguyen, and Khoa Luu. \"Hig: Hierarchical interlacement graph approach to scene graph generation in video understanding.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18384-18394. 2024.", "essential_references_not_discussed": "[1] Nguyen, Trong-Thuan, Pha Nguyen, and Khoa Luu. \"Hig: Hierarchical interlacement graph approach to scene graph generation in video understanding.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18384-18394. 2024.", "other_strengths_and_weaknesses": "Strengths:\n\nThe proposed approach for fine-grained video captioning through scene graph consolidation demonstrates significant strengths, including effective utilization of image-based vision-language models (VLMs) adapted successfully for video domains without requiring task-specific annotations or computationally intensive training.\n\nWeaknesses:\n\n1. Potential reliance on the quality and accuracy of frame-level captions generated by image VLMs, vulnerability to cumulative parsing errors in scene graphs.\n2. Limited ability to capture subtle temporal dynamics between frames due to the discrete nature of the frame-level graph merging process.\n3. Paper was written in a rush.", "other_comments_or_suggestions": "See Weaknesses.", "questions_for_authors": "See Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p3nKCHQAzs", "reviewer": "Reviewer_38AV", "summary": "This paper introduces a novel zero-shot video captioning approach that leverages scene graphs to bridge image and video understanding without requiring paired video-text data. The four-step process involves: generating frame-level captions using an image VLM, converting these into scene graphs, consolidating the graphs into a unified video-level representation through a novel merging algorithm, and finally generating captions using a lightweight graph-to-text model trained only on text corpora.Evaluated on MSR-VTT and ActivityNet Captions datasets, the method outperforms existing zero-shot baselines while reducing computational costs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Reliance on the quality of frame-level captions and vulnerability to cumulative parsing errors in scene graphs**\n\n\n*  Error propagation is common in other algorithms:\nThe primary objective of this work is to understand video content and generate fine-grained, video-level captions without encoding entire videos at once. This approach is especially effective for handling long videos, which require division into multiple segments or frames. However, consolidating information from these segments inevitably leads to cumulative errors—a challenge common to various algorithms despite their efforts to minimize it.\n\n*  Error quantification is difficult:\nDue to the zero-shot nature of our method, directly quantifying the cumulative error from abstracting segment-level (or frame-level) visual information into textual captions and extracting video-level representation based on a scene graph generation and consolidation is challenging. This is because existing methods employ fundamentally different algorithms, including variations in encoder and decoder choices for representation and captioning.\n\n*  An attempt to estimate the impact of cumulative parsing error:\nOne way to isolate the parsing error in scene graph construction is to compare performance between LLM summarization and our approach, SGVC. Note that LLM summarization follows the same pipeline as SGVC up to the segment-level (or frame-level) captioning stage but consolidates multiple captions directly using an LLM. As shown in Tables 1, 2, and 3 of the main paper, SGVC significantly outperforms LLM summarization, indicating that scene graph consolidation effectively integrates information without introducing substantial errors.\n\n*  Artifact from the generation of segment-level (frame-level) captions:\nAnalyzing the impact of segment-level (or frame-level) caption generation is more complex. However, converting video segments (or frames) into text is sufficiently effective for merging semantic information across multiple segments (or frames), facilitating more coherent video-level understanding. Note that the counterpart, maintaining visual representations for individual segments (or frames), is not interpretable and requires an additional module for consolidating the information from multiple segments (or frames).\n\n\n**2. Low performance of 'LLM summarization’ in Table 2**\n\nWe appreciate the suggestions for additional references [1, 2, 3] to compare on ActivityNet Captions. However, we clarify that these methods are extensively trained on large-scale video-text datasets, including ActivityNet Captions, whereas our approach operates in a zero-shot setting without using target dataset annotations during training. Therefore, direct comparisons between [1, 2, 3] and SGVC are not meaningful.\n\n**3. Evaluation on YouCookII**\n\nWe agree that evaluating on a dataset with longer videos, such as YouCookII [4], is meaningful. We are currently conducting experiments on YouCookII and expect that the benefits of SGVC will be particularly evident in datasets with long videos, similar to ActivityNet Captions. However, due to time constraints, we were unable to include these results in our initial response.\n\n[1] Huang, B. et al. VTimeLLM: Empower LLM to Grasp Video Moments. In CVPR 2024.\n\n[2] Tang, Y. et al. Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding. In AAAI 2025.\n\n[3] Kim, M. et al. HiCM2^2 2: Hierarchical Compact Memory Modeling for Dense Video Captioning. In AAAI 2025.\n\n[4] Zhou, L. et al. Towards Automatic Learning of Procedures From Web Instructional Videos. In AAAI 2018.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, the proposed methods make sense to me.", "theoretical_claims": "This paper doesn't contain formal mathematical proofs or theoretical claims requiring verification.", "experimental_designs_or_analyses": "The experimental design is sound overall, with appropriate datasets, baselines, metrics.", "relation_to_broader_scientific_literature": "The use of scene graphs as structured intermediate representations connects to prior work on scene graph generation and reasoning in the image domain. The graph consolidation approach extends ideas from temporal video understanding while addressing the data scarcity problem in video captioning through a zero-shot framework.\n\nThe work also relates to recent efforts bridging image and video domains using large language models and creating more structured representations for video understanding beyond simple frame averaging.", "essential_references_not_discussed": "Some important methods are not discussed, such as [1, 2, 3], and some important benchmarks were not used for evaluation, such as [4].\n\n[1] Huang, B., Wang, X., Chen, H., Song, Z., & Zhu, W. (2024). Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14271-14280).\n\n[2] Tang, Y., Shimada, D., Bi, J., Feng, M., Hua, H., & Xu, C. (2024). Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding. arXiv preprint arXiv:2403.16276.\n\n[3] Kim, M., Kim, H. B., Moon, J., Choi, J., & Kim, S. T. (2024). HiCM2^2\n2: Hierarchical Compact Memory Modeling for Dense Video Captioning. arXiv preprint arXiv:2412.14585.\n\n[4] Zhou, L., Xu, C., & Corso, J. (2018, April). Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence (Vol. 32, No. 1).\n\nPlease also see the \"Other Strengths And Weaknesses\" part.", "other_strengths_and_weaknesses": "Strengths: \nS1: The approach cleverly addresses the scarcity of video-text paired data by leveraging existing image VLMs and text-only training.\nS2: The scene graph consolidation represents an elegant solution to maintain consistency in object identity across frames, addressing a common issue in naive frame-based approaches.\n\nWeaknesses: \nW1: The approach relies on the quality of the initial frame-level captions and scene graph parsing, creating potential error propagation throughout the pipeline.\nW2: The performance of 'LLM summarization' on ANet Caps shown in Table 2 seems to be too low. Some LLM-based methods mentioned in \"Essential References Not Discussed\" part can achieve better performance than that.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Q1: How robust is the scene graph consolidation algorithm to errors in the initial scene graph parsing? Since the method relies on accurate object detection and relationship parsing, I'm curious whether you've analyzed how errors in this initial stage propagate through the pipeline.\n\nQ2: The ActivityNet videos are relatively short compared to many real-world video understanding scenarios. Do you consider evaluating your method on benchmarks with longer videos, such as the [4] mentioned in \"Essential References Not Discussed\" part?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "23LHrSPBwy", "reviewer": "Reviewer_JfkZ", "summary": "This paper proposes a zero-shot video captioning method utilizing frame-level scene graphs obtained from image-based Visual-Language Models (VLMs). The authors consolidate these frame-level graphs into a unified video-level representation using Hungarian matching, followed by generating video-level captions through a lightweight graph-to-text decoder trained only on text corpora. Experimental evaluations on standard benchmarks (MSR-VTT, ActivityNet Captions) demonstrate superior performance compared to existing zero-shot methods, approaching supervised results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Inference cost**\n\nSGVC is efficient in terms of latency, comparable to Decap and C3. However, direct comparisons are challenging as each method utilizes different backbone models for image captioning (SGVC: BLIP, BLIP2, LLAVA-NEXT-7B; Decap: CLIP; C3: ImageBind). In contrast, Video ChatCaptioner and [Tewel et al.] are significantly slower, by at least an order of magnitude, due to the former requiring multiple evaluation rounds and the latter involving test-time optimization.\n\nFor the performance of SGVC with BLIP and BLIP2, please refer to our response for Tables A and B in #2. These models actually outperform our implementation with LLAVA-NEXT-7B.\n\n**2. Different VLM captioning models**\n\nWe evaluate our approach using various VLM backbones, including image-based models (BLIP, BLIP-2, LLaVA-NEXT-7B) and a video-based model (InternVL2.5-1B [1]), on the MSR-VTT (Table A) and ActivityNet Captions (Table B) datasets. The results consistently demonstrate that SGVC outperforms its LLM summarization counterpart, regardless of the chosen VLM backbone. Note that the SGVC results with LLaVA-NEXT-7B in Table B differ slightly from those in the original manuscript, as we extended fine-tuning on Visual Genome paragraphs from 500 to 3200 iterations.\n\nTable A. Ablation of VLM backbones on MSR-VTT test set.\n|Method|Backbone VLM|Using ref.|B4|METEOR|CIDEr|$P_\\text{BERT}$|$R_\\text{BERT}$|$F_\\text{BERT}$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|VidIL|BLIP+CLIP|$\\checkmark$|13.3|20.3|19.4|0.452|**0.553**|0.486|\n|LLM summarization|BLIP||9.6|21.6|10.8|0.313|0.516|0.390|\n|SGVC|BLIP||**17.9**|**22.4**|**24.6**|**0.485**|0.536|**0.509**|\n|Video ChatCaptioner|BLIP2||13.2|22.0|16.5|0.396|0.510|0.436|\n|LLM summarization|BLIP2||11.5|**23.1**|15.4|0.308|0.528|0.389|\n|SGVC|BLIP2||**18.6**|23.0|**26.9**|**0.475**|**0.540**|**0.505**|\n|LLM summarization|LLAVA-Next-7B||15.3|**23.8**|19.5|0.338|0.535|0.414|\n|SGVC|LLAVA-Next-7B||**17.1**|23.0|**24.0**|**0.455**|**0.547**|**0.497**|\n|LLM summarization|InternVL2.5-1B||15.1|**23.6**|19.6|0.339|0.532|0.414|\n|SGVC|InternVL2.5-1B||**17.7**|23.5|**26.7**|**0.481**|**0.550**|**0.513**|\n\nTable B. Ablation of VLM backbones on ActivityNet Captions val set.\n|Method|Backbone VLM|B4|METEOR|CIDEr|$P_\\text{BERT}$|$R_\\text{BERT}$|$F_\\text{BERT}$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|LLM summarization|LLAVA-Next-7B|4.7|11.0|10.3|0.297|0.303|0.300|\n|SGVC|LLAVA-Next-7B|**7.4**|**12.5**|**22.0**|**0.342**|**0.308**|**0.324**|\n|LLM summarization|BLIP|4.0|10.1|8.7|0.304|0.277|0.290|\n|SGVC|BLIP|**7.1**|**11.9**|**17.5**|**0.366**|**0.288**|**0.322**|\n|LLM summarization|BLIP2|4.8|11.0|10.5|0.319|0.300|0.309|\n|SGVC|BLIP2|**7.9**|**12.9**|**22.6**|**0.369**|**0.310**|**0.337**|\n|LLM summarization|InternVL2.5-1B|5.4|11.5|13.0|0.324|0.320|0.322|\n|SGVC|InternVL2.5-1B|**8.4**|**13.4**|**25.6**|**0.348**|**0.326**|**0.337**|\n\n**3. Failure cases**\n\nWe provide two failure cases given by hallucinations in the initial image-based caption:\n\nExample 1.\n* Reference captions: [\"A group of people dressed in all of the colors of the rainbow sing a happy song.\", \"Two elderly women dancing with a group of men.\", …]\n* SGVC output: \"Two guys in multi-colored tops dance in front of a wall.\"\n\nWhile the caption accurately captures specific visual details such as \"multi-colored tops\", \"wall\", and \"dance\", the VLM hallucinates the number of individuals (reporting \"two guys\" instead of the actual group of five people).\n\nExample 2.\n* Reference captions: [\"A man fixes a piece of machinery that appears to be a miniature tank.\", \"A guy fixing his camera equipment.\", … ]\n* SGVC output: \"A man is holding a drill in his hand while working on machinery.\"\n\nThe object in the person's hand is a camera, but the initial frame-level captioner incorrectly identified it as a \"drill\", influenced by the surrounding context. This hallucinated detail was propagated to the final consolidated caption.\n\n**4. Impact of threshold $\\tau$ in graph merging**\n\nWe conducted an ablation study by varying the cosine similarity threshold, $\\tau$, as shown in Table C. The results demonstrate stable performance within the range $\\tau \\in [0.80, 0.95]$.\n\nTable C. Ablation study on the threshold $\\tau$.\n|$\\tau$|B4|METEOR|CIDEr|$P_\\text{BERT}$|$R_\\text{BERT}$|$F_\\text{BERT}$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|0.95|17.1|23.0|24.0|0.455|0.547|0.497|\n|0.90|17.3|22.5|24.5|0.463|0.542|0.499|\n|0.85|17.2|22.8|23.8|0.455|0.545|0.496|\n|0.80|16.9|22.8|23.0|0.445|0.546|0.490|\n\n**5. Frame selection criteria**\n\nIn our experiment for Table 3, we uniformly sampled frames to ensure coverage of the entire video content.\n\n**6. Quantitative evaluation of hallucination reduction**\n\nThank you for your suggestion. However, conducting a quantitative evaluation within a short time frame is challenging due to the lack of necessary annotations and evaluation metrics.\n\n[1] Chen, Z. et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. In arXiv 2024.", "claims_and_evidence": "Most claims are supported with clear empirical evidence, such as thorough experimental results using multiple established metrics (BLEU, METEOR, CIDEr, BERTScore). However, the claim regarding inference-time efficiency is not sufficiently supported by explicit timing comparisons with baseline methods. The claim that the approach reduces hallucinations is qualitative rather than quantitatively supported. Quantitative evaluation would strengthen this claim.", "methods_and_evaluation_criteria": "The proposed methods are well-reasoned and effectively address the zero-shot captioning challenge. The benchmarks and evaluation metrics are also appropriate.", "theoretical_claims": "The paper makes no explicit theoretical claims. It is primarily empirical and methodological. The proposed methods are practical, with clear descriptions and validation through empirical studies.", "experimental_designs_or_analyses": "Almost of the experimental designs make sound. However, it is difficult to visually recognize the results of Table 1 with and without reference captions because they are directly attached to the SGVC.", "relation_to_broader_scientific_literature": "The paper situates its contributions within the current literature well, citing recent works on zero-shot video captioning and Dense Video Captioning (DVC) frameworks. Notably, it introduces a novel framework that leverages scene graphs, a method not commonly used in previous zero-shot video captioning research.", "essential_references_not_discussed": "The key contribution of the paper is the generation and consolidation of Vision-Language Model (VLM) captions to enhance zero-shot video captioning. The proposed approach effectively addresses the problem with a well-structured process. There are no critical missing references.", "other_strengths_and_weaknesses": "__Strength__\n\nS1. Novelty in applying scene graph consolidation for zero-shot video captioning.\n\nS2. Clear methodological explanations supported by strong experimental validation.\n\nS3. Effectively addresses practical limitations inherent in zero-shot captioning scenarios.\n\n__Weakness__\n\nW1. Lack of evaluation regarding inference-time cost.\n\nW2. Missing ablation studies examining the impact of different Visual-Language Models.\n\nW3. Can the authors provide the case of captioning failure? (e.g. due to VLM hallucination)Suggestion 1. Can you provide the case of captioning failure? (e.g. due to VLM hallucination)\n\nW4. How sensitive is the performance of your method to varying the threshold (τ) during graph merging?\n\nW5. What are the criteria for selecting frames in an ablation study for the number of frames in Table 3?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m7CkP23Hxr", "reviewer": "Reviewer_3YCD", "summary": "This paper proposes a zero-shot video captioning framework that leverages scene graph consolidation to bridge image-based vision-language models (VLMs) to video understanding without requiring paired video-text annotations.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Categorization of algorithms**\n\nGiven the diversity of zero-shot video captioning approaches, their categorization is not straightforward, and we acknowledge that alternative perspectives exist. However, since SGVC consistently outperforms LLM summarization throughout our experiments, its improved performance is not merely due to the use of LLMs but rather the effectiveness of the proposed algorithmic components. In the revised manuscript, we will reorganize Table 1 to better position SGVC and clarify the sources of its performance gains.\n\n\n**2. Comparison against LLaVA-NEXT-7B itself and VideoPrism-B**\n\nSince LLaVA-NEXT-7B [1] is an image-based captioner, directly evaluating its performance on video captioning is not straightforward. A meaningful approach is to compare LLM summarization and SGVC, as both share the same pipeline for extracting image-level captions using LLaVA-NEXT-7B but differ in their subsequent processing. As shown in Tables 1 and 2, our results clearly demonstrate the superior performance of SGVC over the simple combination of LLaVA-NEXT-7B and LLMs.\n\nVideoPrism-B [2] indeed achieves outstanding performance on MSR-VTT, but it has been trained on an extensive collection of public and private videos with paired captions. As a result, its encoder is specifically optimized for video data, contributing to its superior performance. Additionally, it employs a powerful decoder (PaLM-2) for caption generation, which further enhances its results. Given these factors, we argue that a direct comparison between VideoPrism-B and our method is not entirely fair. Since VideoPrism-B’s model and source code are not publicly available, we instead selected InternVL2.5-1B [3], a video captioner, to assess SGVC’s performance with a video-based model. Our results in Tables D and E demonstrate that SGVC outperforms LLM summarization when integrating a clip-level captioner, highlighting the effectiveness of our graph-based approach.\n\nTable D. Experiments on MSR-VTT, using InternVL2.5-1B as VLM backbone. \n\n\n|Method|B4|METEOR|CIDEr|$P_\\text{BERT}$|$R_\\text{BERT}$|$F_\\text{BERT}$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|LLM summarization|15.1|**23.6**|19.6|0.339|0.532|0.414|\n|SGVC|**17.7**|23.5|**26.7**|**0.481**|**0.550**|**0.513**|\n\n\n\n\nTable E. Experiments on ActivityNet Captions, using InternVL2.5-1B as VLM backbone.\n\n\n|Method|B4|METEOR|CIDEr|$P_\\text{BERT}$|$R_\\text{BERT}$|$F_\\text{BERT}$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|LLM summarization|5.4|11.5|13.0|0.324|0.320|0.322|\n|SGVC|**8.4**|**13.4**|**25.6**|**0.348**|**0.326**|**0.337**|\n\n\n**4. Efficiency**\n\nPlease refer to our response to #1 for Reviewer JfkZ.\n\n\n**5. Additional ablation studies**\n\nPlease refer to Tables A, B, and C for the ablation studies regarding VLM captioner backbones and hyperparameter $\\tau$, respectively. \n\nAdditionally, we analyze the impact of the hyperparameter $k$, which determines the size of the extracted subgraph. As shown in Table F, lower $k$ values yield more concise subgraphs centered on salient objects, enhancing precision-oriented metrics (CIDEr, $P_\\text{BERT}$). In contrast, higher $k$ values produce richer subgraphs that capture broader contextual information, leading to improvements in recall-oriented metrics (METEOR, $R_\\text{BERT}$).\n\nTable F. Ablation study on the hyperparameter $k$, on the MSR-VTT test set.\n\n|$k$|B4|METEOR|CIDEr|$P_\\text{BERT}$|$R_\\text{BERT}$|$F_\\text{BERT}$|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|1|17.1|23.0|**24.0**|**0.455**|0.547|**0.497**|\n|3|17.1|**23.7**|22.1|0.428|**0.551**|0.482|\n\n\n**6. Clarification on the graph-to-text model**\n\nOur graph-to-text model consists of a BERT-based encoder and a T5-base decoder, totaling 235M parameters. Given that the other two-stage video captioning models often rely on LLMs, our approach is significantly more lightweight. We will include detailed information about our architecture in the revised version of our paper. More importantly, we plan to release our source code and models upon acceptance of our paper.\n\n\n**7. Error propagation case study**\n\nAs mentioned in our response to #3 for Reviewer JfkZ, hallucinations in frame-level caption generation introduce errors in the final video-level captions. We will discuss cases of error propagation in the revised manuscript.\n\n\n[1] Liu, H. et al. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/, 2024.\n\n[2] Zhao, L. et al. VideoPrism: A Foundational Visual Encoder for Video Understanding. In ICML 2024.\n\n[3] Chen, Z. et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. In arXiv, 2024.", "claims_and_evidence": "Some claims made in the submission are not sufficiently supported.\n1. Efficiency Claims: While the authors emphasize \"lower inference cost\", LLaVA-NEXT-7B’s computational cost (7B parameters) negates this claim compared to smaller VLMs. VideoPrism-B [1] w/ PaLM-1B achieves higher performance with fewer parameters as reported in Table 11 of [2]. Besides, efficiency metrics (FLOPs, latency, or GPU memory usage) are not reported.\n\n[1] Zhao, Long, et al. \"Videoprism: A foundational visual encoder for video understanding.\" arXiv preprint arXiv:2402.13217 (2024).\n\n[2] Wang, Yi, et al. \"Internvideo2: Scaling foundation models for multimodal video understanding.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.", "methods_and_evaluation_criteria": "While the methods align with zero-shot goals, the quantitative experiments are insufficient. Kindly refer to the Experimental Designs Or Analyses section for more details.", "theoretical_claims": "No theoretical claims are made. The paper focuses on empirical validation.", "experimental_designs_or_analyses": "The paper’s experimental design and analysis suffer from critical flaws that undermine the validity of its claims.\n1. Misleading Categorization & Incomplete VLM Comparisons: While the authors position their method as \"text-only training\" (Table 1), the reliance on LLaVA-NEXT-7B (line 305) for frame-level captioning inherently makes this an LLM-based approach. This misleading presentation undermines the fairness of comparisons in Table 1, where LLM-based methods (e.g., Video ChatCaptioner) are treated as separate categories. Besides, the authors fail to directly compare against LLaVA-NEXT-7B itself or state-of-the-art video-centric VLMs like VideoPrism-B [2], which achieves CIDEr=40.3 on MSR-VTT (vs. 24.0 in this work) as reported in Table 11 of [1]. This omission raises concerns about whether the proposed scene graph consolidation adds value over simply using the base VLM’s outputs.\n2. Efficiency Claims: While the authors emphasize \"lower inference cost,\" LLaVA-NEXT-7B’s computational cost (7B parameters) negates this claim compared to smaller VLMs. As mentioned above, VideoPrism-B w/ PaLM-1B achieves higher performance with fewer parameters, suggesting the scene graph pipeline introduces bottlenecks rather than enhancements. Besides, efficiency metrics (FLOPs, latency, or GPU memory usage) are not reported.\n3. Inadequate Ablation Study: The only ablation study in this paper is about the number of frames, which is not enough to support technical claims. It is necessary to conduct more ablation studies on the VLM backbone (e.g., BLIP-2 vs. LLaVA), hyperparameter k in subgraph extraction, and the graph-to-text model.\n\n[1] Wang, Yi, et al. \"Internvideo2: Scaling foundation models for multimodal video understanding.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[2] Zhao, Long, et al. \"Videoprism: A foundational visual encoder for video understanding.\" arXiv preprint arXiv:2402.13217 (2024).", "relation_to_broader_scientific_literature": "The paper’s contributions align with several key directions in vision-language research, particularly in video captioning and multimodal representation learning.", "essential_references_not_discussed": "This paper has discussed enough essential related works.", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of parsing frame-level captions into scene graphs, then merging them into a unified graph and converting it into a video-level caption is reasonable and interesting.\n\nWeaknesses:\n1. Misleading Categorization & Incomplete VLM Comparisons: While the authors position their method as \"text-only training\" (Table 1), the reliance on LLaVA-NEXT-7B (line 305) for frame-level captioning inherently makes this an LLM-based approach. This misleading presentation undermines the fairness of comparisons in Table 1, where LLM-based methods (e.g., Video ChatCaptioner) are treated as separate categories. Besides, the authors fail to directly compare against LLaVA-NEXT-7B itself or state-of-the-art video-centric MLLMs like VideoPrism-B [2], which achieves CIDEr=40.3 on MSR-VTT (vs. 24.0 in this work) as reported in Table 11 of [1]. This omission raises concerns about whether the proposed scene graph consolidation adds value over simply using the base MLLM’s outputs.\n2. Suboptimal Performance: While adopting a multimodal large language model (MLLM) as the backbone, the performance on zero-shot video captioning is inferior compared to other MLLMs and VLMs as reported in [1]. e.g., 24.0 (this work) vs. 40.3 (VideoPrism-B [2] w/ PaLM-1B) in CIDEr scores (↑)  on MSR-VTT.\n3. Efficiency Claims: While the authors emphasize \"lower inference cost,\" LLaVA-NEXT-7B’s computational cost (7B parameters) negates this claim compared to smaller VLMs. As mentioned above, VideoPrism-B w/ PaLM-1B achieves higher performance with fewer parameters, suggesting the scene graph pipeline introduces bottlenecks rather than enhancements. Besides, efficiency metrics (FLOPs, latency, or GPU memory usage) are not reported.\n4. Inadequate Ablation Study: The only ablation study in this paper is about the number of frames, which is not enough to support technical claims. It is necessary to conduct more ablation studies on the VLM backbone (e.g., BLIP-2 vs. LLaVA), hyperparameter k in subgraph extraction, and the graph-to-text model.\n5. Missing Details on Graph-to-Text Model: The technical details on this module are insufficient, restricting reproducibility. It is reasonable to report detailed network architecture. e.g., feature dimensions and how these features are processed using mathematical notations. Besides, the claim of \"lightweight\" is unsupported without parameter counts or running time metrics versus baselines.\n\n[1] Wang, Yi, et al. \"Internvideo2: Scaling foundation models for multimodal video understanding.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[2] Zhao, Long, et al. \"Videoprism: A foundational visual encoder for video understanding.\" arXiv preprint arXiv:2402.13217 (2024).", "other_comments_or_suggestions": "1. Limitations of Image VLMs: The framework relies on LLaVA-NEXT-7B for frame captioning. A discussion of how errors in frame-level captioning propagate to the final video caption (e.g., via case studies) would improve transparency.", "questions_for_authors": "1. Performance Gap vs. VideoPrism-B: VideoPrism-VideoPrism-B [1] w/ PaLM-1B CIDEr=40.3 on MSR-VTT, far exceeding your CIDEr=24.0. Does this gap stem from limitations in scene graph consolidation, or is it inherent to using LLaVA-NEXT-7B for frame parsing?\n2. Ablation on Subgraph Extraction: How does the choice of k affect caption quality? For instance, does higher k improve coherence at the cost of missing transient objects?\n\n[1] Zhao, Long, et al. \"Videoprism: A foundational visual encoder for video understanding.\" arXiv preprint arXiv:2402.13217 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Paper proposes a video captioning approach that first extracts frame-level captions, parses them into intermediate scene graphs, consolidates those scene graphs and then uses this consolidates scene graph to produce a video-level caption without ever relying on video caption annotations. Paper received 1 x Reject, 1 x Weak Reject and 2 x Weak Accepts. Main concerns of reviewers centered around (1) fundamental limitations of the proposed approach, such as propagation of error and inability to observe and caption subtle actions that require understanding of motion [SHDF, 38AV], (2) lack of performance on certain benchmarks [38AV] or with respect to some SoTA methods [3YCD] (e.g., VideoPrism and VideoCoCa), (3) missing evaluation of inference-time cost [JfkZ], and (4) missing ablations with different VLMs [JfkZ]. Authors have provided rebuttals that successfully addressed many of the concerns, with an exception of (2). The issues brought up by [SHDF] and [38AV] regarding the fundamental limitations of the proposed approach are valid in AC's opinion. At the same time, AC also believes that proposed approach is not necessarily designed to serve as the ultimate solution for the video captioning, but rather as an intriguing benchmark that shows how much is possible with frame-based information extraction. In this context, paper makes an interesting and valuable contribution to the community. As such, AC believes the paper is valuable for the community and would make a good addition to ICML, thereby recommending ACCEPTANCE.
Decision: Accept (poster)