TripleSumm: Adaptive Triple-Modality Fusion for Video Summarization
Authors: Sumin Kim
Hyemin Jeong
Mingu Kang
Yejin Kim
Yoori Oh
Joonseok Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Video Summarization
Video Understanding
Multimodal Learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose TripleSumm, a frame-level adaptive multimodal fusion model, and introduce MoSu, the first large-scale benchmark with all three modalities, achieving state-of-the-art video summarization performance.

Abstract: The exponential growth of video content necessitates effective video summarization to efficiently extract key information from long videos. However, current approaches struggle to fully comprehend complex videos, primarily because they employ static or modality-agnostic fusion strategies. These methods fail to account for the dynamic, frame-dependent variations in modality saliency inherent in video data. To overcome these limitations, we propose **TripleSumm**, a novel architecture that adaptively weights and fuses the contributions of visual, text, and audio modalities at the frame level. Furthermore, a significant bottleneck for research into multimodal video summarization has been the lack of comprehensive benchmarks. Addressing this bottleneck, we introduce **MoSu** (Most Replayed Multimodal Video Summarization), the first large-scale benchmark that provides all three modalities. Extensive experiments demonstrate that TripleSumm achieves state-of-the-art performance, outperforming existing methods by a significant margin on four benchmarks, including MoSu. Our code and dataset are available at https://github.com/smkim37/TripleSumm.

Published as a conference paper at ICLR 2026
TRIPLESUMM: ADAPTIVE TRIPLE-MODALITY FUSION FOR VIDEO SUMMARIZATION
Sumin Kim Hyemin Jeong" Mingu Kang"
Yejin Kim
Yoori Oh!
Joonseok Leef
Seoul National University
{sumink, hyeminjeong, gms5560, a2000yejin, 0203, joonseok)
ABSTRACT
The exponential growth of video content necessitates effective video summariza- tion to efficiently extract key information from long videos. However, current approaches struggle to fully comprehend complex videos, primarily because they employ static or modality-agnostic fusion strategies. These methods fail count for the dynamic, frame-dependent variations in modality saliency inherent in video data. To overcome these limitations, we propose TripleSumm, a novel architecture that adaptively weights and fuses the contributions of visual, text, and audio modalities at the frame level. Furthernore, a significant bottleneck for research into multimodal video summarization has been the lack of compre- hensive benchmarks. Addressing this bottleneck, we introduce MoSu (Most Re- played Multimodal Video Summarization), the first large-scale benchmark that provides all three modalities. Extensive experiments demonstrate that Triple- Summ achieves ttate-ffthheaa performance, outperforming existing methods by a significant margin on four benchmarks, including MoSu. Our code and dataset are available at https: / /gtthttttth///ggg/gg com/ smkimm3 im3 /ttrr//rrrp/pp /rrrrlriieeseeeeeeee Sum.
INTRODUCTION
With recent advances in smart mobile devices and data communication, video content has explo- sively grown across various platforms such as YouTube or TikTok. At the same time, recent trend shows shifted preference towards short-form content, leading to an increased demand for excerpting the main plot of long videos through summaries. Video summarization, the task of extracting key segments that fully represent the content of the original video, serves to meet this demand.
Existing work video summarization (Apostolidis al., 2021b; Son et al., 2024; Kim et al., 2025a) has primarily focused on seeking a model architecture mapping frame-level visual features to their importance scores summary. However, human comprehension of video is inherently multi- modal process that integrates diverse cues beyond the visual. Most existing architectures, which focus solely on the visual modality, therefore overlook complementary information present in other modalities. Fig. 1, for example. illustrates a music audition that the primary modality to understand the content varies across the video. At the point (a), the text modality (speech) is the most informa- tive to grasp the judge's evaluation, while at the point (b), visual-audio cues play more important role to enjoy the robot's performance, At the point (c), all three modalities contribute conjunction. Observing that the modality-specific -secific importance significantly varies even within the same video, are motivated to utilize multiple modalities in an adaptive manner to dynamically weight the most informative modality for more effective video summarization.
Recognizing this, recent work has begun to utilize multimodal signals for video summarization (He al., 2023; Li et al., 2023; Hua et al., 2025; Guo et al., 2025). However, it has been largely underexplored what is the best-suitable way to fuse multimodal signals in the context of video sum- marization. The objective of summarization is not merely predict a true label, but contrast more important frames to less important ones. This necessitates a discriminative approach to appro- priately weigh feature salience by considering two key factors: modal temporal dependency,
"Equal contribution Corresponning authors

---

Published as a conference paper at ICLR 2026
Modallty Impoctance
Visaal
Test
Test
Viaai
Vixual
lest
Audio
Audio
Audio
"Tell us
about you"
(No Text)
X-makon Yee eneg."
May lithtng
Mustle Primce
Stngiig
Musl
Replayed Stats
(a)
(b)
(c)
Time
Figure I: Illustration of Dynamic Importance of Modalities. Text is most salient (a), while visual-audio are dominant at (b), At (c), all three contribute significantly. This highlights the neces- sity for an adaptive model that dynamically weighs saliency of each modality frame-by-frame
which involves comparing the same modality features across adjacent time steps, and inter-modal coherence, which involves comparing different modalities at the same time point. Despite using rich trimodal inputs, most current architectures still employ simple or static fusion mechanisms (e.g., standard self-attention or cross-attention or prioritize a single modality, failing to dynamically fo- cus on the most informative cue at each frame. This results in compromised performance when non-visual cues become highly descriptive.
In this paper, we propose TripleSumm, novel video summarization model that flexibly and adap- tively leverages multimodal features, while being robust in the presence of missing modalities, cov- cring all three key aspects of video: visual, text, and audio. Specifically, TripleSumm employs the following two novel components. First, the Multi-scale Temporal block employs hierarchical sliding window structure with varying window sizes to effectively detect subtle temporal changes without losing the overall narrative of the video. Second, the Cross-modal Fusion block incorporates fusion tokens to explicitly learn which modality to prioritize at each moment.
In addition the model architecture, training data has been another bottleneck for multimodal video summarization. In spite of its importance, there is no publicly available video data equipped with visual-text--nn textadic features and importance score annotations a sufficient scale. Existing datasets are often in a limited scale (Gygli et al., 2014; Song et al., 2015) or with limited set of modalities (Narasimhan et al., 2022; Sul et al., 2023; Argaw et al., 2024). To address this, we introduce large- scale video summarization dataset providing features for all three key modalities, namely, Most Replayed Multimodal Video Summarization (MoSu). Composed of 52,678 in-the-wild videos and watch behavior aggregated from at least 50,000 viewers per video, this new dataset serves as highly reliable training and evaluation set for trimodal video summarization.
Through comprehensive experiments, we verify that TripleSumm outperforms existing models by large margin on multiple benchmarks. Notably, our method robustly generates a reasonable video summary even when one or more modalities are absent, dynamically relying each modality de- pending on the content. Our qualitative analysis clearly demonstrates that the proposed method adaptively fuses multimodal information frame-by by-fae basis.
Our main contributions can be summarized as follows:
We propose the TripleSumm architecture that adaptively fuses visual, text, and audio modalities at frame level. With its temporal and modality blocks, it dynamically adjusts the importance of each modality and effectively captures the micro- and macro-level information of the video.
We present the MoSu, the first large-scale video summarization dataset that provides trimodal features of each video, establishing a reliable foundation for multimodal video summarization.
We demonstrate that TripleSumm achieves the tate-of-the-an on four major video summariza- tion benchmarks, including MoSu, while maintaining parameter efficiency.
RELATED WORK
Video Summarization. Early work in video summarization primarily focused on modeling tempo- ral dependencies within the visual features, utilizing Recurrent Neural Networks or Long Short-term Memory (Zhang et al., 2016; Zhao et al., 2017; 2018; 2021b). More recent models employ a Self- Attention mechanism to capture global, long-range dependencies (Fajtl al., 2018: 8; Jung et al.,
2

---

Published as a conference paper at ICLR 2026
2019; Zhu et al., 2021; Wang al., 2020; Apostolidis et al., 2021b; Jiang & Mu, 2022; Terbouche et al., 2023; Son et al., 2024). Additionally, some works leverage graph- based models (Park et al., 2020: Zhao et al., 2021a; Zhu et al., 2022; Zhang et al., 2024), Generative Adversarial Networks (Mahasseni et al., 2017; Yuan et al., 2019; Apostolidis et al., 2019; 2021a), or diffusion models (Yu al., 2024; Shang al., 2025; Kim et al., 2025a).
Despite these architectural advances, relying only on visual features limits comprehensive under- standing of a video. Shifting towards multimodal approach, textual data such as transcripts or image captions have been incorporated (Narasimhan et al., 2021; ; Huang et al., 2021; Narasimhan al., 2022; Ghauri et al., 2021; Li et al., 2023; Argaw et al., 2024; Qiu et al., 2024), and recently CFSum (Guo et al., 2025) have pioneered to leverage visual, text, and audio signals for summa- rization. However, the majority of these methods still employ simple or static mechanisms (e.g. standard self-attention or fixed cross- cats-attention modules) to fuse multimodal features, without being fully adaptive. Consequently, they either treat modalities uniformly or prioritize visual information, using text as merely supplementary features (Narasimhan al., 2021; Li et al., 2023). Due to this in- herent bias and lack of dynamic weighting, these models fail to adequately summarize videos where non-visual cues (text audio) dominate the content. Large Language Models are also employed to capture the multimodal context, leveraging their powerful reasoning capabilities (Lin et al., 2024; Hua et al., 2025; Lee al., 2025; Kang al., 2025). Meanwhile, audio remains as largely under- utilized modality in video summarization, with a few exceptions in recent work (Badamdorj et al., 2021: Liu et al., 2022: Zhao et al., 2022; Xie et al., 2022). Our work addresses the need for bal- anced utilization of all three primary modalities (visual, text, and audio) achieving comprehensive video summarization.
Video Summarization Datasets. SumMe (Gygli et al., 2014) and TVSum (Song et al., 2015) have been widely used, but they suffer from two critical limitations: extremely small scale (25 and 50 videos, respectively) and unimodal annotation based solely on visual cues, making them suboptimal for multimodal research. large-scale benehmark Mr. HiSum (Sul et al., 2023) addresses the seale issue, but its deliberate exclusion of audio-rich categories such as music leaves the modality gap unaddressed. More recent datasets have focused on incorporating textual information. However, they often introduce new challenges, such as domain bias towards instructional videos (e.g., (Argaw et al., 2024)) or live streams (e.g., (He et al., 2023)), or suffer from sparse ground truths (e.g.. (Qiu et al., 2024)). We address the need for a large-scale, trimodal and diverse dataset by introducing the MoSu dataset, detailed in Sec. 4.
3 TRIPLESUMM: THE PROPOSED METHOD
We present our video summarization model, TripleSumm, depicted in Fig. 2. Beginning with the multimodal feature extraction (Sec. 3.1), we subsequently describe how we fuse the temporal and cross-modal information using our two core components, Multi-scale Temporal block and Cross- modal Fusion block (Sec. 3.2). Lastly, we describe the inference process to predict the frame-level importance scores and to generate the final summary video from them (Sec. 3.3).
3.1 INPUT REPRESENTATION
A raw video consists of multiple modality streams with different sampling rates. We preprocess these streams to produce a set of synchronized feature sequences, each resampled N time steps. (See Sec. 5.1 for details.) At each time step i = ..., N, we are given multimodal signals. Under our trimodal setting, we assume that we have visual = (V I,.,, V. text T = {Ti,... TN ). and audio A {A1,.. .... An J. where V., Tú, and A, are the moddllty-sppeiffi raw data; e.g., V, is a 2D RGB image. We will describe our method based on this trimodal setting, but our method itself can be extended to arbitrary set of multimodal features.
For each item in each modality sequence, we extract feature representations, denoted by X(1A.a) employing modality-speciffe pre-trained encoders:
X v(V) E NXD. X ((T)ERNX,) X = w(A) E E NxD.
(I)
where @ is an image encoder (e.g., (Szegedy et al., 2015; Radford et al., 2021)), & is a text encoder (e.g., (Liu et al., 2019)), and is audio encoder (e.g., (Gong et al., 2021)). Each pre-trained
3

---

Published as a conference paper at ICLR 2026
E
ThE Turrooe Pho Roodaal Eronding
$ Sppoodt
Visual v
H'
Temporal Block
Fusion Block
RCMF
Text g
A spoous
MST
Pred Score
Ha
((s.
Audio
NT
TPE + LME
Cross-modal Fusion Block (CMF)
Multi-scale Temporal Block (MST)
a v.
#MST
Q
"MST
WSA
-
MST
#MST
HMM
HMST
Figure 2: Overall architecture of TripleSumm. Visual, text, and audio features are first encoded and linearly projected, then aggregated into fusion tokens, refined through the Multi-scale Tempo- ral block (MST, lower left), and fused in the Cross-modal Fusion block (CMF, lower right). The fused representation is passed through a prediction head generate frame-level importance scores.
encoder produces embedding of its own size, denoted by Dv, Dr, and Da for visual, text, and audio, respectively.
These modality-specific features reside in different latent spaces. To effectively fuse them in subse- quent modules, we project them into a common embedding space of size D. For this, we apply a linear projection (Linear) and layer normalization (LN) (Ba et al., 2016) to each modality-specine features X (v,t,) to produce per-modality embeddings (l,,,)) E N>D:
E (.t.a) LN (N( (1:44)))
We denote the embedding at time step i, which the i-th row E(r,,a),, as e (1.t.a) On top of these per-modality embeddings, we introduce another cross-modal embedding, denoted by E (ef...., e xx E NxD where ef = Agg(e)" ef). The aggregation function (Agg) can be either deterministic function (e.g., average), main motivation to add this is to avoid potential bias introduced by conventional cross-modal fusion methods, e.g., asymmetric using one modality query to attend on others. Our fused embeddings E' serve as an anchor to integrate all modalities, promoting their equitable engagement.
learnable model (e.g., multi-layer perceptrons). The
Lastly, we construct the final token embeddings by adding temporal positional encoding (TPE) tpe; E D (Vaswani al., 2017) and a learnable modality embedding (LMH) Ime.ee, RD to distinguish each time step and the origin of the modality:
(.E,t,a) e v.v.I tpe + (...a.) for 1,..., N.
(3)
In matrix form, the final input matrix stacked over all time steps denoted by (ffnta)) E RR NDD
3.2 TEMPORAL AND CROSS-MODAL REFINEMENT
The core of our proposed architecture hierarchical 'refine-and-fuse strategy, designed to learn integrated representations from the input sequences (f.tttt))) This is achieved by interleav- ing two key components: Multi-scale Temporal block (MST) for temporal refinement within each modality, and Cross-modal Fusion block (CMF) for inter-modal information exchange. We stack L interleaved layers, where each layer is composed of P temporal blocks and Q cross-modal blocks.
Multi-scale Temporal Block. From the input sequence (f...ta)) the MST learns temporal pat- terns within each modality by employing Windowed Self-Attention (WSA) (Beltagy et al., 2020; Liu et al., 2021), restricting the range self-attention for a query token q, E IR at time step to keys and values within local window of size centered at i. These local keys and values, denoted

---

Published as a conference paper at ICLR 2026
by K2 E R and V, R=xd, constructed by collecting key vectors k; and value vectors V, within a window such that j = l - jl I w, where w E {n E N 1 S n S N and n is odd] } and dg is the dimensionality of the key vector. Formally, the overall process of the MST is as follows:
WSS = Aun(LN((h"))) (N m (ili-))Ssw) (N (::l- (::1-)1(w))) + h""
MSS = FFN(LN( ve ())) + WSA.
(4)
where m E (f),,t,)) Attn Q, K, V) = softmax (QKT/ / /vdz) V is the standard attention (Vaswani et al., 2017), and FFN feed-forward network. The 'Multi-scale" characteristic is achieved by varying the window size w at each layer. to allow the model to capture semantics from local to global scales. Specifically, the initial layers employ smaller window sizes capture fine-grained local dependencies between adjacent frames. Then, later layers progressively expand the windows to capture long-range dependencies across frames. WSA is applied to each modality with shared parameters, enabling the model to capture general temporal patterns while keeping pa- rameter usage efficient. An ablation on parameter sharing is detailed in the App. C.5. Furthermore, WSA is not just crucial for the model to capture multi-granular context across the input video, it is also beneficial for computational efficiency, reducing the complexity from O(N2) for standard self-attention (w N) at each layer.
Cross-modal Fusion Block. Whereas the MST focuses on refining temporal patterns within each modality, the CMF is designed to model the interactions across different modalities, independently at each time step. To allow the model to select the most informative modality at each time step without being biased towards any particular modality, this block employs the cross- noss-attention mechanism single query each time step i, and the corresponding modality- specific tokens, h (ust,a] as the keys and values. The query token then attends to this collection of three context tokens, allowing it to weigh and aggregate information from the most relevant modality at that specific moment.
taking the fusion token h
Formally, given the features (f.r.ta) from the preceding temporal block at time step i, it performs MMST
bf Attn(LN( kisT,)), (Ntte)) LN(h + hAST,
(5)
h CMF.: FFN(LN(6()) + bf,,
(6)
where Attn(Q, K, V) softmax (QKT) / V, and de is the key vector size. The Layer Nor- malization LN(.) is applied within each modality separately. The updated fused embeddings, H are enriched with contextual information selectively drawn from all three modalities.
A key design principle of TripleSumm is the separation of temporal and cross moosl fusion. The temporal context is modeled exclusively by the MST, allowing it to focus solely on integrating modality information at each corresponding time step. On the other hand, the cross-modal fusion is solely performed by the CMF, at each time step independently. This design is not just beneficial for the model to explicitly learn the two orthogonal (temporal and multimodal) patterns in the video, but also allows efficient implementation by momentarily merging the batch and time dimensions, enabling parallel processing across all tokens.
3.3 MODEL TRAINING AND INFERENCE
Finally, a prediction head linearly projects the refined fused features H to the importance score $ E [0, 1] of each frame, interpreted as the probability to be included in the final video summary. More details about the prediction head are described in App. A.I.
The model is trained to minimize the squared L2 loss between the predicted score vector $ = {$1...., SN) } and the ground truth scores S = { ... SN Formally, loss is given by
L(S Š) = 1|S $113 $11 = || Linear(H)
Following established practice in video summarization (Otani et al., 2019; Son et al., 2024; Kim et al., 2025a), the final summary is obtained by selecting set of temporally coherent shots that maximizes the predicted frame-level importance scores under a fixed length budget. The detailed segmentation and selection procedure is provided in App. A.2.
5

---

Published as a conference paper at ICLR 2026
4 MOSU DATASET
Video summarization has been suffering from the data shortage for both quality and scale. For ex- ample, the most widely used public datasets for the last decade were SumMe (Gygli et al., 2014) and TVSum (Song et al., 2015), which consist of just 25 and 50 videos, respectively. Obviously, these are in an extremely small scale, led serious overfitting environment, but no other dataset has been available until recently due to the high labeling cost. Mr. HiSum (Sul et al., 2023) resolved this by taking the 'Most Replayed' statisties from YouTube as reliable proxy for per-frame importance based on collective viewer engagement (see App. B.2 for details). However, despite this advantage, Mr. HiSum remains unsuitable for trimodal video summarization, as it provides only visual fea- tures without accompanying text or audio modalities and additionally removes audio-centric video categories, further limiting its applicability multimodal fusion.
In order to reliably train and evaluate trimodal video summarization models, we introduce a new large-scale called dataset MoSu (Most Replayed Multimodal Video Summarization), curated from YouTube-SM (Abu-El-Haijp et al., 2016), which provides videos with multi-label annotations across 3,862 classes drawn from knowledge graph entities. For annotations, we follow similar procedure to Mr. HiSum (Sul et al., 2023), collecting the 'Most Replayed' statistic. We construct this dataset by filtering videos that satisfy the following criteria: 1) both an English transcription and an audio track are available, either originally provided or automatically generated by YouTube's 's caption translation, to satisfy the trimodal condition, 2) over 50,000 views to obtain the Most Replayed statistics, and 3) at least 120 seconds long to ensure sufficiently long content for meaningful summarization.
Dataset Statistics. MoSu contains 52,678 videos corresponding to nearly 4,000 hours, covering a vast range of topics with 3,406 cat- egories. The average video length in MoSu is 272.3 seconds, ranging from 120 to 501 sec- onds. As compared with other video summa- rization datasets in Tab. 1, MoSu is the first large-scale dataset to provide all three modali- ties (visual, text, and audio). as opposed to pre- vious ones with visual-only (e.g., Mr. HiSum) or bimodal (e.g., MMSum).
Modality
Dataset
Videns
Duration
Category
Visual
Text
Audio
SumMe TVSum MrlliSum
25
1.1
50
3.5
10
31,892
1,788.0
3,509
TLDWW
12.160
628.5
185
BLISS
13,303
1,109.0
LAVS-P
250K
55,416.6
6,700
MMSum
5,100
1.229.9
170
MoSa
52,678
3,983.7
3,406
Table 1: Statistics of various video summariza- tion datasets. Total duration is reported in hours.
Thematic Categories. Motivated by the finding that the complexity of video summarization significantly varies by video topics (Sul et al., 2023), we aim to analyze video summarization quality across various thematic categories on MoSu as well. Since the original 3,406 categories inherited from YouTube--8M are too fine-grained, we cluster them into 10 distinct topical groups. Specifically, we first construct a topic space by applying K-Means clustering to the SBERT embeddings (Reimers & Gurevych, 2019) of the 3,862 entity descriptions from Wikipedia. We also map each video to the same embedding space using its concatenated entity labels, and select the closest cluster centroid as its pseudo-class. See App. B.3 for details of the resulting thematic groups; App. G.2 for summarization performance across clusters, showing how they affect the ease of summarization.
EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Datasets. Our primary evaluation is conducted on our MoSu dataset, introduced in Sec. 4. To further validate generalizability of the proposed model, we also evaluate its performance on three widely-used external datasets: a large-scale Mr. HiSum (Sul et al., 2023), and two human-annotated SumMe (Gygli et al., 2014) and TVSum (Song et al., 2015). We follow the original data split for Mr. HiSum and MoSu. For SumMe and TVSum, we evaluate under a 5-fold cross-validation framework using two distinct protocols: the traditional train/validation (TV) split without a test set (Li et al., 2023; Son et al., 2024; Lee et al., 2025), and train/validation/f (TVT) split (Kim al., 2025a) to correct the overfitting issues inherent in the TV setup. Details for protocols are available in App. B.5.

---

Published as a conference paper at ICLR 2026
Modality
Method
7 1
pt
mAP50 1 MAP15
Params
GFLOPS
VASNet 2018)
0.151
0.219
64.49
31,05
8.13M
1.99G
PGL--UMM (Apooolidis 302169
0.151
0.218
64.97
30.63
5.31M
1.21G
CSTA (Son et 2000
0.291
0.398
71.77
40.65
10.56M
11.37G
A2Summ (fe 2023)
0.181
0.257
66.48
35.70
2.48M
1.35G
SSPVS diieal
0.190
0.271
66.10
32.65
112.81M
43.64G
Joint- VA (Balamdiejj et 2021)
2021)
0.190
0.272
65.68
32.25
4.21M
1.63G
UMT (Lix 2022)
0.239
0.334
68.83
36.73
4.66M
1.39G
CFSum (Gao al, 2025)
0.277
0.374
70.97
38.20
19.83M
8.52G
TripleSumm (Ours)
0.351
0.472
74.72
44.42
1.37M
0.97G
Table 2: Comparison on MoSu. Best and second-best are boldfaced and underlined, respectively.
(a) Mr. HiSum
(b) SumMe
(c) TVSum
TVT
TVT
TV
TVT
TV
Method
Method
P
mAP50
mAP15
P
P
VASNet
0.069
0.102
58.69
25.28
VASNet
0.089
0.099
0.160
0.170
0.153
0.205
0.160
0.170
PGL-SUM
0.097
0.141
61.00
27.45
PGL-SUM
0.104
0.116
0.192
0.213
0.141
0.186
0.157
0.206
CSTA
0.128
0.185
63.38
30.42
CSTA
0.133
0.148
0.246
0.274
0.168
0.221
0.194
0.255
A2Summ
0.121
0.172
63.20
32.34
A2Summ
0.088
0.096
0.108
0.129
0.157
0.206
0.137
0.165
SSPVS
0.078
0.113
59.48
26.35
SSPVS
0.142
0.157
0.192
0.257
0.171
0.226
0.181
0.238
Joint
0.167
0.231
65.88
35.23
0.117
0.129
0.230
0.256
0.142
0.188
0.166
UMT
0.178
35.65
0.165
0:241
0.268
0.144
0.189
0.179
Ours (Visual)
0.187
0.258
67.16
35.57
Ours (Puth
0.162
0.187
0.265
0.296
0.198
0.259
0.211
0.275
Ours (Falt
0,258
0.352
70.72
40.88
Ours (Malaa)
0.172
0.192
0.282
0.314
0.200
0.262
0.217
0.282
Table 3: Performance comparison on (a) Mr. HiSum, (b) SumMe, and (c) TVSum. OurS(rall) refers to the model trained on each respective dataset. (vray) is trained using only visual fea- tures, and OurS(Mosw) is pre-trained on MoSu and fine-tuned on each target dataset.
Data Preprocessing. As mentioned in Sec. 3.1, our model requires temporally aligned feature sequences of the same length. We preprocess each modality in MoSu as follows. For the visual modality, we sample frames at fps and encode them using pre-trained CLIP (Radford et al., 2021), For the text modality, we extract time-stamped transcripts from YouTube and obtain RoBERTa (Liu et al., 2019) features by taking the sentence-level [CLS] token, which is then broadcast to all frames covered by its duration; frames without transcripts are filled with a default embedding vector. For audio, we extract features at 1-second interval using centered-window approach: for each second f, a 10-second segment is cropped from the interval [t 5, + 5] and encoded into a feature vector using a pre-trained Audio Spectrogram Transformer (Gong al., 2021).
On Mr. HiSum, SumMe, and TVSum, we adopt their officially provided visual features for a fair comparison. Since these datasets do not provide text audio streams, we generate frame-level text captions using an image captioning model, Qwen2.5-VL-7 ---2-5--VL-B--nstrrue (Bai et al., 2025), and extract the raw audio directly from the videos. For each modality, we then follow the preprocessing proce- dures originally defined for each dataset. Detailed preprocessing steps are provided in App. B.4.
Implementation Details. We use L interleaved layers, with P Multi-scale Temporal blocks (thus, in total 4 temporal blocks) and Q 2 Cross-mode modal Fusion block. To facilitate local- to- to-global feature capture, we progressively increase the attention window size (w), from up the entire sequence length N. More details are described in App. A.I.
Evaluation Metrics. Following Otani et al. (2019), we primarily report rank-based correlation metrics, Kendall's r (Kendall, 1945) and Spearman's 's (Zwillinger & Kokoska, 1999), on the frame- level importance prediction, being consistent with recent studies (Son et al., 2024; Terbouche et al., 2023; Kim et al., 2025a). On MoSu and Mr. HiSum datasets where the Most Replayed statistics are used as ground- truth, we further assess the highlight detection performance with mean Average Precision (mAP) following Sul et al. (2023). Following this protocol, each video is divided into 5-second segments and ranked by predicted importance. The top 50% (mAP50) and 15% (mAP15) of these segments are then evaluated the predicted highlights against the ground-truth.
7

---

Published as a conference paper at ICLR 2026
Lock House
the Text!
This Cary."
"Caask
M
tho text!
32
"Day, busdio'
(No Text
IC gRggeea here"
Ow teal!
Bpeakiad
ByeakHad
Didteell....
Dasceddon)
Enccerdend
Ma
mmm
Visual Text
Audio
Hous
Text
Audio
(a) Guitar demonstration where attention shifts among visual, text, and audio cues as the content changes.
(b) Playing accordion video with less visual-text infor- mation, where audio cues predominate.
Figure 3: Qualitative Examples on MoSu. The graph in the middle visualizes the fusion token's 's attention weights, illustrating how our model dynamically estimates the saliency of each modality and thus maintains strong summarization accuracy even when some modalities are missing.
5.2 QUANTITATIVE EVALUATION
Evaluation on MoSu. As shown in Tab. 2. TripleSumm achieves clear state-of-ther of-tth-art performance on MoSu, surpassing all unimodal and multimodal baselines by substantial margin across all met- rics. In addition to its accuracy gains, TripleSumm is also highly efficient with only 1.37M pa- rameters, which significantly smaller than strong baselines such as CSTA (10.56M), and UMT (4.66M). The results demonstrate that TripleSumm delivers superior predictive performance while maintaining a markedly lighter and more efficient architecture.
Evaluation on Other Datasets. Tab. 3(a) shows that method achieves state-of-the-at perfor- mance on most metrics on Mr. HiSum. Notably, outperforms all baselines even only with the visual features OurS(visul)- validating the effectiveness of the proposed Multi-scale Temporal block in capturing salient temporal features from a single modality. In addition, our full model significantly outperforms its visual-only version, highlighting the benefit of leveraging multimodal information.
Tab. 3(b-) indicate that our full model outperforms all baselines by large margin when trained and evaluated on SumMe and TVSum, both of which are human-annotated datasets. This is particularly noteworthy, since the ground-truth annotations are solely based on visual information. Our model's performance with three modalities suggests that non-visual modalities provide crucial contextual cues for identifying key visual moments, implying that our trimodal approach learns a synergistic relationship among modalities. Lastly, our model performs even better when pre-trained on MoSu and fine-tuned on the target dataset (the last row, OurSMosa)- This result indicates that the rich representations learned from the large-scale MoSu dataset are effectively transferable.
5.3 QUALITATIVE DEMONSTRATION
We illustrate couple of qualitative examples of the video summary with TripleSumm in Fig. 3. As seen in the graph in the middle, showing the attention weights of the Fusion Token, TripleSumm adapts the saliency of each modality on frame-by-frame basis the content changes. Through these weights, we interpret that our method has properly learned and reflected the relative importance of each modality, aligned well with human intuition, instead of exhibiting chaotic fluctuations or collapsing to a static mean. In guitar demonstration video in Fig. 3a, the model relies mostly on audio signals, while correctly shifting attention to the visual modality for the opening logo (first scene) and to text for the narration (second scene). Fig. 3b shows an accordion performance with minimal visual changes and no text. Despite missing modalities, it maintains strong predictions by properly attending to the audio. This capability stems from our design using the neutral Fusion Token, not a moddllty--ppeciff one, as the query, allowing the model to weigh modalities free from bias. Overall, observe that TripleSumm produces summaries more faithful to the ground truth than the baselines by appropriately weighing modalities at each moment. More qualitative results available in App. H.
8

---

Published as a conference paper at ICLR 2026
(a) Input Modalities
(b) Attention Window Size
p
m0
mAP15
Window Size (w)
T
p
mAP50
mAP15
0.298
0.404
72.20
40.24
Constant (Local)
0.297
0.409
71.61
39.73
0.267
0.364
70.01
37.37
Constant (Wide)
0.304
0.416
72.54
40.26
0.271
0.370
69.96
37.88
Constant (Global)
0.317
0.431
73.46
41.62
0.307
0.416
72.41
41.20
Wide-to-Loca
0.325
0.441
73.20
42.13
0.328
0.442
73.77
42.39
Local-to-Wine Widd
0.335
0.455
73.83
43.41
0.331
0.448
73.85
42.64
Global-to-Loo
0.345
0.461
74.36
43.61
0.351
0.472
74.72
44.42
Local-to-Global
0.351
0.472
74.72
44.42
(c) MST & CMF Block
(d) Modality Fusion
MST CMF
m A500
mAP15
Method
m.A P5A
mA P15
0.250
0.352
70.01
37.35
Static
0.338
0.458
74.08
43.31
0.337
0.452
73.90
43.08
Global
0.344
0.461
74.26
43.59
0.351
0.472
74.72
44.42
Dynamic
0.351
0.472
74.72
44.42
Ablation studies on input modalities, attention window sizes, MST & CMF blocks, and modality fusion methods.
Table 4:
5.4 ABLATION STUDY
We further conduct several ablation studies on MoSu, unless noted otherwise.
Ablation Input Modalities. We evaluate method with all possible combinations of in- put modalities in Tab. 4(a). Among the unimodal settings, the visual modality (V) turns out to be strongest, while the audio (A) slightly outperforms the text (T). This highlights the power of the au- dio stream, which continuously provides rich contextual cues, whereas transcripts are often sparse in fin-the-wild" videos, leaving many segments without semantic information. Each bimodal com- bination outperforms the strongest single modality, demonstrating a significant synergistic effect. Building on this synergy, the full trimodal configuration achieves the best performance, confirming that all three modalities provide unique and valuable signals for summarization.
Ablation Window Size. We compare several hierarchical windowing options for the multi- scale temporal blocks. We denote w E {3, 5, 7} Local windows and its 3-9 times larger size as Wide windows. A Global window indicates w N. where N is the total number of frames in the video. Using these window sizes, we experiment with various scheduling strategies: using constant window size, and gradually expanding or shrinking. For instance, Local-to-Gloo -Golbal means starting with Local window size at the first layer, and adopts gradually larger sizes in the next layers, reaching to the whole frames at the last one; e.g., [3, 9, 27, N] or [5, 15, 45, Nj.
We first observe in Tab, 4(b) that relying Here, the Constant (Global) is equivalent to the standard self-attention, using all frames at all layers, but it underperforms the following mixed strategies. Among the hierarchical ones, observe two patterns. First, narrower-tor soroower--o-wideee todderrr strategies (Local-to- Wide/Global tend to outperform its opposite (Wide/Globallto-Looe)
constant window size yields suboptimal performance.
demonstrating that bottom-up approach to learn fine-grained details first and establish broader context based on them is more appropriate for video summarization. Also, having Globally-windowed layer slightly improves performance for both bottom-up and top-down strategies, at the cost of computational overhead. We draw these conclusions based on more fine- grained experiments with various choices of w, detailed in App. C.2.
Ablation on MST & CMF Blocks. We conduct an ablation study to verify the effect of the two components of our method: the Multi-scale Temporal (MST) block and the Cross-modal Fusion (CMF) block. The results in Tab. 4(c) clearly show that both blocks are essential to achieve the full performance of our model, observing a significant performance drop when one of them removed. Comparing the effect of the two components, we observe a more substantial degradation when MST removed. This highlights that the MST block the most critical module for accurately captur- ing the long-range temporal dependencies and multi-scale context required for precise frame-level importance prediction. In conclusion, the CMF block indeed provides the final multimodal boost by dynamically fusing the processed features, and the MST block forms the foundational backbone essential for leveraging the complex, long-range temporal structure of the video, confirming their respective roles in proposed architecture.
9

---

Published as a conference paper at ICLR 2026
Ablation Modality Fusion Method. A main hypothesis of our paper is that the saliency of each modality varies frame by frame. To verify its validity, we conduct an ablation study on its design. The simplest approach to fuse trimodal tokens would be simply taking average over them with same weights (Static). A slightly advanced variant would learn a single scalar weight for each modality by averaging cross-modal attention scores time (Global). These global weights are then uniformly applied across the video, allowing adaptation across modalities but not across frames. Finally, Dynamic fusion computes cross-altention scores independently every frame and uses these scores to determine the relative contribution of modality features on a per- frame basis.
As shown in Tab. 4(d), the performance gradually improves as modality and temporal adaptivity are added (Dynamic >> Global > Static). This result confirms that the model effectively leverages frame-level flexibility without collapsing to trivial solution, validating that our Dynamic fusion mechanism is stable and essential for effective multimodal summarization.
5.5 ZERO-SHOT PERFORMANCE ON LONG-FORM VIDEOS
For a more rigorous evaluation of the scalability and generalization capabilities of our model, we conduct another zero-shot experiment on 50 signif- icantly longer videos unseen at training. On aver- age, these test videos are 70,4 minutes long, cover- ing a wide range of topics. More details about this test set are provided in App. B.6.
Method
T
p
mAP50 mAP15
Random
0.000
0.000
50.59
16.18
VASNet
0.024 0.036
54.15
18.95
PGL-SUM
0.024 0.035
54.29
17.55
CSTA
0.083 0.123
58.09
22.26
A2Summ
0.042 0.062
54.04
18.66
SSPVS
0.033 0.048
53.60
18.45
All models have been trained on MoSu, and Tab. 5 reports their zero-shot inference performance on this long video set. The results clearly demon- strate the superior generalization of TripleSumm, particularly in the demanding long-form setting. On these extremely long videos, our model signifi- cantly outperforms all baselines on the rank-based correlation metrics. Specifically, our TripleSumm achieves the highest scores in Kendall's T (0.128) and Spearman's (0.189). This confirms that our adaptive fusion architecture generalizes far more effectively to complex, long-form content, maintaining the most accurate frame-level importance prediction even when faced with entirely new domains and semantic structures.
Joint- Vint-VVA
0.052 0.077
53.99
19.90
UMT
0.066 0.097
56.05
23.10
CFSum TripleSumm 0.128 0.189
0.061 0.089
56.32
20.59
59.70
23.27
Table 5: Zero-shot performance on long videos. All models trained on MoSu and tested directly on long video dataset.
To the best of our knowledge, this is the first summarization benchmark on hours-long videos. Considering the fact that video summarization gets more challenging but meaningful on these longer videos with more complex story-telling structures, we believe that this experiment also significantly contributes the community in advancing video summarization in real-world scenarios.
CONCLUSION
We primarily explore deep integration of three modalities, visual, text, and audio, for the task of video summarization. Our proposed method, TripleSumm, dynamically assesses the saliency of each modality different moments and adaptively utilizes them to produce a superior sum- mary video. To facilitate further development, we also introduce MoSu, a new large-scale trimodal dataset. Our experiments show that TripleSumm achieves highly competitive results on multiple benchmarks. These findings suggest the potential benefits of integrating all three modalities for more comprehensive understanding of video content. We hope that the proposed TripleSumm model and the MoSu dataset can contribute to future advancements in multimodal video summarization.
While our work adheres to the standard three-step protocol (frame importance scoring, temporal segmentation, and segment selection) for fair comparison existing benchmarks, we believe that developing a fully end-to-end trainable model would be a promising future direction. Exploring methods that learn select coherent summary clips directly, rather than only learning frame-level scores, would present a valuable opportunity to advance the field.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was also supported by the SOFT Foundry Institute at SNU, Samsung Electronics, Youl- chon Foundation, National Research Foundation of Korea (NRF) grants (RS-2021-NR05515, RS- 2024-00336576, R8S2023-00222633, RS-2025-25399600
RS-2025-254216420 and the Institute for Information Communication Technology Planning Evaluation (ITTP) grants (RS-2022- 11220264, RS-2024-00355131) funded by the Korean government.
LARGE LANGUAGE MODEL USAGE
We acknowledge the of large language models enhance readability, correct grammar, and refine sentence structure during the preparation of this paper. All intellectual contributions, including the core ideas, methodology, and analysis, are solely the work of the authors.
REFERENCES
Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaynnarasimhan. Youtube-8m. 8m: A large-scale video classification benchmark. arXiv: 08677,, 2016.
Evlampios Apostolidis, Eleni Adamantidou, Alexandros I Metsai, Vasileios Mezaris, and Ioannis Patras. Unsupervised video summarization via attention-driver adversarial learning. Proceed- ings of the International Conference Multimedia Modeling (MMM), 2019.
Evlampios Apostolidis, Eleni Adamantidou, Alexandros I. Metsai, Vasileios Mezaris, and Ioannis Patras. Ac-sum-gan gan: Connecting actor-critic and generative adversarial networks for unsuper- vised video summarization. IEEE Transactions on Circuits and Systems for Video Technology 2021a.
(TCSVT),
Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and Ioannis Patras. Combining global and local attention with positional encoding for video summarization. In Proceedings of the IEEE International Symposium on Multimedia (ISM), 2021b.
Dawit Mureja Argaw, Seunghyun Yoon, Fabian Caba Heilbron, Hanieh Deilamsalehy, Trung Bui, Zhaowen Wang, Franck Dernoncourt, and Joon Son Chung. Scaling up video summarization pre- training with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv: 1607.06607 2016.
Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng. Joint visual and audio learn- ing for video highlight detection. Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), 2021.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv:2502. 13923, 2025.
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The dong-document transformer. arXiv:2004.051500 2020.
Jiri Fajtl, Hajar Sadeghi Sokeh, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino. Summarizing vidcos with attention. In Proceedings of the Asian Conference on Computer Vision (ACCV), 2018.
Junaid Ahmed Ghauri, Sherzod Hakimov, and Ralph Ewerth. Supervised video summarization via multiple feature sets with parallel attention. arXiv:210..11530, 2021.
Yuan Gong, Yu-An Chung, and James Glass. arXiv:2104.01778. 2021.
Ast:
Audio spectrogram transformer.

---

Published as a conference paper at ICLR 2026
Maosheng Guo, Yu Zhang, and Ting Liu. Gaussian transformer: lightweight approach for natural language inference. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2019.
Yaowei Guo, Jiazheng Xing, Xiaojun Hou, Shuo Xin, Juntao Jiang, Demetri Terzopoulos, Chenfanfu Jiang. and Yong Liu. Cfsum: A transformer-basee multi-modal video summarization frame- work with coarse-fine fusion. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2025.
Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating summaries from user videos. In Proceedings of the European Conference Computer Vision (ECCV), 2014.
Bo He, Jun Wang, Jielin Qiu, Trung Bui, Abhinav Shrivastava, and Zhaowen Wang. Align and attend: Multimodal summarization with dual contrastive losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
Hang Hua. Yunlong Tang, Chenliang Xu, and Jiebo Luo. 12xum-llm: Cross-mode modal video sum- marization with temporal prompt instruction tuning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2025.
Jia-Hong Huang, Luka Murn, Marta Mrak, and Marcel Worring. Gpt2mvs: Generative pre-trained transformer-2 for multi-modal video summarization. In Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR), 2021.
Hao Jiang and Yadong Mu. Joint video summarization and moment localization by cross-task sam- ple transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2022.
Yunjae Jung, Donghyeon Cho, Dahun Kim, Sanghyun Woo, and So Kweon. Discriminative feature learning for unsupervised video summarization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2019.
Jeonghun Kang, Soonmok Kwon, Joonseok Lee, and Byung-Hak Kim. DIAMOND: An LLM- driven agent for context-aware baseball highlight summarization. Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Workshop for Research Agent Language Models, 2025.
Maurice G Kendall. The treatment of ties in ranking problems. Biometrika, 33(3):239-255) 1945.
Kwanseok Kim, Jachoon Hahm, Sumin Kim, Jinhwan Sul, Byunghak Kim, and Joonseok Lee. Summdiff: Generative modeling of video summarization with diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025a.
Minsun Kim, Dawon Lee, and Junyong Noh. Generating highlight videos of user-specified length using most replayed data. In Proceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI), 2025b.
Min Jung Lee, Dayoung Gong, and Minsu Cho. Video summarization with large language mod- els. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2025.
Haopeng Li, Qiuhong Ke, Mingming Gong, and Tom Drummond. Progressive video summarization via multimodal sulf-supe learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
Jingyang Lin, Hang Hua, Ming Chen, Yikang Li, Jenhao Hsiao, Chiuman Ho, and Jiebo Luo. Videoxum: Cross-modal visual and textural summarization of videos. IEEE Transactions Multimedia (TMM), 26:5548-5560, 2024.
Ye Liu, Siyuan Li. Yang Wu, Chang Wen Chen, Ying Shan, and Xiaohu Qie. Umt: Unified multi- modal transformers for joint video moment retrieval and highlight detection. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
12

---

Published as a conference paper at ICLR 2026
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 11X1v:1907.11992. 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Behrooz Mahasseni, Michael Lam, and Sinisa Todorovic. Unsupervised video summarization with adversarial Istm networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell. Clip-it! language-guided video summa- rization. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Medhini Narasimhan, Arsha Nagrani, Chen Sun, Michael Rubinstein, Trevor Darrell, Anna Rohrbach, and Cordelia Schmid. TI: dw? summarizing instructional videos with task rele- vance and mooss-modal saliency. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Mayu Otani, Yuta Nakashima, Esa Rahtu, and Janne Heikkila. Rethinking the evaluation of video summaries. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2019.
Jungin Park, Jiyoung Lee, Ig-Jae Kim, and Kwanghoon Sohn. Sumgraph: Video summarization via recursive graph modeling. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Danila Potapov, Matthijs Douze, Zaid Harchaoui, and Cordelia Schmid. Category-specific video summarization. In Proceedings of the European Conference on Computer Vision (ECCV), 2014.
Jielin Qiu, Jiacheng Zhu, William Han, Aditesh Kumar, Karthik Mittal, Claire Jin, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Ding Zhao, et al. Mmsum: A dataset for multimodal summarization and thumbnail generation of videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. 1.X1:1198..1088 2019.
Zirui Shang, Yubo Zhu, Hongxi Li, Shuo Yang. and Xinxiao Wu. Video summarization using denoising diffusion probabilistic model. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), 2025.
Noam Shazeer. Glu variants improve transformer. arXiv:2002.05202. 2020.
Jaewon Son, Jaehun Park, and Kwangsu Kim. Csta: Con-based spatiotemporal attention for video summarization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Yale Song. Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tysum: Summarizing web videos using titles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Jinhwan Sul, Jihoon Han, and Joonseok Lee. Mr. hisum: A large-scale dataset for video highlight detection and summarization. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
13

---

Published as a conference paper at ICLR 2026
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2015.
Hacene Terbouche, Maryan Morel, Mariano Rodriguez, and Alice Othmani. Multi-annotation atten- tion model for video summarization. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems (NIPS), 2017.
Junyan Wang, Yang Bai, Yang Long, Bingzhang Hu, Zhenhua Chai, Yu Guan, and Xiaolin Wei. Query twice: Dual mixture attention meta learning for video summarization. In Proceedings of the ACM International Conference on Multimedia (MM), 2020.
Jiehang Xie, Xuanbai Chen, Shao-Ping Lu, and Yulu Yang. A knowledge augmented and multimodal- framework for video summarization. In Proceedings of the ACM International Conference Multimedia (MM), 2022.
Minghao Xu, Hang Wang. Bingbing Ni, Riheng Zhu, Zhenbang Sun, and Changhu Wang. Cross- category video highlight detection via set-based learning. In Proceedings of the IEEF/CVF Inter- national Conference Computer Vision (ICCV), 2021.
Qinghao Yu, Hui Yu, Ying Sun, Derui Ding, and Muwei Jian. Unsupervised video summarization based on the diffusion model of feature fusion. IEEE Transactions on Computational Social Systems (TCSS), 11(5):6010- 102100212222 2024.
Li Yuan, Francis EH Tay, Ping Li, Li Zhou, and Jiashi Feng. Cycle-sum: Cycle- consistentt adversar- ial Istm networks for unsupervised video summarization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2019.
Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman. Video summarization with long short- term memory. In Proceedings of the European Conference Computer Vision (ECCV), 2016.
Yunzuo Zhang, Yameng Liu, Weili Kang, and Ran Tao. Vss-net: Visual semantic self-mining net- work for video summarization. IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 34(4):2775 27785 2024.
Bin Zhao, Xuclong Li. and Xiaoqiang Lu. Hicrarchical recurrent neural network for video summa- rization. In Proceedings of the ACM International Conference on Multimedia (MM), 2017.
Bin Zhao, Xuelong Li, and Xiaoqiang Lu. Hsa-rnn: Hierarchical structure-adaptive rnn for video summarization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2018
Bin Zhao, Haopeng Li, Xiaoqiang Lu, and Xuelong Li. Reconstructive sequence-graph network for video summarization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2445)22793-2801, 2021a,
Bin Zhao, Xuelong Li, and Xiaoqiang Lu. Tth-rmn: Tensor-traie hierarchical recurrent neural net- work for video summarization. IEEE Transactions on Industrial Electronics (TIE), 68(4):6229 3637, 2021b.
Bin Zhao, Maoguo Gong, and Xuelong Li. Hierarchical multimodal transformer to summarize videos. Neurocomputing, 468:360-369, 2022.
Wencheng Zhu, Jiwen Lu, Jiahao Li, and Jie Zhou. Dsnet: A flexible detect-to- -ummmmaio- network for video summarization. IEEE Transactions on Image Processing (TIP), 30:948-966 2021.
Wencheng Zhu, Yucheng Han, Jiwen Lu, and Jie Zhou. Relational reasoning over spatial-temport graphs for vidco summarization. IEEE Transactions on Image Processing (TIP), 31:3017-3031, 2022.
14

---

Published as a conference paper at ICLR 2026
Daniel Zwillinger and Stephen Kokoska. CRC standard probability and statistics tables and formu- lae. Cre Press, 1999.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A IMPLEMENTATION DETAILS
A.I EXPERIMENTAL SETUP
The model's ' hyperparameters are detailed in Tab. I. The architecture consists of 2 interleaved layers with 2 multi- scale temporal blocks and 2 cross-modal fusion blocks. Totally, the architecture con- sists of 4 multi-scale temporal blocks. To capture features from local-to-global scales, the attention window size w is progressively increased across these blocks, from local self-attention (a window size of 5) up to 15, 45, and N, where N is the input sequence length. We use the Swish-Gated Lin- ear Unit (SwiGLU) (Shazeer, 2020) for the Feed-Forward Network (FFN) layers. The final score prediction head comprises sequence of linear projection, GeLU activation, layer normalization, another linear projection, and a final sigmoid activation. The hidden dimension within this head is set to 192. Architectural parameters include embedding dimension (D) of 128. The model was trained for 100 epochs using the AdamW optimizer. The learning rate was initialized at x 10" and adjusted via Cosine scheduler. All experiments were conducted single NVIDIA RTX 100.
Category
Hyperparameter
Value
Embedding Dimension (D)
128
Number of multi-scale temporal block (P)
2
Number of cross-modal fusion block (Q)
2
Model Architecture
Number of interleaved block (L)
2
Number of attention heads
4
Hidden dimension of prediction head
192
Epoch
100
Batch Size
64
Training Details
Dropout rate
0.1
Initial Learning Rate
x 10
Table I: Hyperparameters for the proposed model.
A.2 SUMMARY GENERATION PROCEDURE
To construct the final summary from the predicted per-frame importance scores S, we follow the standard pipeline used in prior work (Son et al., 2024; Kim et al., 2025a). Each input video is first partitioned into temporally coherent shots using Kernel Temporal Segmentation (KTS) (Potapov et al., 2014; Zhang et al., 2016). The importance score of each shot is then computed as the mean of the predicted frame-level scores within that shot. Next, we select an optimal subset of shots that maximizes the total shot score while satisfying predefined length budget (e.g., 15% of the original video duration); this selection is formulated as a 0/1 knapsack problem with shot length as the weight and shot score as the value. Finally, the chosen shots are concatenated in their original temporal order produce the final summary video.
B DATASETS DETAILS
B.1 DETAILED STATISTICS OF THE MOSU DATASET
We provide more detailed characteristics of the MoSu dataset, introduced in Sec. 4. First, Fig. I shows the distribution of video duration in this dataset, ranging from 120 to 500 seconds. Tab. II offers statistics of the video duration, text tokens, and audio tracks. As filter out all videos without a valid audio track (e.g. a scenario that can occur if creator mutes or removes the audio), all videos in this dataset have audio information, making it suitable for triple modality fusion.

---

Published as a conference paper at ICLR 2026
Statistic Category
Value
Duration Statistics
Avg. Duration
272.25
Std. Deviation
102.43 sec
Min Duration
120,00 sec
Max Duration
501. .00 sec
3322
Textual Statistics
Total # of Tokens
32.6M
Requency
3230
Avg. # of Tokens per Video
619.1
Transcript Density
61.84%
Audio Statistics
300
Audio Availability
100%
230
Table II: Detailed Statistics of the MoSu Dataset. Transcript density means the average ratio of video duration with valid text.
30
Video Duration (sec)
Figure I: Video Durations in MoSu Dataset.
B.2 GROUND-TRUTH PREPROCESSING
The 'Most Replayed' statistic from YouTube serves as the ground-truth for identifying key moments in video. It is normalized frequeney of replay counts collected from tens of thousands of views, provided as a sequence of 100 scores corresponding to uniformly segmented clips. Following the precedent set by Sul et al. (2023); Kim al. (2025b), exclusively use videos with over 50,000 views to ensure statistical reliability.
Handling YouTube Most Replayed Bias. A common artifact was observed in these statistics; anomalous spike in replay scores within the initial few seconds of a video, often occurring before the primary content begins. This phenomenon, referred to YouTuue Most Replayed Bias, is likely attributable to viewers' heightened attention or initial buffering adjustments at the start of a video. If left unaddressed, this bias could mislead model into associating high importance with temporal position rather than semantic content. To mitigate this effect and ensure the model learns content- based importance, the ground- -ouut scores corresponding the first five seconds of each video were zeroed out.
B.3 THEMATIC CATEGORY CLUSTERING
Tab. III shows the number of videos and the top five most frequent entity labels for each identified cluster. A high degree of semantic coherence is observed between the top entities and assigned clus- ter topics, which validates the effectiveness of the clustering approach. The dataset was partitioned into train, validation, and test sets by applying a stratified split with an 8:1:1 ratio to each cluster. This ensures that the proportional representation of each cluster is preserved across all data subsets.
B.4 DATA PREPROCESSING DETAILS
To ensure reproducibility, we provide the detailed settings for our feature extraction pipeline, which processes all videos into temporally aligned feature sequences I-second interval.
Visual Features. For our MoSu dataset, we sampled one frame per second and used the official pre- trained CLIP model to extract 768 768-dimensional feature vector for each frame. For the externa benchmarks, to ensure fair comparison, we directly utilized their officially provided visual features: Inception-v3 features for Mr. HiSum, and GoogLeNet features for SumMe and TVSum.
The text features were derived from two sources: YouTube transcripts for the MoSu dataset and generated captions for external benchmarks. For the MoSu dataset, time-stamped tran- scripts provided by YouTube were utilized. English transcripts were used directly, while English ones were translated via YouTube's automated translation API. Videos without any transcripts were excluded. For external benchmarks, caption was generated for each frame using a Vision-Language
Text Features.
ii

---

Published as a conference paper at ICLR 2026
Cluster
Videos
Top Most Frequent Entities
Video Games
11,578
Game, Video gume, Action-adventure game, Call of Duty, Strategy video
Musical Instruments
7,647
Guitar, String instrument, Musician, Acoustic guitar, Electric guitar
Fashion Materials
5,588
Cosmetics, Hair, Hairstyle, Fushion,
Automobiles
5,207
Vehicle, Car, Engine, Driving, Sports
Electronic Devices
4,291
Gadget, Mobile phone, Smariphone, Samsung Galaxy, IPhone
Food & Cooking
3,919
Food, Recipe, Cooking, Dessert, Dish (Food)
Sports
3,784
Game, Weight training. Basketbull, Gym, Basketbull moves
Animation Comics
3,752
Cartoon, Comedy (drama), Animation, Art, Comedian
Vchicles & Transportation
3,731
Vchicle, Motoreycle, Cycling, Bieycle, Motoscycling
Animals
3,181
Animal, Pet, Dog, Fishing, Fish
Table III: Cluster sizes and their top 5 most frequent entities.
model?. All acquired text was then subjected to a unified encoding process. First, non-verbal an- notations (e.g. [Music], [Applause]) were removed. Each sentence was subsequently encoded into a 768-dimensional feature vector using the [CLS] token embedding from pre-trained RoBERTa model This vector was then broadcasted across 1-second timestamps spanned by its dura- tion. Timestamps with no associated text were assigned a default vector, precomputed by passing <pad> token through the encoder.
Audio Features. To process the audio data, we extracted feature vector for each second of the audio stream. This is achieved using centered window approach designed capture sufficient semantic context and match the model's standard imput size. Specifically, for each second t, 10- second segment extracted from the symmetric interval [t- 5, ++5]. 5]. This bidirectional windowing ensures that the representation for time is informed by both preceding and succeeding events. These segments are subsequently encoded into a fixed-size feature vector by utilizing the output embedding of the [CLS] token from pre-traine pre-trained Audio Spectrogram Transformer model'. This process results in a sequence of contextualized representations for the entire audio stream.
B.5 DATA SPLIT IN SUMME AND TVSUM
For completeness and reproducibility, we provide detailed descriptions of the two evaluation proto- cols used in the main paper. Both protocols operate under 5-fold cross-validation framework but differ significantly in how the data partitioned for model development and evaluation.
Traditional Train/Validatinn Validation (TV) Split. Following standard practice in earlier video summariza- tion research, the dataset is randomly partitioned into five equal folds. For each iteration, four folds are used for training and the remaining one fold is used for evaluation, with the final results aver- aged across all five folds. However, this protocol lacks separate, dedicated test set. Consequently, hyperparameter tuning and early stopping decisions are effectively made based on the performance of the held-out evaluation fold, making the model highly susceptible to overfitting.
(TVT) Split. To address the overfitting concerns inherent in the traditional TV setup, we also evaluate our method using stricter TVT split within the SFCV framework, as adopted by Kim et al. (2025a). In this setting, the five folds explicitly divided into training, validation, and test sets (typically utilizing three folds for training, one for validation, and one for testing, approximating 6:2:2 ratio). The validation fold is strictly used for hyperparameter tuning and early stopping, ensuring that the test fold remains completely unseen during the model devel- opment phase. This protocol provides a more rigorous and reliable evaluation of the model's true generalization capability.
TraainValidation//
#Qwen/Qwen2-VL-: -L-7T--nsttrttt
iii

---

Published as a conference paper at ICLR 2026
B.6 LONG VIDEO TEST SET
In order to evaluate the video summarization models in a more realistic and challenging setting, we collect dedicated evaluation with significantly longer videos than any other public datasets for video summarization. Traditional video summarization benchmarks like SumMe (Gygli et al., 2014) and TVSum (Song et al., 2015) include videos with duration of 2-5 minutes on average. Recent datasets like LIVS-P (Argaw et al., 2024) and MMSum (Qiu et al., 2024) have encreased it to 13- 13-14 minutes on average, as shown in Tab. However, these datasets are still far from long videos in the real-world, g.g movies or documentaries that are hours-long.
We collected 50 diverse, long-fon videos curated challenge models with domains not present in the MoSu training set. While MoSu covers wide range of categories (e.g., Video Games, Fashion, Animals), this dataset features complex narrative structures and specialized content. For example, it includes domains such full-length films and documentaries, professional technical tutorials, full-length sports matches and multilingual talk show. Ground truth for these videos have been obtained using the same 'Most Replayed' scores, same as MoSu. Key statistics of this new test set are summarized in Tab. IV. Notably, the average video duration is 70.4 minutes, 10-20 times longer than the training videos from MoSu.
Statistic
Duration (sec)
Avgerage Duration
4224.0
Stdandard Deviation
1263.6
Min Duration
2413.0
Max Duration
7207.0
Table IV: Statistics for the long video dataset.
C ARCHITECTURE ABLATION STUDY
C.I MODEL LAYER
Number of Layers
pt
mAP50
mAP15
Params |
Overall (L)
MST
CMF (Q)
8
8
8
0.326
0.442
73.41
42.2
17.15M
4
0.346
0.465
74.39
43.34
4.53M
2
2
2
0.351
0.472
74.72
44.42
1.37M
2
2
1
0.330
0.451
73.67
42.96
1.11M
2
1
2
0.339
0.461
74.41
43.33
1.11M
2
1
0.340
0.461
74.38
43.40
0.85M
0.314
0.430
73.4
41.61
0.58M
Table V: Ablation study on the number of model layers. Overall denotes the total number of layers, while MST and CMF refer to the Multi-Scale Temporal block and Cross-Modal Fusion block, respectively.
To investigate the influence of network depth, we vary both the total number of layers and the allocation of layers between the Multi-scale Temporal (MST) block and the Cross-modal Fusion (CMF) block, as reported in Tab. V. Increasing the number of CMF layers provides no additional benefit and results in a slight decline in overall performance across all evaluation metrics, whereas decreasing the MST depth consistently degrades performance. These results indicate that temporal modeling requires moderately deep MST for effective representation, while a single fusion layer sufficient for cross-modal integration.
C.2 WINDOW SIZE
To analyze the impact the temporal receptive field on summarization performance, we experiment with various window size configurations for the window-hased self-attentio mechanism, as shown
iv

---

Published as a conference paper at ICLR 2026
Window Size (10)
T
p
mAP50
mAP15 1
Baselines Standard SA
0.317
0,431
73.46
41.62
Fixed Window 5, 2,5,5,
0.297
0.409
71.61
39.73
15, 15, 15, 15
0,318
0,430
72.64
41.15
45, 45, 45, 45
0.322
0,435
72.95
42.10
Global-to-Local N727,9.0 9,3
0.340
0,462
74.22
43.09
N, 45, 15,5
0.345
0,461
74.36
43,61
N, 63, 21,7
0.347
0,468
74.52
43.54
Local-to-Global
3, 29,27,N
0.346
0,466
74.58
43,91
5, 15, 45, N
0,351
0.472
74.72
44.42
2, 63,
0.348
0.468
74.67
43.92
Table VI: Ablation study on different window size contigurations. N denotes the full sequence length (global attention).
in Tab. VI. Specifically, we compare fixed window sizes and hierarchical configurations (Global-to-
Local and Socallto-Globare The results demonstrate that the Local-to-Globa strategy. specifically sequence of [5, 15, 45, N). yields the best performance across metrics. This configuration progressively expands the receptive field from the lower to higher layers, allowing the model to capture fine-grained local temporal dynamics in the early stages and integrating global context in the later stages. Conversely, fixed small windows fail to capture long-range dependencies, while starting with global attention (Global-to-Local) appears less effective at preserving essential local details. These findings confirm that hierarchically expanding window structure is optimal for modeling the multi-scale nature video content.
C.3 LEARNABLE WINDOW
Window Size (w)
T
P
mAP50 f
mAP15
Standard SA
0.317
0.431
73.46
41.62
Learnable
0.318 (+0.001)
0.429 (-0.002)
73.65 (+0.19)
41.44 (-0.18)
Glocal-to-Local
0.345
0.461
74.36
43.61
Learnable
0.344 (-0.001)
0.465 (40.004)
74.49 (+0.13)
43.77 (+0.16)
Local-to-Global
0.351
0.472
74.72
44.42
+ Learnable
0.353 (+0.002)
0.469 (-0.003)
74.83 (+0.11)
44.32 (-0.10)
Table VII: Effect of Learnable Gaussian modulation on different attention strategies.
Ideally, it would be the best to adaptively choose the set of window sizes from the data. However, learning these from scratch was extremely unstable from our initial experiments, probably due to the excessive degree of freedom. Instead, adopt a learnable window configuration based on Gaussian-moduule modulated self-attention mechanism, conceptually similar to (Guo et al., 2019), initializ- ing the model with a pre- pre-trained one with fixed window sizes. The core idea is set the parameters of the Gaussian distribution (including the standard deviation a)) as learnable values, allowing the model to adaptively adjust the width (or scale) of the receptive field at each layer. Specifically, by modulating the attention scores with the Gaussian mask Gi,j. the parameter directly controls the decay rate of attention weights with respect to the temporal distance - j]. Mathematically, smaller causes the exponential term to vanish rapidly for distant frames, effectively enforcing locality, whereas a larger a yields flatter distribution, allowing the model to capture global context.
As shown in Tab. VII, this adaptive mechanism does not yield substantial difference. Its effect negligible across all configurations. including Standard SA, Global-to-Local. and our Local- to- to-Global strategy, with some metrics even degrading. This suggests that manually designed

---

Published as a conference paper at ICLR 2026
Local-to-Glober structure is already highly effective at capturing the necessary multi-scale temporal dependencies. The additional complexity of learning the receptive field variance does not offer a tangible benefit over carefully tuned fixed windows.
C.4
EMBEDDING DIMENSION
D
P
mAP50 t
mAP15 f
Params 4
GFLOPs L
768
0.234
0.324
68.25
34.50
39.71M
31.77G
512
0.313
0.424
72.40
40.84
18.07M
14.25G
384
0.344
0.464
74.44
43.18
10.42M
8.10G
256
0.347
0.467
74.54
44.1
4.85M
3,67G
0.350
0.471
74.59
43.84
2.85M
2.10G
128
0.351
0.472
74.72
44.42
1.37M
0.97G
96
0.348
0.467
74.56
43.8
0.84M
0.57G
Table VIII: Ablation study different embedding dimensions. D denotes the embedding di- mension.
We further investigate the effect of the embedding dimension D on both model efficiency and sum- marization accuracy. As presented in Tab. VIII, we observe that the model performs best around D 128 256. The performance slightly degrades up to D 384, and an larger D sub- stantially overfits the model. Even at an extremely low dimension of = 96, the model maintains competitive performance, demonstrating the robustness and compactness of our feature represen- tation. Based on these results, we select D 128 as the optimal setting, striking good balance between lightweight architecture and high-quality summarization.
C.5 SHARED PARAMETERS
Setting
T
P
mAP50 1
mAP15
Params |
w/o Shared Parameters
0.343
0.459
74.24
43.24
2.95M
w/ Shared Parameters
0.351
0,472
74.72
44.42
1.37M
Table IX: Effect of shared parameters on performance.
To evaluate the impact parameter sharing in the Multi-scale Temporal block, we perform an ablation study (Tab. IX). The w/o Shared Parameters configuration assigns independent temporal block to each modality, whereas the w/ Shared Parameters configuration employs single block shared across visual, text, and audio streams.
Sharing parameters reduces the number of learnable weights from 2.95 to 1.37 M, nearly three- fold decrease, yet yields consistently higher scores all evaluation metrics. This result indicates that parameter sharing enables the temporal block to capture multi-scale patterns common to differ- modalities. Training this shared block on modalities simultaneously also exposes roughly four times many training sequences as independent block for each modality. The combination of greater data exposure and the inherent ability of the multi-scale temporal structure to model both long- and short-range dynamics provides clear explanation for the superior performance of the shared-re shared-parameter parameter design, despite its substantially smaller parameter count.
C.6 FUSION TOKENS
We explore three ways aggregate trimodal features (e", e et) into the fusion token ef: visual-centric baseline, taking only the visual feature (ef = e No Fusion), 2) treating the fusion token itself as learnable parameter (ef e I: Learnable) and 3) a simple average over the visual, text, and audio features (c/ (c) cf + c Average).
Surprisingly, Tab. 4(d) concludes that the simple Average initialization yields the highest perfor- mance. The No Fusion baseline, relying mainly visual information, may be vulnerable when
vi

---

Published as a conference paper at ICLR 2026
non-visual cues such speech ambient sound dominate or when certain modalities missing. The Average, assigning equal contribution to all modalities, simple but effective to balance all modalities to serve as query.
Method
P
mAP50 f
mAP15
No Fusion
0.337
0.451
73,96
43,48
Learnable
0.341
0.462
74.41
43.71
Average 0.351
0.472
74.72
44.42
Table X: Ablation study on fusion token aggregation.
D COMPUTATIONAL COST ANALYSIS
Method
T
1
Params
GFLOPs !
Inference Time !
VASNet
0.151
0,219
8.31M
1.99G
7.36ms
PGL-SUM
0.151
0.218
5.31M
1.21G
13.55ms
CSTA
0.291
0.398
10.56M
11.37G
9,48ms
A2Summ
0.181
0.257
2.48M
1.35G
44.2 44.29ms
SSPVS
0.190
0.271
112.81M
43.64G
14.27ms
Joint-VA
0.190
0.272
4.21M
1.63G
2.58ms
UMT
0.239
0.334
4,66M
1.39G
4,94ms
CFSum
0.277
0.374
19.83M
8.52G
3.87ms
TripleSumm 0.351
0.472
1.37M
0.97G
2.8lms
Table XI: Comparison of computational cost and performance. We report rank correlation scores (T, p). model size (Params), inference cost (GFLOPs) and inference time evaluate the trade-off' between efficiency and accuracy.
We compare TripleSumm with other methods in terms of computational complexity and inference speed to validate its suitability for real-world applications. Tab. XI reports the summarization accu- racy (r and p), number of learnable parameters (Params), GFLOPs and inference time of competing models. TripleSumm sets eters and 0.97 GFLOPs, which significantly lower than other baselines. Despite its lightweight, model outperforms all baselines by large margin in correlation metrics Furthermore, TripleSumm achieves remarkably fast inference time of 2.81 ms, making it compara- ble the fastest method Joint-VA) while delivering superior summarization quality. This analysis confirms that TripleSumm successfully breaks the trade-off between performance and efficiency, offering highly scalable solution for video summarization.
stae--ofthhe-eee in efficiency, requiring only 1.37M learnable param-
0.361, P 0.484).
E ADDITIONAL ANALYSIS ON ADAPTIVE LEARNING DYNAMICS
E ABLATION ON THE LEARNED MODALITY SALIENCY
To verify if our model effectively assigns attention weights the truly important modalities each frame, we conduct another ablation study on the MoSu test set. At inference, we utilize the attention weights assigned by the Fusion Token the three modalities (Visual, Text, and Audio) at every frame. We then rank them in the order of weights: highest (Rank 1), second-highest (Rank 2), and third-highest attention weights (Rank 3), respectively, each time step. Using these ranks, we selectively take only subset of modalities (e.g.. "Rank only" "Rank + 2").
As shown in Tab. XII, the full TripleSumm model, which always utilizes all modalities, achieves the best performance across all metrics, confirming that the weighted contribution of all three modalities the optimal strategy. As expected, the performance gradually degrades as the model is forced to ignore more highly-weighted modalities, showing worst performance when restricted to the least- important modality ("Rank only"). This demonstrates that the attention mechanism successfully suppresses noise and prioritizes informative cues. The substantial performance improvement from
vii

---

Published as a conference paper at ICLR 2026
"Rank only" to "Rank + 2" validates that the second-moss important modality provides crucial complementary information. In conclusion, this ablation study confirms that the Fusion Token's attention mechanism successfully learns the frame-dependent saliency of each modality, leading to the superior performance of TripleSumm.
Method
7
mAP50 1
mAP15
Rank 3 only
0.058
0.087
59.44
25,20
Rank 2 only
0.136
0.194
63.34
29.99
Rank 2
0.158
0.220
64.06
30.81
Rank 1 only
0.196
0.273
66.09
33.28
Rank 1 +
0.273
0.373
70.28
38.69
TripleSumm (Ours)
0.351
0.472
74.72
44.42
Table XII: Ablation dropping modality features based estimated saliency.
E.2 ANALYSIS OF ATTENTION WEIGHT DISTRIBUTION
To investigate the distribution of the attention weights learned by the Fusion Token, we conduct another ablation study similar the above (App. E.I), but using specific threshold 0 instead of relative importance. This experiment is designed to assess whether the learned weights are heavily concentrated few dominant modalities or if they are distributed more broadly across the input features.
As depicted in Fig. II, the analysis utilized two filtering strategies. "Over 6" retains only the weights 2 0 and sets others to zero), while "Under ( retains only the weights s 0 and sets others to zero). The performance metrics, Kendall's and Spearman's p. exhibit remarkably smooth and linear inverse relationship as the threshold 0 is varied from 0.0 to 1.0. This linearity suggests that the attention weights are not concentrated on a small, fixed subset of features. the weights are sparse, we would expect to see sharp, non-linear changes in performance. Instead, the gradual degradation of "Over 8" performance and the corresponding gradual improvement of "Under 6" performance (as 0 increases and includes more subtle weights) indicate that the Fusion Token distributes its attention weights broadly and subtly. This broad distribution confirms that the model captures complementary information from wide range of features and modalities, validating that TripleSumm effectively utilizes distributed saliency rather than relying on a single dominant cue.
0.50
0.35
0.45
0.40
Spearman's
0.35
Kendall's
Over Under 6
0.30
Over Under e
0.20
0.25
0.15
0.20
0.10
0.15
0.10
0.0
0.2
0.3
0.5
0.5
0.7
0.8
0.9
1.0
0.0
0.1
0.2
0.3
0.5
06
0.7
0.8
0.0
1.0
Threshold 0
Threshold
(a) Kendall's with threshold
(b) Spearman's p with threshold
Figure II: Performance evaluation by thresholding modality attention weights. The figure illus- trates the model performance, measured by (a) Kendall's and (b) Spearman's p. when a threshold 0 is applied to the Fusion Token's 's learned modality attention weights. The Over 0 line shows the performance when only weights 2 are retained, while the Under line shows performance when only weights < 0 are retained.
viii

---

Published as a conference paper at ICLR 2026
F
FULL RESULTS OF EXTERNAL DATASET
F.1
TVSUM AND SUMME
SumMe
TVSum
Method
T
p
Random
0,000
0.000
0.000
0.000
Human
0.205
0.213
0.177
0.204
SUM-GAN(Malasse et 2017)
0,049
0.066
0.024
0,031
VASNetraaid 2018)
0.160
0.170
0.160
0.170
DSNet-AFiZui et al,, 2021)
0.037
0.046
0.113
0.138
DSNet-ABeZhu et al, 2021)
0.051
0.059
0.108
0.129
AC-SUM-GAAWWpporroldllt et al., 202
0.102
0.088
0.031
0.041
DMASum(Wang al., 2020)
0,063
0.089
0.203
0.267
CLIP- CLIP-Incarasiimaan 2021)
0.108
0.147
VGLLSUUMApsstttiiiiicc 2021b)
0.192
0.213
0.157
0.206
(L al., 2022)
0.241
0.268
0.179
0.235
IPTNet(Giag & Mu, 2022)
0.101
0.119
0.134
0.163
A2SummcHo
0.108
0.129
0.137
0.165
AAAMTTerenc al., 2023)
0.169
0.223
MAAM(Terbonchee et al., 2023)
0.179
0.236
VSS-Nct/hang et al,, 2024)
0.190
0.249
Joint- (aia-VA(Badamddorr al., 2021)
0.230
0.256
0.166
0.220
SSPVS 2023)
0.192
0.257
0.181
0.238
CSTA al., 2024)
0,246
0,274
0.194
0.255
LLMV et al., 2025)
0.253
0,282
0.211
0.275
SummDiffiKin et al., 2025a)
0.256
0.285
0.195
0.255
OurS
0.265
0.296
0.211
0.275
OurS
0.282
0.314
0.217
0.282
Table XIII: Performance comparison on SumMe and TVSum datasets. Results are reported under the 5-fold cross-validation protocol using the traditional train/validation (TV) setup.
Tab. XIII presents the performance comparison on the SumMe and TVSum datasets using the tradi- tional train/validation (TV) split under the 5-fold cross-validation (SFCV) framework. These results are included primarily to ensure comparability with prior work, as this evaluation setting has been predominantly used in previous literature.
While the traditional TV split offers comparability with existing baselines and demonstrates the ro- bustness our approach across different random partitions, the TVT split provides more reliable measure of true generalization performance by eliminating potential test set contamination during hyperparameter optimization. Across both evaluation settings, TripleSumm consistently surpasses previous methods, with particularly strong improvements when pre-trained on our MoSu dataset. This consistent performance gain across different evaluation protocols demonstrates the effective- ness of our multimodal fusion design and validates the benefits of large-scale pre-training data.
ix

---

Published as a conference paper at ICLR 2026
F.2 MR. HISUM DATASET
Method
T
PT
mAP50
mAP15
SUM-GAN (Mahasseni al,, 2017)
0.067
0.095
56.62
23.56
VASNet (Fajil 2018)
0.069
0.102
58.69
25.28
AC-SUM-GAN (Apostolidis et 2021a)
0.012
0.018
55.35
21.88
SL-module (Xu et al., 2021)
0.060
0.088
58.63
24.95
PGL-SUM (Apostolidis et al., 2021b)
0.097
0.141
61.60
27.45
Joint- VA (Badamdorj
2021)
0.161
0.231
65.88
35.23
iPTNet (Jiang Mu, 2022)
0.020
0.029
55.53
22.74
UMT (Liu et al., 2022)
0.178
0.253
66.81
35.65
A2Summ (Ile et al., 2023)
0.121
0.172
63.20
32.34
SSPVS ((.i 2023)
0,078
0.113
59.48
26.35
CSTA
2024)
0.128
0.185
63.38
30.42
Ours (Visual)
0.187
0.258
67.16
35.57
Ours (Tull)
0.258
0.352
70.72
40.88
Table XIV: Comparison on Mr. HiSum dataset.
To provide a complete benchmark, we report detailed results on the Mr. HiSum dataset Tab, XIV. TripleSumm achieves the highest scores across all correlation metrics, surpassing previous state-of- the-art approaches by a clear margin.
G DATASET ABLATION STUDY
G.1 DATA SCALING
0.5
0.4
0.3
Score
0.2
0.1
Kendail's
Spearman's
0.0
1000
5000
10000
15000
20000
25000
30000
35000
40000
Training Data Size
Figure III: Evaluating the effect of MoSu dataset seale on TripleSumm performance.
To examine the impact of training data volume performance, TripleSumm trained MoSu subsets of progressively increasing size. As illustrated Fig. III, both Kendall's 7 and Spearman's 's p exhibit clear positive correlation with the number of training samples. Notably, the performance curve does not reach saturation, even when the entire MoSu dataset is utilized, which suggests that further performance gains could be achieved with additional data. This finding highlights the data scarcity of conventional benchmarks like SumMe and TVSum and underscores the necessity of large-scale datasets such as MoSu for advancing the field of video summarization.
G.2 CLASS DISTRIBUTION
To assess label balance, the distribution of semantic clusters within the MoSu dataset was analyzed (App. B.3). The dataset is composed of ten distinct clusters of varying sizes and themes; however, the sample distribution is sufficiently balanced to prevent any single category from dominating the training process.

---

Published as a conference paper at ICLR 2026
Kendall's
Spearman's
3.
Kardafs pp
0.
-24
Video Games
Musical iastruments
Fashien & Matarials
Automobiles
Elactronte Devices
Food & Cooking
Sports
Animation Comics
Vahicles Transportatio Animats
Figure IV: Per-cluster performance the MoSu dataset. Boxplots show the distribution of Kendall's (blue) and Spearman's 's (orange) for each semantic cluster.
per-cluster performance evaluation, measured by correlation metrics, further substantiates this balance (Fig. IV). The median scores across clusters remain proximal to the overall mean and ex- hibit no clear correlation with sample size. For instance, clusters with fewer samples often achieve performance comparable
that of larger ones, while some well-represented clusters show wider variance or slightly lower median scores. Notably, the 'Sports' and 'Animation & Comics' clusters achieve higher medians with tighter distributions, suggesting more consistent model predictions for these categories. These results suggest that model accuracy is more influenced by the difficulty of the content in each cluster (e.g., visual complexity, audio variation) than by the amount of data.
xi

---

Published as a conference paper at ICLR 2026
H
MORE QUALITATIVE RESULTS
7
(No
"Can't
+ [Tire Screech)
[Tire Screech)
[Laughter]
[Engine Sound)
Attention Weight
Fusion Token
Visual
Text
Audio
Ours
CSTA
SSPVS
UMT
A vehicle overcoming an obstaele, where the audio of tire screech critical to understanding the action.
SVS
T
(No Text)
IT right..."
II!
(No Text)
Y
(No Text)
[Lyric]. Jy
[lyricc Au
[llytic]//
[Lyric] Alttic
Attantion Weight
Fusion Taken
Wma
Visual
Text
Audio
GT
Ours
CSTA
SSPVS
UMT
(b) A music video centered its audio track, where visual and text cues of secondary importance.
Figure V: More Qualitative Examples MoSu. The graph in the middle visualizes the fusion token's 's attention weights, illustrating how our model dynamically estimates the saliency of each modality and thus maintains strong summarization accuracy even when some modalities are missing.
xii


Reviews:
[{"review_id": "PEUuYbZ1CW", "reviewer": "Reviewer_bwyx", "summary": "This paper proposes TripleSumm, a trimodal video summarization framework that adaptively fuses visual, textual, and audio information to better capture the dynamics of real-world videos. The authors argue that existing methods are limited by unimodal or bimodal inputs and cannot adapt when the dominant modality changes over time. TripleSumm introduces two key components: a Multi-scale Temporal (MST) block that progressively refines attention from global to local temporal contexts, and a Cross-modal Fusion (CMF) block that computes per-frame adaptive weights to determine the relative importance of each modality. To support large-scale multimodal learning, the authors also construct MoSu (Most Replayed Multimodal Video Summarization), the first large trimodal dataset containing over 52,000 YouTube videos with synchronized visual, text, and audio signals and “most replayed” statistics as pseudo-ground-truth for frame importance. Extensive experiments across four benchmarks (MoSu, Mr. HiSum, SumMe, and TVSum) demonstrate that TripleSumm achieves state-of-the-art results with strong generalization and efficiency, supported by comprehensive ablations validating its dynamic fusion design and multi-scale temporal modeling.", "questions": "For MoSu, non-English transcripts are machine-translated; for other sets, text is generated via image captioning. How sensitive are results to transcript quality and captioning choice? Any robustness analysis?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clear, modular architecture that explicitly separates temporal refinement (MST) from cross-modal fusion (CMF).\n\n Strong, consistent gains across multiple datasets/metrics with a small model size.\n\n Careful ablation validating design choices (windowing schedule, dynamic fusion, modality combos).", "weaknesses": "Reliance on “Most Replayed” as ground truth, while pragmatic, can encode popularity/behavior biases; human alignment on MoSu isn’t quantified beyond transfers to SumMe/TVSum. It would be better to human-study agreement or correlation with editorial summaries.\n\n Selection uses standard KTS + knapsack; could the model be trained end-to-end with a differentiable or learning-to-select objective, and would that change results?\n\n While parameter-efficient, the inference cost with four MST blocks and CMF per frame isn’t fully benchmarked vs. strong baselines at equal latency. A runtime/memory comparison would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rCprmhAEu5", "reviewer": "Reviewer_hyzT", "summary": "This paper tackles multimodal video summarization by proposing TripleSumm, an architecture that performs frame-level adaptive fusion of three modalities. To address the widely acknowledged shortage of suitable evaluation resources in this area, the authors also introduce MoSu, a new large-scale benchmark providing synchronized tri-modal data. Experimental results show that TripleSumm yields clear state-of-the-art gains across four benchmarks, including MoSu, indicating the effectiveness of the fusion strategy and the value of the new dataset.", "questions": "Please refer to the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Well-motivated architecture design — TripleSumm performs adaptive, frame-level fusion of visual, audio, and text signals using specialized Modality and Temporal Blocks, enabling both fine-grained and long-range semantic capture.\n2. Benchmark contribution — MoSu fills a critical gap by offering the first large-scale trimodal dataset for video summarization, which meaningfully advances evaluation and reproducibility in this field.\n3. Strong empirical results — TripleSumm achieves state-of-the-art performance on four public benchmarks (including MoSu) with notable improvements and does so without excessive model size, demonstrating both effectiveness and efficiency.", "weaknesses": "1. Dataset Details and Quality Control\n The paper should include more comprehensive details about the proposed dataset, such as the average and variance of summary lengths, textual and audio statistics, and the distribution of video durations. Moreover, the authors need to justify why the generated summaries can be considered high-quality representations of the original videos under the current construction pipeline. It is strongly recommended that the summary quality be validated through human evaluation to ensure reliability.\n2. Model Design Rationale\n Some architectural design choices lack clear justification. For instance, the model employs larger temporal window sizes in the early layers and smaller ones in the later layers. This design appears counterintuitive, as shallow layers typically lack the capacity to model long-range temporal dependencies. The authors should clarify the motivation behind this choice or consider the more conventional approach of using smaller windows in early layers and progressively larger ones in deeper layers.\n3. Clarity of Figure 2\n Figure 2 is somewhat misleading. The main text states that the four modalities are processed independently by the MST, whereas the figure depicts them as being input jointly, suggesting possible cross-modal interaction within MST. The authors should revise the figure or clarify in the text how the modalities are actually handled to avoid confusion.\n4. Generalization Evaluation\n To better demonstrate the generalization capability of the model and the utility of the proposed dataset, it is recommended to train the model on the new dataset and evaluate it on other benchmark datasets (with or without fine-tuning). Such experiments would significantly enhance the credibility and impact of the proposed dataset within the research community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6jazzNrJ5P", "reviewer": "Reviewer_dGFC", "summary": "TripleSumm proposes an adaptive multimodal video summarization framework that integrates visual, audio, and textual cues through a Multi-Scale Temporal Block and a Cross-Modal Fusion Block. It also builds a new large-scale trimodal video summarization dataset with aligned video, transcript, and audio data. Experiments show that the method effectively models both temporal and multimodal patterns, achieving strong summarization performance.", "questions": "- Q1: Is the reported performance overly dependent on the newly introduced MoSu dataset, whose annotation process (based on replay statistics) may introduce bias and limit reproducibility, potentially misaligning with true human summarization preferences?\n\n- Q2: Is there any discussion of computational cost, such as runtime or FLOPs comparisons, to support the claimed efficiency of the method?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- S1: The separation between temporal modeling (MST) and cross-modal fusion (CMF) is elegant and easy to interpret.\n- S2: The framework effectively captures both intra-modal temporal dependencies and inter-modal relationships.\n- S3: The new dataset provides valuable resources for future research on multimodal summarization.", "weaknesses": "- W1: This paper shows relatively weak originality. The proposed model mainly focuses on optimizing multimodal (visual, text, audio) feature representations through standard attention operations, where self-attention is used to enhance intra-modal features and cross-attention is used for inter-modal fusion. The motivation and methodology closely resemble those of earlier works such as UMT [1] and CFSum [2], without demonstrating a clear conceptual or technical advancement beyond them.\n\n- W2: This paper’s methodological contributions lie almost entirely in representation learning, without offering task-specific innovations for video summarization itself. The proposed feature enhancement techniques could be applied broadly to other video-related tasks such as action recognition, rather than being tailored to the summarization objective. Moreover, this work is not the first to integrate three modalities (visual, textual, and audio), as prior studies have already explored trimodal fusion for similar purposes.\n\n- W3: The “multi-scale” temporal modeling relies on manually predefined window sizes (e.g., N, 45, 15, 5) rather than adaptive or learnable scales. This heuristic choice may not generalize well to videos of different lengths or dynamics, limiting flexibility and scalability.\n\n- W4: The paper does not include detailed ablation studies isolating the effects of each module (e.g., MST vs. CMF, scale depth, or fusion token design).\n\n[1] UMT: Unified Multi-modal Transformers for Joint Video Moment Retrieval and Highlight Detection\n\n[2] CFSum: A Transformer-Based Multi-Modal Video Summarization Framework With Coarse-Fine Fusion", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ODL5Lx5NrY", "reviewer": "Reviewer_rpGf", "summary": "TripleSumm introduces an adaptive tri-modal fusion architecture for video summarization that integrates visual, textual, and audio features at the frame level. The model interleaves two components: a Multi-Scale Temporal (MST) block, which captures temporal patterns with hierarchical windowed self-attention, and a Cross-Modal Fusion (CMF) block, which dynamically weighs modality importance using cross-attention. To support large-scale multimodal training, the authors also release MoSu, a new dataset of 52,678 videos featuring synchronized trimodal data and “Most Replayed” annotations from YouTube. TripleSumm achieves state-of-the-art results on four benchmarks (MoSu, Mr. HiSum, SumMe, and TVSum), outperforming prior unimodal and bimodal approaches by notable margins while remaining computationally efficient.", "questions": "- How well would TripleSumm adapt to domains beyond YouTube-style videos, such as surveillance, documentaries, or multilingual news—where audio-text alignment and modality saliency differ significantly?\n\n- How sensitive is performance to hyperparameters like window size, number of fusion layers, or modality embedding dimension? Would a learned or data-driven windowing schedule outperform the fixed [N,45,15,5] setup?\n\n- Does the adaptive attention occasionally overfit to dominant modalities (e.g., always preferring audio in music videos)? Could regularization or entropy constraints improve stability?\n\n- How do biases in the “Most Replayed” signal affect what the model learns to consider “important”? Has any qualitative audit been performed on viewer-driven biases (e.g., clickbait or thumbnail influence)?\n\n- What is the real-time performance profile (e.g., inference time per video minute) compared to unimodal transformers, and could TripleSumm be deployed on edge devices or streaming platforms?\n\n- Could the fusion and temporal mechanisms be generalized to other multimodal sequence tasks (e.g., event detection, captioning, or multimodal retrieval)? How would the model handle more than three modalities, such as motion or depth cues?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The CMF block effectively learns frame-wise modality weighting, overcoming the common bias toward visual features in prior multimodal architectures.\n\n\n- The MST block’s multi-scale attention enables both global context understanding and fine-grained temporal sensitivity.\n\n\n- Extensive quantitative and qualitative evaluations across four datasets, plus thorough ablations (on modalities, window sizes, and fusion strategies), strengthen empirical claims.", "weaknesses": "- The adaptive fusion’s learning dynamics are empirically motivated but not theoretically analyzed (e.g., modality weighting stability, attention interpretability).\n\n\n- Using “Most Replayed” statistics as pseudo-ground truth may encode platform or viewer biases, which are only partially mitigated.\n\n\n- Despite its scale, MoSu is heavily drawn from YouTube-8M and English-transcribed videos, potentially limiting generalization to non-English or low-resource domains.\n\n\n- The multi-scale temporal attention, though efficient per layer, could still pose latency issues for very long videos or real-time summarization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)