HiVid: LLM-Guided Video Saliency For Content-Aware VOD And Live Streaming
Authors: Jiahui Chen
Bo Peng
Lianchen Jia
Zeyu Zhang
Tianchi Huang
Lifeng Sun
Venue: ICLR 2026 Poster (2026)
Keywords: Video Streaming
Highlight Detection
Large Language Model
Time Series Forecasting
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Practical video streaming with LLM-based highlight detection

Abstract: Content-aware streaming requires dynamic, chunk-level importance weights to optimize subjective quality of experience (QoE). However, direct human annotation is prohibitively expensive while vision-saliency models generalize poorly. We introduce HiVid, the first framework to leverage Large Language Models (LLMs) as a scalable human proxy to generate high-fidelity weights for both Video-on-Demand (VOD) and live streaming. We address 3 non-trivial challenges: (1) To extend LLMs' limited modality and circumvent token limits, we propose a perception module to assess frames in a local context window, autoregressively building a coherent understanding of the video.
(2) For VOD with rating inconsistency across local windows, we propose a ranking module to perform global re-ranking with a novel LLM-guided merge-sort algorithm.
(3) For live streaming which requires low-latency, online inference without future knowledge, we propose a prediction module to predict future weights with a multi-modal time series model, which comprises a content-aware attention and adaptive horizon to accommodate asynchronous LLM inference. Extensive experiments show HiVid improves weight prediction accuracy by up to 11.5\% for VOD and 26\% for live streaming over SOTA baselines. Real-world user study validates HiVid boosts streaming QoE correlation by 14.7\%.

Published as a conference paper at ICLR 2026
HIVID: LLM-GUIDED VIDEO SALIENCY FOR CONTENT-AWEE AWARE VOD AND LIVE STREAMING
Jiahui Chen', Bo Peng', Lianchen Jia', Zeyu Zhang?, Tianchi Huang', Lifeng Sun 133+ Tsinghua University. 2 The Australian National University
3 Key Laboratory of Pervasive Computing, Ministry of Education, Corresponding Author chenjiah22 2mamai2mmmmmmmmmmmmmm is wew.ttsingghhh. www.ggghhaaeeuu....oooooo cn, sunlf@tsinghua edu.
ABSTRACT
Content-aware streaming requires dynamic, chunk-level importance weights optimize subjective quality of experience (QoE). However, direct human annota- tion is prohibitively expensive while vision-saliency models generalize poorly. We introduce HiVid, the first framework to leverage Large Language Models (LLMs) as scalable human proxy to generate high-fidelity weights for both Video-on- Demand (VOD) and live streaming. We address 3 non-trivial challenges: (I) To extend LLMs' limited modality and circumvent token limits, we propose a perception module to assess frames in local context window, autoregressively building a coherent understanding of the video. (2) For VOD with rating incon- sistency across local windows, propose ranking module to perform global re-ranking with a novel LLM- merge-sort algorithm. (3) For live stream- ing which requires low-latency, online inference without future knowledge, we propose a prediction module to predict future weights with a multi-modal time series model, which comprises content-aware attention and adaptive horizon accommodate asynchronous LLM inference. Extensive experiments show HiVid improves weight prediction accuracy by up to I 11.5% for VOD and 26% for live streaming over SOTA baselines. Real-world user study validates HiVid boosts streaming QoE correlation by 14.7%.
INTRODUCTION
Optional
Content-awure QoE
T My (i-(..L...)
Video
Summary
Highlight
Vidao Inderstanding ng
QuE -I
Ry -0.8
aK
Detection
Human Rating
Proposal
MPC
LLM-basod
Pensieve
Higher Wi Higher Quality
Video Chunks/Frames
Predict Chunk Weights
AHRs
Figure/ I: Overview of content-aware streaming. The estimated chunk weights w; are incorporated into QoE and optimized by ABRs. Higher weights would render better viewing experience.
Content-aware video streaming improves quality of experience (QoE) by allocating higher bitrates to more important video chunks guided by user-perceived priority weights Zhang et al.] (202T). As shown in Figure [ with available video chunks and optional text description, we can estimate the saliency score and incorporate into existing QoE model. Following past work on highlight detection Moon et al. (2023); Xiao al. (2024), here denote the saliency as the overall content importance score for each video chunk. We distinguish from visually salient regions within frame in classic video saliency prediction tasks.
The adaptive bitrate (ABR) algorithms Chen et al. (2024a) then optimize the bitrates with preference priority such that higher weights incur higher quality and less rebuffering, thus rendering better subjective experience. However, such numan-centric and content-dependent saliency task brings new challenges to existing paradigms.

---

Published as a conference paper at ICLR 2026
1.0
VideoLLa/R3 0.318
VASNee 084
4000
105.2
VILA:
module 0.125
4002
Time Cost
0.8
Flamingo:
DETR: 0,522
Sallenoy Store
MLP 0.013
GroundTRuth
0.6
Pdi SUME: 0.09
3002
Coat Per Video
0.4
2002
1000.0
40
0.2
0.0
2.4
140
Chank-loodl Rating Metrics (b) Overhead per video of 201s
(a) Saliency score of 8 baselines
Figure 2: Inaccurate saliency of previous work and significant overhead of human ratings.
Why LLM and Its Constraints. The most intuitive solution is computer vision (CV) based highlight detection like DETR Moon et al. (2023), which learns identify per-chunk tempo- ral saliency scores from training videos. However, these models are too small to capture the complex semantic content and generalize across diverse video categories. Alternatively, large video under- standing models like VideoLLaMA3 Zhang et al. (2025) excel in objective question answering and captioning tasks, but they often yield invalid and inaccurate responses when it comes to zero-shot subjective rating. We present a case study in Fig. (a) (refer to experimental setup). We can find that neither paradigm can fit the ground truth with high PLCC correlation (legend value). On the con- trary, (Zhang et al. (022)) conducts offline crowdsoureing ratings with human involvement, which is accurate but expensive and time consuming (78 minutes and 100$ per video). Therefore it's impractical for large-scale deployment, especially for live streaming, as shown in Fig. [b).
Challenge 1:
To enable both accuracy and efficiency, we can harness LLMs for zero-shot subjective reasoning as human proxy. However, video modality is unavailable for most SOTA LLMs like GPT-4o, which motivates us to assess only anchor frames from each chunk. Moreover, the limited input tokens (c.g., 128k) prohibit memorizing all historical context when dealing with long videos (LLM+Context in Fig. (b)). Therefore, we can break down the frames via local sliding window to enable fine- grained rating and global summarization with minimal overhead (LnM+ in Fig. (b).
Challenge 2: Rating Discrepancies in video-on-demand (VOD). Due to the lack of global context across frames in Chal- lenge 1, the LLM rating distribution may vary significantly across different local windows. We present an example in Fig. 3 with sliding window length m 10. During frames 79-80, the scoring scene receives the most attention (the highest ground truth) but the rating only reaches 65-70, while the less intense celebration during frames 108-109 is rated 75-85. This because the results from one window only manifest local importance without global reference. To enable con- sistency, an appropriate re-ranking across chunks can eliminate context bias and rating discrepancy.
70y (75 85) Inconsistent Rating Without Enough Context
SirRuaing: THI iecruaiia vipence
rakking appeling to vicwos Senierr
(Hlating cleeacic excler
(ittttte rnansctig fer vianeer
Figure 3: Inconsistent rating distribution.
Challenge 3: Uncertainty in Live Streaming. Different from VOD, live streaming requires real time decision without future chunks' knowledge. To this end, we can only predict the future weights based on historical ratings via forecasting models. However, the LLM inference latency is variable and dependent on the input tokens (see Table Therefore, a robust prediction must adjust the future horizon to cover the time interval gap for chunks that are not yet rated. Only then can ABRs optimize the future chunks' weighted QoE to decide the optimal bitrate. In addition, the inherent multi-modal time series also calls for new content-aware forecasting model to further boost accuracy.
In response, we propose HiVid, the first systematic framework that harnesses the power of LLMs as judge for content-awand aware streaming with tailored modules. To address challenge 1, we propose perception module to derive overall video description and chunk-level saliency scores. We leverage L.MMs to assess sampled anchor frames from each chunk via local sliding window. The response
2

---

Published as a conference paper at ICLR 2026
comprises frame group ratings with periodical video summary compact historical context for subsequent windows. In this way, HiVid is adaptive to arbitrary video length without token limits.
To address challenge 2, we propose ranking module top of the previous perception. With the global video summary and group ratings, we propose to re-rank the groups with a novel variant of merge sort algorithm, which encompasses LLM-guided comparison capable of sorting multiple frames. In this way. we obtain globally consistent saliency map without distribution discrepancy, while the overall summarization from perception module also guides the LLM reasoning.
To address challenge 3, we propose a prediction module in parallel with perception module. Upon each response of previous group rating, we leverage novel multi-modal time series forecasting model to predict the future chunk weights that are yet to arrive. We align frames and periodical text summary with CLIP Radford et al. (202I), and then we propose content-ave -wwrre attention to capture the impact of multi- video statistics on time series evolution. To further meet the strict latency, we dynamically adjust the prediction dimension asynchronously depending on LLM and forecasting latency. In this way, we achieve real-time streaming by pre-generating the future weights.
We conduct extensive experiments on 3 well-known highlight detection datasets. Regarding VOD, Vid surpasses 8 SOTA highlight detection and video understanding models by 11.5%, 6% and 14.7% in terms of correlation, mean average precision (mAP) and mean opinion score (MOS) aceu- racy. Regarding live streaming, HiVid also outperforms 9 SOTA forecasting by 26% while guaran- teeing real-time tame-tmme latency. We summarize our contributions as follows:
We present Vid, the first coherent LLM-guided pipeline for content-aware VOD and live stream- ing. We identify 3 key challenges: (1) Constrained LLM modality and context length; (2) Rating distribution discrepancy in VOD; (3) Unavailable future chunks and strict latency requirement in live streaming.
We address the issues with 3 modules: (1) Perception module that assesses sampled frames via context windows to iteratively generate video summary and saliency scores; (2) Ranking module that leverages LLM-guided merge sort algorithm re-rank all the frames with global video summary. (3) Prediction module that leverages a multi-modal time series model to predict future weights, compounded by novel content attention mechanism and adaptive forecasting dimension.
HiVid achieves the SOTA across 17 baselines in extensive experiments on public datasets. Real world user study in streaming QoE also validates the effectivvness.
2
RELATED WORK
2.1
CONTENT-AWARS STREAMING
Traditional video streaming leverages ABRs like heuristic MPC Yin et al. (2015) to decide bitrates of chunks to maximize objective QoE metrics Duanmu et al. (2019), i.e. higher visual quality, lower
rebuffering, etc. Content-aware streaming Zhang et al. (2021) improves upon additive chunk-level QoE Mao et al. (017) by incorporating the subjective content preferences as:
N
QoE = Σ
We *
(1)
where wi and qi denote the chunk weight and objective metrics above. To derive w. SENSEI Zhang et al. (202T) leverages crowdsourcing ratings on videos with different low-quality chunks and then infers the optimal weights. However, such human rating process amounts to expensive cost with significant delay, which is not scalable for VOD and live streaming.
As an alternative, highlight detection Xu al. 2021) like DETR Moon et al. (0023) tends to predict the chunk saliency score from video features by training neural networks like transformers. Video summarization Apostolidis et al. (202I) like VASNet Fajtl et al. (2019) achieves a similar goal by inferring chunk importance to the whole video. However, these small models exhibit poor semantic understanding and generalization ability, especially for unseen videos. Recently, large video models like VILA Lin et al. (2024) have enhanced the performance of various understanding tasks. How- ever, they suffer from hallucination and often yield invalid and inaccurate responses when dealing with subjective but quantitative rating task.
3

---

Published as a conference paper at ICLR 2026
Ranking: Weights via MergeSort (Section 3.3)
Periodical Video Summary
Updale
LLM
Frame Ranking
L.M
Vidco Summary
VOD
(A,R)
(AMB)
Ratings
Smoothing
JC Merge
1- I-m: [30,
wi= vi=(0.1.....
2m
Perception: Summarization (Section 3.2)
A
ABR
QoF=I witqi
For Live
Sliding Window
Sampled
Data Input
Images Iaput
Multi-Modal Forecasling
nil
Future Weights wi- .i.
LLM Outpur
g1 Chunk Quality
Adaptive Output
(A) Ordered Group
wi
Chunk Weight
Prediction: Live Streaming via Forecasting (Section 3.4)
Figure 4: Overview HiVid. The perception module generates a video summary with group ratings. The ranking module yields ranking list via variant merge sort algorithm for VOD streaming. The prediction module predicts future weights via adaptive forecasting for live streaming. The final weights W, incorporated into the QoE model.
2.2 LLMs FOR SUBJECTIVE REASONING
On the contrary, LLMs Achiam et al. (203)) have exhibited better semantic reasoning compared with video understanding models. LLMs have been adapted as agent Ge al. (2023) to perform various understanding Jin et al. (2024) and scheduling tasks Lai et al. (2023), while several studies Park al. (2023): (Hussain et (2024): Liu et (2025) also demonstrate the correlation between LLMs and human behavior regarding subjective perception assessment. However, it has not been explored how LLMs empower video-level highlights rating because video modality not directly supported, and the limited context prohibits an entire video input, presenting significant gap.
PROPOSED METHOD
3.1 OVERVIEW OF HIVID
Built upon previous insights, present novel framework HiVid which comprises 3 modules in Fig. 4 The perception module quickly iterates through the video and generates a summary with group ratings via sliding window. To adapt to VOD and eliminate rating discrepancies, the ranking module leverages an LLM-guided merge sort algorithm to rank all the frames, with guidance from previous summary. The final smoothing further refines the oscillating ratings. To adapt to live streaming with latency constraints, the prediction module utilizes adaptive multi-modal forecasting predict future weights in asynchronous manner. Together the 3 modules enable efficient and effective content-aware video streaming.
3.2 (BASIC) PERCEPTION MODULE
In response to Challenge 1, we propose to leverage LLM to understand and rate the video chunks via sliding window. To align with image modality LLMs, we directly sample the anchor frame as the first frame of each chunk to reduce redundancy and computation overhead, while other sampling like the last frame also suffice (see Appendix FF. Unless specified, we estimate the chunk weight with the sampled frame rating. For a video of D chunks and window of m length, we upload the m frames along with periodical summary to the LLM. The prompt instructs (see Appendix J} the LLM to rate the images based existing context and then update the summary:
Rh 1ik-1m++1: Skm LLM((PM))) ( (k-1)m+1: ((--1mm))) k: [1,
4

---

Published as a conference paper at ICLR 2026
where Ri, F. St denote rating and frame group from i to j and periodical summary before i frames, j = min(j, D). The initial summary Sy is the basic title and background of each vidco. In this way, we iteratively derive the overall summarization and all the frame ratings, with only [ LLM calls. m
3.3 (VOD) RANKING MODULE
In response to Challenge 2, we propose to re-rank the grouped chunks to eliminate context discrep- ancies. To this end, we leverage variant merge sort algorithm but with LLMs as the comparison function, which capable of sorting m frames in (m) time.
Merging Two Groups. Built upon perception module, to merge two sorted group frames, A SPT and = SFT2 we pick 2
m frames from each group to form a new m list for sorting. We then extract the first sorted frames and put the rest back to the original group, which can be formulated as:
km (SF
S ) LLM SSF Sp)
where SF. denotes sorted frames from to j. SD is the overall summary from Equ. 2 When either group is exhausted, we directly append the remaining sorted frames to the final list. By repeating Equ. until groups A and B are both exhausted, we derive the final sorted na n2 frames.
Sorting All Groups. For a video of D chunks and [D groups of no more than m frames, we first obtain the SF from sorting R in perception module. Then follow typical binary recursion algorithm to iteratively merge groups to obtain sorted D frames, which represent overall content preferences with global context from both frames and text summary. To evaluate the worst merging overhead, we derive the following formula:
k 2
D
т(k) = T(
)
T
) + 2h-
1,k
(4)
where TKk) is the number of LLM calls for sorting k groups and T(1) = 1. After obtaining the two sorted halves of D frames, we need merge them into the final list. While the worst scenario is when neither half is exhausted faster than the other during Equ. 3 Therefore each LLM sorting ma extracts 2 rendering
D = 2 [ = 2k calls, except that for the last time, we can directly sort the remaining S m frames without putting the second half back.
The T(k) complexity of Equ. is Okk k), rendering total complexity of ranking module Olk log k) Okk), where ((k) is the overhead of sliding window from perception module.
Gaussian Smoothing. With the final ranking SFP, we normalize the index to [0, 1] as chunk weights wi, To better fit the smooth ground truth distribution, e.g., in Fig. 3 we further apply Gaussian smoothing to alleviate the oscillation as w, GS(ss, a, w;), where kernel size s = D and is the standard deviation. We present the final algorithm in Appendix B
3.4 (LIVE) PREDICTION MODULE
In response to Challenge 3, propose to leverage time series forecasting to predict fu- ture weights in parallel with perception module.
Chunk
-Upload Chunks
EIM
Sliding Window
Retuurn Weights
We illustrate the scenario in Fig. 5 Upon each frame upload, the LLM response may arrive later after a token related interval. Therefore, to predict future N weights from previously rated m chunks, the output dimension should cover the time gap for previous n m chunks with- out ratings and N future chunks. Only then can ABRS optimize Equ. D to decide the bitrate of chunk n + 1. In this way, we eliminate the sig- nificant delay by asynchronous prediction and achieve real-time streaming.
C: Upload
Reseive Ee
@ Predict Weigite
Mill Forecesting
chinks
n
D+N-
Player
Chink
Receive Weights
Server
ABR
Figure 5: We predict future weights upon LLM response. The future horizon is latency-adaptive.
5

---

Published as a conference paper at ICLR 2026
Images [3 4 If W])
Series (fm))
Ti [o...3....
Wo
64
Query
CLIP Vision
delp
Avg
datp
Wi
Key
Multi-head Attention (32)
Laur
Time Series (Pouc)
64
P/Attetttn
Texts ( (auzt)
Car
"The vidco begins by
Text
detip
Wir
64
Value
m
Tensor of
Dim
Trainable
Figure 6: Multi-modal forecasting with content attention.
Forecasting Model. Different from traditional forecasting, we have as input not only the series data Tw E R but also historical frames 2; Ri x3x W and video summary It R To incorporate 3 modalities, we leverage a well-known CLIP Radford et al. (202T) to align the further propose
image and text features. To capture the complex interdependent relationships, novel content-aware attention mechanism. We project the time series features as Query (Q), and the concatenated image and text content features are projected as Key (K) and Value (V):
Qw ow Veat
Alln (1F,)) F(Featt, (x soʃ soflmax
Vd
where F(zw) denotes time series features and F = (at(CLIP,, CLIP((z:)). In this way, we motivate the model learn an attention pattern that specifically answers: Given the his- torical video content, what context is most relevani if the time series weights evolve as such? The detailed network architecture is in Fig. series with length Lin. while the general video summary has constant length Leeat- We leverage frozen CLIP to derive the latent features and then average the image to align the dimension. Then project the 3 modality features into Q. K. V in Equ. 5 The multi-head attention is followed by linear layer and finally predicts the future weights with length Lout-
Each previous frame coupled with rating as time
To enhance the prediction performance, typical because the weights distribution represents relative preference. Therefore we propose novel corre- lation loss guide the model follows:
squared error (MSE) loss does not suffice,
loss MSE(I, Fg)) + +1 E(x - Me) Mage)
(6)
where I and Zgt are the predicted and ground truth weights, and are the mean and standard deviation.
Algorithm 1: Forecasting with adaptive output
Input: constant parameters d, m, N, current chunk number i, global variable future weights W Output:
Future weights 1++
i%m = then submit m frames to LLM
Equ. 2
if LLM response is updated then
determine Loue(d, (d, m, N) by Equ.
submit time series Forecasting T.j. m, Tsout)
end
forecasting results w is updated then update w into W
if w in W then return w H+N else return [1] N
w
b weighted QoE in Equ. b original QoE without in Equ. 1
Adaptive Prediction. The core idea is to predict longer future weights that include the model inference time, as shown Fig. Therefore, the prediction dimension adaptive to the LLM and forecasting latency. Assume each chunk is of duration d, submit frames (as chunks) to LLM at t = tm in Equ. but receive at time f = t,, we have LLM interval At t, - t.p and forecasting latency ö, rendering elapsed chunks without rating
A++

---

Published as a conference paper at ICLR 2026
Table 1: Saliency accuracy of 2 method diagrams. Blue and Red denote the best and worst.
Large Model-based
Vision Saliency-hasee
Datasct
Metris
HiVid
LMMA
VILA
Flamingo
ML.P
PGI-SUM
VASNet
St-module
DETR
PLCCT
0.54
0.55
0.59
Youtube-8M 8M
SRCC
0.67
0.55
0.54
0.41
0.60
0.54
0.56
0.60
0.58
mAP501
0.86
0.77
0.73
0.56
0.81
0.80
0.80
0.81
0.81
mAPI51
0.53
0.45
0.44
0.33
0.49
0.46
0.46
0.49
0.45
PLCCT
0.50
0.41
0.37
0.32
0.44
0.39
0.45
0.43
0.42
TVSum
mAP501
SRCCT
0.67
0.52
0.52
0.41
0.37
0.53
0.47
0.30
0.62
0.43
0.59
0.40
0.66
0.45
0.57
0.43
0.63
0.44
mAPI51
PLCCT
0.47
0.35
0.35
0.31
0.37
0.33
0.37
0.39
0,38
SumMe
SRCC?
0.35
0.36
0.30
0.37
0.34
0.37
0.39
mAP501
0.62
0.49
0.55
0.39
0.52
0.53
0.57
0.53
0.61
mAP151
Moreover, since LLMs are called every m frames at é t= tomm the response also arrives periodically rather than per frame frequency. Hence need secure the future weights for those without LLM call or response, i.e. chunk number m. Finally, ABR algorithm typically requires N future chunk weights to optimize the QoE model, which incurs the final prediction dimension follows:
Lout 1 + d
+
Live Streaming Pipeline. The detailed process is in Algorithm Since At + 5 may vary dynami- cally, we first train several models with randomized different Lost During inference, for the initial At++ + m without LLM response, we pad the chunk weights with default I (line 8). For chunks d
current chunk i, we upload to the LLM if group of window length m is complete (line 1). This rating process is executed asynchronously from current video playback. Then we perform rating forecasting when the latest LLM response available, which equals time interval of Át. Given this LLM latency and estimated prediction time 8, we can derive the required adaptive output by Equ. 7 Then we pick trained model with minimum output dimension satisfying Lout to ensure the highest accuracy (line 3). This forecasting process is also executed locally and asynchronously (line 4). Upon new future weights w (from last forecasting submission), we cache the result a global weight pool W for future reference (line 6). Finally check the latest future N weights for content-aware QoE model (line 7) if available.
4 EXPERIMENT
Datasets and Metrics. We conduct experiments on Mr.Hisum from Youtube-8M Sul et al. 2023)) TVSum Song et al. (2015) and SumMe Gygli et al. (204)) which includes 1953, 50, 25 videos respectively. We sample 7:1.5:1.5 for training, validation, and testing, respectively. For saliency scores, leverage correlation-basel Pearson's linear correlation coefficient (PLCC) and Spear- man's 's rank correlation coefficient (SRCC), and we also include highlight detection metrics mAP50 and mAP15 for comprehensive comparison. For forecasting, we leverage typical mean absolute error (MAE), root mean square error (RMSE) and also PLCC and SRCC.
Parameter Setting. The default LLM used for HiVid is GPT-4o unless specified. Video chunks D depends on the test video length, window length = 10 unless specified, chunk duration d = 1s, Gaussian smoothing kernel size
1, Lin m, pretrained models with Toout (1. 2, 3) Lin. ABRs' decision horizon N 5. We also fix the ABR as RobustMPC Yin et al. (2015) since the QoE can be dynamically adjusted by w; during each optimization. The QoE model is the same Pensieve Mao et al. (2017) unless specified. The network trace dataset is FCC Commisssio (2016) and 3G/HSDPA Rüssr et al. (2013) for later user study.
a forecasting loss
8 Saliency Baselines. We select 2 highlight detection methods SL-module Xu al. (202 and DETR Moon et al. (2023), 2 video summarization based PGL-SUM AApsstoiidis et al. 202 and VASNet al. (2019) and MLP based network. We modify the loss function MSE E to learn the exact saliency score, and we also concatenate the same Gaussian smoothing after each model for fairness. We also include 3 SOTA video understanding models, VideolLLMMA Zhang et al. 2025

---

Published as a conference paper at ICLR 2026
VILA Lin al. (2024) and Flamingo AlayingoAAlaarac al. (2022). We leverage sliding window like Hi due to invalid response on entire video rating.
9 Time Series Forecasting Baselines. For uni-modal baselines, we compare HiVid-U (built only MLPs without image and text modalities) with 6 SOTA methods, iTransformer Liu et al.
20233
TimeMixer Wang et al. (2024), TimesNet Wu al. (2022). Crossformer Zhang & Yan 2023)
Nae et al. (2022) and FiLM Zhou et al. (2022).
We also include two efficient architectures
RNN /Sherstinsky (020)) and LSTM Zhao al. (2017). For multi-modal baselines, there are no
methods that incorporate image modality. Therefore we compare HiVid-M (with 3 modalities) with
LLM-based method, where we input all the series data, historical frames and text summary with a
prompt instruction for forecasting.
4.1
VOD: SALIENCY SCORE EVALUATION
Saliency Score Accuracy. To demonstrate our perception and ranking modules, we first eval- uate saliency score and present the results in Table D We can find that Hidid outperforms with 1T.5% and 6% improvement on average PLCC and mAP50 compared with the second SL-module respectively, thanks to our video summary and robust ranking. The latest model DETR Moon et al.] (202)) ranks only the mid- dle, which demonstrates that even the SOTA
Table 2: Overhead comparison of different m. (mini) denotes GPT-4o-mini.
Meeries
m
2(mini)
2
8
10
Token Secee,
15
278
292
314
321
335
API
Average latencyls Lasency Std/s]
3.14
6.4
8.14
9,83
Perception Calls,
100
100
25
Video
Kanking Calls
Cost5]
1358
8.12
3,68
2.41
1.71
242
Tocal Time Costhi
1.21
1.26
0.73
0.67
0.60
0.54
saliency method cannot fully capture video semantic content due to model scaling. In addition, video models like VILAA also exhibit lower accuracy due to inferior reasoning compared with large- scale L.M.s. For a more illustrative case study, we present a saliency distribution in Appendix
Overhead Analysis. We present time and mon- etary costs for different window lengths m in Table 2 Per API call, higher means more input tokens and hence higher score and higher latency. However, higher m also performs bet- ter which renders much fewer API calls as val- idated by Equ. and Equ. Therefore, the total cost per video of 201s is generally lower.
1.0
FLCC
0.8
9/ 945
CA CAS
0.8.
0.4
0.0 G7
Flymte55 10.5
5UM
5t modul
Sallency
User Study. To demonstrate real world stream- ing performance, we evaluate the QoE when Figure 7: MOS correlation of saliency models.
combining saliency score from 10 different baselines including ground truth. We leverage Ro- bustMPC ABR to optimize dynamic QoE model from Mao et al. (2017) with the saliency weights. We sample 10 eategory-varying test videos from Youtube-88 8MM encoded at {300, 750, 1200, 1850, 2850, 4300) kbps.. Note that we extract only 10s clips around the highest score for viewers. We run ABRS with 4 random network traces from FCC [ommiissoon (2016) and 3G/HSDPA Risser et al. 2003). For each viewer, we have 320 10-seconds clips.
We recruit 10 volunteers to evaluate the above clips and rate each from to 100. We randomly shuffle all the clips with the same video to en- sure fair rating without prejudice. Fi- nally, we compute the correlation be- tween weighted QoE model and aver- aged MOS. The results are in Fig. 7 We can find that Hid outperforms with 0.1-0.19 higher PLCC than SL- module and Flamingo, which vali- dates the high PLCC in Table In summary. HiVid achieves the SOTA in both weight correlation and user experience.
Paralal Cala
2.20
IMSSU
1.0
0.0
2.15
gad
No C
9.18-
004
2.45
5:0
(a) Utilityt and calls]
Windax langth (m)
(b) Overall PLCC
Miniiom to 1e Ied
Figure 8: Forecasting performance in live streaming.
8

---

Published as a conference paper at ICLR 2026
Table 3: Time series forecasting w/o LLM output. The results are averaged on datasets among Lin {8, 10) and for each Lin, Lout (,,2,3 * Lin For RNN and LSTM, Lout {1) Lin.
Models
Uni-Modal
Multi-Mddl
Metrics
ITransformer
TimeMixer
TimesNer
Crosstormer
ParchTST
FILM
RNN
1.TTM
HiVid-U
1.M
HiVid-M
MAR]
0.09
0.09
0.18
0.08
PLOCC
PLCCT
0.15
0.14
0.15
0.23
0.15
0.15
0.09
0.09
0.24
0.17
0.29
SRCC
0.11
0.13
0.13
0.22
0.12
0.11
0.08
0.10
0.22
0.18
0.27
MARI
0.08
0.88
0.09
0.09
0.88
0.09
0.19
0.18
0.08
0.13
0.08
RMSEL
0.14
0.14
0.22
SRCC
0.08
0.08
0.08
0.13
0.09
0.08
0.04
0.06
0.16
0.18
0.20
Time/ms)
22
26
48
16
24
22
8134
1350
Table 4: Saliency accuracy of HiVid w/ open-source multi-modal LLMs.
Metric
HiVid I.lama3..21 Vision Instract
Qwen3 VL. Instruct
InterVVL.3314
gemoma-3- 3.11bb...
PLCC
0.66
0.58
0.60
0.64
0.61
SRCCT
0.67
0.60
0.61
0.64
0.62
mAP507
0,86
0.80
0.81
0.84
0.82
mAP151
0.53
0.45
0.50
0.52
0.50
4.2 LIVE: FORECASTING EVALUATION
Forecasting Metrics. We present the time series performance in Table 3 HiVid-M outperforms all the SOTA baselines with the highest PL.CC=0.29 and also the lowest MAE=0.08, thanks to our novel content-aware attention. Even in uni-modality scenario, HiVid-U also achieves the best performance with PLCC=0.24 and MAE=0.08. This demonstrates that more complex models do not necessarily lead to better performance without tailored design. In addition, the improved accuracy when combined with correlation loss in Equ. has also validated our novel design.
LLM-based method performs worse because such task would require sufficient training data, rather than subjective reasoning. As for the time overhead, HiVid-U is the fastest due to simple MLP concatenation, HiVid-M may exhibit cost but it can be circumvented by the asynchronous pipeline in Equ. 7
Streaming Metrics. To demonstrate HiVid's application, we present the forecasting utility (ratio of chunks with available future weights) and overall correlation (forecasting with LLM rating as input) in Fig. We can find that for higher m, the utility decreases due longer initial LLM response interval in Equ. 7 while parallel calls also decrease which minimizes the risk of response blocking, i.c. early calls arriving later.
As for the overall PLCC/SRCC, they are bottlenecked by the forecasting accuracy, even with accu- rate historical weights. Therefore the performance better with higher m. but with the upper bound from forecasting PLCC=0.29 (for HiVid-M).
For end-to-ent latency in real ABR streaming, we present the time overhead in Appendix D Overall Hi imposes near-zero latency original ABR, thanks to our asynchronous LLM and forecasting inference. general, HiVid also outperforms all the baselines forecasting accuracy and latency.
4.3 ABLATION STUDY
To demonstrate the generalization of HiVid, we also apply the ranking module on open-source multi-modal
Table 5: Ablation of HiVid modules in VOD.
Performance
Per Vidao
LLMs (mllm) Dubey et al.] (2024); Chen et al. (2024b); Yang et al. (2025); Team et al. (2025) in Table It is as expected that GPT-4o still outperforms with up to 13.7% PLCC improvement. However, lo- cal milm can guarantee consistency
Model
PLCC1
SRCC*
mAP501
mAP15
Conr/]
Time Costitl
HiVid
0.660
0.674
0.860
0.536
1.35
0.54
HIVid w/ m-2 (mini)
0.645
0.611
0.848
0.511
0.44
1.21
Gemini 2-1lash
0.604
0.592
0.812
0.503
0.03
0.63
Grok2
0.506
Chadde--hhill
0.807
0.477
0.41
H&Vid wio Perception
0.632
0.653
0.852
0.520
1.22
0.49
HiVid wio Ranking
0.611
0.617
0.830
0.498
0.13
0.154
Hidid GS
0.619
0.621
0.835
0.514
1.35
0.54

---

Published as a conference paper at ICLR 2026
across runs and can be further fine-tuned for specific tasks, though it requires significant local com- putation.
To demonstrate the effectiveness of each mod- ule, we conduct ablation study in Table [ Note that all LLM backbones adopt window length m. = 10 for fair comparison. We can find that HiVid (with GPT-4o, m 10) outperforms with the best performance and lowest time over- head, while Gemini achieves the lowest cost, exhibiting different advantages.
Table 6: Overall PI.CC (w/ LLM rating input) for adaptive and constant Lout.
PLOC1
2(mini)
2
10
HiVid-M
0.12
0.15
0.18
0.20
Lout m
0.05
0.05
0.07
0.08
0.08
0.10
0.08
0.09
0.10
0.15
0.17
Lont 3m
0.09
0.09
0.12
0.15
0.17
0.18
By removing the perception module, HiVid cannot capture the global text summary which hinders some improvement. However, without the ranking module, there are significant rating discrepancies as shown in Fig. 3 which leads to much lower performance. Finally, the Gaus- sian smoothing also refines the coarse saliency score distribution to some extent.
Moc
3/8
5.5 8:79
(a) KSQI QoE
(b) Comyco QoE
To demonstrate our adaptive prediction, we leverage constant Lout and append the rest with 1 when necessary as required in Equ. [ The results in Table To show that neither baseline can reach our accuracy. Because dummy future weights directly degrade the correlation, while longer Lout also means inferior model performance, given that Lin. is constant for each m.
Figure 9: MOS correlationf of other QoE models.
To demonstrate the generalization of HiVid, we conduct additional user study with different QoE models in Fig. i.c. Kuammu et (019) and Comyco Huang et al. (2019). HiVid still outperforms with 0.1 and 0.09 improve- ment on PLCC, which stems from our content-
Table 7: LLM robustness for ambiguous videos.
Catepory
PLCC
SRCC
m.AP50
mAP15
Meua 1
Std..,
MMan
Sid
Meus
Sti
MMan 1
Sil
Polntics
0.73
0.08
0.74
0.05
0.92
0.00
0.63
0.44
People
0.69
0.03
0.69
0.04
0.88
0.00
0.00
0.2
bebucation
0.63
0.04
0.62
0.06
0.83
0.02
0.5
0.2
only saliency assessment. We also present more comparisons with different ABRs and parameters
in Appendix Fan
To validate the robustness against hallucination, we average the rating of LLMS (GPT-4o, Gemini and Claude) ambiguous vidcos. The results in Table 7 show that even sensitive categories yield stable accuracy with low deviation, thanks to our robust ranking. In general, all the modules in Hi contribute to the overall accuracy and efficiency.
To demonstrate the large-scale application of HiVid in live streaming, we present detailed cost and PLCC of an example video hours in Table Since HiVid applies sliding win- dow for live streaming without future chunks, the cost increases linearly with video length, i.c. m=2 (mini)
Table 8: Performance of longer live streaming.
Window Length
Matric
2 min
30 min
hour
6 hour
24 hour
m=10
Cost/S
PLOCC
0.08
1.20
0.24
4.81
14.43
57.74
Cost5
PLOC 1
0.13
0.16
0.15
LLM rating and forecasting once per m video
chunks. For longer window lengths m = 10, the performance is better but with higher cost (57.74S per 24 hours), yielding controllable tradeoff. However, we argue that we only need to process the source video in real-world one-to-many streaming, therefore the cost is actually negligible.
5 CONCLUSION
We introduced HiVid, the first systematic framework to leverage LLMs for content-aware streaming. We identify the critical trade-off between the inaccuracy of vision-based models and the prohibitive cost of human annotation. We addressed 3 core challenges: (1) perception module that updates summary and ratings via a sliding window to extend modality and context limitations; (2) an LLM- guided ranking module that ensures globally consistent saliency scores for VOD; (3) prediction module with content-awan aware model based on adaptive dimension to meet the strict real-time live streaming. Extensive experiments public datasets and user study demonstrate our effectiveness.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGEMENTS
This work was supported by Key Laboratory of Data Intelligence, Beijing.
ETHICS STATEMENT
This paper does not raise any ethical issues regarding human subject or dataset usage.
8 REPRODUCIBILITY STATEMENT
We have provided a code example of basic ranking module in the Supplementary Material in OpenReview. It includes how to combine merge sort with LLMs and how to compute the total API calls for overhead analysis. The attached json file an example of how store the response of each sliding window. In this way, we can cache the previous results and resume the ranking in case of API disconnection. While the periodical video summary can also be updated by querying the last json results and uploading LLM input.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmid,, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 03::::2233.0.774,2222
Dean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Evlampios Apostolidis, Georgios Balaouras, Vasileios global and local attention with positional encoding for video summarization. In 2021 IEEE inter- national symposium on multimedia (ISM), pp. 226-234. IEEE, 2021.
Mezaris, and Ioannis Patras. Combining
Tianyu Chen, Yiheng Lin, Nicolas Christianson, Zahaib Akhtar, Sharath Dharmaji, Mohammad Hajiesmaili, Adam Wierman, and Ramesh K Sitaraman. Soda: An adaptive bitrate controller for consistent high-quality video streaming. In Proceedings the ACM SIGCOMM 2024 Conference, 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185 24198, 2024b.
Federal Communications Commission. Raw data measuring broadband america. 2016. URL ftttps//wwwt wcc.gooo/.........oooooooo.o www.ooo/rrposssssoo.
Zhengfang Duanmu, Wentao Liu, Diqi Chen, Zhuoran Li, Zhou Wang, Yizhou Wang, and Wen Gao. A knowledge-driven -ffliy-of-experrinnc model for adaptive streaming videos. arXiv preprint /9/1:9944 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. 2407, 2024.
Jiri Fajtl, Hajar Sadeghi Sokeh, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino. Summarizing videos with attention. In Computer Vision-ACCV 2018 Workshops: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers 14, pp. 39-54. Springer, 2019.
11

---

Published as a conference paper at ICLR 2026
Hao Fei, Shengqiong Wu, Hanwang Zhang. Tat-Seng Chua, and Shuicheng Yan. Vitron: A uni- fied pixel-level vision Ilm for understanding, generating, segmenting, editing. arXiv preprint arXiv:2412.19806 2024.
Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. Openagi: When IIm meets domain experts. Advances in Neural Information Processing Systems, 36:5539-5568, 2023.
Google. Overview of gemini. 2025. URL htt.h:ttpp/aaa.. .oo..looollloooo ( www.eeewwwweeeeeeeeeeeeeeeeeee http:p////....oooo..eeeee//////a
vision Lhhgppyhhhhh =en
Michael Gygli, Helmut Grabner, Hayko Riemenschneiderr and Lue Van Gool. Creating summaries from user videos. In European conference computer vision, pp. 505-520. Springer, 2014.
Tianchi Huang, Chao Zhou, Rui-Xiao Zhang, Chenglei Wu, Xin Yao, and Lifeng Sun. Comyco: Quality-aware adaptive video streaming via imitation learning. In Proceedings of the 27th ACM international conference on multimedia, pp. 429 027,437,45 2019.
7ak Hussain, Marcel Binz, Rui Mata, and Dirk U Wulff. A tutorial open-source large language models for behavioral science. Behavior Research Methods, 56(8) 2024.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13700- 13710, 2024.
Terry K Koo and Mae Y Li. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of chiropractic medicine, 15(2):155-163, 2016.
Siqi Lai, Zhao Xu, Weijia Zhang, Hao Liu, and Hui Xiong. Limlight: Large language models as traffic signal control agents. arXiv preprint arXiv:2312 16044, 2023.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of IEEÉ/CVF conference on computer vision and pattern recognition, pp. 26689-26699, 2024.
Shicheng Liu, Siyuan Xu, Wenjie Qiu, Hangfan Zhang, and Minghui Zhu. Explainable reinforce- ment learning from human feedback to improve language model alignment. Advances in Neural Information Processing Systems, 38, 2025.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023.
Hongzi Mao, Ravi Netravali, and Mohammad Alizadeh. Neural adaptive video streaming with pen- sieve. In Proceedings the conference of the ACM special interest group on data communication, pp. 197-210, 2017.
Meta. Overview of llama. 2025. URL httpt:tt:/// //t::///wwww... www.. 11 aaaam........ ccmm
WonJun Moon, Sangeek Hyun. SangUk Park, Dongchan Park, and Jae-Pil Heo. Query-dependent video representation for moment retrieval and highlight detection. In Proceedings of the 2023.
IEEE/CVF conference computer vision and pattern recognition, pp.
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.147330 2022.
Openai. Overview chatgpt. 2025. URL httttttss///peeaat
Joon Sung Park, Joseph O' O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and
Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. Proceedings of the 36th annual acm symposium on user interface software and technology, pp. 1-22, 2023.
12

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PmLR, 2021.
Haakon Rüssr, Paul Vigmostad, Carsten Griwodz, and Päl Halvorsen. Commute path bandwidth traces from 3g networks: analysis and applications. In Proceedings of the 4th ACM Multimedia Systems Conference, pp. 114-118, 2013.
Alex Sherstinsky. Fundamentals recurrent neural network (mnn) and long short-term memory (Istm) network. Physica D: Nonlinear Phenomena, 404:132306, 2020.
Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing web videos using titles. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 5179-5187, 2015.
Jinhwan Sul, Jihoon Han, and Joonseok Lee. Mr. hisum: A large-scale dataset for video highlight detection and summarization. Advances in Neural Information Processing Systems, 36:40542- 40555, 2023.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19788 2025.
Shiyu Wang, Haixu Wu, Xiaoming Shi, Tengge Hu, Huakun Luo, Lintao Ma, James Y Zhang, and Jun Zhou. Timemixer: Decomposable multiscale mixing for time series forecasting, arXiv preprint arXiv:2405. 14616, 2024.
Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. Timesnet: Tem- poral 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02166 2022.
Yicheng Xiao, Zhuoyan Luo, Yong Liu, Yue Ma. Hengwei Bian, Yatai Ji, Yujiu Yang, and Xiu Li. Bridging the gap: A unified video comprehension framework for moment retrieval and highlight detection. In Proceedings of the IEEE/CVE conference on computer vision and pattern recogni- tion, pp. 18709-18719, 2024.
Minghao Xu, Hang Wang, Bingbing Ni, Riheng Zhu, Zhenbang Sun, and Changhu Wang. Cross- category video highlight detection via set-based learning. Proceedings of the IEEF/CVF Inter- national Conference on Computer Vision, pp. 7970-7979. 2021.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 050.002505.09388, 2025.
Xiaoqi Yin, Abhishek Jindal, Vyas Sekar, and Bruno Sinopoli. A control-theoretic approach for dynamic adaptive video streaming over http. In Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication, pp.
2015.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint a3Xiv:2501.13106, 2025.
Xu Zhang, Yiyang Ou, Siddhartha Sen, and Junchen Jiang. (SENSEI): Aligning video streaming quality with dynamic user sensitivity. In 18th USENIX Symposium on Newworked Systems Design and Implementation (NSDI 2/), pp. 303-303333 323-320,,20 2021.
Yunhao Zhang and Junchi Yan. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. The eleventh international conference on learning representations, 2023.
Zheng Zhao, Weihai Chen, Xingming Wu, Peter CY Chen, and Jingmeng Liu. Lstm network: deep learning approach for short-term traffic forecast. IET intelligent transport systems, 11(2): 68-75. 2017.
13

---

Published as a conference paper at ICLR 2026
Tian Zhou, Ziqing Ma. Qingsong Wen, Liang Sun, Tao Yao. Wotao Yin, Rong Jin, et al. Film: Frequency improved legendre memory model for long-term time series forecasting. Advances in neural information processing systems, 35:12677-12690, 2022.
14

---

Published as a conference paper at ICLR 2026
Video Summarization
**Frame 59 making equally engaging. **Rating: 70
93h*; ..
impending action clevate its exciting nature. **Riing:: 85**
begins showcasing Maradona's
legendary free kick,
BEmI
** Frames 1-7**; ...duc to their less dynamic nature and viewer engngement.
6-
116**; moment attractive compared to the match. "Raiing: 40tH
**Rating"::
Timos
MergeSort via LLM
Figure 10: Example of saliency score distribution.
A LLM USAGE STATEMENT
We clarify that this paper does LLMs for research ideation or paper writing.
B DETAILED ALGORIHTM
Algorithm 2: MergeSort
Input:
sorted frame groups SFbm (k-1)m-11
&e [1, [2]] video summary Sp
Output:
sorted frames S.PPP
mid=
2m Sorted=[];
SF. -MergeSor((.mid))
Binary recursion
B S -MergeSor(mid+1), 1, [2])
while A and B are not exhausted do
c²a Olhm =LLM(A, Sp):
kgp
> Equ.
Velen(A + B) S m then Sorted.aapeedd(("))
else Sorted Sprted.append((**))
Put Cfhm back to and B; end
10 Sorted.append (Remaninng SF); return Sorted
The detailed ranking process is in Algorithm 2
We leverage the typical binary recursion to iterate the frames. To merge sorted groups, we first select m/2 frames from each group, and then query the L.LM to update the total m frames. Following typical merge-sort, we only save the first half of the sorted frames enable subsequent LLM comparison.
C CASE STUDY OF SALIENCY SCORE
We present an example of the estimated score in a soccer game Fig. I0 We can find that all the baselines fail to fit the actual distribution. The plain curve indicates that saliency models only overfit the training videos without truly learning from semantic content, while video understanding models also overlook the most appealing parts. On the contrary, HiVid generally understands the video content with excellent video summary and frame analysis, e.g., low rating (30) during interview and high score (85) during shooting.
15

---

Published as a conference paper at ICLR 2026
Table 9: Time overhead for ABRs w/ different module delays when end-to-end latency T 5s.
Baselines
Raw ABR
HiVid
LLM
prediction
w/ both
ABR Time/s 4.
0.486
0.494
10.316 206.32%
1.836 36.72%
11.616 232.32%
Proportion of T
9.72%
9.88%
Table 11: Ablation study HiVid w/ different settings and parameters. Blue and Red denote the best and worst for each metric across the HiVid variants. The default standard deviation o 5.
Metric
Youtube-SM
TVSum
SumMe
Method
MAP15
HiVid
0.66
0.67
0.86
0.53
0.50
0.52
0.67
0.40
0.47
0..7
0.62
0.37
HiNid Hi Vid w/ middle frame
0.65
0.66
0.84
0.50
0.51
0.52
0.68
0.41
0.47
0.47
0.61
0.36
HiVid 2
0.68
0.69
0.89
0.55
0.52
0.55
0.70
0.41
0.49
0.49
0.64
0.38
HiVid w/ g
0.62
0.62
0.81
0.50
0.48
0.38
0.44
0.59
0.35
D END-TO-END LATENCY
To demonstrate our adaptive prediction for real time requirement, we present the time overhead in Table[9 We can find that HiVid only imposes additional 8ms which stems from asynchronous rating and prediction. Therefore HiVid achieves the same latency as the original ABR. However, without our adaptive prediction, we would have to await the significant delay to prepare all the historical input for forecasting, which leads to much higher latency (11.616s). For an example of latency T 5s, the decision time of HiVid+ABR does not impact overall experience, while the delay waiting would completely block the ABR decision.
E MORE ABLATION STUDY: DIFFERENT ABRS
Fig. 7 in our main paper has shown that HiVid (w/ MPC) surpasses various saliency model baselines with higher PLCC mean opinion score (MOS) correlation. We also ap- ply HiVid on RL-based Pensieve Mao et al. (2017). IL-based Comyco Huang al. (2019) and compare with traditional Qoł--rree buffer- based (BB) and rate-based (RB) ABRs. For RL and IL-based,, we incorporate the future N chunk weights as input and train the model to capture the content-aware preference in QoE. We also modify the reward into weighted QoE to guide the ABR exploration. The MOS correlation for different ABRs is in Table 10
Table 10: MOS correlation f w/ different ABR algorithms.
ABR
PLCCT
SRCC1
w/ QoE model
w/o QoE model
PiViieeeeniiee Pensieve HiVid+Comyco Comyco
0.78
0.76
0.76
0.85
0.85
0.81
0.81
BB
0.68
0.69
RB
0.73
0.72
Note that and BB do not incorporate QoE model and thus cannot combined with our QoE weights from HiVid, We can find that ABRs with HiVid enhancement outperform those without our method. HiVid with Comyco performs the best because the ABR itself achieves the SOTA traditional QoE optimization Huang et al. (2019). BB is the worst because it only applies buffer to monitor and predict future evolution based on a heuristic parameter.
F MORE ABLATION STUDY: DIFFERENT PARAMETERS
We also conduct experiments for different anchor frame choices and Gaussian smoothing param- eters in Table [1 The results demonstrate that frame sampling method only introduces limited performance deviation, because a video chunk of I second often comprises many semantic-similar frames, regardless of the specific position. While the Gaussian smoothing present significant impact. Overall, higher deviation ø means smoother curve and more stable ratings and hence better performance. However, significantly high also eliminate the original information of
16

---

Published as a conference paper at ICLR 2026
Table 12: Accuracy consistency across runs.
Metric
HiVid w/ GPT-4o
HiVid w/ Gemini-2-flash
HiVid w/ Grok2
HiVid w/ Claude-3-haiku
PLCC1
0.661:0.33
0.60:10.03
0.62100.2
0.53:10.05
SRCCT
0.6710.03
0.59
0.61
0.55+:0.04
mAPS01
0.86=0.02
0.81=0.05
0.82:10.02
0.81+0.03
mAP15t
0.53+0.01
0.50=10022
0.51
0.48=10.01
ranking module and thus render plain We choose for more stable performance without losing the greigg details our ranking.
G MORE ABLATION STUDY: RANKING CONSISTENCY
We have demonstrated that HiVid can be com- bined with local milm in Table 4 which ensures consistency across runs by setting do_somple = False and seeds.
Table 13: Accuracy consistency across models.
Metric
GPT-4o & Gemini2
Gemini2 Grok2
GPT-4o & Grok2
evaluate
PLCC SRCC
0.87.10.06 0.89++0.05
0.9110.04 0,90+0.04
0.8910.04 0,90+00.22
the robustness of HiVid with proprietary LLMs across runs Table [2] The low standard
variance proves the stable correlation and detection accuracy within each model. This means that LLMs generate most of the chunk-level scores via confident zero-shot subjective reasoning rather than random guess, e.g. the justified rating for each frame in Fig. I0
While Table 13 further evaluates the accuracy between models, where Gemini and Grok exhibit similar results with PLCC==0911. However, this does necessarily imply better performance on video saliency score, i.e. GPT-4o outperforms other LLMs in Table
H MORE ABLATION STUDY: DETAILED PERFORMANCE FOR LIVE STREAMING
We have demonstrated the clean forecasting performance in Table 3 We also present the prediction with actual LLM rating against the second SOTA Crossformer in Table [4] It shows that inaccurate LLM output (as forecast- ing input) degrades the overall accuracy com-- pared with clean GroundTruth input, especially for correlation PLCC. This is expected since we trained the models with clean data, because using eration.
Table 14: Forecasting ablation with LLM output.
Metric
Forecast-only
Forecast w/ LLM output
HiVid-M
Crossformer
Crosformer
MAE 1 RMSE
0.08
0.09
0.13
0.16
0.12
0.12
0.18
0.22
PLCC
0.20
0.15
SRCC
0.27
0.22
0.19
0.13
L.LM output requires significant offline rating gen-
We also present the forecasting accuracy with module ablation in Table 15 We find that the our content attention effectively learns the interdependent relationships from the various modalities (PLCC accuracy gains 0.03), while the only uni-modal model performs the worst without specific guidance from video content.
Table 15: Forecasting module ablation.
Metric
HiVid-M
HiVid-M w/o Attention
HiVid-U
MAE
0.08
0.08
0.08
RMSE
0.12
0.12
0.12
PLCC 1
0.29
0.26
0.24
SRCC
0.27
0.25
0.22
I MORE ABLATION STUDY: MOS CONSISTENCY AND STATISTICAL SIGNIFICANCE
17

---

Published as a conference paper at ICLR 2026
Table 17: Statistical Significance of MOS correlation from 30 participants.
Metric
GT
HiVid
DETR
VideoLLaMA3
Value
0.88
0.76
0.61
0.63
PLCC
value
10-38
10-26
10-14
10-18
CI
[0.83,0.92]
[0.67.0.82]
[0.48,0.73]
[0.51,0.71]
Value 0.91
0.77
0.65
0.65
SRCC
value
10-46
10-31
10-18
10 20
Table 16: Inter-rater consistency of 30 partici- pants.
To demonstrate the robustness of user study, recruit more volunteers and conduct inter-rater consistency check in Table [T6] Coefficient of Variation (CV) equals to (std/mean)) 100, and derive CV for each video and then compute the averaged CV. We also leverage Intraclass Correlation Coefficient (ICC) to analyze the robustness. ICC(2.k) evaluates the relative rating con- sistency while ICC(A,k) is more strict that includes the inter-rater variance. We can find that < 20% CV and 0.82 > 0.75 are considered "good" according Koo & Li (2016). While ICC(A,k) also "moderately" good, demonstrating valid user study.
Metric
Coefficient of Variation. ICC(2,))
ICC(A.K)
Result
19.76%
0.82
0.73
In addition, we also present the PLCC and significance for MOS correlation in Table T7 We can find that the results are similar to that in Fig. 7 and our values for both PLCC and SRCC across different baselines are near zero, demonstrating convincing user rating and our model QoE.
J PROMPT INSTRUCTIONS
We present the prompts used during perception module for basic periodical summary and group ratings.
Prompt:
have uploaded [len(image path)} frames, each representing a video chunk of second. You first extract the frame number attached below the image content. These frames exhibit continuous (len(image_paati) seconds video clip. The original video background for title and category are (info). Before this video clip, the periodical video summary is: {story Jast].
Your task is as follows:
1. Based the frames, periodical summary and background, summarize what story this video has conveyed so far and output your answer as "story_totall. (No more than 100 words)
2. Based the summary and frames, scale of integer (0,100), rate all the (len(image _ath)) } frames such that higher score exhibits higher interestingness score. Different frames can yield the same scores.
Your answer must be json format like this: json
("story partial"; "xxx"),
("story total"; "xxx"),
json
("frame": xxx, "rating": xxx),
("frame": xxx, "rating": xxx),
18

---

Published as a conference paper at ICLR 2026
C"frame": Xxx, "rating": xxx)
K DISCUSSIONS
We present some clarification for HiVid regarding design and evaluations.
1. The usage of LLMs simulates human ratings regarding QoE model.
As explained in Related Work Section 2.2, LLMs have been widely leveraged for subjective tasks. For example, Fei al. (2024) propose Vitron to perform image editing based prompt understanding and image analysis. has also been demonstrated in Hussain et al. (2024) how LLMs correlate human behavior regarding subjective perception, including the overall understanding of real world information like images and texts.
Therefore, our proposal leverage LLMs for subjective video rating also makes sense regarding human-centric QoE modeling
2. We leverage existing SOTA LLMs instead of training from scratch.
As explained in Section 2.2, proprietary SOTA LLMSs such as GPT-4o Openai 2025) and Gemini Google (005) have dominated the text generation domain. Training open-source models like Llama Meta 2025) also suffices but would require tremendous time and computation overhead. More importantly, we focus on real world application where ABRs are deployed at the client side. These devices possess limited CPU/GPU resources and cannot afford local LLMs inference.
Nonetheless, remains a promising direction to iin-tune new LLM specifically on various datasets like Youtube-8M Sul et al. (2023) in future work.
L LIMITATION AND FUTURE WORK
Despite the SOTA results of our HiVid on multiple datasets, HiVid comprises 2 limitations: (1) The overall inference time overhead for VOD is slightly longer than sliding window iteration, because the ranking module adopts binary recursion to enable sufficient comparisons among different frames to ensure accuracy. One future direction is deriving better LLM-guided sorting algorithm that achieves Onn) time complexity at the cost of more storage like Bucket Sort.
(2) We only test HiVid the video streaming scenario. However, the preference weights in video also benefit other vision-lant language-aattteet applications where human judgment also plays vital role. Therefore, another future direction is applying our core idea of LLM-guided rating and ranking in video compression. For example, can derive frame-level saliency D-À parameters. In this way, we reallocate the bitrates in regions that are appealing to humans.
to apply different R-
19


Reviews:
[{"review_id": "YriZWg8oIr", "reviewer": "Reviewer_zB8q", "summary": "This manuscript makes valuable contributions to content-aware streaming, a critical area for optimizing subjective QoE. The proposed HiVid framework addresses two long-standing pain points in streaming weight generation—prohibitive human annotation costs and poor generalization of vision-saliency models—by innovatively leveraging LLMs as a scalable human proxy. The work is theoretically motivated, methodologically rigorous, and experimentally comprehensive, with clear validation of performance gains for both VOD and live streaming scenarios.", "questions": "See the weakness.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "HiVid’s core idea—using LLMs to replace costly human annotation for streaming chunk importance weighting—is both creative and problem-driven. Unlike prior work that relies solely on vision-based saliency or limited human labels, the framework bridges LLMs’ semantic understanding capabilities with streaming’s practical needs, addressing a critical scalability gap. The three modules (perception, ranking, prediction) are tightly aligned to solve non-trivial, scenario-specific challenges:", "weaknesses": "Clarify LLM implementation details: The manuscript does not specify which LLM(s) were used (e.g., GPT-4, open-source models like LLaMA) or how LLM inference latency was managed for live streaming. Adding these details will improve reproducibility and help readers assess HiVid’s computational feasibility for edge deployment.\n\nElaborate on module ablation studies: While the overall performance gains are reported, a brief summary of ablation experiments (e.g., how removing the LLM-guided merge-sort affects VOD accuracy, or the impact of content-aware attention on live prediction) would reinforce the contribution of each module.\n\nExpand on video diversity: The manuscript does not specify the types of videos evaluated (e.g., sports, movies, animations). Adding a note on whether HiVid generalizes across diverse content genres would further highlight its robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ZUmjZ573Q", "reviewer": "Reviewer_y7h1", "summary": "This paper proposes using a multimodal LLM as a substitute for human judgments to predict temporal visual saliency in videos, with adaptive bitrate (ABR) as the primary application.\n\nBecause most current multimodal LLMs do not support video inputs and cannot take all video frames as context, the paper evaluates representative frames within local windows. The local-window constraint harms global consistency of saliency scores, which the paper attempts to fix by re-ranking with the LLM used as a comparison function.\n\nThe paper also considers a live streaming setting. Since saliency-based ABR there requires predictions for future chunks, they propose a CLIP-based forecasting module.", "questions": "- Table 3 reports latency for forecasting. What compute environment is used for this inference, and how intense the runtime compute is? This matters because it is likely to run alongside high-load tasks such as video playback.\n- The table on page 15 appears disproportionately large. It might be better to consider resizing the table size.\n- The detail of implementations of the forecasting model is hard to understand. This would cause reproducibility problem. Especially, In Figure 9 the legend shows MLP/Attention. Are MLPs also used for the QKV projections? Are CLIP weights updated? Would clarity improve if the CLIP vision encoder and text encoder were depicted separately?\n- It seems the positions of Table 3 and Table 4 swapped.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Comprehensive pipeline design spanning VOD and live streaming settings\n- The proposed method is evaluated against other methods using metrics derived from volunteer human ratings, and it shows improvements.", "weaknesses": "- The proposed method is based on proprietary LLMs. This makes the approach not robust to change in the proprietary LLM's service specifications and operating conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4PM15rfWCW", "reviewer": "Reviewer_44Re", "summary": "This paper proposes HiVid, a three-stage framework that leverages Large Language Models (LLMs) as human proxies to estimate content-aware saliency weights for adaptive video streaming (both VOD and live).\nThe framework comprises:\n- a Perception module, which assesses sampled frames through a sliding-window prompting strategy to generate local saliency scores and iterative video summaries;\n- a Ranking module, which applies a LLM-guided merge sort to globally re-rank frames and eliminate local inconsistencies; \n- a Prediction module, which performs multi-modal time-series forecasting to estimate future saliency weights in live-streaming settings.\n\nExperiments on TVSum, SumMe, and YouTube-8M show improved correlation metrics (up to +11.5% PLCC for VOD and +26% for live streaming) compared to video summarization and highlight detection baselines. A small user study reports better correlation between predicted QoE and human MOS.", "questions": "-\tHow do you ensure that LLM-generated scores are consistent across runs and models?\n-\tCould the authors clarify the exact nature of the “ground truth” used for correlation computation? Are the reference saliency weights derived from human annotations, subjective MOS labels, or pseudo-labels generated by the LLM?\n-\tCould the authors report results for a “forecast-only” setting, where the predictor operates without periodic LLM refresh? This would help understand how much the system relies on LLM inference in practice.\n-\tGiven the small sample (10 participants × 10 videos), could the authors provide information on inter-rater consistency or statistical significance of the reported PLCC improvements?\n-\tThe overhead analysis focuses on a 201-second clip. Have the authors explored or estimated how the cost and latency would scale for longer or continuous live streams?\n-\tCould the authors elaborate on how the LLM-based ranking is implemented in practice? For instance, are the pairwise comparisons deterministic (e.g., with fixed temperature and prompt order), and was any measure of ranking consistency across runs or window pairs evaluated?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Interesting and timely idea: leveraging LLMs as scalable surrogates for subjective human judgments is an emerging and relevant research direction. Applying this concept to video streaming optimization is novel and has clear practical implications.\n\n- Modular design: the separation into perception, ranking, and prediction modules is conceptually clean and covers both offline and online (live) cases.\n\n- Novel use of LLM reasoning: using an LLM as a semantic comparator in a merge-sort procedure is unconventional and potentially generalizable to other ranking tasks.", "weaknesses": "- Conceptual confusion or inconsistent terminology: The paper repeatedly refers to “video saliency prediction” or \"saliency score\", but most baselines (e.g., DETR, VASNet, PGL-SUM) are video summarization or highlight detection methods, not visual saliency models. While the \"saliency\" term is commonly associated in literature with spatial or spatio-temporal saliency maps that highlight visually regions within frames or videos, here it is used to denote subjective importance or priority score assigned to temporal chunks for bitrate allocation in streaming. This broader use of the \"saliency\" term may be misleading, especially readers familiar with classical video saliency prediction tasks. The paper would benefit from clearly defining \"saliency score\" early on and explicitly distinguishing its intended meaning from the traditional notion of saliency map. Providing alternative terminology (such as \"importance score\" or \"temporal relevance weights\") for the chunk-level scores could improve clarity and avoid confusion. \n\n- A potential limitation of the proposed ranking module lies in its reliance on LLM as the comparator function within the merge sort algorithm. While innovative, this design assumes that LLM can consistently and reliably perform pairwise comparisons that satisfy the properties required for sorting (e.g. transitivity and anti-symmetry). However, LLM outputs may be inherently variable, subjective, sometimes inconsistent, especially in tasks involving nuanced semantic judgments. There is no formal guarantee that the LLM will always induce a valid total order, which may lead to instability or errors in the final global ranking. The paper lacks a detailed analysis or empirical evidence on the robustness and consistency of the LLM-guided comparison. Addressing this aspect with more thorough evaluation or fallback mechanism would strengthen the approach. \n\n- Limited scientific contribution for ICLR: the paper primarily presents an application-driven pipeline that leverages existing LLMs as a zero-shot reasoning tool for relevance score assessment, combined with a learning-based forecasting module for live streaming weight prediction. While the system is creative, it is mostly engineering-driven, and the technical novelty in terms of learning methodology remains limited. This raises concerns about whether the contribution advances fundamental learning representation or model innovation, which constitutes the core criteria for ICLR acceptance.\n \n- The reported user study involves only 10 participants, which is a small sample size for drawing statistically reliable and generalizable conclusions regarding subjective Quality of Experience (QoE) in video streaming. Such a limited number of users increases the risk that individual preferences and variability disproportionately influence the results, reducing statistical power and limiting meaningful subgroup analyses. Therefore, this small sample size represents a methodological limitation of the study, and expanding the participant pool with a more diverse and larger user base would strengthen the validity and impact of the empirical evaluation.\n\n- Some implementation aspects of the Live Prediction Module are explained in more detail in the appendices (e.g., the use of both CLIP’s image and text encoders, the cross-attention fusion of modalities, and the training with randomized latency Δt to enable variable-length prediction). These details are only briefly mentioned or omitted in the main text (§3.4), making it difficult for readers to fully understand the model’s structure and training procedure without consulting the supplementary material. Providing a more self-contained description in the main paper—especially of how the adaptive decoding works—would substantially improve clarity and reproducibility.\n\n- The current evaluation mainly focuses on hyperparameter variations (e.g., window size, prediction horizon) but does not report experiments that isolate the contribution of the main components—Perception, Ranking, and Live Prediction. A more explicit analysis of how each module affects overall QoE correlation or latency would help clarify the role of individual stages and strengthen the empirical validation. \n\n- Evaluation of forecasting autonomy:  Since the forecasting module is intended to approximate the LLM’s saliency outputs, it would be informative to assess how well the system performs when operating autonomously—using the predictor without periodic LLM updates.\n Including such an experiment could clarify whether the proposed approach meaningfully reduces dependence on LLM inference and would provide stronger evidence for its real-time applicability.\n\n- Scalability and efficiency not demonstrated at realistic scale: The overhead analysis is performed on a single 201-second video, which does not convincingly demonstrate the scalability of the pipeline for longer or continuous live streams. Evaluating cost, latency, and performance over larger datasets or multi-hour content would strengthen the claims of efficiency and applicability to real-world streaming scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)