Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering
Authors: Jianfeng Cai
Jiale Hong
Zongmeng Zhang
Wengang Zhou
zhannianji
Houqiang Li
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Large Language Model
Hallucination
Activation Engineering
Primary area: deep_learning

Abstract: Multimodal large language models (MLLMs) have achieved remarkable progress in video understanding. However, hallucination, where the model generates plausible yet incorrect outputs, persists as a significant and under-addressed challenge in the video domain. Among existing solutions, activation engineering has proven successful in mitigating hallucinations in LLMs and ImageLLMs, yet its applicability to VideoLLMs remains largely unexplored. In this work, we are the first to systematically investigate the effectiveness and underlying mechanisms of activation engineering for mitigating hallucinations in VideoLLMs. We initially conduct an investigation of the key factors affecting the performance of activation engineering and find that a model’s sensitivity to hallucination depends on $\textbf{temporal variation}$ rather than task type. Moreover, selecting appropriate internal modules and dataset for activation engineering is critical for reducing hallucination. Guided by these findings, we propose a temporal-aware activation engineering framework for VideoLLMs, which adaptively identifies and manipulates hallucination-sensitive modules based on the temporal variation characteristic, substantially mitigating hallucinations without additional LLM fine-tuning. Experiments across multiple models and benchmarks demonstrate that our method markedly reduces hallucination in VideoLLMs, thereby validating the robustness of our findings.

Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering
Jianfeng Cait
Jiale Hong
Zongmeng Zhang?
Wengang Zhout*
Nianji Zhan
Houqiang Lit
Universiity Science and Technology of China
'Shanghai Jiaotong University
Merchants Union Consumer Finance Company Limited
xiaobaical caobmal.aammil.... ustc.edu.... cn
Abstract
Multimodal large language models (MLLMs) have achieved remarkable progress video understanding. However, hallucination, where the model generates plausible yet incorrect outputs, persists as a significant and under-addressed challenge in the video domain. Among existing solutions, activation engineering has proven cessful in mitigating hallucinations in LLMs and ImageLLMs, yet its applicability VideoLLMs remains largely unexplored. In this work, tematically investigate the effectiveness and underlying mechanisms of activation engineering for mitigating hallucinations in VideoLLMs. We initially conduct investigation the key factors affecting the performance of activation engincering and find that model's sensitivity to hallucination depends temporal varia- tion rather than task type. Moreover, selecting appropriate internal modules and dataset for activation engineering is critical for reducing hallucination. Guided by these findings, we propose temporal-aware activation engineering framework for VideoLLMs, which adaptively identifies and manipulates callucinationn-sesitiie modules based on the temporal variation characteristic, substantially mitigating hallucinations without additional LLM fine-tuning Experiments across multiple models and benchmarks demonstrate that our method markedly reduces hallucina- tion in VideoLLMs, thereby validating the robustness of our findings"
the first to sys-
Introduction
Large language models (LLMs) 27 48 72 have witnessed explosive growth in recent years, revolutionizing the field artificial intelligence with remarkable capabilities in understanding and generating human language. This success has naturally extended to multimodal domains, where LLMs are adapted to process and reason about diverse data types beyond text [19 53 32] 64 (8). Particularly noteworthy is the emergence of Multimodal LLMs (MLLMs) in the image domain (ImageLLMs) 40 31 49 5 that integrate visual and spatial perception with language understanding, and in the video domain (VideoLLMs) 6 go 47 [53]. which further capture temporal dynamics to enable comprehensive video understanding through joint spatiotemporal modeling.
Despite their impressive performance, MLLMS inherit and even amplify the hallucination issues prevalent LLMs 76 60 ImageLLMs frequently hallucinate non-existent objects [59).
"Corresponding author.
"Code and dataset are available at huggingface. co/ co/datasets/calj www.aaaannaaaaaaaaaaaaaaaaaaaaae TA-AE
www/ wwcoo/ccaiii....o jww.ffnggnaaa--.. hataa//cccccaa-jaaaenn/TTeta- AE and https://
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

misidentify visual elements [28 39,, or fabricate spatial relationships [70]. In the video domain, the additional temporal dimension substantially exacerbates hallucination problems [52]. VideoLLMs often confabulate events that never appear in the footage 82 misinterpret action sequences [41] or
fail to maintain temporal consistency 17 69, which creates increasingly complex challenges.
To alleviate the hallucination issues in VideoLLMs, researchers have proposed various approaches that broadly fall into two categories: model fine-tuning methods [52
66 38) and train-free
methods [63 79 12). Model fine-tuning approaches necessitate additional training, either by
training on carefully curated datasets [73] or by training auxiliary modules [52] specifically designed
to mitigate hallucination. However, these methods face significant challenges, notably the labor- intensive collection of high-quality datasets and the heavy computational cost of training large models. Consequently, train-free free methods have emerged as alternative solutions, encompassing self-feedback mechanisms [79], contrastive decoding strategies [82), and techniques that leverage auxiliary information [4]] Such train-free approaches offer the advantage of improving hallucination mitigation without necessitating costly retraining or extensive data collection efforts.
Activation engincering [50 85 44 9 is an emerging techniques that manipulate model activations during inference to influence model behavior. This approach has been widely applied in domains such as truthfulness control [45] and sentiment modulation [65] 36 Recently, researchers have begun to
explore its application mitigating hallucinations in LLMs 45 and I [22 ST These works indicate that activation engineering can better leverage correlations, reduce hallucinations without introducing additional side effects, while requiring fewer extra resources. However, its feasibility in the context of Videol LL remains an open question.
In this work, we introduce activation engineering to mitigate hallucinations in VideoLLMs for the first time and conduct a systematic investigation into its feasibility and underlying mechanisms, aiming to identify the key factors that influence its effectiveness. Our findings demonstrate that the two variants of activation engineering exhibit similar performance under any given dataset, indicating no essential difference between them. Furthermore, we reveal that the hallucination sensitivity of internal modules in VideoLLMs is strongly correlated with the temporal variation characteristics of tasks, rather than the task type. Moreover, selecting appropriate modules and datasets for activation engineering is crucial for effectively mitigating hallucinations.
Based on these findings, we propose temporal-aware activation engineering framework that adap- tively identifies and manipulates iallucination-sensitvyy modules according to temporal variation characteristics. Specifically, we divide the temporal variation characteristics of videos into temporal- invariant and lemporal-variant, and design a fully automated pipeline to collect high-quality datasets for each characteristie, Next, we train a lightweight classifier to predict the temporal variation of given input. Guided by the classifier, we select the corresponding dataset to identify hallucination- sensitive modules in the VideoLLM and apply activation engineering mitigate hallucinations. Our experiments demonstrate the effectiveness of our method, yielding consistent gains across multiple benchmarks (e.g., up to +5.52% on VidHallue [4 and up to +24 +-2.211% 21% on EventHallusion [82] and across diverse models, including Video-ILLaVA [47, Videol.LaMA2 [6.. and Qwen2.5-VI. 6
In summary, our primary contributions are follows:
We the first to introduce activation engineering to mitigate hallucinations in VideoLLMs and conduct a systematic investigation into its feasibility and intrinsic mechanisms.
Our study reveals that the hallucination sensitivity of internal modules in VideoLLMs is strongly correlated with temporal variation, e.g., temporal changes within videos.
We propose an effective activation engineering framework, which employs fully automated dataset construction pipeline and a temporal variation classifier to guide module selection.
Extensive experiments across multiple models and benchmarks demonstrate that our method significantly reduces hallucinations without any additional LLM training, while validating the correctness of our findings.
2 Related Work
In this section, we briefly introduce the background of MILMs, hallucinations in MILMss, and Activation Engineering. A more detailed version is provided in Appendix A

---

2.1 MLLMs for Video Understanding
Following the success of Large Language Models (LLMs) 8 18 26 48 researchers have extended them to incorporate visual inputs, giving rise to multimodal LLMs (MLLMs) 14 43.. In the video domain, frames are typically divided into patches treated as tokens 21], then processed by temporally visual encoder and aligned with the LLM token space via non-linear module 477 6 80] For example, Video-ILLaV
47 uses time-sensitive attention to capture trame-wise dynamics; the 16 80) leverages Q-formers 42 or convolutional downsampling to extract
VideoLLaMA series [8]
temporal features while reducing token count; and Qwen2.5-Vi VL [6] applies dynamic frame rate training with absolute time encoding to support resolution and frame rate adaptation.
2.2 Hallucinations in MILMs
Despite their strong performance, MLLMs often suffer from hallucinations [60] 78 7. generating inaccurate fabricated responses due to misinterpretation of input data. To address this in the image domain, two main strategies have emerged: training-baan based and training- free methods. The former improves alignment through additional data B4 33 24, but incurs high computational costs. The latter mitigates hallucinations during inference without retraining [74] 10 67]. For instance, apply contrastive decoding [46) to suppress hallucinated tokens. Other approaches include hallucination detection and correction [15 29 and boosting visual grounding via auxiliary signals like attention maps
Volcano [38] use self-feedback to refine outputs, while VCD
39
When extending MLLMs to the video domain, hallucination becomes more challenging due to temporal dynamics [52 69). To address this, researchers have improved image-based techniques, such
as collecting high-quality datasets for fine-tuning [52] 38) applying self-feedback [79], leveraging contrastive decoding 82 correcting hallucinations [22] and incorporating auxiliary signals to
enhance video understanding [41] For example, Vript 773 retrains the model on a 12k-sample
dataset to reduce hallucinations; TCD [82] uses contrastive decoding suppress hallucinated tokens; and DINO-HEAL [41] re-weights video embeddings using attention maps from a DINO [54] model. Concurrently, benchmarks like VidHal (17). EventHallusion [22 and VidHallue [4 have been
proposed to evaluate hallucination across different aspects. Despite these efforts, hallucination in
Video remains relatively under-explored compared to the image domain.
2.3 Activation Engineering
Activation engineering 65
30
58 refers to manipulating the activations within a model to
control its behavior. Prior works have demonstrated its effectiveness on truthfulness [5). formality
transfer [50]. sentiment control (65]
36 and refusal behavior control [
ITI 45 was the first to apply this technique to mitigate textual hallucinations in LLMs by computing activation vectors from normal samples and pallucination-induciiin samples, and injecting them into the multi-head attention layers during inference to suppress hallucinated content. More recently, several studies have extended activation engineering to address hallucinations in MLLMs for image- related tasks [2). Specifically, ICT [12] computes image-level and object-level vectors to reduce both global and local hallucinations. VTI [51] builds upon this by injecting activation vectors into both the vision encoder layers and the LLM layers, thereby mitigating hallucinations more effectively.
However, the feasibility of applying it to VideoLLMs remains an open question. In this paper, we explore the upplicability of activation engineering for mitigating hallucinations in VideoLLMs.
3 Temporal Variation Matters in Activation Engineering
In this section, we investigate the internal mechanisms of activation engineering and how key factors influence its effectiveness mitigating VideoLLM hallucinations under varying conditions.
3.1 Preliminary
Activation Engineering. Activation engineering mitigates hallucinations by computing offset vectors between normal and ietllucinationinnduiiii prompts, then injecting these averaged vectors into specific modules in the model during inference to guide the model toward non non-hallueinated outputs.
3

---

Transfomer layer times
Transfommer
talen
Rg
MLP
Ag
MIP m
m
Figt
(a) A ttaation Head Engineecing
(b) Layer Engineecing
Figure I: Overview of two common activation engineering variants.
where on and sy To enable it, we need to construct dataset D = ((. , ()), , normal and hallucination-inducing prompts, respectively, paired with shared ground-truth response y Each pair is fed into the model, and the vectors at the final token position are extracted
from N modules, yielding (t.f) and (/(-)) for module j = (1,..., N]. We define the offset vector (((-)) (i.i) (i-)) to capture the latent deviation induced by hallucinations. Aggregating across the dataset gives V() {u(+))) J. after which an average offset vector v(f) = [P] ((1))) //D| is computed per module and later injected during inference mitigate hallucinations.
As shown in Figure [ activation engineering is divided into Attention Head Engineering and Layer Engineering, based on the target modules. The former operates at the attention head level, and the latter operates at the transformer layer level. Taking Attention Head Engincering as an example, for LLM with L layers and M heads per layer, L x M offset vectors be computed. Prior work [45] suggests only a subset of heads are strongly all heads, selectively focus on the most sensitive ones to reduce computation.
thus, instead of using
To identify these, we train binary classifier for each attention head j using its vector v0 (v% ((J) (1.)). The top-K heads with the highest classification accuracy are retained, and offset vectors are computed and applied only to these during inference. See Appendix B for more details.
oh
Temporal Variation Characteristic (61 86]. A video is considered to exhibit temporal-invariant characteristic when it primarily involves fine-grained events without significant scene changes or temporal dynamics. Conversely, a video is regarded as having temporal-varian characteristic when it contains multiple events with substantial scene transitions and temporal variations.
Evaluations. Due to its diversity and comprehensive coverage, we adopt the VidHallue [1 bench- mark evaluation benchmark. Within VidHallue, BQA and MCQ emphasize fine-grained action comprehension and thus possess temporal-invariant characteristic, while STH and TSH target temporal scene transitions and thus possess temporal-variant characteristic. Additionally, these tasks span distinct task types: BQA (binary judgment), MCQ (multiple choice), STH (question answering). the base model for evaluation.
and TSH (ranking). We adopt VideoLLaMA2 41
3.2 Frame Reduction Induces Hallucination
Activation engineering requires collection normal prompts and hallucination-inducing prompts to compute ac- tivation vectors. Following TCD [82]. for given sample (m, g. y), where denotes the input video, q represents the question, and y is the corresponding response, the normal prompt v, is constructed by concatenating the video and the question, e.g., In = (m, g). To generate a hallucination- inducing prompt 24, apply the frame downsampling tech- nique to the video m and use the downsampled video m. combined with the same question q as 2h, e.g. JP (ma, q)-
8.9
0.
0.7
Accuracy
3.5
0.4
R
Ratend
Rater8
0.1
16
BQA
MCQ
ISH
Figure 2: Frame reduction amplifies hallucinations in VideoLLM. Rate de- notes the frame downsampling rate.
To validate the effectiveness of this strategy, we evaluate VideoLLaMA2 the VidHallue under different frames. As

---

shown in Figure the model's performance consistently degrades the number of frames de- creases, indicating an increased degree of hallucination. This supports the validity of using frame- downsampled videos as hallucination-inducing prompts.
3.3 Hallucination Sensitivity Correlates with Temporal Variation Not Task Type
Classitatin
Clasdiiate
Mia Yax
Tranforme Last
Tranformct Layer
(a) Layer-wise mean clasry-c accuracy across attention heads trained on vector sets Vnh..-
(b) Layer-wise distribution of binary-classifier accuracy across attention heads trained on Vun.s.
Figure 3: Binary-classife performance trained on vector sets Vnh,a from different task types. (b)
shows results for Vnn,s with the remaining results presented in Figures and 9f Appendix
To further investigate, we take Attention Head Engineering as a case study. Specifically, we sample a fixed number of examples from each of BQA, MCQ, STH, and TSH, and construct corresponding datasets D. = (m, ge y.). where * {B, M, S.T. For each example, we use the original video and question as the normal prompt In,. (m, 9-), and construct halltuciationniidduinn prompt Th.. = (mL, g.), where m is obtained by downsampling the original video frames by a factor 4.
We then use D. to extract the vectors from each attention head in VideoLLaMA2 and obtain the (1-j) (t.5)) for every head j. Each vector set nh. is split into training set vector set ()()) =dh- {u %h,*
and a validation set. For each attention head j. we train binary classifier using the training set prompt. Finally, we evaluate each classifier the corresponding validation set. The results of binary classifiers and a).
to determine whether the input vector v originates from a hallucination-indueeie
individually trained on each the four vector sets Van,- are shown in Figures
We observe two key findings from the results. First, based on the results shown in Figures b)) 8 and and))a) ) for all tasks types, the attention heads within each transformer layer exhibit highly similar performance. That is, the classification accuracy of the M attention heads within the same layer is nearly identical. This suggests that the attention heads within a transformer layer share similar capacity for perceiving ballucinations, which contrasts with prior studies 45
Second, as illustrated in Figure a). for tasks that have the same temporal variation characteristics but different task types, such as BQA and MCQ which possess temporal-invariant characteristic, the attention heads demonstrate similar capabilities in hallucination detection. Specifically, the accuracy trends of their corresponding classifiers are consistent. This indicates that the internal modules' hallucination sensitivity is task type agnostic, meaning their ability perceive hallucinations does not depend on the task type (c.g., binary classification or multiple-choice). Instead, these modules' hallucination sensitivity is correlated with the temporal variation characteristics of tasks, indicating that internal modules of the model (e.g., attention heads or layers) that effectively detect hallucinations vary according to the temporal variation characteristics of tasks. We also provide an intuitive explanation of this phenomenon in Appendix
3.4 Module Selection Determines Activation Engineering Performance
Next, we evaluate the Attention Head Engineering on the VidHalluc using each dataset D.. Specifi- cally, we select the top-K attention heads with the highest classification accuracy based on the binary classifiers introduced in Section 3.3 For each head jk, k 21,2, ..., K), we compute the offset vector set ((in)) using the corresponding vector set V(ik) nh... During inference with the VideoLLM, we apply activation engincering to the selected K attention heads. The results are presented in Table ]
Unless otherwise specified, Baseline in this paper refers direct evaluation of the VideoLLM.
5

---

Table 1: Results of Attention Head Engineering and Layer Engineering VidHalluc using different datasets. "Attn Head" denotes Attention Head Engineering, while "Layer" denotes Layer Engineering.
BQA
MCQ
STH
TSH
Datasets
Attn Head
Layer
Attn Head
Layer
Attn Head
Layer
Attn Head
Layer
Baseline
75.77
75.77
83.55
83.55
56.55
56.55
58.17
58.17
DB
76.82
74.77
83.70
83.33
54.28
49.55
56.83
55.33
DM 78.29
76.07
83.77
83.70
54.38
46.59
56.67
54.83
Ds 77.22
76.13
83.89
83.75
66.37
60.52
59.00
58.83
Dr 79.45
81.57
83,49
83.58
64.88
66.38
58.50
57.83
It can be observed that when Attention Head Engineering is performed using datasets with the same temporal variation characteristics the task, such as using DB for the MCQ task, it consistently produces positive effects. In contrast, when using datasets that have different temporal variation characteristics than the task, the effectiveness becomes inconsistent. Specifically, compared to the Baseline, using Dn Dar for the STH or TSH tasks leads to a decline in performance, whereas applying Ds or Dr for the BQA or MCQ tasks results in improved performance.
Further examination of Figure (aa) reveals an asymmetry in hallucination sensitivity across tasks with different temporal variation characteristies. The attention heads that are sensitive to hallucinations in the STH or TSH tasks also exhibit sensitivity to hallucinations in the BQA or MCQ tasks. However, the attention heads that are sensitive BQA or MCQ tasks do not exhibit the same sensitivity in STH or TSH tasks. Therefore, the offset vector sets Vs and Vr remain effective when applied to BQA and MCQ tasks, while the VB and Va are ineffective for STH and TSH tasks.
To further validate this hypothesis, we apply Attention Head Engineering the STH and TSH tasks using the selected at- tention heads As from Ds and the corresponding offset vector sets VB and Var. The results are presented in Table 2 observed that Attention Head Engineering continues to yield positive effects. This indicates that the offset vector sets com- puted from attention heads with strong hallucination sensitivity across tasks with different temporal variation characteristics. such as those in the deeper layers shown in Figure 3j). can contrast, offset vector sets derived from attention heads with temporal variation specific hallucination sensitivity, such as those in the shallow layers shown in Figure 3 a), cannot be used across tasks with different temporal variation characteristics. Therefore, carefully selecting appropriate attention heads for each task based on its temporal variation characteristics is essential for the effectiveness of activation engincering.
Table 2: Results of Attention Head Engineering on STH and TSH tasks using different dataset and selected attention heads combinations.
be
Datasets
STH
TSH
Baseline DB As
56.55
58.17
59.37
58.50
Dar + As
65.98
59.33
be used interchangeably.
3.5 Attention Head Engineering and Layer Engineering Exhibit Similar Trends
Next, we evaluate the Layer Engineering on the VidHallue using each dataset D.. The results are shown in Table T It can be observed that the performance of Layer Engineering closely resembles that of Attention Head Engineering. Furthermore, as shown in Section [3.3] attention heads within the same layer exhibit similar sensitivity to hallucinations. This suggests that Attention Head Engineering and Layer Engineering may be intrinsically equivalent, as both methods achieve consistent effects when applied during model inference using the same dataset.
3.6 Mixed Dataset Produces Moderate Effects Across All Tasks
Finally, we combine the four datasets D, into mixed dataset DD and evaluate the Attention Head Engineering on the VidHallue using D. As shown in Table 3 the mixed dataset leads to consistent performance improvements across all four tasks. However, the magnitude of improvement is smaller compared to the results reported in Table
Table 3: Results of Attention Head Engineering on VidHallue using mixed dataset.
Datasets
BQA
MCQ
STH
TSH
Baseline D
75.77
83.55
56.55
58.17
76.02
83.58
59,46
58.33

---

We further visualize the classification performance of each attention head based on the mixed dataset, as indicated by the purple curve (labeled VAA) in Figure a(... can be observed that the attention heads identified as sensitive to hallucinations from DD partially overlap with those selected from DB and DM. and partially with those from Ds and Dr. As a result, when applying activation engineering. only a subset of the selected attention heads aligns with those effective for each specific task. This partial mismatch ultimately leads to less significant improvements compared to those in Table [ This indicates that naïvely mixing dataset from tasks with different temporal variation characteristies does not lead to further performance gains across tasks. Instead, compared to using task-specific datasets, it results in a trade-off that balances performance between tasks.
Consequently, it is necessary to construct separate datasets for tasks with different temporal variation characteristics to enable the selection of effective modules and the computation of corresponding offset vector sets to fully leverage the potential of activation engineering.
4 Tewpe Activation Engineering: Method and Experiments
Iriend Sumplng
Gir--oo bes becp
Autorsiee Duta Cxinkin
Chass Serting
Isknna Sangling
CPT 46
(Fanss <Ti
(s) A Aubomatzdd Datzect Collection Pipeline for Diverse Tasks
(b) Classiiier Traiinn Pramowork
Figure 4: Automated dataset collection pipeline and temporal variation classifier training framework.
In this section, we construct temporal-aware activation engineering framework based on the findings of Section 3 The overall framework is illustrated in Figure We then evaluate our method through comprehensive experiments across various models and benchmarks, demonstrating its effectiveness in reducing ballucinations, thereby substantiating our findings.
4.1 Temporal-Aware Activation Engineering Framework
Basic Task Dataset Processing. Unlike earlier studies 12 51 that draw only from benchmark
subsets, we leverage existing open-source VQA dataset to preserve data diversity while preventing the performance inflation that arise from benchmark leakage. Specifically, we refrain from reusing the benchmark sources employed in Section 4.2 and instead adopt the ShareGPT4Video dataset [13 as our base collection, denoted as Do = {(m,, Gi 3i)) H According to Section 3.3 hallucination sensitivity of the model's internal modules is task type agnostic. Therefore, directly keep the original video description as the task type in our dataset.
Consistent with the finding in Section 33 the decisive factor for the hallucination perception capability of modules is the temporal variation characteristics of tasks. We therefore divide video understanding tasks into two sub-tasks: the task with camporal-invariant characteristic and the task with temporal-variant characteristic, and we prepare distinct datasets for each task.
Since activation engineering requires only a small amount of high-quality data, we clean Do to obtain approximately 10k samples in D,, with details Appendix D.I Then the videos in Dj are then sorted by frame count. To capture wide temporal span without processing the entire D,, we select samples at uniform intervals, yielding 1k videos that form the candidate pool for the temporal-varian variant task, denoted D, For the temporal-invariant -invariant task, whose videos are typically short, we discard any video in D2 whose frame count exceeds a threshold T, and again apply uniform-interval sampling to obtain 1A candidates, resulting in the dataset D.. In our experiments, T' is set to 200.
Filtering Datasets for Distinct Tasks. The two candidate datasets D, and Dy undergo an additional refinement stage. Guided by the prompt illustrated in Figure 7 in Appendix D.2 GPT-4o [32 evaluates every sample in D, and D,, judging whether its characteristic temporal-invarien or temporal-variant. Any entry for which the model assigns a Yes/No confidence below is removed to eliminate ambiguous cases. D., only samples classified as temporal-invane are preserved, yielding D/. In contrast, D; retains solcly the temporal- variant samples; the temporal-ver variiant items
7

---

that are excluded from Da are also merged into this set, producing D/d Statistics regarding D/d and Df can be found in Appendix D.3
Temporal Variation Classifier Training. To deter- mine the temporal variation characteristics of given (m,.) pair, we train a temporal variation classifier on DÍ and D.D. We randomly sample 400 instances from each dataset the training set and use the remaining samples for validation. Each (m, q) is fed into the clas- sifier, which optimized with a binary cross-entropy loss. The model that attains the highest validation ac- curacy is adopted as the final classifier.
Viscal Frooder
Tempara Variation Classfaer
Vidrod....
LLM
offeet
o/ of
Figure 5: Inference process of the temporal- aware activation engineering framework.
To avoid additional lateney during VideoLLM infer- ence, the classifier's execution is overlapped with the model's forward pass. Specifically, (m, q) is supplied in parallel to both the VideoLLLM visual encoder and classifier 0. Once the classifier returns classification result, the corresponding pre-computed offset vector set from either Dí or D selected and injected into the LLM continue the inference process. Because these offset vector sets are computed only once per VideoLLM, no extra runtime overhead is introduced. The complete inference process is illustrated Figure
4.2 Experimental Settings
Benchmarks. We assess the effectiveness of our method on two representative benchmarks. (a) Vid- Hallue [41] is a comprehensive benchmark for measuring hallucination in VideoLLMs. It covers three dimensions, action, temporal sequence, and scene transition, and comprises four task types: BQA, MCQ, STH, and TSH. For STH, the evaluation metric is an overall score obtained by a weighted combination of cinary-classffication task and the description task; the other three subtasks are evaluated purely by the accuracy of their respective tasks. (b) EventHallusion
82) is a recently proposed benchmark targeting hallucinations about events. It contains three subtasks, Entire, Mix, and Misleading, whose evaluation metric is binary-classification accuracy.
Models and Baselines. We adopt three VideoLLMs, Video-LLaVA 47 VideoLLaMA2 [16], and Qwen2.5-VL [6). as baseline models. For comparison, we include two inference-time hallucination- mitigation methods, TCD [2) and DINO-HEAL [4]. TCD applies contrastive decoding to Vide- oLLMs, whereas DINO-HEAL re-whights video features using attention weights produced by DINO. In addition, employ CGD (20) and VTI [5]], two representative contrastive decoding methods in the image domain, as comparative methods for evaluation on the VidHalluc benchmark. Since both CGD and VTI were originally proposed for image-based scenarios, extend them the video domain with several adaptations. Detailed implementation procedures are provided Appendix F
Implementation details. Unless explicitly stated otherwise, all experiments employ Attention Head Engineering as our primary method. For each VideoLLM, we pre-compute the Top-K attention- and the corresponding offset vector sets from D, and D. To ensure reproducibility, all methods use greedy decoding. Hyperparameters are tuned via grid search to systematically explore critical settings. Additional experimental details are provided in the Appendix E.
4.3 Experimental Results and Analysis
Results on VidHalluc. Table presents the performance on VidHallue. noteworthy finding is the robustness of our approach: it delivers substantial improvements over the baseline models across nearly all subtasks and models. For VideoLLaMA2, the gains on the MCQ, STH, and TSH subtasks are considerably larger than those obtained when the model is supplied with each subtask's original dataset (Table clearly demonstrating the value of our dataset. Relative to the compared mitigation methods, our approach achieves the best Overall performance on all models and attains the highest results in almost every subtask, with most margins exceeding 5%. In particular, on the Overall metric, our method surpasses TCD by up to 6.15%, DINO- by up to 4.5%, CGD by up to 3.88% and VTI by up to 2.16%. On BQA and MCQ tasks, the maximum improvements reach 2.32% and 0.87% over TCD, 6.41% % and 9.26% over DINO-HEAL, 3. 3.06% and 9.77% over CGD, and 0.95% and 5. 13% over VTI respectively. While on STH and TSH tasks, the corresponding gains
8

---

Table 4: Results VidHalluc benchmark.
Model
Variant
BQA
MCQ
STH
TSH
Overall
Baseline
75.77
83.35
56.55
58.17
68.46
TCD
76.77
83.65
55.19
56.67
68.07
DINO-HEAL
75.79
83.35
56.32
57.67
68.28
Vide
CGD
76.03
82.76
59.14
59.02
69.24
VTI
78.14
83.67
62.47
60.28
71.14
Ours
79.09
84.07
65.14
61.67
72.49
Baseline
73.50
63.66
60.55
59.17
64.22
TCD
75.49
74.35
46.81
65.83
65.62
DINO-HEAL
69.77
65.69
60.97
59.83
64.07
Qwen2.5-VL
CGD
74.23
65.18
60.97
59.86
65.06
VTI
75.49
69.82
61.74
60.97
67.01
Ours
76.18
74.95
61.81
61.33
68.57
Baseline
67.75
66.60
21.80
46.83
50.75
TCD
70.40
65.97
21.77
42.33
50.12
DINO-HEAL
70.82
67.19
26.47
47.50
53.00
Video-LLaVA
CGD
67.42
65.79
29.17
47.29
52.42
VTI
68.23
66.14
33.24
48.96
54.14
Ours
67.70
66.84
41.16
49.50
56.30
are up 19.39% and 7.17% over TCD, 14.69% and 4.00% over DINO-HEAL, 11.99% and 2.67% over CGD. and 7.92% and 1.39% over VTI. These consistent improvements over both video-native (TCD, DINO-HEALL HEAL) and image-originated (CGD, VTI) methods highlight the effectiveness and generalizability of our approach.
Table 5: Results EventHallusion benchmark.
Model
Variant
Entire
Mix
Misleading
Overall
Baseline
38.60
62.18
46.08
51.59
TCD
42.98
75.65
54.90
61.37
Lad22LLaMA2
DINO-HEAL
38.60
62.69
46.08
51.83
Ours
43.86
70.47
57.70
59.17
Baseline
71.93
39.38
98.04
63,09
TCD
69.30
43.01
97.06
63.81
Qwen2.5-VL
DINO-HEAL
80.70
35.75
95.10
63.08
Ours
89.47
40.93
100.00
69.19
Baseline
41.23
37.82
69.61
46.70
TCD
46.49
54.92
79.41
58.68
Video-LLaVA
DINO-HEAL
39.47
48.71
79.41
53.79
Ours
79.83
55.96
90.20
70.91
Results EventHallusion. EventHallussin results shown in Table indicating that our method yields consistent enhancements across all models and subtasks. Compared with the other approaches, again achieves the best performance on nearly every subtask, with most improvements exceeding 33.34%. These findings underscore the generalizability of our method and validate the effectiveness of categorizing video tasks by temporal variation characteristics, which enables more accurate guidance in selecting hallucination-senseee attention heads for each sample.
Table 6: Results VidHallue benchmark with large VideoLLM.
Model
Variant
BQA
MCQ
STH
TSH
Overall
Baseline
81.17
75.65
66.75
76.00
74.89
Wwen2.5-VL-72B
Ours
84.25
86.16
69.43
78.52
79.59

---

Scalability to Large Vid We further conducted additional experiments to verify the scalabil- ity of our method on the VidHallue benchmark using the Siven2.5-VL-725 model. The results are shown in G Firstly, it can be observed that while (wee2.55VVI-722) outperforms smaller models like Video models do indeed exhibit some mitigation of hallucination issues, hallucination problems still persist. Furthermore, for certain specific tasks (c.g., MCQ), the hallucination is even more severe compared to the VideoLLaMA2-7B. This highlights the persistent challenge of hallucination in VideoLLMs.
on most subtasks of the VidHallue benchmark, indicating that more powerful
Secondly, the results demonstrate that our method also achieves significant performance improvements on the Qwen2.5-VV VLLB-B model, demonstrating consistent hallucination mitigation capabilities across all subtasks. This proves the effectiveness and scalability of our method in large models.
Dataset statistics. We plot the accuracy of the binary shown
classifier trained DD using Vid2 in Figure T0 on Appendix[F [F] The accuracy curve follows the same trend as the accuracy curves of the STH (TSH) datasets Dy (Dr), which validates the effectiveness of our data-collection pipeline.
101
3307
femporal-lmannee
Temporal Narient
precisely gather samples with different temporal variation characteristics.
Number
25
Temporal Variation Classifier effectiveness. We next amine the performance of the trained classifier 0. Figure 6 reports its predictions the subtasks of both benchmarks. In VidHalluc, BQA and MCQ are temporal-invarian tasks, and A assigns more than 97% of their samples the temporal-invariant category. In contrast, STH and TSH temporal-variant tasks, and 0 assigns more than 94% of their samples the temporal-variant category. A similar pattern appears in EventHallusion events and thus align with temporal-invarianl tasks, whereas the Mix involves multiple events and therefore aligns with temporal-varte variant tasks. The classifier successfully captures these distinctions.
D0A
3400
5111
1311
Inar
Mix
Makkdnn
Figure 6: Temporal variation classifier predictions for each subtask.
the Entire and Misleading focus on single
Table 7: Results VideoMMMU benchmark.
Model
Variant
Perception
Comprehension
Adaptation
Overall
Baseline
59.83
46.12
32.25
46.07
Vide
Ours
59.47
46.24
32.25
45.99
Baseline
57.29
42.93
37.16
45.79
Qwen2.5-VL
Ours
57.46
42.57
36.98
45.67
Baseline 39.82
31.67
30.18
33.89
Video-LLaVA
Ours
39.82
31.59
31.09
34.17
Preservation of Model Capabilities. We conducted further evaluation on the VideoMMMU bench- mark, a multi-moda and multi cullii-dssciplinryy video benchmark that evaluates VideoLLMs' knowledge acquisition capability from educational videos, and the results are shown in Table The results show that the performance of our method on all subtasks of the VideoMMMU benchmark is comparable to Baseline, demonstrating that our approach does not impair the model's capabilities. Additional analysis and experimental results can be found in Appendix
Conclusion
In this study, we conduct the first systematic investigation into activation engineering feasibility and underlying mechanisms for mitigating hallucinations in VideoLLMs. We first perform an experimental analysis to identify the key factors that determine the effectiveness of activation engineering. Guided by these insights, we develop temporal-aware activation engineering framework that dynamically identifies and manipulates hallucination-ssnnse sensitive modules based on the temporal variation characteristics of task, without requiring additional LLM fine-tuning. Experimental results show that our method delivers significant performance improvements across all tasks, confirming its robustness in alleviating hallucination in VideoLLMs. Limitations can be found in Appendix G
10

---

Acknowledgement
This work was supported by the National Natural Science Foundation of China under Contract 623B2097, the Youth Innovation Promotion Association CAS. It was also supported by Merchants Union Consumer Finance Company Limited, and the GPU cluster built by MCC Lab of USTC the Supercomputing Center of USTC.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-2533 2022.
[3] Anthropic. The claude 3 model family: Opus, sonnet, haiku.
[4] Kyungho Bae, Jinhyung Kim, Sihaeng Lee, Soonyoung Lee, Gunhee Lee, and Jinwoo Choi. Mash-vim: Mitigating action- scene hallucination in video-Ilms through disentangled spatial- temporal representations. arXiv preprint arXiv:2503. 15871, 2025.
[5] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308, 12966, 1(2):3, 2023.
[6] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
[7] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey, 2025.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
[9] Zouying Cao, Yifei Yang, and Hai Zhao. Nothing in excess: Mitigating the exaggerated safety for Imms via safety-cone conecccoo activation steering. arXiv preprint arXiv:2408. 1491, 2024.
[10] Yue Chang, Liqiang Jing, Xiaopeng Zhang, and Yue Zhang. A unified hallucination mitigation framework for large vision-language models. Transactions on Machine Learning Research, 2024.
[11] Beitao Chen, Xinyu Lyu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Alleviating halluci- nations in large vision- lisoun-aanguage models through calluuenation-indueed induced optimization. In The eiiht Annual Conference on Neural Information Processing Systems, 2024.
[12] Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xuming Hu. Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large languaelag models. arXiv preprint 2xXiv:2411.15200 2024.
[13] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Zhenyu Tang, Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472 19495, 2024.
[14] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.000 2022.
11

---

[15] Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Unified hallucination detection for multimodal large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2225-3252, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[16] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in lideo-Ilms. arXiv preprint arXiv:2406.074700 2024.
[17] Wey Yeh Choong, Yangyang Guo, and Mohan Kankanhalli. Vidhal: Benchmarking temporal hallucinations in vision Ilms. arXiv preprint arXiv:2411. 2024.
[18) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[19] Wenliang Dai. Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[20] Ailin Deng. Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding. arXiv preprint arXiv:2402.15300 2024.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[22] Jinhao Duan, Fei Kong, Hao Cheng, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Truthprint: Mitigating Ivim object hallucination via latent truthful- truthful-guided pre- rre-interventinn arXiv preprint (rXiv:2503.1060) 2025.
[23] Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained model editing for language models. In The Thirteenth International Conference on Learning Representations, 2025.
[24] Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14303-14312, June 2024.
[25] Jiztom Kavalakkatt Francis and Matthew J Darr. Interpretable ai for time-series: Multi- model heatmap fusion with global attention and nlp-generated explanations. arXiv preprint arXiv:2507.00234 2025.
[26] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120030))e2300001622, 2023.
[27] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arxiv:2407.21783, 2024.
[28] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large dision-languaa models. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14375-14385, June 2024.
12

---

[29] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 38, pages 1135-181433 2024.
[30] Evan Hernandez, Belinda Z Li, and Jacob Andreas. Inspecting and editing knowledge represen- tations in language models. In First Conference on Language Modeling.
[31] Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv. Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408. 16500, 2024.
[32] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
[33] Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Versatile vision encoders for multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27992-28002, 2024.
[34] Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27036-27046, 2024.
[35] Prannay Kaul, Zhizhong Li, Hao Yang, Yonatan Dukler, Ashwin Swaminathan, CJ Taylor, and Stefano Soatto. Throne: An object-based hallucination benchmark for the free-form generations of large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27228-27238, 2024.
[36] Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, and Tobias Hecking. Style vectors for steering generative large language models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 782-802, 2024.
[37] Ming Kong, Xianzhou Zeng, Luyuan Chen, Yadong Li, Bo Yan, and Qiang Zhu. Mhbench: Demystifying motion hallucination in videollms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 4401-4409, 2025.
[38] Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. Volcano: Mitigating multimodal hallucination through self feedback guided revision. In Kevin Duh, Helena Gomez, and Steven Bethard, editors. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume I: Long Papers), pages 391-404, Mexico City, Mexico, June 2024. Association for Computational Linguisties.
[39] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13872-13882, 2024.
[40] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning, 2023.
[41] Chaoyu Li, Eun Woo Im, and Pooyan Fazli. Vidhalluc: Evaluating temporal hallucinations in multimodal large language models for video understanding. arXiv preprint arXiv:2412.03735, 2024.
[42] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[43] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: image pre-training for unified vision-language understanding and generation. In International conference machine learning, pages 12888-12900. PMLR, 2022.
Bootstrapping language-
13

---

[44] Kenneth Li, Aspen K Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Emergent world representations: Exploring sequence model trained on synthetic task. ICLR, 2023.
[45] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451 -41500, 2023.
[46] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12286-12312, Toronto, Canada, July 2023, Association for Computational Linguistics.
[47] Bin Lin. Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:231 10122, 2023.
[48] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
[49] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tnning. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 26296-26306, June 2024.
[50] Sheng Liu, Haotian Ye, Lei Xing, and James Y Zou. In-context vectors: Making in context Iearn- ing more effective and controllable through latent space steering. In International Conference on Machine Learning, pages 22287-323077 PMLR, 2024.
[51] Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in large vision-language models via latent space steering. In The Thirteenth International Conference on Learning Representations, 2025.
[52] Fan Ma, Xiaojie Jin, Heng Wang. Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hallucination in video language models via equal distance visual tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13151-13160, 2024.
[53] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan.
Towards detailed video understanding via large vision and language models. Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 12585-12602, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[54] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2024. Featured Certification.
[55] Yein Park, Chanwoong Yoon, Jungwoo Park, Minbyul Jeong, and Jaewoo Kang. Does time have its place? temporal heads: Where language models recall time-specific information. arXiv preprint 252.v22502.14258. 2025.
[56) Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo Maria Ponti, and Shay Co- hen. Spectral editing of activations for large language model alignment. Advances Neural Information Processing Systems, 37:56958-56987, 2024.
14

---

[57] Ruchit Rawal, Reza Shirkavand, Heng Huang, Gowthami Somepalli, and Tom Goldstein. Argus: Hallucination and omission evaluation in video-Ilms. arXiv preprint TrXiv:2506.0737/ 2025.
[58] Nina Rimsky, Nick Gabrieli, Julian Schulz. Meg Tong, Evan Hubinger, and Alexander Turner. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504-15522. 2024.
[59] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Ellen Riloff, David Chiang, Julia Hockenmaier, and innichhi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 0035-4045, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.
[60] Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. comprehensive survey of ballucination large language, image. video and audio foundation models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguisties: EMNLP 2024, pages I 11709-1199 11724, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[61] Weichao Shen, Yuwei Wu, and Yunde Jia. Temporal invariant factor disentangled model for representation learning. In Pattern Recognition and Computer Vision: Second Chinese Conference, PRCV 2019, Xi' an, China, November 8-11, 2019, Proceedings, Part II 2, pages 391-402. Springer, 2019.
[62] Shashwat Singh. Shauli Ravfogel, Jonathan Herzig, Roee Aharoni, Ryan Cotterell, and Pon- nurangam Kumaraguru. Representation surgery: Theory and practice of affine steering. In Forty-first International Conference on Machine Learning, 2024.
[63] Yiwei Sun, Zhihang Liu, Chuanbin Liu, Bowei Pu, Zhihan Zhang, and Hongtao Xie. Halluci- nation mitigation prompts long-term video understanding. arXiv preprint 2xXiv:2406.11333 2024.
[64] Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal lmss. Advances in Neural Information Processing Systems, 11:87310-87355 2024.
[65] Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. Activation addition: Steering language models without optimization. arXiv e-prints, pages arXiv-2308, 2023.
[66] Nasib Ullah and Partha Pratim Mohanta. Thinking hallucination for video captioning. In Proceedings of the Asian Conference Computer Vision, pages 3654-3671, 2022.
[67] David Wan, Jaemin Cho, Elias Stengel-Eskin and Mohit Bansal. Contrastive region guidance: Improving grounding in Dision-languee models without training. In European Conference on Computer Vision, pages 198-215. Springer, 2024.
[68] Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. In ACL (Findings). 2024.
[69] Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models. arXiv preprint arXiv:2406. 16338, 2024.
[70] Jiarui Wu, Zhuo Liu, and Hangfeng He. Mitigating hallucinations multimodal spatial relations through constraint-awar prompting. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Findings of the Asxociation for Computational Linguistics: NAACI. 2025, pages 3450-3468, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
15

---

[71] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zet, and Christoph Feichtenhofer. VideoCLIP: Contrastive pre-training for zero-shot video-text understanding. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6787-6800, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[72] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2.5 technical report. arXiv preprint arXiv:2412. 15115, 2024.
[73] Dongjic Yang. Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. Vript: A video is worth thousands of words. Advances in Neural Information Processing Systems, 37:57240-572611 2024.
[74] Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. Mitigating hallucination in large vision- language models via modular attribution and intervention. In The Thirteenth International Conference on Learning Representations, 2025.
[75] Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, and Dongsheng Li. Mitigating halluci- nations in large llnguage models via dpo: On-policy data hold the key. arXiv preprint arXiv:2501.09693 2025.
[76] Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: A review of hallucinations in large language models. arXiv preprint wrXiv2339.066794 2023.
[77] Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(1):2220005, 2024.
[78] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 129533, 2024.
[79] Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an EOS decision perspective. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17766-1781,, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[80] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprini arXiv:2501. 13106. 2025.
[81] Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA- An instruction-tumne language model for video understanding. In Yansong Feng and Els Lefever, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 543-553, Singapore, December 2023. Association for Computational Linguistics.
visuul
[82] Jiacheng Zhang. Yang Jiao. Shaoxiang Chen, Na Zhao, Zhiyu Tan, Hao Li, and Jingjing Chen. Eventhallusion: Diagnosing event hallucinations in video 1lms. arXiv preprint a40i 16597, 2024.
[83] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivlms through iillucination-awaan aware direct preference optimization. arXiv preprint arXiv:2311.1683 2023.
[84] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing lanionnl understanding with advanced large language models. The Twelfth International Conference Learning Representations, 2024.
16

---

[85] Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, al. Representation engineering: top-down approach to ai transparency. arXiv preprint arXiv:2310.01405. 2023.
[86] Will Y Zou, Andrew Y Ng. and Kai Yu. Unsupervised learning of visual invariance with temporal coherence. In NIPS 2011 workshop on deep learning and unsupervised feature learning, volume 2011.
NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We describe all claims in Abstract and Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion on the limitations in Section G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
17

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification: We fully disclose all information necessary to reproduce the main experimental results and validate the key conclusions of this paper, including the complete datasets and detailed hyperparameters deseribed in Section 4 and Appendix E datasets will be released upon acceptance.
The source code and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
18

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the complete settings used for all experiments in Section 4.1 and Appendix to reproduce all our experiments with instructions. All hyperparameters are described in Appendix The source code and datasets will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
guudes/CooeSSumisse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssionooiicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the complete settings used for all experiments in Section 4.1 and Appendix to reproduce all our experiments with instructions. All hyperparameters are described in Appendix E The source code and datasets will be released upon acceptance.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
19

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We conduct experiments across multiple models and benchmarks to validate the effectiveness of our method. Due to constraints in time and computational resources, error bars not included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide an estimate of the required computational resources in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:p//wearrrpp.../ppp///tttaaa
Answer: [Yes]
Justification:
We have read the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
20

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss both potential positive societal impacts and negative societal impacts in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We focuses mitigating hallucinations in existing open-source LideoLLMs using activation engineering during inference, without altering model generation behavior or releasing new models datasets. Therefore, all existing safeguards (if any) remain place.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
21

---

Answer: [Yes]
Justification: We explicitly cite dataset in Section 4.1 And all models and benchmarks are listed and cited in Section 4.2
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release our code and datasets for easy via Github and Hugging
Face.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We don't involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
22

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We don't 't involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We provide detailed descriptions of the usage of existing LLMs in Section and Appendix
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non nta-standard components.
Please refer to our LLM policy (https: //ttrr/nesipssp cc. Cc/Coonferencee/2255//// for what should or should not be described.
23

---

A Related Work
A.1 MLLMs for Video Understanding
Following the remarkable success of Large Language Models (LLMs) in natural language process- researchers have explored the integration of visual information into LLMs to endow them with broader capabilities, giving rise to the concept of multimodal LLMs (MLLMs) 19
ing I 27 g 18
26
48
72
64 43 84). In the image domain, common approach is to divide an image into patches, treat each patch as a token [21], and employ a visual encoder fol- lowed by non-linear alignment module to align the visual representation with the LLM's token space [31 40 5 49 More recently, this paradigm has been extended to the video domain, referred to [0). Unlike images, videos introduce temporal dimension, prompting the development of various methods for handling temporal information. For instance, Video-I.LaVa [47] employs a time-sensitive attention mechanism model frame-wise variations; the VideoLLaMA series [8] 16 80] leverages Q-formers [42] or convolutional downsampling to model temporal features while reducing the number of generated tokens, thereby mitigating computational overhead; and Qwen2.5-VL. [6] utilizes dynamic frame rate training along with absolute time encoding to support native dynamic resolution and frame rate adaptation.
53
2
32
14
VideoLLM 153
A.2 Hallucinations in MLLMs
Despite their impressive achievements, MLLMs prone to severe hallucination issues 60] 78 7 76). where the model fails to correctly interpret input data and generates inaccurate or entirely fabricated responses. To mitigate hallucinations in the image domain, variety of approaches have been proposed, which can generally be categorized into two types: additional training-based
and training-frre free methods 774 10, 38 77 67]. The first category focuses on constructing supplementary training data to further align model outputs. albeit at the cost of significant computational resources. The second category aims to reduce hallucinations during inference without requiring model retraining. For example, Volcano (38 and Woodpecker [77] adopt self-feedback mechanisms to iteratively refine model responses, while VCD [39 and ICD
methods [34 75 35 24
68
employ contrastive decoding [46] to suppress the probability of hallucinated tokens. Other approaches include hallucination detection and correction
[S] (2). as well as enhancing the model's attention to image content using auxiliary signals such as attention weights 83
When extending MILMMs to the video domain, hallucination becomes even more complex due the inherent temporal dynamics of video data [52] [82 17 69). To address this, researchers have adapted and extended image- techniques, including collecting high-quality datasets for fine-tuning [4]
66 [2 [38] applying self-feedbact mechanisms [79), leveraging contrastive decoding strategies [82).
detecting and correcting hallucinations (22] and utilizing auxiliary information strengthen the
model's 's understanding of video content 41 For instance, Vript (73) retrained the model on a 12k
example high-quality dataset diminish hallucination behaviour. VISTA-LLAMA [52] removed relative position encodings between visual and textual tokens and performs further training so that visual tokens exhibit a stronger influence during text generation. Zihao et al. 77 insert a special EOS token that encourages the model to terminate its output and self-assess the completeness of the generated content. TCD [82] adopts contrastive decoding to decrease the probability of producing hallucinated tokens. DINO-HEAL [1] re-weights video embeddings with attention maps generated by an auxiliary DINO (54 model, thereby attenuating the impact of irrelevant information. In parallel, range of benchmarks has been introduced to evaluate hallucination in
VideoLLMs 41 69 73 VidHal [77 focuses on temporal hallucination, EventHallusion [82 targets hallucination related to events, and VidHalluc [II provides comprehensive evaluation of hallucinations in motion details and spatiotemporal consistency. Nevertheless, compared to the image domain, research on hallucination in VideoLLMs remains relatively underexplored.
A.3 Activation Engineering
Activation engineering 65
refers to manipulating the activations within model to control its behavior. Prior works have demonstrated its effectiveness on truthfulness 45). formality
85
30
58
transfer [50]. sentiment control 65 36 and refusal behavior control [9]
24

---

ITI [45] was the first to apply this technique to mitigate textual hallucinations in LLMs by computing activation vectors from normal samples and hallueination-induee
samples, and injecting them into the multi-head attention layers during inference to suppress hallucinated content. More recently, several studies have extended activation engineering to address hallucinations in MLLMs for image- related tasks [ST] [2]. Specifically, ICT [12) computes image-level and object-level vectors, and leverages the ITI method to reduce both global and local hallucinations. VTI [51] builds upon this by injecting activation vectors into both the vision encoder layers and the LLM layers, thereby mitigating
hallucinations more effectively.
However, the feasibility of applying activation engineering to VideoLLMs remains an open question. In this paper, we explore the applicability of activation engineering for mitigating hallucinations in VideoLLMs. In addition, we examine the key factors that influence its effectiveness.
A.4 Activation Editing Methods
Recent works have explored activation editing techniques from spectral, geometric, and constrained perspectives. Spectral Editing of Activations (SEA) [56] leverages SVD-based inference- ttimeeeee-mmeeeee projec- tions cross-covariance matrices, achieving data-efficient edits without training. AlphaEdit [23] introduces null-space constrained weight updates, providing theoretical guarantees against catas- trophic forgetting during sequential edits. Singh et al. [62] formulates steering transformations as optimal affine mappings over hidden states, effectively addressing biases and toxic outputs. Unlike these approaches, our method dynamically identifies ialluuiinationssesitiiy modules in VideoL.LM, via temporal-aware activation editing, adapting inference- edits specifically to temporal variation characteristics in video tasks.
B Basic Process of Activation Engineering
Activation engineering requires the construction of curated dataset D = ((=.1) ( ())),, ((1))((z)) (1 ).
(1) denotes kallucination-induee prompt, and ())) is the corresponding ground-truth response. For each data pair, both (an , () and (xr)) ()) passed through the LLM, and the model's internal vectors from N distinct modules at the position of the final token are extracted, denoted as v(3) = j (1,2, N). This process yields a set of vectors for each module: (t (5.1) and o/
i {1,2, .... [P]), where (i) denotes normal prompt, 2/
(((5)))) } and (i) -00 (11)), J. where
nh {up (3.1) 1/ (15)). Since
(6.5) represent the normal latent states and hallucination -arnniinnatoo-onnnneeee latent states respectively, an
offset vector ((b.)) on (i-5) u/ ((.5) is computed to characterize the transition from the hallucinated latent space to the normal latent space. Aggregating these across the dataset yields offset vector sets vi) = (v()))), and an average offset vector w/6)) = E! ((6d)// ||D| is computed per module to obtain the final offset vector set V = {2()), with one vector per module. During inference, these offset vectors ماui are injected into their corresponding modules within the LLM to suppress hallucination behaviors.
As illustrated in Figure 1 activation engineering can be categorized into Attention Head Engineering and Layer Engineering, depending on the position of the modules selected for injection. The former targets individual attention heads, while the latter operates at the level of transformer layers. Taking Attention Head Engineering as example, for LLM with L transformer layers and M attention heads per layer, a total of N = L x M offset vectors need to be computed. Prior studies have shown that different attention heads exhibit varying levels of sensitivity to hallucinations. Therefore, instead of applying offset vectors to all attention heads, we selectively compute offset vectors only for those with strong hallucination sensitivity, thereby reducing unnecessary computation.
(4-) (1.5 to Specifically, for each attention head j. we utilize its associated vector set D(و = (ulf- .b train a binary classifier that determines whether the head is sensitive to hallucinations. We then select the top- K attention heads with the highest classification accuracy and apply offset vector injection only to these heads during VideoLLM inference.
25

---

C Intuitive Explanation of Hallucination Sensitivity and Offset Injection
Task-agnostic but temporally correlated hallucination sensitivity. Firstly, of the chal- lenges for VideoLLMMs is effectively capturing and integrating the temporal dynamics of videos. When temporal information is incomplete or misinterpreted. the model is more prone generating hallucinations [37 (7). Conversely, task types often represent a higher-level semantic abstraction that is largely independent of the underlying video's intrinsic temporal characteristics. For given video, the same underlying temporal events can be probed with different task formats. For example, multiple-choice question can be easily rephrased into true/false question without altering the video's 'ss temporal content.
The effectiveness of injecting offset vectors. Secondly, regarding the effectiveness of injecting offset vectors, we can view the attention heads and layers within a VideoLLMM as "experts" that process information different levels of abstraction. Some of these "experts" may be specifically responsible for handling temporal dynamics or integrating information across time, which is consistent with previous findings 25 (55) When these "experts" receive ambiguous or inconsistent temporal signals (e.g., due to frame downsampling), they may produce biased internal representations, leading hallucinations. By calculating and injecting "offset vectors", we are essentially introducing "correction signal" into the internal workspace of these critical "experts". This signal pushes the activations towards state closer the true. non-hallucinated output, thereby correcting the "latent deviation" caused by incomplete or misleading temporal information. This has also been This targeted intervention, combined with our discovery of the modules' sensitivity to temporal variations, enables activation engineering to efficiently and specifically mitigate hallucination problems in L.Ms
demonstrated in previous works I2 45
Temporal Variation Class ification Prompt
This video cones with description sumnarizes any "'significant scene changes** based on both the description and the extracted franes.
analyze whether
A **scene change** is defined one of the following:
1. **Event change**; Different activities or events occur across frames, such as switching to place,
time, or different plot segnent.
2. **Viewpoit or canera change**; The same event is shown, but from a noticeably different perspective
(e.g., switching from stage view audience view, from wide shot close-up).
Here is description: "(description)"
Please answer in *"strict JSON format"", with the following fields:\n
{(
\'scene ""scene_change""" true or false,
>rreason"": V'A short explanation your reasoning. }}
...
Return only valid (video)
include any other commentary explanation.
Figure 7: Temporal variation classification prompt for identifying the temporal variation characteris- ties of each sample.
D Additional Details of Temppral-Aware Activation Engineering Framework
D.I Data Cleaning
Although ShareGPT4Video originally contains 40k (video, question, answer) triples, activation engi- neering benefits more from a compact, high-quality corpus. begin with dataset cleaning through a multi-step process. First, duplicate videos are identified and removed to ensure uniqueness. Next, we filter out clips with low visual quality (e.g., excessive blur, compression artifacts) or incomplete con- tent that may hinder understanding, For the remaining videos, inspect associated metadata, such
26

---

as titles, descriptions, and timestamps, and discard entries with missing inconsistent information. Finally, we apply automated heuristics to detect and eliminate residual noisy or anomalous samples, such as videos with corrupted frames or mismatched audio-visual alignment. These operations reduce the collection to 10k items, which we denote as 1
D.2 Filtering Datasets for Distinct Task Types
Figure illlusratee the temporal variation classification prompt we used to judge whether a task is temporal Innprral-invariant invariant temporal- varriat--variant.
D.3 Dataset Statistics
In our experiments, т is 0.8. The dataset D contains 422 sample instances, while the dataset Df contains 574 sample instances. The average video length varies between the two datasets: D videos have an average length of 104.37 frames, whereas Df videos have an average length of 786.49 frames. These statistics highlight the rationality our constructed datasets, where Df videos are shorter and exhibit temporal-invariant invariant characteristic, while D/ videos are longer and demonstrate temporal-vane variant characteristic.
E
Detailed Experimental Settings
we use concatenating the Normal and hallucination-inducing prompt. Consistent with Section 3.2 original video and the question the normal prompt æ; and use concatenating the 4x downsampled video and the question as the hallucination-inducing prompt Th-
Extending CGD and VTI to the video domain. Since both CGD and VTI primarily image-based methods, we adapted them to the video domain with the following improvements:
For CGD, we replaced the CLIP model used for similarity score calculation with VideoCLIP model [71]
For VTI, masked all frames of each video to generate a set of corresponding random masked videos. Additionally, for each video caption x, we used GPT-4o [3] to generate its corresponding hallucinated version T.
For fair comparison, we used the union of Dí and D' generate VTT's visual and textual directions.
All other hyperparameters adopted the optimal values provided in their respective papers.
Hyperparameter tuning, In our experiments, we employ greedy decoding to ensure reproducibility and perform grid search for hyperparameter tuning, systematically exploring combinations of key parameters. The main hyperparameters include K, the number of top-ranked attention heads selected, and a, the scaling factor applied to the offset vectors. The search space defined as the Cartesian product:
332, 64, 128, 256}
x {8, 16, 22)
where resulting in 4 x 4 16 unique configurations, a relatively small search space. For compared methods, we also apply grid search for fair comparison. For TCD, we tune the frame downsampling rate r, well as the contrastive decoding parameters and B, over the space:
22 x {0.25, 0.5, 0.7,,.7..11. 1.0) x (0.1,
(2)
yielding 3 x 4 x 2 = 24 configurations. For DINO-HEAL, we search over combinations of normalization (enabled not) and three DINO variants with or without registess,,, to 2x2x3 12 configurations in total.
Temporal Variation Classifier. Owing to VideoL.LaMAT strong spatiotemporal representation and low inference latency, we utilize its submodules to construct 0. Specifically, we select V- LaMA2's visual encoder and the first transformer layer of its LLM as the backbone for 0 followed
27

---

by a linear classification layer that performs classification based on the output of the last token. For classifier training, to reduce computational overhead, we freeze the backbone of the classifier and only train its final classification layer. The learning rate is set to 1 x 10-5, followed by cosine learning rate schedule with an initial warmup of 10 steps and a batch size of The classifier trained for 5 epochs.
All experiments are conducted on single machine with 8 NVIDIA A 80 GB GPUs.
F
Additional Experimental Results
1
9
Clssifittonn Aocoee
2
E
Trndormer Layaz
Tradoomer Lagar
(a) Layer-wise distribution of binary- classifier accuracy across attention heads trained on V.s.B.
(b) Layer- wise distribution of dinary-classinee accuracy across attention heads trained on V.k,
Figure 8: Binary classifier performance trained on vector sets Vnn,B and Vnn.M.
T T
9 2 9 26
Cheiffatie Accra
1
Claiffation
Dr
(a) Layer-wise distribution of classier accuracy attention heads trained V.A.T.
(b) Layer- wise distribution of binary- classifier accuracy across attention heads trained on V.n.
Figure 9: Binary classifier performance trained on vector sets VnA.T and Vnk-
Table 8: Ablation results on hyperparameter sensitivity in the VidHalluc benchmark.
a Range/Top-K Attention Head Selection Number
32
64
128
256
8
71.25
71.56
71.94
71.47
16
71.28
71.79
72.08
71.63
24
71.72
71.98
72.17
71.84
32
71.74
72.03
72.49
71.96
Ablation hyperparameter sensitivity. We further conducted ablation study on the value range of the injection weight (a) and the number of Top-K attention heads selected. Specifically, following the grid search settings the paper, we performed grid search for the injection weight in the range [8, 16, 24, 32) and for the Top-K attention head selection number in the range [32, 64, 128, 256], using the VideoLLaMA2 model on the VidHallue benchmark. The experimental results are shown in Table T (using Overall as the evaluation metric). As can be seen, while the injection weight (a) and the number of Toop-K attention heads do influence model performance, the sensitivity not drastic. Within reasonable range, model performance generally improves as both the injection weight and the number of selected Top-K attention heads increase. However, when more attention heads (256) that are less sensitive to hallucinations are included, there is slight decrease in performance. In practical applications, an accuracy threshold strategy can also be adopted to select attention heads, further enhancing the robustness of the model.
28

---

1.0
Classification Accuracy (%)
0.8
0.7
Vah.
Vus, T
0.5
Vah,
10
15
20
25
30
Transformer Layer
Figure 10: Layer-wise mean binary-classifier accuracy across attention heads trained on vector sets
Vul,a from Ds, Dr, and D
Table 9: Results on VidHalluc and EventHallusion benchmarks with LaMA2.
VidHalluc STH
EventHallusion
Method
BQA
MCQ
TSH
Overall
Entire
Mix
Mislcading
Overall
Ours
79.09
84.07
65.14
61.67
72.49
43.86
70.47
57.70
59.17
Ours (Orucle Classification)
78.46
83.98
64.88
60.50
72.00
42.98
67.87
54.90
57.70
Ablation without pre-classification. We conduct an additional study in which we skip the classifier and directly select the data source according to the above analysis in Section 4.3 Specifically, for BQA and MCQ in VidHalluc and Entire and Misleading in EventHallusion, we perform activation engineering using Da; for STH and TSH in VidHalluc and Mix in EventHallusion, we use Di, denoting this setting as Ours (Oracle Classification). shown in Table 9 the results of Ours (Oracle Classification) closely match those of Ours, further confirming both the quality of our dataset and the effectiveness of the classifier
Table 10: Per-sample inference time on the VidHalluc benchmark (unit: seconds).
Model
Variant
BQA
MCQ
STH
TSH
Baseline
1.44
1.66
1.87
1.92
Lid2
Ours
1.44
1.66
1.86
1.93
Baseline
0.99
1.00
1.35
1.10
Qwen2.5-VL.
Ours
0.99
1.00
1.34
1.10
Baseline 2.43
2.49
3.18
3.86
Video-LLaVA
Ours
2.43
2.49
3.19
3.88
Inference efficiency. We further evaluated the pam-se inference time of both our method and the Baseline on the VidHallue benchmark. To ensure fairness, each model was tested on a single 3090 GPU, eliminating any additional overhead from distributed inference. As described in Section 4.1 our method reduces inference-ting overhead by executing the classifier and visual encoder in parallel. The results, presented in Table TO also show that the per-sample inference time of our method is comparable to that the Baseline, confirming the efficiency our parallel design.
Classifier sample efficiency. We further tested the classifier's 's sample efficiency by reducing the number of training samples. Specifically, we trained with 100, 200, 300, and 400 samples respectively, and recorded the classification accuracy of the classifier on the VidHalluc benchmark for each quantity.
29

---

Table 11: Sample efficiency results of the classifier the VidHallue benchmark.
Sample Size
BQA
MCQ
STH
TSH
100
90.17%
89.63%
88.09%
88.83%
200
92.13%
91.47%
91.01%
90.33%
300
94.26%
95.39%
92.58%
92.50%
400
97.63%
97.47%
94.83%
94.17%
The experimental results are shown in Table 11 increases, the classifier's accuracy gradually improves. However, with only 100 training samples, the classifier's accuracy still exceeds 88%. This demonstrates that our classifier possesses high sample efficiency.
can be observed that as the number of samples
Table 12: Hallucination consistency in failure cases on VidHalluc benchmark.
Model
BQA
MCQ
STH
TSH
Overall
Qwen2.5-VL-7B
95.24%
92.37%
96.13%
95.92%
94.92%
Failure cases. We further analyzed the specific changes in model output when using the Baseline versus our method, across all subtasks' failure cases on the VidHalluc benchmark with the Qwen2.5- VL-7B. Specifically, we used GPT-4o determine whether the hallucinations produced by the Baseline and our method's outputs were consistent, and we calculated their hallucination consistency rate. The experimental results are shown in Table 12 As can be seen, the hallucination consistency rate of our method with the Baseline exceeds 90% across all subtasks' failure cases. This indicates that in the vast majority of failure situations, our method gracefully degrades to the original model's behavior and does not further exacerbate the hallucinations.
Ground Truth: Scene change: Yes, Locations: from stage in the woods.
Model Output (Baseline): Scene change: Yes, Locations: from dark room to car.
Model Output (Ours): Scene change: Yes, Locations: from performing stage to being in the woods.
Figure 11: Case study of hallucination reduction on the STH description task.
Case Study. We present a concrete example to illustrate how our method reduces hallucinations. Specifically, since most subtasks selection/classiifcation tasks, we will focus providing specific examples from the STH's description task. We demonstrate the specific changes in model output after using the baseline and after applying approach mitigates hallucinations. The example shown in Figure TT As can be seen, the Baseline exhibited hallucinations regarding the order of scene changes and scene recognition, while our method effectively reduced these hallucinations.
method to more intuitively understand how
G
Limitations
Despite the substantial improvements achieved by our method, several limitations remain to be addressed. First, our current approach adopts a sourse-grained taxonomy that partitions video tasks into only two categories based on their temporal variation characteristics: semporal-invarianl and temporal-variant While this dichotomy effectively captures broad distinctions in hallucination sensitivity, it oversimplifies the inherent heterogeneity within and across tasks. We hypothesize that developing a more fine- anee-rraied taxonomy could further enhance the adaptability and efficacy of activation engineering in mitigating hallucinations.
Second, although we employ optimization strategies such as pre-computing offset vectors and overlapping the classifier's execution with the model's forward pass, our method still introduces slight computational overhead during inference. Addressing this limitation calls for future research into more efficient activation engineering paradigms.
30

---

H Societal Impacts
H.1 Positive Impacts
The proposed temporal-aware activation engineering framework significamtly mitigates hallucinations in VideoLLMs without requiring additional model fine-tuning. This advancement enhances the reliability and factual accuracy of Al-generated video understanding, benefiting applications such as video-based education, content summarization, and numan-computer interaction. By addressing hallucination issues, the work promotes safer deployment of AI in sensitive domains like healthcare, security surveillance, and autonomous systems, where erroneous outputs could have serious conse- quences. Additionally, the method's s train-free nature reduces the environmental and financial costs typically associated with large-scale model retraining, contributing to the sustainable development of Al technologies.
H.2
Negative Impacts
Despite its advantages, the method introduces extra computational overhead during inference, which could limit its scalability in real-time applications or resource-constrale environments, Furthermore, improving the realism and believability of Al Al-generated video outputs might inadvertently facilitate malicious uses, such as generating more convincing deepfakes or disinformation. Finally, while the work mitigates hallucinations, it does not eliminate them entirely, and overreliance on such systems without proper human oversight could lead to unforeseen societal risks.
31


Reviews:
[{"review_id": "RhF0ADkyf9", "reviewer": "Reviewer_mRo1", "summary": "This paper leverages activation engineering to mitigate hallucinations in VideoLLMs. The authors systematically investigate various activation engineering mechanisms and discover that the temporal variation characteristics of tasks play a more significant role than the task type itself. Based on this insight, they propose a temporal-aware activation engineering framework that adaptively manipulates activations according to temporal variation characteristics. Additionally, the authors develop an automated pipeline to collect high-quality datasets for both temporal-invariant and temporal-variant video types. Extensive experiments demonstrate the effectiveness of the proposed method, yielding consistent improvements in hallucination mitigation across multiple benchmarks.", "questions": "It appears that contrastive decoding methods could also be adapted for hallucination reduction. The paper would benefit from a comparison with more advanced contrastive decoding approaches, such as [1-4], which can be readily applied to reduce hallucinations in video generation tasks.\n\n[1] Chen, Zhaorun, et al. \"Halc: Object hallucination reduction via adaptive focal-contrast decoding.\" arXiv preprint arXiv:2403.00425 (2024).\n[2] Deng, Ailin, Zhirui Chen, and Bryan Hooi. \"Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding.\" arXiv preprint arXiv:2402.15300 (2024).\n[3] Liu, Sheng, Haotian Ye, and James Zou. \"Reducing hallucinations in large vision-language models via latent space steering.\" The Thirteenth International Conference on Learning Representations. 2025.\n[4] An, Wenbin, et al. \"Mitigating object hallucinations in large vision-language models with assembly of global and local attention.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- The paper provides novel insights and solid findings regarding the key factors of activation engineering, highlighting that temporal variation characteristics are more crucial than task type.\n- The proposed temporal-aware activation engineering framework is simple yet effective in reducing hallucinations in VideoLLMs.\n- The paper is well-written and easy to follow.\n\nWeaknesses:\n- The scalability of the proposed method is not clearly demonstrated. Since the most powerful models (like Gemini, Qwen2.5-VL-72B) may not suffer from severe hallucination issues, the effectiveness of the proposed approach in such models remains uncertain and may be limited.", "was_revised": true, "final_justification": "After considering the rebuttal, my concerns are fully addressed, which leads to the keep of positive rating. Since paper presents a novel temporal-aware activation engineering framework that addresses the problem effectively, the paper should be accepted.", "rebuttal": ">**Q1: The scalability of the proposed method is not clearly demonstrated. Since the most powerful models (like Gemini, Qwen2.5-VL-72B) may not suffer from severe hallucination issues, the effectiveness of the proposed approach in such models remains uncertain and may be limited.**\n\n**A1**: Thank you for your question. We further conducted additional experiments to verify the effectiveness of our method on the VidHalluc benchmark using the **Qwen2.5-VL-72B-Instruct model**. The results are shown in the table below:\n\n|Model|Variant|BQA|MCQ|STH|TSH|Overall|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL-72B|Baseline|81.17|75.65|66.75|76.00|74.89|\n||Ours|**84.25**|**86.16**|**69.43**|**78.52**|**79.59**|\n\nFirstly, it can be observed that while the Qwen2.5-VL-72B model outperforms smaller models like Qwen2.5-VL-7B and VideoLLaMA2-7B on most subtasks (BQA, STH, TSH) of the VidHalluc benchmark, indicating that more powerful models do indeed exhibit some mitigation of hallucination issues, **hallucination problems still persist**. Furthermore, for certain specific tasks (e.g., MCQ), the hallucination is **even more severe** compared to the VideoLLaMA2-7B. This highlights the persistent challenge of hallucination in VideoLLMs.\n\nSecondly, it can be seen that our method also **achieves significant performance improvements** on the Qwen2.5-VL-72B model, demonstrating consistent hallucination mitigation capabilities across all subtasks. This proves the effectiveness and scalability of our method in alleviating hallucination in large models.\n\n>**Q2: It appears that contrastive decoding methods could also be adapted for hallucination reduction. The paper would benefit from a comparison with more advanced contrastive decoding approaches, such as [1-4], which can be readily applied to reduce hallucinations in video generation tasks.**\n\n**A2**: Thank you for your suggestion. Due to limitations in time and computational resources, we selected **CGD and VTI** ([2] and [3] from your provided papers) for further comparison experiments on the VidHalluc benchmark, using the VideoLLaMA2-7B, Qwen2.5-VL-7B and Video-LLaVA-7B models.\n\nSince both CGD and VTI are primarily image-based methods, we **adapted them to the video domain** with the following improvements:\n\n- For CGD, we replaced the CLIP model used for similarity score calculation with VideoCLIP.\n\n- For VTI, we masked all frames of each video to generate a set of corresponding random masked videos. Additionally, for each video caption $x$, we used GPT-4o to generate its corresponding hallucinated version $\\tilde{x}$.\n\n- For a fair comparison, we used the union of $\\mathcal{D}_a^f$ and $\\mathcal{D}_t^f$ to generate VTI's visual and textual directions.\n\n- All other hyperparameters adopted the optimal values provided in their respective papers.\n\nThe experimental results are shown in the table below:\n\n|Model|Variant|BQA|MCQ|STH|TSH|Overall|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|VideoLLaMA2|Baseline|75.77|83.35|56.55|58.17|68.46|\n||CGD|76.03|82.76|59.14|59.02|69.24|\n||VTI|78.14|83.67|62.47|60.28|71.14|\n||Ours|**79.09**|**84.07**|**65.14**|**61.67**|**72.49**|\n|Qwen2.5-VL|Baseline|73.50|63.66|60.55|59.17|64.22|\n||CGD|74.23|65.18|60.97|59.86|65.06|\n||VTI|75.49|69.82|61.74|60.97|67.01|\n||Ours|**76.18**|**74.95**|**61.81**|**61.33**|**68.57**|\n|Video-LLaVA|Baseline|67.75|66.60|21.80|46.83|50.75|\n||CGD|67.42|65.79|29.17|47.29|52.42|\n||VTI|**68.23**|66.14|33.24|48.96|54.14|\n||Ours|67.70|**66.84**|**41.16**|**49.50**|**56.30**|\n\nAs can be seen, **our method outperforms both CGD and VTI** in mitigating model hallucinations across various subtasks and models on the VidHalluc benchmark. This fully demonstrates the advantage of our approach in alleviating model hallucinations."}, {"review_id": "k4LrFcliNF", "reviewer": "Reviewer_4MRg", "summary": "The paper studies the causes of hallucination in video LLMs and also proposes a mitigation technique based on the finding. The core idea is to compute internal offset vectors, which means the activations between normal and hallucination-inducing inputs, and inject these offsets during inference to drive the model to the correct outputs and away from the hallucinated outputs.\n\nA key insight is that hallucination is tied to a video’s temporal variation more than the tasks. So the idea is to use temporal-aware activation engineering, which first classifies a video into temporal-variant or temporal-invariant and applies the appropriate offset vector accordingly.\n\nThe experiments are comprehensive, evaluating on two benchmarks, VidHalluc and EventHallusion, using multiple models like VideoLLaMA and Qwen-VL, and show the approach reduces hallucinations significantly and outperforms all baseline methods like TCD and DINO. This method is also training-free and can generalize across tasks and architectures.", "questions": "- Can you provide any theoretical analysis or intuitive explanation for why this mechanism reduces hallucinations effectively?\n- How do you ensure that the offsets generalize to different videos and video types?\n- Why is a binary classification sufficient? What about videos that do not fit cleanly into either category?\n- Are there any known failure cases, and how does the method behave in those situations?\n- Are there other types of hallucinations that are as important as or more important than temporal variation (e.g., factual, spatial, or visual hallucinations)?\n- How are shallow and deep layers defined in your method? Is this based on layer index or some functional behavior?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Observation and insight: Hallucination in VideoLLMs seems to be more related to temporal variation. This observation opens up directions for understanding and mitigating hallucinations.\n- Simple yet effective method: The proposed TA-AE framework is simple—compute and inject offset vectors between normal and hallucination-inducing activations. It is shown to be effective across models and benchmarks, without the need for retraining or fine-tuning.\n- Experimental results are strong on two benchmarks, VidHalluc and EventHallusion, and across multiple architectures including Qwen and VideoLLaMA.\n\nWeaknesses:\n- Lack of theoretical justifications: Why is hallucination more correlated with temporal variations than task types? Why is injecting offsets into certain layers or attention heads effective? The approach is largely empirical and needs more theoretical analysis and justification.\n- Limited hallucination types: The paper focuses on temporally induced hallucinations. But hallucinations in VideoLLMs can take many other forms, such as factual errors, counting mistakes, recognition errors, and image-based errors. It is unclear whether this method generalizes to other hallucination modes.\n- Engineering assumptions: The method uses a simple temporal variation classifier to split videos into two categories and applies different offsets. This seems a bit too simplistic and engineering-heavy. The need for an extra model introduces overhead. The paper also does not show how general the classifier is or what the failure cases are.", "was_revised": true, "final_justification": "After considering the rebuttal, I am updating my score to a borderline accept. The authors addressed key concerns by providing clarifications on the role of temporal variation in hallucinations and supporting their claims with additional experiments across diverse datasets and failure scenarios. While the approach remains largely empirical and some design choices (e.g., binary classification, offset injection) are heuristic, the method is simple, training-free, and shows strong empirical gains. The core insigh that temporal variation is a major driver of hallucination in VideoLLMs, is valuable and may inspire future work. Despite remaining limitations, the paper presents a practical and promising contribution.", "rebuttal": ">**Q1: Lack of theoretical justifications: Why is hallucination more correlated with temporal variations than task types? Why is injecting offsets into certain layers or attention heads effective? The approach is largely empirical and needs more theoretical analysis and justification. Can you provide any theoretical analysis or intuitive explanation for why this mechanism reduces hallucinations effectively?**\n\n**A1**: Thank you for your question. Firstly, one of the core challenges for VideoLLMs is effectively capturing and integrating the temporal dynamics of videos. **When temporal information is incomplete or misinterpreted, the model is more prone to generating hallucinations** [1, 2]. Conversely, **task types** often represent a higher-level semantic abstraction that **is largely independent of the underlying video's intrinsic temporal characteristics**. For a given video, the same underlying temporal events can be probed with different task formats. For example, a multiple-choice question can be easily rephrased into a true/false question without altering the video's temporal content.\n\nSecondly, regarding the effectiveness of injecting offset vectors, we can **view the attention heads and layers within a VideoLLM as \"experts\"** that process information at different levels of abstraction. Some of these \"experts\" may be specifically responsible for handling temporal dynamics or integrating information across time, which is consistent with previous findings [3,4]. When these \"experts\" receive ambiguous or inconsistent temporal signals (e.g., due to frame downsampling), they may produce biased internal representations, leading to hallucinations. By calculating and injecting \"offset vectors\", we are essentially **introducing a \"correction signal\" into the internal workspace of these critical \"experts\"**. This signal pushes the activations towards a state closer to the true, non-hallucinated output, thereby correcting the \"latent deviation\" caused by incomplete or misleading temporal information. This has also been demonstrated in previous works [5, 6]. This targeted intervention, combined with our discovery of the modules' sensitivity to temporal variations, enables activation engineering to efficiently and specifically mitigate hallucination problems in VideoLLMs.\n\n1. Kong, Ming, et al. \"MHBench: Demystifying Motion Hallucination in VideoLLMs.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 4. 2025.\n\n2. Rawal, Ruchit, et al. \"ARGUS: Hallucination and Omission Evaluation in Video-LLMs.\" arXiv preprint arXiv:2506.07371 (2025).\n\n3. Park, Yein, et al. \"Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information.\" arXiv preprint arXiv:2502.14258 (2025).\n\n4. Francis, Jiztom Kavalakkatt, and Matthew J. Darr. \"Interpretable AI for Time-Series: Multi-Model Heatmap Fusion with Global Attention and NLP-Generated Explanations.\" arXiv preprint arXiv:2507.00234 (2025).\n\n5. Li, Kenneth, et al. \"Inference-time intervention: Eliciting truthful answers from a language model.\" Advances in Neural Information Processing Systems 36 (2023): 41451-41530.\n\n6. Chen, Junzhe, et al. \"Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n>**Q2: How do you ensure that the offsets generalize to different videos and video types?**\n\n**A2**: Thank you for your question. Firstly, in Section 4.1, we specifically chose ShareGPT4Video as our data source, which **is different from** the sources of the VidHalluc and EventHallusion benchmarks. This was done intentionally, unlike previous work that only drew from benchmark subsets, to test the ability of the obtained offsets to generalize to different video types and to avoid inflated results (detailed explanation can be found in Lines 263-269 of Section 4.1).\n\nSecondly, we utilized two benchmarks with **different video types**, VidHalluc and EventHallusion. VidHalluc contains a variety of video types such as action and scene, while EventHallusion mainly consists of event-related videos. This further verifies the generalization ability of our method across diverse video types.\n\nFinally, the experimental results in Section 4.3 (Tables 4 and 5) demonstrate that the offsets generated by our method can effectively generalize to different video types.\n\n>**Q3: Why is a binary classification sufficient? What about videos that do not fit cleanly into either category?**\n\n**A3**: Thank you for your question. Firstly, **the binary classification into temporal-invariant and temporal-variant categories is derived from our analysis** in Section 3. At the same time, our experimental results (Tables 2 and 3) also demonstrate the effectiveness of the binary classifier. Therefore, based on the current experimental results, utilizing a temporal binary classifier appears to be sufficient in most cases.\n\nSecondly, as stated in Section 3.1 (Lines 146-149), the definitions of temporal-invariant and temporal-variant are **relatively clear and mutually exclusive**, allowing the vast majority of videos to be categorized into one of these two types. Consequently, we consider this binary classification of videos into temporal-invariant and temporal-variant categories to be reasonable.\n\nOf course, we also acknowledge this limitation in Appendix F (Limitations) of our paper. In future work, we plan to explore more fine-grained classifications.\n\n>**Q4: Are there any known failure cases, and how does the method behave in those situations?**\n\n**A4**: Thank you for your question. We **further analyzed the specific changes in model output when using the Baseline versus our method**, across all subtasks' failure cases on the VidHalluc benchmark with the Qwen2.5-VL-7B model. Specifically, we used GPT-4o to determine if the hallucinations produced by the Baseline and our method's outputs were consistent, and we calculated their hallucination consistency rate. The experimental results are shown in the table below:\n\n|Model|BQA|MCQ|STH|TSH|\n|:-:|:-:|:-:|:-:|:-:|\n|Qwen2.5-VL-7B|95.24%|92.37%|96.13%|95.92%|\n\nAs can be seen, the hallucination consistency rate of our method with the Baseline exceeds $90$\\% across all subtasks' failure cases. This indicates that in the vast majority of failure situations, **our method gracefully degrades to the original model's behavior and does not further exacerbate the hallucinations**.\n\n>**Q5: Are there other types of hallucinations that are as important as or more important than temporal variation (e.g., factual, spatial, or visual hallucinations)?**\n\n**A5**: Thank you for your question. Firstly, in Section 3, our analysis of the VidHalluc benchmark revealed that **temporal variation is the primary cause of hallucination in VideoLLMs**. Even in other types of hallucinations (e.g., action hallucination, which is primarily evaluated by the BQA and MCQ subtasks in VidHalluc), temporal variation remains a significant contributing factor. \n\nFurthermore, our experimental results in Section 4.3 (Tables 4, 5) **provide additional proof of this method's effectiveness on other types of hallucinations**, including EventHallusion, which specifically targets hallucinations about events. \n\nIn summary, our analysis in Section 3 indicates that **temporal variation is one of the most important causes of hallucination** in VideoLLMs. While other types of hallucinations (e.g., factual errors) may also exist, they are often closely related to temporal variations. Therefore, we consider temporal variation to be one of the fundamental types of hallucination.\n\nNaturally, in the future, we will also consider other types of hallucinations and attempt to incorporate them into our activation engineering framework.\n\n>**Q6: How are shallow and deep layers defined in your method? Is this based on layer index or some functional behavior?**\n\n**A6**: Thank you for your question. In the paper, the definitions of \"shallow\" and \"deep\" layers are **simply based on their index**. Layers closer to the input are considered shallow, while those closer to the output are considered deep. We will annotate this on the x-axis in Figures 3, 8, and 9 to provide readers with a clearer understanding."}, {"review_id": "iLLJb2Mqig", "reviewer": "Reviewer_Svjj", "summary": "This paper applies activation engineering to reduce hallucinations in VideoLLMs. The authors propose two variants of activation intervention—editing at attention vs. layer output—and find both exhibit similar effectiveness across different datasets. A key insight is that hallucination sensitivity is strongly correlated with the temporal variation of video inputs rather than task type. To this end, the authors categorize videos into temporal-invariant and temporal-variant types and propose a fully automated pipeline to collect high-quality datasets accordingly. This categorization allows identification of hallucination-sensitive modules, which are then selectively edited to mitigate hallucinations. Experiments on multiple VideoLLM's hallucination benchmark indicates the effectiveness.", "questions": "1. It is easy for LLM's capability to be corrupted after editing. Could you provide the in-task performance for the intervened model after equipping with your approach? Is that any hurt in performance or other model's capabilities?\n2. What is the sample efficiency for training a binary classifier in your case? What is the inference efficiency after equipping with you intervene method?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper uncovers several valuable observations, including: 1) Frame reduction increases hallucination risk. 2)Temporal variation, rather than task type, is a key factor influencing hallucination sensitivity. 3) Editing attention layers performs slightly better than editing layer outputs, consistent with prior findings.\n\n2. The paper extends activation editing to the VideoLLM setting. The method adaptively adjust the intervene by incorporating temporal dynamics via a tailored dataset collection.\n\n3. The manuscript is generally easy to follow and well-motivated.\n\n**Weakness**\n1. While the analysis is thorough, the core technical contribution—applying activation editing to VideoLLMs—builds heavily on existing work without significant methodological advancement in the editing technique itself.\n\n2. The proposed adaptive editing framework involves non-trivial training (e.g., classifier-based module sensitivity detection). Including an analysis of inference-time overhead and classifier sample efficiency would help clarify the practical feasibility of deploying the method.\n\n3. Some recent publicaitons with similar insights should be included in the activation editing part of related works. And they may be more advanced compared to the activation engineering used in this paper.\n\n[1] Spectral Editing of Activations for Large Language Model Alignment, NeurIPS 2024.\n\n[2] AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models, ICLR 2025.\n\n[3] Representation Surgery: Theory and Practice of Affine Steering, ICML 2024.\n\nThese methods introduce new perspectives—spectral, geometric, or constrained projections—that may offer stronger guarantees or improved efficiency and should be discussed in the context of this work.", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1: While the analysis is thorough, the core technical contribution—applying activation editing to VideoLLMs—builds heavily on existing work without significant methodological advancement in the editing technique itself.**\n\n**A1**: Thank you for your question. Firstly, due to modality differences, existing activation editing methods from the image domain cannot be directly applied to the video domain. One of the key contributions of our work is to **identify temporal variation as a crucial factor influencing hallucinations in VideoLLMs**. Only by uncovering this key factor can activation editing methods be effectively transferred to the video domain.\n\nSecondly, while the activation editing method we used in Section 4 is not entirely new, we are **the first to systematically apply it to VideoLLMs to mitigate model hallucinations**, conducting comprehensive experiments and analysis.\n\nFurthermore, based on our analysis, we **adapted the method to the specific characteristics of the video domain**. Instead of using a unified dataset to calculate offsets, which is common in the image domain, we compute appropriate offsets separately based on the temporal characteristics of videos. We have validated its effectiveness through extensive experiments. This work provides a comprehensive foundation and new insights for future research into the application of activation editing in VideoLLMs.\n\n>**Q2: The proposed adaptive editing framework involves non-trivial training (e.g., classifier-based module sensitivity detection). Including an analysis of inference-time overhead and classifier sample efficiency would help clarify the practical feasibility of deploying the method. What is the sample efficiency for training a binary classifier in your case? What is the inference efficiency after equipping with you intervene method?**\n\n**A2**: Thank you for your suggestion. \n\n**Inference efficiency:** Firstly, as mentioned in Section 4.1 (Lines 298-303), we minimize inference-time overhead by running the classifier and visual encoder in parallel. For more implementation details, please refer to the paper.\n\nSecondly, we further **evaluated the per-sample inference time of both our method and the Baseline** on the VidHalluc benchmark. The results are presented in the table below (unit: seconds):\n\n|Model|Variant|BQA|MCQ|STH|TSH|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|VideoLLaMA2|Baseline|1.44|1.66|1.87|1.92|\n||Ours|1.44|1.66|1.86|1.93|\n|Qwen2.5-VL|Baseline|0.99|1.00|1.35|1.10|\n||Ours|0.99|1.00|1.34|1.10|\n|Video-LLaVA|Baseline|2.43|2.49|3.18|3.86|\n||Ours|2.43|2.49|3.19|3.88|\n\nAs can be seen, the per-sample inference time of our method on the VidHalluc benchmark **is comparable to** that of the Baseline, demonstrating the effectiveness of our parallel setup.\n\n**Classifier sample efficiency:** Firstly, as mentioned in Section 4.1 (Lines 293-297), we sampled $400$ instances from each category of the dataset to train the classifier. This is already a relatively small sample size in the current VideoLLM field.\n\nSecondly, we **further tested the classifier's sample efficiency by reducing the number of training samples**. Specifically, we trained with $100$, $200$, $300$, and $400$ samples respectively, and recorded the classification accuracy of the classifier on the VidHalluc benchmark for each quantity. The experimental results are shown in the table below:\n\n|Sample Size|BQA|MCQ|STH|TSH|\n|:-:|:-:|:-:|:-:|:-:|\n|100|90.17%|89.63%|88.09%|88.83%|\n|200|92.13%|91.47%|91.01%|90.33%|\n|300|94.26%|95.39%|92.58%|92.50%|\n|400|97.63%|97.47%|94.83%|94.17%|\n\nIt can be observed that as the number of samples increases, the classifier's accuracy gradually improves. However, **even with only $100$ training samples, the classifier's accuracy still exceeds $88$\\%**. This demonstrates that our classifier possesses high sample efficiency.\n\n>**Q3: Some recent publicaitons with similar insights should be included in the activation editing part of related works. And they may be more advanced compared to the activation engineering used in this paper. These methods introduce new perspectives—spectral, geometric, or constrained projections—that may offer stronger guarantees or improved efficiency and should be discussed in the context of this work.**\n\n**A3**: Thank you for your suggestion. We will add these latest research findings to the related works section in our revised paper and discuss their similarities, differences compared to our method. Specifically, we **will add the following content**:\n\n```\nRecent works have explored activation editing techniques from spectral, geometric, and constrained perspectives. Spectral Editing of Activations (SEA) [1] leverages SVD-based inference-time projections on cross-covariance matrices, achieving data-efficient edits without training. AlphaEdit [2] introduces null-space constrained weight updates, providing theoretical guarantees against catastrophic forgetting during sequential edits. Singh et al. [3] formulates steering transformations as optimal affine mappings over hidden states, effectively addressing biases and toxic outputs. Unlike these approaches, our method dynamically identifies hallucination-sensitive modules in VideoLLMs via temporal-aware activation editing, adapting inference-time edits specifically to temporal variation characteristics in video tasks.\n\n1. Qiu, Yifu, et al. \"Spectral editing of activations for large language model alignment.\" Advances in Neural Information Processing Systems 37 (2024): 56958-56987.\n\n2. Fang, Junfeng, et al. \"AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models.\" The Thirteenth International Conference on Learning Representations.\n\n3. Singh, Shashwat, et al. \"Representation surgery: theory and practice of affine steering.\" Proceedings of the 41st International Conference on Machine Learning. 2024.\n```\n\n>**Q4: It is easy for LLM's capability to be corrupted after editing. Could you provide the in-task performance for the intervened model after equipping with your approach? Is that any hurt in performance or other model's capabilities?**\n\n**A4**: Thank you for your question. Firstly, the STH subtask within the VidHalluc benchmark includes a description task, and the performance improvement of our method on this task initially demonstrates that our approach does not harm the model's capabilities.\n\nSecondly, we **conducted further evaluation on the VideoMMMU benchmark**, a multi-modal and multi-disciplinary video benchmark that evaluates VideoLLMs' knowledge acquisition capability from educational videos, and the results are shown in the table below:\n\n|Model|Variant|Perception|Comprehension|Adaptation|Overall|\n|:-:|:-:|:-:|:-:|:-:|:-:|\n|VideoLLaMA2|Baseline|59.83|46.12|32.25|46.07|\n||Ours|59.47|46.24|32.25|45.99|\n|Qwen2.5-VL|Baseline|57.29|42.93|37.16|45.79|\n||Ours|57.46|42.57|36.98|45.67|\n|Video-LLaVA|Baseline|39.82|31.67|30.18|33.89|\n||Ours|39.82|31.59|31.09|34.17|\n\nAs can be seen, **the performance of our method on all subtasks of the VideoMMMU benchmark is comparable to the Baseline**, demonstrating that our approach does not impair the model's capabilities."}, {"review_id": "D9SZ4mKVUE", "reviewer": "Reviewer_DqVL", "summary": "The authors present their new discovery that the model's sensitivity to hallucinations depends on the temporal feature changes of the video, regardless of the task type. Based on this discovery, the authors propose a temporal-aware activation engineering framework that can identify the temporal categories of the video based on a lightweight classifier, and then add corresponding offset vectors for different categories in the inference phase to suppress hallucinations.", "questions": "1. Consider giving a more detailed description of the sample screening process, such as the specific prompt of GPT-4o, and normal and hallucination-inducing prompts, to enhance the reproducibility of the article.\n\n2. The article mainly uses quantitative accuracy improvement as an evaluation indicator. It is recommended to add specific examples to illustrate the specific details of reducing hallucinations, such as the comparison of model output before and after injection of the offset vector.\n\n3. To enhance the reproducibility of the method, it is recommended to give the value range or step length of the injection weight and the ablation experimental results of the Top-K attention head selection number to clarify the impact of each hyperparameter on model performance in the appendix.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1. This article has rigorous logic and clear structure.\n\n2. The experimental details are rich and complete. Experiments are conducted on two datasets and three models, and the effectiveness of the method proposed by the author is demonstrated.\n\n3. This method only runs lightweight classifiers in parallel during the inference phase, selects and injects offset vectors, and does not significantly increase latency.\n\nWeaknesses：\n\n1. Although the author divides video input into two categories: \"unchanged timing\" and \"mutable timing\", this dichotomy is relatively rough and cannot cover diverse timing characteristics such as scene switching, which may lead to insufficient adaptability of activation engineering on different types of videos.\n\n2. The authors identified key hyperparameters such as injection weights and the number of Top-K attention heads through grid search, but did not evaluate the impact of these parameters on the model hallucination performance within a reasonable fluctuation range, nor did their sensitivity be explained.", "was_revised": true, "final_justification": "I would like to thank the authors for the rebuttal, which has somehow tackled weakness. I will retain my original score.\n\nBest regards,", "rebuttal": ">**Q1: Although the author divides video input into two categories: \"unchanged timing\" and \"mutable timing\", this dichotomy is relatively rough and cannot cover diverse timing characteristics such as scene switching, which may lead to insufficient adaptability of activation engineering on different types of videos.**\n\n**A1**: Thank you for your concern. Firstly, **the binary classification into temporal-invariant and temporal-variant categories is derived from our analysis** in Section 3. At the same time, our experimental results (Tables 2 and 3) also demonstrate the effectiveness of the binary classifier. Therefore, based on the current experimental results, utilizing a temporal binary classifier appears to be sufficient in most cases. \n\nSecondly, as stated in Section 3.1 (Lines 146-149), the definitions of temporal-invariant and temporal-variant are **relatively clear and mutually exclusive**, allowing the vast majority of videos to be categorized into one of these two types. For example, scene switching can be regarded as a type of temporal variation. Consequently, we consider this binary classification of videos into temporal-invariant and temporal-variant categories to be reasonable. \n\nOf course, we also acknowledge this limitation in Appendix F (Limitations) of our paper. In future work, we plan to explore more fine-grained classifications.\n\n>**Q2: The authors identified key hyperparameters such as injection weights and the number of Top-K attention heads through grid search, but did not evaluate the impact of these parameters on the model hallucination performance within a reasonable fluctuation range, nor did their sensitivity be explained. To enhance the reproducibility of the method, it is recommended to give the value range or step length of the injection weight and the ablation experimental results of the Top-K attention head selection number to clarify the impact of each hyperparameter on model performance in the appendix.**\n\n**A2**: Thank you for your suggestion. We further **conducted an ablation study** on the value range of the injection weight ($\\alpha$) and the number of Top-K attention heads selected. Specifically, following the grid search settings in the paper, we performed a grid search for the injection weight in the range $[8, 16, 24, 32]$ and for the Top-K attention head selection number in the range $[32, 64, 128, 256]$, using the VideoLLaMA2 model on the VidHalluc benchmark. The experimental results are shown in the table below (using *Overall* as the evaluation metric):\n\n|$\\alpha$ Range\\Top-K Attention Head Selection Number|32|64|128|256|\n|:-:|:-:|:-:|:-:|:-:|\n|8|71.25|71.56|71.94|71.47|\n|16|71.28|71.79|72.08|71.63|\n|24|71.72|71.98|72.17|71.84|\n|32|71.74|72.03|72.49|71.96|\n\nAs can be seen, while the injection weight ($\\alpha$) and the number of Top-K attention heads do influence model performance, **the sensitivity is not drastic**. Within a reasonable range, model performance generally improves as both the injection weight and the number of selected Top-K attention heads increase. However, when more attention heads ($256$) that are less sensitive to hallucinations are included, there is a slight decrease in performance. In practical applications, an accuracy threshold strategy can also be adopted to select attention heads, further enhancing the robustness of the model. We **will add these findings to Appendix E** to demonstrate the robustness of our approach.\n\n>**Q3: Consider giving a more detailed description of the sample screening process, such as the specific prompt of GPT-4o, and normal and hallucination-inducing prompts, to enhance the reproducibility of the article.**\n\n**A3**: Thank you for your suggestion. Firstly, we have already presented the specific content of the prompts we used in Figure 7 of Appendix C.2, and we guide readers to it in Section 4.1 (Line 284).\n\nSecondly, in Section 3.2, we have described our normal and hallucination-inducing prompts in detail, specifically the prompt designed to induce hallucination through frame downsampling. We **will add the following content** to Section 4.2's Implementation details to enhance reproducibility:\n\n*\"Consistent with Section 3, we use concatenating the original video and the question as the normal prompt $x$, and use concatenating the 4x downsampled video and the question as the hallucination-inducing prompt $x_h$.\"*\n\n>**Q4: The article mainly uses quantitative accuracy improvement as an evaluation indicator. It is recommended to add specific examples to illustrate the specific details of reducing hallucinations, such as the comparison of model output before and after injection of the offset vector.**\n\n**A4**: Thank you for your suggestion. We will **add some concrete examples to Appendix E** to illustrate how our method reduces hallucinations. Specifically, since most subtasks are selection/classification tasks, we will **focus on providing specific examples from the STH's description task**. We will demonstrate the specific changes in model output after using the baseline and after applying our method, allowing readers to more intuitively understand how our approach mitigates hallucinations. Below is an example (since OpenReview cannot display videos, we only show the ground truth text and the model output text):\n\n```\nGround Truth: Scene change: Yes, Locations: from on a stage to in the woods.\nModel Output (Baseline): Scene change: Yes, Locations: from a dark room to a car.\nModel Output (Ours): Scene change: Yes, Locations: from performing on a stage to being in the woods.\n```\n\nAs can be seen, the Baseline exhibited hallucinations regarding the order of scene changes and scene recognition, while our method effectively reduced these hallucinations."}]

Author rebuttal:

Meta-review: The paper studies activation-editing for VideoLLMs and identifies temporal variation (rather than task type) as a key driver of hallucination. Building on this, it proposes a temporal-aware activation-engineering (TA-AE) framework that (i) classifies videos into temporal-invariant vs. temporal-variant categories with a light binary classifier, and (ii) injects offsets at selected layers/heads accordingly. Experiments on VidHalluC and EventHallusion across models (VideoLLaMA2, Qwen-VL, Video-LLaVA) show consistent reductions in hallucination. During the discussion authors added results on Qwen2.5-VL-72B, comparisons to contrastive decoding methods (CGD, VTI) adapted to video, ablations on heads/weights, qualitative examples, and checks that standard capabilities (VideoMMMU) are not hurt. 

Three reviewers (mR01, 4MRg, DqVL) are positive after rebuttal, citing strong empirical gains, a simple and practical framework, and new large-model and baseline comparisons added during discussion. One reviewer (Svjj) remains unconvinced on novelty relative to recent activation-editing literature, though acknowledges the technical soundness and added analyses.
On balance, the paper offers 
(i) a useful empirical insight—temporal variation is a major contributor to VideoLLM hallucinations, and 
(ii) a practical, training-free method that improves over strong decoding baselines and scales to larger models without harming general capabilities. 
*These match NeurIPS standards for an applied contribution.*
Decision: Accept (poster)