Hallucination-aware Intermediate Representation Edit in Large Vision-Language Models
Authors: Wei Suo
Hanzu Zhang
Lijun Zhang
Ji Ma
PENG WANG
Yanning Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: mitigating hallucination
feature editing
LVLMs
Primary area: foundation or frontier models, including LLMs

Abstract: Large Vision-Language Models have demonstrated exceptional performance in multimodal reasoning and complex scene understanding. However, these models still face significant hallucination issues, where outputs contradict visual facts. Recent research on hallucination mitigation has focused on retraining methods and Contrastive Decoding (CD) methods. While both methods perform well, retraining methods require substantial training resources, and CD methods introduce dual inference overhead. These factors hinder their practical applicability. To address the above issue, we propose a framework for dynamically detecting hallucination representations and performing hallucination-eliminating edits on these representations. With minimal additional computational cost, we achieve state-of-the-art performance on existing benchmarks. Extensive experiments demonstrate the effectiveness of our approach, highlighting its efficient and robust hallucination elimination capability and its powerful controllability over hallucinations. Code is available at https://github.com/ASGO-MM/HIRE.

Published as a conference paper at ICLR 2026
HALLUCINATION-AVAI INTERMEDIATE REPRESENTA- TION EDIT IN LARGE LISIO-LANGUAAE MODELS
Wei Suo, Hanzu Zhang, Lijun Zhang, Ji Ma, Peng Wang' & Yanning Zhang School Computer Science, Northwestern Polytechnical University, China. {suowei, peng. wang, ynzhang] Saannngggwwwuu.u. edw. [jo]0688, li jijunzhang, ma ji @mai 1. nwp.. edu.co
ABSTRACT
Large Vision- Vision-Language Models have demonstrated exceptional performance in multimodal reasoning and complex scene understanding. However, these models still face significant hallucination issues, where outputs contradict visual facts. Recent research on hallucination mitigation has focused on retraining methods and Contrastive Decoding (CD) methods. While both methods perform well, retraining methods require substantial training resources, and CD methods introduce dual inference overhead. These factors hinder their practical applicability. To address the above issue, we propose framework for dynamically detecting hallucination representations and performing tations. With minimal additional computational cost, we achieve state-of-the-an performance on existing benchmarks. Extensive experiments demonstrate the effectiveness of our approach, highlighting its efficient and robust hallucination elimination capability and its powerful controllability over hallucinations. Code is available https:/giitub.com/AAGGo.//M////
edits on these represen-
INTRODUCTION
Large Vision- Language Models (LVL.Ms) (Bai et al.) 2023b; Dai al.] 2023 Liu et al. 2024b)
Zhu et al.) 2022) have made remarkable advancements in recent years, demonstrating the ability to generate context-aware language outputs based on visual understanding. By integrating visual information into Large Language Models (LLMs), LVLMs have demonstrated strong capabilities in tasks such as multimodal reasoning (Lu et al. 2022) and complex scene understanding (Luo et al.) 2024). However, LVLMs currently face the issue of hallucination, where the generated responses contradict the actual visual content.
Recently, many approaches (Leng et al. 2024 Liu et al. 2024a Yin al. 2024b Zhao al. 2023
have been proposed to mitigate hallucinations in LVLMs, which can be broadly categorized into retraining methods and Contrastive Decoding (CD) methods. As illustrated in Fig (a), retraining methods aim to alleviate ballucinations by constructing specialized datasets that target hallucination phenomena and introducing new training paradigms (Jiang et al. 2024a]] Fu et al., 2024 Lu et al. 2023). In contrast, Fig. (b) illustrates the paradigm of CD methods, which mitigate hallucination during inference. The method can work by contrasting the output token probabilities response with those of weakened variant that is more susceptible to hallucination. CD methods effectively reduce ballucinations without requiring retraining or additional data (Leng et al. 2024
the original
Huo et al. 2024 Manevich & Tsarfaty 2024
Although both methods have shown promising results in hallucination mitigation, they still face some notable limitations: 1) Retraining methods require substantial data collection and computational resources for partial or full model fine-tuning. However, both the high cost of dataset construc- tion and the heavy computational burden (i.e., they need retrain the LVLM) limit the practical applicability of these methods. 2) Contrastive decoding methods, while more efficient in design, introduce high computational overhead. They require forward passes per inference- one for generating the original output and another for producing a deliberately weakened variant prone to
"Corresponding authors.

---

Published as a conference paper at ICLR 2026
Mexsoras: image festures halling
Reyporse: man halding glun wins enjoys ha puzs and posto
glas of wine, knjle and food
Amardadigg pltto wntreeeerien wert weld
Back propegatise
Soan lagts
AA leghtgy 1-
layer
LVLM
layer f
center
Lotal Tareage holling pf
LVLM
LVLM
layer
cctruder
MaxullTed uniiad Atzucatiin
LVLM *
Metse discvibe Bhss emoge dataal
Please disembee
Datra Comttaatiio
Hiex image deank
a. Retraining method
b. Contrastive decoding
c. Our Method
Figure Comparison of mainstream hallucination mitigation paradigms. (a) Retraining- methods: Constructing hallucination-sseciinn datasets and training frameworks. (b) Contrastive decoding methods: Comparing the original probability distribution with a perturbed one. (c) Our
method: Editing intermediate representations of LVLMs.
hallucination-whiie significantly increases latency. Moreover, they uniformly adjust the probability of all tokens without distinguishing whether a token is actually prone to hallucination. For instance, common tokens like "in," "on," or "from" rarely cause such issues. The one-size-fits--e fftts-all adjustment leads to unnecessary computation and may harm output coherence. 3) Finally, hallucinations are not inherently harmful in tasks like creative writing (Jiang et al., 2024b), where appropriate hallucination can enhance expressiveness. However, most existing methods lack the ability to control the degree of hallucination, limiting their applicability in broader use cases where flexible generation is desired.
Recent studies on LLMs (Azaria & Mitchell 2023 Chen et al. 2024b) have shown that model's internal representations encode cues about the truthfulness of statements, enabling hallucination detection without external knowledge. Building on these insights, recent research on LVLMMs (Li al., 2024 Duan et al. 2025) demonstrates that similar techniques can be applied to multimodal settings, where hallucination detection can be effectively performed through simple classifiers trained on intermediate features. This suggests that authentic and hallucinatory features in LVLMs are clearly separable in the latent space. This raises natural question: Can we leverage this inherent separation to effectively eliminate hallucinations?
Based the above discussion. propose HIRE (Hallucination-awal) Intermediate Representation Edit). a feature-editing framework that dynamically detects and mitigates hallucinations at the intermediate representation level. As shown in Fig. (c). instead of costly retraining, propose mitigate hallucinations at the feature level without modifying the LVLM's weights. Specifically, we first introduce module, Editor, that learns to identify and isolate halluce components by modeling both semantic invariance and hallucinatory differences between authentic and hallucinated responses, while preserving the underlying semantic information. Moreover, since uniformly editing all tokens leads to unnecessary computational overhead, we design a lightweight Router to selectively edit only tokens with high hallucination risk. Lastly, enable more flexible hallucination control, we introduce Hallucination Regulator, which allows dynamic control over hallucination levels via simple hyperparameter.
Overall, our main contributions are: (1) We introduce new paradigm for hallucination mitiga- tion, which reduces fabricated content by modifying representations without retraining models or doubling inference costs. (2) We propose new framework. HIRE, that dynamically detects and edits intermediate representations with high hallucination. Meanwhile, our method can control the requirements by adjusting the editing intensity. (3)
degree of hallucinations to adapt to different We validate the effectiveness of the proposed method through extensive experiments and achieve state-of-the. the performance on three benchmarks.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
2.1 LARGE VISION-LANGUAGE MODELS
Building on the success of Large Language Models (LLMs) (Bai et al., 2023a Brown al. 2020
Chiang et al. [023 Touvron et al. 2023) and cross-modal learning (Radford et al. 2021 Dosovit skiy et al. 2020). Large Vision- Vision-Language Models (LVLMs) achieve breakthrough performance by integrating visual perception and language generation capabilities, excelling in tasks such as image captioning (Hossain et al.) 2019), visual question answering (Antol et al. 2015), and multimodal reasoning (Lu et al. 2022). A typical LVLM architecture comprises three core components: visual encoder for hicrarchically image feature extraction like CLIP (Radford et al. 2021); ) a cross-modn alignment module implemented via linear projection layers or advanced architecture like Q-Former (Li et al.) 2023b); a large language model fine-tuned through instruction tuning for context-aware text generation. Despite these advancements, LVLMs inherently suffer from hallucination, where the generated text exhibits semantic inconsistencies with the input visual content.
2.2 MITIGATING HALLUCINATIONS IN LVLMS
Recently, multiple strategies have been proposed to address hallucination in LVLMs, which can be broadly categorized into retraining methods and Contrastive Decoding (CD) methods. Retraining methods retrain LVLMs using carefully constructed datasets and training paradigms specifically designed to address hallucinations. HA-DPO (Zhao et al. 2023 fine-tunes LVLMs using style- consistent hallucination sample pairs to promote non-hale non-hallucinatry outputs. HDPO (Fu et al. 2024) further targets diverse causes of hallucinations by constructing specialized preference pairs for visual distraction, long-context generation, and multimodal conflicts. In contrast, CD methods mitigate hallucinations by comparing the output distributions from the original and perturbed inputs, without requiring any model parameter updates. VCD (Leng et al.) (0244) introduces noise into visual inputs amplify hallucinations and suppresses them by contrasting perturbed and original token distributions. SID (Huo et al. 2024) retains only the least important visual tokens after shallow processing to amplify vision-texx association hallucinations. Unlike previous works, we propose to mitigate hallucinations by detecting and editing hallucinated representations, avoiding the need for heavy training resources and the cost of dual inference.
PRELIMINARY
LVLMss have garnered significant attention due their capacity to combine visual and textual information for generation tasks. These models take both an image and a textual prompt as input. Given an image, it is firstly processed by a visual encoder and a cross- interface. Subsequently, the visual information U and the text-based query g combined and fed into the LLM for further processing. The LLM Po is a parameterized model that maps the input to a probability distribution the next token. Generally, the LLM architecture typically consists of stacked Transformer layers (Vaswani et al.) 2017). each comprising multi- head self-attention and a forwafd network (FFN). After passing through all L layers, the final hidden states are projected into the vocabulary space produce a probability distribution over the next token. The above process can be formulated as:
(e Bet),
where y/ is the t-th generated token, and you represents the previous tokens. However, the generated sentences often suffer from hallucinations, where the output contradicts the real visual input.
4 METHOD
Current methods primarily employ retraining methods (Fu et al. 2024 Zhao et al. 2023 CD methods (Leng et al.) 2024 Wang et al.] (024) to mitigate hallucinations. However, retraining methods suffer from training resource burdens, while CD methods ineur dual computational cost during inference. Moreover, these approaches are incapable of achieving controllable generation of hallucinations. Motivated by the separability of hallucinated and truthful representations in the latent
3

---

Published as a conference paper at ICLR 2026
bicycle rood, with some cars and motorsycles
bicycle pedestrions corrying becipacks the
bicycle
D00 00 000 00
with some
reconstructed activations
motorcycles on the
Edit A
Edit policy
Edit policy H CHAIR: 0.2
CHAIR: 0.4
CHAIR/: 0.8
Decoder
layer
Now Aun
contrastive learning
Aast Newt
FFN
DPO
Enc S
20
Enc H
layer
Router
Editor
semrnntc invariance
hallucinatory difference
Edit Signal
layer
ATTN
0
0|
on
Only layer has Router
positive text activations
negative text activations
LVLM
different visual input
Generate candidate sentences
Q: Please describe this image in detail.
beyydl
bityole the rood,
the road, motoorcycle btc...
samee
Original image
Prompt
captions
mitorryccee
Overview of HIRE. Our framework consists of two key components: the Editor, which learns semantic invariance and hallucinatory difference through contrastive learning, and the Router, which learns efficient editing strategies through DPO.
Figure 2:
space (Li et al. 2024 Duan et al. 2025), we propose feature editing approach HIRE. As shown in Fig HIRE dynamically identifies hallucination-pree -pone components within representations and edits them by projecting features onto low-halluce www-hallucination directions in the representation space.
To eliminate hallucination through editing and identify optimal editing strategies, must address two key challenges: (1) how to determine the direction for obtaining hallucination-reduuee representations, and (2) how to identify which token representations require editing. For the first challenge, we propose constructing high-hallucination and low-hallucination representations and then editing along this divergence direction. For the second challenge, our method autonomously learns effective editing strategies, enhancing the success rate of edits while minimizing ineffective changes. The following sections will detail our approach in terms of model architecture and optimization.
ietttuiinatiio-reddeee representations by analyzing the divergence between
4.1 MODEL STRUCTURE
Editor: Based on the established correlation between hallucination phenomena and attention distri- butions (An et al. 2024 Huo et al. 2024). we identify the attention-layed representations at each transformer layer our editing targets. The further analysis about the effect of editing different layers can found in Appendix D.I However, due to entanglement between representations of hallucinated and non-hallucinated texts (Jiang al. (2024a). direct manipulation may disrupt semantic integrity(Li et al. [203cc] Chen et [024c] /Zhang al. 2024). Inspired by (Zhang et al.) 2024), utilize a dual-encoder autoencoder G that enables hallucination suppression while pre- serving semantics. Specifically, the G2 consists a semantic encoder Exems a hallucinatory encoder Fmat multi-head attention module faun and decoder D. Given the f-th token in attention-layer representations from 1-th layer hu, the input representations are processed by both encoders to yield semantic representations hessem and hallucinatory representations hatbal, which can be formulated as:
bat.scm Esem (ha))
hathaal
Epal (ha).
4

---

Published as a conference paper at ICLR 2026
To ensure correct editing, compute the ceneeiinntionrrdduttin direction ót by averaging token- level differences between authentic and hallucinated representations within the hallucinatory subspace. The semantic and hallucinatory representations are fused and decoded to yield optimized editing direction Au, which is the direction of targeted feature editing. The Au are obtained as follows:
Au = D (hel,ssm + satt (eed, hat.hal + 61)) D (heetsem + Sann (her,sem; hut 8t)),
(3)
where the satn denotes attention fusion of het,sem (serving as query) and hat (serving key and value). Overall, the 6 aims to identify hallucination-reduuee directions that are irrelevant to specific tokens within the hallucinatory subspace, while the Du leverages the dj obtain token-specific hallucination-reduced directions in the original representation space.
To avoid unnecessary editing operations on hallucination-fre representations, we introduce lightweight module, Router Rg. Motivated by the observed layer redundancy in LVLMs, where deeper layers tend to exhibit high similarity (Wu et al., 2023 Suo et al.) all] 2024). as well as by recent findings indicating that shallow layers preserve more informative representations (Wang et al. 2025 Chen al. 2024a), we employ an MLP that takes the first-layer transformer representation h40 as input and produces a binary decision c. This signal determines whether editing is required: if c = L, the Editor is activated for all subsequent layers; otherwise, no editing is performed. Further experiments exploring alternative layer-wise decision strategies are provided in Appendix D.2 With an editing strength a introduced, the editing process can be formulated as follows:
Router.
hh + a Au tha
if c 1, if O,
het.ang
(4)
where meg denotes edited representations, and the a E [-1, controls the direction and intensity of editing. A positive a suppresses hallucination by steering features toward low-hallucination directions, while negative a amplifies them, enabling flexible control over hallucination levels.
4.2 MODEL OPTIMIZATION
The above computation reveals that the Editor Gg and the Router Rg are critical to detecting and correcting ballucination--rlatee features. However, optimizing them challenging due to the lack of labeled training data. Therefore, we introduce contrastive learning and DPO strategy to train the Editor G2 and the Router Rg, respectively. To disentangle hallucinatory patterns from semantic content, we use contrastive learning to optimize the Editor G Specifically, Hisemm (he I (T denotes the number of tokens) should have high similarity to semantically identical representations and low similarity to those with different meanings. Conversely, H hha )= exhibits high similarity within group similarity (hallucinated or authentic), but low cross-group similarity regardless of token semantics. To optimize the Router Rg, we maximize the likelihood ratio between effective edits and ineffective ones by introducing DPO (Rafailov et al. 2023), which learns from pairwise data by increasing the likelihood of preferred responses while suppressing less desirable ones.
4.2.1 DATA CONSTRUCTION
Based on the finding that visual uncertainty amplifies hallucinations in LVLMs (Leng et al. 2024), we obtain authentic and hallucinated representations by pairing identical textual inputs with both intact and visually-
images. The positive sample uses the original image to produce more faithful outputs, while the negative sample uses a noised version of the image, weakening visual grounding and increasing hallucination. The resulting intermediate representations are denoted as H = {h and H, = [h 1 Our framework is also compatible with other hallucination induction techniques; comparative experiments are provided in the Appendix D.3]
To obtain pairwise preference data, we generate N candidate captions for each image by applying different editing action sequences during greedy decoding, where each sentence S, of length T,, has corresponding representations {hor)?=, and editing decision trajectory {c:TT, We quantify hallucination severity using the CHAIR, metric (Rohrbach et al. 2018). The most and least faithful sentences used construct preference pairs ()+ ct and (h .c for the Router training.
5

---

Published as a conference paper at ICLR 2026
4.3 TRAINING PROCESS
We apply contrastive learning to both the semantic and hallucinatory encoders. The semantic encoder aims preserve semantic information by maximizing the similarity between corresponding tokens in positive and negative samples, while minimizing the similarity for non-matching tokens. In contrast, the hallucinatory encoder focuses solely on hallueinatioon-elattee signals and is trained to separate positive and negative samples regardless of their semantic similarity. The loss formulations are defined as follows, where negative samples are formulated similarly:
Editor:
L sl.sem LinfoNCE (hit, (Mil,sem*
Det.sem= H (.sem):
(5)
LIffNNC (hit Hijha H(,mal
where LIsONCE (Oord et al., 2018) encourages the anchor to be closer positive samples than to negatives, hi, sem and h *1 hal denote the semantic and hallucinatory embeddings of token t at layer 1, and Hi Hinh the sets of token embeddings hu. and hi The learned semantic and hallucinatory representations original feature. To ensure both faithful reconstruction and effective editing, a reconstruction loss and an editing loss can be formulated follows, where negative samples are calculated similarly:
fused through multi-heat attention fau and decoded to predict the
rlccoo = MSB(h)) D(htt + facto (aato(ttiissm semm h Lii MSB( (h + farto (hit((hrit,mm 11
(6)
where hit denote the original representations of the token at layer of positive and negative samples. Averaging the sum of four loss terms across tokens and layers yields the final training objective Le-
Router. To guide the model towards optimal editing strategies, we adopt Direct Preference Optimiza- tion (DPO) (Rafailov et al. 2023), which reformulates reward maximization as policy likelihood ranking problem. Unlike standard DPO that requires pairwise comparisons between a learnable policy Ty and reference model Tee.- implementation eliminates the reference model based on recent findings (Meng et al. (2024) demonstrating its removability. The simplification is suitable for training the Router from seratch rather than fine-tuning existing policy. Given paired state-action sequences (h+,ct) (preferred) and ( (non-preferred), the optimization objective becomes:
= -EEhe) [log a (3 (log (h c+) log T T ,]]]]
where &, denotes the Router's training loss, xo represents the learnable policy (Router), a(.) ) the sigmoid function, and 3 = 0.1 serves scaling factor regulating optimization intensity. Finally, the total loss for optimizing both the Editor and the Router can be formulated as:
Listal = Le + fr-
(8)
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Dataset. We evaluate our method on three benchmarks: CHAIR (Rohrbach et al. 2018), POPE (Li al. 2023d), and AMBER (Wang et al.) 2023). CHAIR measures object hallucination in image captions, calculating the proportion of objects mentioned in the text but missing from the ground-trui -ruth annotations. It provides object-level (CHAIR)) and sentence-level (CHAIRS) scores, with lower values indicating fewer hallucinations. We evaluate on 500 images randomly sampled from the MSCOCO (Lin et al.) 2014) dataset. POPE assesses object hallucination in multimodal question answering using three object-sampling strategies (random, popular, adversarial) and binary queries like "Is there a <object>> in the image?". Evaluation based on 9,000 question-anse ansserr pairs from MSCOCO. AMBER evaluates hallucinations in LVLMs, using 1,004 generative instances and 14,216 discriminative instances annotated for existence, attribute, and relation hallucinations, providing a comprehensive framework for hallucination identification and classification..

---

Published as a conference paper at ICLR 2026
Table Evaluation results on the CHAIR benchmark with LLaVA-1.5.
Method
LL et al.
2024b)
InstructBLIPP Dai et al.
2023)
CHAIRs + CHAIR, TFLOPS.
CHAIR: + CHAIR, + TFLOPS)
baseline
51.3
16.8
10.23
51.0
24.2
2.77
Max new tokens set 512
OPuRA et al.
2024
45.2
12.7
47.4
12.9
Huo et al. 2024
44.2
12.2
42.3
12.4
ICD Wang et al 2024
47.4
13.9
20.63
46.3
15.3
5.64
VCDi
2024
46.8
13.2
20.46
44.0
13.6
5.61
M3ID
2024
48.3
13.5
20.71
45.2
13.9
5.70
AVISC Huo ct al. 2024
45.2
13.4
20.08
43.9
13.5
5.63
Octopus et 2025
39.2
11.1
21.39
40.6
12.1
5.87
Projectaway
42.0
12.2
43,8
12.5
ViI al. 2025
35.8
11.1
43.4
11.8
(uISE
30.2
9.7
11.81
39.0
11.5
3,45
Max tokens set 64
baseline
20.4
6.2
8.91
25,4
8.2
1.89
M3ID+DPO Favero et al. 2024
13.5
5.7
Nullu Yang el al. 2025)
17.0
5.9
Ours (HIRE)
15.2
5,4
9.18
14.8
5.4
2.32
Implementation Details. We construct the training samples for both the Editor and the Router using the training split of the MSCOCO dataset (Lin et al.) (2144). Specifically, for the Editor, training performed for epochs using a subset of 2,000 samples. For the Router module, we train for 100 epochs using 8,000 samples. Both modules employ the SGD optimizer (Bottou 2010) with initial learning rate of x 10-2, followed by a Cosine Annealing scheduler reaching a minimum learning rate of 10-3 For the Router, we specifically set the group size 10. We report the results of our method with consistent edit strength a set to 1. All experiments were performed on four 3090 GPUs. Further discussions on the different training configurations are provided Appendix D.5
5.2 RESULTS ON BENCHMARKS
We conduct experiments on LLa 1L [Liu et al. (024b) and InstructBLIIP (Dai et al. 2023) across three widely used benchmarks: CHAIR (Rohrbach et al. a 2018). POPE (Li et al. 2023d), ) and AMBER
(Wang et al. 2023), with related results from (Suo et al. 2025 Xing et al. 2024 Huo al.) 2024 Jiang et al. 2024a Liu et al. Jo25ggiang et al.). Overall, our method consistently outperforms existing approaches while maintaining efficient inference. As shown in Table I it reduces sentence-level and instance-level hallucinations by 440% and ~50% respectively on both LLLVA-115 and InstructBLIP in the long description scene (max new tokens set to 512). Additionally, in the short description scene (max new tokens set to 64), our method also demonstrates reliable hallucination mitigation capabilities. Table shows that our method achieves improvements of 1.48/3.79 and 0.48/1.99 over the state-of-the-art method Octopus (Suo et al. 2025) on the two models on the accuracy and Fl score, demonstrating superior performance in discriminative tasks. As shown in Table 3 our method achieves the highest AMBER scores-ar an indicator that averages performance across generative and discriminative tasks. Compared the baseline, it significantly improves AMBER scores by 7.54 on 1L. and 6.38 on InstructBIIIP In addition, benefiting from the lightweight nature of our editing strategy and the selective token editing enabled by the Router, our method introduces minimal inference overhead. In summary, and discriminative tasks, with only a small additional overhead. Additional analysis and results on generalization and stability be found in Appendix D.6 and Appendix D.7
approach effectively mitigates hallucination in both generative
5.3 HALLUCINATION REGULATOR
Recognizing that hallucinations can have positive effects in certain scenarios like creative writing (Tiang al. 2024b), is important have controllable generation of hallucinations. However,
7

---

Published as a conference paper at ICLR 2026
Table Evaluation results on the POPE benchmark on the MS COCO datasets with LLa and InStructBLIP.
Method
Random
Popular
Adversarial
ALL
TFLOPs/
Acc.
F1
Acc.
F1
Acc.
F1
Acc.
F1
LLaVA-L.5-7B
83.77
81.94
82.57
80.86
79.77
78.47
82.04
80.42
8.13
Referenced
Results
+HACL
88.59
88.70
87.84
87.36
86.54
85.73
87.66
87.26
+VTI
89.50
88.89
87.36
86.69
82.57
82.11
86.48
85.90
Comparable Results
+ICD
87.51
83.28
83.15
83.91
79.13
80.41
83.26
82.53
+Con Vis
84.70
83.20
81.10
83.00
+OPERA
84,40
83,40
81.20
83.00
+VCD
85,43
83.99
83.17
81.94
80.27
79.49
82.96
81.81
16.26
+M3ID
86.13
81.85
82.07
80.77
79.50
78.15
82.57
80.26
16.26
+AVISC
84.67
82.21
83.67
81.27
81.83
79.55
83.39
81.01
16.26
+Octopus
87.51
85.40
85.20
84.19
82.22
81.44
85.79
83.44
16.34
+Ours
90.37
90.25
87.70
87.86
83.73
83.56
87.27
87.23
10.62
InstructBLIP
81.53
81.19
78.47
78.75
77.43
78.00
79.14
79.31
1.08
+ICD
84,36
83.82
77.88
78.70
75.17
77.23
79.14
79.92
+OPERA
84.57
83.74
78.24
79.15
74.59
76.33
79.13
79.74
+VCD
82.03
81.56
79.13
79.20
77.23
77.72
79.46
79.49
2.16
+M3ID
82.33
81.53
80.90
80.42
78.53
78.49
80.59
80.15
2.16
+AVISC
86.03
84.41
84.27
82.77
81.83
80.67
84.04
82.62
2.16
+Octopus
86.63
85.30
84.90
83.55
82.83
81.43
84.79
83.43
2.27
+Ours
90.30
89.83
84.03
84.25
81.47
82.17
85.27
85.42
1.25
Table 3: Evaluation results on the AMBER benchmark with LLaV5-1.5 and InstructBLIP.
Generative
Discriminative
Model
Setting
AMBER
CHAIR
Covert
Hal
Cogl
TFLOPs.,
Acc.1
F1f
TFLOP
baseline
44.5
31.0
2.2
9.89
67.00
71.10
8.07
81.58
VCD
6.7
46.5
27.8
2.0
19.55
67.30
71.10
16.14
82.20
LLaVA-1.5
M3ID
6.0
48.9
26.0
1.5
19.81
67.25
70.90
16.14
82.25
AVISC
6.3
46.6
25.6
2.0
19.45
70.70
75.45
16.14
84.60
Octopus
49.2
23.4
1.2
20.48
76.70
82.70
16.35
88,95
Ours
4.6
49.9
20.4
1.5
11.73
79.20
82.83
10.58
89.12
baseline
8.4
46.4
31.1
2.6
2.93
68.20
74.60
1.04
83.10
VCD
7.6
47.7
29.9
2.2
6.11
69.65
75.90
2.08
84.15
InstructBLIIP
M3ID
6.9
47.2
27.5
2.2
5.87
69.05
75.25
2.08
84,20
AVISC
46.7
28.0
6.09
72.60
78.60
2.08
85,95
Octopus
6.1
48.5
22,2
1.3
6.47
74.00
79.70
2.19
86.80
Ours
5.3
49.3
23.8
1.8
3.57
78.34
84.26
1.21
89.48
existing methods exhibit unstable hallucination control capabilities. Detailed experimental results are provided in Appendix DD8 Hence, conduct experiments on the edit strength hyperparameter in formulation to investigate critical role in hallucination control for L.V.Ms. As shown in Figure we conduct experiments with different values of ranging from -0.7 to 1.0. It can be found that positive a value can mitigate hallucination at both sentence-level and object-level granularities, and the reduction of hallucination increases with higher values. Conversely, negative a values demonstrate a proportional amplification effect on hallucination rates, with lower values inducing stronger hallucinatory responses. From the figure, it can be observed that method has excellent controllability in hallucination generation.
5.4
ABLATION STUDY
8

---

Published as a conference paper at ICLR 2026
Table Ablation study on the impact of encoder architecture and the Router module. Results show the effects on hallucination mitigation performance and computational cost.
Semantic encoder
Hallucinatory encoder
Router
C
TFLOPs|
51.3
16.8
10.23
37.0
12.2
48.6
13.0
30.4
9.5
16.23
30.2
9.7
11.81
LLaVA- imnage
Oare The imsee feaan bubry sittng un Harket on tae gaas smiliz hobing a cell ploat. The baly is wesrita only dape and esmorded by blankii. basy sypears enjoyiz axseees, poselly in park The is placed estertinmnnt communicatoo purposes.
enjoying oendor eestrvemure bockgroand. thrme well. Additionally there to lottls placed near the buby,, polsiity belonging baby other viatom the part.
Assurn UawA--1 One
powittl
beby, possiily
Q: Plaaee descrie this
image danill
Qt thore pion image?
LLaVA-1 S: imuge captares alarchorde perforsing&
purk,
mnddle
dhowcciing thiir scills Them wwond other the
than warcing the
park, watching then claser stanboorder while others
sundbaax det's pes Rartaazee The skate palk features varkey of ramps and stractures, peoviding an exciting and dialleninng cevinoment for akanchordes lo practiee their mecko The seene a livels with people rejoying destling straegheer dondhonders improsive diglly tolon.
LLaV--155 Oure Na
skatcheender's are funher away. banch baakggourdd providing pluce for spectatura stt and enjoy the okourhonting oegnnnngan setiin.
Qr la thert in the imagd??
Q Pirao desorhe this trage desel.
Figure 4: Some examples of generative and discriminative tasks on the MSCOCO dataset, with hallucinated content highlighted in red and newly added correct content displayed in green.
We conduct ablation studies on CHAIR using LLaVA- 1.5 to validate each module's contribution. As shown in Table the original model's results are provided first. Adding only the semantic encoder hallucina- tory encoder individually yields limited gains. Com- bining both achieves the lowest hallucination rate, demonstrating their complementary roles. Further- more, integrating the Router reduces computational cost by ~30% without performance loss, underscor- ing its efficiency. These results confirm the necessity of both encoders for effective hallucination mitigation and the Router's role in maintaining efficiency.
CHESIR Mariis Shreeth
613
618
Figure 3: Control hallucination via a..
5.5 QUALITATIVE EVALUATION
To better illustrate the effectiveness of method, Fig. presents representative examples from both generative tasks and discriminative tasks on the MSCOCO dataset. Hallucinated content in the figure highlighted in red. It can be observed that our method eliminates hallucinations and provides more accurate interpretation of the image. More results are provided in the Appendix
To demonstrate our method's effectiveness in sup- pressing hallucination within the representation space, Figure 5 compares distributions of non- hallucinated (green), hallucinated (red), and edited (blue) feature. Non-hallucinated features are ex- tracted using the image and its ground-truth caption. Following Wang et al. (2024). we introduce halluci- nation into features via disturbed prompt and hal- lucinated caption to avoid image perturbation bias. Edited representations result from applying Ed-
Figure 5: Distribution of original, halluci- nated, and edited representations.

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation results on the CHAIR
Table 6: Evaluation results on the CHAIR
with Max New Tokens set to 512.
with Max New Tokens set to 64.
Method
C
CHAIR,
Method
CHAIRs
CHAIR,
LLaVA-
51.3
16.8
LLaVA-L5--BB
20.4
6.2
Prrojectaway
43.8
12.5
+Nullu
17.0
5.9
+Ours
30.2
9.7
+Ours
15.2
5.4
+Ours & Projectaway
27.6
8.3
+Ours & Nullu
13.2
4.6
Table 7: Comparison of general capabilities on MME and SEED-Bench.
Method
MME t
SEED f
LLaVA-1.5 7B
1751.64
64.3
+Ours
1751.99
63.8
itor hallucinated ones. The left subfigure shows clear separation between hallucinated and non-hallucinated clusters in the hallucinatory subspace. On the right, edited representations shift toward the non-hallueinated cluster and exhibit overlap and converge, confirming that our editing effectively reduces hallucination.
5.6 COMPATIBILITY WITH STEERING-BASED METHODS
Several steering- beased methods exist for hallucination mitigation. For example, Nullu (Ynng et al. 2025) operates by editing model weights and Projectaway (Jiang et al.) removes hallucinations from image features. In contrast, our method focuses on the textual feature space and further making fine-grained editing decisions per token. Hence, approach can be integrated with other steering- based methods, offering a composite solution. Following the experimental settings of these methods, evaluate the combined performance using the CHAIR metric across different max token constraints. As shown Table 5 and Table 66 our method can be combined with steering-hased methods yield stronger hallucination suppression.
5.7 EVALUATION ON GENERAL CAPABILITIES
We evaluate the general capabilities of the model equipped HIRE, using LLaVA-115-77 repre- sentative model on two challenging benchmarks: MME (Yin et al. 2024a) and SERD-Bench (Li al. 2023a).As shown in Table 7 can be observed that our method preserves the model's general capabilities on these benchmarks.
CONCLUSION AND LIMITATIONS
In this paper, we propose novel adaptive feature-editing framework that dynamically detects and calibrates hidden-layer activations to either suppress enhance hallucinations, forming flexible, input-aware workflows. Our method operates in a single inference pass without updating parameters of LVLMs, offering exceptional efficiency while maintaining strong performance. It is also easily extendable to tasks requiring flexible control over hallucinations. We expect that this work will provide general, practical paradigm for mitigating hallucination across diverse scenarios.
Despite these advantages, we note a limitation of the current framework. Our method employs all tokens and layers for training, process that be susceptible noise. Training selectively on the most critical hidden states presents a significant opportunity for enhancing data efficiency. This work, initial step, aimed primarily validating the feasibility of the proposed editing paradigm. We will explore the aforementioned direction in future work.
ACKNOWLEDGEMENT
This work is supported in part by the Natural Science Basic Research Program of Shaanxi Province (No. -
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, Qian Ying Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention. arXiv preprint arXiv:2406.1277 2024.
Stanislaw Antol. Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrencee Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433 2015.
Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it's lying- In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Compu- tational Linguistics: EMNLP 2023, pp. 967-976, Singapore, December 2023, Association for Computational Linguistics. doi: 10. 10111866//1/// 1/20232222222 findings-emmll.66. URL https
//aclanchology. hwwlaanholoog..org/222222222 org/2023, ftt.:23...iiii.. Findings emnlp. 687
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023b. URL http:////.wwww.....aa//a/ao/ee// /bb 2308 12966
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 13Xiv22502.13923, 2025.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010: 19th International Conference on Computational StatisticsParis France, August 22-27, 2010 Keynote, Invited and Contributed Papers, pp. 177-186. Springer, 2010.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
2020.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024a.
Nuo Chen, Ning Wu, Shining Liang, Ming Gong, Linjun Shou, Dongmei Zhang, and Jia Li. Is bigger and deeper always better? probing llama across scales and layers, 2024b. URL https
//arxiv. oraiii.orrg//baa.. 2312. 04333
Zhongzhi Chen, Xingwu Sun, Xianfeng Jiao, Fengzong Lian, Zhanhui Kang, Di Wang, and Chengzhong Xu. Truth forest: Toward multi-scale truthfulness in large language models through intervention without tuning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 20967-20974, 2024c.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph F. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https
//mmys..rg//balll..m.. 2003 03 30- vicuna/
Wenliang Dai, Jumnan Li. Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards ceneral-purpose vision-language models with instruction tuning, 2023.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint PrXi::2010.11920 2020.
NR Draper. Applied regression analysis. McGraw-Hill. Inc, 1998.

---

Published as a conference paper at ICLR 2026
Jinhao Duan, Fei Kong, Hao Cheng, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Truthprint: Mitigating Ivlm object hallucination via latent truthful-guided pre-intervention. arXiv preprint arXiv:2503. 2025.
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary. Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. Multi-modal hallucination control by visual information grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition, pp. 14303-14312, 2024.
Yuhan Fu, Ruobing Xie, Xingwu Sun, Zhanhui Kang, and Xirong Li. Mitigating hallucination in multimodal large language model via hallucination-targeee direct preference optimization. arXiv preprint arXiv:241 I. 10436, 2024.
MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CsUR), 51(6):1-36, 2019.
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang. Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and vetrospection-alllcatie on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
In Proceedings of the IEEFJCVF Conference
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Fushuo Huo, Wenchao Xu, Zhong Zhang, Haozhao Wang. Zhicheng Chen, and Peilin Zhao. Self- introspective decoding: Alleviating hallucinations for large vision- saiiu-la models. arXiv preprint 0:Xi::2408.02032, 2024.
Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27036-27046, 2024a.
Nicholas Jiang, Anish Kachinthaya, Suzanne Petryk, and Yossi Gandelsman. Interpreting and editing vision-language representations to mitigate hallucinations. In The Thirteenth International Conference on Learning Representations.
Xuhui Jiang, Yuxing Tian, Fengrui Hua, Chengjin Xu, Yuanzhuo Wang, and Jian Guo. A survey on large language model hallucination via a creativity perspective. arXiv preprint 47X602240..06647, 2024b.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vaniunel models through visual contrastive decod- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13872-13882, 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Pro- ceedings of Machine Learning Research, pp. 19730- 19742. PMLR, 23-29 Jul 2023b. URL https: https://proceedi http:///oooeeeeiin.cc ml w.w..ppesssee.. pevess/v202// 23g html
12

---

Published as a conference paper at ICLR 2026
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-tinee intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451-41558 2023c.
Qing Li, Jiahui Geng, Chenyang Lyu, Derui Zhu, Maxim Panov, and Fakhri Karray, Reference-free hallucination detection for large vision- languala models, 2024. URL
abs/ 22..22200000557
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXi::2305. 10055 2023d.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer vision- ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pp. 740-755. Springer. 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination large multi-modal models via robust instruction tuning. In ICLR, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296 26306, 2024b.
Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in large vision-language models via latent space steering. In The Thirteenth International Conference on Learning Representations, 2025.
Jaaying Lu, Jinmeng Rao, Kezhen Chen, Xiaoyuan Guo, Yawen Zhang, Baochen Sun, Carl Yang, and Jie Yang. Evaluation and enhancement of semantic grounding in large vision-language models. arXiv preprint arXiv:2309.0400 2023.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Sheng Luo, Wei Chen, Wanxin Tian, Rui Liu, Luanxuan Hou, Xiubao Zhang, Haifeng Shen, Ruiqi Wu, Shuyi Geng, Yi Zhou, et al. Delving into multi-modal multi-task foundation models for road scene understanding: From learning paradigm perspectives. IEEE Transactions on Intelligent Vehicles, 2024.
Avshalom Manevich and Reut Tsarfaty. Mitigating hallucinations in large vision-language models (Ivlms) via kunguage-contraste decoding (led). In Findings of the Association for Computational Linguistics ACL. 2024, pp. 6008-600 0008-602,0400 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. Advances in Neural Information Processing Systems,
2024.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- 7748-8763.. PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-555 53741, 2023.
13

---

Published as a conference paper at ICLR 2026
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint arXiv:1809.02/56. 2018.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Wei Suo, Ji Ma, Mengyang Sun, Lin Yuanbo Wu, Peng Wang, and Yanning Zhang. Pruning all- rounder: Rethinking and improving inference efficiency for large vision language models. arXiv preprint arXiv:2412.06445 2024.
Wei Suo, Lijun Zhang, Mengyang Sun, Lin Yuanbo Wu, Peng Wang, and Yanning Zhang, Octopus: Alleviating hallucination via dynamic contrastive decoding. arXiv preprint arXiv:2503.000 200360031,, 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23X2::2302.3971, 2023,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An llm-free multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:2311.07397 2023.
Sudong Wang, Yunjian Zhang, Yao Zhu, Jianing Li, Zizhe Wang, Yanwei Liu, and Xiangyang Ji. Towards understanding how knowledge evolves in large vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29858- 0158822988,..8 2025.
Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv: 1403. 2024.
Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. Don't miss the forest for the trees: Attentional vision calibration for large vision language models. arXiv preprint arXiv:2405.1788 1884405.7820... 2024.
Qiong Wu, Wei Yu, Yiyi Zhou, Shubin Huang. Xiaoshuai Sun, and Rongrong Ji. Parameter and computation efficient transfer learning for vision languala pre-trained models. Advances in Neural Information Processing Systems, 66:41034-4105 2023.
Yun Xing, Yiheng Li, Ivan Laptev, and Shijian Lu. Mitigating object hallucination via concentric causal attention. Advances in Neural Information Processing Systems, 37:92012-92033. 2024.
Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 14635-14644 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 1 1112))mwae403, 2024a.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105, 2024b.
Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alleviating hallucinations by editing large language models in truthful space. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8908 8908-8949, 2024.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucimations: Enhancing Ivims through hallucination-awwe aware direct preference optimization. arXiv preprint arXiv:2311. 1..839. 16839, 2023.
14

---

Published as a conference paper at ICLR 2026
Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14288 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models, arXiv preprint arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2026
A DETAILED EXPERIMENTAL SETTINGS
A.1 VISUAL PERTURBATION METHODS
We follow VCD (Leng al. 2024) to apply visual perturbations by introducing noise to images. For LLaVA-1.5, set the noise step to its maximum value of 999, which lies within its supported range of 0-999, For InstructBLIP, considering that the Q-former module relies on interactions between textual and visual imputs, excessively corrupted images introduce unwanted degradation into the representations. To mitigate this, we limit the noise step to 600.
B LICENSE OF ASSETS
LLaVA-1.5 (Liu et al. 2024b) is available under the Apache-2.0 License and InstructBIIIP (Dai et al. 2023) is available under BSD-3-Clause License. CHAIR is under the BSE 2-Clause License, POPE is available under the MIT License and AMBER is available under Apache-2.0 License.
C EVALUATION METRIC
AMBER. We follow the experimental setup in (Suo et al. 2025). In the generative task of AMBER, we report four metrics: CHAIR, Cover, Hal, and Cog.
CHAIR measures the proportion of objects mentioned in the generated sentences but not present in the ground-truie trund labels. Specifically, given the list of generated objects Gouj = {obb] obj2 ..... obj and the list of annotated objects Aohi = {objí' objı. ..... .bbff,.,......... CHAIR is calculated by the following formula:
Len(G (Goby n Aobj)
CHAIR
Gobs
Cover measures the ratio between the correctly mentioned objects in responses and the total num of objects in the annotations:
len(Gabj n Aobs) len(Aot))
Cover
(10)
Hal indicates whether a response contains hallucination. For each response, Hal is defined as I when the CHAIR score is greater than 0, and 0 otherwise, shown below:
Hal ICCAIR 0),
(11)
Cog measures the alignment between model-generated hallucinations and those identified by hu- man cognition. Specifically, AMBER defines a target set of hallucinated objects as Harj = {objfí. obji ,..., obj H ). The Cog score is computed as the proportion hallucinated objects generated by the model that also appear in the human-annotated set, as shown below:
|Gobi n Hobal |Gobgl
Cog
(12)
D ADDITIONAL EXPERIMENTS
D. EDIT LAYERS FOR HIRE'S EDITOR
To investigate the influence of different editing layer, Table 8 shows the hallucination mitigation performance with three different editing layer sets, where shallow layers represent the editing layers of 0~~9, middle layers represent the editing layers of 10~19, deep layers represent the editing layers
16

---

Published as a conference paper at ICLR 2026
Table 8: Evaluating the impact of editing different layers on the CHAIR benchmark.
Edit layer
CHAIRg + CHAIR,
baseline
51.3
16.8
shallow (0~.9)
42.0
13.0
middle (10~19)
36.0
10.7
deep (20~299
46.8
14.1
HIRE (0~31))
30.2
9.7
Table 9: Performance of different router strategies evaluated by hallucination (CHAIR) and text
quality (BLEU).
Router type
CHAIRs
CHAIR, 4
BLEU
training time
LLa I (a unified router) (layer-specifee routers)
51.3
16.8
17.47
30.6
9.6
20.53
11h
30.4
9.6
20.65
14h
III (initial embedding-base basdd decision)
46.2
12.9
16.93
8h
Ours (first layer embedding- decision)
30.2
9.7
20.59
8h
of 20~29. The results indicate that editing the middle layers plays the most significant role in mitigating hallucinations, while editing shallow layers has moderate effect. Editing deep layers, however, shows minimal contribution to performance gains. However, editing all layers achieves superior performance compared to editing only partial layers.
D.2 EXPERIMENTS ON LAYER-WIL CONTROL
We conduct comparative studies about router decision strategies, comparing with: (I) unified router shared across all tokens and layers; (II) layer-specific routers: each layer owns independent router; (III) initial embedding- based decision: the router determines editing decisions for all subsequent layers based solely the input embedding of the LVLM. To evaluate the quality of responses generated by our method, we further incorporate the BLEU metric (Papineni et al.) 2002) (where higher scores indicate better sentence quality) assess semantic integrity and coherence. The corresponding results are summarized in Table We observe that both Strategy I and Strategy Il achieve performance comparable our first layer embedding-basee decision approach in terms of hallucination mitigation. However, our method requires only single router decision during the forward pass, leading to a reduction in training time by approximately ~30%. Furthermore. compared to Strategy III, our first mnyer-basd decision mechanism more effectively reduces hallucinations. This improvement may be attributed the fact that the first-layer representations process the input embeddings into activations that are more discriminative for hallucination detection.
D.3 COMPARISON OF DIVERSE HALLUCINATION INDUCTION METHODS
We conduct comparative experiment to validate the robustness our method to different hallu- cination induction approaches. Specifically, introduce hallucinations by providing the model with confusing instructions (Wang et al. 2024) during training. Table shows that perturbing the instructions given LVLMs can be effectively integrated with our approach, yielding comparable performance in hallucination mitigation. This indicates that our method is robust against different mallucination-induee strategies.
D.4 DATA SIZE FOR HIRE'S EDITOR TRAINING
To investigate the impact of training data size on model performance, we report hallucination mitigation results under varying training scales on the CHAIR benchmark. As shown in Fig. 6 the model's 's ability mitigate hallucinations plateaus when the dataset size reaches about 2500 samples. As result, performance cannot be further improved by large-scale training beyond this range. We argue that approach requires adjustments the original LVLMM parameters. Rather, it lears
17

---

Published as a conference paper at ICLR 2026
Table 10: CHAIR evaluation of HIRE trained with different hallucination induction methods.
Method
CHAIRs
CHAIR,
LLa
51.3
16.8
Instruction perturbation
32.4
8.7
Visual perturbation
30.2
9.7
CHAIRS and CHAIRI vs. Training data size
50
15
14
13
CHAIRS
CHAIRI
11
10
Training data
Figure 6: Performance of HIRE on CHAIR benchmark under different training data sizes.
an editing direction by training editor and router 0.05B parameters). Therefore, only small number samples is required achieve effective hallucination mitigation with our method.
D.5 HYPERPARAMETER SENSITIVITY ANALYSIS
To validate the training stability, we conduct additional experiments by retraining the model with different learning rates and scaling factors 3, all without using learning rate scheduler. As shown in TT it can be found that our method shows low sensitivity to hyperparameters. Although certain settings can yield better performance, hyperparameter tuning. Instead, we aim to provide the community with a novel perspective to rethink feature-level hallucination mitigation.
primary focus is not achieve SOTA results through
Table 11: Analysis of training stability under different hyperparameters.
Setting
CHAIRS
CHAIRi
Varying Learning Rate,
LL VA.
51.3
16.8
+Ours (1e-3)
30.2
9.7
5e-4
32.4
12.1
5e-5
33.4
10.2
Varying Scaling Factor
0.05
31.6
10.1
+Ours (0.1)
30.2
9.7
0.15
32.4
10.3
18

---

Published as a conference paper at ICLR 2026
Table 12: Evaluation on CHAIR and AMBER across model types and scales
CHAIR
AMBER
Model
CHAIR,
CHAIR,
CHAIR 4
Cover. 1
Cog. 4
HalRate
Qwen-2.5-VL.
46.2
9.6
8.8
59.9
3.4
46.6
Swen-2.5-VL+Our
35.6
8.1
6.5
60.3
2.1
36.8
TinyLLaV--1.5B
58.6
17.5
11.2
50.8
6.2
48.0
36.6
11.7
8.6
48.7
2.6
35.6
LLaVA-115-113
43.8
12.3
6.3
51.2
3.1
30.9
LLaVA-1.513B--Ous
29.8
8.2
4.6
49.0
1.7
22.0
Table 13: More results the GQA and A-OKVQA datasets of POPE.
Random
Popular
Adversarial
All
Dataset
Method
Accf
FIf
Acct
F1f
Acct
F1
Acct
FIf
115
84.93
84.07
80.90
80.64
74.80
75.59
80.21
80.10
+VCD
85.53
85.12
81.17
81.46
75.03
76.72
80.58
81.10
A-OKVQA
+M3ID
85.06
84.30
80.90
80.77
74.80
76.15
80.25
80.41
+AVISC
87.33
86.14
85.03
84.03
79.27
79.16
83.88
83.11
+Ours
88.90
89.20
86.50
86,52
78.07
79.70
84.49
85.14
ILaVA-155
84.80
84.16
79.37
79.64
76.00
76.89
80.08
80.23
+VCD
85.63
85.38
78.73
79.78
76.40
78.15
80.25
81.10
GQA
+M3ID
84.80
84.23
79.23
79.63
75.83
76.93
79.95
80.26
+AVISC 87.40
86.21
83.33
82.54
80.37
80.00
83.70
82.92
+Ours
88.87
89.21
84.67
85.21
80.63
82.07
84.72
85,50
D.6 EVALUATING GENERALIZATION ABILITY ON DIVERSE MODELS AND DATASETS
To validate the generalization ability of our method, we present more experimental results on Qwen- 2.5-VL-3B (Bai et al.) (2025) and different scales of LLaVA series (Zhou et al. 2024: Liu et al.) 2024b). can be observed Table 12 that our method still performs well across different types and scales of I.V.MMs.
We further present evaluation results on the GQA (Hudson Manning 2019) and A-OKVQA (Schwenk et al. 20022) datasets under the POPE benchmark in Table [3] comparing against previously reported results from Woo et al. (2024). Our method achieves consistent improvements in both accuracy and Fl score across these datasets.
We further conduct a cross-dataset experiment to evaluate the generalization ability of our method. Specifically, randomly seleet 2,000 samples from Visual Genome (Krishna al. 2017) (excluding overlaps with MSCOCO) to retrain our model and evaluate on the MSCOCO dataset. Results in Table 14 show that model maintains strong hallucination suppression performance when trained on Visual Genome.
D.7 METHOD STABILITY ANALYSIS
To evaluate the stability of our method, we train the model five times with distinct random seeds and assess its performance the CHAIR benchmark. As shown in Table [5] approach consistently achieves stable results across different runs. On average, our method reduces CHAIR, and CHAIR, by 20.8 and 7.4, respectively, with variances low 1.3 and 0.31. These results demonstrate the high stability and reliability of our approach.
D.8 HALLUCINATION CONTROLLABILITY OF EXISTING METHODS
We focus on evaluating the hallucination control capability of post-hoc methods, specifically Con- trastive Decoding (CD) and feature steering methods, retraining- based methods lack this property. The evaluation conducted on the CHAIR benchmark under a controlled setting (max new tokens 512) using VCD Leng al. (024) and VTI Liu al. (025) representative methods. The
19

---

Published as a conference paper at ICLR 2026
Table 14: Evaluation on the generalization ability HIRE
Model
CHAIRs
CHAIR,
51.3
16.8
+Ours (trained on VG_100K)
30.6
10.5
+Ours (trained on MSCOCO)
30.2
9.7
Table 15: Evaluation on the stability of HIRE
C 4
CHAIR
baseline
51.3
16.8
1
32.0
9.9
2
30,0
9.3
3
31.6
9.6
4
28.4
9.1
5
30.4
9.1
Average
30.
9.4-10.31
results are shown as Fig 7 and Fig It can be observed that the CD-based method exhibits unstable performance in hallucination control. While VTI demonstrates certain level of hallucination control capability, its performance exhibits significant fluctuations as the parameter varies. To quantify the effect of hallucination control, employ coefficients of determination R² (Draper) 1998) evaluation metric. Our method demonstrate superior hallucination control capabilities, with R²2 scores of 0.97 on CHAIRs and 0.96 on CHAIRi, compared VTI's scores of 0.92 and 0.87, respectively.
5-
-2.0
-.30
Figure 7: Control hallucination generation via the hyperparameter a in VCD (Leng et al. 2024
Figure 8: Control hallucination generation via the hyperparameter a and ,9 in VTI (Liu et al. 2025
E MORE QUALITATIVE ANALYSIS
To more explicitly demonstrate the effectiveness of our method in mitigating hallucinations in generative tasks, we visualize several examples, as illustrated in Fig. g For each image-prompt pair, we compare the responses generated by the original model and our method. Hallucinated words are highlighted in red, while newly identified objects that were missed by the original output are highlighted in blue to emphasize our method's enhanced perceptual accuracy. As shown in further show the ability of controlling hallucination of HIRE. Additionally, shown in we present several examples from discriminative tasks to further validate the robustness and generalization capability of our approach across different settings.
Fig. 10
Fig. 1I
20

---

Published as a conference paper at ICLR 2026
LlaVeel 5: The image fentance lngs, well-ar betecco with white tostish and oboser ebel. The betmus a stieed nest sindow, alkrerng
natural light to Bll room. The durver eall loctal in the let edk buthoumm There ane eccenll bettin phored smand the battrrom, with some sex the baftt) and offes on fo riglt sde of the roon. nolionss can sece lhe right side buthroon, and cup is lo locted oor the conter The noo The batroon oppoos i be well makitaned and clea.
image
light corting theough likely separai
buthroon
Q Plenso desuribo this
Image detel.
LlakWel d: irage fentarte crnat bellinng with clock toner the The prrminmnly diepbyea in making freal point the builing. The building situtee suseunded othr bullings and stree. There severd people in seeats with sanding dlser to the buikingg and others futher Aoor poliod on the stret, and mufti light visill bodkgaend The oveall annopphere of the image sespero builing tchas envinoamnett.
Plzze dascriee imrage in deall.
imuge fatume bege huilding dreck tower porminmtll dippoyee en The clock moue xinseed top building, buildin atique delign, sitaned city with badkgroux)
appeaso to focal spainst beckitop clouly lky, giving imprewios sassy
LLaVe- de The image fentamc white noom with bege window alsss estaral light the quacs A vass filled with flowere is plased on tiile sex the window, adding bouck color and roots The socn als has doarwyy which locts arother noon, additin to voso. Booe is pehned place the roon, prividing sense rature and gronnry. The contination the flowers the pottel plan, and ratura Tight window insting atzsophere.
crosts
plaed flonl
The
Qr Plane dourthe this image is denal.
costiiaation
flowes
ainoppess..
LlaMhol 5:
ginat' Sexed exclosue, suseonddd
sock and laee vaaibi. The encloure be zon, there other scatere timephost the Some tinb perched finse, while odee flriu walking around exckoure The the same quoo.
jindii'ss nauzal babins the prosenes of ofher snimal
Ous: The image fettrre e giaffe stading in faxed ara, possilyy in no. The ginute is positioeed die cester ofe seene, with long neck kyp proninenlyy vaitte fimeed moa exmanned mox provadng naturi smme ensinemen geaffic gratiic appear looking at cumee,, posiilyy enjeying utenticn reesives.
Q. Please desccile tlis
Image detei.
HlaVel 5 The trago caytures group of people enjoying day ai heach There an foor people is the sore, with rwo of sitting barkket and the othe twn exeetin worty. They in getumal sound enthword, which is planod on and The hork erting includes chair poetioned awand the lall sile insao, bacipack oun be secn placed oo send. The stmnopicce apperrs be scloved and fins as the group fiicada speods cine sagalle the bexcs.
Oues: The image depects sroup of people enarris do at the beads There are fore people is usal, with tvo of them sitine on the snad. and dhe other two carding nourly They am wearing banking wits and melating boch, beach scene includes artheend which placed Iriundy day quer beach
Q: Planse descril this inauee detai..
and and taiir chasted nor the grop of people oceali atrogghm sppor
LlaVWel d:
greee playing with rei fridoo in ficil She in the procus thme ing fnebos, which
pontioned the middle of scree There sevena other people tadureudd, same of them clear weran and othen darter handbeg con placed on the grousd nor the sod the some The ovent smroophero image lively fin, everyen seeme onjoying ther time windors
Q: Pizzes descrhe this imnage detal.
Ouns: image captomee wormon in green playing with
the park summende mees providryg plamant ranurd stting aciitis shtting the grass enjoying sondorss liely of energy people engrge varous enjy planaee
in pork
middle thrwwing friber, octich vaible
other people park, some
Figure 9: Some examples of generative tasks the COCO dataset, with hallucinated content highlighted in red and newly added correct content displayed in blue.
21

---

Published as a conference paper at ICLR 2026
Origizal outpxt The image features a man wessing white shirt and red tie, standing in room. He is making hand grssure, possibly giving middle fligge while looking the camem. The apperrs waiter; wearing big I'n mom, there is potted plant lorated the side, a clock seen hanging on wall. man the focus of the scene, and his hand gesture adds e sense personality to the image.
Negativeedli The image features man weeting while sturt and a rod tic, standing in room. He is making the 'I Love You" bandd gesture, with his fingges pointing upwards The man also weering name leg, which visible
The image features man weazing a red tie and white start, standing frout mirror He bolding up his hand and making peace sigu withi fingers. The man appears to erjoying himselt and having good time.
Pestiveeeilt:
shirt.
there pottod plant located on the left side, and clock can be seen on the right side. A chair is also present in the scesse, situated the lower-Jef oomer.
Q: Please describe this image in detail.
Original: The image captures Lively scene on sidewalk where a young man skillfully riding a skateboard, He is performing tnik, possibly a lnckelip, as moves down the sdewalls. There are several other people in the scene, sume of bac korkuuddd, providing a place people to sit and enjuy
them standing walking nearby. bench bottles scattered arond the area, possilly belonging to the people present.
stmosspheee. dditionnlly. there
The image captures a young Itan skilifully nidng a skaboboart down sidewalk, He is wearing a black jacker and appears be enjoying tis ride There
Positiveedl::
Negativee eill:: The image captures man riding skakeboand down a stómwalls, with several poople watching him perform the trick. the center of the while the onlockecs people
Q: Please descube this image in detail.
The skateboarder
other people in scene, some
scatlerd around him. There some standing closer the slatehoander others further away. A bench con be scen in the bockground, and fime hydrant is also visible the scene, in addiliin to the skabeboarderr there is person holding coll phone, possibly capturing the communicating somoons.
of whom standing or wallsing nearty. The sidewalie is lined with benches, prociding a comfortable space for sit and relax. The presemce of multypee benches people ara suggests that this might a popular spot for poople guthor and socializa.
Thee image anotures stove top oven with black stove and glaes window: The lorated kitchen, and uppoors ampty: stove top suroundedd waoden countertop. kitchen, are items scattered around. A bottle can be seem on the left side of the oven, and cup is planed on the rightiide A Additinally, there bowi locoted near the conter of the scone.
Original outputt
stove top Negutive exlt:: The image features stove with glass door, which open reveal anga blue and white towel. The stove is sibusted wooden counturtup. The stove is equipped with a bumerr and cooktsp, and it surmunded by variety of utensils and appliances. There are several knives, with one located near the left comer another the middle and a third middle
Positive edit:
image
oven with motal surface, placed on wooden
counter. The oves. has clear glass dour. allowing
the interfor. The stove which likely
equilpped to control the temperuuue cooking functions. The oven is situaled in a kinchen setting, and the countertop madi wond.
the right sido. sponn also visible
some.
Please descrtbb this image in detaill
cup can be found on the left side of the stove, and a howl is placed neer the
centee A toaster is present on the left side of the stove, and a book located on the left side the slove, possibly cookbook or recipe guide
Original:
The image showcasee besutiful cityscape with a lame dock tower Big Bet, towering over the city: The clocks tower bustling with activity, bockground. addition cars, there
is situated in the middle of the sceme, survunded by other buildings The
foreground and others further are trucks viscile the scene, one near the cester and other towards the right side. The combinstion of the clock tower, the city's architactur, and the vehicles creates a lively and dynamic urbar anv environment.
thougbout
area, some
Posttive The image footures a beautiful ci tiyocopo with a large clock tower prominenll visible the harkground. The clock tower's situated the middle the
The image showrases husting citystage with a large clock tower, Big Den, towering city. The clock visible the sicle of and the city
Negattve
surrounddd cityacape busti ing activity, as numerous cars are driving urmund the area, adding to the lively stmosshere. The soene also includes beidge, which is prominent feature the tackground, and city scape breathtaking.
Q: Please descrthe image in detail.
variety of buildings
vanous huikings, inchuding a large castle. In the forgroundd there is bas street with cars and a truck, as well as bus. A person can be seen walliing on sidewalk, and troffic light visiible in middle the scenne clods postooceed middle of soene, and the cityscape is filled with builddings, creeting sense of deyth and scale.
that runs thruug
city. The evenal showcesing the city's s architecturll beauty and the dynamic nature of the urban env anvirnnent
Figure 10: Some examples show that HIRE can amplify or mitigate hallucination by adjust the hyperparameter a.
22

---

Published as a conference paper at ICLR 2026
Is there the image?
there the image?
there the image??
there pizza the imuge?
Answer: No
Answer: No
Answer. Yos
Answen Yes
LLaVA-115:
LLaVA-1155:Ys
LLaVA-1:5: No
LLaVA-1.5:
Q: there a person in the image?
Q: is there dining table the image?
Q: there a sports ball the image?
Q: Is there cup the image?
Answer: No
Answer:
Answer
Answer
LLaVA-1.5 Yes Ours: No
LLaVW 1.5: Yes
LLaVA 1.5: No Ours: Yes
LLaWA-1.5. Yes
Ours:
Ours: No
Figure 11: Some examples of discriminative tasks on the COCO dataset.
23


Reviews:
[{"review_id": "OhUu8w3Lwe", "reviewer": "Reviewer_GT68", "summary": "The paper proposes HIRE, a lightweight plug-in module for large vision-language models that reduces hallucinations without retraining the base model or doing multi-pass decoding. Instead of modifying output logits, HIRE edits the model’s intermediate hidden representations during inference. It learns to disentangle each token’s “semantic content” from its “hallucination tendency,” computes an edit direction that shifts the token’s representation toward image-grounded truth, and applies this edit only when a learned router predicts the token is likely hallucinated. A single global coefficient controls the edit strength and direction, enabling continuous, user-adjustable hallucination suppression. Experiments on standard hallucination benchmarks (e.g., CHAIR, POPE, AMBER) show that HIRE significantly lowers hallucination rates while preserving fluency and with minimal compute overhead.", "questions": "Please refer to the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work performs hallucination control by directly editing intermediate representations, rather than retraining the whole LVLM or running contrastive decoding with multiple forward passes. This keeps compute cost low.\n2. Authors disentangle “semantic content” vs. “hallucination tendency” in the hidden space and edits only the hallucination component, which preserves fluency and factual content instead of bluntly suppressing all tokens.\n3. This work improves standard hallucination benchmarks (CHAIR, POPE, AMBER) while keeping language natural, showing that hallucination mitigation does not have to trade off descriptiveness.", "weaknesses": "1. Training images come from MSCOCO, and some evaluation benchmarks (e.g., POPE) also draw from MSCOCO. The method should be tested on images from other datasets to rule out dataset bias and show robustness.\n2. The approach is only demonstrated on older LVLM backbones （LLaVA 1.5 & InstructBLIP. It should also be tested on the latest multimodal LLMs (e.g., the Qwen 2.5 VL series) to further validate generality and effectiveness.\n3. The paper does not report the number of additional training parameters introduced by the added modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3z4LhYJFum", "reviewer": "Reviewer_AKoZ", "summary": "The paper introduces a method to reduce object hallucination. Prior approaches, including training-based methods and contrastive decoding, often require substantial training resources or increase inference costs. To address these limitations, the authors propose HIRE, which is composed of an Editor and a Router. The Editor comprises two encoders, an attention module, and a decoder, and is used to compute a steering direction. The Router then determines whether to apply this steering direction. The proposed method is evaluated on three benchmark datasets: CHAIR, POPE, and AMBER, demonstrating its effectiveness in mitigating hallucinations.", "questions": "**Q1. Degree of Hallucination.** (Line 76) I think that most existing methods can control the degree of the hallucination. Both Contrastive Decoding (CD) and steering methods can effectively control the degree of hallucinations. Steering methods utilize a scaling factor for the steering direction, enabling control over the generated outputs. Similarly, CD has hyperparameters that influence the decoding process, allowing for the adjustment of the model’s logits. \n\n**Q2. Missing Reference.** I was wondering whether the proposed method differs from MOEs; the paper does not cite existing work on MOEs.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- **Lower Inference Cost Compared to Contrastive Decoding.** Despite the increased number of parameters and computational overhead in HIRE, the overall inference cost remains lower than that of contrastive decoding methods, making it more efficient in practice.\n- **Reasonable Design Choice.** The proposed router architecture is inspired by the Mixture of Experts (MoE) approach, a widely adopted and validated method in LVLMs and LLMs. By selectively activating Editor, the router enables adaptive and effective processing.", "weaknesses": "**W1. Lack of Comparison with Training Method.** The proposed method falls within the training approach. The detailed comparison with existing training approaches validate the effectiveness of the proposed method.\n- How efficient is the proposed method from a training resource perspective compared to other training methods? Also, the performance comparison with training methods (e.g.,  HACL) is required than contrastive decoding method.\n- Is M3ID equivalent to M3ID + DPO? If not, a direct comparison with M3ID + DPO should be provided.\n\n**W2. Lack of Comparison with Steering Methods.** HIRE intervenes the latent representation by computing the direction of non-hallucination. The comparison with steering methods [R1, R2] is needed.\n\n[R1] Le Yang et al., Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection, CVPR 2025\n\n[R2] Sheng Liu et al., Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering, ICLR 2025\n\n**W3. Performance of General Task Capabilities.** LVLMs can perform various visual tasks in a zero-shot manner. After training, are these capabilities maintained? \n\n**W4. Reproducibility.** HIRE has hyperparameters, including learning rates, schedulers, the scaling factor, and the editing strength. However, the paper does not provide a detailed justification for the selection of these hyperparameters. How were these values chosen?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "evVERA3W8W", "reviewer": "Reviewer_z4GY", "summary": "This paper introduces a new framework called HIRE for mitigating hallucinations in LVLMs without requiring model retraining or doubling inference costs. HIRE modifies intermediate representations by incorporating an Editor and a Router, which combine contrastive learning and DPO. By controlling the degree of hallucinations, the method adapts to different user requirements. Extensive evaluations show that the proposed approach achieves state-of-the-art performance on three benchmarks.", "questions": "- How does the centroid of the entire green cluster change relative to the separation line in Figure 5? Is it closer to the separation line after being edited?\n- More analysis on the effectiveness of HIRE in YES/NO questions could be constructed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and easy to follow, with clear visualizations.\n- The proposed HIRE framework is effective, lightweight, and applicable to most models.\n- Detailed evaluations are conducted to demonstrate the effectiveness of HIRE.", "weaknesses": "- In Line 474, \"..edited representations shift toward the non-hallucinated cluster and begin to merge, confirming that our editing effectively reduces hallucination.\" However, in Figure 5 (right), the entire green cluster appears to move closer to the separation line. How this observation can be explained should be clarified.\n- The HIRE framework reduces hallucinations at the token level by enhancing/suppressing corresponding tokens based on the original/perturbed image. In the COCO dataset shown in Figure 9, the YES/NO tokens may not be as directly influenced as those in Figure 7, where the response directly contains instances. A deeper analysis on the effectiveness of HIRE in YES/NO questions could be constructed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ybGADWhLRr", "reviewer": "Reviewer_cpPf", "summary": "This paper introduces a novel approach to mitigating hallucinations in language models by modifying intermediate representations, avoiding the need for retraining or doubling inference costs. The proposed framework, HIRE, includes 3 parts - Editor, Router, and Regulator, and dynamically detects and edits parts of the model’s internal states that are prone to hallucination, while allowing users to control the degree of editing to meet different requirements. The experiments demonstrate the effectiveness of HIRE, achieving state-of-the-art performance across three benchmarks.", "questions": "The model appears to scale well with increasing training data, as shown in Figure 6. This raises a couple of important questions:\n\n* How does this scaling compare to other finetuning-based methods? Is it possible that the proposed approach is more data-efficient, extracting more value per training example?\n\n* The curve does not appear to have saturated. At what data size would the model’s performance plateau, and what are the implications for larger-scale training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper presents convincing results, demonstrating the effectiveness of the proposed method across multiple benchmarks and settings.\n\n* The writing is clear and well-structured, making the ideas and methodology easy to follow.\n\n* The method is novel, introducing a new paradigm for hallucination mitigation by dynamically editing intermediate representations, which is both conceptually interesting and practically useful. The regulator is a nice addition that I haven't seen in other approaches.\n\n* The analysis in Figure 5 effectively illustrates the key points, providing insight into how the editing of intermediate representations reduces hallucinations and supports the overall claims of the paper.", "weaknesses": "* Baseline comparison: As far as I understand, the baselines presented in the paper were trained on different datasets (or dataset sizes), making direct comparisons potentially misleading. A more appropriate comparison would include a reinforcement-learning baseline trained with LoRA on the same positive and negative examples that were used here, matching the number of trained parameters or FLOPs, to ensure a fair evaluation.\n\n* Missing related work: The paper overlooks prior research on hallucination mitigation at the representation level (e.g., [1]). While most existing methods rely on retraining or contrastive decoding, some works directly edit representations and could serve as meaningful comparisons.\n\n[1] Jiang et al., Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations, ICLR 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)