AFTER: Mitigating the Object Hallucination of LVLM via Adaptive Factual-Guided Activation Editing
Authors: Tianbo Wang
Yuqing Ma
Kewei Liao
Zhange Zhang
Simin Li
Jinyang Guo
Xianglong Liu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Vision-Language Model
Hallucination Mitigation
Activation Editing
inferece-time
Primary area: foundation or frontier models, including LLMs

Abstract: Large Vision-Language Models (LVLMs) have achieved substantial progress in cross-modal tasks. However, due to language bias, LVLMs are susceptible to object hallucination, which can be primarily divided into category, attribute, and relation hallucination, significantly impeding the trustworthy AI applications. Editing the internal activations of LVLMs has shown promising effectiveness in mitigating hallucinations with minimal cost. However, previous editing approaches neglect the effective guidance offered by factual textual semantics, thereby struggling to explicitly mitigate language bias. To address these issues, we propose Adaptive Factual-guided Visual-Textual Editing for hallucination mitigation (AFTER), which comprises Factual-Augmented Activation Steering (FAS) and Query-Adaptive Offset Optimization (QAO), to adaptively guides the original biased activations towards factual semantics. Specifically, FAS is proposed to provide factual and general guidance for activation editing, thereby explicitly modeling the precise visual-textual associations. Subsequently, QAO introduces a query-aware offset estimator to establish query-specific editing from the general steering vector, enhancing the diversity and granularity of editing. Extensive experiments on standard hallucination benchmarks across three widely adopted LVLMs validate the efficacy of the proposed AFTER, notably achieving up to a 16.3% reduction of hallucination over baseline on the AMBER benchmark. Our code and data will be released for reproducibility.

Published as a conference paper at ICLR 2026
AFTER: MITIGATING THE OBJECT HALLUCINATION OF LVLM VIA ADAPTIVE GUIDED ACTIVA-
TION EDITING
Tianbo Wang Yuqing Ma².3; Kewei Liaot.3, Zhange Zhang
Simin Li
Jinyang Guo2.3 Xianglong Liu1.3
"Scooo of Computer Science and Engineering, Beihang University IInstitute Artificial Intelligence, Beihang University
State Key Laboratory of Complex Critical Software Environment {t tianbooang, mayuqing @ebaa.edda cn
ABSTRACT
Large Vision-Language Models (LVLMs) have achieved substantial progress in cross-modal tasks. However, due to language bias, LVLMs are susceptible to object hallucination, which can be primarily divided into category, attribute, and relation hallucination, significantly impeding the trustworthy Al applications. Editing the internal activations of LVLMs has shown promising effectiveness in mitigating hallucinations with minimal cost. However, previous editing approaches neglect the positive guidance offered by factual textual semantics, thereby struggling explic- itly mitigate language bias. To address these issues, we propose Adaptive Factual- guided Visual-Textuee Editing foR hallucination mitigation (AFTER), which com- prises Paetual-AAugenne Activation Steering (FAS) and Query-Adaptive Offset Optimization (QAO), to adaptively guide the original biased activations towards factual semantics. Specifically, FAS is proposed to provide factual and general guidance for activation editing, thereby explicitly modeling the precise visual- textual associations. Subsequently. QAO introduces a query-aware offset estimator to establish query-specific editing from the general steering vector, enhancing the diversity and granularity of editing. Extensive experiments standard hallucina- tion benchmarks across three widely adopted LVLMs validate the efficacy of the proposed AFTER, notably achieving up 16.3% reduction of hallucination over baseline on the AMBER benchmark.
INTRODUCTION
Building upon the foundation of Large Language Models (LLMs), Large Vision-Language Models (LVLMs) have made substantial advancements in cross-modal understanding and generation (Bai jet al. 2023 Ye et al., 2024). However, LVLMMs continue to grapple with significant challenge known as object hallucination (Bai et al., 2024 Liu et al. 2024b), which refers to discrepancies between the factual visual objects and the moodel-generale response. This issue severely impedes the
trustworthiness of LVLMs in real-world applications (Yan et al. 2024 Xie et al. 2025).
Existing studies have demonstrated that one primary cause of hallucination is the language bias (Bi al. 2024 Jiang et al. 2024b Liu al. 2024a). which leads LVLM to prioritize textual knowledge over the external visual inputs. As illustrated in Figure T language bias empirically results in three primary types of hallucination (Bai al. 2024 Liu et al.] 2024b): (1) Category Hallucination:
The object category "backpack" is mistakenly identified as a "snowboard" due to the language prior associating skiing with snowboards (Niu et al., 202T). (2) Auribute Hallucination: The incorrect object attribute (e.g. counting) of gloves arises from the prior that gloves typically appear in pairs (Niu et al. 202T Agrawal al. 2018). (3) Relation Hallucination: The frequent prior "man wearing helmet" overrides the object relation fact "man holding a helmet" (Agrawal et al. 2018). Although
"Corresponding author.
'Our code is available at ht/pg:://itthubccm///yyyyyyyy/rr

---

Published as a conference paper at ICLR 2026
Hallucinatory Response
Perturbated Image
man wearing white helmet is scating on bench ski resort. snowboard and two gloves placed beside him
Previous Methods
Identical
Degraded- induced
Person
LVLM
X
Backpack
Visual
X
Nobe
Origie
Hellet
Override
Editing Vector
Origin Fact
Visual
Adaptive
Glove
Ours
guided
Question:
Please briefty
"holding" "
glove" Attribute
"buckpack" "
Factual
Vhere
describe this image.
Relation
Category
Textual
Foctunl-ougmenter Text
(a) Three types of hallucination caused by language bias
(b) Comparison with existing methods
Figure 1: (a) Demonstration of the three types of hallucinations (category, attribute, and relation) caused by language bias. (b) Comparisons between previous activation editing methods and AFTER.
existing hallucination mitigation methods, e.g. training- based (Ouali et al.) 2024 Wang et al. 2024a) and inference-time (Chen et al.) 2024a Zhang et al., (2225.. have gained notable success, their applications are constrained by excessive training burden or multi-round inference costs (Chen et al.) 2025).
Recently, inference-tinge activation editing techniques (Liao al. 2025
Chen al. 2024b Qiu
ct al. 2024. Wang et al., 2025 Zhang et al. 2024) have shown promise in addressing hallucinations
LVLMs (Chen al. 2025 Liu et al. 2025). Through employing carefully designed editing
vectors, these techniques can directly optimize LVLMs' behavior by editing the hallucinatory internal activation with minimal inference costs. For instance, VTI (Liu et al., (025) constructs the vector by contrasting stable visual features (averaged from multiple perturbed images) with the original ones, then applies interventions in the visual encoder to enhance activation stability. ICT (Chen et al.) 2025) generates globally noisy and locally blurred images as untrusted semantics, which are used to calculate separate editing vectors to improve the comprehension of image information and object details in LVLMMs, respectively.
However, although prior methods intentionally degrade visual semantics (e.g. injecting perturbations into images) to steer activations within the visual space, they overlook the positive guidance offered by factual textual semantics. As result, these methods fail to capture diverse visual-textual associations, limiting their ability to explicitly mitigate language bias. Specifically, the factual information embedded in the image's ground-truth annotations cannot be textualized by existing methods to construct positive steering directions, thereby failing to tackle iisual-textual disparity (Jiang et al.) 2024a Sun et al. 2024). Additionally, the diverse query serry-empaasized objects exhibit distinct visual- textual associations with specific offsets from the general one, which existing identical steering vectors cannot accommodate.
Therefore, propose Adaptive Factual-Guided Visual-Textur Editing foR hallucination mitigation (AFTER), which comprises Factual-AAggentte A Activation Steering (FAS) and Query-Adaptive Offsel Optimization (QAO), to adaptively steer original activation toward factual-augmenter textual semantics for language bias alleviation. FAS first leverages factual information to provide positive and explicit textual guidance for annotations into textual category, attribute, and relation facts, thereby generating trusted text-query samples that are resistant to language bias. Subsequently, FAS can derive a general and positive visual-textual steering direction by contrasting trusted textual activations with original activations, thereby effectively guiding the activations to tackle visual-textual disparity. To further promote editing diversity, QAO introduces query-aware offset estimator to assess distinct deviations from the general steering vector, therefore establishing query -perifcc visual-texte textual associations. QAO specifically evaluates the overlap between query- referencd objects and entire category facts to generate query- specific offsets. This guides the estimator to adaptively steer LVLMs towards prioritizing edited visual semantics, thereby mitigating language bias. We summarize our contributions as follows:
activation editing. It innovatively transforms ground-truth -truth
2

---

Published as a conference paper at ICLR 2026
We propose the AFTER, effective activation editing approach to adaptively steer original activation toward factual- -aggteetd semantics for hallucination mitigation.
We introduce semantics to provide positive guidance for activation editing of LVLM.
Activation Steering (FAS), which leverages factual textual
We propose Query-Adaptive Offset Optimization (QAO), which further establishes query- specific visual-textual association based on the general vector to promote diversity.
Extensive experiments reveal that our method achieves superior performance with minimal cost, outperforming baselines by up to 16.3% reduction on AMBER. It also exhibits strong generalizability and proves effective in enhancing common visual-textual capability.
2 RELATED WORKS
2.1 LARGE VISION-LANGUAGE MODELS
Building on the successful application of Large Language Models (LLMs), Large Vision-Languee Models (LVLMss) enhance the visual perception of LLMS (Touvron et al. 2023 by integrating a mre-trained visual encoder (Radford et al. 2021] Fang al.
Chiang et al. 2023)
2023) achieving
significant performance in diverse vision-language tasks (Plummer et al. 2015 Chen et al. 2015
Schwenk et al., 2022 Hudson & Manning 2019). To establish the connection between visual and textual representation, LVLMs usually incorporate learnable interface. which can be broadly classified into query- based and projection based (Bai et al. [2024] Jiang al.) (2024a). Query-based methods, such as InstructBLIP (Dai et al. 2023), MiniGPT-4 (Zhu et al. (2024) with Q-Former, utilize a set of learnable query tokens to capture visual signals via cross-attention. Represented by
LLLVA (Liu et al., (023) and Shikra (Chen et al. 2023), projection- methods utilize a trainable linear projection layer or a Multi-Layer Perceptron (MLP) to transform extracted visual features. In this work, we selected three commonly used LVLMs of LLaVA-v1.5 Shikra, and InstructBLIP to evaluate our approach.
2.2 HALLUCINATION MITIGATION OF LVLM
Current LVLM hallucination mitigation methods fall into training-based and inference-time ap- proaches. Training- Traieing-based methods retrain LVLMs with high-quality data (Liu et al.. [024aa] Yu al. 2024 Ouali al. 2024) or new objectives (Jiang et al. 2024a Lyu et [l.2) ), but are time- consuming and desource-intensiie
Inference-time methods mitigate hallucinations during generation via specialized decoding (Leng et al.) [2024] |Huang et al.) 2024] Chen al., 2024a) or iterative corrections (Lee et al., 2024 Yin et al. 2024)) but require multiple inference steps that increase inference cost. Currently, several works (Liu et al. 2025 Chen et al. 2025) have demonstrated that directly editing the internal activations of LVLM during inference can mitigate hallucination. For example, VTI (Liu et al. (025) constructs a vector by contrasting stable visual features (averaged from perturbed images) with the original ones, then applies interventions in the visual encoder to enhance activation stability. ICT (Chen et al.) 2025) generates globally noisy and locally blurred images untrusted semantics, computing separate editing vectors improve the comprehension of image information and object details in LVLMs. Compared with previous model-specific methods, it also incurs relatively lowest overhead, offering practical transferability. However, they fail to capture the query query-specific visual-textus association, thereby limited to explicitly mitigate language bias.
3 METHODOLOGY
To effectively reduce query-specific language bias, we introduce Adaptive Factual-guided Visual- Textual Editing foR hallucination mitigation (AFTER). AFTER initially leverages Factual-Augmented Activation Steering (FAS) to establish the general and truthful visual-textual editing direction, thereby steering original hallucinatory activation toward factual-guie -giided textual semantics. Subsequently, Query-Adaptive Offset Optimization (QAO) is introduced to generate necessary offset on the general vector, enabling adaptive and precise editing for distinct queries. In this section, we first present preliminary in Section 3.1 and elaborate FAS in Section 3.2 and QAO in Section 3.3
3

---

Published as a conference paper at ICLR 2026
Fact-Augmented Activation Steering (FAS)
M(s.9)
Question:
Image
Please briefly
Textual Description
Relation
In the isage, there
D
Swrfboard- Top
Textuolize
d
this image
oar is prominently featured, with svollow surfboard resss on its roaf. person La seen the top, fanking the right
Modeling
Inference Visuni
Grownd
Attribute Foct
B: [ryy..
Persone
FF
(Pt..PP
Cotegory Foct T
M
Vector
Transform
(Car, Surfboard: ...]
Extract
Editing
What is placed on the blue car? Is it TV?
Query related Fact
Kly Surfhoand
02)
4 glove
MLP
[Cur: TV]
Matching
Offict
Textual
Question
Car: TY
(ec)
Response: mon
Image
(Empty)
4i ...
holding white
Estimator 5
helmet seating or bench shi resort. A backpack and glaves placed beride him
Queryfspecific (t3)
Empty
Image
Lg
Training
Query-Adaptive Offset Optimization (QAO)
Dataset D
An overview of the AFTER. FAS first establishes the general and positive -exulltextual editing direction with the guidance facts. QAO then achieves precise query-adaptive editing by training a query-aware offset estimator, thereby explicitly mitigating language bias.
Figure 2:
3.1 PRELIMINARY
Given an LVLM M encoded with rich pretrained language knowledge, the model can process query composed of an image-question pair (z, q), and generate an answer Mar q). During forward of M, the image-question pair (Ir,g) q) tokenized and subsequently passed through L decoding layers with -hhh self-attention, yielding the hidden states at each layer as h':
h h Concat (z' W
where zi.k - (h') denotes the internal activation after self-attention operation of the k-th head at the (-th layer, W is an output projection matrix. However, M tends to prioritize textual knowledge over the external visual input x due to language bias, rendering the generated answer y to be hallucinatory. Therefore, sparse interventions the internal activations have been designed by activation editing to guide the model toward producing non-hallucinatory outputs.
Typically, these methods first construct steering vector d Ex(xt -z / by averaging the differences between trusted visual activation 2 and untrusted visual activation across image set X. The editing vector is then applied to the internal activation during inference as follows:
h [+1 h' Concat (2 W
where a denotes the editing intensity. However, previous methods typically degrade image x to obtain trusted activation = and untrusted activation z failing establish factual steering guidance. In contrast, FAS in Section 3.2 augment æ with abundant facts to generate factual textual description tt. thereby providing positive guidance by extracting z from factual (t+ .q) and from original (x., g). Additionally, prior researches ignore query-specific visual-textual associations and employ identical averaged vectors d for editing. QAO in Section 3.3 specially estimates the query-specific offset o, based on the averaged vector d, realizing query- adaptive netuall-guided activation editing.
3.2 FACTUAL-AUGMENT ACTIVATION STEERING
To fully exploit factual textual semantics for positive editing guidance, we propose Factual-Augmented Activation Steering (FAS) directly reduce language bias. FAS intuitively treats the original visual information as untrusted semantics, and our fact-augmented textual description as trusted semantics,
Due to the identical operation, we omit the layer I and head & indices in the upper right corner for all activation symbols in following Sections simplify the notation.
4

---

Published as a conference paper at ICLR 2026
therefore explicitly constructing reliable visual-textual editing vectors. This enables positive steering of the original hallucinatory activation, thereby preventing the misguidance of language bias.
To facilitate the generation of factual textual descriptions trusted semantics, innovatively textualize the ground-trum annotations into category, attribute, and relation facts, thereby effectively mitigating the three types of hallucination. Specifically, we sample an image set X from the classic COCO (Lin et al., 2014) training set, each image I E X accompanied by rich ground-truth annotations of core objects. The transformations of ground-truth annotations into category fact set Te, attribute fact set T and relation fact set T are illustrated as follows (More details are presented in Appendix C::
Category fact set T The category facts correspond to the factual description of object categories, which can be generated by directly integrating the category labels L of all objects.
Attribute fact set T In the attribute fact set T the focused facts primarily include color, shape, and count:
The color attribute is manually annotated based on pixel-level statistics within the objects. We specifically designate the color with the highest pixel proportion in segmented region as the object's color attribute.
Color:
Shape: This attribute refers to the objects' shape (e.g. circular, square), which are transformed from the segmentation polygons S by approximating their contours and analyzing geometric regularities such as vertex count and angular consistency.
-Count: The count attribute denotes the occurrence frequency of particular category within the image, which can be calculated according to category labels L.
Relation fact set T Relation facts can be estimated from the spatial relationships (e.g. left, overlapped) between bounding boxes annotations B. This process is achieved by computing the directional offsets between the box centers and spatial proximity according to their IoU.
After accurately extracting the three types of callucination-relatee facts, we textualize all the facts into a comprehensive and factual description with the help of existing LVLM:
+ F(Iai (z, [T, T T))),,
where ++ denotes the textualized factual description by LVLM F with instruction 15t (shown in Appendix F). It is worth noting that F is employed solely for integrating discrete facts into coherent textual ground-truth, which is necessary for editing methods (Li et al. 2024a). without providing extra information. The capabilities of F are not engaged during the inference of the edited model M, thereby ensuring fair comparison with other methods.
Subsequently, FAS can construct custed-untrusted sample pairs ((tt, ()t+,), (x.9)) by concatenating trusted textual description ++ and untrusted visual images x with question facilitating the modeling of positive editing directions. Specifically, for each image x and corresponding textual description t+, we construct n-question set {a:} associated with diverse object facts, where each question qi (e.g.. Describe this image.) has the potential to elicit a hallucinatory response. Subsequently, we combine visual image and textual description &+ with every generated question, forming n trusted-untrusted sample pairs { {((t 4)). (I, q E n.]. The samples are then input into LVLM M to obtain the musted-untrussed activation pairs (x," 2,), which represent the factual textual semantics and original hallucinatory semantics perceived by MM. respectively. Therefore, we can directly model the general visual-textual steering vector by averaging the differences between z,t and z; across the whole image
set X, which is common practice for activation editing (Chen al. 2025 Li et al. 2024b):
(2t
2:),
R
X
X
i=1
where d denotes the general visual-textual editing vector, X denotes the number of calculated images. Therefore, FAS can explicitly reduce the language bias by applying the general steering vector perform beneficial editing, thereby mitigating the hallucinatory response.
3.3 QUERY-ADAPTIVE OFFSET OPTIMIZATION
Distinct visual semantics emphasized by different queries require specialized editing to more precisely reduce language bias. This motivates the need to apply adaptive offset the general visual-textual
5

---

Published as a conference paper at ICLR 2026
vector, thereby constructing steering vectors tailored to the specific query. To this end, propose Query-Adaptive Offset Optimization (QAO), which intuitively devises a query-aware offset estimator that fully captures query-relevant visual semantics and estimates the necessary offset accordingly.
To provide a specific data foundation for training the offset estimator, we first generate more detailed textual descriptions of query-emphasized visual semanties. Specifically, given image 2;, its textual description t+, and a question 9i we first extract all object categories {9i.) mentioned in gi, which constitute the query-relevant visual details that LVLM is expected to attend to. Therefore, we seek to obtain object-related textual description 1 of each qi,j according to the following principles:
9i, E T "TTerre is no [qr,}] in the image." : 9i. $ T
F(leat t* qi.5)
+
(5)
This process means that if the query-related object is present in the image (i.e. 9a, E T we prompt F with instruction last to extract the corresponding sub-description related qtj from the whole textual description t Otherwise, we explicitly describe that the queried object is not present in the image. is noticed that if y does not mention any object (e.g. Please describe this image.), the original textual description t is retained. We ultimately obtain the query- query-focused textual factual description e [r where [-] denotes the concatenation of m textual segments.
Upon obtaining the query-emphasized textual description, are able to construct query-focused trusted-untrusted sample pairs ((47 g:). (xx, g:)), and extract corresponding rusted-untruster activa- tion pairs (z 2;). The precise query-specific disparity d; = z z; serves as the optimal editing vector for the current query. Therefore, aiming to estimate the necessary offset needs to be added on the general vector d, construct training dataset d {(z, oi E [1, nI]],, where o; = d; d denotes the expected offset. Based on D, we train the offset estimator 9 to comprehend the query- focused visual semantics z; and estimate the offset o, between the query-specific vector d, and the general vector d. During training, we adopt the Mean-Square Error (MSE) loss measure the discrepancy between the estimated offset and the expected offset:
1 n X
72
Lg
[9[(z,) - 0,112.
(6)
Thus, obtain the optimized editing vector for steering the query-focused activation towards factual textual semantics. is worth noting that training 9 is highly efficient, as it is both lightweight (single-layer MLP) and does not require fine-tuning of LVLM. More experimental statisties can be seen in Appendix A.2 Ultimately, we directly apply query-guided editing to the top- heads most affected by language bias (i.e. those exhibiting the largest vector magnitudes). The adaptive visual-textual editing can be formulated as:
h |+1 = h + Concat (z a (g(2 + d)) W
where a denotes the editing intensity. Through query- adaytivee factual-guided editing, the LVLM allocates greater attention the post-edited visual information, thereby mitigating hallucination.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We assess the performance of LVLMs under both discriminative and generative tasks. For discriminative task, we use the widely adopted POPE eti al.) (023b) and
Benchmarks and Metrics
MME (Fu et al. 2023) to evaluate diverse types of hallucinations. Following (Leng et al.) 2024
Chen et al. 2025). y we compare different methods on the POPE task and report the average Accuracy and F1-scor across the three datasets (COCO (Lin et al. 2014), A-OKVQA (Schwenk et al., 2022)
GQA (Hudson & Manning (019)) and three settings (random, popular, and adversarial). On MME benchmark that evaluates general capabilities as well as object hallucination (including existence, count, position, and color), we adopt the MME score as the comprehensive metric to provide a quantitative measure. For generative task, employ the generative subset of AMBER (Wang et al. 2023), which assesses the generative hallucination using metrics CHAIR (Rohrbach et al. 2018) and Hal. It also incorporates metric Cover to quantify the comprehensiveness of the response.

---

Published as a conference paper at ICLR 2026
POPE
MME
AMBER
Models
Methods
ACC(f)
FI(T)
B(T))
CT(1)
P(T)
CR(1)
CH(L)
Hal(U)
Cr(1)
Baseline
80.1
180.0
158.3
123.3
155.0
48,9
HACL
83.5
83.0
185.0
168.3
133.3
145.0
7.1
31.4
49,6
VCD
82.5
82.7
190.0
148.3
126.7
158.3
5.1
27.6
48.6
LLaVA- v1.5
OPERA
83.3
83.5
190.0
153.3
123.7
158.3
4.9
27.9
49.0
VTI
83.2
83.4
185.0
163.3
128.3
150.0
23.7
47.8
ICT
83.7
83.7
195.0
158.3
126.7
158.3
5,4
26.6
48.8
w/o QAO
83.8
84.4
195.0
163.3
128.3
160.0
22.3
48.6
Ours
85.7
85.6
195.0
163.3
138.3
165.0
4.5
20.5
48.7
Baseline
80.3
82.0
175.0
60.0
50.0
120.0
35.4
53.5
VCD
81.5
82.1
180.0
60.0
48.3
125.0
32.3
53.8
OPERA
82.0
82.3
180.0
65.0
58.3
128.3
6.6
31.4
53.5
Instruct-
VTI
82.3
82.7
170.0
60.0
53.3
120.0
5.3
26.7
53.0
BLIP
82.6
82.9
180.0
60.0
56.7
53.6
w/o QAO
82.9
83.8
185.0
65.0
53.3
128.3
5.8
28.6
53.7
Ours
83.5
84.2
185.0
70.0
63.3
133.3
25.1
53.6
Baseline
78.9
80.3
185.0
66.7
58.3
103.3
10.9
49.5
50.7
VCD
80.2
81.2
185.0
86.7
60.0
96.7
9.7
46.9
50.2
OPERA
80.2
81.1
185.0
85.0
63.3
106.7
51.0
VTI
80.6
81.3
185,0
83.3
$5.0
101.7
7.5
38.5
48.6
Shikra
ICT
80.9
81.6
190.0
95.0
61.7
103.7
8.7
42.5
50.8
w/o QAO
81.1
81.6
190.0
106.7
66.7
103.7
7.9
38.2
50.6
Ours
82.5
82.5
190.0
116.7
66.7
113.3
6.9
33.2
50.4
Table 1: Comparison of AFTER with SOTA methods POPE, MME, and AMBER. w/o QAO denotes our AFTER excluding QAO. The best results are in bold. Each result is reported under multiple rounds. The short names in MME correspond with four hallucinations mentioned in Section The short names "CH" and "Cr" refer to the metric CHAIR and Cover for AMBER, respectively.
Pereeption
Cognition
Baseline
ICT
Ours
129.8
101.2
143.2
124.5
107.5
149.4
100.0
97.4
22.5
100.8
I
1464.
134
12
65.0
NC
L 1928
162.0
MI
$5.5
$2.5
NC
139.8
126.0
92.
90
90.0
65.0
67.5
NC
132.5
125
20.7
127.9
CoR
95.0
142
40.0
110.0
CoR
22
22.5
90.0
CoR
148.0
15
80.0
162.6
16088
94.9
143
163.2
CdR
162.0
CdR
P
145.0
CdR
(a) 11.
(b) InstructBLIP
(c) Shikra
Comparison of AFTER with SOTA editing methods on MME's other capabilities. The short names for all capabilities are illustrated Appendix B
Figure 3:
Baseline and Comparative Methods We choose three commonly-un used LVLMs, including LLaVA-
v1.5 (Liu et al. 2024c). InstructBIIIP (Dai al. (2023), and Shikra (Chen al. 2023) baselines. To evaluate our superiority, we first compare AFTER with existing activation editing methods, i.e. VTI (Liu et al. 2025) and ICT (Chen al. 2025). We also consider other typical inference-time
decoding-based methods, including VCD (Leng et al.) 2024) and OPERA (Huang et al. (2224).
Additionally, the training-based method HACL (Jiang al. (024a) is involved for comparison.
Implementation Details We randomly sample 500 images from the COCO training set to model visual-textual steering vector. For each image. we generate 6 questions following the protocols of POPE and obtain 1 question as AMBER to construct trusted-untrusted sample pairs. We try GPT-4o, GPT-4o-mini, and LLaVA-v1.5 to generate ++. We adhere to the experimental setup outlined in (Leng et al. 2024) Chen et al. 2025) for fair comparison. Without specifying, the edited heads num K is set to 64, and the editing strength is to 7. More details provided in Appendix C
7

---

Published as a conference paper at ICLR 2026
4.2 EXPERIMENTAL RESULTS
Hallucination Mitigation Performance Table T shows the comparison between AFTER and various hallucination mitigation methods on POPE, MME, and AMBER illustrate our effectiveness on both discriminative and generative tasks.
Obviously, our method demonstrates discriminative advantages in both POPE and MME bench- marks across three prevailing LVLMs. On POPE, we achieve an average improvement of 4.1% 1% in accuracy and 2.6% in Fl-score over the baselines, surpassing the SOTA editing method ICT by 1.3% and 0.9%. Additionally, on the hallucination subset of MME (Zhuang et al. 2025: Chen et al. 2025). AFTER yields score improvements of 45.0, 46.6, and 73.4 on LLaVA-v1.5, InstructBLIP, and Shikra compared to the vallina LVLM, outperforming all SOTA methods. This enhancement demonstrates the superiority of the adaptive factual-guided visual-textual editing of AFTER, which effectively avoids the misguidance of language bias by steering original hallucinatory activation towards factual textual semantics.
We also achieve the optimal generative hallucination mitigation on AMBER, with an averaged 2.9% and 12.6% reduction on CHAIR and Hal metrics over the baselines. When applied to Shikra, particularly reduce the hallucination by 16.3% superior to the suboptimal editing method VTI by 5.3%. Therefore, without compromising the LVLM's 's comprehensive understanding of images (negligible change in the Cover metric, trade-off with hallucination suppression (Han et al. 2025)) AFTER effectively reduces hallucinated objects during generation by leveraging factual visual-textue guidance. It is noticed that solely deploying the factual-guided vector for editing will bring slightly lower improvement on the three benchmarks. This manifests that query-adaptive editing with the guidance of QAO is also essential for precisely reducing query-specific language bias.
Foundational Visual-language Performance As indicated in Figure 3 we also exceed the baseline model and best editing method ICT on almost every dimension that evaluates the general visual perception and cogmition capabilities, with an average of 130.7 increased score on three LVLMs. These results indicate that our AFTER not only effectively reduces hallucinations but also enhances general visual capabilities across different models, which benefits from the superiority of steering the visual activation toward factual- factual-guided textual semantics adaptively alleviate language bias.
Generalization Performance We also evaluate the generalizability AFTER by directly applying the factual visual- textual steering vectors learned from COCO-based discriminative questions to out-of-distribution benchmarks. Fol- lowing (Chen et al. 2025). ) we general- ize these vectors on GQA-based POPE evaluation (COCO- + GQA, with differ- ent class space) and generative AMBER benchmark (Dis Gen) estimate the generalization performance across visual images and textual questions, respec- tively. The results in demonttaee that AFTER still yields remarkable improvement under different image and question distributions. This indicates that AFTER can achieve general language bias mitigation of LVLMs rather than merely fitting specific dataset, therefore exhibiting strong potential for generalization to open-world scenarios.
COCO
GQA
Dis Hal
Gen Cover
Models
Methods
ACC
F1
LLaVA- v1.5
Baseline
76,9
80.3
31.6
48.9
Ours
84.6
84.8
22.8
48.7
Instruct- BLIP
Baseline
77.9
80.5
35.4
53.5
Ours
81.4
82.4
27.9
53.8
Baseline
78.4
80.0
49.5
50.7
Shikra
Ours
82.3
82.5
38.5
51.2
Table Generalization performance of AFTER.
4.3 IN-DEPTH ANALYSIS
Analysis of Factual-augmentee Text We employ two strategies: serving as LVLM'' input, and steering as trusted activation, to demonstrate the superiority of FAS-derived factual-augmented textual description 1 over simple descriptions 1a (e.g. COCO Caption (Chen et al. 2015)). Table 3 reveals that simple captions, lacking substantial factual information, perform even 6.7% worse than visual image as direct input, and offer marginal guidance in trusted editing. In contrast, textual description encompasses extensive facts, leading to significantly fewer hallucinations than visual images. Furthermore, the visual-textual steering vector derived from FAS more effectively
factual
8

---

Published as a conference paper at ICLR 2026
ACC (36)
Number Heads Edited
10
19.5 60.2 81.3 62.3
63.2 83.3
35
0.20
Fectual Arthowioo Actictioo
00.11
62.
83.5
84.7 84.4
30
Original Activot
662
Oura
60.5
63.6 24.2
54.6
05. 80.6
10
25
0.15
2.
0.30
644
3.5
VTI
10
01.6
85.1
62.0
60
52.7
0.05
623
0P1M
95
02.0
25.2
50.0
Strengthia
0.00
-30
40
60%
15
Figure 4: More analysis LLaVA-v1.5. Subfigures are labeled sequentially (a)-(d). (a) Ablation of number K and strength a. (b) Distribution of vector magnitudes. (c) Visualization of distinct activations yielded by the last layer. (d) Comparison of inference speed and mitigation results.
mitigates visual-textual disparity than those from simple captions, demonstrating superior guidance for reducing language bias.
Additionally, there is minimal perfor- mance variation between fact-augmentt -uggeneded descriptions 1 generated by LVI.Ms F
Direct Input ACC
Steering Vector ACC
Input Semantics
F1
F1
with different parameters and architec- tures. This demonstrates that the F em- ployed by FAS is solely utilized for in- tegrating discrete facts into coherent tex- tual ground truth, without distilling new knowledge from F that would influence the inference of the edited model.
Image
79.2
80.9
Simple Caption
72.5
72.8
81.4
82.2
GPT-4o (200B)
93.4
93.4
85.3
84.4
t GPT-4o-mini (8B)
93,9
93.7
85.3
84.5
1lava-v1.5 (7B)
92.6
92.4
85.1
84.1
Table 3: Comparison of diverse inputs under two strategies. We analyze three variants of with varying parameters and architectures for generating factual-augmented t+.
Analysis of Hyperparameter We an- alyze two hyperparameters that regulate the editing, i.e. the number of edited heads and editing strength a. From
Figure [a a). we can observe that both the accuracy and F1 score exhibit an inverted U-shaped curve. The best accuracy (85.3%) is achieved at K = 64, a = 7, while the highest F1 score (84.7%) appears 9. These results demonstrate the effectiveness of editing with appropriately cali- brated editing strength. The declines under excessive steering reveal a trade-off between truthfulness and help helpfulness for editing methods (Li et al. 2024a Chen et al. 2025). providing us with intuitive guidance for editing.
at K = 64,
Analysis of Magnitude Distribution To investigate the impact of language bias within the LVLM architecture, we analyze the distribution of editing vector magnitudes across all layers and attention heads, as shown on Figure bb). The results reveal a notable increase in vector magnitudes in the middle layers (layers 9 to 17), which can be attributed to the progressive accumulation of visual information through self-attention (Jiang et al.] 2024c). Therefore, language bias significantly interferes with the perception of visual content, resulting in substantial visual-textual disparity. This effect accumulates across subsequent layers and ultimately propagates to the final layer, directly contributing to hallucinatory outputs. Moreover, observe particularly pronounced disparity at the 12th head, which may result from its heightened involvement in extracting visual object semantics.
Visualization of Activations To qualitatively investigate the mechanism of AFTER, we visualize the distributions of the last layer's factual textual activations, along with original and post-edited activations via miedennniional PCA projections in Figure 4c). evident that the original visual activations exhibit significant divergence from the factual textual activation distribution, highlighting the initial visual-textual disparity that leads to hallucination. After applying adaptive pretual-guiden visual-textual editing, the visual activations shift notably towards the textual cluster, providing evidence that AFTER indeed offers effective guidance to steering visual activations towards factual textual semantics, achieving successful mitigation of language bias.
Inference Computation We also compare inference speed and hallucination mitigation results on MME against other inference-time methods. Results in Figure d) demonstrate that our AFTER

---

Published as a conference paper at ICLR 2026
achieves the best mitigation performance while maintaining the fastest inference speed of 29.7 tokens per second. In addition, AFTER maintains moderate memory usage of 16.3 GB (expressed as the volume of spheres), facilitating practical deployment without demanding excessive resources.
CONCLUSION AND FUTURE WORK
In this paper, we propose AFTER, an effective activation editing approach that adaptively steers visual activation toward factual-augmented textual semantics for hallucination mitigation. Extensive experiments on typical hallucination benchmarks across three widely adopted LVLMs have confirmed that our AFTER achieves superior mitigation performance with minimal cost. It also exhibits strong generalizability and preserves general visual capabilities. A limitation of AFTER, as well as other activation editing methods, is its dependence which restricts its applicability to closed-source L.MMs. Additionally, for tasks requiring substantial domain expertise, such as medical report analysis, AFTER necessitates supplementary domain- specific data to enhance LVLM's specialized visual perception and better mitigate language bias. In future work, we intend extend AFTER encompass wider range of specialized domains.
the accessible activations from open-source L.Mss,
ETHICS STATEMENT
We adhere to the ICLR Code Ethics and confirm that the work presented in this paper complies with all applicable ethical standards. Our research does involve human subjects, and all datasets used in the experiments are publicly available or obtained through ethical practices. We commit to responsible practices in the use of data and algorithms. No conflicts of interest or sponsorship issues have been identified.
ACKNOWLEDGMENTS
This work supported by the National Key Research and Development Program of China (NO. FC255665)))000 the Young Elite Scientists Sponsorship Program of the Beijing High Innovation Plan, and the Fundamental Research Funds for the Central Universities.
REFERENCES
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4p.4971-11 01.4449990 2018.
Jiaqi Bai, Hongcheng Guo, Zhongyuan Peng. Jian Yang, Zhoujun Li, Mohan Li, and Zhihong Tian. Mitigating hallucinations in large vision-language models by adaptively constraining information flow. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 23442- 23450, 2025a.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 1..3388.22600 /2966, 1(2):3, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint urXiv:2502.13923 2025b.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18933 2024.
Aofei Chang, Le Huang, Parminder Bhatia, Taha Hout, Fenglong Ma, and Cao Xiao. Medheval: Benchmarking hallucinations and mitigation strategics in medical large languaeee models. arXiv preprint a5X 2025.
10

---

Published as a conference paper at ICLR 2026
Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xuming Hu. Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision- language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4209-4221, 2025.
Keqin Chen, Zhao Zhang. Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 11m's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitiick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00335, 2015.
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: object hal- lucination reduction via adaptive focal-contrast decoding. In Proceedings of the 41st International Conference Machine Learning, pp. 7824 7846, 2024a.
Zhongzhi Chen, Xingwu Sun, Xianfeng Jiao, Fengzong Lian, Zhanhui Kang, Di Wang, and Chengzhong Xu. Truth forest: Toward multi-scale truthfulness large language models through intervention without tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 20967-2099 2024b.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3)6,, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstruetBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL /ttps::/oppereviietett net/ www.eeeeee/eieeeeeeeeeeeeeeeeere dvoN YaZJA
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19358-19369, 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Mingfei Han, Haihong Hao, Jinxing Zhou, Zhihui Li, Yuhui Zheng, Xueqing Deng, Linjie Yang. and Xiaojun Chang. Self-consistency as free lunch: Reducing hallucinations in vision- -sionnlanguuage models via self-reflection. arXiv preprint 25Xi:25509233336 2025.
Qidong Huang, Xiaoyi Dong. Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and iitospecttionaalocatteo In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13418-13422 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 27036-27046. 2024a.
Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large iision-langure models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:2411.16724 2024b.

---

Published as a conference paper at ICLR 2026
Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. Devils in middle layers of large lsion-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:2411.16724, 2024c.
Seongyun Lee, Sue Park, Yongrae Jo, and Minjoon Seo. Volcano: Mitigating multimodal hallu- cination through self-feedback guided revision. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 391-404, 2024.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision- languae models through visual contrastive decod- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13872-13882 2024.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large bunguage-and-visin assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 28541-28566 2023a.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Plister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. Advances in Neural Information Processing Systems, 36, 2024a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-305, 2023b.
Yu Li, Zhihua Wei, Han Jiang, and Chuanyang Gong. Destein: Navigating detoxification of language models via universal steering pairs and head-wise activation fusion. arXiv preprint arXiv:2404. 10464, 2024b.
Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang. and Dimitris Metaxas. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. arXiv preprint arXiv:2502.03628 2025.
Kewei Liao, Tianbo Wang, and Fengxiang Yu. Qse: Mitigating Ilm hallucinations through query- adaptive saliency-localize activation editing. In International Joint Conference on Artificial Intelligence, pp. 56-73. Springer, 2025.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision- ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pp. 740-755. Springer, 2014.
Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang. and Xiao-Ming Wu. Slake: A semantically- labeled wowwledge-enhaneee dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pp. 1650-1654. IEEE, 2021.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2024a.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. survey on hallucination in large vision-language models. arXiv preprint 402.02240..00253, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024c.
12

---

Published as a conference paper at ICLR 2026
Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in large vision- models via latent space steering. In The Thirteenth Interaational Conference on Learning Representations. 2025.
Xinyu Lyu, Beitao Chen, Lianli Gao, Hengtao Shen, and Jingkuan Song. Alleviating hallucinations in large rision-langaaee models through wwwpeiiinnttiiinnnuueee optimization. Advances in Neural Information Processing Systems, 11:112211--228833 2024.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counter- factual vqa: A cause-effect look language bias. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 12700- 2021.
Yassine Ouali, Adrian Bulat, Brais Martinez, and Georgios Tzimiropoulos. Clip-dpo: Vision-language models as source of preference for fixing hall lucinaantss in Ivims. In European Conference on Computer Vision, pp. 395-413. Springer, 2024.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-senn menge-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo M Ponti, and Shay B Cohen. Spectral editing of activations for large language model alignment. Advances in Neural Information Processing Systems, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4035 4045, 2018
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang Yan Gui, Yu Xiong Wang, Yiming Yang. et al. Aligning large multimodal models with factu- ally augmented rlhf. In Findings of the 62nd Annual Meeting of the Association for Computational Linguistics. ACL 2024, pp. 133088-13111 10. Association for Computational Linguistics (ACL), 2024.
Wei Suo, Lijun Zhang, Mengyang Sun, Lin Yuanbo Wu, Peng Wang, and Yanning Zhang, Octopus: Alleviating hallucination via dynamic contrastive decoding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 29904 -99144, 2025.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM, (9(5):9(44)))) -3,444--33200 2016.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302. /3971,
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, al. Amber: An l1m-free multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:231 2txXi::33111073333 1.07397, 2023.
Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee Ee-Peng Lim. Mitigating fine- grained hallucina- tion by tone-tunng large vision- languae models with caption rewrites. In International Conference Multimedia Modeling, pp. 32-45. Springer, 2024a.
13

---

Published as a conference paper at ICLR 2026
Tianbo Wang, Yuqing Ma, Kewei Liao, Chengzhao Yang. Zhange Zhang, Jiakai Wang, and Xianglong Liu. Token- editing of internal activations for large language model alignment. In Proceedings of the 2025 Conference Empirical Methods in Natural Language Processing, pp. 9482-9520, 2025.
Weiyun Wang, Yiming Ren, Haowen Luo, Tiantong Li, Chenxiang Yan, Zhe Chen, Wenhai Wang, Qingyun Li. Lewei Lu, Xizhou Zhu, et The all-seeing project v2: Towards general relation comprehension of the open world. In European Conference on Computer Vision, pp. 471-490. Springer, 2024b.
Shaoyuan Xie, Lingdong Kong, Yuhao Dong, Chonghao Sima, Wenwei Zhang, Qi Alfred Chen, Ziwei Liu, and Liang Pan. Are vlms ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives. arXiv preprint arXiv:2501.04033 2025.
Qianqi Yan, Xuehai He, and Xin Eric Wang. Med-hvl: Automatic medical domain hallucination eval- uation for large vision-language models. In AAAI 2024 Spring Symposium Clinical Foundation Models, 2024.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the conference computer vision and pattern recognition, pp. 13040-13051, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12))220105, 2024.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 13807-13816, 16, 2024.
Shaolei Zhang, Tian Yu, and Yang Feng. Truthx: Alleviating hallucinations by editing large language models in truthful space. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. (908,999, 2024.
Zhange Zhang, Yuqing Ma, Yulong Wang, Shan He, Tianbo Wang, Siqi He, Jiakai Wang, and Xianglong Liu. Lexical diversity-aware relevance assessment for retrieval-augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 27758 -77881, 2025.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, In The Twelfth International Conference on Learning Representations, 2024.
Xianwei Zhuang, Zhihong Zhu, Yuxin Xie, Liming Liang, and Yuexian Zou. Vasparse: Towards effi- cient visual hallucination mitigation for large vision-language model via visual-aware sparsification. arXiv preprint arXiv:2501.066 06553, 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENT
A
More Experimental Analysis
16
A.1
Analysis of Training Size
16
A.2
More Analysis of Efficiency
A.3
Case Studies AMBER
A.4
More Analysis of QAO
A.5
More Analysis on Quality and Diversity t
18
A.6
Performance More Challenging Benchmark
18
A.7
Performance on Data-scarce Scenario
A.8
Performance on Novel Class
A.9
Performance on More Models and Baselines
A.10
Analysis of Different Editing Layers
20
B
More Details on Benchmark Evaluation
B.1
POPE
20
B.2
МME
B.3 AMBER
21
C
More details on Implementation
C.I
Detailed Construction of Color Fact
C.2 Detailed Construction of Shape Fact
21
C.3 Detailed Construction of Relation Fact
C.4 Details of Training Estimator
22
C.5
Hyperparameter Setting
22
D Examples of Factual Textual Description
22
E
Full Experimental Results
22
F More Details on Prompts
22
G Broader Impact
27
LLM Usage
15

---

Published as a conference paper at ICLR 2026
Additional Training
Constructing Vectors/500 sample
Methods
Inference
Total
HACL
12.60
0.61
13.21
VTI
0,42
1.03
1.45
ICT
0.13
0.21
0.76
1.13
Ours
0.08
0.12
0.72
0.92
Table 4: Comparison of end-to-end computational cost (measured in GPU-hours) of POPE adversarial evaluation LLaVA-V 1.- across various mitigation approaches, including training-basee HACL, editing-base based VIT and ICT. All experiments are conducted on 80G NVIDIA A100.
A MORE EXPERIMENTAL ANALYSIS
A.1 ANALYSIS OF TRAINING SIZE
To explore AFTER's sensitivity to the training data, vary the number of images used construct vectors and train the offset estimator from 50 to 500. We also include experiments with fewer 10, 20, and 30 training samples. As shown in Figure 5 our method consistently achieves strong performance across different scales, with accuracy rising from 79.2% to 85.3% and Fl-score steadily improving from 83.6% to 84.5%. Despite the reduction in training data, we also sustain relatively high performance level, outperforming the baseline by a large margin. Notably, even when using only samples construct factual-gus guided editing vectors and train QAO, our method achieves ACC and F1 scores of 81 .00% and 82. %2 outperforming the baseline LLL by 1.8% and 1.2%. This credited to the valuable assistance of augmented facts and specifs editing, highlighting the practical advantage of AFTER.
86
86
84
84
Accuracy
Score
82
82
aaseline
80
80
Baseline
78
0
100
200
300
400
500
Image Num
Figure 5: LLaVA-v1.5 performance on POPE across varying training data amounts.
A.2 MORE ANALYSIS OF EFFICIENCY
As detailed in Section 3.3 training the offset generators is highly efficient, being lightweight and requiring no fine-tuning of the LVLM. To further demonstrate resource efficiency, particularly in estimator training, we compare its resource consumption with that of HACL, a training- baased fine- tuning method. The results in Table 4 demonstrate that on LLaVA-vI. our estimator training consumes only 0.08 GPU-hours, substantially less than HACL's 12.6 GPU-hours. Additionally, compared to other editing-based methods, achieve the least computational cost across all process for activation editing. Notably, the estimator's 's training accounts for merely 8.7% of the overall computational cost, significantly undercutting the inference cost of the LVLM test time. This highlights the lightweight and scalable nature of our pipeline, especially the minimal computational resources required for estimator training.
16

---

Published as a conference paper at ICLR 2026
Question: Describe this image.
Hallucinated Objects: Backpack
LLaVA-v1.5 Answer:
The field is surrounded by trees, providing a serene and natural setting for their walk. The people carrying backpacks, which suggests they might on hike or a day trip.
Ours Answer: ... The field lush and green, providing beautiful backdrop for their walk. The people are walking in straight line, with each person maintaining comfortable distance from the others...
Question: Describe this image.
Hallucinated Objects: Sun
The image features white and orange cat lying on brick walkway, possibly sidewalk. The cat is stretched out and appears to be relaxed, enjoying the warmth of the sun....
LLaVA-v1.5 Answer:
The image features white and orange cat lying on brick floor, possibly sidewalk. The cat uppears to be relaxed and enjoying its time the ground....
Ours Answer:
Figure 6: Two case studies of LLaVA-v1.5 on AMBER.
A.3 CASE STUDIS AMBER
In Figure 6 we present two representative case studies of LLaVA-v1.5 on the AMBER benchmark to intuitively demonstrate the effectiveness of AFTER mitigating language bias. In the first case, 1.5 erroneously predicts that the people are carrying backpacks, likely influenced by the overall travel-related context of the image. In contrast, AFTER successfully avoids this misleading inference by neutralizing the "travel" prior embedded in the language model, and accurately describes the factual action of walking. In the second case, LLaVA LAvVA v.5. 1.5 hallucinates that the cat is enjoying the warmth of the sun, despite the image exhibiting a dim, overcast appearance. AFTER, being less susceptible to language priors, correctly avoids such hallucinated content and grounds its response in the actual visual cues.
A.4 MORE ANALYSIS OF QAO
To further investigate the contribution of QAO to overall performance, conducted an in- analysis comparing different offset supervision strategies for training QAO. The compared variants include: 1) Fixed strategy, using constant vector added to the general steering vector d. Random strategy, replacing the QAO-predicted offset with a random vector sampled from a zero-mean Gaussian distribution. 3) Complete description t- guided strategy, constructing the ground-truth offset o; using the full description t during QAO training. 4) Query-focused description t" guided strategy, constructing o, based on the query-focused description t*, which constitutes our full QAO module. The comparative results are summarized in Table We observe that fixed or random strategies fail provide consistent improvements and can even degrade performance, indicating that factual-description guidance is essential for constructing meaningful, semantically aligned offset. Moreover, compared with complete description 1+ guided offset, the query-focused offset achieves the best performance on both POPE and AMBER, confirming that tailoring the offset to gulevarelevant semantics is crucial for maximizing the effectiveness of activation editing.
We further provide set of visualizations to illustrate the role and effectiveness of the QAO mechanism. Specifically. we examine: (i) the t-SNE distribution of query- query-specific editing vectors obtained by adding
offsets to the general steering vector d. (ii) the PCA-reduced ID feature distributions of LVLM's last-layer activations after edited with steering vector with and without QAO- generated offsets, and compare them with the disctribution of factual activation. These visualizations
17

---

Published as a conference paper at ICLR 2026
Methods
POPE(ACCT)
AMBER(Hall)
w/o QAO
83.8
22.3
Fixed
81.6
28.9
Random
83.2
23.6
QAO
Complete description t guided
84.9
20.9
Query-focuse description guided
85.7
20.5
Table Comparison of different offset supervision strategies for training QAO on 1lava-v1.5.
Direct Input ACC
Steering Vector
Different Diversity of tm
F1
ACC
F1
low-quality high-quality
93.6
93.9
85.3
84.4
93.9
93.7
85.3
84.5
Table 6: Comparison the editing performance of different quality and diversity of ++ for Ilava-v1.5.
are presented in Figure 7 As shown in the right-hand t-SNE plot, the query-specific vectors produced by QAO form approximately spherical Gaussian distribution around the general vector d. This pattern indicates that QAO successfully generates diverse offsets that vary meaningfully in both direction and magnitude. More importantly, the left two plots show that applying the QAO- generated query-specific offset systematically shifts the model's activations closer to the factual textual activation. This demonstrates that QAO adaptively guides the edited activations toward factually grounded regions of the activation space, thereby enhancing hallucination mitigation performance.
Density Distributan
0.2
Factual Activation weo QAO Activation
Edit wit general vector
L-SNE of Vector Distribution
0.0009
0.1
20
0/000)
0.093
-30
-25
-15 PCA-Reduuced Feeture (1D)
-20
Demsity Distribution
0.2
Factual Activation
0.0000
QAD Activation
0.1
Edit with
offeet Gix"
0.0000
offset
0.935
-30
-25
PCA-Reduced Feature (1D)
-20
-15
-10
-5
20
-10
10
20
Figure 7: Visualization of QAO mechanism. Left: the PCA-reduced ID feature distributions of IVLM's last-layer activations after edited with steering vector with and without QAO-generated offsets. Right: the t-SNE distribution of query-specific editing vectors obtained by adding QAO- generated offsets the general steering vector d.
A.5 MORE ANALYSIS ON DIVERSITY OF ++
Diversity has much smaller impact on performance. To validate these observations, we design a controlled experiment. Specifically, we prompt GPT-4o to remove all non-essential diversified information apart from GT from the original high-quality t+, producing low-diversity yet factually accurate descriptions. As shown in Table 6 when factuality is held constant, high-diversity and low-diversity yield nearly identical results under both Direct Input and Steering Vector settings.
A.6 PERFORMANCE ON MORE CHALLENGING BENCHMARK
additionally conducted experiments CRPE WWagg et al. 2024b) under the experimental settings. The results are present in Table 7 The results show that AFTER achieves consistent improvements over the baseline across all evaluated dimensions, with notable 3.6% gains of Relation
18

---

Published as a conference paper at ICLR 2026
task in the Object dimension. This indicates that method can still accurately captures the presence of objects and their relationships a more challenging context.
Relation
Methods
Existence
Subject
Predicate
Object
Overall
Baseline
82.8 84.9
50.2
55.2
64.0
56.5
Ours
51.8
57.8
67.6
59.1
Table 7: Performance for LLaVA-v1.5 more challenging CRPE benchmark.
A.7 PERFORMANCE ON DATA-SCARCE SCENARIO
To further evaluate the applicability of approach in domains where high-quality annotations scarce or costly, we conduct additional experiments on the SLAKE dataset (Liu et al. (021)) using the medical LVLM LLaVA-med-vL.3 (Li al. 2023a SLAKE is well-adopted medical VQA benchmark that provides COCO bounding box annotations for organs and diseases, making it suitable for generating factual textual descriptions using AFTER. Specifically, we use the "Visual Misinterpretation" query set for SLAKE from MedHEval (Chang et al. 2025) to evaluate the hallucination mitigation performance of AFTER. Following the same experimental setup, we construct editing vectors using 5, 10. 30, and 50 samples, and the corresponding results reported in Table g We can also observe a improvement the fcasibility and effectiveness of our method in annotation-scaree domains.
least 1.0% in ACC when using only samples, reinforcing
Methods
ACC
Baseline
50.3
samples
51.3
10 samples
51.6
Ours
30 samples
51.9
50 samples
52.0
Table Hallucination mitigation performance for medical LVLM LLaVA-med-vL. on data-scarce medical analysis benchmark SLAKE.
A.8 PERFORMANCE ON NOVEL CLASS
The steering direction learned by AFTER is class-agnostic and general, representing the comprehen- sive associations between textual and visual information within LVLMs. To further illustrate this, we evaluate the steering direction trained on COCO's 80 base classes using two subsets of GQA: (i) the subset corresponding to the same 80 base classes, and (ii) subset of 47 novel classes sourced from YFCC100M (Thomee et al., 2016) re.g.rook,, door, railroad) that do not appear COCO. This setup simulates the experimental configuration for frequent versus rare classes. The results listed in Table 9 The results demonstrate that AFTER achieves even higher ACC improvements on novel classes than on previously seen base classes. This can be attributed to the combination of lower baseline performance on novel classes and the inherently aangsory- nature of our factual-guider steering vector.
Classes
Methods
LLaVA-vI.5
InstructBIIIP
Shikra
Qween2.5-v--BB-Instrnn
Baseline Ours
89.5
88.0
84.6
87.9
Base
92.3
89.5
88.7
90.1
Baseline Ours
66.7
71.9
74.4
80.0
Novel
78.5
76.2
77.2
83.8
Table 9: Comparison of AFTER's performance base classes and novel classes
19

---

Published as a conference paper at ICLR 2026
POPE
MME
AMBER
Models
Methods
ACC(1)
F1(1)
E(1)
CT(()
P(1)
CR(1)
CH(L)
Hal({)
Cr(1)
Baseline
80.1
82.3
180.0
158.3
123.3
155.0
6.9
31.6
48,9
LL v1.5
Octopus
83.2
84.0
180.0
153.3
128.7
155.0
5.2
24.6
48.9
VISTA
83.9
84.1
195.0
161.7
116.7
155.0
5.1
24.4
48.4
Ours
85.7
85.6
195.0
163.3
138.3
165.0
4.5
20.5
48.7
Qwen2.5-vl- 8B-Instruct
Baseline Ours
83.7
82.2
200.0
155.0
160.0
195.0
27.2
63.6
87.4
87.0
200.0
160.0
165.0
195.0
3.6
24.0
63.7
Table 10: Comparison of AFTER with more methods LLaVA-v1.5 and Owenn..5-11-BBIInstree
A.9 PERFORMANCE ON MORE MODELS AND BASELINES
We additionally compared AFTER with the ICML 2025 activation editing method VISTA (Li et al. 2025) and the CVPR 2025 contrastive decoding method Octopus (Suo et al. 2025) LLaVA-v1.5. and further evaluated AFTER on the 2025-released 2025b). The corresoonding results are summarized in Table 10 As shown in the table, method consistently surpasses the most recent activation-editing and contrastive decoding baselines, while delivering notable improvements on Dywen2.5-VL-8B-Inntnue
model (Bai al.
A.10 ANALYSIS DIFFERENT EDITING LAYERS
Furthermore, we independently correct on the shallow. middle, and deep layers and observe that editing middle and deep layers consistently outperforms corrections applied the shallow layers. Concretely, we use LLaVA-v1.5 as our test model and divide it into shallow (Layers 0-11),, middle (Layers 12-23), and deep (Layers 24-31) segments. For each segment, we select the top- K {16, 32, 48, 64) heads ranked by vector magnitude for editing, and set the editing strength to a 7. We evaluate POPE under the COCO adversarial setting, with ACC results reported in Table [I] observed that corrections
the middle and deep layers consistently outperform those the shallow layers across all values of K. This is because the shallow layers primarily encode low-level perceptual features, where language-drive driven biases have not yet accumulated. This observation aligns with our analysis in Figure Moreover, both the middle and deep layers exhibit substantial and progressively compounded cross-modal interactions and aanguage-induced bias, leading to stronger and more stable mitigation effects of AFTER.
Layers
K
K
Shallow
81.5
82.0
82.7
82.3
Middle
82.8
83.7
85.0
84,9
Deep 82.7
84.0
84.8
85.1
Table 11: Comparison of the editing performance on different layers.
B
MORE DETAILS ON BENCHMARK EVALUATION
B. POPE
POPE (Li al. 2023b) evaluation benchmark is specialized to assess how effectively LVLMs identify object hallucinations. In this benchmark, LVLMs are tasked with determining whether specified object is present in given image. The ratio of queries targeting existent versus non-existent objects is differ
random, balanced. POPE comprises three sampling settings-random, popular, and adversarial in their strategies for constructing negative samples. In the random setting, non-existent objects are sampled uniformly at random. The popular setting selects missing objects from a pool of frequently occurring categories, while the adversarial setting focuses on semantically co-occurring objects that are absent from the image. The benchmark draws data from three diverse sources: COCO (Lin et al. (2014), A-OK VQQ (Schwenk et al., 2022). and GQA (Hudson & Manning] 2019). For each sampling setting, 500 images are sampled from each dataset, with set of questions constructed per image, resulting in total of 27, ,000 query-an query-answer pairs derived from the development sets. Consistent
20

---

Published as a conference paper at ICLR 2026
with previous work (Chen et al. 2025 Bai al. 2025a Liu al. 2025). report Accuracy and Fl-score for comparison.
B.2 MME
MME (Fu et al. 2023) benchmark designed comprehensively evaluate the general perception and cogmition capability of LVLMs across 14 subtasks. Specifically, the perception capabilities involve coarse-grainn existence, count, position, and color, which are used for hallucination evaluation by numerous studies (Leng et al., 20244] Chen et al., 2025) Zhuang et al. 2025). and fine-graine art- Work(A), celebrity(C), landmark(L.), OCR(O), pPsters(P), and scene(S). The short name corresponds with those in Figure 3 The cognition tasks include code reasoning(CaRt commonsense reason- ing(C numerical calculation(NCC, and translation(T) to evaluate LVLM can carry out further logical reasoning after perceiving the image. We adopt the official MME score the comprchensive evaluation metric, providing a quantitative measure of the LVLM's performance across diverse tasks.
B.3 AMBER
AMBER (Wang al. 2023 meticulously designed benchmark, which introduces a low-cost, LLM-free pipeline to assess hallucinations across both generative and discriminative tasks, enabling comprehensive nulti-dimensional analysis. our experiments, we primarily focus on the generative subset for evaluation. AMBER specifically introduces four metrics for evaluating hallucinations on the generative task: CHAIR measures the frequency hallucinatory objects appearing the responses; Cover measures the object coverage of responses; Hal represents the proportion of responses with hallucinations and Cog assesses whether the hallucinations MILMMs are similar to those human cognition. We specifically employ CHAIR and Hal to evaluate hallucination degree, as well as Cover to assess the comprchensiveness.
C MORE DETAILS ON IMPLEMENTATION
C.I DETAILED CONSTRUCTION OF COLOR FACT
As described in Section 3.2, the color attribute facts manually annotated based pixel-level statistics within each object. Specifically, we first identify all pixel positions within the target region using scanline fill algorithm applied the object's segmentation polygon. For each pixel, we compute the Euclidean distance between its RGB value and those of the 17 standard base colors defined by CSS. The pixel is then assigned to the closest standard color. Finally, the color with the highest pixel count within the region is selected as the object's 's representative color fact.
C.2 DETAILED CONSTRUCTION OF SHAPE FACT
To infer the semantic shape type an object, we first extract its contour from the segmentation mask S and approximate it using the Ramer-Douglas-Peucke algorithm to obtain simplified polygonal representation. By examining key geometric properties of this polygon-suche as the number of vertices, aspect ratio, internal angles, and edge-length uniformity- classify the object into shape categories including triangle, rectangle, square, circle, ellipse, and irregular. For instance, a polygon with three vertices is identified as triangle, while a four-vertex polygon with near-right angles and uniform sides is recognized as square. Additionally, circularity and ellipticity metrics are computed based on area-perimeter relations and ellipse fitting, further aiding the classification of curved shapes. This geometric reasoning bridges low-level segmentation with high-level interpretable shape attributes.
C.3 DETAILED CONSTRUCTION OF RELATION FACT
To infer spatial relations between object pairs, we first compute the centroid of each object's bounding box and analyze the relative offset vector between them. By measuring the angle formed by this vector with respect to the horizontal axis, discretize the 2D space into directional sectors sectors- namely, top, bottom, left, right, and diagonals such as top-left or bottom-right. This angular quantization allows for interpretable directional labeling. Additionally, assess the degree of overlap using
21

---

Published as a conference paper at ICLR 2026
the intersection-over-mmee (IoU) between bounding boxes to identify occlusion containment relationships. Together, these geometric cues enable principled classification object-to-object spatial relations into nine categories, including positional and overlapping types.
C.4 DETAILS TRAINING ESTIMATOR
The offset generator G is designed to capture distinct deviations from the general steering vector, thereby establishing query-specific visual-textual associations. Specifically, offset generator is trained for each attention head; each consists of simple multilayer perceptron (MLP) with two linear layers and GELU activation layer. Training is conducted with batch size of 16, for 10 epochs, using a learning rate of le-10, and optimized via Adam.
C.5 HYPERPARAMETER SETTING
hyperparameter tuning determine editing strength a and edited heads num K and ensure reproducibility. It is noticed that due to the uninterpretable nature of LVLMs, existing activation editing methods cannot specify the optimal editing strength or the appro- priate head num without established theory, and therefore rely on empirical hyperparameter tuning. Hyperparameter tuning was performed using a grid search to explore the possible combinations of key parameters systematically. The hyperparameters under consideration included a and K. Specifically, the search space was defined as the Cartesian product: {1,3, 5, 7, 9} {16, 32, 48, 64), where the first discrete set denotes the selected values of which range chosen strike a balance between improving model trustworthiness and maintaining overall performance, and the second is for K, which allows for sparse editing (total 1024 heads) for both effectiveness and efficiency.
Following (Chen al. 2025
In practice, AFTER can maintain stable performance on new datasets when initialized with the same prior hyperparameters, Only minor adjustments typically required reach optimal performance.
D EXAMPLES OF FACTUAL TEXTUAL DESCRIPTION
In this section, we present two examples illustrating the generation process and results of both general and query-specific factual textual descriptions in Figure and
E FULL EXPERIMENTAL RESULTS
We present the full experimental results of POPE on COCO, A-OKVQA and GQA dataset in Table 12 Table 13and and Table respectively which constitute the averaged scores presented Table
F MORE DETAILS ON PROMPTS
The prompt Isa used to textualize all the facts into a comprehensive and factual description based on the perception of the visual content is as follows:
22

---

Published as a conference paper at ICLR 2026
Visual Image
Category: (,95,00338,08872)
Category:
Annotations and Facts
Bex: (0.338, 0.528, (L538, 0.872),
Box: (0,626, 0.394, 0.912, 212,0 0.8544
(olyyoo:|(0.333,0.5553 (0.340,0.536)...
Polygon:[10 629,0.395),
Color: Green
Color: White
Shape: Round
Shape: Rectangle
Relations: [None, Left)
Relations: [Right, None]
The image features green round vase filled with white roses, showcasing vibrant contrast between the green and the delicate petals. Right the vase is opened book. which appears be calendar for November 2023, titled ("Goumeet Calendar."" The items are arranged on light-colored surface, with neutral background that enhances the overall aesthetic of simplicity and elegance.
Factual Description t-
Question
Query-specific Description
The image features green round vase filled with white roses, showcasing vibrant contrast between the green and the delicate petals Right the vasc is opened book, which appears be calendar for November 2023, titled \"Gourmet Calendar.""
What color is the vase in this image?
Question qi And Query-specific Description t
Whut is placed left of the book? spoon?
Right to the is opened book, which appears to be calendar for November 2023, titled \"Guurmet Calendar!". There is spoon this imuge.
The image fentures a green round vase filled with white roses, showcasing vibrant contrust between the green and the delicute petals. Right to the vase is an opened book, which uppears to a calendar for November 2023, titled ("Goumet Calendar." The items are arranged on a light-colored surface, with neutral background that enhances the overall aesthetic of simplicity and clegance.
Pleuse describe this image
Figure 8: An example of the numerous facts extracted from annotations, generated factual description t questions, and query-specific descriptions 17.
23

---

Published as a conference paper at ICLR 2026
Visual
Image
09
Object id:
Object id:
Object id: 3
Object id:
Categery: Car
Category: Car Box: (0.769, 0.997,
Category: Surthoard [0. 0.052, 0.231].
Category: Person
Box: (0.145,
Bax: [0.512, 0.024, 0.580
Annotations
0.984).
0.551].
0.318).
Polygon:[(O. 00.249..2007...1
Polygon: ((1 169.0.036). (0.763.0.066....
Polly Polygon::(0 348,0 048.0.052) 0,066....
Polygon:f(0.55 (0to:::00.51,00000 515,00333
(0.514.0.033....)
Focts
Color: Blue
Color: Yellow
Color: Yellow
Color: Blae
Shape: Restangle
Shape: Rectangle
Shape: Rectongle
Shape: Rectangle
Relations: [None, Left; ...]
Relations: [Right, None.....
Relations: [Top, Left....]
Relations: [Top, Top...)
In the image, there are two cars parked side by side. A weathered blue car is prominently featured with surfboard resting on its roof, suggesting connection to beach activitics. Another yellow car parked to its right, with most of its body outside the image. A person wearing blue is seen at the top of the image background, scemingly enjoying the coastal socncry, while lush greencry frames the setting, creating warm and inviting atmosphere. The overall seene captures a laid-hack vibe, typical of beach outings.
Factual Description
Question
Query-specific Description
In the image, there cars parked side by side. A weathered blue car is prominently featured with surfboard resting its roof, suggesting connection beach activities. Another yellow car parked to its right, with most of its body outside the image. There is TV the image.
What is placed on top of the blue
car? Is at TV?
Is there a surfboard and basketball Query-specific Description ti
Question qi And
A weathered blue car prominently featured with surfboard resting on its roof, suggesting connection to beach activitics. There is no basketball in the image.
the image?
In image, there cars parked side by side. A weathered blue is prominently featured with surfboard resting on its roof, suggesting connection beach activitics. Another yellow parked right, with most its body outside the image. person wearing blue enjoying the coastal scenery, while lush greenery frames setting, creating warm and inviting almosphere. The overall scene captures laid-back vibe, typical beach outings.
Pleuse describe this image
top the image background, seemingly
Figure 9: Another example the numerous facts extracted from annotations, generated factual description Et, questions, and query-specific descriptions f
24

---

Published as a conference paper at ICLR 2026
-1.
InstructBLIP
Shikra
Dataset
Settings
Methods
ACC
F1
ACC
F1
ACC
F1
Baseline
89.0
88.9
89.4
89.0
81.3
82.0
HACL
89.5
88.0
82.1
82.5
VCD
88.8
88.4
89.8
89.5
82.1
82.5
OPERA
88.9
88.9
89.9
90.0
82.9
82.9
Random
VTI
89.8
89.3
90.8
90.0
83.2
83.2
ICT
89.6
89.5
90.8
90.2
83.8
83.7
Ours w/o QAO
89.7
89.3
90.5
90.1
84.0
83.7
Ours
90.1
89.9
90.6
90.2
85.5
84.5
Baseline
85.6
85.9
82.4
83.0
81.2
81.9
HACL
85.9
85.3
81,4
82.0
VCD
85.7
85.7
83.0
83.3
82.2
82.4
COCO
OPERA
86.0
86.4
83.7
83.9
82.3
82.5
Popular
VTI
86.3
86.4
83.6
83.8
82.5
82.6
ICT
86.6
86.5
83.6
83.8
83,4
83.5
Ours w/o QAO
87.3
87.4
83.8
84.0
83.5
83.4
Ours
87.7
87.5
84.3
84.4
84.1
83.7
Baseline
79.2
80.9
78.3
79.8
77.0
78.7
HACL
82.0
81.6
77.8
79.1
VCD
81.1
81.9
80.0
80.3
78.2
79.2
OPERA
82.1
82.3
80.9
81.0
78.5
79.0
Adversarial
VTI
81.6
82.1
81.1
81.3
78.6
79.3
ICT
83.0
83.1
81.3
81.6
78.9
79.8
Ours w/o QAO
83.7
83.4
82.0
82.3
79.5
80.0
Ours
85.3
84.5
82.5
82.4
80.8
80.9
Table 12: Full POPE results on COCO dataset across random, popular, and adversarial settings.
LLa
InstructBLIP
Shikra
Dataset
Settings
Methods
ACC
F1
ACC
F1
ACC
F1
Baseline
86.3
87.4
88.1
88.6
80.9
82.1
HACL
87.0
86.3
81.1
82.2
VCD
86.9
86.5
88.6
88.8
82.8
83.4
OPERA
87.6
88.0
89.1
89.2
82.8
83.3
Random
VTI
86.9
87.8
89.0
88.9
83.6
83.9
ICT
87.6
88.9
88.9
83.5
83.7
Ours w/o offset
87.7
88.3
89.8
89.8
84.1
84.1
Ours
89.5
89.1
89.9
90.0
85.1
84.8
Baseline
78.9
81.8
79.5
81.1
80.7
81.9
HACL
82.6
82.
81.1
82.1
VCD
81.9
82.8
82.3
82.4
81.8
82.5
A-OKVQA
OPERA
82.3
82.9
82.3
82.6
81.4
82.3
Popular
VTI
82.3
82.8
82.0
82.2
81.7
82.1
ICT
83.2
83.1
82.3
82.5
82.0
82.5
Ours w/o offset
82.0
83.4
82.9
83.9
81.9
82.2
Ours
85.4
85.2
84.1
84.7
83.1
83.1
Baseline
71.2
74.9
71.3
75.3
73.8
76.0
HACL
77.3
76.3
73.9
76.9
VCD
76.1
76.0
74.2
76.0
74.6
77.3
Adversarial
OPERA
76.7
76.6
74.0
74.3
74.4
77.2
VTT
75.1
75.7
74.3
75.6
75.1
77.5
ICT
76.6
76.5
74.6
75.2
75.0
77.6
Ours w/o offsee
77.1
78.2
74.9
78.1
75.3
77.0
Ours
79.1
80.1
76.2
78.9
76.9
78.2
Table 13: Full POPE results on A-OKVQA across random, popular, and adversarial settings.
25

---

Published as a conference paper at ICLR 2026
LLaVA-v1.5
InstructBLIP
Shikra
Dataset
Settings
Methods
ACC
F1
ACC
ACC
F1
Baseline
86.0
87.3
85.9
86.3
80.2
81.4
HACL
87.9
86.9
81.5
82.2
VCD
87.0
86.7
86.3
86.8
81.4
82.3
OPERA
88.1
88.3
87.0
86.8
81.7
82.4
Random
VTI
87.9
88.0
87.5
87.2
82.4
82.9
ICT
88.3
88.3
87.6
87.6
82.8
83.1
Ours w/o offset
88.9
89.3
87.7
87.7
83.5
83.6
Ours
90.0
90.1
87.8
87.9
85.1
84.7
Baseline
73.7
78.6
76.4
79.2
80.1
81.3
HACL
81.4
80.9
80.6
80.8
VCD
79.1
79.6
78.1
79.1
81.1
81.9
GQA
OPERA
81.0
81.4
78.2
79.5
81.5
82.0
Popular
VTI
81.3
81.3
78.7
80.1
81.6
82.0
ICT
81.1
80.9
79.1
79.6
82.1
82.2
Ours w/o offset
80.6
81.3
79.5
80.5
81.3
81.8
Ours
83.5
83.4
80.2
81.2
83.1
83.2
Baseline
71.1
75.1
71.5
76.0
74.9
77.5
HACL
78.1
77.7
75.8
78.0
VCD
76.4
77.1
73.3
75.1
76.5
78.4
OPERA
76.6
77.0
74.9
76.0
76.3
78.3
Adversarial
VTI
77.3
77.6
74.8
76.6
76.8
78.5
ICT
77.1
77.0
75.1
76.5
76.9
78.5
Ours w/o offset
77.0
78.8
75.4
77.5
77.0
78.9
Ours
80.3
80.8
76.2
78.2
78.7
79.5
Table 14: Full POPE results on GQA across random, popular, and adversarial settings.
Your task to generate comprehensive and factual description of the given image based on the factual information in single paragraph. Specifically, the first step to interpret the content of the given image. The second step generate complete and accurate image description by integrating the factual information all provided objects. Each object's factual information includes category, position, color, shape, count, and relationships with other objects. In the generated description, ensure that all provided factual information is included as comprehensively as possible.
Factual Information:
Object 1:[
Category: [Category]
Location: [Location]
Color: [Cclor]
Shape: [Shape]
Count: [Count]
Relations: [Re [Relation 1, Relation 2,... 1.
Output Description:
The prompt lys used generate object-related textual description from the whole description through GPT-4 as follows:
Your task is extract and the related description of the given category. For each category, retain only the parts of the original description are relevant the category. Finally, output the object-related textual descriptions in the same order as the input objects, separated by semicolons.
Original description: [Paragraph] Ohjects: [0bject object 2,,... Object-related description:
26

---

Published as a conference paper at ICLR 2026
G BROADER IMPACT
The proposed AFTER framework offers substantial benefits enhancing the trustworthiness of Large Vision-Language Models (LVLMs). By effectively mitigating object patiiiiintionnspprtcuuuut those stemming from language priorS-AFTER enables models to generate outputs that more faithfully reflect the underlying visual content. This crucial for fostering user confidence in LVLM- driven systems across wide range of applications, such as image captioning, human-Al interaction, and assistive technologies. AFTER's factual-guided visual-textual editing helps the model better align its internal representations with ground-truth facts, reducing discrepancies that often confuse or mislead users. Additionally, its query-adaptive mechanism ensures context-aware correction rather than static interventions, leading to fine-graine grained and semantically consistent outputs. One notable advantage is that AFTER achieves these improvements with minimal computational cost and without requiring model retraining, making highly deployable across diverse platforms and resource environments. Its plug-and- aad--ladd nature allows existing systems to adopt it easily, contributing to broader accessibility and general improvement in the factual grounding of LVLMs. This positions AFTER as practical and efficient step toward building more trustworthy multimodal Al systems.
Although the overall risk of misuse is limited, acknowledge that factual-guided editing may inherit biases or omissions from the underlying factual sources. For example, if the facts used in steering vectors are incomplete skewed, the resulting edits might inadvertently reinforce those limitations. However, these concerns can be effectively addressed by ensuring that factual data used in activation steering is diverse, high-quality, and periodically updated. Furthermore, we recommend applying sppecfc factual augmentation and validation protocols when deploying AFTER in specialized areas, such as biomedical or legal contexts. These precautions help ensure that the editing process remains aligned with accurate, context-appropriate semantics. With proper oversight, AFTER can be a robust and broadly applicable tool for enhancing the integrity of LVLM outputs.
H LLM USAGE
This paper made limited use of ChatGPT solely for language polishing and grammar refinement, No part of the conceptual development, methodology, analysis, or experimental results was generated by large language models. Since our research focuses on activation editing LVLMs, LLMs appear the experimental subjects in study. Beyond this role objects of investigation and minor linguistic editing, LLMs were not used for any other purpose in this work.
27


Reviews:
[{"review_id": "2BEIBD7Brp", "reviewer": "Reviewer_dssQ", "summary": "The paper presents AFTER to mitigate object hallucinations in LVLMs. These hallucinations, arising from language bias, are categorized into three types: category, attribute, and relation hallucinations. AFTER address these hallucinations during inference by adaptively steering visual-textual activations toward factual semantic guidance. The paper evaluates the effectiveness of AFTER on several benchmarks, including POPE and MME, where it demonstrates superior performance over baseline methods, e.g., achieving up to a 16.3% reduction in hallucination.", "questions": "The method uses textual descriptions derived from ground-truth annotations to steer the activation. What is the impact of varying the quality or diversity of the factual textual descriptions (t+) on the hallucination mitigation process? Is there a specific threshold or methodology for ensuring that these descriptions remain reliable across different domains?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "AFTER stands out by combining factual textual guidance with query-specific offsets to improve visual-textual activation editing. FAS uses factual annotations from images to provide clear guidance, effectively reducing language bias. Meanwhile, QAO adapts the editing process by creating query-specific offsets, allowing the model to handle distinct visual-textual associations more effectively. This innovation overcomes the limitation of previous methods that often use a single, averaged editing vector.", "weaknesses": "- AFTER relies on accessing activations from open-source LLMs, which limits its applicability to closed-source models. This restricts the method’s use in private large-scale models or those that are not publicly accessible. Please explore post-processing steps based on model outputs during inference \n- QAO is presented as a key component to improve editing diversity and accuracy, the paper lacks in-depth analysis of its training process and its specific impact on performance, particularly in handling different queries. Visualizing the offsets for different queries could provide a clearer understanding of how QAO enhances the editing process. \n- The paper mentions tuning the editing strength and the number of edited heads based on empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "69IcyIy44e", "reviewer": "Reviewer_cVcN", "summary": "This paper addresses the issue of object hallucination in Large Vision-Language Models (LVLMs).The authors think  that they arises mainly from language bias and manifests as category, attribute, or relation errors. The authors propose AFTER (Adaptive Factual-guided Visual-Textual Editing for hallucination mitigation), a lightweight inference-time activation editing method. AFTER combines Factual-Augmented Activation Steering (FAS), which transforms ground-truth visual annotations into textual facts to provide positive, factual guidance, and Query-Adaptive Offset Optimization (QAO), which learns query-specific adjustments to better align visual-textual associations.", "questions": "As described in Weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The figures and illustrations are excellent — the visuals are carefully designed and aesthetically pleasing, and they greatly help in clarifying certain issues I encountered during reading.\n2. The writing is strong, and the appendix provides abundant experimental details, which convinces me that the reported results can be reproduced directly based on the information given in the paper.\n3. The motivation is solid and well-founded. Similar concerns have been raised in other works, and the paper rightly emphasizes that language bias may lead LVLMs to overlook important information in the images [1].\n\n[1] Jia H, Jiang C, Xu H, et al. Symdpo: Boosting in-context learning of large multimodal models with symbol demonstration direct preference optimization[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 9361-9371.", "weaknesses": "1. The proposed method is evaluated primarily on relatively simple datasets, whose difficulty appears notably lower than the complexity illustrated in Figure 1. It remains unclear whether the approach would retain its effectiveness on more challenging benchmarks, such as HallusionBench[1] or CRPE[2].\n2. The approach relies heavily on datasets like COCO, which are richly annotated by humans, and thus depends on the availability of high-quality manual annotations. Given that the current results are derived from COCO-like data, it is uncertain whether comparable performance could be achieved on domains where such annotations are scarce or costly—e.g., in medical imaging datasets.\n3. Regarding datasets such as BLINK[3], it is unclear whether the method could deliver strong generalization performance. In the current generalization experiments, COCO is evidently an in-domain dataset, making it difficult to assess how well the method would transfer to truly out-of-domain scenarios.\n\n[1] Guan T, Liu F, Wu X, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14375-14385.\n\n[2] Wang W, Ren Y, Luo H, et al. The all-seeing project v2: Towards general relation comprehension of the open world[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 471-490.\n\n[3] Fu X, Hu Y, Li B, et al. Blink: Multimodal large language models can see but not perceive[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 148-166.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qQ8VcrqFqr", "reviewer": "Reviewer_j2pv", "summary": "This paper tackles hallucination by leaning on straightforward factual textual cues, rather than only nudging activations or poking the image with perturbations. The approach first uses dense ground-truth annotations from the COCO training set to build three fact banks (category, attribute, relation). These facts are turned into text descriptions with an LVLM. For each description, the method creates trusted and untrusted pairs to get positive and negative activation examples, which yield an editing vector. A small offset estimator is then trained for Query-Adaptive Offset optimization, giving a query-specific offset at inference. In practice, it reduced hallucinations and lifts accuracy across POPE, MME, and AMBER on various LVLM.", "questions": "- What explains the drop in Cover, the metric for answer comprehensiveness?\n- Does the learned steering direction favor frequent classes at the expense of rare or novel ones?\n- How does the method handle compositional prompts with multiple objects, attributes, and relations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The trusted vs. untrusted pairing is a smart way to bootstrap supervision without manual labeling at scale.\n- Inference behaves like a lightweight adapter, so the method is efficient.\n- Results are strong across both discriminative and generative benchmarks.\n- Analysis is thorough, with useful ablations, including performance as the image pool grows from 50 to 500.\n- The out-of-distribution check is informative and suggests the method does not degrade under domain or QA-style shifts (discriminative to generative).", "weaknesses": "- The pipeline relies on ground-truth object annotations during preparation, which may be unavailable in some domains. In QAO, query-focused supervision depends on extracting objects from the question and checking membership against the COCO-derived category fact set T_c; if a queried object is not in T_c, they synthesize a negative textual sub-description. Without rich annotations or a compatible taxonomy, this step becomes brittle and hard to port.\n- Implementation detail: the method is model-specific, so you must recompute the steering vector and retrain the small offset estimator for each LVLM, which adds setup overhead and limits portability across architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lyB33AJ3um", "reviewer": "Reviewer_n12p", "summary": "This paper proposes an activation editing method AFTER to mitigate hallucination in vision-language models. It compromises two methods: FAS to provide factual and global guidance for activation editing, and QAO to provide query-specific guidance. Through training a lightweight MLP, they obtain the offset activation vectors, which is added to the original activations for correction. Results show that AFTER outperforms baselines and previous methods.", "questions": "1. Are you correcting activations for all layers? Since the shallow layers are more for perception and deeper layers for cognition, it’s more interesting to see the correction effect on different layers.\n\n2. The models used to generate training data are not at all mentioned.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing is mostly clear and easy to follow.\n\n2. The method seems effective and substantially outperforms baseline.", "weaknesses": "1. In FAS, how do you construct an $n$-question set for each textual description while ensuring that each of $n$ questions are indeed effective? Besides, what is the value of $n$ used in your experiments?\n\n2. What model is used to generate $t^{+}$? Also, despite $t^{+}$ is synthesized by a set of facts, hallucination can still persist in outputs of any LLMs, so validation of  $t^{+}$ is needed.\n\n3. Models and baselines compared are outdated (all in 2024) considering this is a submission in late 2025.\n\n4. The notation is quite confusing for  $t_{i,j}^{+}$ at the end of line 274. Correct me if I’m wrong: $i$ should be from 1 to $n$, denoting the $n$ questions associated with each sample; $j$ should be from 1 to $m$, which is the number of objects mentioned in the question $q_{i}$, not from 1 to $n$.\n\n5. In line 277, $z_{i}^{\\*}$ is obtained from $t_{i}^{*}$, which is a **set** of activations, but $z_{i}$ is a single activation. How is the subtraction between them possible?\n\n6. How does the selection of training sets impact the trained MLP for offset prediction? Since all of the four benchmarks (POPE, MME, AMBER, GQA) contain images from MSCOCO, it’s necessary to use a different set of images for training or benchmarks with different image sources to further prove the generalization of your method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)