PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs
Authors: Yixuan Wu
Yang Zhang
Jian Wu
Philip Torr
Jindong Gu
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal grounding
MLLM hallucination
alignment
Primary area: foundation or frontier models, including LLMs

Abstract: Multimodal Large Language Models (MLLMs) have shown remarkable performance in vision-language tasks, such as image captioning and visual question answering. However, these models often struggle with fine-grained visual understanding and are prone to hallucinations, primarily due to over-reliance on linguistic priors that distract them from leveraging actual visual information. This results in outputs that are often unanchored in the visual content, leading to errors. To address these challenges, we introduce MMGrounded-PostAlign, a post-multimodal alignment framework designed to enhance the visual understanding capabilities of MLLMs and mitigate hallucinations. In the framework, the visual grounding module identifies the referred objects in the image, while the textual grounding module generates the rationale for the final answer. This dual grounding approach ensures that outputs are firmly anchored in both visual and textual evidence. In particular, we incorporate a negative rejection mechanism within the visual grounding module to distinguish between grounded entities and non-existent objects influenced by linguistic biases. Moreover, we propose a selective reasoning mechanism within the textual grounding module to adjust the model’s reasoning strategy based on the complexity of the query. These innovations together work to resolve the issues associated with hallucinations and enhance the overall alignment between visual and textual modalities. Extensive evaluations on benchmarks such as POPE, HaloQuest, ReasonSeg, MME, and MMBench demonstrate significant improvements in fine-grained visual understanding and hallucination suppression, showcasing the effectiveness of our approach in real-world multimodal tasks.

Published as a conference paper at ICLR 2026
POSTALIGN: MULTIMODAL GROUNDING AS A CORRECTIVE LENS FOR MLLMS
Yixuan Wu!
Yang Zhang!
Jian Wu?
Philip Torr
Jindong Gu National University of Singapore
University of Oxford *Zhejjang University
wyx wyx_chloe@zju.ent
ABSTRACT
Multimodal Large Language Models (MLLMs) have shown remarkable perfor- mance in vision-language tasks, such image captioning and visual question answering. However, these models often struggle with fine-grained visual under- standing and are prone to hallucinations, primarily due to over-reliance on linguistic priors that distract them from leveraging actual visual information. This results in outputs that are often unanchored in the visual content, leading to errors. To ad- dress these challenges, we introduce MMGrounded-PP Postdllgg,, a post-multimodal alignment framework designed to enhance the visual understanding capabilities MLLMs and mitigate hallucinations. In the framework, the visual grounding mod- ule identifies the referred objects in the image, while the textual grounding module generates the rationale for the final answer. This dual grounding approach ensures that outputs are firmly anchored in both visual and textual evidence. In particular, we incorporate negative rejection mechanism within the visual grounding module to distinguish between grounded entities and non-existent objects influenced by linguistic biases. Moreover, we propose selective reasoning mechanism within the textual grounding module to adjust the model's reasoning strategy based on the complexity of the query. These innovations together work resolve the issues associated with ballucinations and enhance the overall alignment between visual and textual modalities. Extensive evaluations on benchmarks such as POPE, Halo- Quest, ReasonSeg, MME, and MMBench demonstrate significant improvements in fine-gra ane-grained visual understanding and hallucination suppression, showcasing the effectiveness of our approach in real-world multimodal tasks.
INTRODUCTION
Recently, the rapid development of Multimodal Large Language Models (MLLMs) has significantly advanced visual understanding by improving multimodal alignment between visual and textual representations. This multimodal alignment enables MLLMs to bridge the gap between modalities, leveraging large-scale vision encoders and pretrained language models to achieve remarkable results in tasks such as image captioning, visual question answering, and visually grounded dialogue (Guo
let al. 2025 Team 2024 Wang al. 2024c Achiam al. 2023 Liu al. 2024b 2023c Team
et al.
2023 Zhou et al.] [2025] Feng et al.] 2023 2024). However, as tasks grow more demanding in
terms fine-grained visual understanding and complex reasoning, these models often fail maintain
robust alignment between modalities (Liu et al. 2024a Bai et al. 2024 Zhou et al. 2023 Wu et al.
2024. revealing critical limitations that hinder their robustness and reliability.
A key factor behind these limitations is the model's ''s reliance on spurious correlations (Calude & Longo 2017: Ye et al.) (2024) instead of genuine causal or contextual relationships. Rather than interpreting visual content based on meaningful interactions, MLLMs often default to statistical associations, such as objects frequently appearing together in images. This problem further amplified by linguistic priors (Leng et al. 2024), which shape the model's predictions based on common textual patterns rather than actual visual evidence.
One major issue caused by this bias is hallucination (Liu et al.) 2024a Bai al.) 2024). where the model generates content that does not exist the image. Instead of relying on visual details,
"Corresoonding author.

---

Published as a conference paper at ICLR 2026
Co-occurrence hallucinatiio
What the object next the blankst the sofa?
MLLM
A dog.
Alignment
Visual Grounding & Textual Grounding
animal with & soft fur, floppy cars, and lorge which are typical features of dog.
cat.
MLLM
(a) Current MLLM
Multi-modal input
(b) MWMGrounded-PsstAllgg
Figure 1: (a) Current MLLMs often struggle with spurious correlations, leading to co-occurrence hallucinations, such as incorrectly identifying a cat on the sofa. (b) To address this, we propose MMGrounded-PostAAee which incorporates multimodal grounding module that enhances answer accuracy by aligning and anchoring the final output with actual visual and textual evidence.
selects words that are likely to appear together in text, even if they are incorrect in the given visual context. This reveals a lack of robust multimodal grounding. Another challenge is the over-reliance on high-level visual cues, Instead of recognizing fine-grained object attributes, MLLMs often focus on general patterns like backgrounds dominant colors. Linguistic priors encourage broad, text-driven generalizations that do not always align with the actual scene. As a result, the model struggles with precise visual understanding. MLLMs also face difficulties in complex reasoning tasks. When interpreting object interactions, answering compositional questions, or inferring causal relationships, they often lack logical consistency. Rather than constructing well-grounded multimodal responses, the model may rely on familiar text patterns, leading to reasoning failures that prioritize language biases over actual visual input.
To address these challenges, MLLMs need stronger multimodal alignment to ensure that textual outputs remain faithful to visual input. Specifically, we propose multimodal grounding as the corrective lens to mitigate these limitations and reduce reliance on spurious correlations. Multimodal grounding improves the alignment between modalities, helping the model establish more reliable connections between language and relevant sources of information. Rather than relying on high-level correlations or co-occurrence-basee embeddings, grounding can take various forms. It may link textual descriptions to visual regions, external knowledge bases, contextual cues. By reinforcing consistency across modalities, grounding helps mitigate hallucinations, reducing the impact of spurious correlations and improving output reliability.
In this paper, we propose MMGrounded-PP Poaddd-ottAAigg.. post-multimodal alignment framework designed to enhance the tne-grained visual understanding capabilities of MLLMs and reduce hallucinations caused by over-reliance on linguistic priors. Our framework incorporates a multimodal grounding module that provides two complementary types of evidence: visual grounding, which identifies the referred object in the image, and textual grounding, which generates an interpretable rationale before producing the final prediction. This dual-grounding design ensures that the model's outputs are firmly anchored in real visual cues and contextually appropriate textual reasoning. Within this framework, we further introduce negative rejection mechanism in the visual grounding module. This mechanism explicitly distinguishes grounded visual entities from nonexistent objects and enables the model to reject visual regions that are not supported by the image, thereby complementing multimodal grounding in suppressing hallucinations influenced by language priors. On the textual grounding side, we propose a selective reasoning mechanism that dynamically controls whether the model should engage in textual grounding. The key insight is that not all queries require explicit reasoning: for simple cases, the model can directly produce the final answer without constructing an intermediate rationale, whereas for complex cases, the model benefits from performing textual grounding to organize its reasoning steps. To realize this behavior, we introduce two routing signals, <SIMP LEE> and <COMPLEX>> which guide the model to selectively invoke textual grounding only when necessary. By integrating these components, our framework reinforces multimodal alignment through both visual and textual evidence, leading to more accurate visual understanding and substantially reduced hallucinations in MLLMs.
Our contributions are summarized follows:
We propose Settrrounddd-PostAlleeeee most-multimodal alignment framework that en- hances visual understanding in MLLMs by aligning visual and textual modalities through multimodal grounding, ensuring outputs anchored to actual visual and textual evidence.
2

---

Published as a conference paper at ICLR 2026
We introduce negative rejection mechanism in visual grounding to mitigate hallucinations, allowing the model to differentiate between grounded objects and hallucinated artifacts.
In textual grounding, we propose selective reasoning mechanism that adapts the model's s reasoning strategy based on the complexity of the imput query, improving the model's ability to handle reasoning tasks with various complex levels.
We evaluate our method on benchmarks such as HaloQuest, POPE, ReasonSeg, MMBennch MME, and RefCOCo, achieving significant improvements in visual understanding and hallucination suppression, while preserving the general reasoning capabilities of MLLMs.
RELATED WORK
2.1
VISION-LANGUAGE GROUNDING MODELS
Grounding in vision-language models involves establishing correspondences between textual descrip- tions and image regions (Xie et al. 2023). Early grounding methods (Yu et al. 2018; Yang et al. primarily adopted object detection pipelines, while later models (Kamath et al. [20]] Yao et al.
2019)
2022 Li al. 2022 Liu et al. (202cc) integrated vision-langue -aigunle pretraining for better end-to-end
performance. Beyond bounding box grounding, segmentation-ban based grounding (Wu et al. 2020
2023 2022) addressed coarse granularity by enabling pixel-wise segmentation for finer localization.
With the rise of MLLMs, grounding tasks have been integrated into generalist frameworks. Some models, like Kosmos-2 (Peng al. 2023) and Shikra (Chen et al.
(023b). formulate grounding
as text-based bounding box prediction, while others (Zhang et al.
2024a Zhao et al. [2023a Lai
al. 2024 Xia al., 2024 Rasheed al. 2024 Ren et al.! 2024 Zhang al. 2024b), integrate
separate grounding modules into MLLMs. These methods primarily focus on enhancing grounding accuracy within single-task framework, typically using MLLMs to control guide grounding. In contrast, our work does not aim to introduce a new grounding architecture. Instead, we repurpose visual grounding, together with an additional textual grounding component, as multimodal evidence for post-alignment. Rather than using MLLMs to perform grounding, we leverage grounding outputs to enhance MLLMs' visual understanding and mitigate hallucinations.
2.2 HALLUCINATION IN MLLMS
Hallucination in MLLMs, characterized by contradictions between image input and textual output, has been a prevalent issue (Liu et al. 2024a Bai et al. (024).. To alleviate hallucinated content, existing works can be divided into the following two directions.
The first focuses on post-processing
approaches, including post-hoc corrections (Zhou et al. 2023
Yin et 2024 Lee et al. 2023 Zhou al. 2023) and specialized decoding (Huang al. 2024
Chen et al. 2024 Leng et al. 2024 Zhu et al. 2024 Zhao et al.) 2024 Deng et al. 2024 Wang et al.
2024a). For example, Lee et al. (2023) introduce self-reviiee self-revising mechanism to reduce hallucination.
However, these methods often require increased inference time, limiting their generalizability and scalability across diverse data domains and model sizes.
The second line of work focuses on training-based methods. Some of them focus on data-level improvements, including the introduction of negative data (Liu et al. (2036), counterfactual data (Yu et al. 2024), and reducing noise in existing datasets (Wang et al. [2024b] Yue et al. 2024).. Reinforce- ment learning (RL) has also been explored to guide MLLMs toward hallucination-Ire outputs (Zhao) al. 2023b Li et al. 2023a Gunjal et al. 2024 [unn et al.) 2023). Instead of solely relying data augmentation or reinforcement learning, our method integrates a multimodal grounding module for stronger multimodal alignment, ensuring that model outputs firmly anchored in real evidence.
3
METHOD
In this section, present WeNGrronddd-PostAAlle post-multimodal alignment framework that integrates both visual grounding and textual grounding as multimodal evidence for correcting hal- lucinations in MILMs. Unlike prior grounding-oriented works, our goal is not introduce
3

---

Given an image I and a text query Q, the MLLM Published as a conference paper at ICLR 2026
Visual grounding output
Visual grounding enceder
Multi-task decoder
Mask
Mid timeaal Alignment
Visual gr prunndnng token generation
Loo
LoRA
Decode
To right the blankest, there is on animal with so fur, floppy ears, and large eyes, which are typical
4LOC.
What the object next to the blanket
Textual grounding taken genzration
Yes
sofa?
Multi-modal LLM
Features of a dog. A deg.
Figure 2: The pipeline of our proposed MMGrounnddd-PsstAiinn Given an image and text query, the MILM generates the tokens of visual grounding (<LOC>), textual grounding, and the final answer. The last-layer embedding of <LOC> is fed to multi-task decoder and then decoded into a segmentation mask and bounding box for the target object. When target object exists the image, the visual grounding token is replaced by <REJ>, which is directly assigned an empty mask and bounding box in the multi-task decoder. The textual grounding facilitates the generation of a rationale prior to the final answer for complex queries.
grounding architecture. Instead, we repurpose grounding outputs as a corrective lens: visual ground- ing provides image-gra wooggggrodndd cues reduce the MLLM's reliance linguistic priors, while textual grounding anchors the reasoning process with interpretable rationales. Together, these two signals enhance fine-grained visual understanding and guide the MILM toward evidence-aligned predictions.
3.1 FRAMEWORK OVERVIEW
As shown in Figure 2 framework comprises three key components: A MILM that processes image and text inputs to predict the tokens of visual grounding, textual grounding, and the final answer. (2) A visual grounding encoder that extracts image features. (3) multi-task decoder that performs visual grounding of segmentation mask and detection bounding box, using the visual grounding token as prompts.
Multi-modal Input and Structured Output. tasked with generating a structured output A, as:
A = MLLM(I, Q)
(1)
A= (V,TFF)
(2)
where V denotes the visual grounding token, T represents the textual grounding token for rationale, and F is the final answer token.
Visual Grounding Prediction. The last-layer embedding of the visual grounding token <LOC> extracted from the MILM and passed through MLP projection layer obtain the prompt embeddings for the subsequent visual grounding. Meanwhile, the visual grounding encoder extracts dense visual features from the input image. These dense visual features, along with the prompt embeddings, are fed into the multi-task decoder, which generates the final visual grounding outputs.
Grounding-Augmente Final Answer Generation. The visual grounding outputs and textual grounding rationales act as implicit constraints that guide the MLLM's final answer generation. The textual grounding provides structured framework for generating interpretable rationales, which, when integrated with the visual grounding cues, anchor the model's reasoning in both visual and linguistic contexts. This guidance helps mitigate the model's over-reliance linguistic priors, enhancing its ability to perceive and understand visual information more accurately.
3.2 REJECTING NON-EXISTENT OBJECTS
Despite the advanced understanding capabilities of MLLMs, they are often prone to hallucinating non-existent objects due to strong linguistie priors and co-occurrence biases present in the training
4

---

Published as a conference paper at ICLR 2026
data. These hallucinations manifest in various forms, including incorrect attributes, inaccurate spatial locations, or entirely fabricated objects.
To mitigate this, introduce negative rejection mechanism, which enables the model to explicitly reject non-cxistent objects rather than hallucinating their presence. Specifically, if the referents absent from the image, we enforce the MILM to predict special <REJ> token to replace the corresponding <LOC> token. In the multi-task decoder, the predicted <REJ> tokens are directly assigned empty masks and bounding boxes, effectively bypassing the decoding process. The primary advantage of the negative rejection mechanism is its ability to mitigate the model's 's over-reliance on linguistic priors. Many MLLMs tend associate high-frequee frequncyyy co-occurring objects (e.g.. predicting a "table" when "chair" is present) and produce incorrect outputs. By explicitly learning to reject such negative samples, the model becomes more capable of distinguishing between genuine visual grounding and misleading language priors. Additionally, incorrect attributes and spatial misplacement, such as predicting a "red apple" when only green apples are present, are also suppressed, as the model is penalized for assigning masks or bounding boxes to absent entities.
To further reinforce the model's ability to reject hallucinated objects, we introduce a negative rejection loss, which explicitly penalizes incorrect grounding predictions when the referent is absent. The loss is defined as:
Negative Rejection Loss.
Caj Σ
vr
log roj +(1 v/81 log(1 p(o)]
(3)
where v; 10 denotes whether the referent in sample i should be rejected.
3.3 SELECTIVE REASONING TEXTUAL GROUNDING
Previous work (Zhang et al. 2023) has shown that generating intermediate rationales through textual grounding improves the reasoning capabilities of MLLMs, leading to improved answer accuracy for complex tasks. In our multimodal queries require explicit rationale generation via textual grounding. Instead, selective approach needed to balance the visual output from the visual grounding module with the textual answer generation of the MILM. Our experiments, shown in Table also support this hypothesis. To this end, we propose a selective reasoning mechanism, which allows the MLLM to determine whether explicit contextual reasoning is needed. Specifically, during training, categorize queries into <SIMPLE: LE> and <COMP LEX> types:
sooudding-asssster MLLM framework, we argue that not all
For simple queries, the MLLM directly outputs the visual grounding token and the final answer, skipping rationale generation. For example, for the simple query: "User: <IMAGE> What color is the in this image?" "Assistant: It is <LOC>, <SIMPLE>. The car is red."
For complex queries, the MLLM's output includes the visual grounding token, the con- textual rationale, and the final answer. For example, for the complex query: "USER: <IMAGE> Which food in the picture contains the most protein?" "ASSISTANT: It <LOC>. <COMPLEX>. This image contains oranges, eggs, vegetables, and buns. Among them, eggs are the richest protein."
To enable automatic complexity assessment during inference, we incorporate self-reflection prompting into the user instructions:
Given an image and text query, first assess whether answering the query requires a rationale. If the answer is directly observable from the image without additional reasoning, classify it as <SIMPLES and provide only the final answer. If answering requires logical inference or contextual understanding, classify it <COMPLEX> and generate rationale before providing the final answer.
Selective Reasoning Loss. We also design a selective reasoning loss to supervise the model in identifying whether query requires complex reasoning. Each query is classified either <STMPLE>>
5

---

Published as a conference paper at ICLR 2026
CCOMPLEX>, and the loss is computed as:
N
Ccasson
M tdh log pf + (1- y/=h) log( )] )]
where v E {0, 13 indicates whether query requires generating a rationale.
3.4 TRAINING OBJECTIVE
In summary, we finetune the MLLMs using LoRA Hu et al. (2022) while jointly optimizing the multi-task decoder. The overall loss function is defined as:
C = Coj + A2 + ground
Liext
(5)
where the visual grounding loss Lerouund consists of a joint optimization of detection loss Lder and segmentation loss Loog. formulated as:
Ldtt Lsawwtt-LL (babux: Ubbux) + LatoU (ybbuxx Ybbux)
(6)
Eseg LBCE (umask Ymusk) LDICE (imask Ymnsk))
and Liesi is the cross entropy language modeling loss, defined as:
Luxa LLM( Yea)-
(8)
EXPERIMENT
4.1 SETUPS
Training Data Formulation. We construct diverse multimodal training dataset, where each instance annotated with reasoning-type token: <SIMP LE> or <COMP LEX>, The <SIMPLE> label denotes low-complexity queries requiring minimal reasoning, while <COMPLEX> is used for tasks involving higher-order or indirect reasoning. Additionally, negative samples are included with the <REJ>> token, indicating the absence of visual referent. Detailed data construction and labeling procedures are provided the Appendices.
Network Architecture. Our framework is built upon LLaVA-1.5-78 -LAVA-I.5-133 (Liu et al. 2024b), Qwen2-VL-27 VL-7B (Wang et al. 2024c), Qwen2.5-VL-7 VL-BB (Bai et al. 2025). Intern VL3-14B (Zhu a and Intern 1LB (Wang et al. 2025) the MLLM backbone, and ViT-H SAM (Kir- illov et al. 2023) as the visual grounding backbone. To preserve the pre-trained MILM's knowledge, we employ LoRA (Hu al. 2022) for parameter-efficient fine-tuning, while freezing the visual grounding encoder. The multi-task decoder, along with the LLM token embeddings, LLM head, and projection layer, is fully fine-tuned. The decoder includes both the mask and bounding box decoders. The hidden embeddings of visual grounding tokens (e.g., <LOC>) serve as prompts for SAM, conditioning its mask decoder to predict object masks, while a lightweight MLP regressor predicts bounding box coordinates from SAM-extracted features.
4.2 MAIN RESULTS AND ANALYSIS
In this section, we present results across three benchmark categories: (1) hallucination datasets (Halo- Quest (Wang al. 2024d) and POPE (Li al., (023b)), ) (2) generalization and reasoning datasets (MMBench (Liu et al. 2024d), MME (Liang et al. )024)), and (3) grounding tasks (RefCOCO series (Kazemzadeh et al. 2014) and ReasonSeg (Lai al. 2024h). Our method demonstrates that, while preserving the general reasoning capabilities of MLLMs, it not only mitigates hallucinations but also enhances visual understanding. We then highlight five key findings that identify the primary sources of hallucinations and demonstrate how method addresses them.
Finding 1: Linguistic Priors Override Visual Information MLLMs. Figure 3 investigates how MLLMs rely on linguistic priors over visual inputs. We analyze layer-wise wise token probabilities during decoding in (a)-(b) and the impact visual input on token generation in (c)-(d) (see Appendices for details).

---

Published as a conference paper at ICLR 2026
Tokon Probability across Layers
Token Probability
Describe this image.
Last Layer
caick
Lest Layer
User
Probestalln
(a)
LLayr
Layer
(b)
Proportion Overtap
The image depicts a spacious living room connected to a dining area, featurrng large TV on the left, a bottle on a coffee table, and a dining table surrounded by multiplé chairs. The dining with potted plonts, while woman stands near the window, adding sense of warmth to the scene.
Hollucination Token Overiap withwwthout Image
with/without Image
Different Hallucination
25.8%
Prodecion
Hall Aufaccaaioo
Wi-h
is decorated
Hol buchnattiin
Persietant Hallucination (89.1%)
(d) Figure 3: (a)(b) Token probability distributions across transformer layers, showing distinct trends for hallucinated (pink) and non-hallucinated (green) tokens. (c) Removing visual input yields an 89.2% overlap in hallucinated tokens, supporting the hypothesis that hallucinations arise from linguistic biases rather than visual misinterpretation.
(c)
In (a) and (b), we examine the Top-4 tokens ranked by probability in the final decoding layer. Non- hallucinated tokens like "woman" have high probabilities from the 20th layer, while hallucinated tokens like "bottle" emerge around the 30th layer. Notably, in (a), the probability the ground-truth token "vase" drops sharply after the 25th layer, falling below hallucinated tokens. We hypothesize that linguistic priors gradually suppress visual information, steering the model towards language-driven generation.
In (c) and (d), we validate this by removing the image input and analyzing hallucinated token persistence. Across 500 MSCOCO images, the hallucinated token overlap rate is 89.2%. In (d), when the image input is removed, 89.2% of hallucinations persist, confirming that linguistic priors dominate hallucination generation, often overriding visual evidence.
Finding 2: Explicit Visual Grounding Mitigates Hallucinations. To investigate the extent to
which the visual grounding mechanism enhances MLLM's visual understanding and mitigates visual hallucinations, we first conduct an ablation study on the HaloQuest benchmark. As shown in Table [ the first row presents the baseline where the visual grounding module entirely removed. The results indicate that the baseline exhibits poor performance, particularly in the False Premise and Insufficient Context categories. Introducing the mask grounding (denoted as <SEG>) and the bounding box grounding (denoted as <DET>) significantly improves performance. Furthermore, when both mask grounding and bounding box grounding are incorporated, the MLLM achieves even better results. Notably, the introduction of the <REJ> token for negative rejection further suppresses hallucinations, especially in the False Premise and Insufficient Context categories. This is primarily because these categories contain substantial number of anti-concept scenarios. For example, question such as "What breed of dog is in the image?" may have ground truth response of "There is no dog in the image.", which directly evaluates the MLLM's ability reject misleading premises.
Additionally, as shown in Table 2 we further investigate different grounding strategies. A common approach to integrating MLLM with grounding mechanisms is treating Bounding boxes as language Tokens for unified Learning (BTL). Based on this strategy, we explore two training paradigms: (1) BTI-Generation: The input consists of an image and a referring text, while the MLLM directly generates the bounding box coordinates. (2) BTL.-Cpttio: The input is an image, and the output is a caption describing the image content, where the target object's bounding box is embedded within the generated text. Here, baseline refers to our proposed framework with the visual grounding module removed, while retaining other designs (i.e., selective reasoning). The results on the POPE hallucination benchmark indicate that incorporating gains, while BTL-Caption provides moderate improvement. Combining both approaches further enhances performance. However, our proposed explicit visual grounding module significantly
does not yield performance
7

---

Published as a conference paper at ICLR 2026
Table I: An ablation study evaluates the impact visual grounding tokens (<LOC>) on the HaloQuest benchmark Wang et al. (2024d), where LLaVA-1.5-7B serves the MLLM baseline. Here, <SEG> and <DET> are specialized types of <LOC> tokens, where <SEG> generates segmentation masks and <DET> produces bounding boxes. The results show that combining <SEG> and <DET> enhances visual understanding, while <REJ> effectively suppresses hallucinations in misleading or insufficient context scenarios.
Method
False Premise
Visually Challenging
Insufficient Context
<SBG>
<DET>
<REJ>
Human Eval Auto-Eval Human Eval Auto-Ewe Human Eval Auto-Ewt
x
X
X
2.0
2.3
23.5
23.0
2.5
1.7
X
X
7.2
30.1
30.6
7.4
8.2
X
/
X
8.2
8.9
31.1
31.7
6.6
7.4
/
X
9.9
10.5
33.9
35.0
9.9
11.6
/
33.2
33.9
38.3
37.2
31.4
32.2
Table 2: Experiments the POPE, MME, and MMBench. Results show that explicit visual
grounding module outperforms BTL-based methods by enforcing stronger multimodal alignment.
POPE
Li et al.
MME
MMBench Liu et al.
Ran
Pop
Adv
(Liang et al.
EN
CN
LLaVA-1.5-7B
83.3
80.1
78.2
1504.6
62.2
57.7
+ BTL-Generation
82.7
80.3
79.2
1489.4
59.2
54.2
+ BTL-Caption
84.5
81.0
79.9
1499.6
59.7
54.5
LLaVA-I.--PottAlignn-B
86.6
84.2
82.3
1514.3
63.9
58.7
LLaVA-I.5-13B
85.4
82.2
79.2
1517.4
66.8
62.2
+ BTL-Genertioon
84.4
80.9
78.3
1501.7
65.9
62.4
+ BTL-Caption
85.1
81.7
78.8
1509.2
65.2
61.4
LL M.VAA-L5-PPstAAign--1
88.9
87.3
85.6
1520.3
68.9
63.2
Qwen2-VL Twen2-VL-7B
88.9
86.8
84.6
1717.4
82.4
79.4
Qwen2-VL.- Vl.-PostAlign-B8
90.3
89.2
87.1
1729.9
83.9
80.2
Qwen2.5-VI. VL
87.3
85.1
83.4
1736.8
82.1
82.3
Dwen2.5-VL--PostA-
89.7
87.9
85.2
1750.2
83.3
82.9
Intern VL3-14B
89.1
87.2
84.3
1762.8
84.3
83.1
Intern VL.3-PsttAlign-Iee
91.2
89.0
86.6
1772.3
85.5
84.3
Intern I 5L
87.1
85.4
83.8
1792.2
82.9
82.1
90.6
88.5
86.2
1906.9
84.1
83.4
outperforms these methods. By enforcing a higher degree of multimodal alignment and effectively rejecting negative samples, our method substantially reduces object and attribute hallucinations.
Finding 3: Our MILMM Retains Strong Reasoning and Generalization Abilities. In Table 2 we further investigate whether explicitly incorporating visual outputs affects the general reasoning capabilities of MLLMs. To evaluate this, test method several widely-used datasets for generalization and reasoning, including MME, MMBench, and POPE. Our results reveal that our post- alignment method does degrade the model's reasoning ability compared to the baseline MI.LM. In contrast, we observe that the BTT--Generation training paradigm results in a notable decline the baseline MLLM's reasoning ability. This suggests that the approach used in BTL-Generation may inadvertently bias the model towards overfitting to visual bounding box information at the cost of abstract reasoning, leading to drop in generalization performance.
Finding 4: Selective Reasoning in Textual Grounding Optimizes Efficiency and Accuracy. Previous work (Zhang al.) (2023) has shown that generating intermediate rationales through textual grounding improves the reasoning capabilities of MLLMs, leading to improved answer accuracy for complex tasks. In Table g we compare three textual grounding strategies: (1) Pre-Reasoning: A separate MILM (the same main MILM) generates rationale, which is then fed to the main
MLLM for final answer generation. (2) Inter-Reasoning:: The main MLLM integrates reasoning, simultaneously generating rationale and final answers for each query. (3) Selective-Reasoningg Reasoning: Dynamically assesses query complexity and generates rationale only for complex queries. Here, baseline refers our proposed framework with the selective reasoning strategy removed, while retaining the visual grounding module. Experimental results indicate that pre-reasoning strategy achieves better performance than inter-reasoning strategy; However, it involves multiple inference
8

---

Published as a conference paper at ICLR 2026
Table An ablation study on ReasonSeg Lai et (2024) shows that selective reasoning textual grounding performs best by adapting to different query complexity, which avoids overthinking on simple queries and ensures sufficient reasoning for complex ones.
Easy
Medium
Hard
Method
gloU
cloU
gloU
cloU
gloU
cloU
LLaVA-1.5-7BB + SAM- VIT-H
67.7
66.4
51.2
50.2
47.0
46.3
pre-reasoning (PR)
67.3
66.7
57.2
58.1
57.0
58.3
inter-reasoning (IR)
64.3
64.7
55.5
56.3
53.9
54.8
selective reasoning (SR)
68.9
67.2
58.9
57.2
57.2
57.7
LLaVA-1.5-133 SAM-ViT-H
69.2
70.3
55.2
56.2
51.7
52.2
pre-reasoning (PR)
69.7
69.3
62.7
62.2
61.9
61.2
+ inter-reasoning (IR)
67.2
68.1
60.9
59.2
58.2
57.2
+ selective reasoning (SR)
70.8
71.3
64.2
65.2
62.9
63.8
IJa + SAM-ViT-B SR
67.6
66.0
57.3
55.5
55.9
56.3
LLaVA-1.5-7B + SAM-ViT-L- SR
68.5
66.9
58.2
56.8
57.0
57.1
LLaVA-1.5-78 SAM-VTT-H+
68.9
67.2
58.9
57.2
57.2
57.7
Table 4: Performance comparison on the REC and RES tasks.
RefCOCO REC
RefCOCO+ REC
RefCOCOg
Models
RES
REC
RES
Kosmos-2 (Peng al.
2023
52.3
45.4
60.5
LISA-7B (ft) [Lai et al.
2024
74.9
65.1
67.9
LLLVASeg-77B (tt) Yang cl a
2024
76.2
65.7
69.8
MiniGPT v2-7B (Chen et al.
202.a
88.0
79.5
84.1
Shikra-7B (Chen
20)
87.0
81.6
82.2
Ferret-7B You al. 2023)
87.4
80.7
83.9
LLaVA-Groundne /B (Zhang et al.
2024a)
89.1
77.1
81.6
68.7
84.8
71.5
VisionLLM v2 [Wu
(0255
87.9
76.6
77.6
64.5
82.9
70.7
PixeILLM (Ren et 2024)
73.0
66.3
69.3
GLaMM (Rasheed
2024
79.5
72.6
74.2
GSVA-7B (ft) [Xta ct al.
2024
86.2
77.2
72.8
65.9
81.5
72.7
88.2
77.9
78.4
68.2
83.3
73.2
LLa VA-1.5 1--ot...--- 13B
89.2
79.7
80.1
70.9
85.3
74.8
rounds, which reduces overall efficiency. Our proposed selective-reasoning approach not only maintains strong performance but also operates with single inference round. Also, we augment the original benchmark queries into three difficulty levels (easy, medium, hard) to study the relationship between reasoning strategies and query complexity (see Appendices for details). We observe that for simple queries, excessive reasoning can slightly degrade performance due overthinking. Conversely, for complex queries, explicit reasoning necessary to ensure accurate predictions.
Finding 5: Our Architecture Provides Potential Approach for Visual Grounding with MLLMs. We also evaluate our method's visual grounding performance on the RefCOCO series and ReasonSeg benchmarks. The results surprisingly show that our framework not only improves the MLLM's visual understanding through the visual grounding module, but also that the MLLM can assist the visual grounding module in enhancing its zero-shot grounding capabilities. This suggests a potential approach for visual grounding empowered by MIL.Ms. However, it is important to emphasize that the primary goal of our work is not to optimize for visual grounding tasks directly. Instead, we aim leverage multimodal grounding (including both visual and text grounding) as evidence to suppress hallucinations in MLLMs and improve their visual understanding. Consequently, we do not increase the amount of grounding -peccific training data to further optimize performance on grounding benchmarks. In Table 4 for the REC and RES tasks, our approach achieves competitive results compared specialized MI.LM based grounding models. This highlights the superior pixel-level grounding capabilities of our method. Additional results on other visual grounding tasks can be found in the Appendices.
4.3 QUALITATIVE ANALYSIS

---

Published as a conference paper at ICLR 2026
To investigate how grounding influences the model's attention to image features, we visualize the average attention weights assigned to image features over 500 MSCOCO samples during response generation under two conditions: without grounding and with ground- ing. To emphasize the most salient attention patterns and reduce noise, we apply Top-k Mean Pooling at three levels: For each output token, we select the top-3 highest attention values across all hidden layers and compute their mean. We then aggregate the top-3 at- tention values across all self-attention heads for each token. Finally, to ensure consistency between the two conditions, we average the top-1 attention values across output tokens, where is the minimum number of tokens generated between the without grounding and with grounding cases. The aggre- gated attention maps in Figure 4 show clear distinction between the two conditions. The without grounding attention map exhibits sparse and scattered attention. In contrast, the with grounding map shows significantly more concentrated attention, particularly over relevant image areas. This suggests that grounding effectively guides the model to align textual outputs with pertinent visual features, resulting in more visually grounded responses.
Without grounding
With grounding
Figure 4: Average attention to image features without (left) and with grounding (right).
CONCLUSION
In this paper, we propose SiGGrounded-PostAligee post-multimodal alignment framework designed to enhance the visual understanding capabilities of MILMs. Our approach addresses the challenges of over-reliance on spurious correlations by integrating grounding module that enables both visual and textual grounding. This ensures that the model's outputs are firmly anchored in actual visual and textual evidence. We propose negative rejection mechanism within the visual grounding module to mitigate hallucinations caused by linguistic biases, helping the model distinguish grounded objects from non-existent ones. Additionally, our selective reasoning in textual grounding adapts the model's reasoning strategy based on query complexity, allowing for more accurate and contextually relevant responses for tasks of varying complexity levels. Extensive evaluations on POPE, HaloQuest, MME, MMBench, ReasonSeg, and RefCOCO demonstrate the effectiveness of our approach in improving visual understanding, reducing hallucinations, and enhancing grounding accuracy.
ACKNOWLEDGMENTS
This research partially supported by National Key Research and Development Program of China under Grant No. 2025YFC2423903, National Natural Science Foundation of China under grants No. T2541004, Zhejiang Key R&D Program of China under grant No. 2025C02120, and State Key Laboratory of Transvascular Implantation Devices under Grant No. SKL.1ID2024003.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 230::2333308774 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 2502. 13923, 2025.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint 10X1v:2404.18930. 2024.
Cristian S Calude and Giuseppe Longo. The deluge of spurious correlations in big data. Foundations of science, 22:595-612, 2017.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large
10

---

Published as a conference paper at ICLR 2026
language model as unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023b.
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403000425, 2024.
Ailin Deng, Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding. arXiv preprint arXiv:2402.15500 2024.
Yujie Feng, Zexin Lu, Bo Liu, Liming Zhan, and Xiao-Ming Wu. Towards lIm-driven dialogue state tracking. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 739-755, 2023.
Yujie Feng, Xu Chu, Yongxin Xu, Guangyuan Shi, Bo Liu, and Xiao-Ming Wu. Tasl: Continual dialog state tracking via task skill localization and consolidation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1266-1279, 2024.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, pp. 18135-18143, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Qidong Huang, Xiaoyi Dong. Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang. and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and eetrospectionnallocatiee on Computer Vision and Pattern Recognition, pp. 13418-13427, 2024.
In Proceedings of the IEEE/CVF Conference
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, Nicolas Carion, and Armand Joulin. Mdetr: Modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 1780-1790, 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp- 787-798, 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF international conference computer vision, pp.
2023.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9579-9589, 2024.
Seongyun Lee, Sue Hyun Park, Yongrae Jo, and Minjoon Seo. Volcano: mitigating multimodal hallucination through self-feedback guided revision. arXiv preprint arXiv:2311.07366 2023.
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chumyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision- languge models through visual contrastive decod- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13872-13888 13822, 2024.
11

---

Published as a conference paper at ICLR 2026
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang. Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665 2023a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-ing -mage pre- training. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10965-10975, 2022.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7061-7070, 2023.
Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference Computer, Artificial Intelligence and Control Engineering, pp. 405-409, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision- ECCV 2014 4: 13th European conference, zurich, Swifzerland, September 6-12, 2014, proceedings, part 13, pp. 740-755. Springer, 2014.
Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 13592-236011 2023a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating halluci- nation in large multi-modal models via robust instruction tuning. arXiv preprint arXiv2206.14500 2023b.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. survey on hallucination in large laiguaee models. arXiv preprint arXiv:2402.002:: 2024a.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3444 34916, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15296-26306, 2024b.
Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pp. 38-55. Springer, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangho Zhao, Yike Yuan, Jiaqi Wang, Conghui He. Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233, Springer, 2024d.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
12

---

Published as a conference paper at ICLR 2026
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13009-13018 2024.
Yang, and Fahad Khan. Glamm: Pixel
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pp. 26374-26383, 2024.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint
vrXiv:2405.09818 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Chenxi Wang, Xiang Chen, Ningyu Zhang, Bozhong Tian, Haoming Xu, Shumin Deng, and Huajun Chen. Milm can see? dynamic correction decoding for hallucination mitigation. arXiv preprint arXiv:2410.11779, 2024a.
Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. Mitigating tiee-grained hallucina- tion by fine-tuning large risionIaanguag models with caption rewrites. In International Conference on Multimedia Modeling, pp. 32 45. Springer, 2024b.
Peng Wang. Shuai Bai. Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/91, 2024c.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508. 18265, 2025.
Zhecan Wang, Garrett Bingham, Adams Wei Yu, Quoc V Le, Thang Luong, and Golnaz Ghiasi. Haloquest: A visual hallucination dataset for advancing multimodal reasoning. In European Conference on Computer Vision, pp. 288-304. Springer, 2024d.
Chenyun Wu, Zhe Lin, Scott Cohen, Trung Bui, and Subhransu Maji. Phrasecut: Language- image segmentation in the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 10216-10225, 2020.
Jiannan WU, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4974-4984, 2022.
Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds laagua-language tasks. Advances in Neural Information Processing Systems, 17:69925-69977 2025.
Yixuan Wu, Zhao Zhang, Chi Xie, Feng Zhu, and Rui Zhao. Advancing referring expression segmentation beyond single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2628 26382638.111 2023.
Yixuan Wu, Yizhou Wang, Shixiang Tang. Wenhao Wu, Tong He, Wanli Ouyang, Philip Torr, and Jian Wu. Dettoolchain: A new prompting paradigm to unleash detection ability of mllm. In European Conference on Computer Vision, pp. 164-182. Springer, 2024.
Zhuofan Xia, Dongehen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3858-3869, 2024.
13

---

Published as a conference paper at ICLR 2026
Chi Xie, Zhao Zhang. Yixuan Wu, Feng Zhu, Rui Zhao, and Shuang Liang. Described object detection: Liberating object detection with flexible expressions. Advances in Neural Information Processing Systems, 36.79095-79107, 2023.
Senqiao Yang, Tianyuan Qu, Xin Lai, Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. Lisat+: An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:23/2.17244, 2023.
Yuqi Yang, Peng-Tao Jiang, Jing Wang, Hao Zhang, Kai Zhao, Jinwei Chen, and Bo Li. Empowering segmentation ability multi-modal large language models. arXiv preprint 24Gi:22003111441, 2024.
Zhengyuan Yang, Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1993--663,20000
Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang. Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: Dictionary-enriched visual-concept paralleled pre-training for open- world detection. Advances in Neural Information Processing Systems, 35:9125-9138, 2022.
Wenqian Ye, Guangtao Zheng, Xu Cao, Yunsheng Ma, and Aidong Zhang. Spurious correlations in machine learning: A survey, arXiv preprint urXiv:2402.12715. 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences, 67(12):220105, 2024.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1307-1315, 2018.
Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22944-12955 2024.
Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an eos decision perspective. arXiv preprint arXiv:2402.1 1454555500.. 2024.
Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Leizhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. In European Conference on Computer Vision, pp. 19-35. Springer, 2024a.
Yichi Zhang, Ziqiao Ma, Xiaofeng Gao, Suhaila Shakiah, Qiaozi Gao, and Joyce Chai. Groundhog: Grounding large language models to holistic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14227-14238, 2024b.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning language models. arXiv preprint arXiv:2302.00923; 2023.
Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision-language models via classifier-free free guidance. arXiv preprint arXiv:2402.0888 080.00880, 2024.
Yang Zhao, Zhijie Lin, Daquan Zhou, Zilong Huang, Jiashi Feng, and Bingyi Kang. Bubogpt: Enabling visual grounding in multi-modal 1lms. arXiv preprint arXiv:2307.0885)) 2023a.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucimations: Enhancing Ivims through hallucination-awweee aware direct preference optimization. arXiv preprint 1.tr:21111.68333 2023b.
14

---

Published as a conference paper at ICLR 2026
Yiyang Zhou, Chenhang Cui, Jachong Yoon, Linjum Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00755. 2023.
Zhihan Zhou, Feng Hong. Jiaan Luo, Jiangchao Yao, Dongsheng Li, Bo Han, Ya Zhang, and Yanfeng Wang. Learning to instruct for visual instruction tuning. arXiv preprint arXiv:2503.222
2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
Lanyun Zhu, Deyi Ji, Tianrun Chen, Peng Xu, Jieping Ye, and Jun Liu. Ibd: Alleviating hallucinations in large vision-language models via image- biased decoding. arXiv preprint arXiv:2402. 18476, 2024.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp-
15127, 2023a.
Xucyan Zou, Jianwei Yang. Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jac Lee. Segment everything everywhere all once, Advances in neural information processing systems, 36: 199799997777222 2023b.
15

---

Published as a conference paper at ICLR 2026
A TRAINING DATA AND LABELS
Our training data composition is summarized in Table The annotation of each query with either <SIMPLE> or <COMPLEX> is based on the inherent reasoning difficulty and task design of the source dataset. Queries originating from datasets such COCO-Stuff, ADE20K, Mapillary Vistas, PACO-LVIS, and COCO are typically straightforward and refer to concrete categories without requiring complex reasoning; thus, they labeled <SIMPLE>. On the other hand, datasets like ReasonSeg, RefCOCO, RefCOCO+, RefCOCOg, RefCLEF, gRefCOCO, Visual Genome, and LLaVA-v1.5-mix665 contain expressions that demand contextual reasoning, ambiguity resolution, Furthermore, to train the
or commonsense inference, and are therefore annotated as <COMP model explicitly reject invalid or hallucinated referents, we incorporate negative examples marked with the <REJ> token. These negative samples are sourced from the gRefCOCO dataset, which includes 32,202 expressions that refer non-existent targets in the image. This enables the model to learn to distinguish between visually grounded content and spurious linguistic cues. All samples are formatted following the LLL VA-tlle instruction tuning template to facilitate effective multimodal alignment.
Table 5: Overview of Tasks, Reasoning Token, and Datasets.
Tusk
Reasoning Token
Datasets
Semuntic Segmentation Referring Segmentation
<SIMPLE> <COMP1.R>>
COCO-Stuff, ADE20K, Mupillary, LVIS-PACO
ReasonSeg, RefCOCO, Re(COCO+, Re/COCOg, Re(CLEF, gRe(COCO
Object Detection
<SIMPLE>
COCO
Referring Comprchension
<COMPLEX>
Visual Genome, RefCOco, RefCOCO+, ReCCOCOg ReFCLEF, gRefCOCO LLaVA-v1.5-mix666
Dialogue
<COMPEXX>
B HALLUCINATED TOKEN ANALYSIS
Hallucinated Token Extraction. To investigate the influence of linguistic priors on hallucinations in MLLMs, we randomly sample 500 images from the MSCOCO dataset [Lin et al. (2014). For each image, we generate open-ended responses using the prompt, "D "Describe the image. in order to elicit detailed descriptions from the model. These responses often contain object tokens that are not present in the image, which define as hallucinated object tokens. A hallucinated token is any object that is mentioned in the generated text but does not correspond to any visual evidence in the image. The process of identifying hallucinated tokens involves few key steps. First, we scan the generated response for object names (such as "car", "dog", "table", etc). Next, we cross-reference each identified object token with the objects present in the corresponding image using an object detection model. If an object mentioned in the response is not detected in the image, we categorize it as a hallucinated token.
Moreover, we also extract the textual context immediately preceding each hallucinated token. For example, the model generates the phrase "Additionally, there car." but no car is visible in the image, the hallucinated token would be "car", and its preceding context would be the phrase "Additionally, there a".
Hallucinated Token Overlap Analysis. In order to quantify the influence of linguistic priors on hallucinations, we perform experiment in which we re-input only the preceding textual context (without any visual input) into the model. The key metric, hallucinated token overlap rate, is calculated as the proportion of hallucinated samples in which the hallucinated tokens from the text- image combination also appear in the set generated from the preceding text alone. This overlap rate gives us an insight into how much influence the linguistic priors have on the hallucination generation process, independent of visual evidence. The hallucinated token overlap rate is computed as:
Number of hallucinated samples without image input Total number of initially hallucinated samples
Overlap Rate
(9)
A higher overlap rate indicates that hallucinated tokens persist even when the image is removed, suggesting that these hallucinations are more heavily influenced by the linguistic priors within the model rather than by the visual evidence.
16

---

Published as a conference paper at ICLR 2026
Table 6: Performance comparison on ReasonSeg Lai et (2024) benchmark.
val
test
Method
short query
long query
overall
gloU
cloU
gloU
cloU
gloU
cloU
gloU
cloU
OVSeg Liang al.
2023
28.5
18.6
18.0
15.5
28.7
22.5
26.1
20.8
GRES Liu
2023a
22.4
19.9
15.0
22.6
21.3
22.0
X-Decoder (Zou al.
2023a
22.6
17.9
20.4
11.6
22.2
17.5
21.7
16.3
SEEM (Zou cf al. 2023b
25.5
21.2
20.1
11.5
25.6
20.8
24.3
18.7
Grounded-SAM [Luu ct
2024c)
26.0
14.5
17.8
10.8
22.4
18.6
21.3
16.4
LISA-7B Lai
20.24]
44.4
46.0
37.6
34.4
36.6
34.7
36.8
34.1
LISA-7B (1t) el
2024
$2.9
54.0
40.6
40.6
49.4
51.0
47.3
LISA-13B Lai al. 2024
48.9
46,9
39.9
43.3
46.4
46.5
44.8
45.8
LISA-13B (TT) Lai ct al.
2024)
56,2
62.9
44.3
42.0
54.0
54.3
51.7
51.1
LISA++ LL
Yang et al 2023
64.2
68.1
49.6
51.1
59.3
61.7
57.0
59.5
LLaVASeg-7B Yang
(024)
54.8
49,9
LaaVASe-113B Yang et a (2U24
59.1
52.8
GSVA-7B (ft) Xia (2024)
50.5
56.4
GROUNDHOOG-7BB2 Zhang al. 2024b)
56.2
MMGrounded Postlllgg--tttttt---1
58,3
59.9
44.5
43.9
54.1
55.7
52.0
$2.9
CCGGroudded-PostAlignne
(ft)
64.4
66.2
49.2
50.3
60.1
61.2
58.2
58.9
MMGGroundee-PostAlignn PostAlign-13BP
63.7
65.5
50.6
48.1
59.9
62.8
57.0
58.2
MMGrounded Poeiddd-ostttt-gssee (ft)
69.2
73.8
55.2
54.7
66.2
64.2
63.2
62.9
C REASONSEG BENCHMARK AUGMENTATION
To investigate the relationship between reasoning strategies and query difficulty, we augment the queries in the ReasonSeg benchmark. For the easy category, simplify the text descriptions of all samples in the dataset. Specifically, we use Semini-1.5-Ppp expressions into simpler ones. For example, the query "Which food in the image is the richest in protein?" is transformed into "Eggs". For the medium and hard categories, we first run preliminary evaluation using baseline model (without any reasoning modules) and classify the samples into two groups: the top 50% in terms of accuracy are labeled as medium difficulty, while the remaining 50% are labeled as hard difficulty. This approach allows us systematically analyze the impact of reasoning strategies across varying levels of query complexity.
to convert complex referring
D MORE VISUAL GROUNDING RESULTS: REASONSEG
In T method achieves the best performance ReasonSeg benehmark. Following prior work, we denote "ft" as fine-tuning on the 236 training samples of ReasonSeg. The results indicate that fine-tuning significantly enhances model performance. Compared to RefCOCO, ReasonSeg features textual instructions that are often posed as questions, making them inherently more complex and requiring common- sense reasoning. Experimental results demonstrate that model possesses strong reasoning capabilities and excels at handling complex reasoning scenarios.
E HYPERPARAMETER SELECTION
The weighting coefficients \ and X2 play crucial role in balancing different components of our objective function. To determine the optimal values, we conducted systematic grid search over the set 0.001, 0.01, 0.1, 1 for each hyperparameter. We observed that setting \ 0.1 and X2 0.1 consistently yielded the best performance.
F IMPLEMENTATION DETAILS
Training is conducted using the DeepSpeed engine for efficient large-scale optimization. We employ the AdamW Loshchilov & Hutter (017) optimizer with learning rate of 0.0003 and no weight
17

---

Published as a conference paper at ICLR 2026
decay. The learning rate follows WarnupDecayLR schedule with 100 warmup iterations. We use a per-device batch size of with a gradient accumulation step of 10. For detailed hyperparameter settings, please refer to the Appendices.
G FALSE REJECTION EXAMPLES IN THE REJ MECHANISM
In Figure 5 two examples illustrate the concept of false rejection cases in the context of the <REJ> mechanism. On the left, the question asks about the presence of a fork in the image, and the system incorrectly outputs a <REJ> token despite the fork being clearly visible. This false rejection likely due occlusion, where part of the fork may be obscured by other objects, causing the model to incorrectly infer that the fork is absent or indistinct. Visual grounding mechanisms can sometimes struggle when key objects are partially hidden or not fully visible, leading misclassification.
On the right, the question inquires about the presence of an oven, with the system correctly rejecting the object. However, this case, the failure occurs because the model incorrectly interprets the presence of an oven based on background objects or visual cues that resemble an oven but do not directly correspond to the target object. In this scenario, level features such as colors, textures, spatial patterns that are similar to those typically associated with an oven, but not this context. This could result in the model being more prone to hallucinations or misidentifying objects in cluttered or complex environments.
model might rely too heavily on high-
These examples demonstrate rare instances where the model falsely rejects an object that exists in the image, highlighting the importance of improving the precision of <REJ> outputs. To address these issues, refining the model's ability to handle occlusions and improve its attention to finer visual details could reduce false rejection rates, partieularly in challenging cases where objects are partially visible or when background noise interferes with object recognition.
H MORE EXAMPLES
In Figure 6 give more examples of different grounding choices: (a) The model selects the
<COMPLEX> reasoning category for rationale generation before generating the final answer. (b) For simple queries, it selects the IIIPLE>>>> reasoning category directly generate the final answer. (c) In visual grounding, the model selects the <REJ> category, returning a full background mask and negative answer.
I LIMITATIONS AND DISCUSSION
A key design choice in our framework is the use of dataset-level <SIMPLE> and <COMPLEX> labels support selective reasoning textual grounding. While effective and scalable, this approximation is inherently coarse, as individual datasets may contain a mixture of simple and complex queries. This raises a natural concern about potential mismatches between dataset-level labels and the true per-sample reasoning complexity. We had already considered this issue during dataset construction, and our analysis indicates that this limitation has only a minimal effect on model performance for the following reasons:
Dataset characteristics mitigate most mismatches. practice, the proportion of mislabeled samples is very small because the datasets differ substantially in their native supervision format. Refering/dialogue datasets inherently exhibit multi-step reasoning, whereas detection-style datasets (e.g., COCO) contain only single-wond word category labels, which we template into simple sentences for training. As a result, dataset-level complexity largely aligns with the intrinsic reasoning structure of the samples.
Sample-level relabeling experiment confirms minimal impact. To directly evaluate the reviewer's concern, we conducted fine-grained relabeling experiment using an LLM-based pre-filter followed by human verification. We retrained PostAlign using these sample-level SIMPLE/COMPLEX labels. As reported in Table the performance differences POPE were marginal (e.g., 86.6 to 86.2 on the Random split), indicating that dataset-level labeling provides a sufficiently accurate approximation for our selective reasoning mechanism in practice.
18

---

Published as a conference paper at ICLR 2026
Table Ablation on complexity labeling. Despite the coarseness of dataset-level labeling, mixed- dataset training and soft gating make its practical impact minimal. Sample-level relabeling further confirms that fine- geainer labels yield only marginal gains.
Method
Ran
Pop
Adv
LLaVA-1.5-7B 1aa----.-----PosAAAe PostAAlil + unmixed-dataset
85.3
83.4
81.7
-LL.VA--.5-B--PotAAAl + mixed-dataset (ours) LLa
86.6
84.2
82.3
+ relabeled-dataset (sample-level)
86.2
84.8
82.8
Mixed-dataset training further reduces bias. Our training pipeline always mixes heterogeneous datasets within each batch. This naturally places simple and complex reasoning patterns together during optimization, preventing the model from overfitting to dataset-level priors. This mixed training strategy is part of original design and not introduced for rebuttal. For completeness, additionally trained an momixed-datased variant, which indeed produced worse results in Table 7
Soft gating ensures robustness to borderline cases. The SIMPLE/COMPLEX classifier outputs soft probabilities rather than hard labels. This soft gating mechanism allows PostAlign to adapt to borderline or mislabeled instances, reducing sensitivity to imperfect dataset-level supervision.
These results demonstrate that although dataset-level labeling is coarse, its practical impact on reasoning control is limited under our mixed-datasel training and soft-gating design. Fine-grained complexity labeling remains promising direction for future work, particularly for large-scale heterogeneous multimodal corpora.
J BROADER SOCIETAL IMPACTS
Our proposed framework, MMGrounded-Postte PostAllg,,, has the potential to positively impact range of real-world applications that rely multimodal understanding, such assistive technologies for visually impaired users, educational tools, and human-AI collaboration visual analytics. By enhancing visual grounding and reducing hallucinations, method contributes to building more reliable and trustworthy Al systems that are better aligned with human expectations and safety standards.
However, as with many advanced AI systems, potential risks should be acknowledged. Improved grounding capabilities may inadvertently be misused in applications such as surveillance, automated misinformation generation, or biased decision-makee Moreover, biases present in the training data could still propagate through the model, particularly in visually ambiguous or culturally sensitive contexts. It is important that future work continues to explore robust alignment, fairness auditing, and transparent usage policies to ensure responsible deployment of such technologies.
if deployed without appropriate safeguards.
K THE USE OF LARGE LANGUAGE MODELS
No large language models (LLMs), including but not limited to ChatGPT, GPT-4, Claude, and Llama, were utilized in any stage the research and writing process of this paper. All content presented in this work-encompasiig the formulation of research questions, design of methodology. analysis of experimental data, drafting of the main text, compilation references, and preparation of appendices was independently conceived. developed, and written by the authors.
19

---

Published as a conference paper at ICLR 2026
Is there fork in the image?
Is there a oven in the image?
Figure 5: Two examples illustrate the concept of false rejection cases in the context of the <REJ> mechanism.
Is there something table crinking?
sandaich placed the
What the orplane?
User
Ground
MLLM
[GROUNDING] [GGRUNNINN]]
(SSROUNDING)
[GROUNDING] AET
(REASONINE) CO The dentified oylinmccal shops fuble for halding liuid. (/REASCNING]
[REASONINS] <SIMPLE> (REASONINE))
[REASONING] KSIMPLE UREASONINS)
[FINAL [FINAL ANSWER] The sundwich is placed insie o ned plashic bosket green outdoor tuble. [/FINALANNWEER)
[FNNALANSWERR No airplane viable the mage (/FINAL ANSWER)
[FINAL ANSWER) The dispocable The toble The most Hialy ooject used for drisking [/FINAL_ ANSWSR]
Figure 6: Examples of different grounding choices: (a) The model selects the <COMPLEX reasoning category for rationale generation before generating the final answer (b) For simple queries, it selects the <SIMPLES LE> reasoning category to directly generate the final answer. (c) In visual grounding, the model selects the <REJ> category, returning full background mask and negative answer.
20


Reviews:
[{"review_id": "JmlKsiMPqP", "reviewer": "Reviewer_PNjX", "summary": "The paper proposes MMGrounded-PostAlign, a post-alignment framework that augments an MLLM with  \n(1) visual grounding (segmentation + bounding box) driven by a special LOC token and a negative rejection token REJ, and  \n(2) textual grounding via selective reasoning that emits rationales only for complex queries (SIMPLE/COMPLEX gate).\n\nThe method aims to reduce hallucinations caused by linguistic priors and improve fine-grained visual understanding.  \nExperiments on HaloQuest, POPE, VQAv2, MMBench, MME, RefCOCO series, and ReasonSeg show consistent gains; ablations compare to BTL (boxes-as-tokens) variants.", "questions": "I note that the Related Work section references reinforcement learning (RL) approaches in the vision–language modelling domain, yet the paper does not include any empirical comparison involving RL.  \n\nMy thought is that a more general RL-based training paradigm—one in which the vision-language model learns in a sequential decision-making setting—might offer better generalization across vision-language tasks rather than designing a task-specific chain-of-thought process solely for the grounding task.  \n\nSo my question is: what are the advantages of your approach compared to a more general reinforcement learning method for MLLMs?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Clear motivation: Tackles language-prior-driven hallucination via explicit multimodal grounding; neat idea of \"grounding as a corrective lens\".  \n- practical design: Simple LOC/REJ interface to a multi-task decoder; selective reasoning avoids unnecessary rationale generation.  \n- Broad evaluation: Covers hallucination, general V+L, and grounding benchmarks with meaningful ablations.", "weaknesses": "- Limited generality:Only tested on LLaVA-1.5 (7B/13B) + SAM-ViT-H; cross-backbone evidence (e.g., Qwen-VL, other grounding encoders) missing.  \n\n- The idea of labeling queries as SIMPLE vs COMPLEX is good, but doing so at the dataset level rather than per sample raises concern. Some “simple” dataset queries might still require reasoning, and vice-versa. \n\n-  The REJ token is an interesting idea, but the paper does not show the  cases where referent exists but system rejects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mUMFK4l6he", "reviewer": "Reviewer_WDPN", "summary": "This paper introduces MMGrounded-PostAlign, a framework to reduce MLLM hallucinations by grounding outputs in evidence instead of unreliable text priors. It features a visual grounding module with a \"negative rejection mechanism\" to deny non-existent objects and a textual grounding module that uses \"selective reasoning\" to add rationales only for complex queries , thereby improving visual accuracy and suppressing hallucinations.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper clearly identifies and addresses a critical problem in MLLMs: the over-reliance on linguistic priors, which leads to hallucinations and a failure to ground responses in visual evidence. The proposed \"post-alignment\" framework is a well-motivated and logical approach to re-center the model's outputs on visual information.\n\n2. The experimental analysis provides valuable insights. \"Finding 1\" (Figure 3), which empirically demonstrates how linguistic priors can override visual information in the model's later layers, offers a strong motivation for the method. Furthermore, the ablation studies (e.g., Tables 1 and 2) are thorough, providing a solid comparison of different grounding strategies (segmentation, detection, BTL vs. explicit grounding) and validating the paper's design choices.\n\n3. The `<REJ>` token is a practical and effective mechanism for negative grounding. By giving the model an explicit option to \"abstain\" from grounding a non-existent object, this method directly targets object hallucination.", "weaknesses": "1. Insufficient baseline comparisons: A significant weakness is the lack of comparison against the original, unmodified baseline model, as well as other well-established MLLMs. The model is built on LLaVA-1.5, yet Tables 1-3 primarily compare variants of the proposed method against an internal baseline (the framework with modules removed), not against the original LLaVA-1.5. This makes it difficult to assess the true impact (including any potential performance trade-offs) of the added components. This is particularly concerning given \"Finding 3\" (retaining reasoning abilities), which cannot be fully verified without this comparison. For instance, the reported 63.9 on MMBench-EN (7B) may not be competitive with the public LLaVA-1.5-7B score (64.3).\n\n2. Unclear architectural novelty: The novelty of the visual grounding module's architecture is not well-explained. The Method section (Section 3) describes a model (with SAM-based decoder, `<LOC>` and `<REJ>` tokens) that seems to reimplement established paradigms from prior works like LISA, GLaMM, and GSVA. While these are cited in Related Work, the Method section itself does not attribute these design choices or clearly differentiate what is adopted from prior work versus what is a new architectural innovation. The contribution appears to be more in the application of this module, but the presentation makes the architectural contribution ambiguous.\n\n3. Dataset-level reasoning labels: A significant limitation, as acknowledged by the authors in Appendix, is that the `<SIMPLE>`/`<COMPLEX>` labels for selective reasoning are applied at the dataset level, not the sample level. This is a very coarse heuristic. A dataset labeled \"complex\" may contain many simple queries, and vice-versa. This design choice weakens the \"selective reasoning\" strategy, as the model isn't learning to distinguish query complexity on a case-by-case basis but is instead learning a bias associated with the data source prior.\n\n4. (Minor) Clarity of \"Selective Reasoning\": The \"selective reasoning\" mechanism is not very clearly explained in the Introduction. The Introduction is vague. The reader must wait until Section 3.3 to understand the concrete implementation (i.e., the `<SIMPLE>` and `<COMPLEX>` tokens). Briefly explaining this mechanism earlier would improve the paper's readability and flow.\n\n5. (Minor) Citation formatting: The paper does not consistently follow the ICLR template's citation guidelines (`\\citep` and `\\citet`). This should be corrected for the final version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IpjYCCtomi", "reviewer": "Reviewer_CYza", "summary": "Addressing the hallucination problem and insufficient fine-grained visual understanding of Multimodal Large Language Models (MLLMs) caused by their over-reliance on linguistic priors, this paper proposes the MMGrounded-PostAlign post-multimodal alignment framework. The framework integrates visual grounding (incorporating a negative rejection mechanism to distinguish between real and non-existent objects) and textual grounding (incorporating a selective reasoning mechanism to adjust reasoning strategies based on query complexity). Built on the base models of LLaVA-1.5-7B/13B and ViT-H SAM, and optimized through LoRA fine-tuning and multi-loss function, the framework’s effectiveness—including suppressing hallucinations, enhancing visual understanding, and preserving the reasoning capabilities of MLLMs—has been validated on benchmarks such as HaloQuest, POPE, VQAv2, MME, MMBench, RefCOCO, and ReasonSeg.", "questions": "See above", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It accurately identifies two key issues of Multimodal Large Language Models (MLLMs): \"hallucinations\" (generating non-existent content) and \"insufficient fine-grained visual understanding\", which are caused by the models' over-reliance on linguistic priors. These two types of issues serve as core bottlenecks that undermine the robustness and reliability of current MLLMs in vision-language tasks, making the research direction highly practically significant and necessary.\n- While enhancing visual understanding and suppressing hallucinations, it does not compromise the inherent reasoning and generalization capabilities of MLLMs (e.g., achieving performance equal to or better than the baseline on MME and MMBench). It also avoids the problem of degraded reasoning ability in some grounding methods (such as BTL-Generation) due to overfitting to visual information, demonstrating an excellent balancing effect.", "weaknesses": "- In textual grounding, the <SIMPLE>/<COMPLEX> labels are categorized at the \"dataset level\" (e.g., queries in the COCO dataset are classified as <SIMPLE>, while those in the ReasonSeg dataset are classified as <COMPLEX>), rather than being annotated at the \"sample level\". Although this approach reduces annotation costs and ensures training stability, it fails to handle scenarios where \"simple queries and complex queries are mixed\" within the same dataset. This may lead to inaccurate matching of reasoning strategies for some samples (e.g., complex reasoning queries expressed in a simple form are misjudged as <SIMPLE>).\n- The <REJ> samples used to train the \"negative rejection mechanism\" are only sourced from the gRefCOCO dataset (containing 32,202 queries that \"refer to non-existent objects\") and do not cover more scenarios (such as negative samples of different object types, different image styles, and different linguistic expressions). When facing unseen \"false query-image\" combinations, the effectiveness of the negative rejection mechanism may decrease, and its robustness needs further verification.\n- During training, the model is enabled to automatically judge the complexity of queries through \"self-reflection prompting\". However, the model's judgment of difficulty may also introduce biases and hallucinations. For instance, it is common for the model to be overconfident, which leads to the generation of hallucinations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i5NAeL13K1", "reviewer": "Reviewer_DgK7", "summary": "This paper introduces MMGrounded-PostAlign, a post-multimodal alignment framework designed to mitigate hallucinations and enhance the visual understanding capabilities of Multimodal Large Language Models (MLLMs). The core idea is to use multimodal grounding as a \"corrective lens\" to anchor the model's outputs in actual visual and textual evidence, thereby reducing its over-reliance on spurious linguistic priors.", "questions": "See weakness please", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a novel and compelling perspective. Instead of using MLLMs for grounding tasks (the common approach), it inverts the relationship by leveraging grounding to enhance the MLLM itself.\n- The paper is generally well-written and well-structured. The motivation is clearly established, the framework is explained with the aid of a pipeline diagram, and the findings are presented logically.", "weaknesses": "- the `<SIMPLE>`/`<COMPLEX>` labels are assigned at the dataset level, not per sample. This is a coarse-grained heuristic that may misclassify individual queries.\n- It is uncertain whether the performance gains are due to the novel grounding-as-a-lens concept or simply the introduction of any additional post-processing signal. It is possible that a simpler method like contrastive decoding could achieve similar hallucination suppression on POPE without the need for a full segmentation model.\n- The framework is a multi-component system where the visual grounding, textual grounding, and final answer generation are interlinked. What happens if the visual grounding module fails (e.g., produces an incorrect or low-confidence mask for a present object, or fails to trigger <REJ> for an absent one)?\n\n- Some figures have relatively small, which affects readability. For example, Figure 3(a) and (c).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)