Causal Decoding for Hallucination-Resistant Multimodal Large Language Models
Authors: Shiwei Tan
Hengyi Wang
Weiyi Qin
Qi Xu
Zhigang Hua
Hao Wang
Venue: Accepted by TMLR ()

Abstract: Multimodal Large Language Models (MLLMs) deliver detailed responses on vision-language tasks, yet remain susceptible to object hallucination (introducing objects not present in the image), undermining reliability in practice. Prior efforts often rely on heuristic penalties, post-hoc correction, or generic decoding tweaks, which do not directly intervene in the mechanisms that trigger object hallucination and thus yield limited gains. To address this challenge, we propose a causal decoding framework that applies targeted causal interventions during generation to curb spurious object mentions. By reshaping the decoding dynamics to attenuate spurious dependencies, our approach reduces false object tokens while maintaining descriptive quality. Across captioning and QA benchmarks, our framework substantially lowers object-hallucination rates and achieves state-of-the-art faithfulness without degrading overall output quality.

Published in Transactions on Machine Learning Research (February/202)
Causal Decoding for Hallucination-Resistaan Multimodal Large Language Models
Shiwei Tan"*, Hengyi Wang"", Weiyi Qin Qi Xu²*, Zhigang Hua?*,
Hao Wang!
"Equal Contribution, 'Rutgers University, Meta Ranking AI
Reviewed on OpenRevicw: https///ppaaeeeeeereeee/////teep
Abstract
Multimodal Large Language Models (MLLMs) deliver detailed responses vision-language lasks, yet remain susceptible to object hallucination (introducing objects not presenl in the image), undermining reliability in practice. Prior efforts often rely on heuristic penalties, post-hoc correction, or generic decoding tweaks, which do not directly intervene in the mechanisms that trigger object hallucination and thus yield limited gains. To address this challenge, we propose a causal decoding framework that applies targeted causal interventions during generation curb spurious object mentions. By reshaping the decoding dynamics to attenuate spurious dependencics,
approach reduces false object tokens while maintaining descriptive quality. Across captioning and QA benchmarks, framework substantially lowers object hallucination rates and achieves state-of-the-art faithfulness without degrading overall output quality.
1 Introduction
Large language models (LLMs), such as GPT-4 (Achiam et al. 2023 and LLaMA (Touvron et al. 2023 have been rapidly developed and widely adopted due to their wide range of applications. To extend the capabili- tics of LLMs to visual tasks, multiple MLLMs have been proposed. Mod- els such as LLaVA (Liu et al. 2024c and MiniGPT (Zhu et al.) 2023 typ- ically project visual information into the same representational space tex- tual data, enabling a unified process- ing approach via an internal LLM. Al- though MLLMS have shown impres- sive performance multimodal tasks, including chatbots, visual question an- swering, and image captioning, they remain susceptible to visual hallucination.
Typical MLLM
COAD (Ours)
Hidden States pizza, table, lork
Hidden States lastfe pizza, table, koele
No hallucination from generated text
Hallucinate "foric" from generated "knife"
Previously Generated x: A siice of pizza plate kniife..
Predicted Text yr: and fork on the side.
Previously Generated Text x ... slice of pizza on plate with knife
Predicted Text y: (period, ends]
Figure 1: Simplified causal graphs for typical MLLMs and our COAD. Left: Typical MLLMs implicitly hallucinate objects (e.g., "fork") in the hidden states z due to previously generated text x (e.g., "knifo"). Right: Our COAD porforms causal inference to remove links between the hidden states z and generated text x, thereby avoiding hallucination.
Specifically, hallucinations in LLMs (Huang et al. 2024a) refer to cases where the model generates outputs that appear factual but are actually incorrect or ungrounded. With the introduction of visual inputs, multimodal LLMs (MLLMs) encounter a new category hallucination: visual hallucination (Liu et al. 2024b). Visual hallucination occurs when the MLLM output diverges from the content of the input image. This undermines the reliability of the models and restricts their applicability high-stakes real-world scenarios that demand high precision, such as medical image analysis and legal document generation.

---

Published in Transactions on Machine Learning Research (obruary//020
Recently, a variety of approaches have been proposed to mitigate hallucinations in MLLMs; they can be broadly categorized into two main stralegies: (1) The first strategy improves the model with external
information, such as incorporating additional training data or retrieving knowledge from external source (Liu
ict al.
2024a Yu al. 2024 Wang et al., 2023 Chen et al., 2024a Vu et al. 2023: Gao et al. 2023 Varshney let al.
2023 Although these methods ively reduce hallucinations ions, they often require signilicant effort
in data collection and depend on the quality and availability of external knowledge bases. (2) The second
strategy aims to reduce hallucinations without relying on additional information, instead refining the training
procedures of the model improving the attention mechanisms during inference Yuue et al.] 2024 Han et al,
2024: Shi et al.) 2023: Leng et al. 2023 Liu et al. 2024d: Huang et al. 2024b; Chuang et al. 2024 Deng
et al. 2024a Chen et al. 2024b). However, these methods still fail to model the causal effect from visual
inpul (e.g., images) to the generated response. They are therefore often susceptible to confounding effect Yan
& Wang (2023): ; Wang et al. ( Pearl (2009) or bias brought by the generated text. As a result, they
tend to generate new hallucinated text based on existing hallucinatod text, exacorbating hallucination.
To address these challenges, we propose Causal Object-Aware Decoding (COAD) to reduce hallucination by incorporating causal inference into the model's decoding process; this is inspired by hierarchical Bayesian doep learning Wang & Young (2016 2020); Wang et al. (2024) and the causality literature Pearl (2009): Wang ct al. 2020 Yan & Wang (023). Specifically, we first employ an object detector to identify visual objects in the image, delegating part of the image comprehension task to this specialized component. We then expose these structured detection results to the MLLM by finetuning the MLLM with object detection outputs as additional inputs, alongside the image and previously generated text tokens. Finally, we perform causal inference to effectively integrate the predictions from both the original pretrained model and the finetuned model to generate the response.
COAD's design improves the reliability of the MLLM via enabling targeted interventions in the model's understanding of visual objects. Furthermore, we incorporate causal inference to reduce the model's dependence on self-generated text when processing and describing images, thereby promoting more stable and less hallucinatory outputs. Our contributions are as follows:
We formulate the generation of reliable responses as the estimation of unknown oracle predictions and introduce a new framework, dubbed Cansal Object-Aware Decoding (COAD), to reduce object hallucination.
We introduce a targeted intervention strategy that exposes and leverages visual structure, allowing the model to reason more faithfully about image content.
We provide empirical results to demonstrate the effectiveness of our method in improving generation quality and reducing object hallucination compared to state-of-the-art methods.
2
Related Work
Hallucination Mitigation. A typical strategy to mitigate halluci- nations in MLLMs is to augment the model with external data. One line of work focuses on expanding or refining the training data to enhance grounding and reduce hallucinations (Liu et al., 2024a Yu et al. 2024 Wang et al. 2023 Chen et al.) 2024a). These methods typically involve curating high-quality multimodal instruction data, improving image-text alignment, or re-captioning visual content to ensure consistency with external world knowledge. By exposing the model to more reliable or better-aligned data, such approaches aim to reduce the risk of generating content that deviates from visual evidence or factual reality. Another line of research tackles hallucination at inference time by retrieving relevant information from external knowledge bases or the internet (Vu et al. 2023) Gao et al. 2023 Varshney et al. 2023) These retrieval-augmented generation methods dynamically inject grounded knowledge into the model's context, thereby improving factuality without requiring the model to memorize all details.
External Ghooledge-Angmentie
While both approaches have demonstratod effectiveness, they rely on either significant data curation and annotation efforts or real-time access to high-quality and up-to-date external sources. In many real-world applications, especially those involving specialized or rapidly evolving domains, such requirements may not
2

---

Published in Transactions on Machine Learning Research (obbuaayy/220
always be feasible or reliable, highlighting the need for alternative strategies that improve factual grounding without external dependencies.
Internal Hallucination Mitigation. Other approaches mitigate hallucinations without relying on external data sources or retrieval mechanisms. These methods aim to improve the model's internal decision- maaiinn process by modifying its behavior during training or inference. For instance, FOS (Yue al. 2024) encourages early stopping in sequence generation to prevent over-generation, which is often a source of factual inaccuracy. Skip-\n An (Han et al. 2024 suppresses hallucinations by skipping newline tokens, which are empirically shown fabricated continuations. Several techniques reduce the distraction caused by noisy mislcading text-conditioned inputs by selectively emphasizing attention visual tokens. Examples include CAD (Shi et al. 2023), VCD (Leng et al. 2023), and PAI (Lu et al. 2024d), which implement visual grounding and cross modsl alignment enhancements. ULIP-guided decoding (Deng et al., 2024b) reduces hallucination by incorporating a CLIP--baeed image-text consistency score into sentence- beam search, adjusting the beam scores beyond the MLLM's own likelihood. OPERA (Huang al., 2024b) proposes intervention-based decoding strategy that penalizes overconfident token predictions, which often associated with hallucinated content. DoLa (Chuang al. 2024) improves factual alignment by comparing generation logits from early and late transformer layers, effectively regularizing token prodiction based on layer-wise consistency. In this paper, we build on this line of research by focusing on internal mechanisms to reduce hallucination, without directly relying on external knowledge bases.
to precede low-quality
3 Methodology
In this section, we first introduce our COAD as causal model for the MLLM's next-token generation process, and then describe how we apply causal inference to predict the next token during inference.
3.1 Preliminaries and Key Intuition behind COAD
Problem Setting: Auto-Regressin Generation. We consider an aulo-regressive MLLM that, at each decoding step, receives: (1) model M; (2) an input image S E cxxx where is the number of channels, h is the height of the image, and w is the width the image; and (3) a sequence of previous tokens x (including the prompt and generated tokens so far). The model predicts the next token y by sampling from
y~ PM x, S).
Here M may be pretrained MLLM My, a finetuned model My, or hypothetical oracle M. (introduced later).
Causal Inference. Causal models provide a principled frame- work for distinguishing true causal effects from correlations induced by confounders (Pearl 2009). A central tool is the use of interventions, denoted by do(.), which remove spuri- ous dependencies when analyzing the effect of one variable on another.
Z
x
X
y
(a)
(b)
Illustrative Example. We next illustrate the effect of confounder using the two causal graphs in Figure 2 Let denote temperature, x the hot drink sales, and y the ice cream sales.
Figure 2: Illustration of confounding.
(a) z induces spurious association between and y even without causal effect. (b) Adding x + introduces genuine causal effect, but P(y(z) remains confounded by z.
In Figure 2a the edges reflect the causal structure where both x and are influenced by the confounder z, but there is no direct causal effect from æ to y. In this setting, observing high hot drink sales æ implies that temperature z is likely low, which in turn suggests that y (ice cream sales) is also likely low. Consoquently, even though r does not causally affect y, the conditional probability P(y]z) differs from Ply). This discrepancy reflects a misleading spurious correlation introduced by the confounder
3

---

Published in Transactions on Machine Learning Research (obbuary//220
To isolate the causal inffuence of y, we instead compute the interventional distribution P(y)do(z)) which simulates actively setting x to fixed value while breaking its natural dependence on According to the rules of do-calculus, P(y]do(z)) = P(y) in this case, correctly reflecting the absence of z's causal influence on y. The formal derivation and rules can be found in (Paall 2009).
Figure 2b then adds a direct causal edge x - y on top of Figure 2a In this case, hot drink sales x indeed have a direct causal impact on ice cream sales y (e.g., some customers may buy a cold item after consuming hot drink). However, the conditional probability P(ylz) still overestimates this causal effect, because contimues to influence both I and y. The confounder thus causes P(y|z)) to capture both the genuine causal effect of x on y and the additional dependence mediated by 2, motivating the use of interventional quantities such as P(y]do(r) to isolate the true causal influence.
Analogy to MLLM Next-Token Prediction. The structure in Figure 2b directly corresponds to MLLM docoding. Let x denote the previously generated tokens, y the next token to be prodictod, and z the model's hidden stales representing its belief about which objects are present in the image (note that z is not the image itself, which could be modeled separately). Because is backdoor variable connecting to both x and y, the conditional probability P(y|x) still overestimates the causal effect from x to y, i.e, overestimates the likelihood of certain tokens y, therefore potentially leading to hallucination.
Key Intuition of COAD. To mitigate object hallucination, COAD explicitly models z as a variable representing object beliefs and replaces the standard conditional distribution with the interventional one P(yldo(x), z). This removes the spurions dependence introduced by the confounder z and predicts the next token y based solely on the true causal effects of x and
Generative vs. Recognition Causal Models. The causal perspective adopted in this work follows the recognition iinooreneee/eeeeeeeee view commonly used in recent causal analyses of vision-language models (Mao et al.) 2021: 2022 Instead of modeling how the physical world generates an image (which would typically imply a generative direction such as z + S), our objective is to describe how an MLLM processes an observed image S and pre-existing lext x in order to form internal beliefs and predict the next token.
Under this recognition view, denotes the model's internal belief about which objects are present in the input image, not a latent variable causing the image itself. The information flow from S to z therefore represents the model's inference procedure, consistent with modern archilectures such as LLaVA, where visual features are computed by the vision encoder before any interaction with textual tokens (see Appendix [ This viewpoint provides the conceptual grounding for the causal structures introduced in the following sections.
3.2 Formal Definition of Object Hallucination
Let S be the input image, z" the ground-truth set of visual objects, and po(y)x,S) S) the model's prodictive distribution for the next token y given previous tokens x and the image S. Let p" (y|x, z") denote the "ground-truth" conditional distribution, i..,, the distribution produced by an ideal model that fully respects the true visual semantics of the image.
We define object hallucination as the divergence between these two distributions:
D(pr(r|x, S) I| (v/xx
where D(-II-) may be a KI. or another divergence measure. large divergence indicates that the model assigns high probability to tokens that contradict the true visual object content, thereby producing object hallucination.
In existing multimodal LLMs, hallucination often arises because the hidden states z may encode nonexistent objects based on previous tokens x rather than on the true image content S. This can lead the model to generate tokens y that are not visually grounded. We empirically validate this phenomenon for LLaVA using a linear-probe analysis of its internal object-existence beliefs; see Appendix B for details.
To address this issue, our COAD (i) uses a detector-derived proxy 2 to approximate the visual constraints x² in the ideal distribution p" (w/x, 2"), and (ii) intervenes on both the internal object-related hidden states
4

---

Published in Transactions on Machine Learning Research Sebruary/2020
User:
Describe image in detail. Model: The image features
Pre-trained MLLM
dog: 0.3 cat: 0.6
token probabilities
Causal Fusion
dog: 0.98 cat: 0.01
Fine-tuned MLLM
dog: 0.98 cat: 0.01
dog: 0.70 cat: 0.25
N sets of token probahilities
dog: 0.9 cat: 0.08
Object Detector
P(dog)=0.97
P(cat)=0.26
sample
fdog]
[dog]
output probabilitte
(dog. cat)
object probabilities
batch of N samples
Figure 3: Overview of our COAD. We employ an object detector to identify the objects present in an image. The MLLM is then finetuned to condition its token predictions on both these detected objects and the input image. COAD subsequently use causal inference to combine the output distributions both the pretrained and finetuned MLLMs to generate the final prediction.
z and the previous tokens x to block the spurious information pathway from to y. These interventions reduce the divergence above and therefore mitigate object hallucination.
3.3 Method Overview
Figure illustrates the decoding (i.e., text generation) process of our COAD. It assumes access to an MLLM that can incorporate a set of detected objects as additional context during generation. To achieve this, we finetune a pretrained MLLM with object-level information. During inference, an object detector identifies likely objects in the input image and outputs probability distribution over candidate object classes. We then sample multiple plausible object sets from this distribution.
Each sampled object set is injected as auxiliary input into the finetuned MLLM to produce distribution over the next token. This results in N next-token distributions, which are further combined with the distribution from the pretrained MLLM. Finally, COAD uses causal inference to combine these outputs to generate more robust and object-aware prediction.
3.4 Causal Model of a Standard MLLM
Before introducing the causal model underlying COAD, we first describe the temporal causal structure of a standard MLLM during autoregressive decoding. This view makes explicit how information flows across timesteps and clarifies the relationship between the image, the evolving text sequence, and the model's internal object-related variables.
The input image S and the initial prompt (() remain fixed throughout decoding. The variable represents the model's belief about which objects are present in the image and is determined solely by S; it therefore remains constant across timesteps. Here, is a conceptual variable capturing image-conditioned object beliefs, rather than a token-level hidden state that evolves during decoding (see Appendix A for further discussion). This treatment is consistent. with the architecture of modern vision-language models such as LLaVA, where visual features are produced by the vision encoder before any interaction with textual tokens (see Appendix C
Temporal Structure.
The only time-varying variables are the evolving text sequence x and the next token (+))
Figure 4a illustrates the resulting temporal causal graph, which contains the following edges:
5

---

Published in Transactions on Machine Learning Research (February/20)) 2026)
Z
S
x
((1)
x(2
xx33)
x(T)
yr(
y(a))
y(T-1)
(b) Collapsed version of (a).
(()
(1)
(2)
x()
(t)
x(++1)
(a) Full temporal causal graph of MLLM decoding process, where gray nodes represent observed variables.
y
O.I.....
(c) Time-compressed representation of (a).
Figure 4: Rolled-out causal structures of the MLLM decoding process. (a) Full temporal (rolled- out) causal graph over decoding timesteps: the image S and initial prompt x()) are fixed, denotes the image-derived object-belief variable (constant
time), and x(1), yd) evolve according to x(1), S, -> y( and (*),, ( (1+1), (b) Collapsed version of (a) obtained by treating y an implicit step in the autoregressive update, so that each x(") (t >> 0) is jointly determined by S, z, and x(t-1), (c) Time--ompressed representation in which the accumulated influence of all previous timesteps on x (4) is summarized by dashed edges from (S, z) to x(").
S + indicating that the object-belief variable z is inferred from the image.
x(),S, + (() reflecting that cach predicted token depends the current text, the visual input, and the inferred object beliefs.
y()) x((+1), reflecting the autoregressive update where each new token (0) is appended to the old sequence x(t) to form the new input sequence x((+1), Le., the new input sequence is x(++1)= = [x((),(/) (()]
Collapsed Representation. the intermediate variables (() are treated as implicit steps in the autore-
gressive update, the subgraph (x(9, y00) x (++1) can be collapsed into a single transition x(t) + x ((+1), producing the structure shown in Figure 4b. In this form, each x(1) (t 0) jointly determined by S, z, and (!!1))
Time- Comppeeeeeeee View. Compressing the time axis of Figure 4a yields the representation in Figure 4c
which summarizes the cumulative influence of past timesteps on x( via dashed edges from (S, z) to (!)
This compressed form closely matches the causal structure underlying a single decoding step, and provides the foundation upon which we build the causal model for COAD the next subsection.
3.5 Causal Model of COAD
Figure 5a shows the causal model (as a causal Bayesian network) of our COAD. Given the input image S and the previous text tokens x as observed variables, below are key components COAD's generative process.
Object Variable Our causal model operates at the gramularity of individual token generation. At cach decoding step, given the image S and the preceding (incomplete) lext COAD infers the presence of visual objects in S through a binary variable E 10 where C' denotes the total number of object categories and is fixed by the choice of the dotector. This variable is sampled from the distribution produced by an object detector D:
D(S),
6

---

Published in Transactions on Machine Learning Research (eebruar/2022)
where D(S) E [0, 1] denotes the detector's estimated probability for the presence of each object category in the image.
Dual MLLMs for Generation. To model the next-token prediction, we incorporate two MLLMs into our causal framework: a pretrained model Mp and a finetuned variant My. The pretrained model Mp takes as input the image and the precoding text x, and outputs a distribution over the next token Up- The finetuned model My, adapted from Mp, additionally conditions on the object variable produce a distribution over the next token y
yp PM, (1pp (),/xx,), y5 ~ PM, (2,1
where Pas, ()pix,5 S) is the next token distribution predicted by Mp, and PM, (uylx, S, is the next token distribution predicted by My. In practice, M, and M, share most parameters for efficiency.
Hypothetical Oracle MLLM. To complete the causal graph, we introduce hypothetical oracle model M., which serves idealized reference that always produces the optimal next-token distribution. The token predicted by this oracle, denoted You is generated as follows:
ya PM. (U,
where PM. (v.|x, S, z) represents the oracle's ground-truth distribution conditioned on the previous text x, image S, and object variable z.
Mixturc-Based Generation. We hypothesize that the finetuned model My behaves as mixture of the pretrained model Mp and the hypothetical oracle model M.. At each decoding step, My may generate either the token predicted by Mp the one predicted by M,, with certain probability. Note that this is a natural assumption: Mp serves as the initialization of M,, and during finetuning. My is optimized to better approximate ground-truth signals (as represented by M.) while still inheriting behaviors from the original pretrained Mç-
capture the uncertainty in My's alignment between Mp and M., we introduce a random variable y E [0, I], which governs the mixture proportion. This variable is drawn from a global prior Beta distribution with hyperparameters Ya: 76 E IR+, which fixed across dataset:
Bela(Ta. 76).
and the next-token prediction yy is drawn approximately from mixture of the two sources:
yf ~ CategoricalMixxture(vv.. (p.. h 7])
Categorical(y x y, + 7) x up).
This formulation reflects the intuition that Mj may probabilistically interpolate between following the oracle model and reverting to its pretraining prior (more details in Equation 2 below). It also provides an alternative generative view of yy, which allows us to indirectly infer the oracle token ys in Section 3.6
Complete Causal Graph. Figure 5a summarizes the causal relationships among all random variables introduced in our model. The image and previous lokens x are the only observed variables. Note that x itself may be influenced by S and during previous decoding steps. All other variables conditionally generated from their respective parents according to the mechanisms described above. The dotted connections from Yxo Up, and to By indicate our hypothesis: M can be alternatively interpreted as a probabilistic mixture of M, and Mp-
With the causal graph and the given observed variables, i.c., the image S and previous tokens x, our goal is (approximately) predict the oracle next token y, using causal inference. This will be discussed in Section 3.6 below.
7

---

Published in Transactions on Machine Learning Research (February/ 2026)
S
Yp
S
Yp
ff
Y
ff
Y
X
X
Z
yx *
Z
y
(a) Original causal graph, where gray nodes represent observed variables. The dotted arrows illustrate alternative pathway for generating Vy-
(b) The resulting causal graph after do(x) interven- tion. All edges going into x are blocked by the intervention.
Figure 5: Illustration of our COAD's causal model before and after intervention.
3.6 Inference Process
In this subsection, we describe how COAD employs our causal model address the key challenges in reducing hallucinations of the MLLMs. We start by briefly discussing key components of our method, i.e., Causal Inference of Objects and Estimation of Oracle Predictions, and then derive the corresponding equations that combine these two components.
Component 1: Causal Inference of Objects z. To ensure object beliefs reflect only the image content, we explicitly model them as variable z in our causal framework. Different from existing methods, where object belief is entangled in the hidden state and influenced by previous tokens x, we block this dependency using an intervention do(x) (sce Figure 5b). This treats as externally fixed, forcing the inference of z depend solely the image S and not on prior language outputs
Component 2: Estimation of Oracle Predictions. To approximate the oracle prediction ye, model the finetuned output Vy mixture of the pretrained model M, and the oracle model M,, following our assumption in Section 3.5 While is unobservable, this mixture formulation allows us to estimate it using the available predictions Ys and Up by M, and My, respectively. This provides a principled way to bridge the gap between observed model behavior and the ideal oracle output.
Combining Components & 2 to Derive the Inference Objective. By combining the previous components, inference objective becomes computing the oracle prediction under intervention, i.., d(o))). Using Bayes' rule and standard rules of causal inference (Pearl 2009), have that:
Ply.
(1)
= Σ, 2l),) do(x)
Ply./S.do(x).
(Rule 3)
Σ, P(..IS.xx2)P((SS)
(Rule 2)
This formulation rewrites the interventional query (with do(-)) using standard conditional probabilities (without do(-)), which can be estimated from observable components. We use the object detector to compute P(z[S), which ensures that object beliefs are based solely on the image. The term P(y-IS,x, z) represents the oracle model's prediction, which not directly accessible. To address this, we approximate it using mixture model. Specifically, following our hypothesized relationship between M,, M., and Mp. have that:
P(y, E. + ((,,))
(2)
8

---

Published in Transactions on Machine Learning Research (February/002)
By rearranging Equation can rewrite the prediction from M, in terms of the predictions yy and yf from Mp and My, respectively. Specifically:
P(y.1 S, x22)
(3)
- - P Ply 1 +
+11
7.1h] P( SS
= (1 2t 30 Plyr|S,x,2
Pg S,
Final Inference Objective. After substituting Equation 8 into Equation and rearranging the terms, we can then rewrite our final inference objective as combination of known quantitics:
Ply. ((y.)s.doxx))
(4)
= =3, P(z|s) (1 a) P(y,1S,x, -.,p))
= =1++a) (( x, - -aP(y,lS.x).
where we use the shorthand a 2 77/7/. Since only the ratio a = appears in the final expression, the Beta distribution's parameters (7a:70 (as hyperparameters) effectively have only one degree of freedom during inference. In practice, We therefore treat a as a single global hyperparameter (see Appendix A for additional details). Notably, computing the closed-form expression in Equation $ is equivalent to sampling many values y ~ Beta(Ta: "n)) and averaging their contributions in expectation. This closed-form solution makes COAD both more efficient and conceptually consistent with the underlying mixture interpretation.
Since the dimension of can be large, directly summing all possible object-belicf vectors is computationally intractable. We consider two practical implementations to approxinate the expectation over in Equation 4 (1) Monte Carlo sampling, where we treat the detector's output ズ E [0, 1] as the parameters of C' Bernoulli distributions and sample N binary vectors 2; E {0. 1) (where =1
.2....,N) 1.,,..., Σx P(z(S))((r)) P(y (S, x, z) is approximated by N "E PS 2.); and (2) probabilily-busee approzimation, where we directly feed the probability vector ž into My, which empirically provides an efficient approximation to the same expectation while avoiding sampling. Due to its high efficiency, we adopt the second implementation (although it is an approximation) for all our experiments.
from them, so that the term
Summary of COAD. To summarize, training and inference of COAD consist of the following steps:
1. Modify the pretrained MLLM to accept object belief vector additional input.
2. Finetune the modified MLLM using the object vectors z (predicted by an object detector).
At inference time, compute the next-token probability using Equation approximating the expecta- tion over z via Monte Carlo sampling.
Therefore, COAD enables object-aware dehallucination by explicitly grounding language generation in visual object beliefs. Through causal modeling and intervention, COAD ensures that predictions remain faithful to the image content, reducing reliance on spurious correlations from prior text.
4 Experiments
In this section, compare COAD with existing methods real-world datasets.
4.1 Datasets and Metrics
We use various datasets and metrics below to evaluate the MLLMs.
POPE. The Polling-based Object Probing Evaluation (POPE) (Li al. 2023) employs visual question answering to assess whether MLLM can correctly identify the presence of object in input image.

---

Published in Transactions on Machine Learning Research (obbuaayy/2200
Following the literature (Liu et al. 2024d: Huang et al. 2024b). focus the MSCOCO dataset with 500 images, with each image having 6 questions for each split of POPE. We evaluate the object recognition performance using the Precision, Recall, F-1, and Accuracy metrics.
CHAIR. Caption Hallucination Assessment with Image Relevance (CHAIR) (Rohrbach et al. 2018 is a set
of widely used metrics to evaluate captioning hallucination. Following the literature (Tiu 2024d Huang
let al., 2024b), we use the MSCOCO dataset (Lin al.) (014) that provides annotations for ground-truti objects in images. Specifically, CHAIR includes two metrics:
CHAIRg, which measures the proportion of captions containing hallucinated objects relative to the lotal number of captions:
C = ccaptions with hallucinated objects / |all captions].
CHAIR,, which measures the proportion of the hallucinated objects relative to the total number of mentioned objects:
CHAIR, hallucinated objects| / |all mentioned objects].
MMHal-Benchh MMHal-Bench (Sun et al., 2023 is a dataset designed to evaluate MLLMs on diverse questions where they may produce false claims about image content. The benchmark spans eight hallucination dimensions that assess different aspects of visual grounding: (1) Object aftribute: the ability to correctly describe properties such as color or shape; (2) Adversarial object: the ability recognize when a queried object is not present rather than hallucinating it; (3) Comparison: the ability to compare attributes or properties across multiple objects; (4) Counting: the ability to estimate the number of referred objects; (5) Spotial relation: the ability to reason about relative positions among objects; (6) Environment: the ability to infer aspects of the surrounding scene or background; (7) Holistic description: the ability to provide accurate global descriptions of the entire image; (8) Others: the ability to recognize text or symbols and reason based on observable visual information. Scores for each dimension are computed as the proportion of responses deemed non-hallucinatory. Higher scores across these dimensions indicate better grounding and fewer hallucinations. Following the MMHal-Beuchh evaluation protocol (Sun al. 2023 we use GPT-4 to perform this judgment.
4.2 Baselines
We use VA [Liu et al. 2024c) the base model for all evaluated methods. For COAD, we use RTMDet (Lyu et al. 2022) the object detector D.
As discussed in Section existing hallucination-miiiigeee mitigation approaches fall into two broad categories: (1) external-Knowlldgeebsss methods and (2) internal ceciitectuue/eecoddiin modifications. Since COAD belongs to the second family and does not rely on external data retricval, all basclines evaluated here are drawn from this internal-mechhanism category to ensure a fair comparison. Under this setting, we compare COAD with state-of- state-of-the-art the-art meternal-mechanism methods, including Decoding by Contrasting Layers (DoLa) (Chuang ct al. 2024 Paying More Attention to Image (PAI) [Lu et al. 2024d). End-of-Sentenee Decision (EOS) (Yuc
et al. 2024 Over-trust Penalty and Retrospetioon-Allccttoo (OPERA) (Huang et al. 2024b). Visual
Contrastive Decoding (VCD) (Leng et al. 2023), Context-Awan Decoding (CAD) (Shi et al. 2023) and Object Hallucination Reduction via Adaptive Focal-Contrast Decoding (HALC) (Chen et al. 2024b
4.3
Implementation Details
We finetune COAD on subset of MSCOCO images sourced from the LLaVA datasot. To enable the model to incorporate the auxiliary input z, we introduce a two-layer MLP projector (with hidden size 256) that maps z into the token embedding space, following LLaVA's multimodal token integration approach. We employ LoRA (r = 128, a 256), cosine lcarning rate schodule with an initial lcarning rate of 4c-5, 5, a batch size of 128, and train the model for 1 epoch. During inference, we use sampling by default with temperature 0.2 and maximum of 512 output tokens. See Appendix A for more details.
10

---

Published in Transactions on Machine Learning Research (eebruary20224
Prompt: Generate concise description for the image.
z-vector: (pizza: 0.89, person: 0.74, oven: 0.42, bowl: 0.36, ... knife: 0.18, fork: 0.09, ...)
LLaVA: A person is cutting a
COAD: A pizza with one slice missing is being cut by person wearing nail polish.
pizza with knife and fork, and the pizza is placed on tray. The pizza has cheese and potatoes on it.
Green: Correct Objects
Red: Hallucinated Objects
Figure 6: Case study on caption generation. MSCOCO objects mentioned in the text are highlighted red (hallucinated) or green (correct). We compare the baseline LLaVA with our COAD-enhanced model. While LLa hallucinates nonexistent objects (e-g-, knife and fork), the z-vector produced by the object detector suggests that these objects are absent. By leveraging this signal, COAD produces faithful caption grounded in the actual image content, consistent with the improvements shown in CHAIR metries.
Table 1: Comparison of different methods in terms of CHAIR metrics. Boldface and underlining denote the best and the second-bes best performance, respectively.
Method
Base
PAI
DoLa
VCD
CAD
OPERA
EOS
HALC
COAD
CHAIR, +
9.9
5.8
13.0
11.4
9.9
4.5
5.8
5.2
3.4
CHAIRs +
29.6
11.3
37.0
32.5
28.0
7.4
10.6
11.1
5.3
4.4 Main Results
In this section, we compare COAD with different baselines across various datasets and metrics.
Free-Form Generation Evaluation CHAIR. We first evaluate COAD the CHAIR benchmark, which measures hallucination rates in free- form image captioning. The CHAIR benchmark includes two sub-metries: CHAIR, (instance-level) and CHAIRs (sentence-level), Lower CHAIR, and CHAIRs indicate fewer hallucinated mentions.
As shown in Table T COAD achieves the best performance across all three CHAIR metrics, significantly reducing hallucinations. Specifically, it achieves 3.4 and 5.3 in terms of CHAIR, and CHAIRs, respectively, outperforming all existing baselines. This demonstrates that our causal object-aware decoding effectively reduces hallucination of generated captions.
Figure 0 shows qualitative example comparing the baseline LLaVA and our COAD. Here, LLaVA hallucinates nonexistent objects such as a knife and fork, while our COAD correctly suppresses them and generates a more faithful caption This illustrates how causal object-aware decoding helps mitigate hallucination in practice. Additional case studies are provided Appendix F
Multimodal QA Evaluation MMHal-Bench. Table 2 shows the results MMHal-Bench. COAD achieves the highest average score (2.52) and the lowest hallucination rate (0.52), significantly outperforming all baselines. The strong performance is consistent across multiple benchmark subsets, particularly in the Attribute, Comparison, and Relation categories, indicating improved factual accuracy and reasoning. These results further demonstrate that incorporating object-level cues effectively reduces hallucination while maintaining or enhancing generation quality.
Object Probing Evaluation on POPE. Table shows the POPE evaluation results across three settings. COAD achieves the highest accuracy (79.8) and F1 score (81.2) on the Adversarial subset, outperforming all baselines, indicating better robustness to prompts designed to induce hallucination. In the Popular and Random subsets, it performs comparably to state-of-the-art methods in F1 while maintaining low
While COAD successfully removes the hallucinated objects such the "knife" and "fork", its output also includes the phrase "one slice missing", which may be inaccurate description of the pizza. This type of error concerns the attribates of already-present present object rather than the presence additional objects. Since COAD designed specifically to mitigate object hallocination, such attribute-level inconsistencies fall outside the scope what method targets.
11

---

Published in Transactions on Machine Learning Research (eebruar/2022)
Table Evaluation on MMHHal-Bench across hallucination dimensions: attributes (attr), adversarial objects (adv), comparison (emp), counting (cnt), spatial relations (rel), environment (env), holistic/ /vverall description
(hol), and others (oth). Boldface and underlining denote the best and the second- best performance, respectively.
Method
Avg. Score
Hall. Rate
attr
adv
cmp
ent
rcl
env
hol
oth
Base
1.88
0.68
2.33
1.25
2.67
0.83
1.75
3.17
1.42
1.58
PAT
2.10
0.65
1.92
1.33
2.25
2.17
2.17
3.67
1.75
1.58
Dola
2.01
0.62
2.08
1.42
2.75
1.67
1.17
4.00
1.75
1.25
VCD
1.98
0.67
2.17
1.83
1.83
1.33
2.42
3.33
1.33
1.58
CAD
2.00
0.64
2.50
1.25
2.42
0.75
1.33
3.83
1.83
2.08
OPERA
2.09
0.65
2.58
1.67
2.67
2.50
1.58
3.08
1.17
1.50
EOS
2.08
0.62
2.67
1.33
2.67
1.00
1.83
3.17
1.58
2.42
HALC
2.12
0.64
2.33
1.67
3.00
2.25
1.67
3.42
1.33
1.33
COAD
2.52
0.52
3.58
1.83
3.33
2.08
2.08
3.50
1.33
2.42
Table 3: POPE evaluation results the MSCOCO dataset. Boldface and underlining denote the best and the second-best performance, respectively.
Random
Popular
Adversarial
Method
Acc
P
R
F1
Yes
Acc
P
R
F1
Yes
Acc
P
R.
F1
Yes
Base
89.0
89.3
88.6
89.0
49.6
85.0
82.6
88.7
85.5
53.7
78.8
74.0
88.8
80.8
60.0
PAI
89.3
89.6
88.9
89.2
49.6
86.1
84.2
89.0
86.5
52.9
78.9
74.4
88.3
80.7
59.4
Dola
86.3
85.3
87.7
86.5
51.4
83.0
80.5
87.1
83.6
54.1
78.2
73.9
87.4
80.1
59.2
VCD
88.8
88.8
88.7
88.8
50.0
85.4
83.6
88.1
85.8
52.7
79.2
74.5
88.6
81.0
59.4
CAD
88.6
88.7
88.5
88.6
49.9
84.8
82.5
88.3
85.3
53.5
78.5
74.0
87.9
80.3
59.4
OPERA
89.4
89.7
89.0
89.3
49.6
85.9
83.9
89.0
86.4
53.1
79.1
74.3
89.0
81.0
59.9
EOS
85.4
81.5
91.7
86.3
56.3
81.2
75.8
91.7
83.0
60.5
75.9
69.6
91.9
79.2
66.0
HALC
88.7
89.9
87.1
88.5
48.5
85.8
84.8
87.1
86.0
51.4
79.1
75.0
87.1
80.6
58.1
COAD
89.0
89.6
88.3
89.0
49.3
85.5
84.0
87.6
85.8
52.1
79.8
75.8
87.5
81.2
57.7
hallucination ratio. These results confirm that approach effectively reduces hallucinations while preserving factual precision across diverse input types.
Ablation Studies. We conduct two ablation studies on CHAIR to better understand the source our improvements. Specifically, we compare our full COAD with (1) "COAD (M, Only)", which only uses the finetuned model M, without applying our causal decoding procedure and (2) "COAD (w/o z)", where we train M, without and perform causal decoding using this modified My. Table g shows the results. The gap between COAD and "COAD (M, Only)" verifies the effectiveness of our causal decoding algorithm, while the gap between COAD and "COAD (w/o z)" verifies the important role of COAD (see more discussion Appendix D
Table 4: Results of COAD and abla- tions CHAIR. "M, only" means only using the finetuned model M, for generation; "w/o z" means replacing M₁ by a normally finetuned MLLLM and applying COAD, without any z vectors involved in the whole process.
Method
CHAIR, 1 CHAIRs
COAD (Full)
3.4
5.3
COAD (My only)
5.4
10.8
4.5 Runtime and Computational Overhead
COAD (w/o z)
6.9
18.1
Beyond hallucination metrics, we also compare the computational overhead of COAD with existing methods. To ensure consistency with the CHAIR evaluation, we select 100 images from the MSCOCO subset used in CHAIR experiments and ask each method to generate free-form descriptions of the images using the
12

---

Published in Transactions on Machine Learning Research (February/202)
Table 5: Decoding throughput of different methods on 100 MSCOCO images used in the CHAIR evaluation.
We report the number of tokens per second during generation (higher is better).
Method
Base
COAD (Ours)
OPERA
PAI
DoLa
VCD
CAD
EOS
HALC
#tokens/s f
24.37
10.49
4.52
43.62
29.38
7.98
7.92
9.91
7.32
same decoding setting in Section 4.4 All methods are evaluated on single GPU. We measure decoding throughput in tokens per second, which reflects the effective per-token computational cost of each method. Table shows the results.
Detector Overhead. COAD invokes the object detector only once per image before docoding. Since auloregressive decoding dominates the total computational cost for MLLMs, this one-time delection overhead is relatively small: the RTMDet detector used in our implementation processes each image in about 0.10 scconds, which is negligible compared with the cumulative cost of token generation long outputs.
Dual-Model Decoding. COAD evaluates both the pretrained model and the object-aware finetuned model at each decoding step. When executed sequentially on a single GPU, this dual-model decoding results in roughly half the throughput the base LLaVA model (10.49 vs. 24.37 tokens/s in Table 5). However, the two forward passes are independent and can be executed fully in parallel on different GPUs, allowing throughput close to that of single-model decoding practical multi-GPU deployments.
Compared other hallucinatioon-iiiggaaee methods, COAD remains computationally competitive. In particular, it is significantly faster than multi-step refinement and beam-search/ /backttaakkiiggggggggeaa approaches such as OPERA (4.52 tokens/s), and comparable to other decoding-modificatie and HALC. Overall, COAD achieves strong reductions in hallucination with a moderate and well-characterized runtime overhead.
methods like VCD, CAD, EOS,
5 Conclusion
In this paper, we propose COAD, novel approach to reducing object hallucination in MLLMs. By combining object detection and causal inference, COAD improves the quality of generated captions and reasoning outputs. Extensive experiments on various benchmarks show that COAD consistently outporforms state-of-the-art. dehallucination methods across diverse metrics and settings.
Future work may include more sophisticated object representations and extend our causal modeling framework additional multimodal tasks. Since the number of object categories determined by the detector, it would also be interesting explore open-vocabulary detectors (e.g., GLIP (Li et al. 2022)), which may allow COAD to operate over substantially richer and more flexible object space. Moreover, we plan to investigate the integration of temporal and spatial priors to further enhance the causal grounding of visual elements. Another promising direction is to incorporate user feedback or human-in-the-lod dynamically refine the intervention policy during inference. Finally, we aim to explore the scalability of COAD in real-world applications such as assistive vision systems and visually grounded dialogue.
supervision to
In terms of limitations, like many other MLLMs, maliciously manipulated inputs could affect COAD's performance. Another limitation that. COAD primarily targets object hallueination; extending the causal modeling framework to other forms of hallucination, such attribute, relational, and global-scene inconsistencies, remains important direction for future work. We defer detailed discussion of limitations and potential mitigations to Appendix
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877, 2023.
13

---

Published in Transactions on Machine Learning Research (obbuary//220
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback, 2024a. URL
10081
Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint arXiv:2403.0044 2024b.
Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models, 2024. URL https: orw. 2309.03883
Ailin Deng, Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding, 2024a. URL
15300
Ailin Deng, Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding. Xiv preprint arXiv:2402. 0etpe/222110300 15300, 2024b.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Authony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. Rarr: Researching and revising what language models say, using language models, 2023. URL https: pestps://arxxee
Zongbo Han, Zechen Bai, Haiyang Mei, Qianli Xu, Changqing Zhang, and Mike Zheng Shou. Skip A simple method to reduce hallucination in large vision-language models, 2024. URL https:
abs/2402.01344
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, November 2024a. ISSN 1558-2868. doi: 10.1145/370311 URL http://dx doi. http///.doo.o///....///o// 1145/3703155
Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust. penalty and retrospection-alloaatio, 2024b. URL https: htaps:///rxxvv wwwpxiiv.org/aaa. wttg///b///...11999999
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations large vision- language models through visual contrastive decoding, 2023. URL
https://arxiv.orr//b//22311111 16922
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 19955-10975, 2022.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object. hallucination in large vision-languaae models. arXie preprint arXiv:2305.1088 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva. Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, Seplember 6-12, 2014, proceedings, part v 13, pp. 740 755. Springer, 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large mulli-modal models via robust instruction tuning, 2024a. URL https://a/xxx org/abs......o... /306. 14565
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongiun Li, and Wei Peng. A survey on hallucination in large vision-language models, 2024b. URL https:
//arxiv.orge
Haotian Liu, Chumyuan Li, Yuheng Li, and Yong Jac Lec. Improved basclines with visual instruction tuning, 2024c. URL http:///aaxivv..r/abbaass 20t../2300.0000 03744
14

---

Published in Transactions on Machine Learning Research (February/202)
Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training- free method for alleviating hallucination in Iylms, 2024d. URL https://arxxi..rrg/ 2tclbb//442..2
Chengqi Lyu, Wenwei Zhang, Haian Huang, Yue Zhou, Yudong Wang, Yanyi Liu, Shilong Zhang, and
Kai Chen. Rtmdet: An empirical study of designing real-time object detectors, 2022. URL https: //arxiv. wwwpxiv.org//bbse. wttg/bb//22...//////32
Chengzhi Mao, Augustine Cha, Amogh Gupta, Hao Wang, Junfeng Yang, and Carl Vondrick. Generative interventions for causal learning- In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 3947 3956, 2021.
Chengzhi Mao, Kevin Xia, James Wang, Hao Wang, Junfeng Yang, Elias Barcinboim, and Carl Vondrick. Causal transportability for visual recognition. In Proceedings of the IEEE/CVE /CV Conference on Compuler Vision and Pottern Recognition, pp. 7521-7531, 2022.
Judea Pearl. Causality. Cambridge university press, 2009.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprinl Xrv: 02156, 2018.
Weijia Shi, Xiaochnang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tan Yih. Trusting your evidence: Hallucinate less with context-aware decoding, 2023. URT. https://arxiv.. org/ aa/////aa/////// 2305
14739
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually auginented rlhf. arXiv preprint arXiv:2309.144 14525, 2023.
Hugo Touvron, Thibaut Lavril, Gantier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaune Lample. Llama: Open and elficient foundation language models, 2023. URL
https://arxiv wwwpp/////..b...oo.o//// 13971
Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. stitch in time saves nine: Detecting and mitigating ballucinations of Ilms by validating low-confidence generation, 2023. URL
https://arxiv.ot org/ arggabs// abs/ 211.22277.00093
Tu Vu, Mohit. Tyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. Freshlims: Refreshing large language models with search engine augmentation, 2023. URL https://arxiv.. wwwpo//gga/...ooooe///r 22s.//221100001 03214
Hao Wang and Din-Yan Yeung. Towards bayesian deep learning: A framework and some existing methods. TDKE, (3(12):3395-3333 2016.
Hao Wang and Dit-Yan Yeung. survey on bayesian deep learning. CSUR, 53(5):1-33 37, 2020.
Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. Mitigating fine-graince hallucination by finc-tuning large vision-language models with caption rewrites, 2023. URL https://aaxii: org/ abs/ 2b12
01701
Yibin Wang, Haizhou Shi, Ligong Han, Dimitris Metaxas, and Hao Wang. Blob: Bayesian low-rank adaptation by backpropagation for large language models. In NeurIPS, 2024.
Yuhao Wang, Vlado Menkovski, Hao Wang, Xin Du, and Mykola Pechenizkiy. Causal discovery from incomplete data: A deep learning approach. In AAAI StarAl Workshop, 2020.
Jingquan Yan and Hao Wang. Self-interpretable time series prediction with counterfactual explanations. In ICML, 2023.
15

---

Published in Transactions on Machine Learning Research (obbuarr//2200
Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and
Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data, 2024. URL
https://arxiv. /tttg://arxxiv.rrg//bb///// http//bb/22311.1800 13614
Zihao Yue, Liang Zhang, and Qin Jin. Less is more: Mitigating multimodal hallucination from an eos decision perspective, 2024. URL https: pettps//arrxvy wwwaaiiww...aaaaa....a. wwtp/b///////a../. 2402. 14545
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhosciny. Minigpt-4: Enhancing vision- language understanding with advanced large language models, 2023. URL lnttps://araa org/ 2304
10592
A Details on Implementation and Causal Graphs
Implementation Details. We finctune COAD on subset of MSCOCO images sourced from the LLaVA dataset. To enable the model to incorporate the auxiliary input z, we introduce two-layer MLP projector (with hidden size 256) that maps into the token embedding space, following LLaVA's multimodal token integration approach. We employ LoRA (r = 128, = 256), cosine learning rate schedule with an initial learning rate of 4c-5, a batch size of 128, and train the model for 1 epoch. To mitigate the model's dependence on prior context, we apply Gaussian noise (a 0.005) to the embeddings of previous tokens with a probability of 0.5 during training. During inference, we use sampling by default with temperature 0.2 and a maximum of 512 output tokens. In implementing Equation | we find that it more effective to perform fusion the logit space rather than the probability space. Therefore, we replace P(yy|S, x, z) and P(yp|S,)) x) with their corresponding logits before computing the fused output, which is subsequently converted back to the probability space via softmax.
All experiments were conducted on a single machine with 8 NVIDIA RTX A5000 GPUs (24GB each), an AMD EPYC 7282 16-Core Processor (64 threads), and 256GB RAM. Finetuning typically took around 16 hours per model. Caption generation on 5,000 images took between 30 minutes and hours, depending on the generation length.
For evaluation, use sampling to generate outputs for all baseline methods, except for OPERA. Since OPERA is built on top of beam search, its outputs are generated using beam search with beam size of 3 instead. For the hyperparameter o in COAD, we set it to 1.5 for text generation tasks (CHAIR and MMHHal-Bench) and 0.1 for POPE.
Causal Graphs and Hidden States. There are two different hidden states:
The image-based, static hidden state z, which corresponds to the output of the MLLM's vision encoder (i.c., H, that we will further explain in Appendix Ch. In a causal MLLM such as LLaVA, the input follows the structure "image user prompt generated tokens", and the hidden state of a Loken position affected only by tokens that appear before it. Therefore, here determined solely by the image and aligns most closely with our interpretation of the object belief variable in causal graphs in Figure 4 and Figure 5 It is not influenced by
The response-based, dynamic hidden state h (or h)), which corresponds to the hidden states when the MLLM generates the t-th response token. These hidden states are influenced by x.
We further clarify the role of the dynamic hidden state (t) in the causal graph shown in Figure 7 In
Figure
the image S generates the image- static hidden state z,
S, z, and (() then jointly generate (influence) the response-based, dynamic hidden state h()) and
h0 then influences the generated token (t))
As shown in Figure 7b we can actually collapse h(()) and the red part of Figure to have an equivalent
causal graph in Figure 7b which matches our causal graphs in Figure 4 and Figure
16

---

Published in Transactions on Machine Learning Research (February/ 2026)
S
S
Z
Z
(t)
(t))
h
y(1)
(tt)
0,1, T
t =
(a) Casual graph with hidden states h (1), The hidden states that decide y are affected by S, and x().
(b) Collapsing hidden states h()) into the generation
of y²) The resulting graph corresponds to the cansal graph COAD Figure 5
Figure 7: The role of hidden states in the decoding causal graph of an MLLM. This figure illustrates causal graph that explicitly includes token-level hidden states h( during generation (a), and shows that collapsing these hidden states into the generation of (t) yields equivalent causal graph consistent with the one used by COAD (b).
B Evidence of Hallucinatory Object Beliefs in LLaVA
examine whether a multimodal LLM (MLLM) can form incorvecl internal beliefs about object existence (corresponding to an incorrect z in our formulation), we conduct linear-probe experiment on LLaVA. The goal is to determine whether LLaVA forms internal object-cxistence beliefs that deviate from the actual content of the image.
B.1 Experimental Setup
We train a linear classifier to probe whether LLaVA internally believes each object category to be present in the image. The classifier
takes as input the embedding for each token, and
outputs a C-dimensional vector o E [0, 1J indicating the estimated probability that each of the C' objects exists the image.
Here, reflects the MLLM's object-existence belief in its hidden states, which mentioned in Appendix
Input to the Linear Probe. For each token in the dialog sequence, LLaVA produces a hidden state after every transformer decoder layer. We concatenate these hidden states together with the embedding before the first layer to obtain the probe input. Thus, each token in the prompt-repponse sequence contributes one training sample for the linear probe.
Conversation Sampling. We collect dialogs generated by LLaVA using MSCOCO images and the prompt "Please describe every object in this image in detail. After LLaVA completes the response, extract hidden states for all tokens (including image tokens, query tokens, and generated response tokens) to construct probe input samples.
Target Labels. For each image, we run LLaVA once with the same prompt and use a rule-based method determine whether each object category mentioned in the resulting text description. These detected mentions serve labels for training the linear probe. The same label assigned all probe samples within given image-prone pronge-prompt-esposeeeee tuple.
17

---

Published in Transactions on Machine Learning Research (ebruary/202)
Discussion the Relation between Input to the Linear Probe and the Hidden States. In causal MLLM such as LLaVA, the input follows the structure "image user prompt generated tokens", and the hidden state of token position is affected only by tokens that appear before it. Therefore:
when the imput to the linear probe is the hidden state in the "image" part, the output serves estimation of the image-based, static hidden state z;
when the input to the linear probe is the hidden state in the "generated tokens" part, the output o serves estimation of the response-based, dynamic hidden state h (or h(?).
B.2 Findings
We apply the trained linear probe to a sampled conversation and visualize the predicted object probabilities over time (Figure
The input image contains a person riding skateboard with no benches in the background.
The probe output in Figure (a) indicates that LLaVA internally assigns high probability (i..., brighter, yellow color in the heatmap) to nonexistent bench around certain image-token positions.
Figure 8(b) shows LLaVA's probed probability of the object "bench" " over the image tokens (patches), demonstrating high "bench" probabilities in some image tokens (highlighted in yellow); this indicates that the baseline LLaVA's hallucinated belief may originate from the pereeppernnttoon stage.
Figure
c) shows that LLaVA hallucinates the object "bench" " in the response.
In contrast, COAD produces a response that does not mention any benches, as shown in Figure g(d).
B.3 Conclusion
This experiment demonstrates that LLaVA can form incorrect internal beliefs about object existence, i.e., it may estimate an incorrect z for certain objects, which subsequently leads to hallucinations. These results provide direct empirical evidence for the assumption illustrated Figure 1 of the main paper.
C LLaVA Architecture and the Formation of Visual Features
To clarify why the object-belief variable z is treated as time-invariant in the causal model of a standard MLLM (Section 3.4]. we briefly describe the relevant components of the LLaVA architecture. A schematic diagram shown Figure g
C.1 Visual Feature Extraction Before Language Interaction
In LLaVA, the input image first processed by a vision encoder (e.g., CLIP's VIT backbone) to produce a set of image features, denoted Z.. These features are then passed through a projector layer to obtain the final visual embeddings H,, which are fed into the language model.
Importantly:
Both the vision encoder and the projector operate independently of the language model.
No textual tokens are involved when computing Z, or H..
The visual embeddings H, remain fixed throughout decoding.
The projected embeddings H, are inserted into the token sequence as "image tokens," and only after this insertion does the multinodal Transformer start attending jointly image tokens and text tokens.
18

---

Published in Transactions on Machine Learning Research (February/002)
Object Probability
trfff Sgle
200 9421
OK
S-nou
Ingl
Img]
Token
[mg]
]mg]]
3mq]
[mg]
Ing]
Ing]
des
Jocaied
0.0
Linear-pres probe output for object probabilities in LLaVA.
image captures a skateboarder in mid-air, performing trick ramp. The skateboarder wearing helmet for safety while executing the jump. The skateboard is Additionally, there are two benches the scene, lo cated near the left edge of the image and other towards the right side.
LLaVA:
In the image, young person is performing skateboard trick, flying through the while riding the skateboard. The skateboard is clearly visible beneath the skateboarder, and the person appears to in mid-air, showcasing their skill. The scene captures the excitement and athleticism of the sport.
COAD:
(b) Visualization of probed probability of the object "bench" " over all image tokens.
(c) LLaVA's response hallucinates the object "benches".
(d) COAD eliminates the bench- related hallucination.
Figure Visualization of LLa hallucinatory object. beliefs. This figure provides empirical evidence that LLaVA can form incorrect internal object-existence beliefs (corresponding to an incorrect z) and shows how COAD corrects the resulting hallucination. (a) Lincar-probe output of LLaVA's object probabilities for all tokens over the full dialog, showing that the probed probability for the object "bench increases both around certain image tokens and around the tokens where LLaVA actually generates the word "benches" (in red boxes). (b) The baseline LLaVA's probed probability of the object "bench" " over the image tokens (patches), showing high "bench probabilities in some image tokens (highlighted in yellow); this indicates that the hallucinated belief may originate from the image-perception stage. (c) L.L.VA hallucinates the objoct "bench" in the response. (d) COAD's response, which climinates the bench hallucination.
19

---

Published in Transactions on Machine Learning Research (February/202)
Language Response
X
Language Model fØ
Projection W
H
Hgg
Z
Vision Encoder
X v Image
X q Language Instruction
Figure 9: Simplified LLaVA architecture (Liu et al. 2024c). The image is encoded by vision encoder and then projected into the language model's embedding space. The resulting visual embeddings H. are inserted as image tokens. Because He is computed independently of textual tokens, the object-belief variable z corresponding to H, remains fixed during decoding. (This figure is adapted from (Liu et al. 2024ch.
C.2 Relation to the Object-Belief Variable z
In our causal formulation, the variable z represents the model's internal belief about which objects exist in the image. Conceptually, corresponds to the portion of the visual embeddings H, that encode object presence absence. Since He is generated solcly from the image (via the vision encoder and projector) and does not depend on previously generated text tokens, z is naturally treated as static variable. This matches the assumption used in Section 3.4. where only (t) and ((t) evolve over time.
D Further Analysis of Ablation Studies
Effect of Finetuning and Effectiveness of Our Causal Decoding Algorithm. Since COAD involves finetuning an MLLM, natural question is whether the observed gains are simply due to finetuning rather than our proposed causal decoding algorithm. To examine this, we directly evaluate the finetuned model M, without applying our decoding strategy. As shown Table I My alone achieves only part of the improvements, indicating that finctuning by itself cannot account for the performance COAD and verifying the effectiveness of our causal decoding algorithm.
Role of the Vector Another question is whether the improvements and Mp during causal fusion, regardless of our vector z. To verify this, remove when training My (iee. a standard finetuning setting) and then apply our causal decoding procedure using this variant of My. The results from Table show a clear drop compared to COAD, demonstrating that z plays an essential role in enabling effective causal fusion.
merely from contrasting My
E Limitations and Future Work
Dependence on Finetuning. Our approach currently requires a LoRA finetuning step for adaptation. While this is practical in many settings, reduces the plug-and-play convenience of COAD. We experimented with a training-free variant that injects the causal vector directly as a prompt, which alrcady yiclds strong improvements on POPE benchmarks (e.g., FI-Rand 95.4, F1-Pop 90.0, F1-Ady 85.8), outperforming all baselines. However, this variant is more sensitive to detector errors and less robust captioning tasks. These results nonetheless highlight the generality of COAD and suggest promising directions for reducing computational cost, such as improving detector reliability or designing dedicated training methods that allow a single MLLM to simulate the causal signal.
Domain Mismatch. COAD relies on detectors trained on specific distributions. the test image domain diverges significantly, the causal signal may become insufficient. One direction is to investigate zero-shot domain-adaptive detectors to mitigate this issue.
20

---

Published in Transactions on Machine Learning Research (obbraary/2000
Adversarial Vulnerabilityy Maliciously manipulated imputs detector outputs could affect COAD's performance. However, its modular design allows for safeguard components (e.g., adversarial detection at the detector level), which we leave as future extensions.
Residual Text Priors. In rare cases with extremely strong linguistic priors, causal interventions may not fully suppress hallucinations. In rare cases with extremely strong linguistic priors, causal interventions may not fully suppress hallucinations. Future improvements may involve designing stronger intervention mechanisms or complementary signals that better counteract such priors.
Scope and Generality. Our study mainly targets object hallucinations. Broader validation other types of hallucinations and across more MLLMs is a promising next step, facilitated by COAD's general token-based interface.
F More Qualitative Examples
We provide more qualitative examples in Figure 10 and Figure 11 where MSCOCO objects mentioned in the text are highlighted in red (hallucinated) or green (correct).
21

---

Published in Transactions on Machine Learning Research (February/200) 2026)
z-vector: [person: 0.93, handbag: 0.76, cell phone: 0.72, potted plant: 0.5, clock: 0.26, bottle: 0.24, bench: 0.2, ...]
z-vector: [toilet: 0.86, cat: 0.84, book: 0.46, dog: 0.17, ...]
LLaVA: A dog is standing on a toilet, sniffing the toilet bowl. The toilet lid open, and the dog appears to be curious about the water inside.
LLaVA: A woman is sitting on a bench, holding cell phone and talking to another woman. They are both wearing sandals.
COAD: A cat is sitting on the toilet and appears to be drinking water from the toilet bowl.
COAD: A woman in pink shirt holds cell phone, while another woman in a dress sits next to her.
Midi, Decomber SNCE Class D8, Slemens Destro
z-vector: [train: 0.91, chair: 0.26, bird: 0.21, bench: 0.2, person: 0.16, clock: 0.15, traffic light: 0.14....
zevector: [banana: 0.87, apple: 0.87, orange: 0.83, bowl: 0.33, dining table: 0.28, wine glass: 0.11, cup: 0.05, ...]
LLaVA: A yellow and white train is traveling down the tracks, pulling into a station. The train is positioned next to a building, and there is bench nearby for passenger to sit. The scene is captured on November 1, 2012.
LLL A clear glass plate is filled with a variety of fruits, including apple, orange, and banana. The plate is placed on table, and the fruits are arranged in visually appcaling manner.
COAD: A clear glass plate holds a variety of fruits, including banana, apple, pears, and an orange.
COAD: A train is stopped at train station, and there is yellow line the platform.
Figure 10: Case studies on hallucination reduction.
22

---

Published in Transactions on Machine Learning Research (February/20)) 2026)
z-vector: [laptop: 0.88, bed: 0.86, suitcase: 0.63, keyboard: 0.44, mouse: 0.42, backpack: 0.36, hand- bag: 0.33, cell phone: 0.31, remote: 0.15, person: 0.14, ...]
z-vector: [hot dog: 0.91, person: 0.77, sandwich
0.31, bottle: 0.16, pizza: 0.07, dining table: 0.04, ...]
LLa VA: A person is holding hot dog with cheese and ketchup on bun. The hot dog is placed on table, and the person's hand is visible in the foreground.
LLaVA: bed with a laptop and cell phone on
it. The bed is unmade and has striped blanket.
There is also a suitcase the bed, and pair of
glasses nearby.
COAD: A hot dog with ketchup and cheese being held by a person.
COAD: laptop is sitting on top of a bed next.
to a suitcase.
LCC
z-vector:
[bicycle: 0.89, person: 0.83, motorcycle: 0.75, backpack: 0.34, bottle: 0.27, handbag: 0.18, chair: 0.05, ...]
z-vector: [person: 0.85, bird: 0.83, cup: 0.83, sandwich: 0.64, bench: 0.55, dining table: 0.5, fork: 0.33, chair: 0.32, knife: 0.22, ...]
LLaVA: A man is working on motorcycle in red stool to reach the motor- cycle. He is wearing jeans and black shirt. The motorcycle is parked on lift, and there are other molorcycle in the background.
LLaVA: A bird is standing on a plate with half- eaten sandwich, which is placed on dining table. The bird seems to be interested in the sandwich, possibly trying to get bite. The scene takes place near body of water, with a bench nearby.
garage, standing
COAD: A man is working on a motorcycle, lifting the front wheel off the ground using lift.
COAD: A half-caten sandwich sits on plate with ketchup, and bird is standing nearby, possibly interested in the remaining food.
Figure 11: Case studies on hallucination reduction.
23


Reviews:
[{"review_id": "tAY8jmS6JN", "reviewer": "Reviewer_XSGh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to address visual hallucinations in MLLMs by explicitly modeling the hidden state corresponding to the object attributes in the images and then finetuning the MLLM to accommodate this hidden state during its generation. During inference, this fine-tuned MLLM is incorporated with the pre-trained using a mixture assumption to obtain samples from the unobserved oracle generative model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "There is no explicit claim made in the paper. The authors may point it out if I missed it. The implicit claim that I have understood from the paper is: we can mitigate visual hallucination in MLLM by explicitly modeling the hidden state $z$ that generates the hallucination by being causally affected by the prior language output $x$. This claim is partially supported by the current evidence.\n\nHere, I have two concerns:\n\n1. The claim rests on the assumption that there is a $z$ that incorrectly captures the object attributes. The proposed method, COAD, then fixes this $z$-estimation. But there is no evidence in the work showing that an incorrect $z$ is estimated by regular MLLMs. While I believe that showing the estimated $z$ is incorrect is difficult, it also leaves the claim on a bit shaky ground. See C1 in the requested changes.\n2. In Figs. 1 and 4, the causal graphs show $s\\rightarrow x\\leftarrow z$. How is this graph justified? In the experiments and in a practical case, the prior text will initially be just the query, which may not contain any information about the objects in the image. E.g., \"Describe the image in detail.\" (Fig. 3). Then, as the model generates outputs, $z$ estimation can go wrong if it is influenced by the previously generated tokens. But for that $x\\rightarrow z$, correct? See C2 in the requested changes.", "audience_yn": "Yes", "audience_explanation": "This work is definitely of interest to the ML audience, as hallucination mitigation is a very popular research goal.", "requested_changes": "**Major changes**:\n\nC1. Is it possible to show that the scenario depicted in Fig. 1 (where some hidden state $z$ hallucinates objects) indeed happens?\n\nC2. Can you add a causal graph with $s$, $z$, $x$, and $y$ rolled out over time? I suppose $s$ and some prior text $x_0$ (such as the query \"describe the image\") will be fixed, as they are provided by the user. But $z$, $x$, and $y$ change with time. The causal edges between variables at timesteps $t$ and $t+1$ will clarify my confusion.\n\nC3. How are $\\gamma_a$ and $\\gamma_b$ decided? Are they learned, and, if so, how? If they are not learned, is it varied per sample, dataset, or MLLM?\n\n**Minor changes**:\n\nC4. The claim needs to be slightly adjusted, as visual hallucination generally includes hallucination about the objects, their attributes, their counts, their positioning, etc., while the proposed method (at least in its current form) can only address hallucination about the object categories and their attributes. This is also evident in Tab. 1, where COAD does not outperform the baselines in \"cnt\", \"rel\", etc.\n\nC5. I saw the new Sec. 3.6 in the revised version. The section cleared a major confusion for me. Please move it ahead to Sec. 3.1 so that readers can focus on Sec. 3 without wondering how a causal edge could just be \"added\", as Fig. 2 says.\n\nC6. Is the number of object categories, $C$, fixed depending on the detector?\n\nC7. I would like to know the authors' opinion on the spiritual relation between this submission and [R1]. Both works use an external detector (CLIP in [R1]) to ground the generated text. [R1] is a post-hoc filtering scheme, while this works implicitly filters out hallucinations through fine-tuning.\n\nC8. In the final algorithm, how exactly are samples from $z$ used? Are there $N$ samples from $z$, or is the probability vector $\\tilde{z}$ used as the input and then multiple $y$'s are sampled from $M_f$? (Related to page 7 in the revised PDF, just before \"Summary of COAD.\")\n\nC9. In Fig. 5, COAD says, \"a pizza with one slice missing.\" But the pizza is not missing a slice.\n\n**References**:\n\n[R1] Deng et al., \"Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding,\" arXiv 2024.", "broader_impact_concerns": "I do not have any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3jLw40cIBq", "reviewer": "Reviewer_8MLm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes COAD, a method to mitigate visual hallucinations in MLLMs. The authors argue that hallucinations arise from spurious correlations where generated text ($x$) incorrectly influences the latent states ($z$) representating visual objects. To address this issue, they propose an intervention-based framework that conditions generation on an explicit objective belief vector derived from an external object detector, which blocks the counding causal path from $x$ to $z$. \n\nThe experiments are strong and demonstrate SOTA performance on hallucination benchmarks, e.g., POPE and CHAIR; however, from my point of view, the analytical framwork, especially the causal formalism is flawed. A major issue is that the paper conflates the data-generating process (a hypothetical belief, i.e., how real data is produced) with the inference process (how the inference model processes data flow). \n\n**Strengths:**\n1. The visual hallucination problem is an important and timely topic in vision-language modeling.\n2. The empirical execution appears robust and the results support the efficacy of the proposed mechanism. \n\n**Weaknesses:**\nMy primary concern lies in the construction of its causal graph and problem formulation. The authors attempt to ground their method in causal reasoning, but the formulation in the paper violates standard principles of structual causal modeling and causal representation learning. \n1.  Misconcepturalization of causal graph (e.g., Figure 1 & 4). This paper presents a \"causal model\" where the input image $S$ and previous text $x$ are treated as the parents of the latent obejctive variable $z$. However, in a vliad causal generative model, latent variables are the cause, and observations are the effect. Therefore, the data-generating process should be $z\\to S$, not $S\\to z$. It seems that the authors have confused the influence flow with the data mechanism.\n\n2. Following the previous confusion, the formulation in Section 3.3 asserts the latent variable $z$ is sampled as $z\\sim D(S)$. In formal causal literature (e.g., Pearl's causal formalization, or Scholkopf's causal representation learning view), $z$ is an ubobserved confounder that exists independently of observation. \n\n3. The definition of the hallucination problem in introduction and Section 3.1 relies heavily on illustrative anecodotes rahther than analytical objectives, which lacks a formal definition based on probability or information theory.", "claims_and_evidence_yn": "Yes", "claims_explanation": "While the submission presents convincing empirical evidence for the efficacy of the proposed method, the claims regarding the causal nature of the framework are not supported by accurate formulation, as noted in the weaknesses above.\n\n--- \n**Update:** In the revisied maniscript, my initial concerns on formality are largely addressed. Therefore, I have changed my evaluation to Yes.", "audience_yn": "Yes", "audience_explanation": "The paper addresses visual hallucination, which is currently a high-priority challenge in the field of vision-language modeling. As MLLMs are increasingly deployed in real-world scenarios, ensuring reliability and reducing hallucination is a critical research direction. The authors provide extensive experiments on standard benchmarks (e.g., CHAIR, POPE) and demonstrate that their method achieves state-of-the-art performance in improving faithfulness without degrading general captioning quality. The empirical success of using auxiliary object detectors to guide decoding provides a valuable data point for researchers working on robust generation strategies.", "requested_changes": "1. Redraw the key figures (e.g., Figure 1 and Figure 4) to improve presentation clarity. See previous comments for detailed suggestions.\n\n2. Refine terminology: Avoid stating \"observation $S$ causally generates latent $z$\". The off-the-shelf detector can instead be framed as inferring a proxy variable $\\hat{z}$ for the true latent obects $z$, and this proxy can then be used to intervene on the generation process, blocking the spurious path from previous tokens to the object representations.\n\n3. Formalize Section 3: Replace the illustrative knife/fork example with a formal objective focused  on the devergence between the model's output and the true conditional distribution give the (proxy) ground-truth objects.", "broader_impact_concerns": "No ethical concerns were identified.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nDn9affn4w", "reviewer": "Reviewer_WeCW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes Causal Object-Aware Decoding (COAD) to reduce visual hallucinations in multimodal large language models (MLLM). It uses an objectdetector to detect objects, which are used to fine-tune MLLM . It performs causal inference to combine the predictions from the original pretrained model and the object-aware finetuned model, and uses this mixture distribution to generate the final response. At inference time, COAD applie cauual inference for both the objective and the estimation of oracle predictions using an unified inference objective.. Implemented on LLaVA-1.5, the method shows consistent reductions in hallucination rates on image captioning and VQA benchmarks while largely preserving or improving overall task performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper claims that COAD reduces visual hallucinations while improving task performance. This claim is supported by experiments on Free-Form Generation on CHAIR dataset, Multimodal QA on MMHal-Bench dataset with eight quantitative metrics, Object Probing (POPE), and ablation studies, all conducted against strong baselines. \n\nHowever, the paper does not clearly explain the eight hallucination dimensions used in MMHal-Bench, and a more thorough justification of these dimensions would further strengthen the paper.", "audience_yn": "Yes", "audience_explanation": "The paper addresses a key limitation of MLLMs, namely visual hallucinations, and provides a reasonable integration of causal inference to improve vision–language models, which would interest the TMLR audience.", "requested_changes": "1. In Section 3.1, please clarify or revise the notation, where t denotes hidden states about object, h is previous tokens, i is the next token. This is confusing as h usually denotes hidden states, while t is usually for time.\n\n2. In section 3.1, it states that t as an unobserved confounder and leads to spurious correlations in p(i | h), while the next token i is not directly depended on previous token h. However, it is a an assumption proposed by this paper, which is conflicted with the established auto-regressive assumption. The authors may briefly justify why it is reasonable for MLLM.\n\n3. Please provide a clearer discussion of the computational complexity and runtime overhead of COAD. We have the extra cost introduced by the object detector and by running both the pretrained and finetuned models during decoding (e.g., FLOPs, latency, throughput), thereby authors should compare with the main baselines to show trade-offs.\n\n4. Please provide a brief explanation for the eight hallucination dimensions in MMHal-Bench, so that readers can better understand why these metrics are appropriate for measuring hallucination rate.\n\n5. Since the Introduction distinguishes two main categories of hallucination-mitigation methods (external knowledge based vs. architecture/decoding modifications), it would be helpful if Section 4.2 explicitly grouped the baselines according to these two categories. The details of baseline methods can be provided in appendix. \n\n6. We may consider adding a small qualitative case study in the main paper to illustrate how COAD affects generated outputs, with the full case-study results placed in the appendix.", "broader_impact_concerns": "NA", "additional_comments": "NA", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)