Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding
Authors: Yixiong Fang
Ziran Yang
Zhaorun Chen
Zhuokai Zhao
Jiawei Zhou
Venue: NeurIPS 2025 poster ()
Keywords: vision language models
multimodal AI
trustworthy
hallucination reduction
uncertainty quantification
dropout decoding
inference time algorithm
Primary area: deep_learning

Abstract: Large vision-language models (LVLMs) excel at multimodal tasks but are prone to misinterpreting visual inputs, often resulting in hallucinations and unreliable outputs. We present Dropout Decoding, a novel inference-time approach that quantifies the uncertainty of visual tokens and selectively masks uncertain tokens to improve decoding. Our method measures the uncertainty of each visual token by projecting it onto the text space and decomposing it into aleatoric and epistemic components. Specifically, we focus on epistemic uncertainty, which captures perception-related errors more effectively. Inspired by dropout regularization, we introduce uncertainty-guided token dropout, which applies the dropout principle to input visual tokens instead of model parameters, and during inference rather than training. By aggregating predictions from an ensemble of masked decoding contexts, we can robustly mitigate errors arising from visual token misinterpretations. Evaluations on benchmarks including CHAIR, THRONE, and MMBench demonstrate that Dropout Decoding significantly reduces object hallucinations (OH) and enhances both reliability and quality of LVLM outputs across diverse visual contexts.

Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding
Yixiong Fang" Carnegie Mellon University yixiongfèes cmu. edu
Ziran Yang" Princeton University
Zhaorun Chen University of Chicago
ziranyOprinceton. edu
Chaorun@uchicagoo edu
Zhuokai Zhao
Jiawei Zhou'
University of Chicago zhuokai@un zhuokaiBuchi @uchicag..iccccocooooco edu
Stony Brook University
j jiawei. zhou. 18stonybrook. edu
Abstract
Large vision- vision-language models (LVLMs) excel at multimodal tasks but are prone misinterpreting visual inputs, often resulting in hallucinations and unreliable outputs. We present DROPOUT DECODING, novel inference-time approach that quantifies the uncertainty of visual tokens and selectively masks uncertain tokens to improve decoding. Our method measures the uncertainty of each visual token by projecting onto the text space and decomposing into aleatoric and epis- temic components. Specifically, we focus on epistemic uncertainty, which captures perception-related errors more effectively. Inspired by dropout regularization, we introduce uncertainty-guide token dropout, which applies the dropout principle input visual tokens instead model parameters, and during inference rather than training. By aggregating predictions from an ensemble of masked decoding texts, we can robustly mitigate errors arising from visual token misinterpretations. Evaluations benchmarks including CHAIR, THRONE, and MMBench demon- strate that DROPOUT DECODING significantly reduces object hallucinations (OH) and enhances both reliability and quality of LVLM outputs across diverse visual contexts. Code is released at https: //t/thub...t..t// thub. com/kigh Dtwraagb/roooouecodiingg
Introduction
Recent advancements in large vasion-language models (LVLMs) have demonstrated impressive capabilities [ 2 3 in tasks such as image captioning, visual question answering (VQA), and multimodal reasoning [4] 5 77. However, LVLMs still face challenges in accurately perceiving and interpreting visual imputs, leading to inaccurate outputs and hallucinations [8]. These issues often stem from LVLMs misrepresenting key image elements overlooking critical details 9 [0]. In practice, LVLMs typically process visual inputs token by token [I]. which we refer to as visual f This can fall short in effectively focusing the most informative parts of the visual context. While attention mechanisms are designed to prioritize relevant information, they are not always perfect [12 13 14, especially when the inputs complex ambiguous for the model, in other words, of high uncertainty. Existing methods to address these challenges in the training stage often using additional supervision signals especially at lower level to guide the model 20 21 However, these approaches are resource-intensive
tokens
involve fine-tuning specific tasks [15] 16 17 18 19]
"Equal contribution. Work done during their research internship Stony Brook University.
Joint last author.
We specifically refer the tokens are already the input prompt to the text decoder. Concrete definition
is in 3.1
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and not easily extensible to new tasks, Alternative inference-tine strategies rely on attention or logits-
based mechanisms but typically use heuristic designs and increase inference cost [2] 23 24 25}.
Therefore, enhancing the trustworthiness of LV (26] and reducing hallucinations 24 require more principled methods that can more effectively emphasize the most informative parts of the visual input.
To address this challenge, we propose novel approach that quantifies uncertainty in visual token contexts and removes uncertain tokens, both directly at inference time to improve the reliability of LVLM outputs. Inspired by traditional dropout [27] techniques typically applied to model parameters but difficult to implement directly in pretrained LVLMs [28, 29] we
we introduce token dropout, which applies the dropout principle to imput context tokens instead of model parameters.
Furthermore, it is applied to regularize the inference process instead of training, by introducing randomness in decoding contexts to reduce overfitting to noisy visual tokens.
Our method measures the uncertainty of each visual token by projecting it into the text token space through the text decoder directly, and decomposing this uncertainty into two components: aleatoric (data-related) and epistemic (model-related) [30 31 (2). By focusing on epistemic uncertainty, which reflects the model's lack of knowledge, we identify visual tokens with high uncertainty and selectively target them for suppression. At inference time, we adjust the visual inputs by selectively suppressing tokens with high epistemic uncertainty. Specifically, we create an ensemble of predictions by generating multiple subsets of visual inputs, each with different combinations of high- -ncertnnryy tokens dropped out. These subsets are processed independently, and their corresponding outputs are aggregated using majority voting to produce the final prediction.
Our method, termed DROPOUT DECODING, enhances the reliability and accuracy of LVLM outputs without modifying the underlying model parameters or requiring additional training. Experiments on LVLM decoding benchmarks including CHAIR [33], THRONE (34), and MMBench (35) demon- strate the effectiveness of our approach. In summary, we make the following contributions. First, we introduce a novel approach that quantifies and decomposes uncertainty on tokens in the visual inputs at inference time without additional supervision, by projecting visual input tokens onto text token interpretations. Second, we propose decoding strategy that uses epistemic uncertainty mea- surements to guide the selective dropout of high-uncertaing to performing dropout on the model but applied to the input tokens and during inference. And finally, comprehensive experiments are conducted on various benchmarks, showing significant reductions in OH and improved fidelity pre-trained LVLMs without additional fine-tuning.
visual tokens in the context, analogous
Related Work
Reliable Generation. Hallucinations in LLMs where models generate irrelevant or incorrect information [66 377 [38 -aiserise
arise from data [19], training, and inference issues [39], with attention mechanisms exacerbating them [40]. To address this, factual-nucleus sampling 44 balances diversity and accuracy. While [42] guide decoding with quantified uncertainty, our approach quantifies uncertainty at the visual input level, not requiring model ensembles.
OH in LVLMs. Object hallucination (OH) is common in LVLMs, where models generate incorrect object descriptions. CHAIR (33) and POPE (43 evaluate OH, while THRONE (34 offers a more holistic approach. We use CHAIR and THRONE to assess OH in our work.
OH Reduction. Methods addressing OH in LVLMs include internal signal guidance (e.g.. OPERA (44), contrastive decoding (e.g., VCD (45)), and selective information focusing (e.g.. HALC (4) AGLA [46] mitigates hallucinations by enhancing visual grounding through global and local attention, while Memory-Space Visual Retracing [47] refines multimodal alignment via iterative visual reference retrieval. In contrast, DROPOUT DECODING 1) selects visual tokens dur- ing generation, 2) uses uncertainty for token selection without external models, and 3) employs a token-level majority voting strategy.
Preliminaries
3.1 Vision-Language Model Decoding
Widely adopted LVLM architectures 48 49
typically include a vision encoder, a vision-text interface module, and branef LLM decoder. As we mostly focus on the decoder side inference, we assume the LLM decoder parameterized by 0.
17

---

The visual input, such as an image, is segmented into patches and processed by the vision en- coder followed by the vision-text interface module, to produce a sequence of visual tokens rr" ( ........ :.)) Each token x. is contextualized embedding of an image patch, serv- ing as the direct input to the text decoder. The text input such as query or instruction is r¹ = : -1 The input to the text decoder is denoted as x [=", which is the concatenation of visual and text tokens. At this point, the visual and text tokens are aligned and serve as sequential input to the LLM decoder. During autoregressive decoding, the decoder generates out- put text tokens y = (2/1, 3/2, .) as continuation from prompt z; following the conditional probability distribution
(1) where リーク = (yi,..., 1/) is the sequence of previously generated tokens, Lo denotes the LLM forward pass to produce hidden states hy E He on top of the Transformer layers, Wy E R is the output projection matrix onto the text vocabulary V. and y V the output token j-th step.
,=, h2 = fo(x", x², ,yz)). po(ys xx, x', ,y4)) = softmax Why))
3.2 Uncertainty Quantification
Our approach quantifies the information uncertainty of visual tokens used for decoding by adapting the concept of epistemic uncertainty for measurement, as detailed in $5 and drawing inspiration from classical uncertainty decomposition [31 50 32]. To provide the necessary background, we first introduce the concept of uncertainty decomposition.
Uncertainty decomposition separates the total uncertainty of a model's prediction into two compo- nents: aleatoric uncertainty, which is inherent to the data, and epistemic uncertainty, which relates to the model's lack of knowledge. The Bayesian framework offers a principled way to quantify uncertainty about some candidate model with weights w, through the posterior estimation over the hypothesis space for a given dataset D.. The Bayesian model average (BMA) predictive distribution is defined as
plu x. D) = p(u x. w)p(w D) dw.
The total information uncertainty is measured by the entropy of BMA: Hip(y x. D)], which equals the posterior expectation of the cross-entropy between the predictive distribution of the candidate model and the BMA distribution:
Hppp x, D) Ep(e(o) [CE[p(y æ, w), ply x. D)]]
Total Uncertainty
=E Bp((ID) [Ply x, w))] + (D( (PKI(p(V) x. (.ww) ) p(y x. D))
Alcatoric Uncertainty
Epistemic Uncertainty
The epistemic uncertainty, expressed as the KL divergence between candidate models' predictive
distributions and the BMA, has proven effective in various applications 51 28 53]. Our approach,
adopts similar formulation for uncertainty quantification, calculating the KL divergence between candidate prediction distributions on individual visual tokens and an aggregated average distribution.
4 Textual Interpretation of Visual Tokens
As discussed in I identifying the visual tokens that carry significant information and quantifying their uncertainty is critical for improving the reliability of LVLMs. We propose supervision-fis free approach that maps visual tokens to text token space for improving LVLM reliability by identifying significant visual tokens and quantifying their uncertainty. This mapping leverages the LVLM's inherent ability to align visual and textual contexts.
Text-space projection of visual tokens. While LVLMs are trained to generate text only after processing all visual tokens x" and text instruction tokens z", the hidden representations h on top of the text decoder layers inherently capture textual semantics. This is due to their proximity to the text vocabulary projection, even at visual token positions where the model is not explicitly trained to generate text.
*We assume general Transformer architecture for vision encoder well. Our approach could also apply to other types of vision encoders.
"pyy x, w. D) p(y x, w) because of conditional independence.
3

---

Building on this intuition, we adopt a heuristic approach to interpret visual tokens by projecting them onto the text vocabulary at the top Transformer layers. In particular, for each visual token z at position : we obtain its textual projected distribution over the vocabulary V from the last layer of the LLM decoder in the LVLM as:
h fo(2"))) P = po(. x = ma
where h2 is the LLM decoder top-layer hidden representation aligned at the i-th visual token positions, F22i denotes the visual tokens up until index i! This approach is also generally referred to as logit lens [54] in mechanistic interpretability for LLMs.
Here, of f0 which we refer to as visual-textual distribution, represents the projection of the visual input onto the text space. It encapsulates the model's interpretation of the i-th vi- sual token. This projection offers a text-based summarization, akin an unordered cap- tion or a "bag-offwords" [I5] representation of the visual content. As we will demon- strate $6 this heuristic method serves as an effective proxy for uncertainty estimation.
An illustrative example with projection uncertainty. Figure J our projection method by process- ing an image into patches and projecting five selected patches into the text space, retrieving their top-5 text tokens. Informative patches yield specific tokens like "Berlin," "computer," or "map," which are less frequent in the vocabulary and capture unique visual contexts. In con- trast, patches producing common words (e.g., "u," "u,"""a "the," "on"") convey less specific information. This suggests that projected text tokens effectively proxy the information content of visual tokens.
Tea Magpetee Fout Takees Frees LLM Deceser
Taptop Méc
alay
stnoot pocket
bead
Leveraging this, we introduce uncertainty measures from the textual projection distributions q to quantify each visual token's 's uncertainty, as depicted in the figure. Fol-
lowing classical uncertainty quantification ($3.2), we de- compose total uncertainty into aleatoric (data-related) de- rived directly from g." and epistemic (model-related) by comparing mr to an average distribution ( (.).).. As illus- trated, epistemic uncertainty aligns well with the informa- tion content of visual tokens: high epistemic uncertainty corresponds informative patches (e.g.. "Berlin"), and vice versa (e.g., "the"). In contrast, aleatoric and total uncertainty do not show this correlation. This finding motivates focus epistemic uncertainty as a reliable indicator of the significance of visual information.
Figure I: An illustrative example where vi- sual tokens are projected into the text space, bold words indicate highly informative pro- jections, and red words mark misalignments. Dotted lines show average uncertainties; high epistemic uncertainty correlates with infor- mative patches.
5 Method
We propose DROPOUT DECODING, which leverages visual uncertainty to selectively drop out visual tokens and guide decoding. As shown in Fig, 2 and Algorithm|| [ our approach comprises two stages: uncertainty quantification (I.I) before decoding and imcertainty-guidee token generation ($5.2) for decoding.
5.1 Uncertainty Quantification Before Decoding
Average visual-textual distribution. We begin by defining the averaged distribution goo) which represents the overall projection of the entire visual input (c.g. an image) into the text space. Using the projected distribution defined in Eq. B), we define the average projection distribution over all
"Note that indexes are only used over visual tokens æ", not text tokens sf or generations y. For the models we use, the visual tokens x" are all placed before the text tokens x' in the concatenated sequence x. æx,, purely visual tokens. But approach also applies to other cases.

---

Before Decoding
Decoding Time tolen
In acdiion to the book, there & lotopp poslio roatioane to the right the ocok
Dynomially rask viaa
Eppstemic Uncertaitty (1)
Sanaaa
Uncertainty dudded Masks
Our Dropout Decoding (Noitellciintton)
Lroket Guantiarion
Ensamble prodict oo
adaition there as laptop and mouse on the desie
Prejection Dietr on
Relevant takkens
Original Generation (Halll on "mouse"
Compann distane [ ceatlag
1 83 Fiild
ecticn
Forwond with wwwee Iopur
(Ophonal!
User: Desol be the image. Aotettnt: -3 -or
Tehanized Visual Zoputs LM Deccder
Irputs Generation
Vieual Inputs
Figure 2: An overview of our DROPOUT DECODING, The method includes uncertainty measurement of visual tokens (under "Before Decoding") and uncertainty- -uudddd visual context dropout decoding algorithm (under "Decoding Time"). ). The pseudocode is in Algorithm| T
visual tokens as:
1 N Σ D
proj E.
where 2 represents the text-space projection of the i-th visual token, and N is the total number of visual tokens. Note that the subscript i indicates different distributions rather than elements within single distribution. This provides with "baseline" representation of the visual input, against which we can quantify the surprisal of a specific visual token. This idea is grounded in classical uncertainty decomposition where a Bayesian average distribution is needed to quantify epistemic uncertainty 32]
Uncertainty measurement for visual tokens. We aim to quantify the uncertainty associated with each visual token at inference time. To distinguish from those uncertainty terms in classical settings as introduced in $3.2 we refer to ours as perception uncertainty. We start by quantifying the perception total uncertainty of the visual input as the entropy of the average visual-textual distribution H [qa0o]] Then, to attribute this total uncertainty to individual visual tokens, we decompose it (details in Appendix A by:
Ustal
H m
E2
CF
Proj
Further decomposing the cross-entropy (CE), the perception total uncertainty can be expressed as: m
Urotal E,
H
+ DKL
pmo || (r
= E, aU (i) + (i)
Here we have the perception aleatoric uncertainty of the i-th visual token (ate(i) = TH o
capturing the inherent noise or ambiguity of the i-th token, and the perception epistemic uncertainty
Uep (i) DKL
(g.
|l g
quantifying the divergence between the visual token's ' textual projection and the overall projection. It indicates how much the model's belief about this token differs from its belief about the entire visual input. A higher (i) suggests that the i-th visual token conveys information that is surprising or not well-represented in the overall visual content, which can be critical for identifying tokens that might introduce uncertainty in the decoding process.
5.2 Uncertainty-Guider Decoding
During the text decoding process, we leverage the computed uncertainty measures to guide the generation of each token. Our method involves two main steps for each generated text token: (1) identifying relevant visual tokens (optional), and (2) performing loken dropout with uncertainty- guided masking. The first step optional, designed to enhance decoding by retaining more relevant visual tokens.
5

---

Identifying relevant visual tokens (uptional), We selectively retain only the most relevant visual tokens from the context, which are excluded for dropout. When generating each output text token, yj+ we first perform preliminary forward pass generate an initial prediction token with.
Vj init po(- | x=, x², ;B55)
(7)
Next, we determine the set of visual tokens that are relevant to this initial prediction. Specifically, a visual token zi is considered relevant if the initial prediction yltht appears among the top-k tokens of its visual-textual projection o."" Formally, the set of relevant visual tokens for the j-th generation is:
S 1 y) E TopK(g, (n."
(8)
where (-p denotes the function returning the top-k entries of a given distribution.
To illustrate the intuition behind this step, consider an image depicting a cat. Suppose the model correctly predicts the token "cat" during the preliminary forward pass. In that case retain the visual tokens associated with "cal" and drop out among the remaining visual content. Conversely, if the model incorrectly predicts "dog" or unrelated tokens irrelevant an object, these predictions will not align with the top text projections of any g. if the visual interpretation is accurate. In such cases, no visual tokens are retained due to a lack of clear relevance, and dropout is applied across the entire visual context as the best alternative.
It is worth noting that this step is optional. Omitting it can improve efficiency by reducing the computational overhead of the preliminary forward pass. As shown by the ablation studies in 7 while skipping this step may lower performance on certain benchmarks like THRONE 34 it still achieves comparable results on others such as CHAIR (33
Visual token dropout with uncertainty guidance. Using the epistemic uncertainty measurements (i) from Eq. (6) we introduce dropout masks over visual tokens. As illustrated in Fig, 4. the projected visual-textual distributions sometimes misalign with the image content, and regions of high information can lead to substantial errors, resulting in hallucinations. Based this intuition, selectively target visual tokens with high epistemic uncertainties for dropout.
Specifically, we formulate a controllable series of sample distributions for visual token dropout based on Uepi(? (i), for each visual position i:
Uapi (i) U epá ((k)
min
droppour (z+) = 7ik))
Ummax
U munn
where Urmin, Urmax are the minimum and maximum epistemic uncertainty values across all visual tokens, and (0) and g(k) are hyperparameters controlling the probability range of the dropout. By adjusting the values of (A) and 5kk), we can modulate the intensity of visual token dropout. For further discussion on hyperparameters, see 7..2
With the dropout distributions, we can sample dropout masks for each visual token independently. Denote the binary mask as (() E {0, 1) consisting of binary indicator M, for each visual token x.. where the conrresponding visual token is retained if M = 1, and dropped if M 0. The dropout mask sampling follows P P(M( 0) droponi (x.), and the sampling is done for each visual token position independently. A higher value of P Parupouu (x:) indicates that x. is more likely to be dropped out. If we performed the optional preliminary forward pass to identify relevant visual token set Sy. these visual tokens are never dropped, i.c.,. Yay E Sy. set M,A
(k)
directly.
Ensemble- based reliable generation. Our inference-time context dropout introduces stochasticity, so we employ ensemble decoding approach by independently sampling K distinct dropout masks, (M 1 to enhance generation quality. Since the masks are independent, the text generative distribution from K masks can be efficiently computed in a parallel forward pass
y
(k) Decoding
po 2'/artn x², (y3))
(10)
where M/) denotes the visual tokens after applying dropout mask (4)), and Decoding denotes invariance to the decoding algorithm used (e.g., greedy search in our implementation, though others applicable).

---

Algorithm Pseudocode of DROPOUT DECODING.
1: Input: visual tokens x", Text tokens æ", Number dropout masks Generation length
2: Output: Generated sequence y
3:
4: Before Decoding:
5: Obtain visual text projecting distributions pro)
(Eq
6: Compute average distribution gTT)) [Eq.
Compute epistemic uncertainty Upp (1).
8: for j = 1 to do
9: Identifying relevant visual tokens (optional):
10:
Generate preliminary token vith. (Eq.
)
11:
Get relevant tokens S, with t and o (Eq. (8)
12:
Visual token dropout with uncertainty-guuddnee:
13:
Get K dropout prob p(k) with (i) (Eq. (9)
14:
Generate dropout masks (2) based on p(o) while retain relevant tokens S,.
15:
Forward candidates y, with masks M (8), (Eq. 11
16:
Majority voting on Vj (k) and get V/--
17:
end for
Return Generated sequence y
Each U/ serves as a candidate prediction for the next text token, with the final token yj selected via majority voting among the K masked inputs. In case of a tie, we choose the prediction from the forward pass with the fewest dropped tokens, it retains the most information and is deemed more reliable. By forming an ensemble of predictions derived from various subsets of the visual input, enabled through token dropout, This diversity mitigates the impact of any single misinterpretation, ultimately leading to more reliable and robust generation, which is also observed in other ensemble-basee based methods 24 55 56
diversify the model's perspective
the visual content.
57 58
Experiments
We evaluate the proposed DROPOUT DECODING from two aspects: OH reduction and overall generation quality. For OH, we use the CHAIR 33 and THRONE 34 metrics to assess the performance of different decoding methods on the MSCOCO dataset. Additionally, employ MMBench (35 to evaluate the overall generation quality and general ability of these methods.
6.1 Experimental Setup
Base LVLMs. We evaluate all methods on three representative LVLMs: LLaVA-1.5 449 Instruct- BLIP [59] and LLa VA-NETT [16]. VL- and LLaVA-NEXT use hundreds to thousands of visual tokens for detailed representation, while InstructBLIP employs just tokens but with higher information density. This showcases the flexibility of our approach, effective across models with varying token counts.
Hallucination reduction baselines. addition to the original LVLM outputs, we compare our method with beam search well state-of-the-art decoding methods: VCD 455 which
contrasts original and distorted visuals to reduce hallucinations, and OPERA penalties and token adjustments for better grounding.
44 which applies
6.2 CHAIR
CHAIR [33 is a benchmark for evaluating object hallucination in image captioning. It includes two metrics: the sentence-level CHAIRs, measuring the frequency captions with hallucinated objects, and the object-level CHAIR, calculating the proportion of hallucinated objects among all objects.
Results. As shown in Table I DROPOUT DECODING consistently outperforms baseline approaches across various models, demonstrating its reliability and effectiveness in image captioning. Especially on InstructBLIP, CHAIR, and CHAIR improve by 16% and 12% respectively over the second-best method. Furthermore, DROPOUT DECODING reduces the generation of hallucinated objects without compromising the inclusion of relevant objects. These improvements align with expectations that token dropout reduces generated objects.
7

---

Model
Method
CHAIR
THRONE
CHAIRS L CHAIR;
Fai
Fis
Part
Rurt
Greedy
42.20=2.
12.83
0.795
0.784
0.772am01s
0.817
Beam Search
46.33
13,9,0.
0.750
0.772am.04
0.75%,n.03
1..62
LLAVA-1.5
OPERA
4147ca.2
49.20
12.37.0.22
14.872047
0.78640.012
28002...000
0.782:u000
8.759
0.854
0.854
DROPOUT DECODING
39 39.00
11.782002
0,804
0.796.a.00
0.790
0.851
DROPOUT DECODING (wis prelimm)
39
12,20.0
0,759
0.794
0.791
0.843
Greedy
27.87
10000
0.826...00
0.832.0000
Ream
25.87=2m7
6.9
48002
0.836-20005
20.00
InstruciBLIP
OPERA
VCD
28.07 28.07
39.33
0.737 19
8,23
0.737
0.805
40
0.824g0000
0.746ga.00
0.751
0.83020004
0.757
0.757
0,798
0.798
DROMUU DECOBING
24.53
6,63
0.814
0,833_.00
0,838
0,508 10.036
DROPOUT DECODING prelim)
26.2g2.00 0
7.10
40.0e
g.823
0.82720010
2.804
Greedy
28.80
8.10
0.81520.012
0.832a.0nn
0.830
0.79920.c0
Beam Search
28,06
1.10
0.816
0.834ga.00
0.834.0.031
0,501
LLVAA-EEX
OPERA
VCD
29.06
33.19
8.10
8.06
1..14
1.88
0.832
a.822
0.831
0.808..0.00
0.799 0.799.0.cmm
.o.c00
DROPOUT DECODING
26.26:2.4
7.39
0.82120.210
0.840=5.0m
0.842=n0022
s0.010
DROPOUT DECODING (wlo prelim)
_7.0_1..0
7.53.c.60
1.814
0.835
0.837 0.837
0.793
Table I: Comparison of methods CHAIRs, CHAIR,, F 1.5, IIstructBLIIP, and VA-VAA- Details of the experimental setup and the interpretation of the standard deviation can be found in the appendix. Details of the experimental setup and the interpretation of the standard deviation can be found in the appendix.
F Pan, and Ran metrics for LL-
(c) CHAIR
(b) CHAIR,
(c) Pan
(3) Ran
Figure 3: Comparison of CHAIRs, CHAIR,, Pau and Ran scores with standard deviations across
different candidate numbers.
6.3 THRONE
THRONE (34 assesses hallucinations in LVL--generated responses, covering both "Type I' (men- tions of non-existent objects, like CHAIR) and "Type II" (accuracy of object existence, like POPE 4 It uses Pan (Precision), Ran (Recall), Fl and F0.5, Additionally, it employs Fa. which combines Pan and Ranl with the parameter 3 controlling the weight of Ran relative to Pan: F& (1 B²)
Pa Pu)+
The test results in Table Iillustrate that DROPOUT DECODING surpasses nearly all baseline methods across various metries, highlighting its effectiveness in reducing both Type I and Type hallucinations. Specifically, DROPOUT DECODING demonstrates notable strengths in InstructBLIP, excelling in the Pai metric and achieving the highest performance in Ran- Across models, Pau metric achieves larger improvement while the Rai score also exceeds that of the Greedy method, confirming that retaining overlap tokens effectively preserves relevant objects. The large increase in F.05 further shows its comprehensiveness.
Results.
6.4 MMBench
MMBench (35 is a comprehensive benchmark designed to evaluate the multimodal capabilities of LVLMs across various tasks and data types. Since the prompt length limits in MMBench ceed InstructBLIP's token allowance, we report results only on LLaVA-1.5 and LLAVA-NEXT.
Mehhol
Original VCD OPERA DROPOUT DECODING
Results. As shown Table DROPOUT DECODING out-
LLaVA-15
72.35 73,86
74.01
performs all the other baselines on LLLVAA-155, which demonstrates its robustness and adaptability across a broader range of multimodal tasks.
LLaVA NEXT 74.57 69.65 74,54
74.31
Table
Results on MMBench. Higher is better.
7 Analysis and Ablation Studies
7.1
Efficiency Analysis
We conducted thorough analysis computational overhead, measuring throughput and wall-time to evaluate efficiency. Table 7.1 summarizes these results. Our method introduces additional overhead
8

---

primarily in two aspects: (1) preliminary forward pass for identifying relevant visual tokens, (2) performing K parallel forward passes using varied dropout masks.
The preliminary forward pass, though beneficial, is optional. Omitting it results in only approximately 7% throughput reduction compared to greedy decoding, while still consistently improving perfor- mance metrics across benchmarks. Furthermore, the method efficiently handles the K parallel passes by batching identical inputs with distinct dropout masks into a single batched operation, significantly reducing additional computational overhead.
In terms of GPU memory, verified efficiency under realistic conditions. Using vLLM and LL. on 4xA800 80GB GPUs, GPU memory usage was 38. 12 GB with efficient KV caching, unchanged between greedy decoding and our method without preliminary passes. Confirmatory experiments under Huggingface Transformers similarly demonstrated minimal GPU memory increase (from 14.02 GB to 15.31 GB), indicating negligible impact on inference constraints.
The cost of computing uncertainty metrics is explicitly included in our benchmarks and remains negligible. With LLa VA-1.5 and 576 image tokens, computing uncertainty adds only 73.30 ms per input, a minor cost amortized across the batched forward passes.
Overall, our approach effectively balances efficiency and performance.
Metric
Greedy
Ours w/o prelim
Ours w/ prelim
Throughput (tok/s) f
37.0
34.1 (-7.8%
20.1 (-5.77%)
Wall-time (per 50 tok) +
1.35
1.47 (+8.9%)
2.49 (+84.4%)
Table 3:
Computational overhead analysis.
7.2 Parallel Dropouts Hyperparameters
As in $5.2 we generate K candidate predictions using token dropout. This section examines how varying the hyperparameters impacts generation quality. We fix 866) = 0.1 and adjust 7(6)) based on predefined order: (1) = 0.3. 7 0.5, and 79( = 0.7. However, setting (1) = 0.9 excessively
drops visual tokens and degrades InstructBLIP's performance, so we set (,(4 = 0.1. Moreover, our majority voting favors candidates with fewer dropped tokens ties. To avoid identical outputs when comparing only two candidates, we remove Candidate 1 in the second round.
As shown in Fig. (a) and (b), both CHAIRs and CHAIR, scores peak K = for LLaVA- and InstructBLIP. Increasing K to 4 introduces a masked candidate that slightly negatively impact our method's ss effectiveness in reducing hallucinations. Conversely, using fewer candidates (e.g.. only Candidate 1 1/2) lacks the balance needed for stable voting outcomes, resulting in increased randomness. Similarly, Fig. (3 and (d) shows that THRONE'S Rall and Pau metrics also perform best K 3. Overall, we find that selecting three candidates strikes the optimal balance between increased certainty from additional votes and the controlled uncertainty introduced by candidate dropout probability, allowing DROPOUT DECODING to achieve more trustworthy and stable generation results.
7.3 Preliminary Forward Pass
As discussed in $5.2 DROPOUT DECODING may employ a preliminary forward pass to retain most relevant objects during generation, which helps reduce hallucinated objects while maintaining high- quality outputs. contrast, bypassing this step risks masking relevant visual tokens during the token dropout phase, potentially degrading overall performance. However, incorporating preliminary forward pass roughly doubles the computational cost per generation. Specifically, our goals are: 1) to confirm the effectiveness of the preliminary forward pass, and 2) to explore more efficient alternative when computational resources are limited.
As shown in Table I including the preliminary forward pass consistently improves most metrics, with particular notable gains in the Fan score on THRONE. Interestingly, for LLLVA-1.5 and LLa NEXT, the variant without the preliminary pass still outperforms other baselines in most metrics. We hypothesize that this discrepancy arises from differences the abundance of visual tokens, as LLaVA-1.5 5.5 and I.L.V--NEX" have hundreds or thousands of visual tokens. While InstructBIIIP only has 32, making each token's 's contribution more critical. Consequently, omitting the preliminary forward pass in InstructBLIP risks losing critical information, lowering performance. These findings

---

suggest that while a preliminary forward pass is highly beneficial for LVLMs, models with more tokens may achieve better efficiency and performance by skipping this step.
7.4 Necessity of Uncertainty Guidance on Masking
As discussed in $5.1 DROPOUT DECODING incorporates epistemic uncertainty in the masking process. To validate the necessity of this approach, we compare it with a random masking strategy, which replaces uncertainty with random method. As shown in table Table 4 although random masking performs better on the CHAIR metric, it struggles with BLEU and fails to compute the THRONE metric. We find that random masking causes repetitive token generation (e.g., "apple apple upple.."). artificially inflating the CHAIR score. This happens because random masking disrupts contextual information, leading to faulty generation. In contrast, our uncertainty based approach selectively masks uncertain tokens, preserving the context and ensuring more coherent and accurate sequences.
CHAIR
BLEU
THRONE
Model
Method
CHAIRs
CHAIR,
BLEU f
THRONE
Greedy
42.20
7.90
11.62
Table
LLaVA-1.5
Uncertuinty-guiie
39. 80_80.2..
11.73+0.26
11.6440.12
Table
Random
35.93
8.57_00.5
+1.11
Emor
Greedy
27.87
12.70 7.90_o.as
Table
InstructBI.IP
Uncertainty guided
24.53
6.63_2.66
12.30
Table
Random
23.80
5.07
10.88
Error
Table 4: Comparison of masking strategics on CHAIRs, CHAIR,, BLEU and THRÖNE metrics for LLa InstructBLIP.
7.5
High-Confidence Token Masking Analysis
To further investigate the robustness of our uncertainty-guided masking, we conducted additional ablation experiment focusing on the opposite mondition-masking high-confidence tokens instead of low-confidence ones.
Following the identical experimental setup as in Table (our main CHAIR and THRONE evaluation), we replaced low-confidence masking with high-confidence masking. The results summarized in
Table 7.5
Model
CHAIR J
CHAIR,
Fa 7 Fo.s.an
Pai
Rall f
LLaVA
41.62
12.00
0.798
0.784
0.774
0.858
LLaVA-NEXT
27.61
0.803
0.823
0.828
0.789
InstructBLIIP
29.50
9.01
0.808
0.825
0.825
0.794
Table Effect of masking high-confidence tokens on CHAIR and THRONE metrics.
As shown, the results are generally worse than masking low-confidence tokens (i.e., masking high- uncertainty tokens as in our proposed method) and remain close to the greedy decoding baseline. This suggests that dropping high-confidence tokens has limited influence on generation quality-these tokens correspond to regions where the model is already certain, typically associated with background or redundant patches. Consequently, their removal produces only minor perturbations.
In contrast, masking low-confidence tokens directly influences the model's 's generative process, as these tokens are uncertain yet potentially informative (often corresponding to salient or ambiguous visual regions). Masking them introduces meaningful variability, thereby improving robustness and reducing hallucination frequency. This further validates our uncertainty-guicn guided masking strategy as both effective and theoretically grounded.
8 Conclusion
We introduce DROPOUT DECODING, a novel inceertaitt-guuddeee context selective decoding approach aimed at enhancing the reliability of LVLMs. After quantifying the uncertainty in visual inputs, DROPOUT DECODING accordingly drops out visual tokens to regularize uncertainty and employs an ensemble- decoding approach to stabilize generation. Extensive experiments on CHAIR, THRONE, and MMBench validate the effectiveness with consistent improvements over existing methods in both hallucination reduction and general multimodal capability.
10

---

References
[1] Yifan Du, Zikang Liu, Junyi and Wayne Xin Zhao. A survey vision-language pre- pre-trained models. arXiv preprint arXiv:2202. 10936, 2022.
[2] Wanhua Li, Renping Zhou, Jiawei Zhou, Yingwei Song, Johannes Herter, Minghan Qin, Gao Huang. and Hanspeter Plister. 4d langsplat: 4d language gaussian splatting via multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22001-22011, 2025.
[3] Zhuokai Zhao, Harish Palani, Tianyi Liu, Lena Evans, and Ruth Toner. Multimodal guidance network for missing-modality inference in content moderation. In 2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pages 1 1-4, IEEE, 2024.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[5] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vision-language foundation models reason compositionally? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10910-10921, 2023.
[6] Devaansh Gupta, Siddhant Kharbanda, Jiawei Zhou, Wanhua Li. Hanspeter Pfister, and Donglai Wei. Cliptrans: transferring visual knowledge with pre-trained models for multimodal machine translation. In Proceedings of the IEEF/CVF international conference on computer vision, pages 2875-2886, 2023.
[7] Wanhua Li. Zibin Meng, Jiawei Zhou, Donglai Wei, Chuang Gan, and Hanspeter Pfister. Socialgpt: Prompting Ilms for social relation reasoning via greedy segment optimization. Advances in Neural Information Processing Systems, 37:2267-222 1191722291,220 2024.
[8] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang. Liping Hou, Rongjun Li, and Wei Peng. A survey on hallucination in large vision-language models. arXiv preprint arXiv: 2024.
[9] Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. arXiv preprint (tti:2300000334 2023.
[10] Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li. Halle-switch: Controlling object hallucination large vision language models, arXiv e-prints, pages arXiv-2310, 2023.
[11] Yanhong Li, Zixuan Lan, and Jiawei Zhou. Text pixels? evaluating efficiency and understanding of Iims with visual text inputs. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 2025. Association for Computational Linguistics.
[12] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localization. Interna- tional journal of computer vision, 128:336-358 2020.
[13] Yixiong Fang, Tianran Sun, Yuling Shi, and Xiaodong Gu. Attentionrag: Attention-guided context pruning in augmente generation, 2025.
[14] Seil Kang, Jinycong Kim, Junhyeok Kim, and Seong Jac Hwang, See what you are told: Visual attention sink in large multimodal models. In The Thirteenth International Conference Learning Representations, 2025.
[15] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why languae models behave like bags-of ofodss and what to do about it?, 2023.
[16] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oer, and world knowledge. January 2024.
[17] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselines with visual instruction tuning, 2024.
[18] Weiyun Wang. Zhe Chen, Wenhai Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language 2024.
models via mixed preference optimization. arXiv preprint 1.
[19] Mingyang Song. Xiaoye Qu, Jiawei Zhou, and Yu Cheng. From head to tail: Towards balanced representa- tion in large vision- vision-language models through adaptive data calibration. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages
2025.
11

---

[20] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang. and Yinfei Yang. Ferret: Refer and ground anything any anyeeere any granularity. arXiv preprint urXiv:2310007044 2023.
[21] An-Chich Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang. Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision language model. arXiv preprint arXiv:2406.0158: 2024.
[22] Kelvin Xu, Show, attend and tell: Neural image caption generation with visual attention. arXiv preprint 2015,
[23] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra, Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[24] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Hale: Object hallucination reduction via adaptive focal-contrast decoding. In Forty-first International Conference on Machine Learning.
[25] Jianyi Zhang, Da- Da-Cheng Juan, Cyrus Rashtchian, Chun-Sung Fern, Heinrich Jiang. and Yiran Chen. Sled: Self logits evolution decoding for improving factuality in large language models. arXiv preprint arXiv:241 1.004333 2024.
[26] Tanqiu Jiang. Jiacheng Liang, Rongyi Zhu. Jiawei Zhou, Fenglong Ma. and Ting Wang. Robustifying vision-language models via dynamic token reweighting. arXiv preprint 2505:2505..7133 2025.
[27] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky. Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-15) 1958, 2014.
[28] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA, 20-22 22 Jun 2016. PMLR.
[29] Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
[30] Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of general- ization. Advances in neural information processing systems, 33:4697-4708, 2020.
[31] Eyke Hüllermeier and Willem Wacgeman. Alcatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning, 11003)457-506, 2021.
[32] Kajetan Schweighofer, Lukas Aichberger, Mykyta lelanskyi. and Sepp Hochreiter. Introducing an improved information-thcorete measure of predictive uncertainty. arXiv preprint 0rXiv:23/L.08300 2023.
[33] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning, 2019.
[34] Prannay Kaul, Zhizhong I.i, Hao Yang, Yonatan Dukler, Ashwin Swaminathan, C. J. Taylor, and Stefano Soatto. Throne: An object-based hallucination benchmark for the free-form generations of large vision- language models, 2024.
[35] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024.
[36] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles. taxonomy, challenges. and open questions, 2023.
[37] Zhuokai Zhao. Enhanced data utilization for efficient and trustworthy deep learning. 2024.
[38] Chaoqi Wang. Zhuokai Zhao, Chen Zhu, Karthik Abinav Sankararaman, Michal Valko, Xuefei Cao, Zhaorun Chen, Madian Khabsa, Yuxin Chen, Hao Ma, et al. Preference optimization with multi-sample comparisons. arxiv preprint 240 2024.
12

---

[39] Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli, Hallucination is inevitable: An innate limitation of large language models, 2024.
[40] David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention. In Smaranda Muresan, Preslav Nakow, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 7654-7664, Dublin, Ireland, May 2022, Association for Computational Linguisties.
[41] Nayeon Lee, Wei Ping. Peng Xu. Mostofa Patwary, Pascale Fung. Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation, 2023.
[42] Esteban Garces Arias, Julian Rodemann, Meimingwei Li, Christian Heumann, and Matthias ABenmacher. Adaptive contrastive search: Uncertainty-guider decoding for open-ended text generation. arXiv preprint arXiv:2407. 18698, 2024.
[43] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large laon-laa models, 2023.
[44] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He. Jiaqi Wang, Dahua Lin. Weiming Zhang. and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation 2024.
[45] Sicong Leng, Hang Zhang. Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding, 2023.
[46] Wenbin An, Feng Tian, Sicong Leng, Jiahao Nic, Haonan Lin, QianYing Wang, Guang Dai, Ping Chen, and Shijian Lu. Agla: Mitigating object hallucinations in large vision-language models with assembly of global and local attention, 2024.
[47] Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng. Junkai Chen, Chang Tang. and Xuming Hu. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models, 2024.
[48] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022.
[49] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.
[50] Kajetan Schweighofer, Lukas Aichberger, Mykyta lelanskyi, Günter Klambauer, and Sepp Hochreiter. Quantification of uncertainty with adversarial models. Advances in Neural Information Processing Systems, 36: 19444199444 2023.
[51] Ian Oshand, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn, 2016.
[52] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation, 2018
[53] Zhuokai Zhao, Yibo Jiang, and Yuxin Chen. Direct acquisition optimization for low-budget active lcarning. arXiv preprint arXiv:2402.06045, 2024.
[54] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi. Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens, 2023.
[55] Lior Rokach. Ensemble-based classifiers. Artificial intelligence review, 33:1-39, 2010.
[56] Mudasir A Ganaic, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai N Sugan- than. Ensemble deep learning: A review, Engineering Applications of Artificial Intelligence, 115:105151, 2022.
[57] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
[58] Stanislav Fort, Huiyi Hu, and Balaji akksminnraaynaan Deep ensembles: A loss landscape perspective. arXiv preprint arxiv: 1922.0277, 2019.
[59] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language language models with instruction tuning. 2023.
13

---

A Details of Uncertainty Decomposition
detailed derivation Eq. (5)
Uwial
TH m
Σ q (3)) log gani
VEV
B2 1. ((o0o(())) log (a)oe))
NEV
E,
B. (1) log of (eo
E,
CE pmij .prmm)))
(11)
E2 THT proo + DKL ( (1,00) || (000))
E, [Uate (i) + Uppi(i)]
B Implementation Details
Our experiment is conducted on the MSCOCO 2014 test set, where we randomly sample 500 images across random seeds. The average and standard deviation result table. The prompt used for the images is "Describe the image."
these seeds are reported in our
The experimental setup of DROPOUT DECODING is shown in Table 6 We set the maximum tokens to 512 to ensure the complete generation of models, therefore achieving more reliable results from CHAIR and THRONE. In MMBench, all questions single-chiiee questions, set the maximum new tokens to for a more precise evaluation. We set other parameters in generation to greedy for more stable and repeatable results.
Parameters
CHAIR
THRONE
MMBench
512
512
256
Top-k
False
Top-p
Temperature
Number Beams
Table 6: Parameter settings used in our experiments.
In addition to general generation settings, DROPOUT DECODING includes hyperparameters specified in 5.2 The details of these hyperparameter settings are provided below:
Top-k in identifying relevant visual tokens. Before the decoding process, we first obtain g which is then used in the decoding process for generating the relevant visual tokens. The higher the top-k is, the more visual tokens are expected to be kept during the decoding process. In LLaVA-1.5, set k InstructBLIP derives from the informative level of each visual token, where in LLaVA-1 each visual token carries less information than in InstructBI.IP, which only contains 32 visual tokens.
and in InstructBLIP,
set & 10. The difference of & between LLaVA-115 and
Number of mask K. refers to the number of predictions that will join the majority vote progress. We set K = 3 in our experiment settings.
(k) and (Ak) in uncertainty-guidee masking We set ((8) 0.1, = 0.2 * + 0.1;k
1,2, K; K 3 in our experiment settings.
Moreover, we provide the hyperparameter settings of our baselines. OPERA's hyperparameters can be referred to Table 7 VCD's hyperparameters can be referred to Table g

---

Parameters
Value
Self-attention Weights Scale Factor 0
50
Attending Retrospection Threshold
15
Beam Size
3
Penalty Weights
Table 7: OPERA hyperparameter settings.
Parameters
Value
Amplification Factor a
Adaptive Plausibility Threshold
0.1
Diffusion Noise Step
500
Table 8: VCD hyperparameter settings.
C Details of Masked Tokens
We measured the average masked visual tokens of the selected candidate in decoding, demonstrating the effectiveness our method. For reference, the total number of visual tokens varies across models: LLaVA uses 576 visual tokens. InstructBI.IP processes 32 visual tokens, while L.L.VA-EXT typically generates around 2,000 tokens depending on the resolution and patching strategy.
LLaVA
LLaVA-NEXT
InstructBLII
Masked Tokens
244.14
879.53
13.67
Table 9: Average number of masked tokens in the selected candidate
Impact Statement
This paper aims to advance the field of Machine Learning. While our work has potential societal implications, we do not identify any immediate concerns that require specifie attention.
Limitations
While DROPOUT DECODING improves reliability by suppressing uncertain visual tokens during inference, our method has several limitations. First, the additional forward pass in decoding, which introduces computational overhead. Second, the effectiveness of epistemic uncertainty estimation relies on the quality of text-space projections, which may influenced by the quality of projector.

---

D Generation Examples and Analysis
D.1 LLaVA-1.5 Examples
Generated Caption:
Greedy Generated caption:
The image features a black and white photograph of a small boat with three people rowing it down river. The boat is be- ing pulled by the people, and there are two swans swimming alongside the boat. The swans are swimming in the water near the boat, creating a peaceful and serene scene.
DROPOUT DECODING Generated caption:
The image features group of people in a small boat, pad- dling down a river. There are at least four people visible in the boat, with some of them sitting closer to the front and others towards the back. The boat is filled with people, and they seem to be enjoying their time together the water.
Generated Caption:
Greedy Generated caption:
The image features a kitchen countertop with a microwave oven and large, wrapped object sitting top of it. The object is wrapped in aluminum foil, which is covering large rectangular box container. The microwave oven is positioned on the left side of the countertop, and the wrapped object is placed on the right side. The countertop is also adorned with a few utensils, such as spoon and fork.
DROPOUT DECODING Generated caption:
The image features kitchen counter with metal pan or tray sitting on top of it. The pan is covered in aluminum foil, which is commonly used to keep food warm to prevent it from sticking to the pan. The kitchen counter appears to part of larger countertop, and the scene suggests that the pan is being prepared for cooking or baking.
Generated Caption:
Greedy Generated caption:
The image features dining table set with white plate con- taining variety of food items, including meat, vegetables, and bread. The plate is placed on a wooden table, and there glass wine next to The table set for a meal, with fork and knife placed on the table. The food items on the plate are arranged in visually appealing way.
DROPOUT DECODING Generated caption:
The image features dining table with a white plate contain- ing delicious meal. The plate is topped with a variety of food items, including piece of meat, roll, and a side of vegetables. There are also couple carrots on the plate. In addition to the plate, there is wine glass placed on the table, suggesting that the meal accompanied by glass of wine. A chair is positioned near the table.

---

D.2 InstructBLIP Examples
Generated Caption:
Greedy Generated caption:
The image features a table with laptop and a plate of food, including bowl of salad and plate of food. The food is arranged on the table, with fork and a knife nearby. The laptop is placed on the table, likely used for work or en- tertainment. The table is set up for a comfortable dining experience, with variety of food options available.
DROPOUT DECODING Generated caption:
The image features plate of food on the table aestheti- cally pleasing and well-balanced. There is plate of food, including a salad, and bowl of chicken, and a fork. The plate is placed the table, which is also adorned with laptop and a computer. The laptop is placed on the table, and the computer is positioned to the left of the plate.
Generated Caption:
Greedy Generated caption:
The image features red fire hydrant placed in front of stone wall. The hydrant positioned the middle of the scene, with a window above it. There are two benches placed near the fire hydrant, one on the left and another on the right side of the image. The benches are positioned close to the fire hydrant, with one closer to the window and the other further away.
DROPOUT DECODING Generated caption:
The image features red fire hydrant is placed in front of stone building, with window above it. The window is located on the second floor of the building, and the fire hydrant is positioned in front of the window. There are two metal bars on either side of the fire hydrant, which are likely used secure the hydrant in place.
Generated Caption:
Greedy Generated caption:
The image features dining table with white plate topped with a slice of pie, a spoon, and cherry on top. The pie is surrounded by various utensils, including fork, knife, and spoon. There are also two cups on the table, one on the left and one on the right, each containing a beverage. The dining table is set with a variety of dishes, including bowl, cup, and a spoon. The scene is set for delicious meal, with the pie and beverages adding to the overall ambiance.
DROPOUT DECODING Generated caption:
The image features a dining table is set with variety of dishes, including plate with slice of pie, a cherry, and spoon. There are also several cups of coffee and bowl of soup. The table is set for meal, and the dishes arranged in way that suggests person is about to enjoy their food.

---

D.3 LLaVA-NEXT Examples
Generated Caption:
Greedy Generated caption:
In the heart of a bustling pizza shop, a woman stands, her attention captivated by the slice of pizza she holds in her hand. The pizza, with its golden crust and vibrant toppings, tantalizing sight. She's dressed casually, her green jacket contrasting with the warm tones of the shop.Thu The shop itself feast for the eyes. A large refrigerator stands in the back- ground, its surface adorned with various signs and posters. The words "We guarantee the best pizza" are prominently displayed, testament to the shop's commitment to quality.
DROPOUT DECODING Generated caption:
In the heart of a bustling pizza shop, a woman stands, her attention captivated by the slice of pizza she holds in her hand. The pizza, with its golden crust and vibrant toppings, tantalizing sight. She's dressed casually, her green jacket contrasting with the warm tones of the shop.Thu The shop itself feast for the eyes. Boxes of pizza are neatly stacked on shelves, their white and red colors standing out against the darker tones of the shop. sign on the wall proudly declares "We guarantee the best pizza in town", testament to the shop's 's confidence in its offerings.......
Generated Caption:
Greedy Generated caption:
The image depicts cozy indoor setting with a focus wooden table. The table set with a few items, including plate of food, glass of water, and small bowl. There are also couple of utensils, a fork and knife, placed on the table. The table is positioned in front of large window that offers view of patio area with a bamboo fence and a small table. The room has warm and inviting atmosphere, with wooden floor and a beige cushion visible in the foreground. The lighting suggests might be daytime. There are visible texts or distinctive brands in the image. The style of the image a candid, real-life photograph with focus the table setting and the view outside the window.
DROPOUT DECODING Generated caption:
The image depicts cozy indoor setting with focus wooden table. The table set with a few items, including a plate with small portion of food, glass of water, and a pair of chopsticks. There are also couple of small bowls and a spoon the table. The table is positioned in front of large window that offers a view of wooden deck outside. The deck is furnished with bench and small table, and there are bamboo plants and a tree visible in the background. The room has warm and inviting atmosphere, with wooden floor and a cushion on the floor to the left of the table. The lighting suggests might be daytime, and the overall style of the image is natural and serene.
5

---

D.4 Analysis
In examples, red denote hallucinations. Specifically, DROPOUT DECODING addresses hallucinations in two key ways: eliminating non-existent objects and correcting the generation of misidentified objects. For instance, in the third example of LLaVA-1.5, the Greedy method hallucinates a non-existent object, a knife, whereas DROPOUT DECODING effectively removes this hallucination. Similarly, the second example of InSIructBLIP,, the Greedy method misidentifies metal bars as benches, but DROPOUT DECODING successfully corrects this, accurately recognizing the metal bars

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims in the abstract and introduction accurately reflect the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are listed in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification:
We provide the full set of assumptions and a complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We fully disclose all the information reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
8

---

Answer: [Yes]
Justification:
We provide open access to the data and code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeeee 7/nips.com htte://lllllliiiiee////eeeuaasss ssiniiiiccypy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We include no training in our work, and we provide all test details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the error bars in the experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the efficiency analysis for time of execution.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaarrrsccccoooaaa//aaaaaa
Question:
Answer: [Yes]
Justification:
We follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work has societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
10

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper for each asset.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work doesn't It release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
12

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Our method development does not involve LLMs any important, original, or non-standard components.
Justification:
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
13


Reviews:
[{"review_id": "yumxcjcAFl", "reviewer": "Reviewer_TrMw", "summary": "This paper introduces Dropout Decoding, an inference-time strategy to reduce hallucinations in large vision-language models (LVLMs) by masking uncertain visual tokens. Uncertainty is quantified by projecting visual tokens into the text token space via the decoder and decomposing the resulting distribution into aleatoric and epistemic components. The method focuses on epistemic uncertainty to guide selective dropout of visual tokens during decoding. Rather than modifying the model or retraining, the approach samples multiple dropout masks (applied to input tokens) and aggregates the resulting outputs. The method shows strong performance improvements on CHAIR and THRONE benchmarks, with limited gains on MMBench. The core contribution lies in treating uncertainty not just as a signal to directly determine important visual tokens. The method is model-agnostic, lightweight, and requires no additional supervision or fine-tuning.", "questions": "1. What happens if you ensemble over masks that ablate low uncertainty tokens?\n\n2. How does performance vary with mask sparsity?\n\n3. Why exactly can’t the THRONE metric be used with random dropout?\n\n4. Could a simpler uncertainty metric (e.g., entropy or low LM confidence) perform comparably?", "limitations": "There is no dedicated limitations section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "$\\textbf{Strengths}$\n\n- Dropout Decoding is effective and efficient, requiring no model fine-tuning or retraining.  Furthermore, ensembling over dropout masks is performed in a single parallel forward pass, which avoids the costly repetition of model runs. Dropout Decoding provides moderate gains on CHAIR and THRONE with mixed results on MMBench. \n\n- The paper carefully distinguishes epistemic from aleatoric uncertainty and shows empirically that epistemic uncertainty better captures misaligned or misleading tokens (e.g., visual patches labeled “Berlin”). The authors measure epistemic uncertainty via the KL divergence between a token's text-space projection and the average distribution, effectively surfacing perceptually informative outliers. This was well explained and is grounded nicely in theory.\n\n- The two-step method with a potential preliminary forward pass is cleanly modular. The preliminary forward pass typically improves performance, but it is not absolutely needed in the method. This paper provides nice baselines with greedy and random dropout schemes. \n\n$\\textbf{Weaknesses}$\n\n- It would be valuable to see what happens when high-confidence tokens are dropped. Does the method still work? Do the regions that are preserved correspond to attention outliers, which have been shown to play an important role in VLMs (Golovanevsky et al. 2025 https://arxiv.org/abs/2406.16320)\n\n- The epistemic uncertainty measure, based on KL divergence from a global average distribution, is complex. A simpler baseline besides greedy or random is not tested (e.g., softmax entropy or low token confidence).\n\n- How sparse must the dropout mask be for the method to remain effective? No sensitivity analysis is provided.\n\n- The method underperforms or is inconsistent on MMBench, suggesting it may be less effective for general multimodal reasoning tasks.\n\n- In Section 7.4, the paper states the THRONE metric cannot be used with random dropout, but it’s unclear why—this undermines some comparisons.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the strengths of our work, particularly the effectiveness and efficiency of Dropout Decoding as a lightweight, model-agnostic inference-time strategy that leverages epistemic uncertainty to improve hallucination mitigation without requiring any model training.\n\nThe concerns proposed by the reviewer are very valuable, and we address them below.\n\n\n### Q1: What happens when high-confidence tokens are dropped? Does the method still work, and do preserved regions correspond to attention outliers?\n\nWe conducted experiments to specifically address this question. In this experiment, we follow the identical experiment setting in Table 1 (the main experiment result table with CHAIR and THRONE results), just changing the low-confidence into high-confidence. Results are shown below:\n\n\n| Mask high-confidence           | CHAIR_s | CHAIR_i | F     | F0.5  | P     | R     |\n|----------------|---------|---------|-------|-------|-------|-------|\n| LLaVA          | 41.62   | 12.00   | 0.798 | 0.784 | 0.774 | 0.858 |\n| LLaVA-NEXT     | 27.61   | 7.82    | 0.803 | 0.823 | 0.828 | 0.789 |\n| InstructBLIP   | 29.50   | 9.01    | 0.808 | 0.825 | 0.825 | 0.794 |\n\nAs we can see, the results are worse than dropping the low-confidence tokens(i.e., dropping high uncertainty tokens as in our method), and quite close to the greedy decoding baseline. We believe that masking the high-confidence tokens are making small effects, as model has already been confident about these tokens, indicating these tokens are actually “unimportant” during generation, like background patches. In contrast, masking low-confidence tokens is more likely to influence the generation process because the model inherently considers these tokens uncertain yet potentially informative(like object patches). Therefore, masking them tends to alter the original generation outcomes. \n\n### Could a simpler uncertainty metric (e.g., entropy) perform comparably?\n\nWe have also implemented an entropy-based baseline for comparison, similarly, we follow the identical setting in Table 1, the results are shown below:\n\n\n| Entropy Mask   | CHAIR_s | CHAIR_i | F     | F0.5  | P     | R     |\n|----------------|---------|---------|-------|-------|-------|-------|\n| LLaVA          | 42.21   | 11.12   | 0.800 | 0.799 | 0.805 | 0.845 |\n| LLaVA-NEXT     | 27.23   | 7.52    | 0.792 | 0.818 | 0.836 | 0.778 |\n| InstructBLIP   | 28.04   | 8.00    | 0.797 | 0.819 | 0.825 | 0.784 |\n\nAs observed, entropy-based masking consistently underperforms uncertainty-guided Dropout Decoding method. Across all three models, CHAIR_s increases (worsens) by up to +3.5, the F1 score decreases by around 0.03, and all other metrics (CHAIR_i, F0.5, P, R) similarly decline. This further validates the effectiveness of our proposed approach of utilizing uncertainty to mask tokens. We will include these results in our paper with more discussion. \n\n### Q3: Method inconsistency on MMBench?\n\nThe inconsistency observed in MMBench primarily arises because MMBench mainly consists of multiple-choice tasks, which typically involve single-token outputs, similar to the POPE benchmark. Our method is more suited to open-ended generation tasks. Indeed, our evaluation already covers both close-ended and open-ended benchmarks, such as THRONE, providing a comprehensive assessment of our method's applicability.\n\n### Q4: Sensitivity analysis regarding dropout mask sparsity?\n\nWe acknowledge the importance of mask sparsity sensitivity. In our default hyperparameter setup, we collect the generation trajectory of LLaVA-1.5,  running on MSCOCO 2014 train dataset (we sampled 500 images), around 45.6% of tokens are masked during each generation step. Further sensitivity analysis across different sparsity levels could indeed be insightful, and we will explore this in future work.\n\n### Q5: Clarification on THRONE metric incompatibility with random dropout (Section 7.4)?\n\nFrom our experiments, random dropout often leads the model to repetitively generate identical tokens (e.g., continuously outputting \"apple\") until reaching the maximum token limit (512 tokens). This scenario produces nonsensical results and computational issues (e.g., out-of-memory errors even with substantial resources like 4 × A100 GPUs), thus making the THRONE evaluation impractical and unreliable in this context.\n\nThank you again for these insightful comments, which significantly enhance the clarity and depth of our analysis."}, {"review_id": "RIFUllTIg6", "reviewer": "Reviewer_YE1U", "summary": "The paper introduces a new inference time method to mitigate hallucinations in LVLMs based on dropout decoding. The core idea behind the dropout decoding is to mask visual tokens based on the uncertainty of their predictions on LLM decoding and using ensemble method to generate uncertainity aware predictions. The empirical results demonstrate clear improvements in reducing object hallucinations and improving the generation quality across various LVLMs.", "questions": "- It would be good to present the failure cases of the proposed method especially cases where the method generates incorrect responses. What are the scenarios where this uncertainity mechanism yields factually incorrect response through LVLMs. Are there scenarios where relevant tokens are masked despite model generation correct initial response or vice versa.\n- Can the authors provide citations for the introduced uncertainty formulation from previous works? If its introduced in this paper, can the authors back it up from theoretical point of view?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Novelty: The use of dropout mechanism on the input visual tokens during inference guided by uncertainity metric is used for first time in context of improving LVLM reliability.\n- Empirical results show that the method consistently improve generation capability of LVLMs by reducing hallcinations and improved reliability on benchmarks such as CHIAR, THRONE, MMBench.\n- The application of epistemic uncertainity derived from textual interpretation of visual tokens is interesting. The authors provide ablations to highlight its necessity.\n- The proposed method in inference time method and thus do not require any retraining, which makes it easy to apply.\n\nWeakness:\n- Computational overhead: Though the method is used at inference time but K parallel decoding passes are computationally expensive especially for large LVLMs. \n- Different LVLMs require different hyperparams for masking, makes this method less user-friendly.\n- It's not clear how robust the method is to projection of visual tokens onto text space since uncertainty is derived from that. The visual text alignment is a crucial requirement for this method, which may not be true for certain visual concepts.", "was_revised": true, "final_justification": "I would stick to my original rating after authors responses based on the originality and significance of the paper.", "rebuttal": "Thank you for recognizing the novelty of our approach—applying a dropout mechanism guided by epistemic uncertainty on input visual tokens during inference to reduce hallucinations in LVLMs. We also appreciate your acknowledgment of our consistent empirical improvements across CHAIR, THRONE, and MMBench, as well as the practicality of our method given its plug-and-play nature without requiring any retraining.\n\nRegarding the concerns about our work, we find it mainly derives from misunderstandings of our work and we address them as below.\n\n\n### 1. Computational overhead\n\nWe collect the computation overhead data in our main experiments (aligned with the experiment setting in Table 1) and calculate the throughput and wall-time. As shown in the table, throughput only decreases by approximately 7% for the method without preliminary forward, and the wall-time impact remains minor. More importantly, performance consistently improves across various benchmarks, indicating a successful trade-off between efficiency and performance.\n\n| Metric                    | Greedy | Our method w/o prelim | Our method w/ prelim  |\n|---------------------------|--------|------------------------|------------------------|\n| CHAIRS ↓                  | 42.20  | 39.73 (−2.47)          | 39.80 (−2.40)          |\n| CHAIRI ↓                  | 12.83  | 12.20 (−0.63)          | 11.73 (−1.10)          |\n| F1all ↑                   | 0.795  | 0.799 (+0.004)         | 0.804 (+0.009)         |\n| F0.5all ↑                 | 0.784  | 0.794 (+0.010)         | 0.796 (+0.012)         |\n| Pall ↑                    | 0.772  | 0.791 (+0.019)         | 0.790 (+0.018)         |\n| Rall ↑                    | 0.847  | 0.843 (−0.004)         | 0.851 (+0.004)         |\n| Throughput (tok/s) ↑      | 37.0   | 34.1                   | 20.1                   |\n| Wall-time (per 50 tok) ↓  | 1.35   | 1.47                   | 2.49                   |\n\n### 2. Hyperparameter sensitivity\n\nWhile our method introduces hyperparameters, we have provided the ablation study of hyperparameters shown **in Figure 3 of the paper**, yielding consistently strong performance across various models and benchmarks, indicating that our method is not sensitive to the hyperparameters. Also, our default setting of hyperparameters **shown in the Appendix Section B** (we set Top-K=5, number of candidates to 3, $\\gamma^{(1)}$=0.3, $\\gamma^{(2)}$=0.5 and $\\gamma^{(3)}$=0.7 ) show the consistent performance on each evaluation scene.\n\n\n### 3. Robustness to visual-text alignment\n\nWe acknowledge the potential imperfections in projecting visual tokens onto the text space. However, this is an inherent limitation of adapter-based or late-fusion LVLMs themselves rather than our decoding method. Thus, we argue this architectural limitation should not detract from the significant contributions of our proposed test-time decoding approach, especially given its demonstrated effectiveness in practice.\n\n\nThank you again for your valuable comments."}, {"review_id": "DwR32svcRZ", "reviewer": "Reviewer_eLAj", "summary": "This paper introduces Dropout Decoding, a method for mitigating object hallucinations in large vision-language models (LVLMs). It quantifies the uncertainty of each visual token and selectively masks those with higher uncertainty to improve decoding accuracy. Experiments on CHAIR, THRONE, and MMBench demonstrate that the proposed approach effectively alleviates hallucinations in LVLMs and enhances the reliability of their responses across diverse scenarios.", "questions": "1. Could the authors provide results or insights on how the proposed method performs on open-ended benchmarks such as CHAIR, LLaVA-Bench, and AMBER, and standard benchmarks such as POPE?\n2. Could the authors provides quantitative efficiency metrics (e.g., FLOPs, throughput, or inference time) of the propsed approach and compare them with VCD or similar baselines?", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. This work addresses the hallucination problem from an uncertainty perspective, which is both intuitive and straightforward.\n2. Experimental results on three benchmarks validate that the proposed approach can enhance the reliability of LVLM responses.\n3. The paper is well-structured and clearly presented.\n\n**Weaknesses:**\n1. The experimental results are not sufficiently comprehensive; evaluations on open-ended benchmarks such as CHAIR, LLaVA-Bench, and AMBER should be included to better validate the generalizability and robustness of the proposed approach. Additionally, performance comparisons on the POPE benchmark should be included. It would also be valuable to evaluate the proposed approach on more recent LVLM architectures, such as the Qwen-VL series, to assess its effectiveness and compatibility with state-of-the-art models.\n2. The proposed approach bears similarities to existing work on visual token reduction [1, 2], and a clear discussion is needed to highlight the key differences. Additionally, qualitative visualizations of the retained and masked visual tokens should be included to enhance the interpretability of the method.\n3. Lines 302–303 state, \"*Compared to methods requiring multiple generation rounds, our approach offers a lightweight and effective trade-off.*\" To substantiate this claim, please include efficiency metrics such as FLOPs or throughput, and compare them with other hallucination mitigation approaches like VCD.\n4. More experimental details for the experiments in Section 7.4 should be provided. Additionally, the results seem to suggest that random masking may be more effective than the proposed approach in reducing hallucinations, despite yielding a slightly lower BLEU score on LLaVA-1.5. This is highly unintuitive. If this is the case, further clarification and discussion are needed.\n\n[1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. *ECCV '24*.\n\n[2] VisionZip: Longer is Better but Not Necessary in Vision Language Models. *CVPR '25*.\n\n\n**Minor issues:**\n1. Line 65, wrong citation format.\n2. The paper lacks sufficient discussion of related work on mitigating hallucinations in LVLMs. A more comprehensive review of existing approaches is recommended.", "was_revised": true, "final_justification": "The authors’ rebuttal has addressed some of my concerns, such as efficiency and the unintuitive results of random masking. However, I still find the evaluation somewhat weak and suggest that the authors compare with more recent baselines on additional benchmarks and newer models (such as Qwen-2/2.5-VL) in future revisions.", "rebuttal": "We appreciate your detailed and constructive feedback and respectfully address your concerns below:\n\n### 1. On Evaluation using Open-Ended Benchmarks:\n\nWe clarify that our paper indeed includes evaluations on open-ended benchmarks such as CHAIR, as reported explicitly in **Table 1 and Figure 3** (metrics: CHAIR-S and CHAIR-I). Additionally, we have evaluated our method extensively using the **THRONE benchmark**, an open-ended evaluation setting. Regarding benchmarks like LLaVA-Bench, AMBER, and POPE, and the evaluation of recent architectures such as Qwen-VL, we acknowledge your valuable suggestions and will incorporate these additional evaluations in our future revisions to further validate our approach's generalizability and robustness.\n\n### 2. Differences with Concurrent Work: \n\nWe thank the reviewer for pointing out these concurrent works and will cite and discuss them in our revision. However, we believe our contribution remains distinct. While those works also explore token dropout to mitigate hallucination or improve efficiency, their methods rely heavily on empirical observation of the attention mechanism rather than being derived from a principled foundation. In contrast, **our approach is grounded in uncertainty quantification**, providing a more theoretically motivated and generalizable framework. This difference in motivation and methodology highlights the novelty and rigor of our work.\n\n### 3. Efficiency:\n\nWe collect the computation overhead data in our main experiments (aligned with the experiment setting in Table 1) and calculate the throughput and wall-time. As shown in the table, throughput only decreases by approximately 7% for the method without preliminary forward, and the wall-time impact remains minor. More importantly, **performance consistently improves** across various benchmarks, indicating a successful trade-off between efficiency and performance.\n\n| Metric                   | Greedy | Our method w/o prelim | Our method w/ prelim  |\n|--------------------------|--------|------------------------|------------------------|\n| CHAIRS ↓                 | 42.20  | 39.73 (−2.47)          | 39.80 (−2.40)          |\n| CHAIRI ↓                 | 12.83  | 12.20 (−0.63)          | 11.73 (−1.10)          |\n| F1all ↑                  | 0.795  | 0.799 (+0.004)         | 0.804 (+0.009)         |\n| F0.5all ↑                | 0.784  | 0.794 (+0.010)         | 0.796 (+0.012)         |\n| Pall ↑                   | 0.772  | 0.791 (+0.019)         | 0.790 (+0.018)         |\n| Rall ↑                   | 0.847  | 0.843 (−0.004)         | 0.851 (+0.004)         |\n| Throughput (tok/s) ↑     | 37.0   | 34.1                   | 20.1                   |\n| Wall-time (per 50 tok) ↓ | 1.35   | 1.47                   | 2.49                   |\n\n### 4. Random Masking Effectiveness:\n\nYou mentioned the unintuitive result that random masking might appear more effective at reducing hallucinations, despite lower BLEU scores. As clearly discussed in Section 7.4, random masking often results in repetitive token generation (e.g., repeatedly generating “apple”), which artificially inflates CHAIR scores but reduces BLEU scores and **fails to yield meaningful THRONE metrics** due to impractical computational overhead. Thus, random masking actually provides invalid generation.\n\nThank you once again for your constructive comments and valuable suggestions, which greatly enhance our manuscript's quality and comprehensiveness."}, {"review_id": "pWDe3IJmdP", "reviewer": "Reviewer_LP6F", "summary": "This paper addresses the problem of mitigating hallucinations in Large Vision-Language Models (LVLMs). By quantifying the uncertainty of visual tokens and selectively masking those deemed unreliable during inference, the proposed method reduces errors stemming from misinterpreted visual information. Extensive experiments across diverse benchmarks demonstrate that this strategy significantly enhances the reliability and overall output quality of LVLMs.", "questions": "Typos in Lines 65 and 138.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. By projecting visual tokens onto the text space and masking out unreliable tokens, this paper provides a new perspective for mitigating hallucinations for LVLMs.\n2. Experiments validate the effectiveness of the proposed method.\n\nWeaknesses\n1. For generative tasks such as CHAIR, different decoding steps require focusing on different image areas. However, the proposed method masks image tokens only once for the entire decoding process, which may cause information loss.\n2. Generating K candidate predictions introduces additional inference overhead.\n3. The method involves tuning multiple hyperparameters, which complicates deployment and may reduce practicality.\n4. According to Table 3, the performance gain of the proposed uncertainty-guided decoding over a simple random masking baseline is marginal.", "was_revised": true, "final_justification": "Thanks for your responses. I have increased my score. Furthermore, since the random masking baseline is fundamentally flawed on the CHAIR evaluation, it would be better to choose another dataset and metrics to show the effectiveness of the proposed selection strategy.", "rebuttal": "Thank you for highlighting our core contribution—mitigating hallucinations in LVLMs by projecting visual tokens into the text space and masking uncertain ones during inference—as well as recognizing the strong empirical results across diverse benchmarks. We appreciate your acknowledgment of both the methodology and its practical importance in improving LVLM reliability.\n\nWe also appreciate the concerns proposed by the reviewer, and we believe they mainly come from the misunderstanding of our method, which we have clarified and solved below.\n\n\n### 1. Single-shot Masking May Lose Information:\n\nThank you for raising an important point regarding dynamic attention across different decoding steps. We agree that, in principle, different decoding steps may focus on different regions. Our design still works for three reasons, supported by the evidence below.\n\n- (1) **Masked tokens are statistically redundant**\n  \n  On 500 MS-COCO images (LLaVA-1.5), we computed uncertainty for every visual token before decoding and split them into the 50 highest-uncertainty (“pruned”) and the rest (“kept”).  Key statistics show the pruned set is dominated by outliers that contribute little to confident generation:\n\n| Statistic (mean)       | Kept | Dropout | Δ (P–K) |\n|------------------------|------|---------|--------|\n| Embedding norm         | 38.8 | 49.9    | +11.1  |\n| Distance to centroid   | 32.8 | 51.2    | +18.4  |\n| Logit entropy ↓        | 4.63 | 3.43    | –1.20  |\n\nHigher norm, larger distance, and lower entropy indicate that these tokens are both atypical and uninformative, aligning with earlier findings that image features are highly redundant.  Similar gaps appear in principal-component projections and variance measures.\n\n- (2) **Pruning once is already dynamic “enough”**\n\n  Because we evaluate uncertainty on-the-fly after the text prompt is appended but before generation starts, the retained set still depends on the input question.  During generation the model attends freely within this reduced set; we merely remove the outliers once.  \n\n  In practice the model ignores ≈45.6 % of image tokens per example, leaving ample capacity for spatial shifts across steps.\n\n- (3) **Step-wise re-masking is costly but yields no gain**\n\n  We implemented a step-wise variant that recomputes uncertainty every 5 decoding steps (“prelim” in our paper).  Throughput dropped from 34.1 tok/s to 20.1 tok/s (–41 %), wall-time per 50 tokens rose from 1.47 s to 2.49 s, yet CHAIR-S improved by only 0.07 pt over our single-shot version—well within noise.  Thus the added compute is not justified.\n\n  Taken together, these results show that a single uncertainty-guided pruning pass removes statistically redundant tokens, preserves the informative ones needed across decoding steps, and avoids the steep runtime penalty of continuous re-masking.\n\n### 2. Computational Overhead:\n\nRegarding the computational overhead, we provide a theoretical analysis in Section 7.1, noting that candidate generations are independent and can be efficiently parallelized via batch generation. To further support this, we conducted empirical evaluations using vLLM with LLaVA-1.5 (identical experiment setting in Table 1, the main experiment result table), where we measured both throughput and wall-time per 50 generated tokens. As shown in the table below, our method without the preliminary forward pass achieves strong performance gains on CHAIR and THRONE, while incurring only **a modest 7% drop in throughput** and a slight increase in wall-time. We believe this reflects a favorable tradeoff between computational cost and generation quality, demonstrating the practical efficiency of our approach.\n\n| Metric (50 tokens)       | Greedy | Our method w/o prelim | Our method w/ prelim  |\n|--------------------------|--------|------------------------|------------------------|\n| CHAIRS ↓                 | 42.20  | 39.73 (−2.47)          | 39.80 (−2.40)          |\n| CHAIRI ↓                 | 12.83  | 12.20 (−0.63)          | 11.73 (−1.10)          |\n| F1all ↑                  | 0.795  | 0.799 (+0.004)         | 0.804 (+0.009)         |\n| F0.5all ↑                | 0.784  | 0.794 (+0.010)         | 0.796 (+0.012)         |\n| Pall ↑                   | 0.772  | 0.791 (+0.019)         | 0.790 (+0.018)         |\n| Rall ↑                   | 0.847  | 0.843 (−0.004)         | 0.851 (+0.004)         |\n| Throughput (tok/s) ↑     | 37.0   | 34.1                   | 20.1                   |\n| Wall-time (per 50 tok) ↓ | 1.35   | 1.47                   | 2.49                   |\n\n### 3. Hyperparameter Tuning Complexity:\n\nAlthough our method introduces several hyperparameters, our experimental results of ablation study on hyperparameters (as detailed in Figure 3 of our paper) demonstrate robust performance improvements across various models and benchmarks, demonstrating that our method is not sensitive to hyperparameters. Also, our default setting of hyperparameters shown in the Appendix Section B(we set Top-K=5, number of candidates to 3, $\\gamma^{(1)}$=0.3, $\\gamma^{(2)}$=0.5 and $\\gamma^{(3)}$=0.7 ) the **consistent performance on each evaluation scene**. The setting maintains consistency across models and benchmarks.\n\n### 4. Misunderstanding on Random Masking Baseline:\n\nWe appreciate the reviewer’s feedback; however, we believe this concern stems from a misunderstanding of the random masking results. As shown in Section 7.4, random masking can produce superficially high scores on CHAIR by exploiting the evaluation metric—for instance, by repeatedly generating a single word like “apple” hundreds of times. While this may appear effective under certain benchmarks, it **fails to produce meaningful outputs**. Moreover, due to excessive token generation (up to the 512-token limit), it consistently causes out-of-memory (OOM) issues on THRONE, even when evaluated on 4×A100 GPUs. Therefore, we would like to clarify that the **random masking baseline is fundamentally flawed, despite its misleading performance on CHAIR**.\n\nThank you again for your valuable insights and constructive feedback."}]

Author rebuttal:

Meta-review: The paper received borderline reviews. While concerns around baseline evaluations (e.g., dropping out high confidence tokens and entropy based uncertainty measures) and running time were mostly addressed in the author response some concerns remain. These relate to the choice of benchmarks and the need to test against newer models. Nevertheless the principled approach linking to previous works that link dropout to uncertainty and the novelty of the approach were appreciated. Moreover the method appears general and to be effective. On the balance the Area Chair believes that the paper makes a valuable contribution with ideas that are of interest to the community.
Decision: Accept (poster)