MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibration
Authors: Zhitao He, Sandeep Polisetty, Zhiyuan Fan, Shujin Wu, Yuchen Huang, Yi R. Fung
Accepted at: acl25

MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibration

Zhitao He1  Sandeep Polisetty2*  Zhiyuan Fan1  Yuchen Huang1
Shujin Wu3*  Yi R. (May) Fung1
1Hong Kong University of Science and Technology
2UMass Amherst  3University of Southern California
{zhebu, yrfung}@cse.ust.hk

Abstract
In recent years, multimodal large language models (MLLMs) have made significant progress but continue to face inherent challenges in multimodal reasoning, which requires multi-level (\emph{e.g.}, perception, reasoning) and multi-granular (\emph{e.g.}, multi-step reasoning chain) advanced inferencing. Prior work on estimating model confidence tends to focus on the overall response for training and calibration, but fails to assess confidence in each reasoning step, leading to undesirable hallucination snowballing. In this work, we present MMBoundary, a novel framework that advances the knowledge boundary awareness of MLLMs through reasoning step confidence calibration. To achieve this, we propose to incorporate complementary textual and cross-modal self-rewarding signals to estimate confidence at each step of the MLLM reasoning process. In addition to supervised fine-tuning MLLM on this set of self-rewarding confidence estimation signal for initial confidence expression warm-up, we introduce a reinforcement learning stage with multiple reward functions for further aligning model knowledge and calibrating confidence at each reasoning step, enhancing reasoning chain self-correction. Empirical results show that MMBoundary significantly outperforms existing methods across diverse domain datasets and metrics, achieving an average of 7.5% reduction in multimodal confidence calibration errors and up to 8.3% improvement in task performance\footnote{Work done as a visiting student at HKUST.}.

1 Introduction
Although multimodal large language models (MLLMs) demonstrate exceptional abilities in cross-modal reasoning, the reliability of their responses remains uncertain due to the inherent challenges of multimodal reasoning (Zhou et al., 2023b; Huang et al., 2024b; Chen et al., 2024; Zhang et al., 2025). In particular, erroneous knowledge can occur not only at the cross-modal reasoning level but also in the early stages of visual perception. However, MLLMs typically fail to explicitly indicate their uncertainty to avoid the propagation and amplification of knowledge errors (Liu et al., 2024c; Huang et al., 2024c; Bai et al., 2024; Guan et al., 2024; Huang et al., 2024a). Therefore, it is crucial to enable MLLMs to accurately express confidence for each reasoning step during inference, enhancing

![Confidence calibration on reasoning step enables MLLMs to express natural language confidence statements during inference, enhancing self-correction of low-confidence steps and ultimately reasoning toward correct answers. Traditional methods calibrate model confidence solely on entire response, which can lead to incorrect answers with high confidence. Due to space limitations, only the reasoning chain of our method is presented. The red and purple colors indicate incorrect knowledge and confidence estimates, respectively.](page_1016_370_393_393.png)

Figure 1: Confidence calibration on reasoning step enables MLLMs to express natural language confidence statements during inference, enhancing self-correction of low-confidence steps and ultimately reasoning toward correct answers. Traditional methods calibrate model confidence solely on entire response, which can lead to incorrect answers with high confidence. Due to space limitations, only the reasoning chain of our method is presented. The red and purple colors indicate incorrect knowledge and confidence estimates, respectively.
reasoning chain self-correction.

Prior work on estimating model confidence tends to focus on the overall response for training and calibration (Yang et al., 2023; Zhang et al., 2024a; Lyu et al., 2024; Xu et al., 2024). However, these methods fail to enable the trained models to express confidence estimates for different knowledge within generated content. As shown in Figure 1 (upper part), the trained MLLM generates incorrect information at the visual perception level (i.e., misidentifying the "drum" as a "shield") without expressing its uncertainty, causing significant deviations in reasoning chain and ultimately producing an incorrect answer. Moreover, due to the logical coherence of the reasoning, the model still generates a high confidence score in its overall response.

Therefore, in this work, we propose MMBoundary, a reinforced fine-tuning framework for advancing MLLM knowledge boundary awareness by reasoning step confidence calibration. Our method enables the model to express natural language confidence statement for each generated sentence, enhancing reasoning chain self-correction by scaling inference-time. Specifically, we introduce a confidence estimation module that integrates three effective text-based uncertainty methods—namely, length-normalized log probability, mean token entropy, and tokenSAR—and incorporates cross-modal constraint (i.e., CLIPScore) to model the self-rewarding confidence signal from the perspective of its internal states. Then, we propose a mutual mapping between the detected score and predefined confidence statements to achieve two objectives: (1) by inserting confidence statements after the associated knowledge and training the model via supervised learning, we enable the model to naturally generate natural language statements for each sentence, similar to human expression; (2) by integrating internally detected confidence scores and those converted from model expressed statements into the reward modeling for reinforcement learning, we can achieve further confidence calibration, reducing the inaccuracy of model-expressed confidence. Moreover, we annotate the reference reasoning chain of training data to facilitate rigorous evaluation of MLLMs’ knowledge at different reasoning levels, and incorporate model knowledge calibration into the reward modeling, encouraging MLLMs to faithfully express confidence while improving response quality.

Experimental results from both automatic and human evaluations across diverse domain datasets demonstrate that MMBoundary significantly reduces confidence calibration errors while simultaneously enhancing task performance.

The contributions of our work can be summarized as follows:

• We present a novel framework, MMBoundary, for advancing the knowledge boundary awareness of multimodal language models through reasoning step confidence calibration.

• We propose to integrate both textual and cross-modal self-rewarding signals for confidence estimation. Beyond supervised fine-tuning for initial confidence expression warm-up, we introduce a reinforcement learning stage with multiple reward functions to align model knowledge and calibrate confidence, enhancing self-correction in reasoning chain.

• Empirical results show that MMBoundary significantly outperforms existing methods, achieving an average reduction of 7.5% in multimodal confidence calibration errors and up to 8.3% improvement in task performance.

2 Problem Formulation

Given a multimodal model \( \pi_\theta \) with parameter \( \theta \), prior work focuses on enabling the model to output a confidence estimate for its entire response \( y \), formalized as:

\[
y = [z_1, z_2, \ldots, z_T, c]
\]

Here, \( z_t \) represents the t-th generated sentence, \( c \) denotes the overall confidence estimate, \( T \) is the total number of sentences in the response. However, the trained model often assign high confidence incorrectly. Therefore we aim to train models to express fine-grained confidence estimate for each sentence during inference for enhancing reasoning chain self-correction. Thus, the output:

\[
y = [z_1, c_1, z_2, c_2, \ldots, z_T, c_T]
\]

Each pair (\( z_t, c_t \)) represents the t-th sentence generated by the model and its corresponding confidence statement, respectively.

3 Methodology

Our framework consists of two stages: the confidence expression warm-up stage and the reinforcement learning stage, as shown in Figure 2.
Figure 2: The overview of MMBoundary, which consists of two stages. The initial stage trains MLLMs via supervised learning to generate natural language confidence statement for each sentence, similar to human expression. The second stage employs reinforcement learning with three intuitively designed reward functions to further calibrate the expressed confidence estimates and enhance knowledge alignment. ■ represents the internal states (i.e., the log probability of tokens) of model and the estimated internal confidence.

3.1 Confidence Expression Warm-Up

3.1.1 Internal Confidence Estimation

In this section, we propose to leverage multiple text-based uncertainty methods and incorporate visual constraint to estimate MLLM’s confidence. Previous work primarily relies on model response consistency as a confidence indicator. However, these methods fail to assess confidence across distinct knowledge in generated content and do not consider the correlation between the response and the visual information, limiting their applicability in multimodal scenarios. Drawing on recent research (Xiao et al., 2022; Fadeeva et al., 2023; Vashurin et al., 2024), we utilize the following efficient and effective uncertainty estimation methods to create our confidence indicator:

(1) Length-normalized log probability calculates the average negative log probability of the tokens generated:

\[
U_{LNLP}(\mathbf{y}, \mathbf{x}; \theta) = \exp \left\{ -\frac{1}{L} \log P(\mathbf{y} \mid \mathbf{x}, \theta) \right\},
\]

where \( \mathbf{x} \) denotes the input, \( \mathbf{y} \) denotes the output, and \( \theta \) represents the model parameters.

(2) Mean token entropy (Fomicheva et al., 2020) computes the average entropy for each token in the generated sentence:

\[
U_{MTE}(\mathbf{y}, \mathbf{x}; \theta) = \frac{1}{L} \sum_{l=1}^{L} \mathcal{H}(y_l \mid y_{<l}, \mathbf{x}, \theta),
\]

where \( \mathcal{H}(y_l \mid y_{<l}, \mathbf{x}, \theta) \) is an entropy of the token distribution \( P(y_l \mid y_{<l}, \mathbf{x}, \theta) \).

(3) TokenSAR (Duan et al., 2024) computes the weighted average of the negative log probability of generated tokens, considering their relevance to the entire generated text. For a given sentence similarity function \( g(\cdot, \cdot) \) and token relevance function \( R_T(y_k, \mathbf{y}, \mathbf{x}) = 1 - g(\mathbf{x} \cup \mathbf{y}, \mathbf{x} \cup \mathbf{y} \setminus y_k) \), the resulting estimate is computed as:

\[
U_{TokenSAR}(\mathbf{y}, \mathbf{x}; \theta) =
- \sum_{l=1}^{L} \tilde{R}_T(y_l, \mathbf{y}, \mathbf{x}) \log P(y_l \mid y_{<l}, \mathbf{x}, \theta),
\]

where \( \tilde{R}_T(y_k, \mathbf{y}, \mathbf{x}) = \frac{R_T(y_k, \mathbf{y}, \mathbf{x})}{\sum_{l=1}^{L} R_T(y_l, \mathbf{y}, \mathbf{x})} \).

(4) CLIPScore (Hessel et al., 2021) evaluates the relevance between the generated sentence and input image. Since CLIP’s vision encoder aligns with the target MLLM’s, we employ CLIPScore to represent the sentence-image uncertainty. For an image with visual CLIP embedding \( v \) and a sentence with textual CLIP embedding \( s \):
<table>
  <tr>
    <th>Score</th>
    <th>Confidence Statement</th>
  </tr>
  <tr>
    <td>1</td>
    <td>but I can’t confirm this. / I’m uncertain about this. (…)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>and it may need checking/it might not be right. (…)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>but I can’t guarantee perfection. / I can’t be entirely sure (…)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>and this seems trustworthy. / and I believe this is right. (…)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>with total certainty. / with no doubts at all. (…)</td>
  </tr>
</table>

Figure 3: We preset a confidence statement pool for each confidence score. The five levels correspond to uncertain, slightly uncertain, moderately confident, highly confident, and fully confident. More statements are shown in Appendix A.

\[
U_{CLIP} \text{Score}(\mathbf{v}, \mathbf{s}) = \max (\cos(\mathbf{v}, \mathbf{s}), 0)
\] (6)

We normalize \( U_i \) across the entire dataset using min-max normalization to ensure their values are within the range [0, 1]. Then, we compute the final weighted average as:

\[
U_{\text{Final}} = w_0 U_{LNLP} + w_1 U_{MTE}
+ w_2 U_{TokenSAR} + w_3 U_{CLIP} \text{Score}
\] (7)

where \( w_i \) are the respective weights for each component. The closer \( U_{\text{Final}} \) is to 0, the greater the certainty of the model. Then, we use the distribution of \( U_{\text{Final}} \) to define confidence levels for the model, considering the uneven distribution of \( U_{\text{Final}} \). Confidence levels \( C_v \) from 5 to 1 correspond to the intervals of \( U_{\text{Final}} \) as [0, \( \mu - \sigma \), \( \mu + \sigma \), \( \mu + 2\sigma \), \( \mu + 3\sigma \), 1], with higher confidence levels indicating greater model confidence. Here, \( \mu \) and \( \sigma \) represents the average and the standard deviation of \( U_{\text{Final}} \). We further validate the effectiveness of this confidence level classification method in Section 5.2.

3.1.2 Confidence Score-Statement Mapping

This module, as shown on the right side of Figure 2, aims to establish a mutual mapping between the detected score and predefined confidence statements. First, we construct statement pools for each confidence level, as shown in Figure 3. These statements can be naturally appended to the end of sentences, providing a concise expression of the model’s confidence estimates, similar to human expression. During the Confidence Expression Warm-Up Stage, we randomly select statements from the corresponding pools based on the detected scores and insert them into the model’s original response to create data for fine-tuning. In the Reinforcement Learning Stage, after obtaining the confidence statements from the model’s output, we encode these statements into vectors using an encoder model\( ^2 \) and compute the cosine similarity with all embeddings in the different confidence pools to achieve reverse mapping of statements to confidence scores.

3.1.3 Supervised Fine-Tuning

Specifically, the model undergoes fine-tuning on our constructed data \( \mathcal{D} \) consisting of tuples: (x, y), where the input x comprises an image I and a question Q. At step \( s_t \), the sentence with its confidence statement (\( z_t, c_t \)) are generated by the model’s policy \( \pi_\theta \). The next state \( s_{t+1} \) is:

\[
s_{t+1} = \begin{cases}
x, & t = 0 \\
[s_t, z_t, c_t], & 1 \leq t \leq T
\end{cases}
\] (8)

We fine-tune the vanilla model via supervised learning. The loss function can be written as:

\[
\mathcal{L}_{FT}(\theta) = -\mathbb{E}_{(\mathbf{x}, \mathbf{y}) \sim \mathcal{D}} \left[ \sum_{t=1}^T \log \pi_\theta(z_t, c_t | s_t) \right]
\] (9)

3.2 Reinforcement Learning

As noted by Xu et al., 2024, the model undergoing supervised training tends to generate uniform confidence levels, which may impact task performance. Therefore, we employ reinforcement learning with reward signals involving model knowledge alignment, internal confidence and external confidence calibration to encourage model to faithfully express confidence while simultaneously improving the quality of responses. Specifically, we sample questions from the training data and prompt the model to generate responses.

(1) Knowledge Accuracy Reward evaluates whether the knowledge in generated response is aligned with annotated reference chain, thereby ensuring the reliability of the generated content. Specifically, if the t-th generated sentence \( z_t \) matches the knowledge in reference chain, \( R_{KA_t} \) is 1. Refer to the "Step Matched" example in Figure 6. After evaluating all generated sentences, the reward is normalized:

\[
R_{KA} = \frac{1}{T} \sum_{t=1}^T R_{KA_t}
\] (10)

where \( T \) is the total number of sentences.

\( ^2 \)https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
(2) Expected Calibration Reward is consistent with Xu et al. (2024), but we extend it to sentence-level. This reward function measure the correlation between the expressed confidence and the ground truth. The Expected Calibration Reward (\( R_{EC} \)) is fundamentally consistent with Expected Calibration Error (ECE). We expect the model’s confidence scores to properly reflect answer quality, i.e. lower confidence for poor-quality answers and vice versa. The reward function is formalized as follows:

\[
R_{EC} = \frac{1}{T} \sum_{t=1}^{T} [1 - 2 \cdot (\mathbb{I}(z_t) - \mathrm{EV}(c_t))^2]
\] (11)

where \( \mathbb{I}(\cdot) \) is the indicator function that returns 1 if the sentence is correct compared with reference chain, and 0 otherwise. \( \mathrm{EV}(c_t) \) represents the expressed confidence score, which is obtained by mapping and normalizing the confidence statements generated by the model. The confidence score is normalized between 0 and 1.

(3) Confidence Self-Calibration Reward is based on the consistency between the expressed confidence and internal confidence of MLLMs:

\[
R_{CS} = \frac{1}{T} \sum_{t=1}^{T} [1 - 2 \cdot (\mathrm{IV}(z_t) - \mathrm{EV}(c_t))^2]
\] (12)

where \( \mathrm{IV}(z_t) \) represents the internal confidence score, which is estimated by our method in Section 3.1.1. This reward encourages the model to express its confidence level as accurately as possible, aligning its external expression with internal belief. Thus, the overall reward for response is:

\[
R = \alpha R_{KA} + \beta R_{EC} + \gamma R_{CS}
\] (13)

Lastly, we employ the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017) for training. The model’s policy objectives is:

\[
\mathcal{L}_{RL}(\theta) = -\mathbb{E}_{x \sim \pi_{\theta_{old}}} \left[ \min \left( \frac{\pi_{\theta}(z_t, c_t|s_t)}{\pi_{\theta_{old}}(z_t, c_t|s_t)} \hat{A}_t, \right. \right.
\]
\[
\left. \left. \mathrm{clip} \left( \frac{\pi_{\theta}(z_t, c_t|s_t)}{\pi_{\theta_{old}}(z_t, c_t|s_t)}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_t \right) \right]
\] (14)

The advantage estimate \( \hat{A}_t \) (Schulman et al., 2015) is derived by calculating the difference between the anticipated future rewards under the current policy and the baseline or value function. Implementation and data details can be found in Appendix B.

4 Experiments

4.1 Dataset

In order to evaluate the model’s robustness and generalizability across diverse scenarios, we select the following datasets from different domains: A-OKVQA (Schwenk et al., 2022), a general domain dataset designed to evaluate models on complex visual question answering tasks involving multi-hop reasoning, commonsense understanding, and external knowledge integration; ScienceVQA (Lu et al., 2022), a large-scale multimodal dataset designed for science question answering, featuring questions across natural science, social science, and language science; CulturalVQA (Nayak et al., 2024), a visual question-answering benchmark evaluating MLLMs on understanding geo-diverse cultural concepts beyond general scene understanding.

4.2 Reasoning Chain Annotation

To simultaneously calibrate the model’s knowledge and confidence levels, we conduct detailed reasoning chain annotations for each question in the training dataset. As shown in Figure 5, for each question, we prompt the GPT-4o to generate analysis (inference chain) structured in the perception and reasoning level. The former identifies key visual elements in the image that are most relevant to the question and answer, while the latter provides granularity reasoning that justifies why the answer is correct. Each level should include concise, interconnected sentences, with each sentence conveying a single piece of knowledge. Then, we perform filtering and quality evaluation to ensure the accuracy and consistency. Due to space limitations, please refer to Appendix C for more details.

4.3 Evaluation Metrics

Consistent with previous research (Chen et al., 2022; Xu et al., 2024), we evaluate models from three perspectives using six different metrics:

(1) Confidence Calibration Performance: We adopt 3 calibration metrics. First, we use the Expected Calibration Error (ECE) score (Guo et al., 2017). Then, we extend the ECE score to measure the confidence calibration error of each knowledge within reasoning chain, which we refer to as Multi-granularity Expected Calibration Error (MECE). The MECE score evaluates the correlation between the confidence estimates expressed in generated sentences and their corresponding correctness, as shown in Figure 6. Details of MECE computation
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="5">A-OKVQA</th>
    <th colspan="5">ScienceVQA</th>
    <th colspan="5">CulturalVQA</th>
  </tr>
  <tr>
    <th>ECE (↓)</th>
    <th>MECE (↓)</th>
    <th>Acc (%)</th>
    <th>F1 (↑)</th>
    <th>ECE (↓)</th>
    <th>MECE (↓)</th>
    <th>Acc (%)</th>
    <th>F1 (↑)</th>
    <th>ECE (↓)</th>
    <th>MECE (↓)</th>
    <th>Acc (%)</th>
    <th>F1 (↑)</th>
  </tr>
  <tr>
    <td>DPV</td>
    <td>0.563</td>
    <td>0.582</td>
    <td>0.650</td>
    <td>0.512</td>
    <td>0.593</td>
    <td>0.611</td>
    <td>0.582</td>
    <td>0.414</td>
    <td>0.624</td>
    <td>0.650</td>
    <td>0.334</td>
    <td>0.482</td>
  </tr>
  <tr>
    <td>DPS</td>
    <td>0.511</td>
    <td>0.554</td>
    <td>0.675</td>
    <td>0.535</td>
    <td>0.574</td>
    <td>0.575</td>
    <td>0.575</td>
    <td>0.427</td>
    <td>0.572</td>
    <td>0.594</td>
    <td>0.354</td>
    <td>0.485</td>
  </tr>
  <tr>
    <td>SC</td>
    <td>0.435</td>
    <td>0.492</td>
    <td>0.701</td>
    <td>0.548</td>
    <td>0.463</td>
    <td>0.534</td>
    <td>0.602</td>
    <td>0.442</td>
    <td>0.491</td>
    <td>0.554</td>
    <td>0.371</td>
    <td>0.511</td>
  </tr>
  <tr>
    <td>Multisample</td>
    <td>0.413</td>
    <td>0.430</td>
    <td>0.683</td>
    <td>0.543</td>
    <td>0.446</td>
    <td>0.500</td>
    <td>0.596</td>
    <td>0.434</td>
    <td>0.463</td>
    <td>0.505</td>
    <td>0.362</td>
    <td>0.538</td>
  </tr>
  <tr>
    <td>SaySelf</td>
    <td>0.345</td>
    <td>0.384</td>
    <td>0.734</td>
    <td>0.603</td>
    <td>0.386</td>
    <td>0.462</td>
    <td>0.633</td>
    <td>0.483</td>
    <td>0.375</td>
    <td>0.437</td>
    <td>0.417</td>
    <td>0.571</td>
  </tr>
  <tr>
    <td>Conf-CSR</td>
    <td>0.408</td>
    <td>0.437</td>
    <td>0.785</td>
    <td>0.618</td>
    <td>0.453</td>
    <td>0.503</td>
    <td>0.694</td>
    <td>0.502</td>
    <td>0.472</td>
    <td>0.513</td>
    <td>0.435</td>
    <td>0.582</td>
  </tr>
  <tr>
    <td>RCE</td>
    <td>0.361</td>
    <td>0.394</td>
    <td>0.788</td>
    <td>0.620</td>
    <td>0.413</td>
    <td>0.475</td>
    <td>0.671</td>
    <td>0.497</td>
    <td>0.408</td>
    <td>0.453</td>
    <td>0.412</td>
    <td>0.577</td>
  </tr>
  <tr>
    <td>DRL</td>
    <td>0.395</td>
    <td>0.453</td>
    <td>0.746</td>
    <td>0.614</td>
    <td>0.476</td>
    <td>0.513</td>
    <td>0.654</td>
    <td>0.485</td>
    <td>0.453</td>
    <td>0.502</td>
    <td>0.392</td>
    <td>0.564</td>
  </tr>
  <tr>
    <td>MMBoundary</td>
    <td><b>0.316</b></td>
    <td><b>0.304</b></td>
    <td>0.835</td>
    <td><b>0.661</b></td>
    <td>0.354</td>
    <td><b>0.392</b></td>
    <td>0.703</td>
    <td><b>0.565</b></td>
    <td><b>0.337</b></td>
    <td><b>0.361</b></td>
    <td><b>0.448</b></td>
    <td><b>0.665</b></td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="13">Warm-Up Stage</th>
  </tr>
  <tr>
    <td>w/o \( U_{LNLP} \)</td>
    <td>0.327</td>
    <td>0.343</td>
    <td>0.815</td>
    <td>0.642</td>
    <td>0.369</td>
    <td>0.421</td>
    <td>0.698</td>
    <td>0.548</td>
    <td>0.356</td>
    <td>0.397</td>
    <td>0.423</td>
    <td>0.639</td>
  </tr>
  <tr>
    <td>w/o \( U_{MTF} \)</td>
    <td>0.337</td>
    <td>0.332</td>
    <td>0.824</td>
    <td>0.653</td>
    <td>0.385</td>
    <td>0.441</td>
    <td>0.682</td>
    <td>0.536</td>
    <td>0.349</td>
    <td>0.378</td>
    <td>0.430</td>
    <td>0.643</td>
  </tr>
  <tr>
    <td>w/o \( U_{TSAR} \)</td>
    <td>0.324</td>
    <td>0.358</td>
    <td>0.813</td>
    <td>0.627</td>
    <td>0.352</td>
    <td>0.417</td>
    <td>0.694</td>
    <td>0.546</td>
    <td>0.361</td>
    <td>0.403</td>
    <td>0.438</td>
    <td>0.655</td>
  </tr>
  <tr>
    <td>w/o \( U_{CLIPS} \)</td>
    <td>0.337</td>
    <td>0.354</td>
    <td>0.806</td>
    <td>0.631</td>
    <td>0.372</td>
    <td>0.435</td>
    <td>0.681</td>
    <td>0.532</td>
    <td>0.358</td>
    <td>0.394</td>
    <td>0.426</td>
    <td>0.637</td>
  </tr>
  <tr>
    <td>w U<sub>Max</sub></td>
    <td>0.362</td>
    <td>0.386</td>
    <td>0.774</td>
    <td>0.583</td>
    <td>0.391</td>
    <td>0.467</td>
    <td>0.663</td>
    <td>0.494</td>
    <td>0.378</td>
    <td>0.425</td>
    <td>0.403</td>
    <td>0.589</td>
  </tr>
  <tr>
    <td>w/o S-S<sub>Mapping</sub></td>
    <td>0.340</td>
    <td>0.362</td>
    <td>0.793</td>
    <td>0.634</td>
    <td>0.377</td>
    <td>0.443</td>
    <td>0.684</td>
    <td>0.531</td>
    <td>0.365</td>
    <td>0.398</td>
    <td>0.427</td>
    <td>0.602</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="13">Reinforcement Learning Stage</th>
  </tr>
  <tr>
    <td>w/o \( R_{KA} \)</td>
    <td>0.325</td>
    <td>0.347</td>
    <td>0.802</td>
    <td>0.629</td>
    <td><b>0.334</b></td>
    <td>0.410</td>
    <td>0.686</td>
    <td>0.535</td>
    <td>0.348</td>
    <td>0.370</td>
    <td>0.437</td>
    <td>0.632</td>
  </tr>
  <tr>
    <td>w/o \( R_{EC} \)</td>
    <td>0.332</td>
    <td>0.357</td>
    <td>0.819</td>
    <td>0.635</td>
    <td>0.363</td>
    <td>0.426</td>
    <td><b>0.712</b></td>
    <td>0.548</td>
    <td>0.359</td>
    <td>0.392</td>
    <td>0.422</td>
    <td>0.648</td>
  </tr>
  <tr>
    <td>w/o \( R_{CS} \)</td>
    <td>0.343</td>
    <td>0.368</td>
    <td><b>0.857</b></td>
    <td>0.648</td>
    <td>0.372</td>
    <td>0.449</td>
    <td>0.693</td>
    <td>0.556</td>
    <td>0.368</td>
    <td>0.417</td>
    <td>0.436</td>
    <td>0.640</td>
  </tr>
  <tr>
    <td>w/o RL</td>
    <td>0.392</td>
    <td>0.427</td>
    <td>0.768</td>
    <td>0.581</td>
    <td>0.419</td>
    <td>0.481</td>
    <td>0.663</td>
    <td>0.495</td>
    <td>0.408</td>
    <td>0.456</td>
    <td>0.419</td>
    <td>0.595</td>
  </tr>
</table>

Table 1: The evaluation results of models and various ablations of our framework. CulturalVQA is the out-of-distribution dataset. w/o \( U_{LNLP} \), w/o \( U_{MTF} \), w/o \( U_{TSAR} \), and w/o \( U_{CLIPS} \) represent MMBoundary without the three text-based uncertainty estimation methods and visual information uncertainty estimation, respectively; w \( U_{Max} \) indicates the confidence determined using the max pooling method from the four uncertainty estimation scores; w/o S-S<sub>Mapping</sub> denotes MMBoundary without confidence score-statement mapping; w/o \( R_{KA} \), w/o \( R_{EC} \), and w/o \( R_{CS} \) represent MMBoundary without knowledge accuracy reward, expected calibration reward, and confidence self-calibration reward, respectively; w/o RL denotes MMBoundary without reinforcement learning.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">A-OKVQA</th>
    <th colspan="4">ScienceVQA</th>
    <th colspan="4">CulturalVQA</th>
  </tr>
  <tr>
    <th>Faithful</th>
    <th>Concise</th>
    <th>Granular</th>
    <th>Avg.</th>
    <th>Faithful</th>
    <th>Concise</th>
    <th>Granular</th>
    <th>Avg.</th>
    <th>Faithful</th>
    <th>Concise</th>
    <th>Granular</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Multisample</td>
    <td>4.20</td>
    <td>5.17</td>
    <td>4.06</td>
    <td>4.47</td>
    <td>4.77</td>
    <td>5.24</td>
    <td>4.53</td>
    <td>4.85</td>
    <td>3.91</td>
    <td>5.63</td>
    <td>4.72</td>
    <td>4.75</td>
  </tr>
  <tr>
    <td>SaySelf</td>
    <td>7.28</td>
    <td><b>7.49</b></td>
    <td>6.47</td>
    <td>7.08</td>
    <td>7.49</td>
    <td>7.18</td>
    <td>6.28</td>
    <td>6.98</td>
    <td>7.12</td>
    <td>6.81</td>
    <td>6.58</td>
    <td>6.83</td>
  </tr>
  <tr>
    <td>Conf-CSR</td>
    <td>6.47</td>
    <td>5.73</td>
    <td>5.82</td>
    <td>6.01</td>
    <td>6.74</td>
    <td>5.61</td>
    <td>6.40</td>
    <td>6.23</td>
    <td>6.38</td>
    <td>5.45</td>
    <td>5.86</td>
    <td>5.89</td>
  </tr>
  <tr>
    <td>RCE</td>
    <td>6.73</td>
    <td>6.58</td>
    <td>7.41</td>
    <td>6.90</td>
    <td>7.55</td>
    <td>6.92</td>
    <td>7.12</td>
    <td>7.19</td>
    <td>6.84</td>
    <td>6.19</td>
    <td>6.82</td>
    <td>6.62</td>
  </tr>
  <tr>
    <td>DRL</td>
    <td>6.54</td>
    <td>6.13</td>
    <td>6.95</td>
    <td>6.54</td>
    <td>6.81</td>
    <td>5.97</td>
    <td>6.34</td>
    <td>6.37</td>
    <td>6.55</td>
    <td>5.63</td>
    <td>6.07</td>
    <td>6.08</td>
  </tr>
  <tr>
    <td>MMBoundary</td>
    <td><b>7.83</b></td>
    <td>7.25</td>
    <td><b>8.18</b></td>
    <td><b>7.75</b></td>
    <td><b>8.35</b></td>
    <td><b>7.46</b></td>
    <td><b>8.02</b></td>
    <td><b>7.94</b></td>
    <td><b>7.66</b></td>
    <td><b>7.17</b></td>
    <td><b>8.26</b></td>
    <td><b>7.69</b></td>
  </tr>
</table>

Table 2: The human evaluation results of strong baselines and our framework. We provide a panel of three graduate students with 50 random entries from each setting, asking them to evaluate whether each entry meets the criteria (Faithful, Concise, Granular) and to give a score from 1 to 10. The final result is the average score.

process is in the Appendix D.1. For all responses \( A \) generated by MLLMs:

\[
\text{MECE} = \frac{1}{|A|} \sum_{a \in A} \frac{1}{|a|} \sum_{(z,c) \in a} |\mathbb{I}(z) - \operatorname{conf}(c)|
\]

Here, \( a \) represents the model’s response to the question, while \( z \) and \( c \) represent the sentences in the response and their corresponding confidence statements, respectively. \( \operatorname{Conf}(\cdot) \) represents the numerical value of the confidence statement.

(2) Task Performance: We adopt 2 metrics. First, we measure the typical Accuracy. Second, to identify model responses containing erroneous knowledge and mitigate the risk of them being assigned high confidence, we evaluate the quality of the model’s reasoning chain by employing the metric in Reasoning Chain F1 score (Ho et al., 2022). This metric compares the information contained in the predictions and references. We present implementation details in the Appendix D.3.

(3) Human Evaluation: Automated model evaluation may not accurately capture the subtle differences between different responses (Goyal et al., 2022; Ho et al., 2022; He et al., 2023). Therefore,
we conduct additional manual evaluation. We provide a panel of three graduate students with 50 random entries from each setting, asking them to evaluate whether each entry meets the following criteria and to give a score from 1 to 10, consistent with (Xu et al., 2024). 1) Faithful: whether the response faithfully expresses the confidence; 2) Concise: whether the response conveys necessary information clearly and without excess; 3) Granular: whether the response contains confidence estimates for distinct knowledge. The final result is the average score of these criteria.

4.4 Baselines
We compare with the following methods: (1) DPV: directly prompting the vanilla MLLMs to give a response with a confidence score; (2) DPS: direct prompting the vanilla MLLMs to give a response with a confidence statement; (3) SC (Xiong et al., 2023): deriving the confidence estimates of MLLMs based on diverse sampling; (4) Multisample (Yang et al., 2023): training MLLMs to generate confidence estimates that align with the confidence derived from self-consistency; (5) SaySelf (Xu et al., 2024): analyzing inconsistencies in multiple sampled responses, with the resulting data used for supervised fine-tuning and then confidence estimates calibrated through reinforcement learning based on task supervision; (6) ConfCSR: converting the calibrated self-reward (Zhou et al., 2024) of each sentence into the model’s confidence score and utilize DPO (Rafailov et al., 2024) for optimization; (7) RCE: training the model to first generate a complete response and then produce confidence estimates for each sentence; (8) DRL: directly employing our reinforcement learning method to train model. We use LLaVA-NEXT 7B (Liu et al., 2024d) as backbone model for all methods to ensure a fair comparison. To prove that our method can generalize across models, we also conduct experiments on Qwen2VL 7B (Wang et al., 2024) in Appendix E.

4.5 Main Experimental Results
Confidence Calibration Performance. We present the ECE and MECE results in Table 1 and the AUROC results in Appendix (Table 8), which measure the correlation between the expressed confidence and the ground truth. The findings indicate that MMBoundary outperforms other methods in reducing confidence calibration errors and enhancing the ability to distinguish confidence between correct and incorrect answers (AUROC). This conclusion is validated on both in-distribution datasets (AOKVQA and ScienceVQA, with a 7.5% improvement in the MECE score and out-of-distribution datasets (CulturalVQA, showing an increase of 6.6%), highlighting the generality of our framework. Compared to baseline methods (DRL, RCE, and SaySelf) that also require annotated data during the RL phase, our method achieves the best performance, improving MECE and F1 scores by 5.2% and 7.8%, respectively. When no annotated data is used, our method (w/o RL) surpasses the baselines (Multisample and Conf-CSR) by up to 5.7% (MECE) under the same settings.

Task Performance. We comprehensively evaluate the task performance of the model using the final answer accuracy and the Reasoning Chain F1 score, as presented in Table 1. The results show that our method surpasses other baselines across three datasets, achieving up to 8.3% improvement in CulturalVQA. Unlike Conf-CSR and SaySelf, which rely solely on task-oriented reward or the expected calibration reward, our approach integrates knowledge alignment along with internal and external confidence calibration into reward modeling. The results demonstrate that our framework improves the model’s knowledge boundary awareness while simultaneously enhancing its task performance. We conduct paired t-tests on the experimental results of MMBoundary, showing significant advantages over the baselines (p-value < 0.05).

Human Evaluation. We conduct human evaluation of the responses generated by our method and other strong baselines across the dimensions of faithfulness, conciseness, and granularity, with the results shown in Table 2 and Figure 7. We observe that our framework demonstrates statistically significant improvements over three dimensions. SaySelf performs well in the concise dimension for content, but it is designed only to estimate confidence for the entire response, lacking the ability to generate confidence for each step of the reasoning process. We perform a Kappa test on the faithfulness evaluation results to assess inter-annotator agreement, obtaining a Kappa value of 0.79.

5 Discussion

5.1 Influence of Different Components
We conduct an extensive ablation study to verify the effectiveness of different components, with re-
<table>
  <tr>
    <th>Dataset</th>
    <th>Mean</th>
    <th>Var</th>
    <th>Std</th>
    <th><0.1</th>
  </tr>
  <tr>
    <td>A-OKVQA</td>
    <td>0.0443</td>
    <td>0.0015</td>
    <td>0.0397</td>
    <td>96%</td>
  </tr>
  <tr>
    <td>ScienceVQA</td>
    <td>0.0578</td>
    <td>0.0014</td>
    <td>0.0374</td>
    <td>93%</td>
  </tr>
  <tr>
    <td>CulturalIVQA</td>
    <td>0.0522</td>
    <td>0.0015</td>
    <td>0.0397</td>
    <td>94%</td>
  </tr>
</table>

Table 3: Comparison between our internal confidence estimation (ICE) and widely adapted self-consistency-based estimation (SCE). We compute \( |C_{ICE} - C_{SCE}| \) to demonstrate the correlation between the two methods.

sults shown in Table 1. Compared to the version without RL, supervised fine-tuning enable model to express confidence during inference. Incorporating RL with our reward signals further improves confidence precision, with an average 9.2% increase in the MECE score. Each reward term encourages the model to focus on distinct aspects (e.g., knowledge alignment or confidence calibration). When \( R_{KA} \) is removed (leaving only confidence calibration rewards), the model prioritizes confidence calibration, leading to improved Expected Calibration Error performance on ScienceVQA and resulting in an average performance drop of 3.1%. Conversely, removing \( R_{EC} \) or \( R_{CS} \) shifts the model’s focus toward knowledge alignment (as the weight of knowledge alignment rewards becomes dominant), thereby improving accuracy (Acc) on A-OKVQA. The removal of \( R_{EC} \) and \( R_{CS} \) leads to a maximum decrease of 6.4% in the confidence calibration performance. Furthermore, the results show that all four selected uncertainty estimation methods enhance model performance, with the \( U_{TSAR} \) having the most significant impact on confidence calibration. Moreover, the conversion between confidence scores and statements (S-SMapping) positively impacts the model’s confidence calibration, resulting in an average improvement of 4.8%.

5.2 Effectiveness of Confidence Estimation

To evaluate the effectiveness of our proposed internal confidence estimation (ICE), we compare our method with the self-consistency-based confidence estimation method (SCE) (Yang et al., 2023; Xu et al., 2024). We randomly sample 50 data from three datasets and compare the confidence scores of the model’s responses from approaches above. We compute \( |C_{ICE} - C_{SCE}| \). The results are shown in Table 3. We observe that the confidence estimation bias between the two methods is small for the vast majority of samples. On the ScienceVQA dataset, the average difference in confidence scores between the two methods is 0.0578, indicating that for a given answer from the model, our method has only about 6 instances of deviation compared to the post-hoc confidence estimation method.

5.3 Effectiveness of Confidence Calibration

We further investigate the confidence calibration (MECE) and task performance (F1) of our method across different reasoning levels in MLLMs, specifically focusing on visual perception and cross-modal reasoning. The results is presented in Figure 4. Our method achieves a significant improvement in confidence calibration at the perception level (an increase of 20.4%), which contributes to a 38.5% improvement in the accuracy of the reasoning chain. Furthermore, at the reasoning level, benefiting from the strengthened knowledge boundary in the visual understanding stage, both the confidence calibration score and the reasoning chain F1 score show improvements, surpassing the strongest baseline by 19.7% and 27.4%, respectively.

![Bar chart showing Model Improvement Across Inference Levels for Perception and Reasoning](page_1012_186_410_312.png)

Figure 4: Performance improvement of strong baselines and our model compared to the base model in visual perception and cross-modal reasoning level of MLLMs. We report the results on ScienceVQA.

6 Related Work

Hallucinations and Uncertainty Estimation. In MLLMs, hallucinations refer to model responses that are misaligned with the visual modality (Chen et al., 2024; Liu et al., 2024c; Huang et al., 2025b), which can arise due to insufficient capabilities in visual perception and knowledge reasoning. Various efforts have been made to evaluate hallucination in MLLMs (Liu et al., 2024a; Gunjal et al., 2024; Zhang et al., 2024b; Wu et al., 2024; Xu et al., 2025). As a fundamental approach to detecting model hallucination, uncertainty estimation (UE) has long attracted significant attention,
falling into two main types: black-box and white-box. Black-box methods only require the generated text, and most of these methods are based on self-consistency (Fomicheva et al., 2020; Kuhn et al., 2023; Lin et al., 2023), inspired by its success in textual domain reasoning (Wang et al., 2025; He et al., 2024b; Jin et al., 2024). White-box methods rely on access to logits and internal layer outputs. They encompass information-based, density-based and sample diversity-based approaches (Malinin and Gales, 2020; Kadavath et al., 2022; Vazhentsev et al., 2023; Kuhn et al., 2023; Fadeeva et al., 2024; Duan et al., 2024). Instead of simply relying on self-consistency prompting, we leverage the internal state of model to quantify the confidence of each reasoning step.

Confidence Calibration of Language Model. Existing research has highlighted the tendency of LLMs to fabricate information when faced with unknown questions (Hu et al., 2023; Amayuelas et al., 2023; Liu et al., 2024b; Huang et al., 2025a; Fan et al., 2025). As a result, increasing attention has been directed toward enhancing the models’ awareness of their knowledge boundaries and enabling them to express their confidence in outputs when encountering uncertainty (Lin et al., 2022; Xiong et al., 2023; Yang et al., 2023; Lyu et al., 2024; Xu et al., 2024; Zhang et al., 2024a). Zhou et al. (2023a) empirically finds that injecting uncertainty expressions into prompts significantly increased the accuracy of GPT-3 responses and improved calibration scores. Zhang et al. (2024a) introduce R-tuning to encourage LLMs to express “certain/not certain”. Xu et al. (2024) goes further to teach the model to express more fine-grained confidence estimates along with self-reflective rationales. However, these methods focus solely on the entire response, which can lead to incorrect answers with high confidence. Therefore, we propose MMBoundary to train MLLMs to express fine-grained confidence estimates for each reasoning step, enhancing reasoning chain self-correction.

7 Conclusion

In this work, we present MMBoundary, a novel framework that advances the knowledge boundary awareness of multimodal models through reasoning step confidence calibration. We incorporate complementary textual and cross-modal self-rewarding signals to estimate confidence at each step of the MLLM reasoning process. In addition to supervised fine-tuning MLLM for initial confidence expression warm-up, we further introduce a reinforcement learning stage with multiple reward functions for further calibrating model confidence. Empirical results demonstrate that our framework significantly outperforms existing methods, achieving an average reduction of 7.5% in multimodal confidence calibration errors and up to 8.3% improvement in task performance.

Limitation

Our framework aims to enable MLLMs to autonomously generate natural language confidence statements during inference, enhancing reasoning chain self-correction. A limitation of this work is that our proposed method involves using the model’s internal states and uncertainty methods to assess the model’s confidence. However, more research is needed to determine whether uncertainty methods can accurately reflect the model’s confidence in its output. Ablation experiments on the uncertainty methods indicate that the four carefully selected methods provide gains for the model. Additionally, we explore the correlation between the proposed internal confidence estimation method and the self-consistency method. The results show that our metric, without requiring multiple samples, achieves performance comparable to methods that rely on multiple samples. To further advance our method, our future work will concentrate on the following areas: Firstly, we aim to incorporate multimodal self-correction mechanisms (He et al., 2024a), leveraging diverse data sources to augment the model’s capabilities. Secondly, we plan to explore synthetic data pre-training techniques (Qin et al., 2025) to address data scarcity and improve the model’s generalization ability across various project scenarios.

References

Alfonso Amayuelas, Kyle Wong, Liangming Pan, Wenhui Chen, and William Wang. 2023. Knowledge of knowledge: Exploring known-unknowns uncertainty with large language models. arXiv preprint arXiv:2305.13712.

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930.

Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jin-
jie Gu, and Huajun Chen. 2024. Unified hallucination detection for multimodal large language models. arXiv preprint arXiv:2402.03190.

Yangyi Chen, Lifan Yuan, Ganqu Cui, Zhiyuan Liu, and Heng Ji. 2022. A close look into the calibration of pre-trained language models. arXiv preprint arXiv:2211.00151.

Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kaikhura, and Kaidi Xu. 2024. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5050–5063.

Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, et al. 2024. Fact-checking the output of large language models via token-level uncertainty quantification. arXiv preprint arXiv:2403.04696.

Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al. 2023. Lm-polygraph: Uncertainty estimation for language models. arXiv preprint arXiv:2311.07383.

Zhiyuan Fan, Yumeng Wang, Sandeep Polisetty, and Yi R. Fung. 2025. Unveiling the lack of lvlm robustness to fundamental visual variations: Why and path forward. Preprint, arXiv:2504.16727.

Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. 2020. Unsupervised quality estimation for neural machine translation. Transactions of the Association for Computational Linguistics, 8:539–555.

Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356.

Tianrui Guan, Fuxiao Liu, Xiayang Wu, Ruqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. 2024. Hallucionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375–14385.

Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18135–18143.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR.

Jiayi He, Hehai Lin, Qingyun Wang, Yi Fung, and Heng Ji. 2024a. Self-correction is more than refinement: A learning framework for visual and language reasoning tasks. Preprint, arXiv:2410.04055.

Zhita He, Pengfei Cao, Yubo Chen, Kang Liu, Ruopeng Li, Mengshu Sun, and Jun Zhao. 2023. Lego: A multi-agent collaborative framework with role-playing and iterative feedback for causality explanation generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9142–9163.

Zhita He, Pengfei Cao, Chenhao Wang, Zhuoran Jin, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojiang Jiang, Kang Liu, and Jun Zhao. 2024b. Simucourt: Building judicial decision-making agents with real-world judgement documents. arXiv e-prints, pages arXiv–2403.

Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718.

Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujie Lu, and William Yang Wang. 2022. Wikiwhy: Answering and explaining cause-and-effect questions. arXiv preprint arXiv:2210.12152.

Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. 2023. Won’t get fooled again: Answering questions with false premises. arXiv preprint arXiv:2307.02394.

Junsheng Huang, Zhita He, Sandeep Polisetty, Qingyun Wang, and May Fung. 2025a. Mac-tuning: Lim multi-compositional problem reasoning with enhanced knowledge boundary awareness. Preprint, arXiv:2504.21773.

Kung-Hsiang Huang, Hou Pong Chan, Yi R Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, and Heng Ji. 2024a. From pixels to insights: A survey on automatic chart understanding in the era of large foundation models. IEEE Transactions on Knowledge and Data Engineering.

Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024b. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427.
Shijie Huang, Hongru Wang, Wanjun Zhong, Zhaochen Su, Jiazhan Feng, Bowen Cao, and Yi R. Fung. 2025b. Adactrl: Towards adaptive and controllable reasoning via difficulty-aware budgeting. Preprint, arXiv:2505.18822.

Wen Huang, Hongbin Liu, Minxin Guo, and Neil Zhenqiang Gong. 2024c. Visual hallucinations of multi-modal large language models. arXiv preprint arXiv:2402.14683.

Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint arXiv:2406.10890.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334.

Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187.

Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yaoob, and Lijuan Wang. 2024a. Mitigating hallucination in large multi-modal models via robust instruction tuning. Preprint, arXiv:2306.14565.

Genglin Liu, Xingyao Wang, Lifan Yuan, Yangyi Chen, and Hao Peng. 2024b. Examining llms’ uncertainty expression towards questions outside parametric knowledge. Preprint, arXiv:2311.09731.

Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024c. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024d. Llava-next: Improved reasoning, ocr, and world knowledge.

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521.

Qing Lyu, Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, and Chris Callison-Burch. 2024. Calibrating large language models with sample consistency. arXiv preprint arXiv:2402.13904.

Andrey Malinin and Mark Gales. 2020. Uncertainty estimation in autoregressive structured prediction. arXiv preprint arXiv:2002.07650.

Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd van Steenkiste, Lisa Anne Hendricks, Karolina Stańczak, and Aishwarya Agrawal. 2024. Benchmarking vision language models for cultural understanding. arXiv preprint arXiv:2407.10920.

Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang, Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, and Furu Wei. 2025. Scaling laws of synthetic data for language models. Preprint, arXiv:2503.19551.

Rafael Raffailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer.

Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Akim Tsvigun, Danil Vasilev, Rui Xing, Abdelrahman Boda Sadallah, Kirill Grishchenkov, Sergey Petrankov, et al. 2024. Benchmarking uncertainty quantification methods for large language models with lm-polygraph. arXiv preprint arXiv:2406.15627.

Artem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrankov, Danil Vasilev, Maxim Panov, Alexander Panchenko, and Artem Shelmanov. 2023. Efficient out-of-domain detection for sequence to sequence models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1430–1454.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
Yumeng Wang, Zhiyuan Fan, Qingyun Wang, May Fung, and Heng Ji. 2025. Calm: Unleashing the cross-lingual self-aligning ability of language model question answering. Preprint, arXiv:2501.18457.

Shujin Wu, Yi R Fung, Sha Li, Yixin Wan, Kai-Wei Chang, and Heng Ji. 2024. Macaroon: Training vision-language models to be your engaged partners. arXiv preprint arXiv:2406.14137.

Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. arXiv preprint arXiv:2210.04714.

Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063.

Tianyang Xu, Shujin Wu, Shizhe Diao, Xiaoze Liu, Xingyao Wang, Yangyi Chen, and Jing Gao. 2024. Sayself: Teaching llms to express confidence with self-reflective rationales. arXiv preprint arXiv:2405.20974.

Yao Xu, Mingyu Xu, Fangyu Lei, Wangtao Sun, Xiangrong Zeng, Bingning Wang, Guang Liu, Shizhu He, Jun Zhao, and Kang Liu. 2025. Amplify adjacent token differences: Enhancing long chain-of-thought reasoning with shift-ffn. Preprint, arXiv:2505.17153.

Yuqing Yang, Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. 2023. Alignment for honesty. arXiv preprint arXiv:2312.07000.

Hanning Zhang, Shizhe Diao, Yong Lin, Yi Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2024a. R-tuning: Instructing large language models to say ‘i don’t know’. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7106–7132.

Jianshu Zhang, Dongyu Yao, Renjie Pi, Paul Pu Liang, and Yi R. Fung. 2025. Vlm2-bench: A closer look at how well vlms implicitly link explicit matching visual cues. Preprint, arXiv:2502.12084.

Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R Fung, Jing Li, Manling Li, and Heng Ji. 2024b. Knowledge overshadowing causes amalgamated hallucination in large language models. arXiv preprint arXiv:2407.08039.

Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023a. Navigating the grey area: How expressions of uncertainty and overconfidence affect language models. arXiv preprint arXiv:2302.13439.

Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. 2023b. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.00754.

Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622.

A The Value-Statement Mapping Table

This module aims to establish a mutual mapping between the detected score and predefined confidence statements. We set the confidence levels to five categories (uncertain, slightly uncertain, moderately confident, highly confident, and fully confident), considering that having more levels might lead to overly similar confidence statements between adjacent levels. The confidence statements need to be directly integrable into the model’s generated content without sounding abrupt or redundant, much like human reflective expressions. We have preset 40 concise statements for each level. Table 4 presents additional confidence statements. These statements are concise and express the semantics of the corresponding confidence levels, allowing for seamless integration into sentences generated by the model, making them suitable for training generative language models.

B Implementation Details

Our experiment involves three distinct datasets: AOKVQA (Schwenk et al., 2022), ScienceVQA (Lu et al., 2022), and CulturalVQA (Nayak et al., 2024). The first two datasets are in-domain datasets, and our training data comes from the training sets of these two datasets, while CulturalVQA is an out-of-domain dataset. Since the test sets for all three datasets are not publicly available, we cannot accurately annotate the reasoning chain for MECE and Reasoning Chain F1 evaluation. Therefore, we use the validation sets of AOKVQA and ScienceVQA for in-domain testing of the model. For CulturalVQA, which only has a non-public test set, we manually selected and annotated 800 samples from it to serve as the test set.

For the construction of the warm-up dataset, we deploy the vLLM model with a temperature setting of 0.1 and number of log probabilities to return per output token of 10. We collect a total of 19K Question-Image pairs from the training sets of
<table>
  <tr>
    <th>Score</th>
    <th>Statement</th>
  </tr>
  <tr>
    <td>1</td>
    <td>but I can’t confirm this. / I’m uncertain about this. / I’m not sure about that. / This answer may be wrong. / I can’t guarantee this answer. / I’m unsure about this. / I can’t be sure about this. / This answer is unclear to me. / This might be imprecise. / This could be questionable: (…)</td>
  </tr>
  <tr>
    <td>2</td>
    <td>and it may need checking / it might not be right. / but I’m not sure. / and it might be slightly off. / though it’s not perfect. / but it may need confirmation. / though there’s some doubt. / though it may not hold up. / though I feel a bit unsure. / but there’s minor hesitation. (…)</td>
  </tr>
  <tr>
    <td>3</td>
    <td>but I can’t guarantee perfection. / I can’t be entirely sure / but it’s not beyond all doubt. / though minor errors might exist. / but it’s not fully certain. / though small flaws are possible. / but it’s not completely precise. / but it’s not entirely error-free. / though it’s not fully verified. / though it’s open to review.(…)</td>
  </tr>
  <tr>
    <td>4</td>
    <td>and this seems trustworthy. / and I believe this is right. / and I’m quite confident in this. / and this feels reliable to me. / and I trust this is correct. / and this seems very likely true. / and this appears reliable. / and this fits the context well. / and I’m confident this is right. / and this is well-reasoned. (…)</td>
  </tr>
  <tr>
    <td>5</td>
    <td>with total certainty. / with no doubts at all. / and I’m absolutely sure about this. / and I’m fully confident in this. / with total certainty. / and this is undoubtedly correct. / and this is entirely reliable. / and it’s unquestionably right. / with complete confidence. / and I guarantee this is right. (…)</td>
  </tr>
</table>

Table 4: The confidence score-statement mapping table. The five scores correspond to uncertain, slightly uncertain, moderately confident, highly confident, and fully confident. We preset 40 statements for each score.

A-OKVQA and ScienceVQA. For each Question-Image pair, we prompt the model to generate the reasoning chain and calculate the model’s confidence score for each sentence, resulting in 55K sentences with confidence statements, with \( w_0, w_1, w_2 \) and \( w_3 \) all set to 0.25 in internal confidence estimation module. \( \alpha, \beta, \gamma \) are equal. During the warm-up stage, we use the AdamW optimizer with a 10% warm-up ratio, a learning rate of 1.0e-4, and a batch size of 16. In the reinforcement learning phase, we randomly sample data from the training set for training, for each question, we sample \( N = 3 \), with a learning rate of 1e-5 and a batch size of 16.

C The Reasoning Chain Annotation

To obtain the necessary fine-grained knowledge of visual perception and cross-modal reasoning in visual question-answering for calibrating the multi-level confidence of MLLMs, we conduct reasoning chain annotation on knowledge-extensive datasets from three different domains.

C.1 The Annotation Pipeline

The pipeline of reasoning chain annotation is presented in Figure 5. We first prompt the GPT-4o to generate analysis (reasoning chain) structured in the perception and reasoning level. The former identifies key visual elements in the image that are most relevant to the question and answer, while the latter provides granularity reasoning that justifies why the answer is correct. Each level should include concise, interconnected sentences, with each sentence conveying a single piece of knowledge. As shown in the upper right corner of the figure, the initially obtained reasoning chain may contain redundant information and irrelevant content. Therefore, we use GPT-4o again to correct the content of the reasoning chain, filtering out redundancy and unrelated information to ensure that each sentence is concise and accurate. Then, we conduct annotation quality control to ensure the accuracy and consistency of the data. The prompt is provided in Appendix G.

C.2 Quality Evaluation

After the machine annotation is completed, we randomly selected 50 samples from each of the three datasets and asked two graduate students to evaluate the data quality. The evaluation metrics included: (1)Accurate: the reasoning chain is relevant to the question and contains no wrong knowledge; (2) Concise: each sentence is concise and contains no redundant information; (3) Complete: the reasoning chain formed by each sentence accurately explains the answer to the corresponding question without omitting relevant knowledge. We use a Likert Scale to evaluate each indicator, with a scoring range from 1 to 5, where 1 indicates ‘Strongly Disagree’ and 5 indicates ‘Strongly Agree.’ The results are shown in Table 5. We report the proportion of data with a rating greater than 4 (i.e., Agree). The results indicate that the majority of the data meet the three criteria. We conduct a Kappa test on the accuracy evaluation results of the two graduate students, yielding a Kappa value of
Question: Which city of Turkey is the origin of the performers depicted in the image?
Answer: Konya

<table>
  <tr>
    <th>Image level</th>
    <th></th>
  </tr>
  <tr>
    <td>1. The image shows a group of performers dressed in traditional Turkish attire, likely meant to evoke a sense of historical significance.</td>
    <td style="color:red">Redundant</td>
  </tr>
  <tr>
    <td>2. The performers are engaged in a traditional dance involving drums.</td>
    <td></td>
  </tr>
  <tr>
    <td>3. Turkish folk performers are often accompanied by traditional instruments like the saz, a long-necked stringed instrument.</td>
    <td></td>
  </tr>
  <tr>
    <th colspan="2">Reasoning level</th>
  </tr>
  <tr>
    <td>1. This dance style is associated with the Whirling Dervishes, known for their spinning movements as part of a meditative practice.</td>
    <td></td>
  </tr>
  <tr>
    <td>2. The Whirling Dervishes are followers of the Mevlevi Sufi order, which emphasizes music and dance as spiritual expressions.</td>
    <td></td>
  </tr>
  <tr>
    <td>3. The Mevlevi order originated in Konya, Turkey, which is renowned for its association with Rumi, the famed Sufi mystic and poet.</td>
    <td style="color:magenta">Irrelevant</td>
  </tr>
  <tr>
    <td>4. So the answer is Konya.</td>
    <td></td>
  </tr>
</table>

Figure 5: The Annotation Pipeline. We first prompt GPT-4o to generate an analysis (reasoning chain) structured at the perception and reasoning levels. Then, we have GPT-4o filter and correct the initially annotated chains. Finally, manual data quality control is conducted to ensure accuracy and reliability.

<table>
  <tr>
    <th>Metric</th>
    <th>Accurate</th>
    <th>Concise</th>
    <th>Complete</th>
  </tr>
  <tr>
    <td>Rate (%)</td>
    <td>96.8</td>
    <td>91.3</td>
    <td>93.5</td>
  </tr>
</table>

Table 5: The Likert Scale results of annotated data. We report the proportion of data with a rating greater than 4 (i.e., Agree).

0.75, which indicates a high level of consistency between the evaluators.

D The Details of Evaluation Metrics

D.1 The Multi-granular Expected Calibration Error (MECE)

As shown in Figure 6, after comparing the knowledge contained in the predictions and references, we obtain sentences where the knowledge in predictions and references aligns. Then, we calculate the Expected Calibration Error (ECE) for each sentence one by one, and finally derive the Multi-granular Expected Calibration Error (MECE):

\[
\mathrm{ECE}(a) = \frac{1}{|a|} \sum_{(z,c) \in a} |\mathbb{I}(z) - \mathrm{Conf}(c)| \tag{16}
\]

\[
\mathrm{MECE}(A) = \frac{1}{|A|} \sum_{a \in A} \mathrm{ECE}(a) \tag{17}
\]

Here, \( A \) represents the entire test set, and \( a \) denotes the reasoning chain generated by the model, which consists of multiple sentences. \( (z, c) \) represent a sentence and its corresponding confidence statement, respectively. \( \mathbb{I}(\cdot) \) is the indicator function that returns 1 if the sentence is correct when compared with the reference chain, and 0 otherwise. Conf(\(\cdot\)) represents the numerical value of the confidence statement.

D.2 Area Under the Receiver Operating Characteristic curve (AUROC)

We adopt the AUROC score (Hendrycks and Gimpel, 2016), which measures the ability of models to distinguish between correct and incorrect responses across different threshold settings.

\[
\mathrm{AUROC} = \int_0^1 \mathrm{TPR}(\mathrm{FPR}^{-1}(x)) \, dx \tag{18}
\]

where \( x \) denotes the threshold confidence level, \( TPR \) represents the true positive rate at this threshold, and \( FPR \) indicates the false positive rate cor-
Reference Chain | Predicted Chain | Step Matched | Step ECE
[R1] The image shows a group of performers in traditional Turkish attire | [S1] The image shows three men dancing with a drum and wearing traditional attire, with total certainty. | R2 | ECE(R2,S1)
[R2] The performers are engaged in a traditional dance involving drums | [S2] The setting appears to be a traditional venue, and it need checking. | R3 | ECE(R3,S3)
[R3] This dance style is associated with the Whirling Dervishes | [S3] Given the attire and dance style, it is likely associated with the Whirling Dervishes, but it’s not fully certain. | R6 | ECE(R6,S4)
[R4] Whirling Dervishes are followers of the Mevlevi Sufi order | [S4] So the performers are from Konya, but I’m not sure. |
[R5] The Mevlevi order originated in Konya, Turkey |
[R6] So the answer is Konya |

Figure 6: Example of MECE and Reasoning Chain F1 calculation.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">A-OKVQA</th>
    <th colspan="4">ScienceVQA</th>
    <th colspan="4">CulturalVQA</th>
  </tr>
  <tr>
    <th>ECE (↓)</th>
    <th>MECE (↓)</th>
    <th>Acc (↑)</th>
    <th>F1 (↑)</th>
    <th>ECE (↓)</th>
    <th>MECE (↓)</th>
    <th>Acc (↑)</th>
    <th>F1 (↑)</th>
    <th>ECE (↓)</th>
    <th>MECE (↓)</th>
    <th>Acc (↑)</th>
    <th>F1 (↑)</th>
  </tr>
  <tr>
    <td>Multisample</td>
    <td>0.437</td>
    <td>0.463</td>
    <td>0.665</td>
    <td>0.557</td>
    <td>0.467</td>
    <td>0.492</td>
    <td>0.613</td>
    <td>0.415</td>
    <td>0.443</td>
    <td>0.512</td>
    <td>0.335</td>
    <td>0.513</td>
  </tr>
  <tr>
    <td>SaySelf</td>
    <td>0.324</td>
    <td>0.381</td>
    <td>0.727</td>
    <td>0.632</td>
    <td>0.332</td>
    <td>0.445</td>
    <td>0.628</td>
    <td>0.523</td>
    <td>0.397</td>
    <td>0.426</td>
    <td>0.352</td>
    <td>0.586</td>
  </tr>
  <tr>
    <td>Conf-CSR</td>
    <td>0.413</td>
    <td>0.463</td>
    <td>0.774</td>
    <td>0.623</td>
    <td>0.433</td>
    <td>0.534</td>
    <td>0.702</td>
    <td>0.517</td>
    <td>0.482</td>
    <td>0.503</td>
    <td>0.394</td>
    <td>0.562</td>
  </tr>
  <tr>
    <td>RCE</td>
    <td>0.372</td>
    <td>0.427</td>
    <td>0.793</td>
    <td>0.647</td>
    <td>0.401</td>
    <td>0.483</td>
    <td>0.686</td>
    <td>0.504</td>
    <td>0.436</td>
    <td>0.475</td>
    <td>0.425</td>
    <td>0.597</td>
  </tr>
  <tr>
    <td>DRL</td>
    <td>0.406</td>
    <td>0.442</td>
    <td>0.762</td>
    <td>0.628</td>
    <td>0.435</td>
    <td>0.523</td>
    <td>0.641</td>
    <td>0.487</td>
    <td>0.465</td>
    <td>0.493</td>
    <td>0.384</td>
    <td>0.552</td>
  </tr>
  <tr>
    <td>MMBoundary</td>
    <td><b>0.395</b></td>
    <td><b>0.348</b></td>
    <td><b>0.806</b></td>
    <td><b>0.664</b></td>
    <td>0.346</td>
    <td><b>0.426</b></td>
    <td><b>0.713</b></td>
    <td><b>0.562</b></td>
    <td><b>0.354</b></td>
    <td><b>0.385</b></td>
    <td><b>0.437</b></td>
    <td><b>0.634</b></td>
  </tr>
</table>

Table 6: Experimental results on a different base model, Qwen2VL 7B (Wang et al., 2024).

responding to the threshold. The result is shown in Table 8.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">A-OKVQA</th>
    <th colspan="2">ScienceVQA</th>
  </tr>
  <tr>
    <th>ECE (↓)</th>
    <th>MECE (↓)</th>
    <th>ECE (↓)</th>
    <th>MECE (↓)</th>
  </tr>
  <tr>
    <td>MMBoundary</td>
    <td><b>0.316</b></td>
    <td><b>0.324</b></td>
    <td><b>0.354</b></td>
    <td><b>0.405</b></td>
  </tr>
  <tr>
    <td>w/ UIC</td>
    <td>0.358</td>
    <td>0.387</td>
    <td>0.406</td>
    <td>0.479</td>
  </tr>
</table>

Table 7: The comparative study of confidence level segmentation methods. UIC (uniform interval for confidence level segmentation) means simply divides the interval [0, 1] directly into five equal segments, with each segment corresponding to a confidence level.

D.3 Reasoning Chain F1 Score

We use the Reasoning Chain F1 score (Ho et al., 2022) to evaluate the quality of the reasoning chain generated by the model. We compare the knowledge contained in the predictions and references. First, we split the predicted and reference chains into “steps” by sentence. We then compute a matrix of pairwise similarity scores before using a threshold to classify “matches.” Since a single predicted sentence may contain multiple reference knowledge, we keep separate counts of precise predicted sentences and covered reference sentences. These

![Boxplots of human evaluation scores on the A-OKVQA dataset.](page_1012_1342_388_246.png)

Figure 7: Boxplots of human evaluation scores on the A-OKVQA dataset.
<table>
  <tr>
    <th>Model</th>
    <th>A-OKVQA</th>
    <th>Sci-VQA</th>
    <th>Cul-VQA</th>
  </tr>
  <tr>
    <td>Multisample</td>
    <td>0.5016</td>
    <td>0.5429</td>
    <td>0.4904</td>
  </tr>
  <tr>
    <td>SaySelf</td>
    <td><b>0.6872</b></td>
    <td>0.6118</td>
    <td>0.6261</td>
  </tr>
  <tr>
    <td>Conf-CSR</td>
    <td>0.5238</td>
    <td>0.5713</td>
    <td>0.4931</td>
  </tr>
  <tr>
    <td>RCE</td>
    <td>0.6037</td>
    <td>0.5902</td>
    <td>0.6059</td>
  </tr>
  <tr>
    <td>DRL</td>
    <td>0.4956</td>
    <td>0.5028</td>
    <td>0.4576</td>
  </tr>
  <tr>
    <td>MMBoundary</td>
    <td>0.6635</td>
    <td><b>0.6786</b></td>
    <td><b>0.7108</b></td>
  </tr>
</table>

Table 8: The AUROC experimental results.

counts are then micro-averaged to calculate the overall precision, recall, and F1 scores for the test set:

\[
\text{Precision} = \frac{\text{Matched}}{\text{Prediction}}, \quad \text{Recall} = \frac{\text{Covered}}{\text{Reference}}
\]

(19)

Taking the answer in Figure 6 as an example, we have: Prediction = 4, Reference = 6, Matched = 3, Covered = 3. We then calculate the F1 score:

\[
F1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}
\]

(20)

Drawing on the study of Ho et al. (2022), we select a large RoBERTa model (cross-encoder/stsb-roberta-large) with a similarity threshold of 0.64.

E Experiments of MMBoundary on Qwen

To prove that our method can generalize on multiple models, we also implement the baseline approaches and MMBoundary on Qwen2VL 7B (Wang et al., 2024).

F Analysis of Domain Shift

We select three widely used datasets from distinct domains (general, scientific, and cultural) for separate training and testing. This cross-domain design aims to validate the robustness of our approach in domain-agnostic scenarios. The experimental results in Table 1 and Table 6 demonstrate that: (1) Our confidence estimation method, which integrates textual and cross-modal signals based on the model’s internal uncertainty, is inherently domain-agnostic, thereby effectively mitigating domain shift effects; (2) Despite notable distributional differences between domains, our method maintains a mean expected calibration error (MECE) of 0.361 on out-of-domain data, surpassing other strong baselines by over 7%, demonstrating its good adaptability to domain shifts.

G Prompt

G.1 Prompt for Data Annotation

As illustrated in Figure 8, we present the prompt for data annotation. We first prompt the GPT-4o to generate analysis (reasoning chain) structured in the perception and reasoning level. The former identifies key visual elements in the image that are most relevant to the question and answer, while the latter provides granularity reasoning that justifies why the answer is correct.

G.2 Prompt for Data Refinement

The initially obtained reasoning chain may contain redundant information and irrelevant content. Therefore, we use GPT-4o again to correct the content of the reasoning chain, filtering out redundancy and unrelated information to ensure that each sentence is concise and accurate. As demonstrated in Figure 9, we present the refinement prompt designed to guide GPT-4o in filtering and correcting the initially annotated reasoning chains.
Prompt for Annotation

You will receive a question, an accompanying image, the correct answer, and the corresponding rationales. Follow these steps to generate your analysis (reasoning chain) structured in two levels.

Each level should include concise, interconnected sentences, with each sentence conveying a single piece of knowledge. Ensure the reasoning chain covers all necessary knowledge points concisely, with each sentence in this reasoning chain is essential and avoid adding redundant or irrelevant sentences.

The levels are as follows:
**Image level**: Identify key visual elements in the image that are mostly relevant to the question and answer. Format these sentences in JSON, like: ['sentence 1', ..., 'sentence i'].
**Reasoning level**: Based on the extracted visual elements, provide logical reasoning that justifies why the answer is correct. Format these in JSON as well: ['sentence i+1', 'sentence i+2', ..., 'So, the answer is ...'].

The sentences in both levels together should form a coherent chain of reasoning that clearly explains why the answer is correct. Ensure that each sentence builds upon the previous one to complete the reasoning chain. In the final sentence of the reasoning level, provide a clear conclusion with the answer, like: 'So, the answer is ...'.

Refer to the example below:
Please answer the following question:
Image: (Three people in traditional clothing holding drums, performing a form of the 'whirling dervishes' ritual.)
Question: Which city in Turkey is the origin of the performers depicted in the image?
Answer: Konya
Analysis: {'Image_level': [], 'Reasoning_level': []}
Your output:
Analysis: {
    'Image_level': [
        'The image shows a group of performers in traditional Turkish attire',
        'The performers are engaged in a traditional dance involving drums'
    ],
    'Reasoning_level': [
        'This dance style is associated with the Whirling Dervishes',
        'Whirling Dervishes are followers of the Mevlevi Sufi order',
        'The Mevlevi order originated in Konya, Turkey',
        'So the answer is Konya'
    ]
}

Now, please answer the following question:
Image: image
Question: {question}
Answer: {answer}
Analysis:{{{
    'Image_level': [],
    'Reasoning_level': []
}}}

Your output:
Analysis:
Prompt for data refinement

Now, the following analysis (reasoning chain) is structured.
Image_level and Reasoning_level together form a complete reasoning chain.
Please filter out any irrelevant sentences to maintain a concise reasoning chain, including only the essential sentences.

Refer to the example below:
Image: (Three people in traditional clothing holding drums, performing a form of the 'whirling dervishes' ritual.)
Question: Which city in Turkey is the origin of the performers depicted in the image?
Answer: Konya
Analysis:{{
    'Image_level': [
        'The image shows a group of performers dressed in traditional Turkish attire, likely meant to evoke a sense of historical significance.',
        'The performers are engaged in a traditional dance involving drums.',
        'Turkish folk performers are often accompanied by traditional instruments like the saz, a long-necked stringed instrument.'
    ],
    'Reasoning_level': [
        'This dance style is associated with the Whirling Dervishes, known for their spinning movements as part of a meditative practice.',
        'The Whirling Dervishes are followers of the Mevlevi Sufi order, which emphasizes music and dance as spiritual expressions.',
        'The Mevlevi order originated in Konya, Turkey, which is renowned for its association with Rumi, the famed Sufi mystic and poet.',
        'So the answer is Konya.'
    ]
}}

Your output:
Analysis:{{
    'Image_level': [
        'The image shows a group of performers in traditional Turkish attire .',
        'The performers are engaged in a traditional dance involving drums.'
    ],
    'Reasoning_level': [
        'This dance style is associated with the Whirling Dervishes.',
        'Whirling Dervishes are followers of the Mevlevi Sufi order.',
        'The Mevlevi order originated in Konya, Turkey.',
        'So the answer is Konya.'
    ]
}}

Now:
Image: {image}
Question: {question}
Answer: {answer}
Analysis: {analysis}
Your output:
Analysis:

Reviewer paper summaries: The paper introduces MMBoundary, which is a framework designed to enhance multimodal large language models' awareness of their knowledge boundaries by calibrating confidence at each reasoning step. MMBoundary incorporates textual and cross-modal self-rewarding signals to estimate confidence for individual reasoning steps to address the error propagation issue in multi-step reasoning chains. This framework involves two training stages: (1) supervised fine-tuning, which make the model generate natural language confidence statements, and (2) reinforcement learning, which aligns the model's confidence expressions with internal knowledge using carefully designed rewards. Experiments on datasets like A-OKVQA, ScienceVQA and CulturalVQA demonstrate significant improvements compared to previous baselines.
Summary of strengths: 1. This paper is very well-written. The motivation, the story, and the methods are very clear to me.
2. The step-level confidence calibration is a novel contribution compared to existing holistic approaches. 
3. Two-stage training pipeline balances initial confidence expression learning with fine-grained calibration.
4. The experimental results demonstrate the efficiency of their work.
Summary of weaknesses: 1. In the experiments, authors only conduct an ablation study of adding or removing rewards. However, they didn't conduct an ablation study about the rewards weights (e.g., $\alpha$, $\beta$, or $\gamma$ in Eq. 12. Also, no ablation study about $w_0$, $w_1$, $w_2$, and $w_3$ in Eq. 7.
2. The authors didn't explain why sometimes removing some rewards can be beneficial to some tasks, e.g., A-OKVQA, and ScienceVQA in Table 1.
3. How does MMBoundary handle conflicting signals between textual and cross-modal uncertainty estimates (e.g., high CLIPScore but low token entropy)? The paper does not discuss such edge cases.
Comments, suggestions and typos: 1. Although the paper mentions that Eq. 10 comes from previous paper. But it still lacks intuitive explanations. Why is the term $1-2 \cdot\left(\mathbb{I}\left(z_t\right)-\operatorname{EV}\left(c_t\right)\right)^2$ chosen for $R_{EC}$?
2. While CulturalVQA is labeled as "out-of-distribution," the paper does not analyze why MMBoundary performs better here or whether domain shifts affect calibration.
Overall assessment scores: 4
Metareview score: 3
Metareview: This paper propose a method called MMBoundary to increase the awareness of MLLMs to their boundary of knowledge, by having the model generate statements with confidence level represented in natural language at each stage of the reasoning chain. They show  that generating uncertainty estimates can improve performance on three benchmarks.
Summary of reasons to publish: The problem studied in this paper is interesting with a novel solution.