TFAR: A Training-Free Framework for Autonomous Reliable Reasoning in Visual Question Answering
Authors: Zhuo Zhi
Chen Feng
Adam Daneshmend
Mine Orlu
Andreas Demosthenous
Lu Yin
Da Li
Ziquan Liu
Miguel R. D. Rodrigues
Venue: Accepted by TMLR ()

Abstract: Recent approaches introduce chain-of-thought (CoT) reasoning to mitigate the challenges, such as hallucination and reasoning deficit in multimodal large language models (MLLMs) and enhance performance. However, existing CoT-based methods often rely on extensive data annotation and training. To overcome these limitations, we propose a training-free framework for autonomous and reliable reasoning (TFAR), which only uses common lightweight vision tools to improve the reasoning ability of MLLMs. TFAR enables an MLLM to autonomously and accurately identify relevant regions of interest (RoIs) and support CoT reasoning, without requiring additional training or annotations, and with low computational overhead during inference.  However, the use of external tools will introduce noise and uncertainty. To mitigate the uncertainty introduced by external tools and select the optimal pathway, we propose a conformal prediction-based uncertainty quantification method that calibrates the outputs from external tools and dynamically selects the most appropriate tool based on the MLLM’s output uncertainty. Experiments across five datasets demonstrate that TFAR improves performance over the base MLLM by an average of 4.6$\%$, in some cases even outperforming fine-tuned baselines, while maintaining low inference cost. These results offer new insights into training-free CoT guidance for MLLMs and underscore the value of reliable visual tools.

Published in Transactions on Machine Learning Research (08/2025)
TFAR: A Training-Free Framework for Autonomous Reliable Reasoning in Visual Question Answering
Zhuo Zhi
zhuo. @uul. uk
Deportment of Electronic and Electrionl Engineering
University College London
Chen Feng
Chen.feng@rull.c..om uk
Department Electronic and Electricol Engineering University College London
Adam Daneshmend
odam, adam..daneshneenddllss.cc
Imperial College Healtheare NHS Trust
Mine Orlu
m. m.ortu@wull.cccoom
UCL School of Pharmacy University College London
Andreas Demosthenous
a.demosthenousseer ucl. uk
Department of Electronic and Electrical Engineering University College London
Lu Yin
Lyin
School of Compuler Science and Electronic Engineering University of Surrey
Da Li
dali. www.enncc@@gmmillccomm com
Samsung Al Centre Combridge Queen Mary University of London
Ziquan Liu
ziquan.liu Www.ollll@@l...omm uk
School Electronic Engineering and Computer Science Queen Mary University of London
Miguel Rodrigues
.rrodriguss@ucl..cc. uk
Department of Electronic and Electrical Engineering Universily College London
Reviewed OpenReview: https: https://openrevieeu
openreview. net/ for/formm? id= cBAKeZN3jy
Abstract
Recent approaches introduce thain-of-thoughtt (CoT) reasoning mitigate the challenges, such as hallucination and reasoning deficit in multimodal large language models (MLLMs)) and enhance performance. However, existing CoT-based methods often rely on exten- sive data annotation and training. To overcome these limitations, we propose a training- free framework for autonomous and reliable reasoning (TFAR), which only uses common lightweight vision tools to improve the reasoning ability of MLLMs. TFAR enables an MLLM to autonomously and accurately identify relevant regions of interest (Rols) and sup- port CoT reasoning, without requiring additional training annotations, and with low computational overhead during inference. However, the use of external tools will introduce
"Corresponding author

---

Published in Transactions on Machine Learning Research (08/2025)
noise and uncertainty. To mitigate the uncertainty introduced by external tools and select the optimal pathway, we propose a conformal prediction-based uncertainly quantification method that calibrates the outputs from external tools and dynamically selects the most appropriate tool based on the MLLM's output uncertainty. Experiments across five datascts demonstrate that TFAR improves performance over the base ML.LM by an average of 4.6%, in some cases even outperforming fine-tuned baselines, while maintaining low inference cost. These results offer new insights into training-free CoT guidance for MLLMs and underscore the value of reliable visual tools.
1
Introduction
To address common challenges in multimodal large language models (MLLMs) such as hallucination (Wu
et al.
2025 and reasoning deficit (Zhang et al. 2023) recent studies have introduced chain-of-thought (CoT) reasoning to improve performance on tasks like visual question answering (VQA) (Shao et al. 2024
Xu et al. 2024 Zhi et al., 2025b). A central component of CoT in MLLMs is guiding the model from coarse- to-iine-grained visual understanding, gradually focusing attention on the most relevant image regions. This process reflects how humans typically interpret images: starting with broad overview and then zooming in specific details (Shao et al.) 2024). For instance, (Shao et al.) 2024) creates a dataset with bounding boxes highlighting question-relevant regions and designs two-stage VQA pipcline. An MLLM fine-tuned on this data first identifies key regions, then generates answers by integrating both local and global cues, as illustrated in Fig. [a]. Similarly, (Xu et al.) 2024) introduced the -Lava-O1-100K dataset, framing VQA at multiple granularities and employing stago-wise wisc beam search. While these models show improved reasoning, they rely heavily on manual annotations and additional training costs that scale with model size and complexity.
example, natural question arises: is it necessary to fine-tune MLLM to teach it to select regions of interest (RoIs), as shown in Fig. aaa Could we instead prompt an off-the-shelf MLLM to select Rols on its own, then combine these with the original image, and perform CoT reasoning without further training? To test this idea, we apply this naive two-round inference method using the setup from (Shao et al. 2024). without any additional tuning. As shown in Table 9 this approach fails to improve and often degrades overall accuracy. By analyzing the failure cases, we identify two key reasons: (1) Poor localization: the MLLM lacks a sense of scale, often misjudging image size, misplacing bounding boxes, including irrelevant background; (2) Lack of instruction following; since the model is not trained to interpret prompts that request Rol selection, it tends to ignore such instructions and produces unreliable results, even format errors often occur. False Rols arising from these factors can mislead the model's reasoning, ultimately resulting in incorrect answers. We show the representative failure examples in Fig.
Taking (Shao et al. 2024
To address these limitations, we propose a training-free approach. We find that using only common lightweight vision tools (i.e., object detectors and segmenters) to provide essential visual information (i.e., the object name with bbox or boundary) can enable the MLLM to autonomously and accurately select RoIs by considering object size, pasition, and semantic relationships. In addition, with this information, MLLM only needs to select the object ID instead of generating a bounding bax or boundary, thus avoiding problems such as format errors caused by inadequate instruction tuning. The selected regions are then combined with the full image to facilitate the final CoT reasoning by following the two-round reasoning in (Shao et al, 2024). Our approach promotes both deep visual understanding and precise RoT selection, without requiring any manual annotations or model retraining. A comparison with training-based methods is shown in Fig, [
However, using external tools introduces new challenge: uncertainty and noise. Additionally, when multiple vision tools are available, it becomes unclear which one to trust. To address these issues, we incorporate uncertainty quantification (UQ) to evaluate both external tools and the MLLM itself, improving the reliability of the overall framework through uncertainty calibration or comparison. For example, Fig. 2 shows a case where an object detector misleads the MLLM and our proposed method based on UQ.
We integrate all the proposed methods into framework named
Free Franework for Au-
tonomous Reliable Reasoning illustrated in Fig. 3 We follow (Shao et al.] 2024 to two-stage
2

---

Published in Transactions on Machine Learning Research (08/2025)
Training
Inference
Our training free method (a) Training- saininggbbasedaee based method
Q: What activity is puppy engaging in?
@: What is the dog doing
Reasoning Round 1 MLLM
Key area
Reasoning Round MLLM
@: Does the bird have blue crownand?
A: The dog playing with a ball.
@: What number is associated with the bus line?
@: What is the dog doing?
Reasoning Round I MLLM
Key area
Reasoning Round 2 MLLM
A: The dog playing with a hall.
NEED
Objectl: xx
bhox:
External
Object2: xx
tools
Figure 1: Comparison of training-based and our training-free methods for navigating CoT reasoning. (a) baiiing-based method (Shao et al. 2024): The MLLM explicitly trained to identify key regions for CoT reasoning. A common approach involves constructing a dataset in which each image annotated with regions relevant to the associated question, typically using bounding boxes. During training, the model learns to locate these regions, which are then extracted and used alongside the original image to support final-stage reasoning. While this strategy extensive manual annotation and incurs significant training costs. (b) Our training-free free method: Instead of training the MLLM to learn Rol selection, we only leverage lightweight vision tools to provide essential visual information (e.g,, object names and bounding boxes). The MLLM then autonomously selects Rols by reasoning over object size, position, and semantic relationships. These regions are combined with the original image to support final-stage CoT reasoning, without the need for manual annotations or fine-tuning.
enhance the reasoning performance of MLLMs, it requires
process. The first stage of TFAR uses fine-grained visual information from external models, calibrated using specially designed conformal prediction (CP)-based method (Vovk et al. 2005) for computational effi- ciency, to guide Rol selection. The MILM uses this information, together with the query and image, to select relevant regions autonomously. In the second stage, the MLLM performs CoT reasoning based on both the original image and selected Rols, To assess the reliability of this stage, we estimate uncertainty using a prediction set size-basod method that considers the number of plausible tokens in top-p sampling (Holtzman al. 2020). This uncertainty is further used select the most reliable vision tool for given reasoning task. Our contributions are summarized follows:
We identify and address two key limitations in existing CoT-based multimodal reasoning pipelines-the need for extensive annotations and model retraining- by introducing a training-free reasoning framework. In this framework, using only common weight visual tools to provide ba- sie information about the image enables MLLM to autonomously and accurately select Rols, which facilitates the final CoT reasoning.
We introduce conformal prodiction-inspired calibration strategy to quantify uncortainty from both external tools and MLLM outputs, ensuring that the two stages of MLLM inference are reliable. The strengths of the two UQ methods over existing UQ methods, such confidence scores and entropy, are validated in our empirical study.
We evaluate TFAR on five datascts, demonstrating that the performance MLLM is significantly improved with an average improvement of 4.6%, which is on par with some methods that require adnitional training and annotations, while maintaining affordable computational cost. These findings
3

---

Published in Transactions on Machine Learning Research (08/2025)
Q: "How many dogs are playing the ball?"
MILM
A: one (x)
(b) Unreliable tool use
A two (US=1.5)
MLLM
Calibration
A: one (US=2.3)
MLLM
MLLM
A: one (x)
(a) Zero-shot
(c) Our method based on UQ
Figure 2: An example illustrating how unreliable visual tools can mislead MLLM reasoning. US refers to the uncertainty score. (a) Zero-shot VQA by the MLLM: The MLLM alone fails to answer correctly in zero-shot mode due to limited visual recognition capabilitics. (b) MLLM with an unrelialle object detector: Incorporating external visual information with noise can lead the MLLM to incorrect conclusions. (c) Our proposed method based on UQ: We calibrate the outputs of external vision tools provide reliable visual cues and estimate the uncertainty of the MLLM's responses identify the most reliable tool, which leads to the correct answer.
highlight the effectiveness our proposed reasoning framework and the importance of incorporating visual tools reliably.
We review related work in See and describe the proposed TFAR framework in Sec Sec presents the experimental settings and results, and we conclude our work in Sec
2 Related work
Vision language models Vision language models integrate textual and visual representations to enable complex reasoning tasks and has always been the focus of much attention in the multimodal field (Zhang et al.] 2024: Zhi et al. 20244; 2025a)). Early approaches primarily focused on joint embedding stralegies and attention mechanisms, where images and text are projected into a shared feature space for downstream tasks such as visual question answering, image captioning, and cross-modal retrieval (Lu al. 2019 Kim
et al. 2021 With the advent of LLM (Achiam al. 2023 Touvron et al. 2023 researchers have explored
ways to merge massive textual encoders/ prooders/deodderss with robust vision encoders, aiming to leverage LLMss powerful linguistic reasoning while preserving rich visual features. One line of work investigates contrastive pre-training strategics, popularized by CLIP (Radford et al.) 2021). Another family of methods explores fusion architectures such as Flamingo (Alayrac et. al.) 22022 and BLIP-2 (Li et al. 2023) where vision encoders are tightly coupled to transformer-based language models to achieve few-shot or even zcro-shot performance on tasks ranging from image captioning visual grounding. More recent efforts push the boundaries by aligning large language models with visual embeddings, named MLLM, such LLaVA {Liu et al. 2024b), Qwen-VI.(Bai et al. 2023) and InternVI. (Chen et al. 2024). In our work, we mainly focus on incorporating training-fres methods to improve MLLM reasoning in the VQA task.
Shain-of-thoughh in visual reasoning. The success CoT in LLM because it significantly improves the model's reasoning ability. With CoT, instcad of answering questions directly, L.L simulates the reasoning of human beings and explicitly performs step-by-step reasoning to improve the performance in complex reasoning tasks (Wei et al. 2022 Zhang et al.] 2022 Many works have begun to try to apply CoT in
4

---

Published in Transactions on Machine Learning Research (08/2025)
Stagel
Stage2
Tool call and calibration
Reasoning: ROI selection
ROI extraction
Reasoning: Final answer generation
Calibmned "rame"
MULM
[0]
MILM
A wy:
"boele" "houndayy [194 258) d
The beer on
prompt no
prompy atsteer
the ground
We input an image into (Tool nome). The rature value [Toul miuure suiur).
Given the original
close to chair.
inage and the highllighedd arce. please think
Qui Where is the beer?
given the question. "Wllrre bear?" plaase analyoe the resilts stop by stop. Consider the semuatic.
2.23
step
question: Where to the
boar?" Please view the
location and relaziossipe between objects and how they relate to the question. Please finally provide me with a list containigg only the snigen of objects could help
osiginal image fins. focus
highlighedd saa and please retrieve more information from the oniginal image if
A*:
o
The beer is the chair next to the
needed;
X
f det
Calibrated Rn" 1'4" n. "nane""
answer the question.
(US - 1.5)
"scale", "bhox 1210. 271, 550, S100,
Pav: 1]
MLLM
Figure 3: Overview of the proposed TFAR framework. When answering this example directly, MLLM will give the wrong answer: 'The beer is the ground!. Instead, this figure illustrates how TFAR generates the correct answer through two-stage process. Stage 1: TFAR invokes external tools to obtain fine-grained information and applics CP-based calibration to thoir outputs to improve quality. This calibration mitigates issues such as the segmentation tool misclassifying many pixels as background and the object detection tool missing small objects. Based on the calibrated results, the MLLM autonomously selects RoIs through a reasoning process. Stage The selected Rols are extracted and combined with the original image as the MILM input to perform the final CoT reasoning. The best answer is chosen from all pathways using uncertainty estimation based on the prediction set size. TFAR does not require additional annotation model training. By relying solcly on common, lightweight vision tools to provide essential image information, the framework enables the MLLM to autonomously and accurately select Rols while maintaining efficient inference. Uncertainty quantification at both stages further ensures the reliability and robustness of the entire reasoning process.
MLLM to enhance visual reasoning capabilities. Z. Z et al. (Zhang et al., 2023) propose multimodal- CoT that incorporates language and vision modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. In (hao et al. 2024), the authors create a datasct with bounding boxes for quest relevant regions in the image and design a two-slage VQA pipeline. An MLLM fine-tuned on this dataset automatically locates regions of interest and provides accurate answers by performing CoT with the help of global and local image details. Similarly, (Xu et al.. 2024 introduces the LLavA-O1-100K dataset, which frames VQA at four levels of gramularity and applies stage-level beam search. The resulting A-AvA-01 model displays strong reasoning capabilities. In addition, reinforcement learning is also used to guide the model to adaptively focus on key image areas, thereby achieving efficient multimodal reasoning (Zhang et al.) 2025 Fan et al. 2025). However, these methods rely on extensive manual annotations and additional training, which pose practical challenges as MLLLMs grow larger. In our approach, we show that only involving lightweight vision tools can enable the MILM to autonomously and accurately seleet key regions, which facilitate the construction of the COT. This removes the need for manual annolations and extra training.
Uncertainty Quantification in MLLM The quantification of uncertainty in LLMs has gained significant atlention as a critical component for improving reliability and trustworthiness in AI systems (Xiong et al. 2023 Ye et al. 2025 Lin et al. 2023a). Some recent work has begun to extend to MLLM. In (Kostumov ct al. 2024), conformal prodiction is employed to quantify the uncortainty of 20+ MILMs and conclude that models with the highest accuracy may also have the highest uncertainty. However, they only focus the multiple-choice VQA task. To solve the hallucination of MLL.M under rare images, (Fang et. al.
5

---

Published in Transactions on Machine Learning Research (08/2025)
2024) proposes to use uncertainty-guideet token dropout to mitigate errors arising from visual token misin- terpretation. Unlike in (Groot & Valdenegro-Tori 2024), the verbalized uncertainty of MLLM is estimated via prompting and net calibration error is calculated to measure the direction of miscalibration. (Khan & Fu 2024) proposes using the principle of neighborhood consistency to identify unreliable responses from an MLLLM in QA lasks. Most of these approaches aim at black-box MLLM and do not consider uncertainly using the external tools. In contrast, our TFAR framework estimates the uncertainty of both external tools and the MLLM's outputs, leading to more trustworthy results.
3 Method
In this section, we present the proposed methodology in two main sections: the proposed TFAR framework and the uncertainty quantification approach for it.
3.1 The proposed TFAR framework
We find that by referencing the basic information from external vision tools, MLLM can autonomously and accurately select RoIs without additional learning process. These regions, combined with the original image, are subsequently fed into the MLLM. The MLLM then performs the final CoT reasoning, progressing from a coarse interpretation to fine-grained details, to generate the final answer. To ensure the reliability of both the vision models and the MLLM output, we utilize UQ for calibrating the tools and for selecting the final answer from multiple pathways. The overall framework is depicted in Fig. 3. For example, given an image To and a question Qo, the TFAR framework generates the final answer A by following four main steps:
Tool call and calibration. We call external tools to provide basic information of To, i.e., the object name and bbox/boouddee from the segmentation tool faeg and object detection tool faet by following popular agent-based methods (Zhou el al. 2024 Wang al.] 2024). Note that in prineiple, TFAR can use any number of tools; here we use two tools as examples and show the results of more tools in the ablation study. To ensure the reliability of these tools, we design the calibration algorithms C.ssg and CPaet based on conformal prediction. Thus, we get the calibrated output of tools by Reeg = Cesy (ss)((o)) and Ract = Caeet (ddee(())) Note that Rxeg and Ratt are in text format and objects are described using boundary point coordinates and bounding box, respectively.
Reasoning: Rol selection, Instead of selecting the key area manually (Shao et al. 2024). We leave it to the MLLM fMLLM itself to choose the area that may help to answer the question based on the tool output Reeg and Raet by using the prompt promptros: WWe imput an image into { (segmentation tool/object detection tool). The return value is RRseg/Rdet) Now, given the question Qo, please analyze the results step by step. Consider the semantic, location and size relationships between all objects and how they relate to the question. Pleose finally provide me with list containing only the * ** of the objects that could help answer the question.". In this process, MLLM reasons to fully explore the fine-grained information provided by tools and give the interested object id in the answer Oseg- = (.LLM(BRse) Qo, promlyo)) and Oaet = MMMIM M.I.M(BReee. (Raet, Qo, promptras:. Thanks to the basic information in Raeg/ the MLLM only needs to select the object ID instead of generating a bounding box or boundary. This simplifies the task and helps avoid issues such as format errors that often arisc from insufficient instruction tuning.
Rol extraction. Based on the object of interest obtained in the previous step, we extract the cor- responding region from the original image by froi Eseg (I, Oseg) and /rei = Eace(To, Odes), where Fiseg refers to the area extraction by pixel for segmentation result and Edet is the area extraction by bounding box for detection result.
Reasoning: Final answer generation. We follow Shao et al. 2024 to input the original image with the key area and guide the MLLM to perform CoT reasoning from coarse-grained to fino-grained by using the prompt promptanswer: 'Given the original image and the highlighted area, please think step by step to answer the question (Qa)- Please view the original image first, then focus on the
6

---

Published in Transactions on Machine Learning Research (08/2025)
highlighted area and please retrieve more information from the original image if needed." So far get the final answer for both pathways by Aseg = JMLLAS (To, Qo, T promppaaseeer)) and Adet fMLLLM (Io, Qu. ree promptansuer) In the end, we choose the more trustworthy answer the final answer A = ming (Aa) US is the proposed uncertainty score based on the quasi-conformal prediction that will be introduced in the next section.
Through the previous steps, TFAR provides a more trustworthy answer. We introduce the enpertainty-guuided approach for TFAR in the next section.
3.2 Uncertainty quantification approach for TFAR
External tools can introduce uncertainty, which is a major challenge for TFAR. To extract trustworthy information from these tools and to select the most reliable pathway, we perform uncertainty quantification on both the tool return values and the MLLM outputs, using CP. CP is distribution-Fi frreee uncertainty quantification framework that provides statistically valid prediction sets with guarantced coverage probability (Vovk al., 2005). We describe our uncertainty quantification methods for external tools and the MLLM outputs, respectively.
3.2.1 Calibration of visual models by conformal prediction
Given a pro-trained model and an allowable crror rate (or risk level) a, the inductive CP constructs prediction sets that contain the true label with probability least 1-a, a, where - a is also named as the confidence level. In classification, one chooses nonconformity score s(-) to measure how "unusual" each predicted label is (relative to the ground truth), then estimates threshold d. from a held-out colibration set. Formally, let {(x,, H denote calibration set of size n, where X; is an input sample and Y, is its true label. Given new, previously unseen test input Xtest and its (unknown) true label Yiest, conformal prediction produces sets
c(Xtest) = ( s(X 1) S )a} s.t. P(Yest E C((eest)) 2 1-a,
(1)
(2)
where Xtett denotes the Lest input and Yoss denotes its (unknown) ground-truth label. The resulting set C(Xreat is guaranteed to contain the true label with probability at least 1 - a. qa is the (1 - ququutlee of all calibration scores (x, 1;) and is calculated by
[(n 1) (1)))))
Φ- Quantile { (x K.). U (+20).
(3)
The extra {+0o} and the cciling term [(n + 1)(1 - =a]]/n ensure finite-sample coverage guarantees (An- gelopoulos & Bates 2021). The core principle of CP is that choosing the quantile as shown Equation mathematically ensures that the prediction set of labels on the test data contains the true label with probability 1 For classification model whose output is probability vector over classes, common
choice of nonconformity score is
(X,,Y,)
- p(Y: IX).
(4)
i.e. the probability of the model not predicting the true class. Below, we show how CP can be specialized to (a) a segmentation model (pixel-wise) and (b) an object-detection model boounding-box-wiie))
Calibration of a segmentation tool Segmentation models can be viewed grid of pixel-level classifiers, each outputting a probability distribution over K classes every pixel location. A frequent practical issuc is that many pixels belonging to foreground objects get labeled as background (class 0), causing under- segmentation (Xie et al.] 2021). We calibrate the result using a pixel-wise CP approach that allows the re-labeling of certain background pixels. We define the following notions,
X' be the space of images of dimension H x W;
y'= -0, 10, 1..... K) be the set of semantic labels (including background 0);

---

Published in Transactions on Machine Learning Research (08/2025)
(J), be the calibration set, where Yi(,, v) E 3 is the ground truth at pixel (u, u);
- W be the total number of pixels in the calibration set.
A segmentation model provides a probability vector Pi,a, E [0, 1]K+1 over classes at each pixel (u, v)
image X;. We define the pixel-wise nonconformity score:
s(i, u, v)
(1,
(5)
i.c., one mimus the prodicted probability that (u, v) its true class. Thus we can collect all nonconformity scores from the calibration set:
S { u.
i I....,,
H; 1, I....WW.
(6)
Its (1 quantile da (with sinite-sample correction) is
Qu. = Quantile s u (+oo), [(N 1)(1 - a)]
(7)
N For test image X Xtest, at each pixel (u, v)) we define the conformal prediction set.
C(u, v) {kEy px(u,,) S J
(8)
Thus Clu, u) is all classes whose wise nonconformity scores are no larger than da. We map C(u, v) to a single calibrated label Deal(u) (u, v) via:
arg max Py(u, v), jec(u.y)
if C(u, v) + 0, and & C(u, v), if (u, v) + 0, and C(u, v), if C(C, 0.
Deal (u. v)
arg max (, v), jEC(
(9)
If (background) appears in C(u, v), the pixel can be re-labeled to a foreground class within that set, mitigating over-conservative segmentation.
Calibration of object detection tool We apply bounding-box-wise conformal prediction (Andéol et al. 2023 to calibrate object detections. This addresses extreme cases where tiny objects may require expanding the predicted box lo guarantee covering the true object with high probability. We define the following notions: let
X' be the image space;
{ba) Ik=1 be the set of K ground-truth bounding boxes in image, where each box is given by bt = (attnn Vmnin wha vi
((X, (1)))) be the calibration set, with (b/} the ground-truth boxes of image X;-
A trained detector produces a set of predicted boxes 18) = (# Piin Flancs Hlax)) for image X,. We match predicted boxes to ground-truth boxes via an loU threshold Let M, be the set of matched pairs (6) ot) with (o, ( b/)) 2 For each matched pair, define the additive nonconformity score:
( ( * Prin Mmins
(10)
which measures the coordinate-wise prediction errors. The full set of nonconformity scores is collected across all matched pairs:
S s= U U
(11)
1= (8) B/)M:

---

Published in Transactions on Machine Learning Research (08/2025)
where [m[] is the mth coordinate error. To ensure coverage on all four coordinates, apply Bonferroni correction, splitting risk into a/4 for each coordinate. Let Sm be the mth-ccordinate errors across all matched pairs from all calibration images. Then for each m E (1,2,3,4),
[[|mm]] 1) (1-11000)) a/4)1
aa.m Quantile Sm U (+00),
(12)
[Sml
On test image Xest conformalize each predicted box & by expanding it to
C(b)) = &nin âa,1, Pnin - q0.
(13)
Nmax Ja,,, Pmax + qua]
This guarantees that if & matched to a true box 1², then b remains inside C(P) with probability at least 1-a.
3.2.2 Uncertainty estimation for MLLM output based on prediction set
By using the TFAR framework, answers Aseg and Adst are generated through two pathways. We propose an uncertainty quantification method based on a prediction set in the top-p sampling, corresponding to conformal prediction with fixed non-conformity score threshold. Thus, we call it a quasi-conformal pre- diction UQ for convenience, but note that the top-p sampling does not generate a prediction set with any coverage guarantee. This method measures the number of candidate tokens required to "cover' a predefined probability mass p at each generation step in top-p sampling. For an output sequence A {wi, W2,.... WT): where i; denotes the ith token, compute the uncertainty score in two steps:
(1) (2) pt Pi ...., (V)
Token probability sorting. For each token wa, let
represent its probability
distribution over the vocabulary, sorted in descending order (p:" 2 P;
(2)
P M)
Cumulative thresholding. We compute the minimal number of tokens k; needed exceed threshold p 1]:
k: min k
ΣΦ! ( 2 pPJ
(14)
Here, k; reflects the breadth of plausible alternatives for w;, A large k; indicates high uncertainty, while small one implies confidence (few tokens dominate the probability mass).
Sequence-level aggregation. The final uncertainty score (US) for A is achieved by calculating the mean k; across all tokens:
1 T
US(A)
k
(15)
By applying the proposed quasi-conformal prediction- based uncertainty estimation, we get the uncertainty score of two pathways US(g)))) and US(ed))). The answer with a smaller uncertainty score will be consid- ered as more trustworthy and thus output by the TEAR as the final answer A" -iinn US(AA)
4 Experiment
We first describe the experimental settings, then the results of our method and the baselines across five multinodal reasoning datasets, demonstrating the effectiveness of our approach.

---

Published in Transactions on Machine Learning Research (08/2025)
Table Comparison with baselines on all datasets.
Method
VQA2
VizWiz
GQA
Flickr30K
MMBench
Average
LLaVA-1.5-13B
80.0
53.6
63.3
62.3
67.7
63.4
LLaVA-ov-7B (base model)
79.0
53.6
64.8
61.1
72.3
66.2
SPHINX-11B
78.1 1
52.5
62.6
60.7
66.9
64.2
dicma-7BBVisCooT
77.4
54.8
61.6
67.1
67.3
65.6
TFAR (ours)
82.5
60.1
67.3
67.2
77.1
70.8
-LaVA-ov--B-ViiCool (upper bound)
83.3
57.2
69.6
69.9
76.6
71.3
4.1
Experimental setting
Baselines. We select the following strong baseline for comparison.
Base model: Www.--neVisioo--Oe model to build the TFAR pipeline due to its excellent performance and support for multiple imago imputs, which essential for our implementation.. We refer to this baseline as LLaVA-ov-7B.
(LLaVa-OV) (Li et al. 2024 We selected this base
LLaVA-1.5-13B Liu et al.
2024ah.
LLaVA-1.5-13B is a popular baseline for VQA tasks with an
increased parameter count.
Visual COT (VisCoT) (Shao et al. lunes an MLLM on a manually curated dataset, enabling it to aulomatically identify key areas in an image and facilitate CoT reasoning. We fine-tune the LLaVa-OV model on the VisCoT dataset following the procodure in (Shao al.) (024) and denote the resulting model as LLaVA-ov-7B- VisCoT. Additionally, also compare our method with the Meuna-TB-VisCoT model released in (Shao et al. 2024).
2024). VisCoT is state-of-the-art (SOTA) approach that fine-
SPHINX (Lin al., 2023b SPHINX enhances the MLLM's ability identify regions of interest and guide CoT reasoning by incorporating multiple visual encoders during training. The base model used in SPHINX is Llama2-13B.
Datasets. We select diverse set of VQA datasets to comprehensively evaluate proposed method:
VQA2 (Goyal et al., 2017]: large-scale benchmark for open-ended VQA lasks, featuring 107,391 question-anse answer pairs in the test set that cover diverse topics, including object recognition, counting, and commonsense reasoning.
VizWiz (Gurari et al. 2018): : real-world dataset collected from blind or low-vision users, capturing everyday scenarios. The open-ended questions frequently involve tasks such as text recognition, object identification, and scene understanding, with 8,000 question-answer pairs in the test split.
Flickr30K (Plummer et al. 2015 supports VQA with open-ended question-answer pairs that em- phasize fine-grained object attributes, actions, and contextual scene understanding, The test set comprises 1,546 question-anse annewr pairs.
GQA (Hudson & Manning 2019) provides 12,578 open-ended questions in the test set, with a focus on the relationships among objects, attributes, and spatial arrangements.
MMBench [Lu et al. 2024c]: includes 1,784 questions in the test set that span classification, captioning, and VQA tasks. features both multiple-choice and open-ended questions across diverse visual scenarios.
Metric. All dataset evaluation scripts use accuracy as a metric, more details are given in the Appendix B Tool selection. We choose popular lightweight tools: SEEM (Zuu al. 2023) for segmentation and Yolov11 JJocerr & Qiu 2024 for object detection. We also try other tools in the ablation study.
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table 2: Ablation of UQ approach in TFAR.
Method
VQA2
VizWiz
GQA
Flickr30K
MMBeuch
Average
base model
79.0
53.6
64.8
61.1
72.3
66.2
T-AR-seg- beaebeee
80.3
56.2
65.2
62.3
74.0
67.6
TFAR-det-base
80.7
57.3
65.9
63.4
74.7
68.4
TFA IPAR-seg-CPP
82.0
59.0
66.6
66.1
75.5
69.8
TFAR-det-CP
81.7
58.6
66.2
65.2
75.8
69.5
TFAR (ours)
82.5
60.1
67.3
67.2
77.1
70.8
TFAR setting We set 0.1 for both CPse, and C'dets, following prior work (Kumar et al. 2023 van der Laan & Alaa 2025) and show more results in the Appendix [ under different values of a. Empirically, we set the parameter to 0.9 for the prediction set-based uncertainty estimation and show more results in the Appendix C under different values of p. To compute the nonconformity scores for both tools, we use the COCO-2017 validation dataset (Lin al., 2014), which contains 5,000 images with annotations for both segmentation and object detection. We also try different calibration datasets in the ablation study. The time consumplion for calculating the nonconformity score on the calibration set is 35 minutes and 8 minutes for the segmentation and object detection tools, respectively. For each test image, the average time taken for calibration is about 34ms and 0.2ms, respectively. The hardware and software platforms involved in the experiment are given in the Appendix
4.2 Main result
We present the experimental results of our proposed TFAR method alongside baselines on all datasets in Table [ As shown in the table, TFAR substantially improves the base model's performance by 3.5%, 6.5%, 2.5%, 6.1%, and 4.8% across the five datasets, demonstrating its effectiveness. Notably, VizWiz exhibits the largest performance gain, likely because this dataset contains many blurred images, where TFAR's tool calibration plays a particularly important role. We show some calibrated results from the VizWiz dataset in Appendix C Furthermore, TEAR outperforms all competing methods except for the upper bound, including some methods with more parameters (o.g., I.IVVA-1.--3BB and SPHINX-13B), Remarkably, TFAR even surpasses the upper bound on VizWiz and MMBench, underlining the significance of reliable tool return values.
We show some representative examples in Fig. g to qualitatively analyze TFAR. Due to the limited space, We omit the interaction with MLLM in the figure and highlight the Rols selection and UQ process. In Q1, the MLLM provides an incorrect answer due to interference from the cup pattern. The segmentation tool especially after correction accurately separates the cat from the cup, which helps the MLLM reach the correct answer with high confidence. Although the detection tool correctly identifics both the cup and the cat, its bounding box is too broad, failing to eliminate the interfering information. In Q2, the MLLM is confused by the image and produces incorrect answer. In this case, both the segmentation and detection tools identify the dog; however, the segmentation tool also captures the person and refines the output through the calibration, which helps to better distinguish between the human and the dog and allows the MLLM to generate a more reliable answer. In Q3, MLLM fails to capture the bottle on the grass and gives an incomplete answer. The segmentation model and the detection model also face the same challenge. However, CP-based calibration enables the segmentation model to re-segment the bottle from the background, and the detection model successfully identifies this object. Consequently, MLLM focuses on the ball, the person, and the bottle, which leads to the correct answer. More visualizations are shown in the Appendix
4.3 Ablation study
We perform comprehensive ablation experiments to validate the effectiveness of the proposed approach.
Inference efficiency One potential concern with TEAR is inference efficiency, as involves external tool calls and uncertainty calibration. To assess this, compare the average inference time of TFAR with that
11

---

Published in Transactions on Machine Learning Research (08/2025)
Stagel
Stuge2
27: What is the color cat's nose?
answembble. (US 2.3)
GT: unanswerable. MLLM: Pink.
4: Yellow. (US=3.1)
Q2: What color shoe does the dug wear?
A The dog does not (US - 2.3)
A: The dog wear the white shoes. (US 2.9)
GT: The dog does not wear shoes.
MLLMM: Brown.
Q3: What are the
4: The man ball and knnck over a bettle. (US- 2.5)
possible hazards in
the picture?
4 could be hit by a hall and hit bottle. (US 222
GT: get by ball and hit bottle
MLLM: man get hit by ball.
Figure 4: Visualization of some results. GT refers to the ground truth of the answer. Due to the limited space, We do not show interaction with MLLM, focusing on the Rols selection and UQ process in Stage and Stage 2. Refer to Fig. 3 for more details the framework.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Table 3: Comparison of inference time consumption between LLaVA-ov-7B, TFAR and the LLaVA-ov-7B. VisCoT.
Method
VQA2
VizWiz
VAVo--v-
1.24s
1.33s
TFAR
2.42s
2.63s
VisCooo
2.15s
2.34s
Table 4: Ablation of UQ approach in TFAR.
Method
VQA2
VizWiz
GQA
Flickr30K
MMBench
Average
base model
79.0
53.6
64.8
61.1
72.3
66.2
TFAR-VOG
81.6
58.9
66.4
65.7
75.8
69.7
TFAR (ours)
82.5
60.1
67.3
67.2
77.1
70.8
of the best-performing baseline, LLa LLaVA-or-7B--ViCool and the base model, LLa LLaVA-ov-7B, on the VQA2 and VizWiz datasets. All evaluations are conducted with batch size of 1, and the average inference times across all samples are reported in Table 3 As shown in the table, TFAR's inference time is 12.6% and 12.4% slower than the bascline -LaVA-ov--B--isse overhead remains within an acceptable range, especially considering that -LaVA-ov-7B-VisCa approximately 20 hours fine-tuning with 438k question- question-answer answer pairs, which computationally expensive and resource-intensive. The modest overhcad introduced by TFAR primarily stems from the extended reasoning process in Stage 1, as demonstrated in Appendix C Nevertheless, the integration of lightweight visual
on VQA2 and VizWiz, respectively. This performance
requires
operating with millisecond-level latency incurs minimal computational cost and does not introduce noticeable runtime delays (detailed in Section 4.1 Furthermore, the conformal prediction- based calibration step adds negligible overhcad, as the inconsistency scores required for calibration are precomputed. It is worth noting, however, that the overall reasoning speed of both TFAR and VisCoT is substantially lower than that the base model LLaVA-ov-7B: This reduction is attributed to the two-round dialogue framework employed by both methods to enhance reasoning quality.
Different calibration dataset Another important consideration is the impact of the calibration dataset distribution on overall performance. To investigate this, evaluate TFAR using an alternative calibration set: the PASCAL VOC 2012 validation dataset (Everingham et al.), which contains 1449 images with both segmentation and detection annotations. We denote this variant as TFAR- VOC and compare its performance against the base model and the original TFAR (calibrated with the COCO 2017 validation set), as shown in Table Our results show that TEAR- VOC cxhibits a slight performance drop compared to TFAR, with an average decrease 1.1%. Nonetheless, still outperforms the base model by substantial margin of 3.5%. We attribute this performance gap to the smaller number of samples and more limited category diversity in the PASCAL VOC dataset compared to COCO 2017. Based these observations, recommend using the COCO 2017 validation dataset for calibration to achieve the best overall performance.
Different base models We evaluate the effectiveness of TFAR when integrated with various base models, including both open-source and commercial MLLMs. The open-source models include PPLUl- (7B) (Ye et al. 2024), and Qwen-VL (7B) (Bai et al. 2023). The commercial models include GPT-4o (Wang et al. 2024) and Gemini 2.5 Flash (Team et al. 2024). All of them support multi-image inputs. When combined with the TFAR framework, these are referred to as mPLUG-OWW2-TTAR, Qwen-VL-TFARR, GPT- 4o 4o-TFAR, and Gemini 2.5R respectively. Note that the commercial models do not provide access to raw probability outputs. To address this, we follow heuristic approach inspired by (Zhi et al. 2025b), allowing the LLM to produce its uncertainty score, in line with the concept of self-reftcction (Ji et al. 2023 Specifically, we appended the following instruction to the original prompl: "Note that you need to generate an uncertainty score for your answer, scaled from 1.0 to 5.0. The larger the value, the more certain you are."
We show the detailed prompt in Appendix B, 'Prompt for the commercial models in stage 2'. As shown Table 5 TFAR consistently improves the performance of all base models across five datasets, with average gains of 5.2%, 6.8%, 2.0%, and 1.3% for mPI.UG-Ow12, Qwen-VT., GPT-4o, and Gemini 2.5, respectively.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Table 5: Comparison of different base models in TFAR.
Method
VQA2
VizWiz
GQA
Flickr30K
MMBench
Average
mPLUG-Ow12
78.3
49.8
56.1
58.5
64.5
61.4
mPLUGG-OW2-TEAR
80.3
56.9
62.6
63.8
69.4
66.6
Qwen-VL
79.5
42.6
59.3
56.0
60.6
59.6
Qwen- VI-TFAR
81.9
54.8
65.7
63.2
66.2
66.4
GPT-4o
89.1
72.2
83.3
79.1
83.4
81.5
HOPT-4--TFAL
89.8
76.7
85.1
80.5
85.4
83.5
Gemini 2.5
91.4
75.3
86.2
84.1
86.9
84.8
Gemini 2.5-TFAR
91.9
78.8
86.9
85.3
87.7
86.1
ECE 0.069
ECE 0.073
ECE = 0.052
1.0
1.0
1.0
Perfect Calibration
Perfect Callbration
Perfect Calibration
0.8
Outout
Output Gap
Output Gap
Gap
accuray
0.6
0.6
0.6
0.4
0.4
0.4
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
confidence
confidence
confidence
(a) TFAR-seg-CP
(b) TFAR-det-CP
(c) TFAR
Figure 5: Reliability diagram of TEAR- seg-CP, TEAR-det-CP and TFAR GQA dataset.
These results demonstrate the generalizability of the proposed framework. Notably, the performance gains are more pronounced in open-source models. We attribute the smaller improvements observed commercial models their extensive pretraining on tasks such object detection and segmentation, which equips them with strong visual understanding capabilities, sometimes surpassing those of dedicated vision models. Nevertheless, the significant performance boost observed in smaller open-source models underscores TFAR's practicality and potential in wide range of application scenarios.
Ablation of UQ approach in TFAR We first compare the pipeline without using uncertainty to guide TFAR by disabling the calibration Stage 1 and the uncertainty score calculation in Stage This yields two pathways: TFAR-seg-base (segmentatin
tool) and TFAR-det-base (object detection tool). Next, we enable tool calibration in Stage 1 on top of TFAR-seg- and TFAR-det-base to explore the benefit of the method, rosulting in TFAR-scg-CP and TFAR-det-CP. The performances of these approaches are shown in Table 2 By comparing TFAR-seg-CH TFAR-det-CE and TFAR, it shows that using quasi-conformal prediction for answer selection improves the performance of single pathway, and TFAR outperforms any single-pathway method on all datasets. Compared to the strongest single pathway, TEAR achieves an average improvement of about 1% on five datasets. This suggests that different samples require different vision models and that the uncertainty the MI..M output can serve as a reliable metric for choosing the more appropriate tool.
We show the reliability diagram of TFAR-seg-CHP TFAR-det-CP, and TFAR on the GQA dataset in Fig. to compare the reliability of the answer. A unique scores of 0 and 1 is calculated for each answer in the GQA dataset. We normalize the uncertainty scores of all answers before subtracting them from one to obtain the confidence score. It shows that the expected calibration error (ECE) (Guo et al.) 2017) is reduced in TEAR, indicating the effectiveness of the proposed UQ for MLLM. Further comparison of TEA R-se-- base and TEAR-det- base with TFAR-seg-CG and TFAR-det-CI shows that Ca-based tool calibration yields an improvement of 2.2% and 1.1% for two pathways, respectively. We attribute the higher performance of TFAR-seg- CP its level-level ROI extraction, which excludes more irrelevant regions than box-level extraction and thus reduces the likelihood of introducing noise.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Table 6: Comparison of different ROI representation methods
Method
VQA2
VizWiz
GQA
Flickr30K
MMBench
Average
TFAR
82.5
60.1
67.3
67.2
77.1
70.8
TFAR-
82.0
60.3
66.9
66.8
77.1
70.6
Different UQ approaches in TFAR We compare different UQ approaches. For the external tools, we compare CP with a heuristic method. In the segmentation result, the confidence score for each object is obtained by averaging the softmax probabilities of all pixels belonging to it. In the detection model, it uses the confidence score returned by the model. We refer to these two pathways as T-AR-seg Beg--CS and TeAR-det- CS, respectively, and the overall approach as TFAR-CS. To incorporate the confidence score, we include it in the lool's return value (e.g., "id'": x2; 'name": 'xx', 'confidence score": " ax, "boundary': zI, ...... }') and add the text to promptrai: 'IIz, Consider the semantic, location, size and confidence relationships between all objects and how they relate to the question. TTx', For MLLM's output, we compare with uncertainty estimated by entropy:
U eentropy (A)
Σ
Σ j=1
b ps
logp," 63
(16)
T
We denote this approach as TFAR-US TFAR-USSeettopps where the tool calibration still uses CP. Table presents the results. From Table we see that heuristic-based uncertainty estimation for the tools underperforms the CP-based approach in TFAR, with drops of 1.6% for the segmentation pathway and 0.8% for the detection pathway. This effect is more pronounced in segmentation, likely because the heuristic struggles to handle overly conservative segmentation predictions. We also observe that using entropy for the MLLM output boosts performance by 0.4% compared to the strongest single pathway, but it still trails the quasi-conformal prediction approach by 0.6%. We attribute it to UQ by top-p coverage focusing on major contributors to the probability mass and aligns more closely with practical uncertainty perceptions, whereas entropy factors in the entire long-tail tail distribution and can artificially inflate perceived uncertainty in some cases.
More tools in TFAR In principle, TFAR can be extended to include arbitrary number of external tools. To explore the impact of incorporating additional tools, we integrate Mask2Former (Cheng et al, 2022) as a second segmentation model (denoted as TFAR-seg2) and Internlmage (Wang et al., 2023) as a second detection model (TFAR-det2). The extended version of the framework is referred to as TFAR+ Experimental results are presented in Table S We observe that both TFAR-seg2-CT and TFAR-de(2-CP perform comparably to their original counterparts. TFAR+ yields a modest performance improvement of 0.5%. This result suggests that the MLLM can alrcady identify accurate regions of interest using the essential visual information provided by the initial two tools, without requiring additional computational overhead. These findings highlight that TFAR achieves strong performance with a minimal toolset, reinforcing its efficiency and scalability for practical deployment. Therefore, considering the trade-off between performance and efficiency, we recommend using the original TFAR configuration with two tools.
Different ROI representation method in TEAR In TFAR, we mask out other objects that are not of interest. Here, we use different ROI representation methods where, after the MLLM selects objects of interest, we magnify those objects. Specifically, overlap among multiple objects of interest, input all magnified images separate images alongside the original image into the final MLLM. We refer to this approach as TFAR-zoomin, and evaluated it on all datasets. The results are shown in Table D Our results indicate that TFAR-zoomin performs slightly better than or comparable to TEAR on some datasets (e.g., VizWiz and MMBench), but overall, its performance is slightly lower than TFAR's. We hypothesize that magnifying objects in the middle of the image may disrupt the original spatial relationships, potentially misleading the MLLM's final judgment. Based on these findings, believe that the original TFAR settings, which maintain spatial consistency, remain preferable.
enlarge each object to the original image size. To avoid
15

---

Published in Transactions on Machine Learning Research (08/2025)
Table 7: Comprison of different UQ approaches.
Method
VQA2
VizWiz
GQA
Flickr30K
MMBench
Average
TeAR-seg-CE
82.0
59.0
66.6
66.1
75.5
69.8
TFAR-det-CP
81.7
58.6
66.2
65.2
75.8
69.5
TEAR-seg-CC
80.9
57.1
65.8
63.0
74.3
68.2
TFAR-det-CS
81.0
57.6
66.4
63.4
75.0
68.7
TEAR-CS
81.9
58.3
67.2
64.0
76.0
69.5
TFAR-US TEAR-US_entrrop
82.0
59.4
66.9
66.5
76.3
70.2
TEAR
82.5
60.1
67.3
67.2
77.1
70.8
Table 8: Performance of TFAR+ with 4 tools
Method
VQA2
VizWiz
GQA
Flickr30K
MMBench
Average
TFAR-seg-CP
82.0
59.0
66.6
66.1
75.5
69.8
TFAR-det-CP
81.7
58.6
66.2
65.2
75.8
69.5
TFAR-seg2-CH
81.5
58.6
66.0
65.7
75.0
TFAR-det2-CP
81.9
59.1
66.5
65.6
76.7
70.0
TFAR
82.5
60.1
67.3
67.2
77.1
70.8
TFAR+
83.2
60.4
67.6
67.8
77.4
71.3
5
Conclusion
In this work, introduced TFAR, training-free framework for autonomous and reliable reasoning in the visual question answering (VQA) task. Specifically, we demonstrate that using only common, wight vision tools to extract essential image information enables multimodal large language models (MLLMs) to autonomously and accurately select regions of interest (RoIs), thereby facilitating effective thoughtf-thought (CoT) reasoning. This approach eliminates the need for manual annotation and model fine-tuning. To enhance the reliability of the framework, we incorporate uncertainty quantification (UQ) based on conformal prediction (CP) at both stages of the reasoning process. The CP-based UQ not only improves the robustness of external visual tool outputs but also guides the selection of the most credible reasoning pathway. Extensive experiments five diverse datasets- VQA2, VizWiz, Flickr30K, GQA, and MMBench demonstrate that TFAR consistently improves performance over the base MLLM, with an average gain of 4.6%. In some cases, TFAR even out uulprfforms methods that require additional ional training. Ablation studies on inference efficiency show that TFAR achieves this performance improvement with only a 12.5% increase inference time over the best training-based bascline, offering a favorable tradc-off that avoids the need for dozens of hours of model training and data annotation. Additional ablations confirm the effectiveness of our CP-based UQ stralegy, which outperforms conventional heuristic-based approaches. In future work, plan to extend TFAR to support additional modalities and tasks beyond VQA. We also consider training TFAR to enable the MLLM to autonomously select more suitable tools for a given task, thereby reducing the inference burden.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXim:2303.08721 087744 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 23736, 2022.
Téo Andéol, Thomas Fel, Florence De Grancey, and Luca Mossina. Confident object detection via conformal prediction and conformal risk control: an application to railway signaling. In Conformal and Probobilistic Prediction with Applications, pp. 36-55. PMLR, 2023.
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 2021.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprinl arXiv:2308.1100 2023.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.0522 2024.
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. 2022.
M. Everingham, L. Van Gool, C. K. 1. Williams, J. Winn, and A. Zisserman. PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. sewwork.rgg/chaleengess /eaee/nnge////////////// VOC/vvc2012/ hwrp workshoppindex..ueg
The
http://www. wwtp://lw.ppacall...
Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching- Chen Kuo, Yuting Zheng, Sravana Jyothi Naraya- naraju, Xinze Guan, and Xin Eric Wang.
Grit: Teaching milms to think with images. arXiv preprint
arXiv:2505.156 15879, 2025.
Yixiong Fang, Ziran Yang, Zhaorun Chen, Zhuokai Zhao, and Jiawei Zhou. From uncertainty to trust: Enhancing reliability in vision-language models with uncertainty-guided dropout decoding. arXiv preprint arXiv:2412.06/17 2024.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Tobias Groot and Matias Valdenegro- Toro. Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models. arXiv preprint arXiv:2405.000) 2024.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwia grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-3617, 2018.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yojin Choi. The curious case of noural text degener- alion. In International Conference on Learning Representations, 2020. URL forum?id=re iddryygyyyy
net.
Drew A Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700-6709, 2019.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating 1mm hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1827-1843, 2023.
Glenn Jocher and Jing Qiu. Ultralytics yolo11, 2024. URL https://gittubb.oo//lttrtttttttt ultralytics
Zaid Khan and Yun Fu. Consistency and uncertainty: Identifying unreliable responses from black-box vision- language models for selective visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10854 10863, 2024.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languaae transformer without convolution or region supervision. In International conference on machine learning, pp. 5583 5594. PMTR, 2021.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Vasily Kostumov, Bulat Nutfullin, Oleg Pilipenko, and Eugene Ilyushin. Uncertainty-aware evaluation for vision-language models. Xrviv preprinl or arxiv:242. 14418, 2024.
Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam. Conformal prediction with large language models for multi-choice question answering, arXiv preprint ar (rXi::200..1 18404, 2023.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arxiv preprint or arXiv:2408.0033 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19745 PMLR, 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pp. 740 755. Springer, 2014.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint Xiv: 11105.191111 19187, 2023a.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qin, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07576 2023b.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, Tn Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 26296- 26306, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, al. Mmbench: Is your multi-modal model an all-around player? In European conference computer vision, pp. 216-233. Springer, 2024c.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representation for vision-and-languuage lasks. Advances in neural information processing systems, 32, 2019.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrasc correspondences for richer imago-to-sentence models. In Proceedings of the IEEE international conference on compuler vision, pp. 2641 2641-26499, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PmLR, 2021.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Unleashing shain-of-thoughe reasoning in multi-modal language models, arXiv preprint arXiv:2403.16999 2024.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXio preprint XXxx:4433.00550 2024.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. orXiv preprint arXim:2302.1399 2023.
Lars van der Laan and Ahmed M Alaa. Sclf-calibrating conformal prediction. Advances in Neural Information Processing Systems, 37:107138 0777331177770,, 2025.
Vladimir Vork, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, vol- ume 29. Springer, 2005.
Hao Wang, Jiayou Qin, Ashish Bastola, Xiwen Chen, John Suchanck, Zihao Gong, and Abolfazl Razi. Visiongpl: Llm-assisted real-time anomaly detection for sale visual navigation. Xin preprinl arXiv:2403. 0..122409.22415 12415, 15, 2024.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowci Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognation, pp. 14408 14419, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thoughl prompting elicits reasoning in large language models. Advances in neural in information processing systems, 0122424-24883 2022.
Qiong Wu, Xiangcong Yang, Yiyi Zhou, Chenxin Fang, Baiyang Song, Xiaoshuai Sun, and Rongrong Ji. Grounded chain-of-thought for multimodal large language models. arXiv preprint or Xiv:25003 12799, 2025.
Eaze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in neural information processing systems, 34:12077 12090, 2021.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can 1lms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.1300 2023.
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-ste step.... arXiv preprint 11. 2024.
Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek Wong, Emine Yilmaz, Shuming Shi, and Zhaoopnn Tu. Benchmarking Iims via uncertainty quantification. Advances in Newral Information Processing Systems, 37:15366-15385, 2025.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. owlu: Revolutionizing multi-modal large language model with modality collaboration. In Proceed- ings of the ieee/cuf conference on computer vision and pattern recognition, pp. 13040-13051, 2024.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-languaae models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Xintong Zhang, Zhi Gao, Bofei Zhang, Pengxiang Li, Xiaowen Zhang, Yang Liu, Tao Yuan, Yuwci Wu, Yunde Jia, Song-Chur Zhu, et al. Chain-of-focus: Adaptive visual search and zooming for multimodal reasoning via rl. arXie preprint 1xXb:22505114336 2025.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting large language models. arXiv preprint orXiv:2210.03./ 2022.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. ar preprint urXi::2302.00922 2023.
19

---

Published in Transactions on Machine Learning Research (08/2025)
Zhuo Zhi, Ziquan Liu, Moe Elbadawi, Adam Daneshmend, Mine Orlu, Abdul Basit, Andreas Demosthenous, and Miguel Rodrigues. Borrowing treasures from neighbors: In-context learning for multimodal learning with missing modalities and data scarcity. arXiv preprint arXiv:2403.0990 09428, 2024.
Zhuo Zhi, Yuxuan Sun, et al. Wasserstein modality alignment makes your multimodal transformer robust. Transactions Machine Learning Research, 2025a.
Zhuo Zhi, Qiangqiang Wu, Wenbo Li, Yinchuan Li, Kun Shao, Kaiwen Zhou, et Videoagent2: Enhancing the Ilm-based agent system for long-form video understanding by uncortainty-aware cot. arXiv preprint arXiv:25040444 04471, 2025b.
Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang, Image-of-though prompting for visual reasoning refinement multimodal large language models. arXiv preprint 1ki::2405.18872 2024.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 18-19799-19782 2023.
20

---

Published in Transactions on Machine Learning Research (08/2025)
Appendix
A
Method
We summarize the proposed TFAR framework in Algorithm
Algorithm The Proposed TFAR Framework
Input: Image To, Question Qo, external tools faegs Sant, pre-trained MLLM JMLLM, CP-based calibration methods (CPseg: Cddtt,, the method for UQ of MLLM output US, pixel-based image extraction Eseg and bounding box-based image extraction Eaet-
Stage 1: Tool Call and Calibration
1: Call segmentation tool: Reeg & Ceseg (Feg(())())))
2: Call detection tool: Rast + CP deded
3: Select Rol via MLLM-CoT:
Oseg -- faSLLad (Reeg, Qu, promptroi),
Odet + fMLLMM (Rdets Qo: promptroi)
4: Extract Rol images:
1 seg
Esegl (Ta, Oseg), Mot = Eaae Odet)
Stage 2: Final Answer Generation and Selection
Perform final reasoning with CoT:
Aseg~ JMLLM(UO (Ja, Qo, Troi pronplanswer) Adot SASLLAS (Jo, Qo, Taot promplanswer)
Compute uncertainty scores:
US US(ee))); US = US(Ade))
Choose the final answer with minimal uncertainty:
A" = arg
min
US(AAs)
A ALE(Acc):Aa))
return Final Answer: A
B
Experimental Settings
Metric calculation method All datasets involved in the paper use accuracy as an evaluation metric and calculate it as follows:
VQA2. Fach question in VQA2 is associated with 10 human-provided answers. For a predicted answer, the score is defined as: score = min(1, (number of matching human answers) / 3), which means if at least annotators agree with the predicted answer, receives full credit (a score of 1); fewer matches yield partial credit. The overall accuracy is the average of these per-question scores across the dataset.
VizWiz. Similar VQA2, VizWiz collects multiple answers per question (typically reflecting the diversity in responses from blind users). It uses essentially the same formula as VQA2 comparing the predicted answer against the set of human responses using the min(1, count/3) rule. The per- question scores averaged to produce the final accuracy score.
GQA. For each question, the answer is deemed correct it exactly matches the ground truth provided by the dataset. The primary measure the percentage of questions answered correctly (i.e., the

---

Published in Transactions on Machine Learning Research (08/2025)
mumber of correct answers divided by the total number of questions). Beyond simple accuracy, GQA papers also explore secondary metrics (such as consistency, validity, and reasoning subtasks), but the headline metric remains overall accuracy.
Flickr30K. The evaluation of answers in Flickr30K follows (Shao et al. 2024 Reference answers are provided, and predicted answers evaluated against these references using a GPTT-basdd scoring method. The final performance metric is computed by averaging the scores across all samples.
MMbench. MMBench is designed evaluate multimodal models over a rango of tasks. For each question task, answers are scored on a scale (oflen normalized between 0 and 1 or and 100) based on criteria like correctness, completeness, and relevance. The scoring can be performed using human evaluations, automated assessments (e.g., leveraging language models as evaluators), or a combination of both. Each answer receives a score reflecting how well it meets the task requirements. These per-task (or per-question) scores are then averaged to yield composite performance metric that summarizes the model's overall capability across the diverse tasks included in MMBench.
Prompt for the commercial models in stage The commercial models do not provide access to raw probability outputs. To address this, we follow a heuristic approach inspired by (Zhi et al. 2025b), allowing the LLM to produce its uncertainty score, in line with the concept of refeectin {Ji et al. 2023 We also added JSON template to facilitate parsing answers and uncertainty scores from MLLM responses. The
complete prompt is follows:
Given the original image and the highlighted area, please think step by step to generate the answer [ANSWER] of the question Q. Please view the
original image first then focus on the highlighted area and please
retrieve more information from the original image needed.
Additionally, you need to give me [UNCERTAINTY SCORE] that indicates
your uncertainty in answering the question accurately, on scale from
1.0 to 5.0 and accurate to one decimal place. The larger the value, the more certain you are. YOU MUST output in the JSON format:
ANSSEER' : [ANSWER]
UNNERTAINTY SCORE': [UNCERTAINTY SCORE]
Experimental platform All the experiments run on NVIDIA L40 GPU with CUDA 12.1.
C More experimental results
Directly let MLLM select RoI and reason with the original image We explore not training the MLLM but letting it directly select the region of interest and then perform VQA by following the two-round reasoning framework of VisCoT(Shao et al.] 2024). In reasoning round 1, we use the prompt (abbreviation) Please think step by step and give the bboz of the area in this picture that may help answer the question, and answer the format of [æmin, ymin, æmaz, ymaz). If there are multiple regions, please provide multiple bbores.". In reasoning round we extract the area according to the bbox in the previous answer and combine it with the original image for final answer generation by using the prompt "Given the original image and the highlighted orea, please think step by step to answer the question. Please view the original image first, then focus on the highlighted area, and please retrieve more information from the original image if needed'. We use VLaVA--neVisioon Qweno--b as the base model (because the model can handle multiple image inputs). The method of fine-tuning using the VisCoT dataset to facilitate localization of Rols is called LLaVAAov. 7B- Vi.C.o... We name our approach as CAVVA-o--B--ViCoT---- Flickr30K datasets, using accuracy as the metric. The results are shown in Table g As Table D shows, the results not only fail to improve but often degrade overall accuracy. By analyzing the failure cases, we
We test all methods on the VQA2 and
2

---

Published in Transactions on Machine Learning Research (08/2025)
Q: What is the dog doing? ?
Q: What is the dog doing? ?
MLLM
Reasoning for Rol selection
MELM
The interested area is
The interested area [27, 27
[27, 27, 161, 100]
Unable to parse
(1) Poor localization
(2) Lack of instruction
Figure 6: Failure cases of prompting MLLMs directly for RoI selection. (1) Poor localization. The MLLM lacks an understanding of absolute scale. It often misjudges the image size, resulting in bounding boxes that either miss the target or include excessive background. (2) Lack of instruction tuning for region selection. Since the model was not trained with prompts focused on identifying key regions, it tends to treat such instructions as secondary. This leads to unreliable region proposals and frequent format errors.
summarize two reasons: (1) Poor localisation. The MLLM has no sense of absolute scale. It often guesses the image size incorrectly, draws bounding boxes that miss the target, or includes too much background. (2) No instruction tuning for "find the key region." Because the model was never trained on prompts that emphasise region selection, it treats the request as a side note and delivers unreliable RoIs, and even format errors often occur. False Rols arising from these factors can mislead the model's reasoning, ultimately resulting in incorrect answers. We show the representative failure examples in Fig.
Table 9: Comparison with baselines on all datasets.
Method
VQA2
Flickr30K
LLaVA-ov-77 7BB (base model)
79.0
61.1
78.4
59.2
LLaVA-ov-7B-VisCoo
83.3
69.9
Visualization of more results We show more representative examples in Fig In Q1, the MLLM fails to distinguish between the bowling ball and the person's 's head, leading to incorrect answer. Initially, the segmentation tool identifies only portion of the person's 's region. However, after applying CP, it successfully segments both the bowling ball's and the head's regions. This improvement enables the model to capture the head's details and mitigate the bowling ball's interference, ultimately guiding the MLLM to the correct answer, whereas the detection model is unable to resolve the issue. In Q2, note the airplane toy includes small captain toy on its front a detail that the MLLM initially missed. Although neither the segmentation the detection model recognized this feature at first, the calibrated detection tool eventually identified object in that place, thereby preventing the model from focusing the little girl, which assists the MLLM in arriving at the correct answer. In Q3, the MLLM failed to distinguish between the human and the dog, resulting in an incorrect answer. The segmentation tool only segments the dog while a large area the image is labeled as background. In contrast, the detection tool identifies both the human and the dog, though its detection frame lacked precision. After CP calibration, the detection tool's output changed little, but the segmentation model's calibrated results successfully separated the human and the dog, enabling the MLLM to reach the correct answer with high confidence. In addition, combining Fig. 4
3

---

Published in Transactions on Machine Learning Research (08/2025)
Stagel
Stage2
Q1: What color man's hair?
4 Black. (US - 2.5)
4: Blue. (US - 2.9)
GT: Black. MLLM: Blue.
Q2: What color is the hand of the captain of the toy plane?
A: White (US- - 1.9)
person
person
A: Yellow. (US - 1.3)
GT: Yellow. MILM: White.
Q3: What is in the image?
A A dog and a person. (US 2.8)
4: dog and person. (US - 3.7)
GT: A dog and a
person.
MILLM: dog.
Figure 7: Visualization of more results.
and Fig. 7 we observe that the segmentation model performs belter with overlapping largets, whereas the bounding boxes provided by the detection model struggle accurately separate these targets. Different and in TFAR framework We evaluate the effects of parameters and p in the TFAR framework by conducting experiments with varying settings. Specifically, we examine values at 0.05, 0.1, 0.15, and 0.2, and p values at 0.8, 0.85, 0.9, and 0.95. The performance results are summarized in Table TO and Table From Table I0 we observe that extreme values of a negatively impact the performance. Specifically, very small & values impose stringent error-rate constraints, resulting in wider, overly conservative prediction intervals that lack granularity and informativeness. Conversely, large a values permit higher error rates, yiclding narrower, more precise intervals but significantly increasing the likelihood that the truc value falls outside the interval. Table I1 indicates that TFAR performance remains relatively stable when p is set to 0.95. However, reducing to 0.85 or 0.8 leads to noticeable performance degradation. This decline likely occurs because a lower threshold prematurely discards predictions with closely matched probabilities, thereby truncating the probability distribution and misrepresenting the actual uncertainty. Based on our analysis, we recommend selecting a = 0.1 and = 0.9 for optimal performance of TEAR. Comparison of reasoning length between TFAR and VisCoT To help readers better understand the difference in inference processes between TFAR and VisCoT, we provide the average number of output tokens in the two reasoning rounds the VQA2 dataset, shown in Table 12 The primary difference
4

---

Published in Transactions on Machine Learning Research (08/2025)
Table 10:
The performance of TFAR with different a.
VQA2
VizWiz
GQA
Flickr30K
MMBench
Average
0.05
82.1
59.5
67.0
66.6
77.0
70.4
0.1 ( (used paper)
82.5
60.1
67.3
67.2
77.1
70.8
0.15
82.3
59.8
66.8
66.9
76.5
70.5
0.2
82.0
59.2
66.6
66.5
76.2
70.1
Table 11:
The performance of TFAR with different, p.
P
VQA2
VizWiz
GQA
Flickr30K
MMBench
Average
0.8
81.8
59.0
66.5
65.9
76.2
69.9
0.85
82.0
59.3
66.8
66.6
76.7
70.3
0.9 (used in paper)
82.5
60.1
67.3
67.2
77.1
70.8
0.95
82.5
59.8
67.0
67.0
77.0
70.7
in CoT Chain-of-Thoughtt length between TFAR and VisCoT occurs in the first stage. We illustrate this using the example in Fig. [ In stage 1, TFAR explicitly prompts the MLLM to consider the semantics, size, and spatial relationships of all detected/ segmented objects in the current image and select the index of the object of interest. In contrast, VisCoT has been trained on millions of examples to rapidly locate regions of interest, resulting a shorter CoT in the first round. This difference explains why TFAR is slightly less efficient during inference. In stage 2-when generating the final answer based the original image and the selected region of interest the CoT lengths for both methods are comparable, as they receive similar image inpuls and prompts.
Visualization of detection failure and detection error of tools. Based on our analysis of the exper- imental results, we observe that when a tool fails to detect any relevant information, is often due to the image being inherently complex, blurry, or ambiguous. such cases, the MLLM's own ability to understand the image is also limited. These failure cases can generally be categorized into two types: (1) Detection failure where the tool does not return any useful information. In this situation, we fall back to using the original image as the input region of interest for the MLLM, essentially bypassing the tool. Whether the MLLM can still provide correct answer depends entirely on its own visual reasoning capability. For example, in Fig. g the object detector failed to identify any objects, yet the MLLM was still able to answer correctly. (2) Detection error - where the tool returns incorrect or misleading information.
most of these cases, the MLLM is negatively influenced by the tool's output, highlighting the importance of tool correction. While we provide examples of such failure cases in Fig. 4 and
2 we also include a success case in which the MILM correctly rejects the tool's faulty output Fig. I0 However, such outcomes are rare. When the tool produces incorrect results, it often indicates that the image particularly challenging, making it difficult for the ML.LM to compensate for the error on its own.
Some calibrated results from VizWiz dataset. The VizWiz dataset simulates images captured from the perspective of visually impaired individuals and primarily contains blurry or low-quality images. MLLMs generally exhibit limited robustness when processing such inputs directly. In TFAR, dedicated object detec- tion and segmentation tools can effectively assist in identifying these challenging regions, thereby improving overall accuracy. To better illustrate these results, we show some results here. As shown in Fig. 11 the VizWiz dataset is characterized by frequent motion blur and poor image quality, which present significant
Table 12: Comparison of output token number in two stages between VisCoT and TEAR
Method
Stagel
Stage2
TFAR
132
115
VisCoT
132
5

---

Published in Transactions on Machine Learning Research (08/2025)
Stagel
Stage2
Gives the orginal imuze and the highiighned cees, placee duik step by step to ieswer the coestion Q Please view onginal image fins, than flous on highlighted retieve mure infernation Boo theorigini ('aeeded.
Please boandinghos cocedisato the the help anower the quostion bener.
igital small standing neaching
field
(a) VisCoT
Q: What is the dog playing?
andpyeese
playtti
The highlighest shows clasp- of back and white wabor hall boing soucheal by the dog's This helpe confirn what insenading with, Looking, back the well image, the dog's possune and foess the hall saggost in ie satively playing Sox the dag ie playing with suuper bull.
The intenanaal anak (0,30,000 (.5700990,00000 0.10, 0..00.0.0.15)
MLLM
MILM
We desocue: The nutam walse
the ebject
The imsae hes two detecied objecti
Given the oeginal imuse and the habiisted tres, plaase thikk aep by step to asswer the questoa Pleas view the original image fiss, then nous on highlighted area andppienee resiese more indemation ifacoded.
What the dog playing?
deg and spens bull queetion sdos, "What playingh snower both the object typos and posiions Semeetically. spans commonity
(Too ietumn valuel. Now, given the question Q please asabyre the resuts sup hystep. Consider the somarrtic loomion and size rebitoondpps bawcer all obyraze and how they releic to the quection. Messe finally pronde me with a lis contaning only the aeigeted the objecta that could helpanswer quequettoon
dog
Bused
informatios, The puppy. which
(b) TEAR
play, especialle
indicating is The highlighod prosidio clossey stew this ubjon. The objon clearly ball with distncttvve black and white pozeem This penees is univesslly recogrized that of soccer bell. Combening thee abservation, the dog is playnn with soczer buil on the grawe.
(0.54, 0.44,
object dctccttrr
MLLM
0.60,
1AT Day Mkem
located the
other
be with. cansid both the mensing sputial relaionskip, the sparts ball is likely what the dog playing will. Answai:1].. LT).
Figure 8: An example illustrating the difference between TFAR and VisCoT in terms of two-stage inference length.
stagel
stage2
Q: How many mailboxes the picture?
Given the original image and the highlighted area, please think step by stop answer the question
original image first, then fbeus on the highlighted retreve mare information from the original image nendeed..
andplease
MLLM
There are three mailboxes the
object detector
Figure 9: An example of tool detection failure.

---

Published in Transactions on Machine Learning Research (08/2025)
Stagel
Stage2
Q: What's the water?
We input an image into the object
Given the original image and the highlighted
desector. The reem value is (Toel valse), the question please unalyze results step bystep. Consider the semantic. location and size relationstips
please think step by step to answer the question Q Please view the original image first, that foeus highlighted retrieve more infiemation from the original image F'neeldd
image
jumping
over the water. has cutly and
andplease
is mid-air with its legs stretced cut. There splach behind it, suggesting just through
objects
they question. Please finally provide me with lat containing only the off
water. The object detection result ways it is bear, but this looks like dog. The dog appears playful, possibly chasing something or just enjoynng the water. The scene is outdoons, with the above the water surfsce during the jump.
object dctector
that could help answer question.
MELM
active
SES THERDS Bon. Wooos 50220555540333
MILM
Answer: (0).
Figure 10: An example of tool detection wrong.
pottle e
bottle
Chair
(a) Example
(b) Example 2
Figure 11: Some calibrated results from the VizWiz dataset. Images with blue bounding boxes indicate the outputs from the object detection tool, while red bounding boxes represent the calibrated results after applying our correction method.
challenges for machine vision systems. For instance, in Fig. 11 [a); only small part of bottle was detected by the tool, but proposed conformal prediction method successfully corrected the bounding box to fully cover the object. Similarly, in Fig, []),,, the object detection tool failed to detect the chair and produced an incorrect bounding box for the TV. With the help of our correction process, the chair was accurately identified, and a more reasonable bounding box was assigned to the TV.


Reviews:
[{"review_id": "R0qxdVtD4F", "reviewer": "Reviewer_2GXx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The proposed calibration strategy addresses a key roadblock -- that of making the performance of the individual vision tools in the reasoning framework of MLLM-based VQA reliable -- and thereby makes training-free VQA for MLLMs viable.\n\n2. While demonstrated for VQA, the proposed framework and calibration strategy can be potentially useful for a wide variety of MLLM tasks, such as captioning, summarization, re-identification, and so on.\n\n3. The experimental results are presented in a well-balanced manner, incorporating relevant variants of the proposed implementation of the framework and the calibration strategy, and also clearly delineating the trade-offs in inference efficiency of the proposed training-free approach.\n\n## Weaknesses\n\n1. Sec. 3.2.1 is somewhat hard to follow, and some aspects of the conformal prediction approach can be made clearer:\n\n  - introducing $\\alpha$ before mentioning $1 - \\alpha$,\n\n  - describing what $X_{test}$ and $Y_{test}$ are in Eqns. 1 and 2,\n\n  - instead of repeating Eqn. 4 in words (\"one minus ... \"), providing more insight, such as \"probability of the model *not* predicting the true class\",\n\n  - clarifying the core idea that choosing the quantile as per Eqn. 3 mathematically ensures that the prediction set of labels on the test data contains the true label with probability $1 - \\alpha$.\n\n2. Based on Eqn. 9, each pixel label is determined greedily based on the probability vector associated with only that pixel and does not consider the labels of neighboring pixels. How does this approach ensure that the detected segments continuously span the underlying objects? For example, a segment for one object may erroneously contain within it segments of another semantically similar object due to some image or detection noise. Wouldn't such artifacts make the segments unusable for MLLM reasoning?\n\n3. Additionally, why is a non-background label always forced for segmentation? Could this not lead to overcorrecting and detecting false foreground segments?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a training-free framework for MLLMs to improve reasoning capabilities for the VQA task. To achieve this, the authors leverage available vision tools, such as scene segmentation and object detection, to obtain regions of interest (RoIs) in the queried images, which they incorporate in the context of MLLMs to enhance reasoning capabilities and improve VQA performance. To ensure the reliability of the vision tools in providing the RoIs, the authors propose a calibration strategy using the conformal prediction for each vision tool in the framework. They provide a specific implementation of their framework, incorporating segmentation and object detection as two vision tools, and using the COCO dataset as the calibration dataset for conformal prediction-based calibration of both tools. Through quantitative experiments, qualitative comparisons, and ablation studies, the authors demonstrate the VQA performance of their proposed implementation of the framework as well as its relevant variants.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please include details to address the weaknesses mentioned above.\n\n2. Citations in the main text intermix with the text and do not appear to be appropriately punctuated, and impede readability. My recommendation is to use [] for indirect citations in the main text, for example: Sec. 1, Line 1: \"such as hallucination Wu et al. (2025)\" -> \"such as hallucination [Wu et al. (2025)]\".\n\n3. In Sec. 4.2, to support the claim that \"VizWiz exhibits the largest performance gain, likely because this dataset contains many blurred images\", it would be helpful to at least show qualitative examples of improved performance on blurry images.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qLMAwHJGGB", "reviewer": "Reviewer_V6XM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n**(S1)** The proposed method replaces MLLM fine-tuning with tool calling, so it becomes training free. This is a desirable property for many reasons, including efficiency. Furthermore, we generally want to change large models as little as possible, as this might affect performance in other domains.\n\n**(S2)** While the method is mainly designed and evaluated on VQA tasks, the paper clearly points out that TFAR can be extended to other tools. It also appears possible to integrate the method into interactive settings, where models use additional information about images to provide their answer.\n\n**(S3)** Overall, the study is technically solid, and most relevant details are explained in appropriate depth. Results show that TFAR provides benefits, and motivates further research on tool calling as part of reasoning. The paper furthermore motivates research on improving performance of external tools in such scenarios.\n\n## Weaknesses\n\n**(W1)** The main weakness seems to be the reliance on providing two input images and the necessity to perform a separate inference run for each tool considered. This decreases performance and multiple images as input may not be supported by all models.\n\n**(W2)** It is unclear if the discussion on inference efficiency considers the two inference runs required by the proposed method to select the correct tools. More details on where TFAR and VisCoT differ in terms of output length would help understand this better. Furthermore, the inference speed should also be compared to base models with a single inference run, to understand the overhead better.\n\n**(W3)** An interesting aspect to consider is to let the MLLM decide which tool in wants to invoke, if any tool at all. This could potentially decrease inference latency considerably, if the model is confident it knows the correct answer with any tools.\n\n**(W4)** It is unclear how Uncertainty Estimation works for API models such as Gemini-2.5 and GPT-4o, which do not give access to raw probabilities. This should be explained in the paper.\n\n**(W5)** The evaluated open models are outdated at this point. While it would have been great to see experiments on more recent models, at the rate new models are released currently, this may be of minor importance.\n\n**(W6)** One missing aspect is failure cases where the bounding box detection model or segmentation model does not detect the correct image area at all. It would be interesting to see if the MLLM can reject any tool output, labeling it as insufficient on its own.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a training-free method called TFAR to combine the outputs of bounding box detection tools and image segmentation tools with MLLM Chain-of-Thought reasoning to improve the performance in VQA datasets. First, tools generate segmentation masks or bounding boxes, which are then improved by uncertainty estimation techniques. The outputs are then provided to the MLLM, which selects the appropriate bounding boxes or segmentation masks. These are used to create a second image that only shows the selected regions, and the MLLM uses both images to provide a final answer. Finally, again using uncertainty scores, the answer from the MLLM using bounding boxes or the answer from the MLLM using segmentation masks is selected.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**(R1)** Formatting: Currently, all citations are using `\\citet`, however in many cases, `\\citep` is appropriate. These have to be fixed.\n\n**(R2)** Explain how uncertainty estimation is applied to API models such as GPT-4o that do not give access to raw probabilities.\n\n**(R3)** Expand the discussion on inference efficiency: How do the output lengths of TFAR and VisCoT differ? What is the impact of 2 required inference runs of TFAR?\n\n**(R4)** Provide a few examples of failure cases. To which parts in the pipeline can they be traced (tool output, MLLM region selection, final MLLM reasoning)? Where is the greatest potential for improvement?", "broader_impact_concerns": "This paper does not have a Broader Impact Statement. However, while ethical implications such as bias compounding through the pipeline are conceivable, this work does not target any specifically sensitive applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GApHuusMfV", "reviewer": "Reviewer_694L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper explains everything in a simple manner with examples.\n- Gives thorough theoretical explanation of calculating uncertainty for the tools and MLLM output.\n- Provides thorough experiments with all possible types of ablations to show the effectiveness of the proposed method.\n- The results show improvement over base models and over some fine-tuned models as well.\n\nWeaknesses:\n- The inference time is greater for TFAR than fine-tuned models due to tool calling and uncertainty calibration.\n- The ablation study for TFAR+ doesn't use any new visual tools.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a training free framework for reasoning in Visual QA tasks. The paper introduces:\n\n- Using common lightweight visual tools to provide basic information about the image.\n- Introduces conformal prediction-inspired calibration strategy to quantify uncertainty from both external tools and MLLM outputs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The citations in section 2.2 use non-standard style. Like V.K. et al. Kostumov et al. (2024).\n- For the ablation involving more tools (TFAR+), the authors use the same tools: Segmentation and Detection. But this ablation should include complementary visual tools like zooming in which can provide more information to the MLLM. Using the same tool just makes it redundant.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)