Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models
Authors: Qingni Wang
Tiantian Geng
Zhiyuan Wang
Teng Wang
Bo Fu
Feng Zheng
Venue: ICLR 2025 Spotlight (2025)
Keywords: generative models
calibration/uncertainty
inference methods
Primary area: generative models
TL;DR: We propose a general framework for risk control and assessment, which creates rigorous prediction sets with statistical guarantees calibrated by two user-specified risk levels, applicable to any MLLMs supporting sampling in open-ended settings.

Abstract: Multimodal Large Language Models (MLLMs) exhibit promising advancements across various tasks, yet they still encounter significant trustworthiness issues. Prior studies apply Split Conformal Prediction (SCP) in language modeling to construct prediction sets with statistical guarantees. However, these methods typically rely on internal model logits or are restricted to multiple-choice settings, which hampers their generalizability and adaptability in dynamic, open-ended environments. In this paper, we introduce *TRON*, a **t**wo-step framework for **r**isk c**o**ntrol and assessme**n**t, applicable to any MLLM that supports sampling in both open-ended and closed-ended scenarios. *TRON* comprises two main components: (1) a novel conformal score to **sample** response sets of minimum size, and (2) a nonconformity score to **identify** high-quality responses based on self-consistency theory, controlling the error rates by two specific risk levels. Furthermore, we investigate semantic redundancy in prediction sets within open-ended contexts for the first time, leading to a promising evaluation metric for MLLMs based on average set size. Our comprehensive experiments across four Video Question-Answering (VideoQA) datasets utilizing eight MLLMs show that *TRON* achieves desired error rates bounded by two user-specified risk levels. Additionally, deduplicated prediction sets maintain adaptiveness while being more efficient and stable for risk assessment under different risk levels.

Published as a conference paper at ICLR 2025
SAMPLE THEN IDENTIFY: A GENERAL FRAMEWORK FOR RISK CONTROL AND ASSESSMENT IN MULTI- MODAL LARGE LANGUAGE MODELS
Qingni Wang', Tiantian Geng2.s, Zhiyuan Wang', Teng Wang2.1, Bo Fu'; Feng Zheng?*
University of Electronic Science and Technology of China
Southern University of Science and Technology
"University of Birmingham
"The University of Hong Kong
(gingni1031) genctiant yhzywang, ttengwang @ 1. com
ABSTRACT
Multimodal Large Language Models (MLLMs) exhibit promising advancements across various tasks, yet they still encounter significant trustworthiness issues. Prior studies apply Split Conformal Prediction (SCP) in language modeling to construct prediction sets with statistical guarantees. However, these methods typ- ically rely on internal model logits or are restricted to multiple-choice settings, which hampers their generalizability and adaptability in dynamic, open-ended vironments. In this paper, we introduce TRON, a two-step framework for risk control and assessment, applicable any MLLM that supports sampling in both open-ended and closed-ended scenarios. TRON comprises two main components: (1) novel conformal score to sample response sets of minimum size, and (2) a nonconformity score to identify high-quality responses based on self-consistency theory, controlling eтror rates by two specific risk levels. Furthermore, we investigate semantic redundancy in prediction sets within open-ended contexts for the first time, leading to a promising evaluation metric for MLLMs based on average set size. Our comprehensive experiments across four Video Question- Answering (VideoQA) datasets utilizing eight MLLMs show that TRON achieves desired error rates bounded by user-spedified risk levels. Additionally, dedu- plicated prediction sets maintain adaptiveness while being more efficient and sta- ble for risk assessment under different risk levels.
INTRODUCTION
In recent years, there has been swift progress in the development of Large Language Models (LLMs) and Multimodal Large Language Models (MILMs) (Zhao et al. 2023 Wu al.) 2023). MILMS extend the capabilities of L.MMs by integrating and processing information from multiple modalities, including text, vision, and audio (Bai et al., 2024 Zhu et al. 2024). However, MLLMs are proven to exhibit significant drawbacks in trustworthiness, such hallucination (Rawte al. 2023 Zhang ctt al. 2024 Wang et al. 2024a), which results non non-factur factual information (Ji et al. 2023) and biased generations (Zou al. 2023). These issues have elicited increasing societal concerns regarding the reliable deployment of foundation models in consumer-faaee -aacuunn--acing applications QQia et al. 2024).
Uncertainty estimation (Ye et al., 2024) provides valuable insights into the trustworthiness of model
generations. Prior work performs probabilistic modeling (Li et al. 2024), develops entropy-based measures (Wang et al.) 2024c] Duan al. [2024] |Farquhar al. 2024). and generates multiple sponses to analyze the output space (Lin et |024| Wang et al. 2024b). However, these approaches cannot provide guarantees of the error rate, and forcing measure to predict a single data point is overly restrictive (Cresswell et al. 2024). Split Conformal Prediction (SCP) (Papadopoulos et al., 2002 Angelopoulos & Bates 2021 Campos et al. 2024) is promising candidate to tackle these
"Corresponding co-authors: fubo@uuettc.duu.. f.zhen@@iece.org

---

Published as a conference paper at ICLR 2025
Calibration
Sampled nesponses
Frequency
Prediction set
Step 1:
Calfbrate the number of response samples
Saxophone Bass
Drums Violin Boss Bass
1-3
#/1/N quantile
What kind instrument intirument?
Sprrphootd
Trumped
Gortar
Piano
cunf cunformal scores (r))
00000
Sampled responses
Frequency score
Prediction set
Calibrate the threshold to identiff high-quality resporsess
selli-jo
Piano
M01/8 quantile
Piano
Piano
Flure Piano
Piute
Viatn
Pinto
nemcunforuiity scores Lsg
What kind instrument is the leftest iutrrmmnt?
Figure 1: Pipeline of TRON illustrated by two open-ended VideoQA examples. Step calibrates the minimum number of responses required to sample for each test data point on the calibration set; Step calibrates the threshold to identify high-quality responses. A larger prediction set size reflects higher risk (model uncertainty) during the current VideoQA task.
challenges above, which constructs statistically rigorous prediction sets with the hallmark of risk control (Angelopoulos et al., 2024, Farinhas et al. (024). Furthermore, SCP allows the calibrated set size to vary based on the model's uncertainty regarding particular input.
SCP has been successfully applied language modeling (Campos et al., 2024). However, existing methods either (1) modify the original model outputs ensure factuality with desired probability (Mohri & Hashimoto 2024| Cherian et al., 2024), potentially leading to uninformative and vague generations, (2) rely internal token sequence logits (Quach al., 2024), or (3) are restricted to multiple-choice settings (Kumar et al. 2023] Ye et al., 2024; Kostumov et al. 2024). Adapting SCP for proprietary MILMSs in practical open-ended video question-answering ( (VideoQA) applications presents several challenges: (a) Adaptability. Unlike closed-ended tasks equipped with fixed options covering the correct answers, the output space for each VideoQA task is unbounded, and there may not be an acceptable response within the sampled generations; (b) Reliability. Logits or verbalized confidence levels are often miscalibrated, leading to biased prediction sets; (c) Flexibility. For some API-only MLLMs, users lack access to internal model information like logits.
In this paper, we aim to tackle these challenges by developing a two-step framework for risk control and assessment (7RON). We illustrate the pipeline of TRON via two open-ended VideoQA samples F The workflow begins by determining whether acceptable responses exist in the sampled generations, akin to options in closed-ended tasks. To this end, propose novel conformal score that controls the minimum number of response samples required for each calibration data point. For new data point, we define the number of samples as the approximate 1 quantile of all conformal scores the calibration set, denoted as f. Then, the error rate for the candidate set of each test data point failing encompass acceptable response is controlled by the risk level (i.e., a.).
Once the risk level for acceptable responses being sampled is specified, we evaluate the reliability of each response. Inspired by self-consistency theory (Wang al. 2022 Li et al. 2022b), propose using the frequency each response as a proxy for confidence compensating for the limitations of internal logits information and verbalized confidence scores, and define the nonconformity score of each calibration data as one minus the frequency score of any one acceptable response in the 3 quantile of
candidate set. Then, we specify another risk level and calculate the approximate all nonconformity scores, denoted as &. The quantile serves as threshold to identify high-quality responses within the candidate set of each test data point. Finally, we guarantee that the miscoverage
'Response frequency positively correlates with average real generative probability. As the frequency in- creases, there is corresponding growing level of confidence in the model's responses (Su al. 2024)
2

---

Published as a conference paper at ICLR 2025
rate of acceptable responses in the calibrated prediction sets is statistically controlled by two risk levels, a and 3 (i.e., S a + 3 - a:8), while the set size reflects the model uncertainty to each query.
two closed-ended VideoQA datasets (e.g., Video-MME (Fu et al. 2024) and NEXT-QA (Xiao et al. 202T)), and two open-ended VideoQA datasets (c.g.. MUSIC- AVQA al. 2022a and MSVD (Chen & Dolan 2011)) )) utilizing five open-source MLL.s((e.g.
We evaluate framework
VideoLLaMA (Zhang et al. 2023). PandaGPT (Su et al. 2023), and NEXT-GPT (Wu et al. 2024 ),
and three closed-source MLL.s(e.g.. Gemini (Reid et al. (0244) and GPT-4o mini (OpenAl 2023 ).
Experimental results demonstrate that TRON guarantees the error rates in both steps across various user-specified risk levels. Additionally, the deduplicated average set size provides stable uncertainty estimation of MLLMs across various risk levels in open- open-ended VideoQA tasks, complementing the accuracy metric for more comprehensive evaluations. Furthermore, we investigate other black-box measurements in the development of the nonconformity score, resulting in more efficient predictions and consistently rigorous guarantees. To summarize, our contributions are the following:
We introduce a general framework for risk control and assessment, applicable to any MILM that supports sampling for both open-ended and closed-ended VideoQA tasks.
We propose a novel conformal score to derive the minimum sample size that bounds the miscoverage rate, extending SCP open-ended contexts, and then develop the nonconfor- mity score based on black-box measures, which results in rigorous guarantees as well.
We explore the impact of semantic redundancy on risk assessment leveraging prediction set in open-ended settings, and disclose that the deduplicated set size provides more stable uncertainty estimation across various risk levels, which serves as promising evaluation metric for MLLMs VideoQA tasks.
2 RELATED WORK
Split Conformal Prediction and Risk Control. Our work extends the variants of the SCP frame- work for correctness (coverage) guarantees and risk control in language modeling. Existing methods either modify model generations and achieve conformal factuality guarantees (Mohri & Hashimoto 2024 Cherian et al. 2024) or guarantee the ground-trume coverage in Multiple- Ceattppe-CChoie Choice Question- Answering (MCQA) tasks (Kumar et al.) 2023 Ye et al. 2024 Kostumov et al. 2024). The previous approaches may result in vague and uninformative responses, constrained by the correctness of the original generation, while the latter frameworks are limited closed-ended settings. In open-ended scenarios, (Quach et al., (2024) develops a stopping rule with slightly relaxed coverage criteria uti- lizing the internal logits information, restricting the applicability
API-only models, whereas (Sul et al. 2024) leverages black-box uncertainty but fails to align the nonconformity score with correct- ness, leading to theoretically loose guarantees. Our approach addresses these gaps by introducing a novel two-step framework that first derives the minimum number of response samples to approxi- mate a "closed-ended" condition, where there at least one acceptable option provided, followed by identifying reliable responses based on self-consise self-consistency theory, which features (1) rigor, controlled by two user-specified risk levels, a and B, and (2) versatility, applicable to any MLLM that supports sampling in both closed-ended and open-ended settings.
Risk Assessment and Uncertainty Estimation. Recent work performs probabilistic modeling LLi et al. 2024), develops entropy-based uncertainty measures (Wang et al. 2024c Duan et al. 2024
Farquhar et al. 2024). and generate multiple responses to analyze the output space (Lin al. 2024
Wang et al. 2024b). However, these approaches are heuristic, and forcing a measure to predict single data point overly restrictive (Cresswell al., 2024). Yee et al. 2024) and (Kostumov al., 2024) apply SCP to MCQA tasks, creating conformal sets based on a measure of uncertainty (risk), where the set size represents the confidence of the foundational model regarding a particular input, with larger sets indicating higher risk. In open-ended scenarios, there may be redundant responses within the prediction sets due to the randomness of sampling (Quach et al. 2024).. Our work explores the performance of set size on risk assessment before and after deduplication and observes that the original set size becomes unstable in uncertainty estimation across different MLLMs as the error rate changes, while the calibrated set size reliably evaluates MILMMS across various risk levels.
3

---

Published as a conference paper at ICLR 2025
3 METHODOLOGY
In this section, we introduce how our framework extends SCP techniques to open-ended VideoQA tasks based on novel conformal score for sampling responses. addition, to address the limitation of internal logits on API-only MLLMs, we develop the nonconfornity score for identifying high- quality responses based on plack-box measures.
3.1 PRELIMINARIES
Following standard SCP for risk control (Angelopoulos & Bates 2021 Angelopoulos et al. 2024), calibration dataset Deal {(X,, ==) of N query-answer pairs and independent and identically distributed (i.i.d.) test point (Fitests Itest)- Our goal of risk control is construct prediction sets that are calibrated to bound the correctness miscoverage rate
(Utest & (Iitest) E,
where the user-specified risk level, and C' function formed utilizing Peat- Note that the risk level is marginal (average) over the draw of both the calibration and test data points.
M;
For each query of calibration data X generate Mj response samples
yim
and demand
m
(-00
M.
that there is at least correct response in the candidate set (i.e., Yi
ym
For the test
m=1
point, also generate Mest response samples { (oltest) ) M J Jm=1" Note that currently unsure
if the candidate set of test data encompasses acceptable responses.
3.2 CONFORMAL RISK CONTROL
Sampling. As we construct a prediction set by selecting high-quality responses from the can- didate first perform simple calibration step for the setting of Mrest control the risk of 5m (((eest) }
Mrest
m=1 failing to encompass acceptable responses, so that we can approximate closed-ended settings with specific probability. We develop a novel conformal score for each calibration data
point, which determines the minimum sampling threshold that ensures Y, E
(()) fm
m=1
M
r YX,, sup M, VM, Mi, Y, & ym
(2)
m=1
and define be the [()) quantile of r.. .N (r: r (X,, Y.)): +[(N+)(1-0)-- Then, we set Meest = f and obtain the upper bound of the error rate
P Vtest & ((teeat)
glu
aa
(3)
m=1
complete proof of Eq. 3 presented in Appendix
B.1
Li et al. 2022b Wang et al. 2024b), which states that repetitively sampled response is viewed as a form of consistency linked to higher confidence in the response, perform simple semantic clustering process on the candidate set and obtain the frequency of each response, denoted as F [jm For detailed description of semantic clustering, refer Appendix [.. Then, we define the nonconformity score of the i- th calibration data point as one minus the frequency of the acceptable response, denoted as Dree
Identification. Inspired by self-consistency theory (Wang al. 2022
within the candidate set, which is semantically equivalent to the label Y; (i.e., Fref + Y,) A B
(0))
represents bidirectional entailment (MacCartney & Manning 2015 Mohri & Hashimoto 2024
SSpli conformal prediction requires that the test data points drawn from the same distribution the calibration data points.
4

---

Published as a conference paper at ICLR 2025
Upper Bound
Upper Bound
0.8
0.82
NEXT
NEXT
0.7
Empitical Errar Rater
0.73
MUSC
MSVD
0.64
0.5
0.55
Empirical
0.4
0.37
0.2
0.28
0.18
0.1
0.2
0.3
0.4
0.5
0.6
0.7 0.8
0.9
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.9
Risk Level
Risk Level
(a) EER vs. risk level a.
(b) EER vs. risk level 3.
Figure 2: Results of the EER metric at various risk levels. Each solid line starts at the GPT-4o mini model on the associated dataset. (a), the upper bound corresponds to various values of the risk level a. In (b), fix the risk level in the first stage 0.1 (i.e., 0.1), and the upper bound represents the values e for different values B.
Farquhar et 2024 between A and (i.e.. equivalence). The specific evaluation for bidirectional In this case, the nonconformity score is formulated as entailment is provided in Appendix C.I
(X,, Y,) ==
(ree
(c) Note that Dref refers any acceptable response in the candidate set that is semantically equivalent to the reference answer Y, Similar to f, define š to be the [(N+1((1-)) quantile of s1. 8N (&; = (X,, Y))): & = s $[(++)(1- Then, we construct the prediction set by post-processing the candidate set into set of responses with low non-conformity scores
(xrest) =
(fteat) Um'
E 1m
(test))
Trests (teat) <
(5)
Finally, guarantee the upper bound of the correctness miscoverage
PP (yeett & (Irest)) { (gleest)
Ylest E ggleer))
= IP Yrest &
+ IP Steet
(6)
m=1
m=1
S,, where a + B - 0.3. The complete proof of Eq.
6 given in Appendix B.2
Extensibility. F (-) can be any measure that reflects the reliability of each response. In the case of frequency, appropriate sample size is necessary to approximate the output distribution and accurately represent the confidence level. We also investigate other measurements in Section 4.4 and evaluate their impact on risk control and assessment and prediction efficiency.
4 EMPIRICAL EVALUATIONS
4.1 EXPERIMENTAL SETTINGS
Models. We employ five open-source MLLMs, including VideoLLaMA-7B (Zhang et al. 2023 VideoLLaMA-II PandaGPT-7B (Su et al. 2023), PandaGPT-13B, and NEXT-GPT (u et al. 2024), and three closed-source MLLMs, including Gemini-1.5-Fliar (Reid et al. (024). Gemini- 1.5-Pro, and GPT-4o mini (OpenAl 2023). Note that we utilize all five open-source MLLMs as they API-only MILMs,, i.e., assumes access any internal information of MLLMs.
Datasets. We consider two multiple-choiie VideoQA datasets: Video-MME (VMME) (Fu et al. 2024 and NExT-QA (NEXT) (Xiao et al.) 2021), where only utilize the multiple-choin section of NEXT. and two open-ended VideoQA datasets: MUSIC-AVQA (MUSC) (Li et al. 2022a) and MSVD (Chen & Dolan 2011). More details of the dataset utilization be found in Appendix
5

---

Published as a conference paper at ICLR 2025
V
9
PondscPt-78 or
0.45
2.55
RundecrT:133-0 on
2.90
Average Set Size
Vident sWW 333-00
VidcoLLaMA 18-91
0.41
PandaccP--1B.0
0:40
5
2.73
0.37
Pank
2.70
2
0 40
60
50 100
80 100 Silence Percentage (7))
0 20
40
60
Silence Percentage (1)
(a) ACC vs. SP.
(b) APSS SP.
0.18
0.28
0.37
0.46
0.55
0.64
0.73
0.82
0.91
Risk Level
Figure 4: (a) and (b) show the variation of Figure Comparison APSS before and af- ACC and APSS across various SPs, utilizing the
ter deduplication at different risk levels (c) across VideoLLaMA-7B and PandaGPT-7B models on four open-source MILMs on the VMME dataset. the VMME dataset.
Evaluation Metries. We utilize the Empirical Error Rate (EER) to assess whether we control the risk of miscoverage by the two user-specified risk levels (Angelopoulos & Bates| (021). Note that in the first stage, miscoverage refers to not sampling the acceptable responses, and this EER is controlled by a. In the second stage, miscoverage refers to the calibrated prediction sets failing to cover the acceptable responses, and this EER is bounded by both a and B. Additionally, we adopt the commonly used Accuracy (ACC) and the Average Prediction Set Size (APSS) (Kostumov al. 2024
Ye et al. 2024) to evaluate MLLM predictions. More details can be found in Appendix F
4.2 RESULTS FOR RISK CONTROL
Following prior work (Mohri & Hashimoto 2024 et et al. 2024 Quach et al. (004)) we randomly split our datasets half by default, calibrating the sampling number and threshold for identifying high-quality responses on the first half, i.e., the calibration set, and measure the EER (i.e., miscov- erage rate) at various risk levels (i.e., upper bound) in two steps on the second half, i.e., the test set. We plot the empirical results in the two steps utilizing the GPT-4o mini model on four datasets in Figure 2a and Figure 2b and defer similar plots employing other MLLMS to Appendix G
In the first step, we aim to address the challenge of determining whether acceptable responses covered by the candidate set in open-ended VideoQA tasks. Figure2a demoossttss that by deriving the minimum number of responses required to sample for each test data point on the calibration set based on our developed conformal score, we achieve statistically rigorous guarantees on EER. The marginal correctness miscoverage rates by the sampled responses bounded by various predetermined risk levels. In this way, we can approximate the fixed options provided in closed-ended tasks through sampling, despite the presence of duplicate options.
the test set are consistently
In the second step, we attempt to overcome the limitations of previous approaches, which rely on internal model logits or verbalized confidence scores. We specify the risk level in the first step to be 0.1 (i.e., = 0.1), and then measure the correctness miscoverage rates by the prediction sets, which are calibrated by our devised nonconformity score based on self ooff-oonsstency theory. Figure illustrates the validity of identifying high-quality responses based on their frequency scores with the candidate sets. The miscoverage rates on the calibrated prediction sets are strictly bounded by the risk level (= -+-3-a3) at various settings of B. In this way, obtain intuitive confidence level by analyzing the frequency of each response in the candidate set, adapting our framework to API- only MILMs. In terms of the black-bor reliability measurement, we also evaluate the development of nonconformity score based on semantic diversity in Section 4.4
Empirical evaluations demonstrate that our risk control framework matches theory in practice, as the correctness miscoverage rates in both steps never exceed the user-specified risk level. Note that EER is marginal (average) over the test set, not conditional to individual test data points.

---

Published as a conference paper at ICLR 2025
Table 1: Evaluations of the ACC and APSS metrics utilizing five open-source and three closed- source MLLMs across four open-ended and closed-ended datasets.
ACC (
APSS J
MLLMs
VMME
NEXT
MUSC
MSVD
VMME
NEXT
MUSC
MSVD
Open-source Models
VidcoLLaMA-78
38.10
46,06
78.70
78.29
2.81
2.68
1.97
1.63
LaMAI33
45.33
47.07
79.18
78.91
2.58
2.37
1.59
1.63
PandaGPT-7B
43.45
50.95
81.85
77.29
2.70
2.10
2.02
1.12
PandaGPT-13B
48.75
58.90
82.73
86.04
2.24
2.20
2.21
NEXT-GPT
42.64
42.59
79.84
84.37
2.43
2.45
2.20
1.70
Cloxed-source Models
Somini-I.5-Flasil
72.31
70.80
85.58
87.14
1.22
1.17
1.55
1.08
Gemini-1.5-H
73.11
76.29
85.16
86.67
1.15
1.15
1.59
1.10
GPT-4o mini
73.26
82,50
86.27
85.71
1.14
1.08
1.43
1.21
4.3 RESULTS FOR RISK ASSESSMENT
Conformal sets constructed adaptively to each particular input (Angelopoulos & Bates 2021 1,22), that the set size gives us an indicator of the model's uncertainty (or the risk in the current decision- making). Prior work (Ye et al. 2024 Kostumov et al. 2024) utilize APSS to benchmark LLMs and Vision-Language Models (VLMs) through risk assessment in MCQA tasks. In contrast to closed- ended scenarios, where MLLMs are provided predetermined
of options that encompasses the correct answer and produce prediction set without internal duplicates, open- open-ended VideoQA tasks feature their unfixed and unbounded output spaces. Consequently, duplicate responses are inevitable during the sampling process, which leads to semantic redundancy in the prediction set after filtering for high-quality responses based on the nonconformity score.
In this section, we investigate the impact of semantic redundancy in the prediction set when employ- ing the APSS metric evaluate MLLMs, address the research gap in open-ended settings. We fix the risk level in the first stage at 0.1 (i.e., a 0.1) and measure the APSSs for different values of & (= a + 3 - ar3), utilizing four open-source MILMs on the VMME dataset. As demonstrated in Figure 3 the original APSS is sensitive to the error rate, leading ambiguity in the evaluation of MLLMs at different risk levels, while after removing semantically redundant responses, the final APSS consistently decrease the dational models of varying performance at different risk levels. Additionally, the post-processed prediction sets result in more efficient decision-making Note that still utilize bidirectional en- tailment detailed in Appendix to determine the semantic equivalence between two responses.
rate lowers, and accurately distinguish between foun-
Given the adaptiveness and stability of the deduplicated APSS, we propose utilizing uncertainty/risk, represented by APSS, to evaluate MILMS in open-ended VideoQA tasks for the first time. We spec- ify the risk level be 0.19 (a 3 0.1) and evaluate the ACC and APSS metrics across four datasets utilizing eight MLLMs. Intuitively. MLLMs with higher accuracy should exhibit lower risk when processing given VideoQA dataset. However, as shown Table T higher ACC does not nec- essarily correspond to lower uncertainty and risk. For instance, although PandaGPT-133 achieves an ACC of 86.04% on the MSVD dataset, higher than GPT-4o mini, which has an ACC 85.71 %, PandaGPT-Is 13B exhibits higher APSS, indicating greater uncertainty compared to GPT-4o mini. This demonstrates that even with higher ACC, uncertainty, measured by APSS, may not decrease. Similarly, VideoLLaMA-133 13B and PandaGPT-13B both demonstrate positive correlation between ACC and APSS on the MSVD dataset. These findings highlight the importance of considering uncertainty alongside ACC when evaluating MI.LMs.
Unlike prior work (Kostumov et al. 2024) that evaluates MILMs on vision-language QA datasets, investigate the impact of the audio modality MLLM performance in open-ended VideoQA tasks. Specifically, we extract audio information from the original video, apply varying levels of Silence Percentage (SP), defined the proportion of randomly muted audio segments in video, and then provide the processed audio, along with the corresponding visual and textual information,
In the open-ended VideoQA task, sampling multiple responses from MI. can result in the candidate set containing identical answers answers that syntactically lexically distinct but semantically equivalent.
7

---

Published as a conference paper at ICLR 2025
to MLLMs. As illustrated in Figure incorporating additional modality information enhances the ACC MLLMs to some extent. As the SP increases, we observe significant changes in model uncertainty through the APSS metric, showcasing that introducing audio modality enhances the confidence level of MILMs' decision-making. In light of the observations mentioned above, we hope that our framework through risk assessment, while providing guarantees, thereby complementing the accuracy metric for more comprehensive analysis.
assist in evaluating the performance of MILMS in VideoQA tasks
4.4 SENSITIVITY ANALYSES
Splitting Ratio of Calibration and Test Set. As previously discussed, the calibration set is essen- tially set sufficiently general observation samples, from which we derive the number of response samples required for each test data point and the threshold for identifying high-quality responses in the candidate set based on specific statistical criteria. In this section, we explore the impact of the ratio of sample sizes between the calibration set and the test set the final performance, specifi- cally how many resources are needed for our framework to achieve risk guarantees on the test set. In previous work, set ratio of 0.5 by default. Here,
evaluate the EER when the ratio is set 0.3 and 0.1. As shown in Figure 5 when a and /3 are set to 0.1, resulting in an overall risk level (i.e., 2) of 0.19, we achieve consistently strict guarantees the miscoverage rates across three split ratios utilizing three closed-souree MLLMs on the MUSC dataset, which demonstrates the efficiency and stability of our framework for risk control in practical open-ended VideoQA applications.
Bound-0.19
GPT Gamini-1.5-Flat Gemini-J 1.5 Pro
0.150
Empirical Error
0.125
0.100
0.075
0.050
0.025
0.000
0.3 Split Ratio
Figure 5: EERs on the MUSC dataset utilizing three closed-source MLLMs when varying the split ratio the calibration and test set. risk levels a and B3 are both set to 0.1.
Table 2: Comparison frequency and semantic diversity on the VMME datasets employing four open-source MLLMs. The risk levels and 3 are both set to be 0.1.
APSS
EER
MLLMs
Frequency
Semantic Diversity
Frequency
Semantic Diversity
VideoLLaMA-7B
2.81
2.74
0.1793
0.1766
ViMe-LL
2.58
2.52
0.1767
0.1744
PandaGPT-7B
2.70
2.42
0.1812
0.1675
PandaGPT-113
2.24
2.20
0.1783
0.1674
Reliability Measurement. As discussed Section 3.2 F (.) the second step can be any mea- sure that evaluates the reliability of each response within the candidate set. To address the issues of logit access utilizing API-only MILLMs and the overconfidence in the verbalized reliability scores, leverage the most intuitive frequency score based self-consistency theory. In this section, as the reliability score the m-th response with the i-th candidate set. We set the risk levels a and 3 to be 0.1 (i.c., 0.19), and evaluate the APSS and EER metries the VMME dataset utilizing four open-source
we utilize the semantic diversity, formulated as Egen 4560 S (8) Mm
8

---

Published as a conference paper at ICLR 2025
0.22
Miscoverage Error Ratee
0.20
0.16
0.14
0.12
Froqueney- VMME
Semarte- VMME
Frequency NEXT
--- Semanti: NEXT
Frequenyy MJSC
Semante-
0.08
Frequency MEVO
Semantic MSVD
Set Size
Figure 6: Stratified miscoverage rate at each size of prediction set on four VideoQA datasets uti- lizing the Gemini-1.5-Pra model. The solid line corresponds to Frequency, while the dashed line corresponds to Semantic Diversity.
MLLMs. The comparison results in Table that the employment of semantic diversity results in smaller APSSs across four MLLMs, indicating more efficient predictions, while the EER metric always bounded by the risk level. This finding demonstrates the stability of our framework when using different reliability measures. It also points to promising direction for further research on the impact nonconformity score designs and their ability to represent agrecment between question- and-answer pairs prediction efficiency.
Conditional Performance. As mentioned before, TRON bounds the correctness miscoverage rate marginally over the test set (i.e., average). However, for some critical applications requiring more stringent guarantees, users demand that error rate is bounded for particular test data point. In other words, the risk level should be guaranteed for any subset of the data points. In the most general case, conditional coverage is impossible to achieve (Angelopoulos & Bates 2021). Following prior
work (Kumar et al. 2023 Su et al., 2024). set 3 0.1 (i.e., = 0.19) and evaluate the size-stratified miscoverage rate utilizing the Gemini- 1.5--rr model. As demonstrated in Figure 6 the miscoverage rate fluctuates up and down with the set size, with some values exceeding It worth noting that when we employ Semantic Diversity as the reliability measurement, TRON's conditional performance improves (e.g., the miscoverage rate decreases from 0.2065 0. D.1667 on the VMME dataset). At this point, it's like a bag tricks, and we can enhance the capability of reliability measurement to optimize the conditional performance of risk control.
5 CONCLUSION
Deploying MILMs in practical VideoQA applications requires reliable risk control and assessment. In this paper, we present general two-step framework for risk control by calibrating a confor- mal score to sample new generations and a nonconformity score identify high-quality responses, which provides statistically and theoretically rigorous guarantees on the correctness miscoverage rate. Our method bridges the gap between SCP and closed-source MILMs, and it can also be ap- plied to other generative models (e.g., LLMs) that support sampling in various language generation tasks (see Appendix H). Furthermore, we conduct risk assessments and explore the issue of seman- tic redundancy in prediction sets within open-ended contexts for the first time, which results in promising evaluation metric for MILMs open-ended VideoQA tasks via the average set size after deduplication. Additionally, conformity score for more efficient predictions. We hope that our framework can be helpful for numan-in-the-looe daccision-making and human-Al teams.
analyze reliability measurement in the development of the non-

---

Published as a conference paper at ICLR 2025
LIMITATIONS
In our work, guarantees are marginal over the test set, not conditional to individual data points. In addition, our framework does not deviate from the fundamental exchangeability assumption of SCP, and we will explore risk control under distribution shift conditions in future work.
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (Grant No. 62122035) and the project of the Sichuan Provincial Department of Science and Technology (No. 2023YFQ0011) This research was completed during a visiting period at Southern University of Science and Technology.
REFERENCES
AI@Meta. Llama 3 model card. 2024.
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Con- formal risk control. In The Twelfth International Conference on Leurning Representations, 2024.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, 2015.
Tianyi Bai, Hao Liang, Binwang Wan, Ling Yang. Bozhou Li, Yifan Wang, Bin Cui, Conghui He, Binhang Yuan, and Wentao Zhang. A survey of multimodal large language model from data- centric perspective. arXiv preprint wrXiv:2405.16644 2024.
Margarida M Campos, António Farinhas, Chrysoula Zerva, Mário AT Figueiredo, and André FT Martins. Conformal prediction for natural language processing: survey. arXiv preprint arXiv:2405.01976 2024.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190 200, 2011.
John Cherian, Isaac Gibbs, and Emmanuel Candes. Large language model validity via enhanced conformal prediction methods. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Jesse C. Cresswell, Sui, Bhargava Kumar, and Noël Vouitsis. Conformal prediction sets improve human decision making. In Forty-first International Conference Machine Learning. 2024.
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang. Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 5050-5063, 2024.
António Farinhas, Chrysoula Zerva, Dennis Thomas Ulmer, and Andre Martins. exchaxcha conformal risk control. In The Twelfth International Conference on Learning Representations, 2024.
Sebastian Farquhar, Jannik Kossen. Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. Nature, 630:625-630, 2024.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.2107. 2024.
10

---

Published as a conference paper at ICLR 2025
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding -ehadine bert with disentangled attention. In International Conference on Learning Representations, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. Towards mitigating IIm hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1827-1843, 2023.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), 2017.
Vasily Kostumov, Bulat Nutfullin, Oleg Pilipenko, and Eugene llyushin. Uncertainty-awar evalua- tion for vision-language models. arXiv preprini 24Ki::2002.144188, 2024.
Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and An- drew Beam. Conformal prediction with large language models for multi-choice question answer- ing. arXiv preprint arXiv: /8404, 2023.
Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19108-19118, 2022a.
Haopeng Li, Qiuhong Ke, Mingming Gong, and Tom Drummond. Answering from sure to un- certain: Uncertainty-aware curriculum learning for video question answering. arXiv preprint arXiv:2401.015 2024.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.023388 2022b.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Generating with confidence: Uncertainty quantifi- cation for black-box large language models. Transactions Machine Learning Research, 2024.
Bill MacCartney and Christopher D Manning. Natural logic and natural language inference. Com- puting Meaning: Volume pp. 129-149911 2015.
Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guaran- tees. arXiv preprint (eXXiv:2422.1097 2024.
OpenAl. Gpt-4 technical report, 2023.
Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confi- dence machines for regression. In Machine learning: ECML 2002: 13th European conference on machine learning Helsinki, Finland, August 19-23, 2002 proceedings 13, pp. 345-356, 2002.
Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qa: A multi- modal visual question answering benchmark for autonomous driving scenario. In Proceedings of the AAA/ Conference on Artificial Intelligence, 2024.
Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In The Twelfth International Conference on Learning Representations, 2024.
Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922 2023.
Siva Reddy, Danqi Chen, and Christopher D. Manning, Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 2019.
11

---

Published as a conference paper at ICLR 2025
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gem- ini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 2019.
Jiayuan Su, Jing Luo, Hongwei Wang, and Lu Cheng. Api is enough: Conformal prediction for large language models without logit-access. arXiv preprini
2024.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305. 16355, 2023.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. / /171, 2022.
Yuxuan Wang. Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models, arXiv preprint arXiv:2406.1633 16338, 2024a.
Zhiyuan Wang, Jinhao Duan, Lu Cheng, Yue Zhang, Qingni Wang, Hengtao Shen, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. Conu: Conformal uncertainty large language models with correctness coverage guarantees. arXiv preprint arXiv:2407.00499 2024b.
Zhiyuan Wang, Jinhao Duan, Chenxi Yuan, Qingyu Chen, Tianlong Chen, Huaxiu Yao, Yue Zhang, Ren Wang, Kaidi Xu, and Xiaoshuang Shi. Word-sequence entropy: Towards uncertainty timation in free-form medical question answering applications and beyond. arXiv preprint arXiv:2402. 14259. 2024c.
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and S Yu Philip. Multimodal large language models: survey. In 2023 IEEE International Conference on Big Data (BigData), pp- 2247-2256 IEEE, 2023.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. NEXT-GPT: Any-to-any multi- modal LLM. In Proceedings of the International Conference Machine Learning, pp. 53366- 53397, 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 9777-9786, 2021.
Fanghua Ye, Mingming Yang, Jianhui Pang, Longyue Wang, Derek F. Wong, Emine Yilmaz, Shum- ing Shi, and Zhaopeng Tu. Benchmarking LLMs via uncertainty quantification. In The Thirty- eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.0 02858, 2023.
Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang. Yifan Wang, Huan- ran Chen, Xiao Yang, Xingxing Wei, et al. Benchmarking trustworthiness of multimodal large language models: A comprehensive study. arXiv preprint arXiv:2406.07057, 2024.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong. et A survey of large language models. arXiv preprint arXiv:2303.1 18223, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth Interna- tional Conference on Learning Representations, 2024.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15044 2023.
12

---

Published as a conference paper at ICLR 2025
A BACKGROUND OF SPLIT CONFORMAL PREDICTION
Split conformal prediction (SCP) is model-agnostic and distribution-free, and can offer statistically rigorous guarantees of ground mound-truth coverage on fresh test samples with minimal assumptions (c.g., data exchangeability) given a relatively small calibration set (Angelopoulos & Bates 2021 Campos al. 2024). SCP can transform any heuristic notion of uncertainty (i.e., the nonconformity score) from any model into a statistically rigorous one by constructing a prediction set adaptively to each particular input. Additionally, the size of each prediction set allows into the uncertainty of the current decision-making (i.e., the set becomes larger when the model is uncertain the input is intrinsically hard) Ye et (2024) Kostumov al. (2024).
gain intuitive insights
1i=1" Let's illustrate the framework utilizing classification problems. In the calibration set, {(x, { we define the nonconfornity score (NS), 8,, to identify a heuristic notion of the uncertainty of the decision process of the model for the i-th calibration data. Here, we utilize one minus the softmax output corresponding to the true class to link the NS with the uncertainty condition of correctness closely. Then we calculate the [(x+1]1-a]] {si,... sn in the calibration set and denote as I. Since there are approximately 1 - calibration samples whose uncertainty states corresponding their true class satisfy (i.e., S) the threshold q. we construct the prediction set of each test sample following C (Xtest) We expect that if the class y satisfies (Xtstt y) S q (i.e., the uncertainty state of the current class satisfies the uncertainty condition corresponding to the true labels of approximate 1 calibration samples), y has the probability of 1 a to the true label test-
(i.e., approximate
a) quantile of all NSs (i.e.,
{y E [K] s (Xtest, y) S 4)-
B PROOFS
B. PROOF OF EQUATION
3
This section presents standard proof of validity for the upper bound of the error rate, defined Eq For completeness, we restate definition of the conformal score in Eq We define the
(())) M,
minimum sampling threshold that ensures Y; E 9m
as
m=1
T3 = (Xi,
M,
= sup Mr Mi Y
(000
m=1
[(++1)]1a)]
sort the conformal scores so that ri S... S TN. and calculate the of Fri," .N:
empirical quantile
N
(i (i:ra S srll
[[N + 1) a)
8. = inf r:
N
N
(8)
=[(N+1)(1-a]]-
We proceed by noting the equality of the two events based on the definition in Eq
yeast & (pftestt)
(rrest f}.
m=1
By exchangeability the calibration and test data points, we have
i
IP (Teest 5rr = N 11
(10)
'The standard framework of conformal prediction assumes exchangeability of all data points, and exchange- ability weaker than the independent and identically distributed (i.i..)) assumption.
13

---

Published as a conference paper at ICLR 2025
Then, guarantee the upper bound of the probability that the candidate set fails to encompass acceptable response based on a user-specified risk level, denoted as az
Weett & { (gheert) 1a
(rrest > f)
m=1
= (Ftest > ([(N+1)(1-a))
= 1 - P (Frest S &T((N+)(--)))
(11)
1 [(N + (1)(11111 (1
N ++1
S
B.2 PROOF EQUATION 6
Given that select trustworthy responses from each candidate set to build the prediction set, let us formally prove upper bound of the enror rate (i.e., E) in Eq. [6] building upon the foundation es- tablished in Eq. 3 If there a certain event, and the probability of miscoverage is
acceptable responses in the candidate set, event {ytest C (Itest))
P Vlest & C (Ireet) Ytest & (((teat)
pm
m==1) = (Beest (Beest & (olteae)
& }mme
(12)
m=1
Conversely, if at least one acceptable answer exists the candidate set (i.e., Vtest we then redefine the non-conformity score here. First, we determine the acceptable response of i-th calibration data point (1) which is semantically equivalent to the correct answer Y, Then we formulate the non-conformity score
(lleet) Dm
m=1
8; 8 (X,, Yi)
(13)
Yref
where F (1/0)
(
(i) is the frequency of Fref in the candidate set. Similar Eq. 8 sort the
conformal scores so that 81 ... & SN. and define the [1++))100] quantile of 81, 8Nt
N
inf {i S $
[NN 1) (1 B)1
N
N
(14)
+((N+1)11-8]
Then, construct the prediction set following
(Iirest) =
Um
(test)
E (iftestt)
mm=1 (Irests ) 58)
(15)
(test) fm
Given ytest
mm=1' obtain
(rret & C (Irest)) = (Itest Breet) > 8}
(16)
{stest 8)
In this case, the probability of miscoverage is
PP Ureat & C (Frest)
Utest E
(ttest) Um
m=)) =P Steat > 8 Brest { ) )mm)) (17)
Similarity to Eq.
10 we have
(Stest > 84) = -P (Stest I si)
i
(18)
N
14

---

Published as a conference paper at ICLR 2025
Finally, achieve risk control and guarantee the upper bound of the correctness miscoverage rate:
P (ytest & (Itest))
= Yteat & { (glicce)
+ Stest > 8 Ylest glteet) 1²
m=1
m== [(N + a)]
I(N + 1) cr]
[(N + (1 (111111
+
N +1
N +1
N +1
(19)
[(N + 1) (1 - 331
[NN + 1) a1]
N +1
N ++1
< 1 (1 a1 (1-8)
<a+ a8
C IMPLEMENTATION SPECIFICS
C.I SEMANTIC CLUSTERING PROCESS
Algorithm 1: The pseudo-code for semantic clustering.
Input: The candidate set of the i-th data point with M; response samples (1) ym m==""
C = (). F =[]. C is non-repaating
for m - to M, do
L= -m
for mn' +1 M; do
if m + then
if 7 + im then
L. + L (m])
f+
C+C+(L): F F+
Output: Non-repeating semantic clusters C, frequency of M, responses.
Note that since there may not be semantically equivalent transfer between the sampled responses in open-ended tasks, the sum of the frequency scores corresponding to all responses in the candidate set may be uncertain, and utilize the normalized frequency
for each response.
Bidirectional Entailment. As noted in Section 3.2 and Algorithm I A +> A, represents that 2024 Lin et al. 2024] Farquhar et al.] 2024). we utilize Natural Language Inference (NLI) classifier, with the off-the- shelf DeBERTa-large Ta-raaee-- model (He et al., 202T) as the backbone, for this evaluation. The NLI classifier typically takes [A1, A2] input and predicts logits for three distinct semantic relationships: entailment, neutral, and contradiction. Note that this function has directionality and we only consider A and Ag to be equivalent when the logit scores corresponding to the entailment are at their highest for both inputs [Q UPUA A, Q U Ag] and IQUAp QUA,] (i.c., bidirectional). Here, Q denotes the question, and means evaluate the semantic relationship between query-response pairs.
A1 semantically equivalent A2. Following prior studies (Duan
D DATASETS
Video-MME (VMME) Fu et al. 2024 is multi-modal evaluation benchmark for MLLMs in video analysis, which includes 900 manually selected videos totaling 254 hours and resulting in 2,700 MCQA pairs. NExT-QA (NEXT) (X1oo al., (021) is benchmark for VideoQA that focuses on causal and temporal action reasoning, moving beyond simple scene descriptions. It features both multiple-choice and open-ended QA tasks. However, only selected the multiple-choice
15

---

Published as a conference paper at ICLR 2025
part for experiments. We use the sub-test of MCQA, which consists of 2.312 QA pairs. MUSIC-AVQA (MUSC) (Li et al. 2022a) is an open-ended VideoQA dataset designed to evaluate comprchensive multimodal understanding and spatio-temporal reasoning We use the test set of MUSIC-ACQA, which consists of 9,185 QA pairs. MSVD (Chen & Dolan 2011) is open-ended dataset that provides large-scale, highly parallel text collection. We use 2,519 QA pairs from this dataset. In our experiments, we extracted the audio from the videos and integrated it with the corresponding visual and textual data before inputting them into the MLLMs.
visual scenes.
E HYPERPARAMETERS
We employ multinominal sampling for responses which are used construct prediction sets. The temperature of generation for all MLLMs is set to 1.0. All the experiments are conducted on a server with one Intel Xeon Gold 6326 CPU and NVIDIA A6000 GPUs.
F EVALUATION METRICS
Empirical Error Rate (EER) evaluates the validity of our framework for risk control. In the first stage, we expect that, under the condition of exchangeability, we calibrate the number of response samples of each test data point based on the calibration set so that the candidate set encompasses at least one acceptable response with specific probability. To this end, the EER on the test set, Drest, should be bounded by the risk level a (i.e., I a). At this point, we can approximate the closed-ended settings utilizing the candidate set. We redefine each sample in the test set as (z4, y/) and the candidate set the i-th data point as S (z;). Then, EER is formulated as
EER
Σ
1 Hi 8 (2.))
(20)
DDest
In the second stage, we expect that we can identify the acceptable response within the candidate set, which also said the prediction encompasses the correct answer, based on the nonconformity score with user-specified probability. To this end, the EER on the test set should be bound by and another risk level (i.e., a + 3 a.B.). In the evaluations, we set a 0.1 by default and measure EER at various values of B. At this point, the upper bound of EER is 0.1 + B - 0.133 and EER formulated
EER
Σ
1 1v E C(xs))
(21)
[Deest]
Average Prediction Set Size (APSS) measures the average size of all prediction sets required to guarantee the EER the test set. A larger APSS indicates greater uncertainty/risk, while smaller APSS reflects higher efficiency and confidence. At this point, APSS is formulated as
APSS
Σ
IC(x))
(22)
[Deest|
Accuracy (ACC) assesses the correctness of MLLMs' predictions, serving as a foundational bench- mark for model performance. In our experiments, we employ the most frequent (reliable) response within the candidate set of each test data point, denoted as o12)- as the model output to measure the total accuracy, which is formulated as.
ACC
1
Σ
10
Doee = y/
(23)
[Deest
(D
16

---

Published as a conference paper at ICLR 2025
G ADDITIONAL EXPERIMENTAL ANALYSIS
0.91
Upper
0.91
VNME
NEXT
NEXT
Emprrical Erron Rate
0.73
MUSC
Empiicall Enoor Rade
0.73
MUSC
MSVD
MIVD
0.55
0.56
0.46
0:40
0.37
0.28
0.28
0.19
0.28
0.37
0.46
0.55
0.64
0.73
0.82
0.91
Risk Level
(a) vs.
(b) EER VS. risk level E.
Figure 7: (a) Gemini-1 1.5-Flashh (b) Seminni-1.5-Pri We fix the risk level in the first stage at 0.1 (i.e., 0 0.1) and evaluate the correctness miscoverage rate at various values of B.
H GENERALIZABIILTTY OF TRON
As mentioned before, TRON applies to any generative language models that support sampling (or repeatedly responding to the same query) various open-ended tasks. To evaluate its generaliz- ability, we employ TRON on the (1) open-book conversational QA dataset, CoQA (Redyy et al.) 2019), and the (2) closed-book reading comprehension dataset, TriviaQA (Joshi et al. 2017). utiliz- ing LLMs, 1AMMAA--77B-Im-
(AI@ 2024) and LLaMA-3. 70B-Instrucc Note that we LangChain with DeepInfra for LLaMA-3 3-B- Instruct and LLaMA-3. randomly select 2,000 questions from the development split CoQA and 2,000 questions from the training split of TriviaQA, and set the split ratio between the calibration and test set to 0.5 for both datasets. Empirical results of the correctness miscoverage rate, obtained from 5 trials with randomly allocated calibration and test data various user-specified risk levels are shown in Figure g
Instructtttt [ We
0.91
Upper Bound
LLaMA-3-70B CoQA
LLaMA-3.1-70E CoQA
Empirical Error Rate
LLaMA-3-70B TriviQAA
LLaMA-3.1-70B TriviaQA
0.55
0.37
0.19
0.3
0.7
Risk Level B
Figure 8: Results of the EER metric various risk levels the CoQA and TriviaQA tasks utilizing LLaMA-3-70B-Insstee and LLaMA-3.1-70B-Insttn. Note that we fix the risk level a to O.I.
We also utilize the PandaGPT-13B model and GPTT-40 I on the Visual Question Answering (VQA) dataset (Antol al. 2015) for image understanding tasks. We randomly select 1,200 VQA samples and employ 200 samples calibration data and 1,000 samples as test data. We set to 0.1 and evaluate the ERR metric at various user-specified risk levels, B, as shown Figure
https://dceppiffr.com//// wttp://wwww..oooaaaatoaaaaaaaaaa Maw.e/taalllaaaMMaaaaaaaaaaaaaaa --t.-aaaaaa-------- Instruct
https//dccepinfra.com http://ddegnnttiaaccoommmmmaaaaa hwap://eilllaaaaaaaaaaaaaaaaaaaa 700---mmmmmm111-----m---n
17

---

Published as a conference paper at ICLR 2025
0.91
Upper Bound
PandaGPT-138
0.82
GPT-4o
Empirical Error Rate
0.73
0.64
0.46
0.37
0.19
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Risk Level B
Figure 9: Results of the EER metric, obtained from 5 trials with randomly allocated calibration and test data at various risk levels on the VQA dataset utilizing PandaGPT-133 and GPT-4o.
Empirical results demonstrate that TRON also provide risk management for both MLLMs and LLMs on image understanding, conversational QA, and reading comprehension tasks. Note that while the theoretical guarantee of correctness coverage is rigorous, there be minor fluctuations of empirical miscoverage rates that exceed the user-specified risk levels in practice due to finite- sample variability (Angelopoulos Bates 2021
Ye et al. 2024).
Comparisons on MCQA tasks. We also compare TRON with the existing method for LLMs on MCQA tasks, termed Least Ambiguous set-valued Classifiers (LAC) (Sadinle et al. 2019 Ye al., 2024 which has been proven to produce prediction sets with the smallest average size (Sadinle) jtt al. 2019h. LAC defines the nonconformal score function as one minus the softmax score cor- responding to the true label. We adopt MMLU (Hendrycks et al. 2021 the evaluation dataset utilizing LLaMA-3-8B- (0n------Innstuuc] and LLaMA-3.. 1-8-11Isttuc
To maintain consistency with pre- only select samples that cover least correct response
vious work settings Yee et al. 2024), within the candidate set when M is set to (i.e., a = 0). We set the splitting ratio between calibra- 0.1, and evaluate the APSS metric. Table 3 demonstrates that when model, TRON's performance can already match that LAC, and when M is 20, TRON more predictive efficient than LAC. Additionally, when the sampling size is set to 10, TRON outperforms LAC. Our insight is that since model internal logits can induce miscalibrated biases due to arbitrarily overfit or otherwise untrustworthy issues, we analyze the output distribution of the model through sampling, which is more sound.
tion and test set 0.5 and
sampling 10 times utilizing the L.LMA-3--B-Inst
Model
LAC
TRON (M = TRON (M 10)
TRON (M 20)
LLaMA-3-SB-Instrus
2.93
3.06
2.93(7)
2.76(z)
LLaMA-3.1 1-8B-Instruee
2.57(8)
2.61
2.53
2.50(4) (4)
Table 3: The APSS metric (L) on the MMLU dataset.
18


Reviews:
[{"review_id": "vIL1R9mDA5", "reviewer": "Reviewer_amj3", "summary": "The paper introduces a two step risk control based framework extending split conformal prediction method for open ended and multimodal (videoQA) tasks. The method applies a conformal score to calibrate the minimum number of responses (samples) needed to ensure coverage. This score defines the prediction set’s error rate at a risk level alpha. It then refines the set using frequency or semantic similarity to identify high quality responses controlled by another risk parameter beta. The overall risk is bounded by a function of alpha and beta to provide statistical consistency guarantees using calibration, sampling and refinement steps. The paper builds upon multiple conformal risk control based methods and addresses the shortcomings with this two step method (to maintain smaller average prediction set size (APSS))  with lower risk thresholds) and applying them to multimodal videoQA setting.", "questions": "- Do you foresee any major modifications needed for TRON to control risk in scenarios involving distribution shifts, where calibration and test distributions differ?\n- Would it be feasible to incorporate dynamic adjustments to prediction set sizes based on task difficulty or user preferences in real time? Are there challenges with balancing efficiency and robustness in such a dynamic setting?\n- Could access to model internals like logits help improve TRON's performance?\n- Could reliance on frequency-based nonconformity scores lead to biases in the types of responses included in the prediction set, especially in cases where the model’s sampling is limited?\n- Have you observed any variance in EER across different model architectures or response generation and sampling methods (for example, cases where EER can go outside the bounds set by alpha and beta)\n- How easy is it to generalize this approach beyond VideoQA to other open-ended tasks? Are there any major limitations or requirements for generalization?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper's two-step risk control methodology addresses general shortcoming of the conformal prediction method and provides statistical guarantees for error rates, increasing the reliability of MLLM responses.\n- Since open ended tasks are more challenging due to large number of possible generation, this method (two step approach, use of semantic similarity) seems to dynamically adapt well to provide flexible prediction set sizes for complex and generic generative scenarios (although we need more experimental validation). Error stays within bounds even after filtering step (bounded by alpha + beta - alpha.beta)\n- As shown is Figure 3, deduplication of the semantically similar responses helps with more stable error rates and smaller prediction sets. Experiments suggest that semantic diversity can create smaller, more efficient prediction sets (lower APSS) without compromising on accuracy (EER stays within limits).", "weaknesses": "- Authors already mention under limitations that guarantees are not conditional to individual data points but marginal over the test set. With this limitation, it may still be a bottleneck where risk compliance guarantees are needed for critical applications requiring more stringent guarantees and/or compliance requirements.\n- more open-ended evaluations and experiments on the open-ended datasets would have shed more light on the strengths and weaknesses of the methods (like Fig 4b). This is a key innovative strength of the method to address open-ended tasks (unlike MCQ) and adoption of this method will depend heavily on understand the strengths of this method in more open-ended generation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HDm6dTctcU", "reviewer": "Reviewer_rC7h", "summary": "The authors propose a two-step framework, TRON, for assessing and controlling risk in MLLMs, specifically targeting VideoQA tasks. The framework consists of: (1) Sampling Step: This step involves calibrating a conformal score that defines the minimum number of response samples needed to ensure the inclusion of acceptable responses within a certain risk level. (2) Identification Step: In this phase, a nonconformity score based on self-consistency theory identifies high-quality responses within the candidate set. This score accounts for model uncertainty by estimating reliability via response frequency, introducing another risk level to statistically control errors.\n\nThe authors carry sufficient experiments on four VideoQA datasets with eight MLLMs and demonstrate that TRON achieves desired error rates within the specified risk bounds. The study further highlights TRON’s adaptability and stability through deduplicated prediction sets that provide more efficient and robust uncertainty estimation across various risk levels.\n\nThe authors address limitations in existing SCP methods, which either modify outputs to ensure factuality, rely on internal token sequence logits, or restrict applications to multiple-choice settings. This new approach is versatile, applicable to both open-ended and closed-ended VideoQA tasks, and operates independently of internal model outputs.", "questions": "1. How does TRON handle outliers in response sampling that may disproportionately affect the frequency-based confidence scoring?\n2. Could TRON’s conformal score be further adapted to dynamically adjust the sampling size based on real-time uncertainty measurements?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. TRON’s two-step approach combines conformal scores and self-consistency theory to establish a flexible and robust risk assessment framework for MLLMs, particularly in open-ended scenarios, where traditional SCP methods fall short.\n2. The paper presents extensive experiments across four VideoQA datasets and eight MLLMs, showcasing TRON's effectiveness and consistency in different VideoQA tasks.\n3. By avoiding reliance on model logits, TRON is adaptable for API-restricted MLLMs, expanding its usability in various practical applications.", "weaknesses": "I raise the concern that although TRON is evaluated on diverse datasets, it primarily focuses on VideoQA tasks. Could it be tested on additional multimodal tasks to enhance the generalizability of its risk management capabilities?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pYJeLXytTX", "reviewer": "Reviewer_Tw6D", "summary": "The paper deals with risk control and assessment for MLLMs. To address the issues of existing work relying on the internal model logits and working in the multiple-choice setting, the authors propose a TRON, a two-step framework for MLLMs supporting sampling for both open-ended and close-ended scenarios. TRON allows controlling error rates by sampling response sets of minimum size and identifying high-quality responses using self-consistency theory. The experiments on VideoQA demonstrate the efficacy on eight MLLMs.", "questions": "- It seems that the best practice of the ratio of the calibration and test set is model-dependent. Is there any insight on the ratio selection when applied to different MLLMs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is easy to follow and understand.\n- The proposed method extends SCP to open-ended scenarios by estimating the confidence from frequency.", "weaknesses": "- The confidence estimation in Step 2 relies on the prediction of another model(e.g. DeBERTa-large-mnli). Then it should be at least discussed on the reliability as the semantic classifier. Otherwise, it makes the identification of risk control less convincing.\n\n- It is unclear how the silence percentage is conducted on the audio, and how the conclusion ‘introduce audio modality enhances the confidence level’ (Line371-372) is made. It is shown in Fig. 4 that increasing SPs leads to higher APSS. Also, introducing audio modality seems to only improve VideoLLaMA with SP <50%.   \n\n- It seems that the proposed method is general to LLMs as well. How does the proposed method work for LLMs for both open-ended and close-ended scenarios? The paper claims this advantage but shows no experimental results. This would make the paper more impactful.\n\n- How is the proposed method compared with existing methods for LLMs on such as MCQA? Such a comparison would make the paper more comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vtU1kk2FcC", "reviewer": "Reviewer_pRzK", "summary": "The paper presents **TRON**, a two-step framework designed for **risk control and assessment** in multimodal large language models (MLLMs), particularly for **Video Question Answering (VideoQA)** tasks. Addressing challenges in dynamic and open-ended environments, TRON leverages **Split Conformal Prediction (SCP)**, introducing a **conformal score** for sampling response sets and a **nonconformity score** for identifying high-quality responses. Through experiments on several VideoQA datasets, TRON demonstrates the ability to achieve desired error rates across various **user-specified risk levels**. It is also noted for exploring the concept of **semantic redundancy** in prediction sets as an evaluation metric, an area not previously investigated in open-ended contexts.", "questions": "1. **What is the expected computational impact** of using TRON in large-scale applications or real-time risk assessment tasks? Could you provide a more detailed explanation or metrics regarding the processing time required for each step?\n\n2. **Baseline Method Comparison**: Have you considered comparing TRON with simpler risk control baselines? For instance, would a heuristic-based risk control method suffice for certain types of tasks in closed-ended VideoQA? If not, could you clarify how TRON performs specifically better in such cases?\n\n3. **Semantic Redundancy in Open-Ended Tasks**: How does the semantic redundancy analysis handle responses that may be lexically distinct but only partially semantically equivalent? Could this approach potentially overlook responses with subtle but important semantic differences?\n\n4. **Alternative Conformal and Nonconformity Scoring Methods**: Could the proposed conformal and nonconformity scores be further enhanced by alternative methods, such as clustering-based or confidence-based approaches beyond self-consistency theory? If so, what would be the implications for TRON’s current framework?\n\n5. **Additional Real-World Validation**: Have there been any attempts to validate TRON in real-world or industry-specific VideoQA tasks, possibly in collaboration with industry partners? If so, could you share any preliminary insights on its practical performance and potential adaptations? If not, do you plan to include such validation in future work?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. **Innovative Framework**: The introduction of TRON, a two-step risk control and assessment framework, contributes significantly to the field of MLLM evaluation in both open-ended and closed-ended VideoQA tasks. Its flexibility in applying conformal prediction in open-ended contexts is commendable.\n\n2. **Novel Conformal and Nonconformity Scores**: The paper proposes a unique conformal score for setting the minimum sample size in open-ended tasks and a nonconformity score based on self-consistency theory. These scores provide a rigorous approach to risk control.\n\n3. **Addressing Uncertainty with Redundancy Analysis**: The evaluation of **semantic redundancy** in open-ended settings introduces a new angle to uncertainty measurement, providing a promising metric that complements traditional accuracy.\n\n4. **Comprehensive Experimental Evaluation**: The experiments span multiple datasets, risk levels, and different types of MLLMs, offering a thorough assessment of TRON’s effectiveness in diverse conditions.", "weaknesses": "1. **Limited Discussion of Practicality and Adaptability**: While TRON provides theoretical guarantees, the practical aspects, such as computational overhead and applicability in real-world scenarios, could be discussed in greater depth.\n\n2. **Insufficient Baseline Comparisons**: The paper lacks a comparison with other standard risk control methods or frameworks that may be relevant, particularly in closed-ended settings or previous SCP applications in MLLMs.\n\n3. **Complexity in Method Presentation**: Some sections of the methodology, particularly the derivation of the conformal and nonconformity scores, lack clarity, which could challenge readers unfamiliar with SCP.\n\n4. **Inconsistent Evaluation Details**: Although the experiments are extensive, details on some evaluation metrics, like APSS and their implications for different risk levels, could be better explained. The choice of models and how each metric was applied in open-ended versus closed-ended settings was also not consistently clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)