Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models

## Abstract

Abstract Multimodal Large Language Models (MLLMs) exhibit promising advancements across various tasks, yet they still encounter significant trustworthiness issues.
Prior studies apply Split Conformal Prediction (SCP) in language modeling to construct prediction sets with statistical guarantees.
However, these methods typically rely on internal model logits or are restricted to multiple-choice settings, which hampers their generalizability and adaptability in dynamic, open-ended environments.
In this paper, we introduce TRON , a t wo-step framework for r isk c o ntrol and assessme n t, applicable to any MLLM that supports sampling in both open-ended and closed-ended scenarios. TRON comprises two main components: (1) a novel conformal score to sample response sets of minimum size, and (2) a nonconformity score to identify high-quality responses based on self-consistency theory, controlling the error rates by two specific risk levels.
Furthermore, we investigate semantic redundancy in prediction sets within open-ended contexts for the first time, leading to a promising evaluation metric for MLLMs based on average set size.
Our comprehensive experiments across four Video Question-Answering (VideoQA) datasets utilizing eight MLLMs show that TRON achieves desired error rates bounded by two user-specified risk levels. Additionally, deduplicated prediction sets maintain adaptiveness while being more efficient and stable for risk assessment under different risk levels.

## 1 Introduction

In recent years, there has been swift progress in the development of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) (Zhao et al., ; Wu et al., ).
MLLMs extend the capabilities of LLMs by integrating and processing information from multiple modalities, including text, vision, and audio (Bai et al., ; Zhu et al., ).
However, MLLMs are proven to exhibit significant drawbacks in trustworthiness, such as hallucination (Rawte et al., ; Zhang et al., ; Wang et al., ), which results in non-factual information (Ji et al., ) and biased generations (Zou et al., ).
These issues have elicited increasing societal concerns regarding the reliable deployment of foundation models in consumer-facing applications (Qian et al., ).

Uncertainty estimation (Ye et al., ) provides valuable insights into the trustworthiness of model generations.
Prior work performs probabilistic modeling (Li et al., ), develops entropy-based measures (Wang et al., ; Duan et al., ; Farquhar et al., ), and generates multiple responses to analyze the output space (Lin et al., ; Wang et al., ).
However, these approaches cannot provide guarantees of the error rate, and forcing a measure to predict a single data point is overly restrictive (Cresswell et al., ).
Split Conformal Prediction (SCP) (Papadopoulos et al., ; Angelopoulos and Bates, ; Campos et al., )
is a promising candidate to tackle these challenges above, which constructs statistically rigorous prediction sets with the hallmark of risk control (Angelopoulos et al., ; Farinhas et al., ).
Furthermore, SCP allows the calibrated set size to vary based on the model’s uncertainty regarding a particular input.

SCP has been successfully applied in language modeling (Campos et al., ).
However, existing methods either (1) modify the original model outputs to ensure factuality with desired probability (Mohri and Hashimoto, ; Cherian et al., ), potentially leading to uninformative and vague generations, (2) rely on internal token sequence logits (Quach et al., ), or (3) are restricted to multiple-choice settings (Kumar et al., ; Ye et al., ; Kostumov et al., ).
Adapting SCP for proprietary MLLMs in practical open-ended video question-answering (VideoQA) applications presents several challenges:
(a) Adaptability. Unlike closed-ended tasks equipped with fixed options covering the correct answers, the output space for each VideoQA task is unbounded, and there may not be an acceptable response within the sampled generations;
(b) Reliability. Logits or verbalized confidence levels are often miscalibrated, leading to biased prediction sets;
(c) Flexibility. For some API-only MLLMs, users lack access to internal model information like logits.

Figure: Figure 1: Pipeline of TRON illustrated by two open-ended VideoQA examples. Step 1 calibrates the minimum number of responses required to sample for each test data point on the calibration set; Step 2 calibrates the threshold to identify high-quality responses. A larger prediction set size reflects higher risk (model uncertainty) during the current VideoQA task.
Refer to caption: extracted/5915776/figure/overview.jpg

In this paper, we aim to tackle these challenges by developing a two-step framework for risk control and assessment (TRON).
We illustrate the pipeline of TRON via two open-ended VideoQA samples in Figure 1.
The workflow begins by determining whether acceptable responses exist in the sampled generations, akin to options in closed-ended tasks.
To this end, we propose a novel conformal score that controls the minimum number of response samples required for each calibration data point.
For a new data point, we define the number of samples as the approximate $1-\alpha$ quantile of all conformal scores in the calibration set, denoted as $\hat{r}$.
Then, the error rate for the candidate set of each test data point failing to encompass an acceptable response is controlled by the risk level $\alpha$ (i.e., $\leq\alpha$).

Once the risk level for acceptable responses being sampled is specified, we evaluate the reliability of each response.
Inspired by self-consistency theory (Wang et al., ; Li et al., ), we propose using the frequency of each response as a proxy for confidence(^1^11Response frequency positively correlates with average real generative probability. As the frequency increases, there is a corresponding growing level of confidence in the model’s responses (Su et al., ).), compensating for the limitations of internal logits information and verbalized confidence scores,
and define the nonconformity score of each calibration data as one minus the frequency score of any one acceptable response in the candidate set.
Then, we specify another risk level $\beta$ and calculate the approximate $1-\beta$ quantile of all nonconformity scores, denoted as $\hat{s}$.
The quantile serves as a threshold to identify high-quality responses within the candidate set of each test data point.
Finally, we guarantee that the miscoverage rate of acceptable responses in the calibrated prediction sets is statistically controlled by two risk levels, $\alpha$ and $\beta$ (i.e., $\leq\alpha+\beta-\alpha\beta$), while the set size reflects the model uncertainty to each query.

We evaluate our framework on two closed-ended VideoQA datasets (e.g., Video-MME (Fu et al., ) and NExT-QA (Xiao et al., )), and two open-ended VideoQA datasets (e.g., MUSIC-AVQA (Li et al., ) and MSVD (Chen and Dolan, )), utilizing five open-source MLLMs(e.g., VideoLLaMA (Zhang et al., ), PandaGPT (Su et al., ), and NExT-GPT (Wu et al., )), and three closed-source MLLMs(e.g., Gemini (Reid et al., ) and GPT-4o mini (OpenAI, )).
Experimental results demonstrate that TRON guarantees the error rates in both steps across various user-specified risk levels.
Additionally, the deduplicated average set size provides stable uncertainty estimation of MLLMs across various risk levels in open-ended VideoQA tasks, complementing the accuracy metric for more comprehensive evaluations.
Furthermore, we investigate other black-box measurements in the development of the nonconformity score, resulting in more efficient predictions and consistently rigorous guarantees.
To summarize, our contributions are the following:

- •
We introduce a general framework for risk control and assessment, applicable to any MLLM that supports sampling for both open-ended and closed-ended VideoQA tasks.
- •
We propose a novel conformal score to derive the minimum sample size that bounds the miscoverage rate, extending SCP to open-ended contexts, and then develop the nonconformity score based on black-box measures, which results in rigorous guarantees as well.
- •
We explore the impact of semantic redundancy on risk assessment leveraging prediction set in open-ended settings, and disclose that the deduplicated set size provides more stable uncertainty estimation across various risk levels, which serves as a promising evaluation metric for MLLMs in VideoQA tasks.

## 2 Related Work

#### Split Conformal Prediction and Risk Control.

Our work extends the variants of the SCP framework for correctness (coverage) guarantees and risk control in language modeling.
Existing methods either modify model generations and achieve conformal factuality guarantees (Mohri and Hashimoto, ; Cherian et al., ) or guarantee the ground-truth coverage in Multiple-Choice Question-Answering (MCQA) tasks (Kumar et al., ; Ye et al., ; Kostumov et al., ).
The previous approaches may result in vague and uninformative responses, constrained by the correctness of the original generation, while the latter frameworks are limited to closed-ended settings.
In open-ended scenarios, (Quach et al., ) develops a stopping rule with slightly relaxed coverage criteria utilizing the internal logits information, restricting the applicability to API-only models,
whereas (Su et al., ) leverages black-box uncertainty but fails to align the nonconformity score with correctness, leading to theoretically loose guarantees.
Our approach addresses these gaps by introducing a novel two-step framework that first derives the minimum number of response samples to approximate a “closed-ended” condition, where there is at least one acceptable option provided, followed by identifying reliable responses based on self-consistency theory, which features (1) rigor, controlled by two user-specified risk levels, $\alpha$ and $\beta$, and (2) versatility, applicable to any MLLM that supports sampling in both closed-ended and open-ended settings.

#### Risk Assessment and Uncertainty Estimation.

Recent work performs probabilistic modeling (Li et al., ), develops entropy-based uncertainty measures (Wang et al., ; Duan et al., ; Farquhar et al., ), and generate multiple responses to analyze the output space (Lin et al., ; Wang et al., ).
However, these approaches are heuristic, and forcing a measure to predict a single data point is overly restrictive (Cresswell et al., ).
(Ye et al., ) and (Kostumov et al., ) apply SCP to MCQA tasks, creating conformal sets based on a measure of uncertainty (risk), where the set size represents the confidence of the foundational model regarding a particular input, with larger sets indicating higher risk.
In open-ended scenarios, there may be redundant responses within the prediction sets due to the randomness of sampling (Quach et al., ).
Our work explores the performance of set size on risk assessment before and after deduplication and observes that the original set size becomes unstable in uncertainty estimation across different MLLMs as the error rate changes, while the calibrated set size reliably evaluates MLLMs across various risk levels.

## 3 Methodology

In this section, we introduce how our framework extends SCP techniques to open-ended VideoQA tasks based on a novel conformal score for sampling responses.
In addition, to address the limitation of internal logits on API-only MLLMs, we develop the nonconformity score for identifying high-quality responses based on black-box measures.

### 3.1 Preliminaries

Following standard SPC for risk control (Angelopoulos and Bates, ; Angelopoulos et al., ), we assume a calibration dataset ${\mathcal{D}}_{cal}=\left\{\left(X_{i},Y_{i}\right)\right\}_{i=1}^{N}$ of $N$ query-answer pairs and an independent and identically distributed (i.i.d.)(^2^22Split conformal prediction requires that the test data points are drawn from the same distribution as the calibration data points.) test point $\left(x_{test},y_{test}\right)$.
Our goal of risk control is to construct prediction sets that are calibrated to bound the correctness miscoverage rate

$$ ${\mathbb{P}}\left\{y_{test}\notin{\mathcal{C}}\left(x_{test}\right)\right\} \leq\varepsilon,$ (1) $$

where $\varepsilon$ is the user-specified risk level, and ${\mathcal{C}}$ is a function formed utilizing ${\mathcal{D}}_{cal}$.
Note that the risk level is marginal (average) over the draw of both the calibration and test data points.

For each query of calibration data $X_{i}$, we generate $M_{i}$ response samples $\left\{\hat{y}_{m}^{(i)}\right\}_{m=1}^{M_{i}}$ and demand that there is at least one correct response in the candidate set (i.e., $Y_{i}\in\left\{\hat{y}_{m}^{(i)}\right\}_{m=1}^{M_{i}}$). For the test point, we also generate $M_{test}$ response samples $\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{M_{test}}$.
Note that we are currently unsure if the candidate set of test data encompasses acceptable responses.

### 3.2 Conformal Risk Control

#### Sampling.

As we construct a prediction set by selecting high-quality responses from the candidate set, we first perform a simple calibration step for the setting of $M_{test}$ to control the risk of $\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{M_{test}}$ failing to encompass acceptable responses, so that we can approximate closed-ended settings with a specific probability.
We develop a novel conformal score for each calibration data point, which determines the minimum sampling threshold that ensures $Y_{i}\in\left\{\hat{y}_{m}^{(i)}\right\}_{m=1}^{M_{i}}$

$$ $r\left(X_{i},Y_{i}\right):=\sup\left\{M_{i}:\forall M_{i}^{{}^{\prime}}<M_{i}, Y_{i}\notin\left\{\hat{y}_{m}^{(i)}\right\}_{m=1}^{M_{i}^{{}^{\prime}}}\right\},$ (2) $$

and define $\hat{r}$ to be the $\frac{\left\lceil\left(N+1\right)\left(1-\alpha\right)\right\rceil}{N}$ quantile of $r_{1},\cdots,r_{N}$ ($r_{i}=r\left(X_{i},Y_{i}\right)$): $\hat{r}=r_{\left\lceil\left(N+1\right)\left(1-\alpha\right)\right\rceil}$.
Then, we set $M_{test}=\hat{r}$ and obtain the upper bound of the error rate

$$ ${\mathbb{P}}\left(y_{test}\notin\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{ \hat{r}}\right)\leq\alpha.$ (3) $$

A complete proof of Eq. 3 is presented in Appendix A.1.

#### Identification.

Inspired by self-consistency theory (Wang et al., ; Li et al., ; Wang et al., ), which states that a repetitively sampled response is viewed as a form of consistency linked to higher confidence in the response, we perform a simple semantic clustering process on the candidate set and obtain the frequency of each response, denoted as $F\left(\hat{y}_{m}^{(i)}\right)$.
For a detailed description of semantic clustering, refer to Appendix B.1.
Then, we define the nonconformity score of the $i$-th calibration data point as one minus the frequency of the acceptable response, denoted as $\hat{y}_{ref}^{(i)}$, within the candidate set, which is semantically equivalent to the label $Y_{i}$ (i.e., $\hat{y}_{ref}^{(i)}\Leftrightarrow Y_{i}$).
$A\Leftrightarrow B$ represents a bidirectional entailment (MacCartney and Manning, ; Mohri and Hashimoto, ; Farquhar et al., ) between A and B (i.e., equivalence).
The specific evaluation for bidirectional entailment is provided in Appendix B.1.
In this case, the nonconformity score is formulated as

$$ $s\left(X_{i},Y_{i}\right):=1-F\left(\hat{y}_{ref}^{(i)}\right).$ (4) $$

Note that $\hat{y}_{ref}^{(i)}$ refers to any acceptable response in the candidate set that is semantically equivalent to the reference answer $Y_{i}$.
Similar to $\hat{r}$, we define $\hat{s}$ to be the $\frac{\left\lceil\left(N+1\right)\left(1-\beta\right)\right\rceil}{N}$ quantile of $s_{1},\cdots,s_{N}$ ($s_{i}=s\left(X_{i},Y_{i}\right)$): $\hat{s}=s_{\left\lceil\left(N+1\right)\left(1-\beta\right)\right\rceil}$.
Then, we construct the prediction set by post-processing the candidate set into a set of responses with low non-conformity scores

$$ ${\mathcal{C}}\left(x_{test}\right):=\left\{\hat{y}_{m^{\prime}}^{(test)}\in \left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}:s\left(x_{test},\hat{y}_{m ^{\prime}}^{(test)}\right)\leq\hat{s}\right\}.$ (5) $$

Finally, we guarantee the upper bound of the correctness miscoverage rate

$$ $\begin{split}&{\mathbb{P}}\left\{y_{test}\notin{\mathcal{C}}\left(x_{test} \right)\right\}\\ &={\mathbb{P}}\left(y_{test}\notin\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{ \hat{r}}\right)+{\mathbb{P}}\left(s_{test}>\hat{s}\mid y_{test}\in\left\{\hat{ y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}\right)\\ &\leq\varepsilon,\end{split}$ (6) $$

where $\varepsilon=\alpha+\beta-\alpha\beta$.
The complete proof of Eq. 6 is given in Appendix A.2.

#### Extensibility.

$F\left(\cdot\right)$ can be any measure that reflects the reliability of each response.
In the case of frequency, an appropriate sample size is necessary to approximate the output distribution and accurately represent the confidence level.
We also investigate other measurements in Section 4.4 and evaluate their impact on risk control and assessment and prediction efficiency.

## 4 Empirical Evaluations

Figure: (a) EER vs. risk level $\alpha$.
Refer to caption: extracted/5915776/figure/coverage_alpha.png

### 4.1 Experimental Settings

#### Models.

We employ five open-source MLLMs, including VideoLLaMA-7B (Zhang et al., ), VideoLLaMA-13B, PandaGPT-7B (Su et al., ), PandaGPT-13B, and NExT-GPT (Wu et al., ), and three closed-source MLLMs, including Gemini-1.5-Flash (Reid et al., ), Gemini-1.5-Pro, and GPT-4o mini (OpenAI, ).
Note that we utilize all five open-source MLLMs as if they are API-only MLLMs, i.e., it assumes no access to any internal information of MLLMs.

#### Datasets.

We consider two multiple-choice VideoQA datasets: Video-MME (VMME) (Fu et al., ) and NExT-QA (NEXT) (Xiao et al., ), where we only utilize the multiple-choice section of NEXT, and two open-ended VideoQA datasets: MUSIC-AVQA (MUSC) (Li et al., ) and MSVD (Chen and Dolan, ).
More details of the dataset utilization can be found in Appendix B.

#### Evaluation Metrics.

We utilize the Empirical Error Rate (EER) to assess whether we control the risk of miscoverage by the two user-specified risk levels (Angelopoulos and Bates, ).
Note that in the first stage, miscoverage refers to not sampling the acceptable responses, and this EER is controlled by $\alpha$.
In the second stage, miscoverage refers to the calibrated prediction sets failing to cover the acceptable responses, and this EER is bounded by both $\alpha$ and $\beta$.
Additionally, we adopt the commonly used Accuracy (ACC) and the Average Prediction Set Size (APSS) (Kostumov et al., ; Ye et al., ) to evaluate MLLM predictions.
More details can be found in Appendix E.

### 4.2 Results for Risk Control

Following prior work (Mohri and Hashimoto, ; Su et al., ; Quach et al., ), we randomly split our datasets in half by default, calibrating the sampling number and threshold for identifying high-quality responses on the first half, i.e., the calibration set, and measure the EER (i.e., miscoverage rate) at various risk levels (i.e., upper bound) in two steps on the second half, i.e., the test set.
We plot the empirical results in the two steps utilizing the GPT-4o mini model on four datasets in Figure 2(a) and Figure 2(b), and defer similar plots employing other MLLMs to Appendix F.

In the first step, we aim to address the challenge of determining whether acceptable responses are covered by the candidate set in open-ended VideoQA tasks.
Figure 2(a) demonstrates that by deriving the minimum number of responses required to sample for each test data point on the calibration set based on our developed conformal score, we achieve statistically rigorous guarantees on EER.
The marginal correctness miscoverage rates by the sampled responses on the test set are consistently bounded by various predetermined risk levels.
In this way, we can approximate the fixed options provided in closed-ended tasks through sampling, despite the presence of duplicate options.

In the second step, we attempt to overcome the limitations of previous approaches, which rely on internal model logits or verbalized confidence scores.
We specify the risk level in the first step to be 0.1 (i.e., $\alpha=0.1$), and then measure the correctness miscoverage rates by the prediction sets, which are calibrated by our devised nonconformity score based on self-consistency theory.
Figure 2(b) illustrates the validity of identifying high-quality responses based on their frequency scores with the candidate sets.
The miscoverage rates on the calibrated prediction sets are strictly bounded by the risk level $\varepsilon$ ($=\alpha+\beta-\alpha\beta$) at various settings of $\beta$.
In this way, we obtain an intuitive confidence level by analyzing the frequency of each response in the candidate set, adapting our framework to API-only MLLMs.
In terms of the black-box reliability measurement, we also evaluate the development of nonconformity score based on semantic diversity in Section 4.4.

Empirical evaluations demonstrate that our risk control framework matches our theory in practice, as the correctness miscoverage rates in both steps never exceed the user-specified risk level.
Note that EER is marginal (average) over the test set, not conditional to individual test data points.

### 4.3 Results for Risk Assessment

Figure: Figure 3: Comparison of APSS before and after duplication at different risk levels ($\varepsilon$) across four open-source MLLMs on the VMME dataset.
Refer to caption: extracted/5915776/figure/final_plot.png

Conformal sets are constructed adaptively to each particular input (Angelopoulos and Bates, ), so that the set size gives us an indicator of the model’s uncertainty (or the risk in the current decision-making).
Prior work (Ye et al., ; Kostumov et al., ) utilize APSS to benchmark LLMs and Vision-Language Models (VLMs) through risk assessment in MCQA tasks.
In contrast to closed-ended scenarios, where MLLMs are provided a predetermined set of options that encompasses the correct answer and produce a prediction set without internal duplicates, open-ended VideoQA tasks feature their unfixed and unbounded output spaces.
Consequently, duplicate responses are inevitable during the sampling process, which leads to semantic redundancy(^3^33In the open-ended VideoQA task, sampling multiple responses from MLLMs can result in the candidate set containing identical answers or answers that are syntactically or lexically distinct but semantically equivalent.) in the prediction set after filtering for high-quality responses based on the nonconformity score.

In this section, we investigate the impact of semantic redundancy in the prediction set when employing the APSS metric to evaluate MLLMs, to address the research gap in open-ended settings.
We fix the risk level in the first stage at 0.1 (i.e., $\alpha=0.1$) and measure the APSSs for different values of $\varepsilon$ ($=\alpha+\beta-\alpha\beta$), utilizing four open-source MLLMs on the VMME dataset.
As demonstrated in Figure 4, the original APSS is sensitive to the error rate, leading to ambiguity in the evaluation of MLLMs at different risk levels, while after removing semantically redundant responses, the final APSS can consistently decrease as the error rate lowers, and accurately distinguish between foundational models of varying performance at different risk levels.
Additionally, the post-processed prediction sets result in more efficient decision-making.
Note that we still utilize bidirectional entailment detailed in Appendix B to determine the semantic equivalence between two responses.

Table: Table 1: Evaluations of the ACC and APSS metrics utilizing five open-source and three closed-source MLLMs across four open-ended and closed-ended datasets.

Given the adaptiveness and stability of the deduplicated APSS, we propose utilizing uncertainty/risk, represented by APSS, to evaluate MLLMs in open-ended VideoQA tasks for the first time.
We specify the risk level $\varepsilon$ to be 0.19 ($\alpha=\beta=0.1$) and evaluate the ACC and APSS metrics across four datasets utilizing eight MLLMs.
Intuitively, MLLMs with higher accuracy should exhibit lower risk when processing a given VideoQA dataset.
However, as shown in Table 1, higher ACC does not necessarily correspond to lower uncertainty and risk.
For instance, although PandaGPT-13B achieves an ACC of 86.04% on the MSVD dataset, higher than GPT-4o mini, which has an ACC of 85.71%, PandaGPT-13B exhibits a higher APSS, indicating greater uncertainty compared to GPT-4o mini. This demonstrates that even with higher ACC, uncertainty, measured by APSS, may not decrease. Similarly, VideoLLaMA-13B and PandaGPT-13B both demonstrate a positive correlation between ACC and APSS on the MSVD dataset.
These findings highlight the importance of considering uncertainty alongside ACC when evaluating MLLMs.

Unlike prior work (Kostumov et al., ) that evaluates MLLMs on vision-language QA datasets, we investigate the impact of the audio modality on MLLM performance in open-ended VideoQA tasks.
Specifically, we extract audio information from the original video, conduct varying degrees of Silence Percentage (SP), and then provide the processed audio along with visual and textual information to MLLMs.
As illustrated in Figure 4, incorporating additional modality information enhances the ACC of MLLMs to some extent.
As the SP increases, we observe significant changes in model uncertainty through the APSS metric, showcasing that introducing audio modality enhances the confidence level of MLLMs’ decision-making.
In light of the observations mentioned above, we hope that our framework can assist in evaluating the performance of MLLMs in VideoQA tasks through risk assessment, while providing guarantees, thereby complementing the accuracy metric for a more comprehensive analysis.

### 4.4 Sensitivity Analyses

#### Splitting Ratio of Calibration and Test Set.

As previously discussed, the calibration set is essentially a set of sufficiently general observation samples, from which we derive the number of response samples required for each test data point and the threshold for identifying high-quality responses in the candidate set based on specific statistical criteria.
In this section, we explore the impact of the ratio of sample sizes between the calibration set and the test set on the final performance, specifically how many resources are needed for our framework to achieve risk guarantees on the test set.
In previous work, we set a ratio of 0.5 by default.
Here, we evaluate the EER when the ratio is set to 0.3 and 0.1.
As shown in Figure 5, when $\alpha$ and $\beta$ are set to 0.1, resulting in an overall risk level (i.e., $\varepsilon$) of 0.19, we achieve consistently strict guarantees of the miscoverage rates across three split ratios utilizing three closed-source MLLMs on the MUSC dataset, which demonstrates the efficiency and stability of our framework for risk control in practical open-ended VideoQA applications.

Figure: Figure 5: EERs on the MUSC dataset utilizing three closed-source MLLMs when varying the split ratio of the calibration and test set. The risk levels $\alpha$ and $\beta$ are both set to be 0.1.
Refer to caption: extracted/5915776/figure/split_ratio.png

Table: Table 2: Comparison of frequency and semantic diversity on the VMME datasets employing four open-source MLLMs. The risk levels $\alpha$ and $\beta$ are both set to be 0.1.

#### Reliability Measurement.

As discussed in Section 3.2, $F\left(\cdot\right)$ in the second step can be any measure that evaluates the reliability of each response within the candidate set.
To address the issues of logit access utilizing API-only MLLMs and the overconfidence in the verbalized reliability scores, we leverage the most intuitive frequency score based on self-consistency theory.
In this section, we utilize the semantic diversity, formulated as $\textstyle\sum_{\hat{y}_{j}^{(i)}\neq\hat{y}_{m}^{(i)}}S\left(\hat{y}_{j}^{(i)
},\hat{y}_{m}^{(i)}\right)F\left(\hat{y}_{j}^{(i)}\right)$, as the reliability score of the $m$-th response with the $i$-th candidate set.
We set the risk levels $\alpha$ and $\beta$ to be 0.1 (i.e., $\varepsilon=0.19$), and evaluate the APSS and EER metrics on the VMME dataset utilizing four open-source MLLMs.
The comparison results in Table 2 show that the employment of semantic diversity results in smaller APSSs across four MLLMs, indicating more efficient predictions, while the EER metric is always bounded by the risk level.
This finding demonstrates the stability of our framework when using different reliability measures. It also points to a promising direction for further research on the impact of nonconformity score designs and their ability to represent agreement between question-and-answer pairs on prediction efficiency.

## 5 Conclusion

Deploying MLLMs in practical VideoQA applications requires reliable risk control and assessment.
In this paper, we present a general two-step framework for risk control by calibrating a conformal score to sample new generations and a nonconformity score to identify high-quality responses, which provides statistically and theoretically rigorous guarantees on the correctness miscoverage rate.
Our method bridges the gap between SCP and closed-source MLLMs, and it can also be applied to other generative models (e.g., LLMs) that support sampling in any language generation tasks.
Furthermore, we conduct risk assessments and explore the issue of semantic redundancy in prediction sets within open-ended contexts for the first time, which results in a promising evaluation metric for MLLMs in open-ended VideoQA tasks via the average set size after deduplication.
Additionally, we analyze reliability measurement in the development of the nonconformity score for more efficient predictions.
We hope that our framework can be helpful for human-in-the-loop decision-making and human-AI teams.

## Limitations

In our work, guarantees are marginal over the test set, not conditional to individual data points.
In addition, our framework does not deviate from the fundamental exchangeability assumption of SCP, and we will explore risk control under distribution shift conditions in future work.

## Appendix A Proofs

### A.1 Proof of equation 3

This section presents a standard proof of validity for the upper bound of the error rate, as defined in Eq.3.
For completeness, we restate the definition of the conformal score in Eq.2.
We define the minimum sampling threshold that ensures $Y_{i}\in\left\{\hat{y}_{m}^{(i)}\right\}_{m=1}^{M_{i}}$ as

$$ $\begin{split}r_{i}&=r\left(X_{i},Y_{i}\right)\\ &:=\sup\left\{M_{i}:\forall M_{i}^{{}^{\prime}}<M_{i},Y_{i}\notin\left\{\hat{y }_{m}^{(i)}\right\}_{m=1}^{M_{i}^{{}^{\prime}}}\right\},\end{split}$ (7) $$

sort the conformal scores so that $r_{1}\leq\cdots\leq r_{N}$, and calculate the $\frac{\left\lceil\left(N+1\right)\left(1-\alpha\right)\right\rceil}{N}$ empirical quantile of $r_{1},\cdots,r_{N}$:

$$ $\begin{split}\hat{r}&:=\inf\left\{r:\frac{\left|\left\{i:r_{i}\leq r\right\} \right|}{N}\geq\frac{\left\lceil\left(N+1\right)\left(1-\alpha\right)\right \rceil}{N}\right\}\\ &=r_{\left\lceil\left(N+1\right)\left(1-\alpha\right)\right\rceil}.\end{split}$ (8) $$

We proceed by noting the equality of the two events based on the definition in Eq.2:

$$ $\left\{y_{test}\notin\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}\right \}=\left\{r_{test}>\hat{r}\right\}.$ (9) $$

By exchangeability(^4^44The standard framework of conformal prediction assumes exchangeability of all data points, and exchangeability is weaker than the independent and identically distributed (i.i.d.) assumption.) of the calibration and test data points, we have

$$ ${\mathbb{P}}\left(r_{test}\leq r_{i}\right)=\frac{i}{N+1}.$ (10) $$

Then, we guarantee the upper bound of the probability that the candidate set fails to encompass an acceptable response based on a user-specified risk level, denoted as $\alpha$:

$$ $\begin{split}{\mathbb{P}}\left(y_{test}\notin\left\{\hat{y}_{m}^{(test)}\right \}_{m=1}^{\hat{r}}\right)&={\mathbb{P}}\left(r_{test}>\hat{r}\right)\\ &={\mathbb{P}}\left(r_{test}>r_{\left\lceil\left(N+1\right)\left(1-\alpha \right)\right\rceil}\right)\\ &=1-{\mathbb{P}}\left(r_{test}\leq r_{\left\lceil\left(N+1\right)\left(1- \alpha\right)\right\rceil}\right)\\ &=1-\frac{\left\lceil\left(N+1\right)\left(1-\alpha\right)\right\rceil}{N+1}\\ &\leq\alpha.\end{split}$ (11) $$

### A.2 Proof of equation 6

Given that we select trustworthy responses from each candidate set to build the prediction set, let us formally prove the upper bound of the error rate (i.e., $\varepsilon$) in Eq. 6, building upon the foundation established in Eq. 3.
If there are no acceptable responses in the candidate set, event $\left\{y_{test}\notin{\mathcal{C}}\left(x_{test}\right)\right\}$ is a certain event, and the probability of miscoverage is

$$ ${\mathbb{P}}\left(y_{test}\notin{\mathcal{C}}\left(x_{test}\right)\mid y_{test }\notin\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}\right)={\mathbb{P}} \left(y_{test}\notin\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}\right).$ (12) $$

Conversely, if at least one acceptable answer exists in the candidate set (i.e., $y_{test}\in\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}$), we then redefine the non-conformity score here.
First, we determine the acceptable response of the $i$-th calibration data point $\hat{y}_{ref}^{(i)}$, which is semantically equivalent to the correct answer $Y_{i}$.
Then we formulate the non-conformity score as

$$ $\begin{split}s_{i}&=s\left(X_{i},Y_{i}\right)\\ &:=1-F\left(\hat{y}_{ref}^{(i)}\right),\end{split}$ (13) $$

where $F\left(\hat{y}_{ref}^{(i)}\right)$ is the frequency of $\hat{y}_{ref}^{(i)}$ in the candidate set.
Similar to Eq. 8, we sort the non-conformal scores so that $s_{1}\leq\cdots\leq s_{N}$, and define the $\frac{\left\lceil\left(N+1\right)\left(1-\beta\right)\right\rceil}{N}$ quantile of $s_{1},\cdots,s_{N}$:

$$ $\begin{split}\hat{s}&:=\inf\left\{s:\frac{\left|\left\{i:s_{i}\leq s\right\} \right|}{N}\geq\frac{\left\lceil\left(N+1\right)\left(1-\beta\right)\right \rceil}{N}\right\}\\ &=s_{\left\lceil\left(N+1\right)\left(1-\beta\right)\right\rceil}.\end{split}$ (14) $$

Then, we construct the prediction set following

$$ ${\mathcal{C}}\left(x_{test}\right):=\left\{\hat{y}_{m^{\prime}}^{(test)}\in \left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}:s\left(x_{test},\hat{y}_{m ^{\prime}}^{(test)}\right)\leq\hat{s}\right\}.$ (15) $$

Given $y_{test}\in\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}$, we obtain

$$ $\begin{split}\left\{y_{test}\notin{\mathcal{C}}\left(x_{test}\right)\right\}&= \left\{s\left(x_{test},y_{test}\right)>\hat{s}\right\}\\ &=\left\{s_{test}>\hat{s}\right\}.\end{split}$ (16) $$

In this case, the probability of miscoverage is

$$ ${\mathbb{P}}\left(y_{test}\notin{\mathcal{C}}\left(x_{test}\right)\mid y_{test }\in\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}\right)={\mathbb{P}} \left(s_{test}>\hat{s}\mid y_{test}\notin\left\{\hat{y}_{m}^{(test)}\right\}_{ m=1}^{\hat{r}}\right).$ (17) $$

Similarity to Eq. 10, we have

$$ $\begin{split}{\mathbb{P}}\left(s_{test}>s_{i}\right)&=1-{\mathbb{P}}\left(s_{ test}\leq s_{i}\right)\\ &=1-\frac{i}{N+1}.\end{split}$ (18) $$

Finally, we achieve risk control and guarantee the upper bound of the correctness miscoverage rate:

$$ $\begin{split}&{\mathbb{P}}\left\{y_{test}\notin{\mathcal{C}}\left(x_{test} \right)\right\}\\ &={\mathbb{P}}\left(y_{test}\notin\left\{\hat{y}_{m}^{(test)}\right\}_{m=1}^{ \hat{r}}\right)+{\mathbb{P}}\left(s_{test}>\hat{s}\mid y_{test}\in\left\{\hat{ y}_{m}^{(test)}\right\}_{m=1}^{\hat{r}}\right)\\ &=1-\frac{\left\lceil\left(N+1\right)\left(1-\alpha\right)\right\rceil}{N+1}+ \left(1-\frac{\left\lceil\left(N+1\right)\left(1-\beta\right)\right\rceil}{N+1 }\right)\cdot\frac{\left\lceil\left(N+1\right)\left(1-\alpha\right)\right \rceil}{N+1}\\ &=1-\frac{\left\lceil\left(N+1\right)\left(1-\beta\right)\right\rceil}{N+1} \cdot\frac{\left\lceil\left(N+1\right)\left(1-\alpha\right)\right\rceil}{N+1} \\ &\leq 1-\left(1-\alpha\right)\left(1-\beta\right)\\ &\leq\alpha+\beta-\alpha\beta\\ &\leq\varepsilon.\end{split}$ (19) $$

## Appendix B Implementation Specifics

### B.1 Semantic Clustering Process

Figure: Algorithm 1 The pseudo-code for semantic clustering.

Note that since there may not be a semantically equivalent transfer between the sampled responses in open-ended tasks, the sum of the frequency scores corresponding to all responses in the candidate set may be uncertain, and we utilize the normalized frequency score for each response.

#### Bidirectional Entailment.

As noted in Section 3.2 and Algorithm 1, $A_{1}\Leftrightarrow A_{2}$ represents that $A_{1}$ is semantically equivalent to $A_{2}$.
Following prior work [Farquhar et al., ], we utilize a Natural Language Inference (NLI) classifier for this evaluation.
Specifically, we employ the off-the-shelf DeBERTa-large-mnli model [He et al., ] as the backbone, and the NLI classifier typically takes $\left[A_{1},A_{2}\right]$ as input and predicts logits for three distinct semantic relationships: entailment, neutral, and contradiction.
Note that this function has directionality and we only consider $A_{1}$ and $A_{2}$ to be equivalent when the logit scores corresponding to the entailment are at their highest for both inputs $\left[Q\cup A_{1},Q\cup A_{2}\right]$ and $\left[Q\cup A_{2},Q\cup A_{1}\right]$ (i.e., bidirectional).
Here, $Q$ denotes the question, and $\cup$ means we evaluate the semantic relationship between query-response pairs.

## Appendix C Datasets

Video-MME (VMME) [Fu et al., ] is a multi-modal evaluation benchmark for MLLMs in video analysis, which includes 900 manually selected videos totaling 254 hours and resulting in 2,700 MCQA pairs.
NExT-QA (NEXT) [Xiao et al., ] is a benchmark for VideoQA that focuses on causal and temporal action reasoning, moving beyond simple scene descriptions.
It features both multiple-choice and open-ended QA tasks. However, we only selected the multiple-choice part for our experiments.
We use the sub-test set of MCQA, which consists of 2,312 QA pairs.
MUSIC-AVQA (MUSC) [Li et al., ] is an open-ended VideoQA dataset designed to evaluate comprehensive multimodal understanding and spatio-temporal reasoning over audio-visual scenes.
We use the test set of MUSIC-ACQA, which consists of 9,185 QA pairs. MSVD [Chen and Dolan, ] is an open-ended dataset that provides a large-scale, highly parallel text collection.
We use 2,519 QA pairs from this dataset. In our experiments, we extracted the audio from the videos and integrated it with the corresponding visual and textual data before inputting them into the MLLMs.

## Appendix D Hyperparameters

We employ multinominal sampling for responses which are used to construct prediction sets. The temperature of generation for all MLLMs is set to 1.0. All the experiments are conducted on a server with one Intel Xeon Gold 6326 CPU and 7 NVIDIA A6000 GPUs.

## Appendix E Evaluation Metrics

Empirical Error Rate (EER) evaluates the validity of our framework for risk control.
In the first stage, we expect that, under the condition of exchangeability, we calibrate the number of response samples of each test data point based on the calibration set so that the candidate set encompasses at least one acceptable response with a specific probability.
To this end, the EER on the test set, ${\mathcal{D}}_{test}$, should be bounded by the risk level $\alpha$ (i.e., $\leq\alpha$).
At this point, we can approximate the closed-ended settings utilizing the candidate set.
We redefine each sample in the test set as $\left(x_{i},y_{i}\right)$ and the candidate set of the $i$-th data point as ${\mathcal{S}}\left(x_{i}\right)$.
Then, EER is formulated as

$$ $\mathrm{EER}=\frac{1}{|{\mathcal{D}}_{test}|}\sum_{(x_{i},y_{i})\in{\mathcal{D }}_{test}}\textbf{1}\left\{y_{i}\in{\mathcal{S}}\left(x_{i}\right)\right\}$ (20) $$

In the second stage, we expect that we can identify the acceptable response within the candidate set, which is also said the prediction set encompasses the correct answer, based on the nonconformity score with a user-specified probability.
To this end, the EER on the test set should be bound by $\alpha$ and another risk level $\beta$ (i.e., $\leq\alpha+\beta-\alpha\beta.$).
In the evaluations, we set $\alpha=0.1$ by default and measure EER at various values of $\beta$.
At this point, the upper bound of EER is $0.1+\beta-0.1\beta$, and EER is formulated as

$$ $\mathrm{EER}=\frac{1}{|{\mathcal{D}}_{test}|}\sum_{(x_{i},y_{i})\in{\mathcal{D }}_{test}}\textbf{1}\left\{y_{i}\in{\mathcal{C}}(x_{i})\right\}$ (21) $$

Average Prediction Set Size (APSS) measures the average size of all prediction sets required to guarantee the EER on the test set.
A larger APSS indicates greater uncertainty/risk, while a smaller APSS reflects higher efficiency and confidence.
At this point, APSS is formulated as

$$ $\mathrm{APSS}=\frac{1}{|{\mathcal{D}}_{test}|}\sum_{(x_{i},y_{i})\in{\mathcal{ D}}_{test}}|{\mathcal{C}}(x_{i})|$ (22) $$

Accuracy (ACC) assesses the correctness of MLLMs’ predictions, serving as a foundational benchmark for model performance.
In our experiments, we employ the most frequent (reliable) response within the candidate set of each test data point, denoted as $\hat{y}_{ref}^{(i)}$, as the model output to measure the total accuracy, which is formulated as.

$$ $\mathrm{ACC}=\frac{1}{|{\mathcal{D}}_{test}|}\sum_{(x_{i},y_{i})\in{\mathcal{D }}_{test}}\textbf{1}\left\{\hat{y}_{ref}^{(i)}=y_{i}\right\}$ (23) $$

## Appendix F Additional Experimental Analysis

Figure: (a) EER vs. risk level $\varepsilon$.
Refer to caption: extracted/5915776/figure/gemini-flash.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Do you foresee any major modifications needed for TRON to control risk in scenarios involving distribution shifts, where calibration and test distributions differ?
- Would it be feasible to incorporate dynamic adjustments to prediction set sizes based on task difficulty or user preferences in real time? Are there challenges with balancing efficiency and robustness in such a dynamic setting?
- Could access to model internals like logits help improve TRON's performance?
- Could reliance on frequency-based nonconformity scores lead to biases in the types of responses included in the prediction set, especially in cases where the model’s sampling is limited?
- Have you observed any variance in EER across different model architectures or response generation and sampling methods (for example, cases where EER can go outside the bounds set by alpha and beta)
- How easy is it to generalize this approach beyond VideoQA to other open-ended tasks? Are there any major limitations or requirements for generalization?
soundness: 3
strengths: - The paper's two-step risk control methodology addresses general shortcoming of the conformal prediction method and provides statistical guarantees for error rates, increasing the reliability of MLLM responses.
- Since open ended tasks are more challenging due to large number of possible generation, this method (two step approach, use of semantic similarity) seems to dynamically adapt well to provide flexible prediction set sizes for complex and generic generative scenarios (although we need more experimental validation). Error stays within bounds even after filtering step (bounded by alpha + beta - alpha.beta)
- As shown is Figure 3, deduplication of the semantically similar responses helps with more stable error rates and smaller prediction sets. Experiments suggest that semantic diversity can create smaller, more efficient prediction sets (lower APSS) without compromising on accuracy (EER stays within limits).
summary: The paper introduces a two step risk control based framework extending split conformal prediction method for open ended and multimodal (videoQA) tasks. The method applies a conformal score to calibrate the minimum number of responses (samples) needed to ensure coverage. This score defines the prediction set’s error rate at a risk level alpha. It then refines the set using frequency or semantic similarity to identify high quality responses controlled by another risk parameter beta. The overall risk is bounded by a function of alpha and beta to provide statistical consistency guarantees using calibration, sampling and refinement steps. The paper builds upon multiple conformal risk control based methods and addresses the shortcomings with this two step method (to maintain smaller average prediction set size (APSS))  with lower risk thresholds) and applying them to multimodal videoQA setting.
weaknesses: - Authors already mention under limitations that guarantees are not conditional to individual data points but marginal over the test set. With this limitation, it may still be a bottleneck where risk compliance guarantees are needed for critical applications requiring more stringent guarantees and/or compliance requirements.
- more open-ended evaluations and experiments on the open-ended datasets would have shed more light on the strengths and weaknesses of the methods (like Fig 4b). This is a key innovative strength of the method to address open-ended tasks (unlike MCQ) and adoption of this method will depend heavily on understand the strengths of this method in more open-ended generation tasks.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. How does TRON handle outliers in response sampling that may disproportionately affect the frequency-based confidence scoring?
2. Could TRON’s conformal score be further adapted to dynamically adjust the sampling size based on real-time uncertainty measurements?
soundness: 3
strengths: 1. TRON’s two-step approach combines conformal scores and self-consistency theory to establish a flexible and robust risk assessment framework for MLLMs, particularly in open-ended scenarios, where traditional SCP methods fall short.
2. The paper presents extensive experiments across four VideoQA datasets and eight MLLMs, showcasing TRON's effectiveness and consistency in different VideoQA tasks.
3. By avoiding reliance on model logits, TRON is adaptable for API-restricted MLLMs, expanding its usability in various practical applications.
summary: The authors propose a two-step framework, TRON, for assessing and controlling risk in MLLMs, specifically targeting VideoQA tasks. The framework consists of: (1) Sampling Step: This step involves calibrating a conformal score that defines the minimum number of response samples needed to ensure the inclusion of acceptable responses within a certain risk level. (2) Identification Step: In this phase, a nonconformity score based on self-consistency theory identifies high-quality responses within the candidate set. This score accounts for model uncertainty by estimating reliability via response frequency, introducing another risk level to statistically control errors.

The authors carry sufficient experiments on four VideoQA datasets with eight MLLMs and demonstrate that TRON achieves desired error rates within the specified risk bounds. The study further highlights TRON’s adaptability and stability through deduplicated prediction sets that provide more efficient and robust uncertainty estimation across various risk levels.

The authors address limitations in existing SCP methods, which either modify outputs to ensure factuality, rely on internal token sequence logits, or restrict applications to multiple-choice settings. This new approach is versatile, applicable to both open-ended and closed-ended VideoQA tasks, and operates independently of internal model outputs.
weaknesses: I raise the concern that although TRON is evaluated on diverse datasets, it primarily focuses on VideoQA tasks. Could it be tested on additional multimodal tasks to enhance the generalizability of its risk management capabilities?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - It seems that the best practice of the ratio of the calibration and test set is model-dependent. Is there any insight on the ratio selection when applied to different MLLMs?
soundness: 3
strengths: - The paper is easy to follow and understand.
- The proposed method extends SCP to open-ended scenarios by estimating the confidence from frequency.
summary: The paper deals with risk control and assessment for MLLMs. To address the issues of existing work relying on the internal model logits and working in the multiple-choice setting, the authors propose a TRON, a two-step framework for MLLMs supporting sampling for both open-ended and close-ended scenarios. TRON allows controlling error rates by sampling response sets of minimum size and identifying high-quality responses using self-consistency theory. The experiments on VideoQA demonstrate the efficacy on eight MLLMs.
weaknesses: - The confidence estimation in Step 2 relies on the prediction of another model(e.g. DeBERTa-large-mnli). Then it should be at least discussed on the reliability as the semantic classifier. Otherwise, it makes the identification of risk control less convincing.

- It is unclear how the silence percentage is conducted on the audio, and how the conclusion ‘introduce audio modality enhances the confidence level’ (Line371-372) is made. It is shown in Fig. 4 that increasing SPs leads to higher APSS. Also, introducing audio modality seems to only improve VideoLLaMA with SP <50%.   

- It seems that the proposed method is general to LLMs as well. How does the proposed method work for LLMs for both open-ended and close-ended scenarios? The paper claims this advantage but shows no experimental results. This would make the paper more impactful.

- How is the proposed method compared with existing methods for LLMs on such as MCQA? Such a comparison would make the paper more comprehensive.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. **What is the expected computational impact** of using TRON in large-scale applications or real-time risk assessment tasks? Could you provide a more detailed explanation or metrics regarding the processing time required for each step?

2. **Baseline Method Comparison**: Have you considered comparing TRON with simpler risk control baselines? For instance, would a heuristic-based risk control method suffice for certain types of tasks in closed-ended VideoQA? If not, could you clarify how TRON performs specifically better in such cases?

3. **Semantic Redundancy in Open-Ended Tasks**: How does the semantic redundancy analysis handle responses that may be lexically distinct but only partially semantically equivalent? Could this approach potentially overlook responses with subtle but important semantic differences?

4. **Alternative Conformal and Nonconformity Scoring Methods**: Could the proposed conformal and nonconformity scores be further enhanced by alternative methods, such as clustering-based or confidence-based approaches beyond self-consistency theory? If so, what would be the implications for TRON’s current framework?

5. **Additional Real-World Validation**: Have there been any attempts to validate TRON in real-world or industry-specific VideoQA tasks, possibly in collaboration with industry partners? If so, could you share any preliminary insights on its practical performance and potential adaptations? If not, do you plan to include such validation in future work?
soundness: 3
strengths: 1. **Innovative Framework**: The introduction of TRON, a two-step risk control and assessment framework, contributes significantly to the field of MLLM evaluation in both open-ended and closed-ended VideoQA tasks. Its flexibility in applying conformal prediction in open-ended contexts is commendable.

2. **Novel Conformal and Nonconformity Scores**: The paper proposes a unique conformal score for setting the minimum sample size in open-ended tasks and a nonconformity score based on self-consistency theory. These scores provide a rigorous approach to risk control.

3. **Addressing Uncertainty with Redundancy Analysis**: The evaluation of **semantic redundancy** in open-ended settings introduces a new angle to uncertainty measurement, providing a promising metric that complements traditional accuracy.

4. **Comprehensive Experimental Evaluation**: The experiments span multiple datasets, risk levels, and different types of MLLMs, offering a thorough assessment of TRON’s effectiveness in diverse conditions.
summary: The paper presents **TRON**, a two-step framework designed for **risk control and assessment** in multimodal large language models (MLLMs), particularly for **Video Question Answering (VideoQA)** tasks. Addressing challenges in dynamic and open-ended environments, TRON leverages **Split Conformal Prediction (SCP)**, introducing a **conformal score** for sampling response sets and a **nonconformity score** for identifying high-quality responses. Through experiments on several VideoQA datasets, TRON demonstrates the ability to achieve desired error rates across various **user-specified risk levels**. It is also noted for exploring the concept of **semantic redundancy** in prediction sets as an evaluation metric, an area not previously investigated in open-ended contexts.
weaknesses: 1. **Limited Discussion of Practicality and Adaptability**: While TRON provides theoretical guarantees, the practical aspects, such as computational overhead and applicability in real-world scenarios, could be discussed in greater depth.

2. **Insufficient Baseline Comparisons**: The paper lacks a comparison with other standard risk control methods or frameworks that may be relevant, particularly in closed-ended settings or previous SCP applications in MLLMs.

3. **Complexity in Method Presentation**: Some sections of the methodology, particularly the derivation of the conformal and nonconformity scores, lack clarity, which could challenge readers unfamiliar with SCP.

4. **Inconsistent Evaluation Details**: Although the experiments are extensive, details on some evaluation metrics, like APSS and their implications for different risk levels, could be better explained. The choice of models and how each metric was applied in open-ended versus closed-ended settings was also not consistently clarified.

Meta-review: This paper presents a novel pipeline for risk assessment for open-ended LLM generated responses to visual question answering systems.  This work addresses limitations of the current state of the art method of using Split Conformal Prediction (SCP) to construct estimates of the error rate of statistical prediction methods for LLM assessment.  

Extracting a well written summary from one of the reviewers: "The authors propose a two-step framework, TRON, for assessing and controlling risk in MLLMs, specifically targeting VideoQA tasks. The framework consists of: (1) Sampling Step: This step involves calibrating a conformal score that defines the minimum number of response samples needed to ensure the inclusion of acceptable responses within a certain risk level. (2) Identification Step: In this phase, a nonconformity score based on self-consistency theory identifies high-quality responses within the candidate set. This score accounts for model uncertainty by estimating reliability via response frequency, introducing another risk level to statistically control errors.

The authors carry sufficient experiments on four VideoQA datasets with eight MLLMs and demonstrate that TRON achieves desired error rates within the specified risk bounds. The study further highlights TRON’s adaptability and stability through deduplicated prediction sets that provide more efficient and robust uncertainty estimation across various risk levels.

The authors address limitations in existing SCP methods, which either modify outputs to ensure factuality, rely on internal token sequence logits, or restrict applications to multiple-choice settings. This new approach is versatile, applicable to both open-ended and closed-ended VideoQA tasks, and operates independently of internal model outputs.

Reading this myself, I agree with the  major points the reviewers have identified.  Many weaknesses are discussed, but the problem of estimating the error of a predictor under and unknown distribution shift is difficult and I believe that this paper both takes us a notable step closer to solving the problem and makes clear its limitations.

Decision: Accept (Spotlight)