Evaluating Large Language Models for Confidence-based Check Set Selection
Authors: JANE ARLETH LUBANG DELA CRUZ, Iris Hendrickx, Martha Larson
Accepted at: acl25

Evaluating Large Language Models for Confidence-based Check Set Selection

Jane Arleth dela Cruz      Iris Hendrickx      Martha Larson
Center for Language and Speech Technology
Center for Language Studies
Radboud University, Nijmegen, Netherlands
{jane.arleth.delacruz, iris.hendrickx, martha.larson}@ru.nl

Abstract
Large Language Models (LLMs) have shown promise in automating high-labor data tasks, but the adoption of LLMs in high-stake scenarios faces two key challenges: their tendency to answer despite uncertainty and their difficulty handling long input contexts robustly. We investigate commonly used off-the-shelf LLMs’ ability to identify low-confidence outputs for human review through "check set selection"—a process where LLMs prioritize information needing human judgment. Using a case study on social media monitoring for disaster risk management, we define the “check set” as a list of tweets escalated to the disaster manager when the LLM has the least confidence, enabling human oversight within budgeted effort. We test two strategies for LLM check set selection: individual confidence elicitation – LLM assesses confidence for each tweet classification individually, requiring more prompts with shorter contexts, and direct set confidence elicitation – LLM evaluates confidence for a list of tweet classifications at once, using less prompts but longer contexts. Our results reveal that set selection via individual probabilities is more reliable but that direct set confidence merits further investigation. Direct set selection challenges include inconsistent outputs, incorrect check set size, and low inter-annotator agreement. Despite these challenges, our approach improves collaborative disaster tweet classification by outperforming random-sample check set selection, demonstrating the potential of human-LLM collaboration.

1 Introduction
Large language models (LLMs) have significantly advanced the field of natural language processing (NLP) and made it possible to automate a wide range of NLP tasks such as classification, information retrieval, summarization, and many more (Raian et al., 2024; Lee et al., 2022; Cohen et al., 2022; Yang et al., 2024). LLMs can perform these tasks by following prompts, where the enduser provides task details and input data, and the model generates a text response. However, studies show that end users tend to struggle to identify incorrect LLM responses, a problem that can escalate as larger and more complex LLMs are less likely to refrain answering questions (Zhou et al., 2024).

The adoption of LLMs in high-stakes scenarios continues to be a challenge, as assuming LLM-generated responses to be always correct can have severe consequences, i.e., if incorrect outputs influence decision-making processes. Previous studies evaluated LLMs’ ability to express uncertainty which we refer to as confidence elicitation (Xiong et al., 2024; Lin et al., 2022; Tian et al., 2023; Kadavath et al., 2022). Confidence elicitation methods have shown that uncertainty estimates are closely correlated with the accuracy of the prediction (Tian et al., 2023; Kumar et al., 2023). While LLM’s output is challenging to evaluate automatically in high-stakes scenarios, we investigate if we can surface LLM incorrectness using confidence elicitation techniques.

We introduce the check set for the human-LLM collaboration pipeline. The check set is a list of potentially misclassified predictions by the LLM needing review by the end users. While prior research has evaluated the quality of LLM-generated output for escalation to human review, such efforts have typically relied on separate verifier models (Wang et al., 2024; Varshney and Baral, 2023), task-specific fine-tuning (Xin et al., 2021; Chen et al., 2023), or probing the model (Yoshikawa and Okazaki, 2023). In contrast, this study introduces a novel approach in which the check set is directly selected by the off-the-shelf LLM itself.

In this paper, we investigate the LLMs’ check set selection capability with a case study in the field of disaster risk management. For this use case, the check set is a list of tweets escalated to the disaster manager when the LLM has the least confidence,
Figure 1: Check Set Selection Framework. Two strategies for check set selection (1) Individual Confidence Elicitation - LLM assesses confidence for each tweet classification individually, requiring more prompts with shorter contexts (2) Direct Set Confidence Elicitation - LLM evaluates confidence for a list of tweet classifications at once, using fewer prompts but longer contexts.

enabling human oversight within a budgeted time-frame. LLMs have the potential to assist disaster managers in sifting through massive amounts of online social media data for relevant, critical, and actionable information during disaster events. With the goal of helping disaster managers, we are focused on commonly available LLMs that allow the disaster managers independence from a complex pipeline and the maintenance it implies.

We present two methods for check set selection as seen in Figure 1: (1) individual confidence elicitation: LLM assesses confidence of each tweet classification separately using individual probabilities, requiring more prompts with shorter contexts and (2) direct set confidence elicitation: LLM evaluates confidence for a list of tweet classifications at once which allows for comparison within the list, using fewer prompts but longer contexts. These two approaches attempt to mitigate two underlying problems of LLMs in high-stakes use cases, LLMs refusing to refrain from answering questions they may not know the answers to (Zhou et al., 2024) and LLMs being unable to robustly make use of information in long input contexts (Liu et al., 2024).

Previous work on selection from long-context lists (Hsieh et al., 2022; Gupta et al., 2024; Levy et al., 2024; Laban et al., 2024), has not required the LLM to identify a subset of specific items within a longer list provided in the prompt and has not explored the influence of the referencing method used for the input. Intuitively, more input data and longer contexts provide LLMs more information i.e., the more classifications, the more comparisons LLMs can make to determine the potential incorrect classifications. However, recent studies show that LLMs struggle with long-context tasks where performance is influenced by the input order and context size (Liu et al., 2024; Hsieh et al., 2022; Gupta et al., 2024).

We ran our experiments using both closed and open-sourced off-the-shelf LLMs: gpt-4o-mini (OpenAI, 2024a), gpt-4o (OpenAI, 2024b), llama 3.1 8B-Instruct (Llama Team, 2024), mistral 7B-Instruct v0.3 (Jiang et al., 2023) across check set selection from predictions on two classification tasks: (1) humanitarian aid vs. not humanitarian aid and (2) humanitarian aid information type.

Our key contributions are as follows:

• We introduce direct set confidence-based check set selection, leveraging fewer prompts with longer context input, and compared it to the individual confidence-based check set selection.

• We analyze the long-context capabilities of LLMs in direct set selection, examining the impact of context length, input order, and referencing methods.

Our results show that LLMs have the ability of check set selection using confidence elicitation techniques by outperforming random check set selection. Individual confidence elicitation is found
to be more reliable compared to direct set confidence selection. This demonstrates that the current off-the-shelf LLMs are not sufficiently developed for the direct set selection method. This is evidenced in Section 4 demonstrating the issues in the direct set method such as providing incorrect list sizes, inconsistent outputs across different list-referencing methods, and low inter-annotator agreement. The direct set selection capabilities ought to be further explored as LLMs improve, as the issues pointed out need to be solved, especially when LLMs are to be applied in high stakes scenarios.

2 Method

The study investigates LLM’s ability to select a useful check set from long-context input using confidence elicitation. First, we present the motivation of our approach and how we use LLMs as our disaster tweet classifiers. Then, we demonstrate the two set selection methods. Lastly, we deep dive on the LLMs direct set selection ability from long context input.

Problem Definition. LLMs have been very effective in various natural language tasks. However, adoption of LLMs in high-stake scenarios continues to be a challenge due to two main issues: the larger and more complex the LLMs the less likely they are to refrain from answering questions they do not know the answer to (Zhou et al., 2024) and LLMs struggle with long-context tasks (Liu et al., 2024; Hsieh et al., 2022). We aim to mitigate these problems using check set selection by allowing LLMs to utilize their confidence estimates of their initial predictions to prioritize information needing human review.

LLM as Disaster Tweet Classifier We test the performance of LLMs as disaster tweet classifiers using two classification tasks: Task (1) humanitarian aid vs. not humanitarian aid – asking LLMs if the tweet is useful for humanitarian aid or not and Task (2) humanitarian aid information classification – asking LLMs to classify the tweet based on the type of humanitarian aid information it contains. We ran our experiments on sixteen (16) different disaster events, Task (1) with 6 disaster events with 500 tweets per event and Task (2) with 10 disaster events with 300 tweets per event. More details are found in Section 3.1. The selected check sets are from the initially classified list by these classifiers.

Set Selection using Individual Confidence Elicitation. We make use of an LLM to predict the probability of the initial tweet classification from our disaster tweet classifier to be correct with a value between 0.0 and 1.0, referring to one of the methods by Tian et al. (2023) on confidence elicitation. We select the check set by using the tweet classifications with the lowest probabilities of being correct at the lowest 20% of the tweet classifications. The chosen check set size of 20% corresponds to the estimated effort the disaster managers have budget for, i.e., time and people to review check set. We chose a fixed check set size because it standardizes the effort done by the end users and allows us to compare across different check set selection strategies. For cut-off tweets with the same probabilities, we use random selection.

Evaluating the Reliability of Direct Set Selection To evaluate the reliability of off-the-shelf LLMs in direct set selection, we conducted a comprehensive analysis of three factors: input context length, list-referencing methods, and input list order. For an LLM to be considered reliable, these factors should not significantly affect its performance.

Our experimental design involved prompting the LLM to identify \( k \) tweets with potentially erroneous classification labels from a given list of tweets and classifications provided by an AI assistant. This task requires the LLM to comprehend the initial classification task prompt, access the list of \( k \) tweets and classifications, and subsequently select the check set for end-user review. Figure 7 shows an example set selection prompt.

First, we investigate the influence of context length of the input so we ran prompts with different list context sizes of 25, 50, and 100 tweets and classifications. For the 25-tweet context, we partitioned the 100 tweets into four disjoint groups, each prompt selecting five from the list to create the check set size of 20.

Second, we investigate the influence of list-referencing methods used for the tweet and classification lists. We do these investigations following Mizrahi et al. (2024)’s finding that instruction templates lead to very different performance. Furthermore, as our goal is to allow disaster managers independence from a complex pipeline and optimize resource, the choice of list-referencing method does influence the cost per token (both input and output) and so merits further examination. The four list referencing methods and their rationale are as follows (see Appendix A.4 for examples):
• numerical ID – method commonly used for single retrieval from a list.

• full-text – ensures LLM selects the actual tweets and not hallucinating IDs.

• keywords – similar to how humans recall relevant information from a list of sentences.

• short-uuid (8 characters) – used as key for single retrieval methods that is more robust than numerical IDs as hallucination can easily be detected.

We used multiple prompts (n = 10) for the same disaster event where in every prompt, randomly shuffling the order of tweet classifications in each prompt to assess the influence of list order on selection choices. To select the final check set from the responses of the multiple prompts, we applied majority vote on valid responses.

3 Experimental Setup

3.1 Datasets

Task 1: humanitarian aid vs. not humanitarian aid. We randomly sampled 500 tweets for six different disaster events, i.e., a total of 3000 tweets from CrisisBench (Alam et al., 2021b), a consolidated crisis-related social media dataset for humanitarian information processing. For the LLM prompt design, we renamed the class labels as humanitarian aid and not humanitarian aid from the original broad labels informative vs. not informative to explicate the labeling task.

Task 2: Humanitarian Aid Information Classification. For the humanitarian information classification task, we utilized human-annotated crisis-related tweets from (Alam et al., 2021a). The original dataset had 11 labels, however, we limited our labels to the 5 that were present in all of our selected crisis events, following (Zou et al., 2023) who also reduced their labels. Originally, we experimented with including the labels: other relevant information and not humanitarian, however, our initial experiments showed that such vague and negated labels are too challenging for the LLM. We sampled 300 tweets for each of ten different disaster events, i.e., a total of 3000 tweets.

More information about the datasets used is found in Appendix A.2

3.2 Models

We chose four of the latest commonly used off-the-shelf LLM’s in our experiments. We used gpt-4o-mini (OpenAI, 2024a), gpt-4o (OpenAI, 2024b), llama 3.1-8B-Instruct (Llama Team, 2024), and mistral 7B v0.3-Instruct (Jiang et al., 2023). These models were chosen because they are commonly used by both researchers and the public. We ran our experiments at the temperature setting of 0.0 to make all models deterministic in their prediction. All the other parameters were kept default. The exact model parameters and information are found in Appendix A.3.1.

3.3 Evaluation Metrics

First, we need to evaluate the initial performance of the LLM on classifying single tweets. We use the following metrics for this: Accuracy and Effective Accuracy. We define effective accuracy as the overall performance of the collaboration of the LLM and enduser on the dataset \( D \) of length \( n \), when the enduser is provided with the set size of \( c \) to review. For this scenario, we are working with the assumption that the enduser’s performance on the check set has 100% accuracy. This is computed as follows:

\[
\text{%Eff Acc}_D = \frac{(n - c)}{n} \% \text{Acc}_{LLM} + \frac{c}{n} \% \text{Acc}_{Hum}
\]

To evaluate the LLMs’ ability to select a set from long context input, we introduce the following metrics:

No. of Valid Prompt Response. We test the robustness of all the LLMs on their ability to provide valid prompt responses consistently. We consider an LLM response is considered valid if (1) the set provides the correct number of items requested and (2) all the items in the set come from the long-context input list, i.e., there were no hallucinations. We report valid prompt responses by the 100-tweet partitions of a disaster event (our set largest context-size), i.e., one valid response is equivalent to four valid responses of each disjoint group of context size 25 and two valid responses of each disjoint group of context size 50.

Inter-Annotator Agreement. We used Krippendorff’s alpha (Krippendorff, 1970) to measure the inter-annotator agreement between the multiple prompts with the varying classification list order.

3.4 Prompts

Classifier Prompts. We formulated our classifier prompts with reference to the annotation protocol and the class description provided from the original dataset paper sources. We observed that choice of prompt strategies can influence the relative performance of the model which is in line with multiple
works (Mizrahi et al., 2024; Wei et al., 2024; Gupta et al., 2024). We used as our maximum performance metric Mizrahi et al. (2024), accuracy to select the final prompt templates. The exact prompts can be found in the Appendix A.3

Individual Confidence Set Selection Template Prompts. The set selection prompts consists of the following: (1) individual confidence elicitation task, (2) the classification task prompt and (3) individual tweet and classification. We evaluated different prompt strategies for individual confidence elicitation from Xiong et al., 2024 and Tian et al., 2023 to find the best prompt strategy for our specific tasks. Chain-of-thought (CoT) prompting was not explored for set selection as Tian et al., 2023’s finding suggest that CoT prompting does not improve verbalized calibration for individual confidence prompting. We used as our maximum performance metric (Mizrahi et al., 2024), effective accuracy to select our final prompt. Figure 6 shows the example individual confidence set selection prompt.

Direct Set Selection Template Prompts. The direct set selection prompts consists of the following: (1) the direct set selection task instruction, (2) the classification task prompt and (3) the list of \( k \) tweets and classifications. We manually craft the set selection prompt, where we make explicit the importance of the count of the items that need to be retrieved and that only items in the provided list are to be selected. Specifically, we explored direct set selection and re-ranking the list before selecting the top-\( k \) tweets and classifications. From the evaluated prompt strategies, the choice of prompt strategy also influenced the response, so we used the metric, most number of valid prompt responses, to select our final prompts. Figure 7 shows the example direct set selection prompt.

4 Results

4.1 Disaster Tweet Classification Performance

We ran our experiments on two classification tasks across eight disaster events. The LLMs’ performance for Tasks 1 and 2 are found in Tables 3 and 4 measured in accuracy scores at the column Acc. We observed that the closed-source model, gpt-4o-mini performs well in both tasks, achieving accuracy scores of between 74% and 90% for Task 1 and between 86% and 92% for Task 2. Based on these accuracy scores, we observed that the chosen 20% check set size is the check size that would be needed for a good classifier, if the check set selection is perfect (see column Eff Acc (Max)), the maximum effective accuracies of the LLMs given the check set size in Tables 8 and 9 found in Appendix A.5.3. At the chosen check set size, the Eff Acc (Max) of almost all LLMs reach to above 0.85 across all tasks and all disaster events.

4.2 LLM Individual Confidence Check Set Selection Performance

Using the results from the initial classification tasks, we select our individual confidence check set based on the individual probabilities of each tweet classification of being correct. The effective accuracies of the different models for Tasks 1 and 2 are in Tables 3 and 4 using the individual confidence set selection strategy at column Eff Acc (I). All Eff Acc (I) is higher than the original accuracies of the models, hence improve overall classification performance.

To check the effectiveness of the individual confidence check set selection strategy, we compare Eff Acc (I) with the effective accuracy achieved by the models when selecting a random check set of the same size. We highlighted the instance where the individual confidence check set selection did not outperform random in Tables 3 and 3. We observed that the models, gpt-4o, gpt-4o-mini, and llama individual confidece check set selection outperform random for all the tasks and all the events. Mistral, on the other hand, outperforms random for all except Task (1), Vanuatu cyclone.

We wanted to know if there is an optimal check set size, compared to the current 20%, from our models by mapping the effective accuracies achieved by the models across changing check set sizes as seen in Figure 8 in Appendix A.5.2. These were the average effective accuracies from the four disaster events per task. We found that there is no obvious optimal check set size, with almost all models reaching 100% effective accuracy only when all the tweets are checked.

4.3 LLM Direct Set Selection Performance

LLMs ability to select from a set is influenced by the input context size  As a first step to test LLMs’ check set selection ability using direct set confidence elicitation, we count the number of valid prompt responses LLMs generate. Figure 2 shows the number of valid prompt responses LLMs can generate by context size. We observed that the
Figure 2: Valid prompt responses by context size using the short UUID referencing method. The values are raw counts. One valid prompt response corresponds to one valid check set of size 20 selected from a 100-tweet partition of each disaster event.

input context size influences LLMs’ ability to select a set from a list as seen in Figure 2. We observe that the smallest, 25-tweet context size consistently provides more valid prompt responses accross all models except for llama in Task (2) for the short UUID referencing method.

The list-referencing method used affects LLM’s direct set selection output Figure 3 shows the number of valid prompt responses that LLMs can generate when asked to select 10 tweets from a list of 50 tweets and classifications by list-referencing method used. We observed that the chosen referencing method affects the number of valid prompt responses generated. We observed that providing an index, i.e., either the ID or the short UUID in the list, helps LLMs retrieve a set from the input list. All LLMs struggled in retrieving the full tweet text and keywords, providing invalid responses as outputs.

The input list order influences direct set selection. We observed that the selected check sets vary significantly when we shuffle the order of the input list of tweets and classifications. We present the Krippendorf’s alpha inter-annotator agreement scores for our models in Tasks 1 and 2 in Tables 1 and 2 respectively using the short UUID referencing methods. We do not have agreement scores for some models with insufficient valid prompts. The alpha is computed on the agreement across 100 tweets per disaster event i.e., whether they are included in the check set in each prompt iteration. We must take note that these agreement scores cannot be directly compared across context sizes but are to be evaluated individually. Table 1 shows that only gpt-4o and gpt-4o-mini had agreement scores above 0.60 for the for Task 1, while Table 2 shows only gpt-4o and llama achieve this. This shows that input list order can influence the chosen check set using direct set selection.

Figure 3: Valid prompt responses by list-referencing method at the 50-tweet context size. The values are raw counts. One valid prompt response corresponds to one valid check set of size 20 selected from a 100-tweet partition of a disaster event.

4.4 Individual Confidence is more reliable but Direct Set Confidence merits further investigation

The effective accuracies from the direct set confidence selection are shown in the columns Eff Acc (D - <context size>) in Tables 3 and 4. Effective accuracies for direct set selection across tasks and context sizes are higher than the original accuracies. We note that the effective accuracies for direct set sizes D-50 and D-25 are disadvantaged beforehand compared to the D-100, because they are depen-
<table>
  <tr>
    <th rowspan="2">Event</th>
    <th colspan="4">Task 1: Humanitarian Aid vs. Not Humanitarian Aid</th>
    <th colspan="4">Task 2: Humanitarian Information Classification</th>
  </tr>
  <tr>
    <th>Model</th>
    <th>D-100</th>
    <th>D-50</th>
    <th>D-25</th>
    <th>Model</th>
    <th>D-100</th>
    <th>D-50</th>
    <th>D-25</th>
  </tr>
  <tr>
    <td>California Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.32</td>
    <td>0.55</td>
    <td>0.45</td>
    <td>Canada Wildfires</td>
    <td>gpt-4o-mini</td>
    <td>0.37</td>
    <td>0.37</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td></td>
    <td>gpt-4o</td>
    <td>0.30</td>
    <td>0.33</td>
    <td>0.35</td>
    <td></td>
    <td>gpt-4o</td>
    <td>0.50</td>
    <td>0.47</td>
    <td><b>0.67</b></td>
  </tr>
  <tr>
    <td></td>
    <td>llama</td>
    <td>0.25</td>
    <td>0.25</td>
    <td>0.20</td>
    <td></td>
    <td>llama</td>
    <td>0.00</td>
    <td>0.06</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td></td>
    <td>mistral</td>
    <td>0.25</td>
    <td>0.17</td>
    <td>0.25</td>
    <td></td>
    <td>mistral</td>
    <td>0.00</td>
    <td>0.12</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>Chile Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.25</td>
    <td>0.19</td>
    <td>0.18</td>
    <td>Cyclone Idai</td>
    <td>gpt-4o-mini</td>
    <td>0.36</td>
    <td>0.36</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td></td>
    <td>gpt-4o</td>
    <td>0.55</td>
    <td>0.60</td>
    <td><b>0.64</b></td>
    <td></td>
    <td>gpt-4o</td>
    <td>0.31</td>
    <td>0.46</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td></td>
    <td>llama</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.42</td>
    <td></td>
    <td>llama</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td></td>
    <td>mistral</td>
    <td>0.09</td>
    <td>0.11</td>
    <td>0.16</td>
    <td></td>
    <td>mistral</td>
    <td>0.00</td>
    <td>0.15</td>
    <td>0.32</td>
  </tr>
  <tr>
    <td>India Floods</td>
    <td>gpt-4o-mini</td>
    <td>0.66</td>
    <td><b>0.72</b></td>
    <td><b>0.72</b></td>
    <td>Greece Wildfires</td>
    <td>gpt-4o-mini</td>
    <td>0.35</td>
    <td>0.44</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td></td>
    <td>gpt-4o</td>
    <td><b>0.69</b></td>
    <td><b>0.72</b></td>
    <td>0.71</td>
    <td></td>
    <td>gpt-4o</td>
    <td>0.38</td>
    <td>0.18</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td></td>
    <td>llama</td>
    <td>0.25</td>
    <td>0.00</td>
    <td>0.00</td>
    <td></td>
    <td>llama</td>
    <td>0.29</td>
    <td>0.09</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td></td>
    <td>mistral</td>
    <td>0.10</td>
    <td>0.28</td>
    <td>0.33</td>
    <td></td>
    <td>mistral</td>
    <td>0.00</td>
    <td>0.15</td>
    <td>0.32</td>
  </tr>
  <tr>
    <td>Nepal Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.32</td>
    <td>0.44</td>
    <td>0.44</td>
    <td>Hurricane Harvey</td>
    <td>gpt-4o-mini</td>
    <td>0.30</td>
    <td>0.49</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td></td>
    <td>gpt-4o</td>
    <td>0.35</td>
    <td>0.45</td>
    <td>0.48</td>
    <td></td>
    <td>gpt-4o</td>
    <td>0.30</td>
    <td>0.12</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>llama</td>
    <td>0.00</td>
    <td>0.07</td>
    <td>0.00</td>
    <td></td>
    <td>llama</td>
    <td>0.00</td>
    <td>0.17</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td></td>
    <td>mistral</td>
    <td>0.46</td>
    <td>0.24</td>
    <td>0.33</td>
    <td></td>
    <td>mistral</td>
    <td>0.00</td>
    <td>0.10</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>Pakistan Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.24</td>
    <td>0.38</td>
    <td>0.38</td>
    <td>Hurricane Maria</td>
    <td>gpt-4o-mini</td>
    <td>0.37</td>
    <td>0.47</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td></td>
    <td>gpt-4o</td>
    <td>0.26</td>
    <td>0.41</td>
    <td>0.56</td>
    <td></td>
    <td>gpt-4o</td>
    <td>0.37</td>
    <td>0.47</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td></td>
    <td>llama</td>
    <td>0.07</td>
    <td>0.00</td>
    <td>0.30</td>
    <td></td>
    <td>llama</td>
    <td>0.00</td>
    <td>0.30</td>
    <td>0.23</td>
  </tr>
  <tr>
    <td></td>
    <td>mistral</td>
    <td>0.13</td>
    <td>0.13</td>
    <td>0.18</td>
    <td></td>
    <td>mistral</td>
    <td>0.19</td>
    <td>0.10</td>
    <td>0.20</td>
  </tr>
  <tr>
    <td>Vanuatu Cyclone</td>
    <td>gpt-4o-mini</td>
    <td>0.22</td>
    <td>0.28</td>
    <td>0.28</td>
    <td>Hurricane Matthew</td>
    <td>gpt-4o-mini</td>
    <td>0.25</td>
    <td>0.23</td>
    <td>0.31</td>
  </tr>
  <tr>
    <td></td>
    <td>gpt-4o</td>
    <td>0.43</td>
    <td>0.46</td>
    <td><b>0.66</b></td>
    <td></td>
    <td>gpt-4o</td>
    <td>0.46</td>
    <td>0.19</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td></td>
    <td>llama</td>
    <td>0.07</td>
    <td>0.00</td>
    <td>0.33</td>
    <td></td>
    <td>llama</td>
    <td>0.22</td>
    <td>0.10</td>
    <td>0.30</td>
  </tr>
  <tr>
    <td></td>
    <td>mistral</td>
    <td>0.15</td>
    <td>0.26</td>
    <td>0.21</td>
    <td></td>
    <td>mistral</td>
    <td>0.22</td>
    <td>0.10</td>
    <td>0.30</td>
  </tr>
</table>

Table 1: Inter-annotator agreement between the valid prompts. Krippendorf’s alpha by context size. Bold values indicate high inter-annotator agreement. Short UUID referencing was used.

dent on the luck of the misclassified tweets being evenly distributed across subgroups. When compared with the effective accuracies using random check set, only check set selection using gpt-4o outperforms random across all tasks and context sizes, while gpt-4o-mini, llama and mistral have some events and context sizes that do not outperform random.

We compare the two check set selection strategies and observe that individual confidence check set selection is a more reliable method over direct set confidence selection for having issues across input context length, list-referencing method and input list order. Furthermore, as observed in Tables 3 and 4, only individual confidence outperform random consistently across tasks and events. Full tables of effective accuracies, that also include (Eff Acc (Random)) – effective accuracies for random check set and (Eff Acc (Max)) – maximum possible effective accuracies, can be found in Tables 8 and 9 in Appendix A.5.3.

Table 2: Inter-annotator agreement between the valid prompts. Krippendorf’s alpha by context size. Bold values indicate high inter-annotator agreement. Short UUID referencing was used.

We discovered from our experiments that although we set LLMs to their most deterministic setting, when we do direct check set selection, changing the order of the input context (list of tweets) leads to different check set selections and can even return invalid responses. Invalid responses include providing incorrect number of items from the output list (both more or less than asked), repeating items in the output, editing short-UUID’s where characters can be replaced in items, and editing output full-text tweets to correct the grammar. This observation holds across different input context sizes. We recommend evaluating LLMs with multiple prompts always as we have observed that this issue is under reported.

We performed a sanity check on the check sets selected by the LLMs. Upon manual evaluation of a subset of disaster events for Task 2, we observed that they were commonly tweets that contain multiple information, meaning they can be classified into more than one category.

We did a quick exploration on inference time vs. Effective Accuracy across our two strategies. The disaster manager wants to keep the use of the model cheap, which is probably related to computational/inference time, but more practically it is related to output tokens. Here, we assume that direct set selection has a clear advantage. In terms of inference time a rough estimate on a 100-tweet sample for Llama to select a checks size of 20, for example takes 18.4 seconds inference time on individual confidence (100 prompts), 3 seconds on direct set (D-100), 3.33 seconds on D-50, and 3.65
<table>
  <tr>
    <th colspan="2">Task 1: Humanitarian Aid vs. Not Humanitarian Aid</th>
    <th colspan="6">Task 2: Humanitarian Aid Information Classification</th>
  </tr>
  <tr>
    <th>Event</th>
    <th>Model</th>
    <th>Acc</th>
    <th>Acc (I)</th>
    <th>Acc (D-100)</th>
    <th>Acc (D-50)</th>
    <th>Eff Acc (D-25)</th>
    <th>Model</th>
    <th>Acc</th>
    <th>Acc (I)</th>
    <th>Acc (D-100)</th>
    <th>Acc (D-50)</th>
    <th>Eff Acc (D-25)</th>
  </tr>
  <tr>
    <td rowspan="4">California Earthquake</td>
    <td>gpt-40-mini</td>
    <td>0.74</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.78</td>
    <td>Canada</td>
    <td>gpt-40-mini</td>
    <td>0.92</td>
    <td>0.99</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>gpt-40</td>
    <td>0.67</td>
    <td>0.77</td>
    <td>0.82</td>
    <td>0.81</td>
    <td>0.79</td>
    <td>Wildfires</td>
    <td>gpt-40</td>
    <td>0.92</td>
    <td>0.99</td>
    <td>0.98</td>
    <td>0.98</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.73</td>
    <td>0.79</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.79</td>
    <td>Idai</td>
    <td>llama</td>
    <td>0.86</td>
    <td>0.94</td>
    <td>0.87</td>
    <td>0.89</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.54</td>
    <td>0.67</td>
    <td>0.63</td>
    <td>0.67</td>
    <td>0.67</td>
    <td>Cyclone</td>
    <td>gpt-40-mini</td>
    <td>0.87</td>
    <td>0.94</td>
    <td>0.90</td>
    <td>0.91</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td rowspan="4">Chile Earthquake</td>
    <td>gpt-40-mini</td>
    <td>0.82</td>
    <td>0.92</td>
    <td>0.87</td>
    <td>0.91</td>
    <td>0.83</td>
    <td>Greece</td>
    <td>gpt-40-mini</td>
    <td>0.93</td>
    <td>0.96</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>gpt-40</td>
    <td>0.73</td>
    <td>0.85</td>
    <td>0.91</td>
    <td>0.91</td>
    <td>0.90</td>
    <td>Wildfires</td>
    <td>gpt-40</td>
    <td>0.92</td>
    <td>0.95</td>
    <td>0.97</td>
    <td>0.97</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.73</td>
    <td>0.80</td>
    <td>0.78</td>
    <td>0.78</td>
    <td>0.76</td>
    <td>llama</td>
    <td>0.81</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.65</td>
    <td>0.73</td>
    <td>0.73</td>
    <td>0.75</td>
    <td>0.76</td>
    <td>mistral</td>
    <td>0.58</td>
    <td>0.66</td>
    <td>0.66</td>
    <td>0.67</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td rowspan="4">India Floods</td>
    <td>gpt-40-mini</td>
    <td>0.90</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>Hurricane Harvey</td>
    <td>gpt-40-mini</td>
    <td>0.86</td>
    <td>0.94</td>
    <td>0.89</td>
    <td>0.88</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>gpt-40</td>
    <td>0.87</td>
    <td>0.91</td>
    <td>0.97</td>
    <td>0.97</td>
    <td>0.97</td>
    <td>gpt-40</td>
    <td>0.89</td>
    <td>0.95</td>
    <td>0.93</td>
    <td>0.94</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.64</td>
    <td>0.71</td>
    <td>0.81</td>
    <td>0.79</td>
    <td>0.81</td>
    <td>llama</td>
    <td>0.75</td>
    <td>0.85</td>
    <td>0.80</td>
    <td>0.79</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.80</td>
    <td>0.89</td>
    <td>0.84</td>
    <td>0.89</td>
    <td>0.91</td>
    <td>mistral</td>
    <td>0.64</td>
    <td>0.79</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td rowspan="4">Nepal Earthquake</td>
    <td>gpt-40-mini</td>
    <td>0.82</td>
    <td>0.90</td>
    <td>0.89</td>
    <td>0.90</td>
    <td>0.87</td>
    <td>Hurricane Maria</td>
    <td>gpt-40-mini</td>
    <td>0.88</td>
    <td>0.95</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>gpt-40</td>
    <td>0.74</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>gpt-40</td>
    <td>0.90</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>0.95</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.75</td>
    <td>0.85</td>
    <td>0.81</td>
    <td>0.79</td>
    <td>0.79</td>
    <td>llama</td>
    <td>0.76</td>
    <td>0.88</td>
    <td>0.83</td>
    <td>0.83</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.65</td>
    <td>0.74</td>
    <td>0.73</td>
    <td>0.73</td>
    <td>0.78</td>
    <td>mistral</td>
    <td>0.76</td>
    <td>0.88</td>
    <td>0.83</td>
    <td>0.83</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td rowspan="4">Pakistan Earthquake</td>
    <td>gpt-40-mini</td>
    <td>0.80</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.83</td>
    <td>Hurricane Matthew</td>
    <td>gpt-40-mini</td>
    <td>0.88</td>
    <td>0.95</td>
    <td>0.91</td>
    <td>0.93</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>gpt-40</td>
    <td>0.66</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>gpt-40</td>
    <td>0.91</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>0.96</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.74</td>
    <td>0.79</td>
    <td>0.78</td>
    <td>0.79</td>
    <td>0.78</td>
    <td>llama</td>
    <td>0.77</td>
    <td>0.84</td>
    <td>0.82</td>
    <td>0.82</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.67</td>
    <td>0.75</td>
    <td>0.73</td>
    <td>0.78</td>
    <td>0.78</td>
    <td>mistral</td>
    <td>0.65</td>
    <td>0.72</td>
    <td>0.72</td>
    <td>0.73</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td rowspan="4">Vanuatu Cyclone</td>
    <td>gpt-40-mini</td>
    <td>0.87</td>
    <td>0.94</td>
    <td>0.91</td>
    <td>0.91</td>
    <td>0.88</td>
    <td>Italy Earthquake</td>
    <td>gpt-40-mini</td>
    <td>0.92</td>
    <td>0.94</td>
    <td>0.93</td>
    <td>0.94</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>gpt-40</td>
    <td>0.76</td>
    <td>0.92</td>
    <td>0.93</td>
    <td>0.94</td>
    <td>0.94</td>
    <td>gpt-40</td>
    <td>0.92</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>0.96</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.79</td>
    <td>0.85</td>
    <td>0.83</td>
    <td>0.83</td>
    <td>0.82</td>
    <td>llama</td>
    <td>0.86</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.82</td>
    <td>0.84</td>
    <td>0.85</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>mistral</td>
    <td>0.67</td>
    <td>0.74</td>
    <td>0.74</td>
    <td>0.74</td>
    <td>0.74</td>
  </tr>
</table>

Table 3: Effective Accuracies of the Check Set Selection Strategies. Eff Acc (I) is for the individual confidence and Eff Acc (D) is for direct set confidence and the number indicates the context length size. The highlight indicates when the Eff Acc does not outperform random-short UUID-referencing was used. Full table can be found in Table 8.

seconds on D-25. We recommend a more thorough investigation on these factors as LLMs improve.

6 Related Work

Confidence Elicitation in LLMs. The most common ways to measure confidence in model predictions rely on model’s internal logits. However, with the decoder-only LLMs, it has become less suitable to use these methods. There have been methods in prompting LLMs themselves to express uncertainty in natural language, so called verbalized confidence (Lin et al., 2022). Xiong et al. (2024) defines a systematic framework for LLM uncertainty estimation using prompting, sampling and aggregation strategies and benchmarks these methods in calibration and failure prediction. Tian et al. (2023) showed that large LLMs can express calibrated-confidence (as a probability) more accurately than their raw conditional probabilities suggest. For our individual-based check set selection, we used verbalized numerical confidence.

Selective Prediction for LLMs. Prior research on selective prediction and escalation for human review in LLMs generally follows three main strategies. First, separate verifier models are trained as external classifiers to identify uncertain or potentially incorrect outputs without modifying the base LLM, as explored in human-LLM collaborative annotation frameworks (Wang et al., 2024; Varshney and Baral, 2023). Ma et al., 2023’s filter-then-rerank paradigm employs a separate small language model as a verifier model for the LLM Second, task-specific fine-tuning adapts the LLM itself to better estimate uncertainty by incorporating error regularization or self-evaluation during training (Chen et al., 2023; Xin et al., 2021; Lin and Ma, 2024). Third, model-probing techniques analyze the LLM’s internal signals, for example, Selective-LAMA uses token-level confidence thresholds to filter dubious predictions (Yoshikawa and Okazaki, 2023). Unlike these strategies, which require external verifiers, task-specific training, or manual probing – our approach directly leverages the off-the-shelf LLM’s own confidence estimates to curate

<table>
  <tr>
    <th>Event</th>
    <th>Model</th>
    <th>Acc</th>
    <th>Acc (I)</th>
    <th>Acc (D-100)</th>
    <th>Acc (D-50)</th>
    <th>Eff Acc (D-25)</th>
    <th>Model</th>
    <th>Acc</th>
    <th>Acc (I)</th>
    <th>Acc (D-100)</th>
    <th>Acc (D-50)</th>
    <th>Eff Acc (D-25)</th>
  </tr>
  <tr>
    <td rowspan="4">Maryland Floods</td>
    <td>gpt-40-mini</td>
    <td>0.88</td>
    <td>0.92</td>
    <td>0.90</td>
    <td>0.91</td>
    <td>0.91</td>
    <td>gpt-40-mini</td>
    <td>0.89</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>gpt-40</td>
    <td>0.89</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>gpt-40</td>
    <td>0.77</td>
    <td>0.86</td>
    <td>0.80</td>
    <td>0.82</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.62</td>
    <td>0.73</td>
    <td>0.68</td>
    <td>0.70</td>
    <td>0.69</td>
    <td>llama</td>
    <td>0.85</td>
    <td>0.89</td>
    <td>0.88</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.78</td>
    <td>0.89</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.83</td>
    <td>mistral</td>
    <td>0.85</td>
    <td>0.89</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td rowspan="4">Mexico Earthquake</td>
    <td>gpt-40-mini</td>
    <td>0.91</td>
    <td>0.96</td>
    <td>0.95</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>gpt-40-mini</td>
    <td>0.92</td>
    <td>0.97</td>
    <td>0.97</td>
    <td>0.97</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>gpt-40</td>
    <td>0.91</td>
    <td>0.96</td>
    <td>0.95</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>gpt-40</td>
    <td>0.92</td>
    <td>0.97</td>
    <td>0.97</td>
    <td>0.97</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.85</td>
    <td>0.89</td>
    <td>0.88</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>llama</td>
    <td>0.90</td>
    <td>0.93</td>
    <td>0.91</td>
    <td>0.93</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.82</td>
    <td>0.92</td>
    <td>0.85</td>
    <td>0.86</td>
    <td>0.87</td>
    <td>mistral</td>
    <td>0.82</td>
    <td>0.92</td>
    <td>0.85</td>
    <td>0.86</td>
    <td>0.87</td>
  </tr>
</table>

Table 4: Effective Accuracies of the Check Set Selection Strategies. Eff Acc (I) is for the individual confidence and Eff Acc (D) is for direct set confidence and the number indicates the context length size. The highlight indicates when the Eff Acc does not outperform random-short UUID-referencing was used. Full table can be found in Table 9.
check sets for human review, simplifying deployment and broadening applicability

LLM performance on long-context input text.
For the direct-set check set selection we propose, we explored long-context prompts, which are previously studied in, e.g., Hsieh et al. (2022); Shaham et al. (2023); Levy et al. (2024); Laban et al. (2024). “Long-context” is an umbrella term for use cases of LLMs defined by the total length of the model’s input that may include retrieval, summarization, and information aggregation (Goldman et al., 2024). The most common task that papers evaluate on is the needle-in-a-haystack (NIAH) task, where the LLMs are tasked to retrieve single points (the “needle”) in a long input context (the “haystack”) and asking the LLM to retrieve it given a related question (Kamradt, 2023) and not multiple needles. Hsieh et al. (2022) expands the NIAH task with a comprehensive evaluation of long-context LLMs by creating a new synthetic benchmark revealing that almost all models exhibit large performance drops as context increases. Most papers evaluate LLM performance on synthetic datasets or existing benchmarks (Hsieh et al., 2022; Shaham et al., 2023; Levy et al., 2024; Laban et al., 2024). The study by Gupta et al. (2024) differs by evaluating LLMs in a real-world financial dataset, however, evaluated only the gpt-4 suite of LLMs in solving tasks, as a function of factors such as context length, task difficulty, and position of needle. Our study on the other hand, evaluates both off-the-shelf closed and open-sourced LLMs and considers list-referencing factors in addition to the context length and input list order on real-world crisis-related tweets.

7 Conclusion

In this paper, we investigate the ability of LLMs to identify low-confidence outputs for human review through check set creation, the process of utilizing LLMs to prioritize information needing human review. We run our experiments using a case study for social media monitoring in disaster risk management. We tested two strategies for check set selection: individual confidence elicitation by assessing confidence for each tweet classification and direct set confidence elicitation by evaluating confidence for a list of tweet classifications at once. Furthermore, we examined the impact of context length, input order and referencing methods for direct set selection. Our results show that LLMs struggle in direct set selection as they cannot consistently provide valid prompt responses, being influenced by all the three factors mentioned. Hence, we say that individual confidence set selection is more reliable than direct set selection for our particular setting. However, we observe that the direct set method has potential and could be explored and evaluated further as LLMs continue to improve. Despite these challenges, our approach improves collaborative disaster tweet classification, demonstrating the potential of human-LLM collaboration. Such collaboration is crucial for high stake scenarios where we want the end-user in control of the final decisions.

8 Limitations

We only evaluated four commonly used off-the-shelf LLMs: gpt-4o-mini, gpt-4o, llama and mistral. We only evaluated on the base models to test their check set selection capabilities. Instruction-tuning/fine-tuning these models to specifically do check set selection tasks may lead to more favorable results. Our use case is focused on classification tasks for disaster risk management with text that are only in English language tweets. For the direct set confidence set selection, we only tested context sizes of 100, 50 and 25 tweets. A smaller context size may offer more stable responses from the LLMs. In addition, in selecting the check set from the smaller context sizes, D-50 and D-25, we did not try to optimize which tweets to compare with each other. Our experiments were not performed in a real world application where we had an actual disaster manager perform the manual verification of the tweets in the selected check set. As we assume all wrongly labeled tweets would be corrected in such manual check, our estimations are likely too optimistic.

9 Ethical Considerations

The datasets used in this paper were from publicly available datasets (Alam et al., 2021b,a) which were collected tweets from X (previously, Twitter) using the platform’s streaming API in line with its terms of service.

Our work aspires ultimately to support disaster management in high-stakes scenarios. As such, a potential risk is that readers misinterpret the readiness of the technology for use by disaster managers, and move either too quickly to uptake without guarantees of reliability or pre-maturely abandon the
type of solutions we study. We have attempted to address this point by stating clearly our negative result (i.e., LLMs struggle with long-context set selection) and stating that we find human-LLM collaborations may still hold future potential.

Acknowledgments

This publication is part of the project ‘Indeep: Interpreting Deep Learning Models for Text and Sound’ with project number NWA.1292.19.399, which is partly financed by the Dutch Research Council (NWO).

References

Firoj Alam, Ferda Ofli, and Muhammad Imran. 2018. Crisismd: Multimodal twitter datasets from natural disasters. In International Conference on Web and Social Media.

Firoj Alam, Umair Qazi, Muhammad Imran, and Ferda Ofli. 2021a. Humaid: Human-annotated disaster incidents data from twitter with deep learning benchmarks. Proceedings of the International AAAI Conference on Web and Social Media, 15(1):933–942.

Firoj Alam, Hassan Sajjad, Muhammad Imran, and Ferda Ofli. 2021b. Crisisbench: Benchmarking crisis-related social media datasets for humanitarian information processing. Proceedings of the International AAAI Conference on Web and Social Media, 15(1):923–932.

Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan Arik, Tomas Pfister, and Somesh Jha. 2023. Adaptation with self-evaluation to improve selective prediction in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5190–5213, Singapore. Association for Computational Linguistics.

Aaron Daniel Cohen, Adam Roberts, Alejandra Molina, Alena Butryna, Alicia Jin, Apoorv Kulshreshtha, Ben Hutchinson, Ben Zevenbergen, Blaise Hilary Aguera-Arcas, Chung ching Chang, Claire Cui, Cosmo Du, Daniel De Freitas Adiwardana, Dehao Chen, Dmitry (Dima) Lepikhin, Ed H. Chi, Erin Hoffman-John, Heng-Tze Cheng, Hongrae Lee, Igor Krivokon, James Qin, Jamie Hall, Joe Fenton, Johnny Soraker, Kathy Meier-Hellstern, Kris ten Olson, Lora Mois Aroyo, Maarten Paul Bosma, Marc Joseph Pickett, Marcelo Amorim Menegali, Marian Croak, Mark Díaz, Matthew Lamm, Maxim Krikun, Meredith Ringel Morris, Noam Shazeer, Quoc V. Le, Rachel Bernstein, Ravi Rajakumar, Ray Kurzweil, Romal Thoppilan, Steven Zheng, Taylor Bos, Toju Duke, Tulsee Doshi, Vincent Y. Zhao, Vinodkumar Prabhakaran, Will Rusch, YaGuang Li, Yanping Huang, Yanqi Zhou, Yuanzhong Xu, and Zhifeng Chen. 2022. Lamda: Language models for dialog applications. In arXiv.

Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. 2024. Is it really long context if all you need is retrieval? towards genuinely difficult long context NLP. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16576–16586, Miami, Florida, USA. Association for Computational Linguistics.

Lavanya Gupta, Saket Sharma, and Yiyun Zhao. 2024. Systematic evaluation of long-context LLMs on financial concepts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1163–1175, Miami, Florida, US. Association for Computational Linguistics.

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2022. Ruler: What’s the real context size of your long-context language models?

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Shouk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringler, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language models (mostly) know what they know. Preprint, arXiv:2207.05221.

Greg Kamradt. 2023. Llmtest needle in a haystack: Doing simple retrieval from llm models at various context lengths to measure accuracy.

Klaus Krippendorff. 1970. Estimating the reliability, systematic error and random error of interval data. Educational and Psychological Measurement, 30(1):61–70.

Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam. 2023. Conformal prediction with large language models for multi-choice question answering. In Neural Conversational AI Workshop at ICML 2023.

Philippe Laban, Alexander Fabbri, Caiming Xiong, and Chien-Sheng Wu. 2024. Summary of a haystack: A challenge to long-context LLMs and RAG systems. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages
9885–9903, Miami, Florida, USA. Association for Computational Linguistics.

Mina Lee, Percy Liang, and Qian Yang. 2022. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery.

Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15339–15353, Bangkok, Thailand. Association for Computational Linguistics.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research.

Yu-Xiang Lin and Wei-Yun Ma. 2024. Generating attractive and authentic copywriting from customer reviews. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4629–4642, Mexico City, Mexico. Association for Computational Linguistics.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.

Llama Team. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Yubo Ma, Yixin Cao, Yong Hong, and Aixin Sun. 2023. Large language model is not a good few-shot information extractor, but a good reranker for hard samples! In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10572–10601, Singapore. Association for Computational Linguistics.

Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. State of what art? a call for multi-prompt LLM evaluation. Transactions of the Association for Computational Linguistics, 12:933–949.

OpenAI. 2024a. Gpt-4o mini: advancing cost-efficient intelligence.

OpenAI. 2024b. Gpt-4o system card.

Mohaimenul Azam Khan Raiaan, Md. Saddam Hossain Mukta, Kaniz Fatema, Nur Mohammad Fahad, Sadman Sakib, Most Marufatul Jannat Mim, Jubaer Ahmad, Mohammed Eunus Ali, and Sami Azam. 2024. A review on large language models: Architectures, applications, taxonomies, open issues and challenges. IEEE Access, 12:26839–26874.

Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, Singapore. Association for Computational Linguistics.

Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Raffailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5433–5442, Singapore. Association for Computational Linguistics.

Neeraj Varsney and Chitta Baral. 2023. Post-abstention: Towards reliably re-attempting the abstained instances in QA. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 967–982, Toronto, Canada. Association for Computational Linguistics.

Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. 2024. Human-Ilm collaborative annotation through effective verification of Ilm labels. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery.

Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. 2024. Systematic evaluation of Ilm-as-a-judge in Ilm alignment tasks: Explainable metrics and diverse prompt templates. In Building Trust in LLMs and LLM Applications workshop at ICLR 2025.

Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. The art of abstention: Selective prediction and error regularization for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1040–1051, Online. Association for Computational Linguistics.

Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations.

Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng, Haoming Jiang, Shaochen Zhong, Bing Yin, and Xia Hu. 2024. Harnessing the power of llms in practice: A survey on chatgpt and beyond. ACM Trans. Knowl. Discov. Data, 18(6).

Hiyori Yoshikawa and Naoaki Okazaki. 2023. Selective-LAMA: Selective prediction for confidence-aware evaluation of language models. In Findings of the Association for Computational Linguistics: EACL 2023,
pages 2017–2028, Dubrovnik, Croatia. Association for Computational Linguistics.

Lexin Zhou, Wout Schellart, Fernando Martínez-Plumed, Yael Moros-Daval, Cèsar Ferri, and José Hernández-Orallo. 2024. Larger and more instructable language models become less reliable. Nature, pages 1–8.

Henry Peng Zou, Yue Zhou, Cornelia Caragea, and Doina Caragea. 2023. Crismatch: Semi-supervised few-shot learning for fine-grained disaster tweet classification. CoRR, abs/2310.14627.

A Appendix

A.1 Models
Table 5 contains the information about the 4 LLMs we evaluated and analyzed.

A.2 Datasets
Task 1: humanitarian aid vs. not humanitarian aid
We used data from CrisisBench (Alam et al., 2021b), a consolidated crisis-related social media dataset for humanitarian information processing. We renamed the classes to humanitarian and not humanitarian aid from the original informative vs. not informative classes because these words by themselves were too broad and general. Tweets were annotated as follows(Alam et al., 2021b,a):

• humanitarian aid: tweet is useful for humanitarian aid and

• not humanitarian aid: tweet is not useful for humanitarian aid.

We sampled from consolidated disaster events from CrisisMMD (Alam et al., 2018) dataset specifically from the following crisis events: Pakistan Earthquake 2013, California Earthquake 2014, Chile Earthquake 2014, India Floods 2014, Nepal Earthquake 2014, and Vanuatu Cyclone 2014. We randomly sampled 500 tweets for each disaster event.

Task 2: Humanitarian Aid Information Classification
For the humanitarian information classification task, we utilized human-annotated crisis-related tweets from (Alam et al., 2021a). We sampled across four different disaster types: earthquake, hurricane, wildfire and flood. We chose the event with the highest inter-annotator agreement per disaster type based on (Alam et al., 2021a). The original dataset had 11 labels, however, we limited our labels to the 5 that were present in all of our selected crisis events, following (Zou et al., 2023) who also reduced their labels to 7. Originally, we experimented with including the labels: other relevant information and not humanitarian, however, this seemed to be too challenging for the LLM. The humanitarian aid information labels are as follows:

• Caution and advice: Reports of warnings issued or lifted, guidance and tips related to the disaster;

• Infrastructure and Utility Damage: Reports of any type of damage to infrastructure such as buildings, houses, roads, bridges, power lines, communication poles, or vehicles;

• Injured or dead people: Reports of injured or dead people due to the disaster;

• Rescue, volunteering, or donation effort: Reports of any type of rescue, volunteering, or donation efforts such as people being transported to safe places, people being evacuated, people receiving medical aid or food, people in shelter facilities, donation of money, or services, etc.;

• Sympathy and support: Tweets with prayers, thoughts, and emotional support;

We sampled the test sets of the following crisis events: Canada Wildfires 2016, Cyclone Idai 2019, Greece Wildfires 2018, Mexico Earthquake 2017, Hurricane Matthew 2016, Hurricane Harvey 2017, Hurricane Maria 2017, Italy Earthquake 2016, Maryland Floods 2018, and Sri Lanka Floods 2017. We randomly sampled 300 tweets for each disaster event.

A.3 Prompts
A.3.1 Classification Prompts
The disaster tweet classification prompts are shown in Figures 4 and 5.

A.3.2 Check Set Selection Prompts
The prompts for the two strategies of check set selection are in Figures 6 and 7.

A.4 Output Examples by List-referencing Method
Below are output examples of valid responses by list-referencing method at 50-tweet context size.

Numerical ID:
[366, 191, 233, 356, 74, 149, 80, 242,
Table 5: Information of evaluated and analyzed LLMs

<table>
  <tr>
    <th>Model</th>
    <th>Type</th>
    <th>Size</th>
    <th>Context Length</th>
    <th>Source (OpenAI/Huggingface)</th>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>closed</td>
    <td>-</td>
    <td>128K</td>
    <td>gpt-4o-2024-08-06</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>closed</td>
    <td>-</td>
    <td>128K</td>
    <td>gpt-4o-mini-2024-07-18</td>
  </tr>
  <tr>
    <td>llama 3.1 SB - Instruct</td>
    <td>open</td>
    <td>8B</td>
    <td>128K</td>
    <td>meta-llama/Meta-Llama-3.1-8B-Instruct</td>
  </tr>
  <tr>
    <td>mistral 7B - Instruct v0.3</td>
    <td>open</td>
    <td>7B</td>
    <td>32K</td>
    <td>mistralai/mistral-7B-Instruct-v0.3</td>
  </tr>
</table>

Figure 4: Prompt for Task 1: Humanitarian Aid vs. Not Humanitarian Aid

You will be provided with a tweet. Your task is to classify the tweet as either "humanitarian aid" or "not humanitarian aid" based on its content.
Criteria for Classification:
humanitarian aid:
Classify the tweet as "humanitarian aid" if it contains one or more of the following labels:
Caution and advice: Reports of warnings issued or advice given and tips related to the disaster;
Infrastructure and utility damages: Reports of any type or damage to infrastructure such as buildings, houses, roads, bridges, power lines, communication poles, or vehicles;
Injured or dead people: Reports of people injured or killed due to the disaster;
Rescue, volunteering, or donation effort: Reports of any type of rescue, volunteering, or donation efforts such as people being transported to safe places, people being evacuated, people receiving medical aid or food, people in shelter facilities, donation of money or resources, etc.;
Sympathy and support: Tweets with prayers, thoughts, and emotional support;
Select only one label, even if multiple labels seem to apply. Respond with only the label.
Do not add additional information.
Labels: [select]

Figure 5: Prompt for Task 2: Humanitarian Information Classification

You will be provided a tweet. Based on the tweet’s content, assign one of the following labels related to humanitarian aid that best fits the information provided:
Caution and advice: Reports of warnings issued or advice given and tips related to the disaster;
Infrastructure and utility damages: Reports of any type or damage to infrastructure such as buildings, houses, roads, bridges, power lines, communication poles, or vehicles;
Injured or dead people: Reports of people injured or killed due to the disaster;
Rescue, volunteering, or donation effort: Reports of any type of rescue, volunteering, or donation efforts such as people being transported to safe places, people being evacuated, people receiving medical aid or food, people in shelter facilities, donation of money or resources, etc.;
Sympathy and support: Tweets with prayers, thoughts, and emotional support;
Select only one label, even if multiple labels seem to apply. Respond with only the label.
Do not add additional information.
Labels: [select]

282, 301, 317, 290, 175, 349, 10, 1, 2, 55, 7, 14]
short UUID:
['d8d26064', '88ef4c41', '9cb96943', '41bb8105', '785935c5', 'ea8dfa5b', '4eef9f54', '60df1292', 'b6f5170d', '2b577377']
Key Word:
["distributing commodities", "donate to help", "Hurricane Maria Disaster Recovery", "donate for hurricane relief", "devastated by Hurricane Maria", "damaged Puerto Rico", "death toll climbs", "damaged Arecibo radio telescope", "ruined homes and infrastructure", "donations with what you can"]
Full Text:
[ "80 hours! ! GOD!! https://t.co/sNetLbIskQ", "RT @USER: .@USER Lives may have been saved if Nepal govt prepared people instead of funding animal sacrific Gadhimal ht","RT @USER: 38,000 Nepal youth in Indian Army Gorka Rifles. Over 1.25 lakh veterans. The family will come together in thi hour of c", "Big day for nepal people", "Pulitzer Prize winning Jim Morin’s cartoon on NepalQuake NepalEarthquake URL", "but our farmers issues are gone unnoticed URL", "@USER You are amazing. URL", "Economic Impact Of Nepal Quake Likely To Be Massive: One estimate puts the reconstruction at more than $5 bill.. URL", "@USER Huh. I guess all those Christian missions to Nepal are to protect 7-11’s Himalayan locations.", "12 Things Indians Can Do To Helpâ€Nepal URL" ]
** edited Full Text responses by anonymizing users and URL’s.

A.5 Supplementary Results

A.5.1 Disaster Tweet Classifier Performance

The performance of the LLMs as disaster tweet classifiers are in Tables 6 and 7.

A.5.2 Individual Confidence Elicitation Results

We wanted to know if there is an optimal check set size, compared to the current 20%, from our models by mapping the effective accuracies achieved by the models across changing check set sizes as seen in Figure 8.
Provide the probability that an AI Assistant's response is correct, as a value between 0.0 and 1.0 for the following task.
Give only the probability – no words, explanations, or extra commentary whatsoever.
Respond only with the probability in this format:
<value between 0.0 and 1.0 only>

Task: {Classification Task Prompt}
Tweet: {Tweet}
AI Assistant response: {Classification}

Figure 6: Prompt for Individual Confidence Elicitation

Task Overview:
You are provided with a list of tweets, each labeled with a classification assigned by an AI Assistant (also an LLM). Your role is to identify tweets where the assigned classification may not accurately reflect the content, potentially indicating an error by the AI Assistant.

Instructions:
1. Review the AI Assistant’s Classification Prompt: Refer to the prompt used to instruct the AI Assistant on how to classify the tweets. This prompt outlines the criteria you’ll use to assess the accuracy of each classification.
2. Assess Class Accuracy: For each tweet, determine if the assigned class aligns with the tweet’s content based on the AI Assistant’s classification prompt. Emphasize consistency, especially among tweets with similar themes or content.
3. Identify Misclassifications: Flag tweets where the assigned class does not match the content according to the AI Assistant’s classification prompt.
4. Select Exactly {COUNT} Tweets: Choose “*precisely {COUNT} unique tweets*” with classification errors=** no more and no less**. If you identify more errors than the required count, prioritize tweets that are the most clearly misclassified.
5. Recount Selected Tweets: Include the complete text of each selected tweet verbatim.
6. Use Only Provided Tweets: choose tweets exclusively from the provided list; do not add, modify, or invent tweets.
7. Avoid Duplicates: Ensure each selected tweet appears only once.

Output Format:
Your output must include exactly {COUNT} tweets, formatted as a Python list. Do not add any explanation.
["<tweet1>", "<tweet2>", ..., "<tweet{COUNT}>"]

Failure to provide exactly {COUNT} tweets will be considered incorrect output.

AI Assistant’s Classification Prompt:
{Classification Task Prompt}
Tweet: | Class Assigned by AI Assistant:
{Tweet and Classifications List}

Figure 7: Prompt for Direct Set Selection

A.5.3 Effective Accuracies Full Tables
Tables 8 and 9 show the full tables effective accuracies of all the check set selection strategies.
Table 6: Performance of LLMs on Task 1: Humanitarian Aid vs. Not Humanitarian Aid measured in Accuracy.

<table>
  <tr>
    <th>Event</th>
    <th>Model</th>
    <th>Accuracy</th>
  </tr>
  <tr>
    <td rowspan="5">California Earthquake</td>
    <td>majority class</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td rowspan="5">Chile Earthquake</td>
    <td>majority class</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td rowspan="5">India Floods</td>
    <td>majority class</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td rowspan="5">Nepal Earthquake</td>
    <td>majority class</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td rowspan="5">Pakistan Earthquake</td>
    <td>majority class</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td rowspan="5">Vanuatu Cyclone</td>
    <td>majority class</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.82</td>
  </tr>
</table>

![Line plots showing Effective Accuracy (I) vs. Check Set Size for different models on Task 1 and Task 2](page_120_1042_627_410.png)

Task 1: Humanitarian Aid vs. Not Humanitarian Aid

Task 2: Humanitarian Aid Information Classification

Figure 8: Effective Accuracy (Individual Confidence) vs. Check Set Size. The broken lines represent the Effective Accuracies for the random check set selection.

Table 7: Performance of LLMs on Task 2: the Humanitarian Aid Information Classification task measured in Accuracy

<table>
  <tr>
    <th>Event</th>
    <th>Model</th>
    <th>Accuracy</th>
  </tr>
  <tr>
    <td rowspan="5">Canada Wildfires</td>
    <td>majority class</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td rowspan="5">Cyclone Ida</td>
    <td>majority class</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td rowspan="5">Greece Wildfires</td>
    <td>majority class</td>
    <td>0.40</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td rowspan="5">Hurricane Harvey</td>
    <td>majority class</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td rowspan="5">Hurricane Maria</td>
    <td>majority class</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.9</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td rowspan="5">Hurricane Matthew</td>
    <td>majority class</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td rowspan="5">Italy Earthquake</td>
    <td>majority class</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td rowspan="5">Maryland Floods</td>
    <td>majority class</td>
    <td>0.29</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.62</td>
  </tr>
  <tr>
    <td rowspan="5">Mexico Earthquake</td>
    <td>majority class</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td rowspan="5">Sri Lanka Floods</td>
    <td>majority class</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>gpt-4o-mini</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.9</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.82</td>
  </tr>
</table>
<table>
  <tr>
    <th rowspan="2">Event</th>
    <th rowspan="2">Model</th>
    <th colspan="1">Acc</th>
    <th colspan="1">Eff Acc (Random)</th>
    <th colspan="1">Eff Acc (Max.)</th>
    <th colspan="1">Eff Acc (I)</th>
    <th colspan="1">Eff Acc (D-100)</th>
    <th colspan="1">Eff Acc (D-50)</th>
    <th colspan="1">Eff Acc (D-25)</th>
  </tr>
  <tr></tr>
  <tr>
    <td rowspan="4">California Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.74</td>
    <td>0.79</td>
    <td>0.94</td>
    <td>0.80</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.67</td>
    <td>0.74</td>
    <td>0.87</td>
    <td>0.77</td>
    <td>0.82</td>
    <td>0.81</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.73</td>
    <td>0.78</td>
    <td>0.93</td>
    <td>0.79</td>
    <td>0.71</td>
    <td>0.73</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.54</td>
    <td>0.63</td>
    <td>0.74</td>
    <td>0.67</td>
    <td>0.63</td>
    <td>0.67</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td rowspan="4">Chile Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.82</td>
    <td>0.86</td>
    <td>1.00</td>
    <td>0.92</td>
    <td>0.87</td>
    <td>0.91</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.73</td>
    <td>0.78</td>
    <td>0.93</td>
    <td>0.85</td>
    <td>0.91</td>
    <td>0.91</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.73</td>
    <td>0.78</td>
    <td>0.93</td>
    <td>0.80</td>
    <td>0.78</td>
    <td>0.78</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.65</td>
    <td>0.72</td>
    <td>0.85</td>
    <td>0.73</td>
    <td>0.73</td>
    <td>0.75</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td rowspan="4">India Floods</td>
    <td>gpt-4o-mini</td>
    <td>0.90</td>
    <td>0.92</td>
    <td>1.00</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.97</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.87</td>
    <td>0.90</td>
    <td>1.00</td>
    <td>0.94</td>
    <td>0.97</td>
    <td>0.97</td>
    <td>0.97</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.64</td>
    <td>0.71</td>
    <td>0.84</td>
    <td>0.71</td>
    <td>0.81</td>
    <td>0.79</td>
    <td>0.81</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.80</td>
    <td>0.84</td>
    <td>1.00</td>
    <td>0.89</td>
    <td>0.84</td>
    <td>0.89</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td rowspan="4">Nepal Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.82</td>
    <td>0.86</td>
    <td>1.00</td>
    <td>0.90</td>
    <td>0.89</td>
    <td>0.90</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.75</td>
    <td>0.79</td>
    <td>0.94</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.88</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.75</td>
    <td>0.80</td>
    <td>0.95</td>
    <td>0.85</td>
    <td>0.81</td>
    <td>0.79</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.65</td>
    <td>0.72</td>
    <td>0.85</td>
    <td>0.74</td>
    <td>0.73</td>
    <td>0.78</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td rowspan="4">Pakistan Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.81</td>
    <td>0.85</td>
    <td>1.00</td>
    <td>0.89</td>
    <td>0.86</td>
    <td>0.85</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.66</td>
    <td>0.73</td>
    <td>0.86</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.87</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.74</td>
    <td>0.79</td>
    <td>0.94</td>
    <td>0.79</td>
    <td>0.78</td>
    <td>0.79</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.67</td>
    <td>0.74</td>
    <td>0.87</td>
    <td>0.75</td>
    <td>0.73</td>
    <td>0.78</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td rowspan="4">Vanuatu Cyclone</td>
    <td>gpt-4o-mini</td>
    <td>0.87</td>
    <td>0.90</td>
    <td>1.00</td>
    <td>0.94</td>
    <td>0.91</td>
    <td>0.91</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.76</td>
    <td>0.81</td>
    <td>0.96</td>
    <td>0.92</td>
    <td>0.93</td>
    <td>0.94</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.79</td>
    <td>0.83</td>
    <td>0.99</td>
    <td>0.85</td>
    <td>0.83</td>
    <td>0.83</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.82</td>
    <td>0.86</td>
    <td>1.00</td>
    <td>0.84</td>
    <td>0.85</td>
    <td>0.87</td>
    <td>0.87</td>
  </tr>
</table>

Table 8: Effective Accuracies of the Check Set Selection Strategies. Eff Acc (Random) is the effective accuracy for the task given a random check set, Eff Acc (Max) is the maximum possible effective accuracy for the task, Eff Acc (I) is for the individual confidence elicitation and Eff Acc (D) is for direct set confidence elicitation and the number indicates the context length size. The referencing method for direct set used for this table short-uuid
<table>
  <tr>
    <th rowspan="2">Event</th>
    <th rowspan="2">Model</th>
    <th colspan="1">Acc</th>
    <th colspan="1">Eff Acc (Random)</th>
    <th colspan="1">Eff Acc (Max)</th>
    <th colspan="1">Eff Acc (I)</th>
    <th colspan="1">Eff Acc (D-100)</th>
    <th colspan="1">Eff Acc (D-50)</th>
    <th colspan="1">Eff Acc (D-25)</th>
  </tr>
  <tr></tr>
  <tr>
    <td rowspan="4">Canada Wildfires</td>
    <td>gpt-4o-mini</td>
    <td>0.92</td>
    <td>0.94</td>
    <td>1.00</td>
    <td>0.97</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.92</td>
    <td>0.94</td>
    <td>1.00</td>
    <td>0.99</td>
    <td>0.98</td>
    <td>0.98</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.86</td>
    <td>0.89</td>
    <td>1.00</td>
    <td>0.92</td>
    <td>0.90</td>
    <td>0.88</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.86</td>
    <td>0.89</td>
    <td>1.00</td>
    <td>0.94</td>
    <td>0.87</td>
    <td>0.89</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td rowspan="4">Idai</td>
    <td>gpt-4o-mini</td>
    <td>0.87</td>
    <td>0.90</td>
    <td>1.00</td>
    <td>0.94</td>
    <td>0.90</td>
    <td>0.87</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.89</td>
    <td>0.91</td>
    <td>1.00</td>
    <td>0.96</td>
    <td>0.92</td>
    <td>0.95</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.80</td>
    <td>0.84</td>
    <td>1.00</td>
    <td>0.88</td>
    <td>0.84</td>
    <td>0.83</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.77</td>
    <td>0.75</td>
    <td>0.91</td>
    <td>0.82</td>
    <td>0.76</td>
    <td>0.77</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td rowspan="4">Greece Wildfires</td>
    <td>gpt-4o-mini</td>
    <td>0.93</td>
    <td>0.94</td>
    <td>1.00</td>
    <td>0.96</td>
    <td>0.95</td>
    <td>0.95</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.92</td>
    <td>0.94</td>
    <td>1.00</td>
    <td>0.97</td>
    <td>0.95</td>
    <td>0.97</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.81</td>
    <td>0.85</td>
    <td>1.00</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.84</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.58</td>
    <td>0.66</td>
    <td>0.78</td>
    <td>0.71</td>
    <td>0.66</td>
    <td>0.67</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td rowspan="4">Hurricane Harvey</td>
    <td>gpt-4o-mini</td>
    <td>0.86</td>
    <td>0.89</td>
    <td>1.00</td>
    <td>0.94</td>
    <td>0.89</td>
    <td>0.88</td>
    <td>0.90</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.89</td>
    <td>0.91</td>
    <td>1.00</td>
    <td>0.95</td>
    <td>0.93</td>
    <td>0.94</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.75</td>
    <td>0.80</td>
    <td>0.95</td>
    <td>0.85</td>
    <td>0.80</td>
    <td>0.79</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.64</td>
    <td>0.71</td>
    <td>0.84</td>
    <td>0.79</td>
    <td>0.70</td>
    <td>0.70</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td rowspan="4">Hurricane Maria</td>
    <td>gpt-4o-mini</td>
    <td>0.88</td>
    <td>0.90</td>
    <td>1.00</td>
    <td>0.95</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.90</td>
    <td>0.92</td>
    <td>1.00</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>0.96</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.79</td>
    <td>0.83</td>
    <td>0.99</td>
    <td>0.84</td>
    <td>0.82</td>
    <td>0.82</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.76</td>
    <td>0.81</td>
    <td>0.96</td>
    <td>0.88</td>
    <td>0.83</td>
    <td>0.83</td>
    <td>0.80</td>
  </tr>
  <tr>
    <td rowspan="4">Hurricane Matthew</td>
    <td>gpt-4o-mini</td>
    <td>0.88</td>
    <td>0.90</td>
    <td>1.00</td>
    <td>0.95</td>
    <td>0.91</td>
    <td>0.93</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.91</td>
    <td>0.93</td>
    <td>1.00</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>0.96</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.77</td>
    <td>0.82</td>
    <td>0.97</td>
    <td>0.84</td>
    <td>0.82</td>
    <td>0.81</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.65</td>
    <td>0.72</td>
    <td>0.85</td>
    <td>0.75</td>
    <td>0.72</td>
    <td>0.73</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td rowspan="4">Italy Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.92</td>
    <td>0.94</td>
    <td>1.00</td>
    <td>0.94</td>
    <td>0.93</td>
    <td>0.94</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.92</td>
    <td>0.94</td>
    <td>1.00</td>
    <td>0.97</td>
    <td>0.96</td>
    <td>0.96</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.86</td>
    <td>0.89</td>
    <td>1.00</td>
    <td>0.89</td>
    <td>0.88</td>
    <td>0.89</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.66</td>
    <td>0.73</td>
    <td>0.86</td>
    <td>0.74</td>
    <td>0.72</td>
    <td>0.71</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td rowspan="4">Italy Earthquake</td>
    <td>gpt-4o-mini</td>
    <td>0.88</td>
    <td>0.90</td>
    <td>1.00</td>
    <td>0.92</td>
    <td>0.90</td>
    <td>0.91</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.89</td>
    <td>0.91</td>
    <td>1.00</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.93</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.77</td>
    <td>0.82</td>
    <td>0.97</td>
    <td>0.86</td>
    <td>0.80</td>
    <td>0.82</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.62</td>
    <td>0.70</td>
    <td>0.82</td>
    <td>0.75</td>
    <td>0.68</td>
    <td>0.70</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td rowspan="4">Maryland Floods</td>
    <td>gpt-4o-mini</td>
    <td>0.92</td>
    <td>0.94</td>
    <td>1.00</td>
    <td>0.95</td>
    <td>0.94</td>
    <td>0.95</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.91</td>
    <td>0.93</td>
    <td>1.00</td>
    <td>0.96</td>
    <td>0.95</td>
    <td>0.97</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.85</td>
    <td>0.88</td>
    <td>1.00</td>
    <td>0.89</td>
    <td>0.88</td>
    <td>0.89</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.85</td>
    <td>0.82</td>
    <td>0.89</td>
    <td>0.89</td>
    <td>0.81</td>
    <td>0.81</td>
    <td>0.83</td>
  </tr>
  <tr>
    <td rowspan="4">Sri Lanka Floods</td>
    <td>gpt-4o-mini</td>
    <td>0.92</td>
    <td>0.94</td>
    <td>1.00</td>
    <td>0.97</td>
    <td>0.94</td>
    <td>0.96</td>
    <td>0.96</td>
  </tr>
  <tr>
    <td>gpt-4o</td>
    <td>0.94</td>
    <td>0.95</td>
    <td>1.00</td>
    <td>0.98</td>
    <td>0.98</td>
    <td>0.98</td>
    <td>0.98</td>
  </tr>
  <tr>
    <td>llama</td>
    <td>0.90</td>
    <td>0.92</td>
    <td>1.00</td>
    <td>0.93</td>
    <td>0.91</td>
    <td>0.93</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>mistral</td>
    <td>0.82</td>
    <td>0.86</td>
    <td>1.00</td>
    <td>0.92</td>
    <td>0.85</td>
    <td>0.86</td>
    <td>0.87</td>
  </tr>
</table>

Table 9: Effective Accuracies of the Check Set Selection Strategies. Eff Acc (Random) is the effective accuracy for the task given a random check set. Eff Acc (Max) is the maximum possible effective accuracy for the task. Eff Acc (I) is for the individual confidence elicitation and Eff Acc (D) is for direct set confidence elicitation and the number indicates the context length size. The referencing method for direct set used for this table short-uuid

Reviewer paper summaries: The paper explores the effectiveness of LLMs in selecting least confidence instances of text classification in the context of tweet characterization for disaster management tasks. The goal is to foster human-LLM collaboration and leverage human guidance for task instances where LLMs are least confident in critical domains. The authors propose two selection strategies — individual confidence elicitation, where the LLM assesses confidence for each classification individually, and direct set confidence elicitation, where the LLM evaluates confidence for a list of classifications at once — and conduct experiments with several LLMs to evaluate effectiveness of their proposed approach along different dimensions and experiment conditions.

**Summary**:
This work investigates whether using LLM confidence estimates to mediate human intervention in a set of K inputs is an effective strategy. When evaluated in two classification tasks across four LLMs, the authors show an average improvement of 1.38 and 3.19 average accuracy improvement when compared to a random input selection (irrespective of confidence scores).  

**Main contributions**:
One of the proposed strategies (direct confidence elicitation) is interesting as it sidesteps the need to obtain confidence scores for each candidate, instead relying on the relative confidence across inputs.

**Methodology**
- Starting with a set of n input examples referring to a classification task, the authors begin by obtaining the LLM’s answer for each example. Assuming a scenario of human-AI collaboration, the goal is to determine which of these answers is likely to be wrong and flag that example for human supervision.
- To this end, the authors explore two approaches: (1) *Individual confidence elicitation* (I), where the authors collect (via prompting) confidence scores for all input examples, and automatically assign the lowest 20% for human revision; (2) *direct confidence elicitation* (D-K), where the authors randomly group the n inputs in combinations of K examples and ask the LLM to select the subset of inputs out of the K that are misclassified.
- The authors then test both of these methodologies in terms of final system accuracy, comparing them with a random selection baseline.
- The authors also provide ablation studies for the (D-K) methodology, such as sensitivity to ordering of the inputs, increasing K (25,50,100), and others.

**Question to authors**:
- Q1. The authors motivate this work by referring to two problems with LLM widespread adoption: (1) answering questions in the presence of uncertainty, and (2) difficulty in handling long input contexts. However, it appears that their methodology (and results) mostly tackles the former. Can you comment on that?
        - Q1.a. In lines 115-118 you also mention that “LLM’s ability to select a direct set of information points from long-context [...] is under-explored” but the work by Laban et al 2024 also performs a similar study. Can you comment on the differences? 
- Q2. In the contributions (lines 107-109) you mention that you explore “aggregation methods”. However, I don’t think I’ve seen any aggregation method being explored. Did you mean “reference methods”? 
- Q3. A clarifying question about the chosen models. It appears that across 4 models you selected 3 with instruction following capabilities but end up selecting a base model for Mistral. This was not sufficiently motivated and since we expect lower capabilities to follow instruction, this could affect the results. Can you comment on why you decided to select this model? 
- Q4. Since the proposed methodology is based on the relative ranking of inputs, it seems that one of the limitations of this framework is that it cannot work in an online setting and it always has to wait until K examples are collected, which may introduce latency in a real-world deployment scenario. Can you comment on that?
- Q5. Lines 232-234 claim “short uuid [...] is more robust than numerical IDs as hallucination can be easily detected”. It is not obvious to me that this is the case. Can you elaborate on this reasoning? 
- Q6. The use case you presented for risk management is interesting. However, I noticed that the datasets are a bit older (~ 2021), which could imply that the datasets were contaminated (i.e., present in the training dataset of the evaluated models). This may also explain why the LLMs’ baseline performance is already pretty high. It would be interesting to test your framework in other datasets.
- Q7. You mention that “negated labels are too challenging for the LLM” (line 266). Can you elaborate on that and what experiments you performed?
- Q8. When you mention the use of “maximum performance metric” in line 318, does it mean that for the reported results in Table 2, you use the prompt that achieved the highest accuracy in your ablation experiments? Please clarify.
- Q9. The explanation in lines 343-348 about the valid number of prompts is unclear. You mention that one valid response for D-100 is equivalent to 4 valid responses for D-25. How is that accounted for in Figures 2 and 3? What do the numbers in the y axis mean?
    - Q9.a. How many points are being used to compute the height of each bar. Can you add some measure of uncertainty (std, confidence interval)?
    - Q9.b. For example, why is the height of the blue bar in  Figure 2 - Task 1 (for GPT-4o) between 35 and 40?
- Q10. A few claims from the results section that are not supported or clearly stated:
       - (lines 367-371) “based on these accuracy scores, we observed that the chosen 20% check size is the check size that would be needed for a good classifier”. This claim follows the analysis of the baseline accuracy scores for LLMs (between 72-95 across both tasks). The results in the column Eff Acc(Max) also doesn’t support the claim since only a single check size was reported. In fact, Figure 8 shows that the best check size would be greater than 70 (when models’ accuracy begins saturating).
      - (lines 402-403) “gpt-4o-mini’s and mistral’s outperform random most of the time”. While we observe that the average accuracy improvement wrt to the baseline for gpt-4o-mini is 2.5 points, the average accuracy improvement over the baseline for mistral is only 0.25 points – suggesting very little improvement. 
     - (lines 444-445) “all LLMs struggled in retrieving the full text [...]”. However in Figure 3 we observe that gpt-4o-mini performance is comparable to that obtained with the numerical IDs and short UUID.
- Q11. Could the lower number of valid prompts for mistral in task 1 be related to the fact that it is a base model and fails to follow instructions? Have you manually checked the outputs to observe what is happening? Can you also comment on why LLama D-100 is so low compared to Llama D-50? What are the differences in the output of Llama ?
- Q12. You mention in lines 445-447 that models “mostly provide an incorrect number of tweets” (which goes against the instruction in the prompt). Is this number above or below the requested number? If it is above, then why don’t you truncate the list (programmatically)? And if it’s below, could it be that the model is not able to find the desired number of errors (perhaps because all the tweets were correctly classified)? Can you please provide more insights regarding this?
- Q13. Lines 512-513 “however, with the decoder-only LLMs, it has become less suitable to use these methods” (referring to model’s internal logits to measure confidence). How are the decoding methods related to the use or not of internal logits. Did you mean closed-source APIs? 



**References**:

- [Ramírez et al 2024: Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection](https://arxiv.org/abs/2405.02134v1)
- [Aggarwal et al 2024: AutoMix: Automatically Mixing Language Models](https://arxiv.org/abs/2310.12963)
- [Kamath et al 2020: Selective Question Answering under Domain Shift](https://aclanthology.org/2020.acl-main.503/)
- [Varshney et al 2022: Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settings](https://aclanthology.org/2022.findings-acl.158/)
- [Yoshikawa et al 2023: Selective-LAMA: Selective Prediction for Confidence-Aware Evaluation of Language Models](https://aclanthology.org/2023.findings-eacl.150/)
- [Chen et al 2023: Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs](https://aclanthology.org/2023.findings-emnlp.345/)
- [Madhusudhan et al 2024: Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://arxiv.org/abs/2407.16221)
- [Laban et al 2024: Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://aclanthology.org/2024.emnlp-main.552/)


*Note**: Pending on the author's response I'm willing to revise my score accordingly.

LLMs often exhibit a tendency to provide answers even when uncertain. This work investigates their ability to identify low-confidence outputs that require human review. The project aims to develop a human-LLM collaboration pipeline in which LLMs handle high-volume tasks but flag potentially inaccurate outputs for human verification to maintain quality in high-stakes scenarios. The authors prompt LLMs to assess confidence levels both for individual samples and across a set of samples through two classification tasks and find that sample-level confidence assessment is more reliable.
Summary of strengths: * The focus on human-LLM collaboration for critical tasks with LLMs as affordances to reduce the workload of human reviewers is well-motivated, and the proposed problem formulation is well-designed.

* The study empirically evaluates the two proposed strategies, including comparisons across input sizes, aggregation methods, and different LLMs. The methods used for prompt selection for the indirect and direct check set preparation were rigorous.

* The results provide insights that characterize LLMs’ competency in addressing the proposed problem.

1. Important research question concerning how to delegate low confidence outputs to experts (humans or AI)
2. Methodology is sound (performs various ablations of the prompt)

This paper introduces an essential and practical future capability for LLMs: prompting users (or at least warning them) to review outputs for potential inaccuracies. The paper is well-written, presenting a clear and detailed introduction to the problem, supported by a good literature review. The authors provide a thorough explanation of the data, methodology, rationale, evaluation process, and results.
Summary of weaknesses: * The paper is missing discussion relevant related work. Wang et al. [1] also explores set selection strategies for verification of LLM-generated annotated generations for different tasks and conducts human subject studies to characterize their effectiveness.

* While focus of the work is LLM-human collaboration, the authors did not conduct any human-subject studies to evaluate the strengths and weaknesses of their approach. Such evaluations are required to reliably evaluate the proposed strategies. Similarly, the assumption of 100% accuracy for human review of the check set is unrealistic — agreement of multiple human reviewers may provide more confidence.

* The authors only explored the capabilities for one task, and it’s unclear how generalizable the insights are. For example, Wang et al. [1] explore multiple datasets and tasks.

* The experimental design of only selecting 5 class labels is not well justified. What is the impact of reducing the number of class labels? Do LLMs perform poorly as the number of labels increases?

[1] Wang X, Kim H, Rahman S, Mitra K, Miao Z. Human-LLM collaborative annotation through effective verification of LLM labels. In Proceedings of the CHI Conference on Human Factors in Computing Systems 2024 May 11 (pp. 1-21).

- W1. Relevant work in active learning (uncertainty-based sampling) and selective prediction is not properly discussed. Given the similarities of this work with those settings, it is imperative that the authors differentiate their work from work in these related fields.
- W2. Other relevant baselines were not considered (e.g., sampling-based, probability-based), or even other baselines in the selected prediction work.
- W3. The reported results are based on few data points (n=100), drawn from a single dataset, and have no guarantees of statistical robustness, such as error bars or confidence intervals (see “Questions and comments to Authors”). 
- W4. The random baseline is competitive with the proposed methodology (+- 2%) in most tasks and models, raising concerns about the reliability and robustness of reported results. 
- W5. Some of the claims are not validated by the experiments or current version of the paper (See “Questions to authors” above)

The authors select a large dataset containing over 166K samples for their experiments but utilize only 400 samples for testing. Given the limited size of the dataset used in the experiment, the results cannot be reliably generalized.
Comments, suggestions and typos: see strengths and weaknesses.

## Writing 

While the paper is relatively easy to follow, the tone appears informal at times and there is a mix of verb tenses (e.g., lines 213-216) and disconnected ideas (e.g., lines 219-224), which hinders readability. 

**Add citations**:
- Discussion of relevant work is missing, in particular:
   - the proposed methodology fits the general field of **selective prediction**. Please consider discussing relevant related work and by clearly stating the differences between your approaches and the proposed “selective prediction” approaches. Here are a few non-exhaustive set of references to start with: Kamath et al 2020, Varshney et al 2022, Yoshikawa et al 2023, Chen et al 2023, Madhusudhan et al 2024. 
   - likewise, the idea of routing least confident answers to humans (or AI assistants) shares similarities with the field of active learning and, specifically, with uncertainty-based sampling. Please consider adding a relevant related work section to the paper. Here are a few references to start with Ramirez et al 2024, Aggarwal et al 2024.
- Lines 50-51 state “studies show that endusers tend to struggle to identify incorrect LLM responses” but there is no supporting evidence. Please consider adding citations to previous work corroborating such statements.
- Lines 233-234 state “that is more robust than numerical IDs as hallucination can be easily detected”. This claim would benefit from adding a citation of previous work that supports it.
- Lines 162-164 state that “the larger and more complex the LLMs the less likely they are to refrain from answering [...]”. Please provide supporting evidence.



**Clarity**:
- The formatting of Figures 2-3 can be improved: it’s pixelated.
- The formatting of Figures 4-7 in Appendix can be improved: It’s pixelated and there are some “red wiggly underlines”.
- Figure 8 should include a confidence interval to provide an idea of the variability of the obtained accuracy across different event types.
- The formatting of Table 2 could also be improved (a bit more spacing) between Task 1 and Task 2 results, add the complete name of mistral (Mistral 7B) and llama (llama 3.1-8B) models for clarity. Since accuracy is being reported, you could consider reporting the accuracy as percentage and up to decimal places.
- Lines 226-234: when describing the different reference methods, you could provide examples of what the identifiers look like.
- The paper repeatedly states “individual probabilities are more reliable but direct set confidence does show potential” (abstract, lines 32-34, lines 146-148, etc.). The paper would improve greatly if you could be more specific on what “potential” means. 
     - Also related, in lines 32-34 you refer to the potential of direct set confidence but the next sentence only refers to challenges. This is counterintuitive and hinders readability. Consider revising it.
- Lines 67-69 state “While LLM’s output is impossible to evaluate automatically in the real world setting”, this is too general and not true in practice, since coding and math-related tasks can be evaluated automatically for correctness. Perhaps a better way of phrasing it would be “high stakes scenarios” ? 
- It is unclear what kind of prompt selection algorithms were used. The authors mention Mizrahi et al 2024 on many occasions but the explanations are not evident. It is also not clear what prompt variations were considered. Please consider revising this description.
- The notation in 3.4 is not easy to parse. I suggest you add the coefficients (n-c)/n before the %Acc_LLM and the same for the human accuracy for better readability.
- The results description should be revised to be more objective (avoid using “almost all LLMs reach to” and instead prioritize “x out of 4 models exhibit“).
- Add a comment about what the different highlights in Table 2 represent.



**Coherence**:
- Section 1 needs re-organizing, since you list the contributions before listing and discussing all the relevant work (e.g., you only discuss the long input context  literature after listing the contributions). 
- Section 3 needs re-organizing the sections: Section 3.4 (evaluation metrics) should come before Section 3.3 (prompts) since the authors refer to various evaluation metrics in Section 3.3 before they were introduced. 
- Ideas expressed in lines 203-242 need to be revised for better coherence. The paper starts mentioning the need to test reference methods without introducing what there are or explaining why those may be necessary.

**Typos**: 
- Line 173: “Tweet Classifier” → “Tweet Classifier.” (for consistency)
- Line 187: “make use of an LLM” → “use an LLM” (simplify)
- Line 197: “review check set” → “review the check set”
- Lines 226-234: missing punctuation in bullet points.
- Line 280: “were kept default” → “were left as default”
- Line 299: “task, (2)” → “task instruction, (2)” (for consistency)
- Line 375: “reach to above” → “achieve up to”

There are a few other inconsistencies and grammar inconsistencies which I did not make a note due to time constraints.

The paper would benefit from proofreading in sections such as lines 213-255, where some parts are unclear or ambiguous.

Line 68: Stating "impossible" may be an overstatement here, as the size of LLMs varies significantly.

Lines 158-172: It's great that you highlighted the problem statement in this section, but it repeats information already covered, which makes it less helpful. Similar repetitions can be found in other areas as well.

Lines 216-219: I understand that the probability scale across disjoint groups may not be synced or normalized. However, selecting the bottom 20% of samples from each disjoint group to create the check set seems unfair, as the lowest probability in one group could be 0.8, while the highest in another group could be 0.2. This suggests that samples should be selected across all groups.

Line 235: The phrase "multiple prompts (n = 10)" is ambiguous. Do you mean you inputted the same prompt 10 times while shuffling the data?

Line 246: Please specify the original dataset size after "randomly sampled 100 tweets from." Why was only 100 tweets selected when the original dataset contains over 166K samples?

Line 267: Why sample only 400 samples from such a large dataset?

Line 327: Please specify what "accuracy" refers to in this context.
Overall assessment scores: 2.5

2

3