Title: Predicting the Performance of Black-box Language Models with Follow-up Queries
Venue: NeurIPS 2025
Accepted: yes
Keywords: monitoring

language models

black-box language models
Abstract: Reliably predicting the behavior of language models---such as whether their outputs are correct or have been adversarially manipulated---is a fundamentally challenging task. This is often made even more difficult as frontier language models are offered only through closed-source APIs, providing only black-box access. In this paper, we predict the behavior of black-box language models by asking follow-up questions and taking the probabilities of responses _as_ representations to train reliable predictors.
We first demonstrate that training a linear model on these responses reliably and accurately predicts model correctness on question-answering and reasoning benchmarks. 
Surprisingly, this can _even outperform white-box linear predictors_ that operate over model internals or activations.
Furthermore, we demonstrate that these follow-up question responses can reliably distinguish between a clean version of an LLM and one that has been adversarially influenced via a system prompt to answer questions incorrectly or to introduce bugs into generated code. 
Finally, we show that they can also be used to differentiate between black-box LLMs, enabling the detection of misrepresented models provided through an API. 
Overall, our work shows promise in monitoring black-box language model behavior, supporting their deployment in larger, autonomous systems.

Predicting the Performance of Black-box Language Models with Follow-up Queries

Abstract

Reliably predicting the behavior of language models-such as whether their outputs are correct or have been adversarially manipulated-is a fundamentally challenging task. This is often made even more difficult as frontier language models are offered only through closed-source APIs, providing only black-box access. In this paper, we predict the behavior of black-box language models by asking follow-up questions and taking the probabilities of responses as representations to train reliable predictors. We first demonstrate that training a linear model on these responses reliably and accurately predicts model correctness on question-answering and reasoning benchmarks. Surprisingly, this can even outperform white-box linear predictors that operate over model internals or activations. Furthermore, we demonstrate that these follow-up question responses can reliably distinguish between a clean version of an LLM and one that has been adversarially influenced via a system prompt to answer questions incorrectly or to introduce bugs into generated code. Finally, we show that they can also be used to differentiate between blackbox LLMs, enabling the detection of misrepresented models provided through an API. Overall, our work shows promise in monitoring black-box language model behavior, supporting their deployment in larger, autonomous systems.

1 Introduction

Reliably predicting the behavior of a language model (e.g., whether its outputs are correct, or whether it has been adversarially manipulated) is a fundamentally challenging task. This is made even more challenging as many of the most capable large language models (LLMs) lie beyond closed-source APIs [Achiam et al., 2023, Team et al., 2023], providing only black-box access through inputs and outputs. As a result, recent advances in understanding these models through model internals or from mechanistic viewpoints [Olsson et al., 2022, Nanda et al., 2022] are no longer applicable. The inability to rely on LLMs remains a roadblock for their widespread deployment in high-stakes settings or in agentic and autonomous frameworks [Xi et al., 2023, Robey et al., 2024].

In spite of only having black-box access, a promising direction in understanding LLMs is to leverage their ability to interact with human queries and provide useful responses. Recent work in the whitebox setting (i.e., having access to model internals) has demonstrated that a language model's hidden state contains low-dimensional features of truthfulness or harmfulness [Zou et al., 2023a], and has analyzed learning sparse dictionaries and activations on certain input tokens [Bricken et al., 2023]. While significant progress has been made on these fronts, these approaches all require white-box access to these models. This raises the question, ' How well can we predict a language model's behavior with only black-box access? "

In this paper, we propose to predict model behavior by looking at their responses to follow-up questions. After receiving an initial generation or answer from an LLM, we ask a set of follow-up

Black-Box Representation

Q: 'Does the world cup

final go to penalties?'

Q: 'Are you confident in

your answer?'

Q: 'Are you able to explain

your answer?'

A: 'Yes'

P('Yes') = 0.95

P('Yes') = 0.80

. . .

Detect Adversary-Influenced Models

Distinguish between models

Predict model performance!

System: you are a

helpful assistant

System: answer

questions incorrectly!

GPT-3.5

vs.

questions, such as, 'Are you able to explain your answer?' We then take the probability of the 'Yes' token of its response as our features for predicting model behavior. Our hypothesis is that the distributions over answers to these questions meaningfully vary with correctness, model families, and model scale. A key advantage of our approach is that, because it only relies on model outputs, it is also model-agnostic and broadly applicable. In cases where top-k probabilities are not available, we can approximate them via sampling. We provide a theoretical result on how quickly using this approximation converges to the approach that has the true underlying probabilities from the LLM.

Our experiments demonstrate that querying a model with follow-up questions yields features that are highly predictive of performance on LLM benchmarks. We show that simple linear models trained on these features accurately predict instance-level correctness on question-answering and reasoning tasks. Surprisingly, our black-box approach often matches-or even outperforms-white-box methods that operate over the language model's hidden state, across a range of different language models and benchmarks. Furthermore, we demonstrate that our predictors admit nice generalization properties due to their low-dimensional nature and perform well on out-of-distribution data (e.g., transferred to new model scales or new datasets) due to our approach's generality.

Beyond predicting performance on benchmarks, our approach provides insights into other model behaviors. For instance, these follow-up questions can be used to reliably detect when an LLM (e.g., GPT-4o-mini) has been adversarially influenced via a system prompt to generate incorrect answers or introduce hidden bugs into code. We also demonstrate that these follow-up question responses can be used to accurately distinguish between different black-box LLMs; this is useful in auditing if cheaper or smaller models are falsely being provided through closed-source APIs. Together, these results highlight the promise of our approach in predicting and monitoring the behaviors of black-box language models, supporting their future use in large systems.

2 Related Work

Predicting Model Performance As previously mentioned, predicting the performance deep learning models is challenging due to their difficult-to-interpret nature. Existing work looks to assess the performance of models by directly operating over the weight space [Unterthiner et al., 2020] or ensembles of multiple trained models [Jiang et al., 2021]. Specifically for language models, prior work has primarily focused on predicting task-level performance on new tasks; for instance, developing predictors of task-level performance that use the performance on similar or related tasks [Xia et al., 2020, Ye et al., 2023]. Other work attempts to predict the performance of models as

they scale up computation (often in terms of data and model size) [Kaplan et al., 2020, Muennighoff et al., 2024]. In contrast, our work focuses on instance-level prediction-i.e., determining whether a model's response to a specific input is likely to be correct. Furthermore, we operate in a black-box setting, using only input-output behavior, rather than internal model parameters or activations.

Uncertainty Quantification in LLMs A related line of work is assessing the calibration or ability of a language model to represent its own uncertainty [Xiong et al., 2023]. Some work investigates LLMs' ability to verbalize confidence or self-assess the quality of their outputs [Kadavath et al., 2022, Kapoor et al., 2024], and others explore prompting techniques to elicit richer uncertainty estimates-e.g., distinguishing between epistemic and aleatoric uncertainty via iterative queries [Yadkori et al., 2024]. Our approach is related in that we ask follow-up questions (e.g., 'Are you confident in your answer?') to elicit indicators of model uncertainty. However, we differ in our use of these responses: rather than relying on a model's verbalized confidence alone, we extract token-level probabilities as features and train simple linear classifiers to predict correctness. We further show that these features generalize across models and settings, and are useful for a large set of tasks that go beyond the set of calibration metrics focused on in the uncertainty quantification literature. In fact, we provide a comparison with a variety of uncertainty quantification methods, empirically showing many benefits of extracting additional information with multiple follow-up queries.

Extracting Features from Neural Networks Many other works have explored approaches to extract representations from neural networks. A related line of work looks to train neural networks (specifically image classifiers) to extract a small set of discrete, interpretable concepts, which can be passed through a linear probe to recover a classifier [Koh et al., 2020]. In our case, we leverage the ability of the LLM to understand language and can circumvent this need for training, extracting features in a task-agnostic manner. Prior work has studied how to extract useful representations for downstream tasks [Wang et al., 2023, Zou et al., 2023a], although they operate in the fundamentally different white-box setting where you can access model internals. Perhaps the most related work employs a similar strategy of asking questions, specifically to detect instances where a model is untruthful [Pacchiardi et al., 2024]. Our work encompasses the much broader task of predicting model behavior and performance.

3 Predicting Performance with Follow-up Queries

Without any access to language model internals, we propose to elicit useful features about its behavior by asking follow-up questions about its generations. This is completely black-box as we only look at the model's outputs, or more specifically, its topk probabilities over the next token. We feed these as features into simple linear classifiers for some downstream task (e.g., predicting performance). For some APIs, we do not have access to the LLM's topk probabilities, so we theoretically analyze predictors trained on sampled approximations of these probabilities.

3.1 Predictive Features through Follow-up Responses

We consider a set of follow-up queries Q = { q 1 , ..., q d } and some autoregressive language model, which models some distribution P over sequences of text. We also consider a dataset D = { ( x 1 , y 1 ) , ..., ( x n , y n ) } , where x i is a sequence of tokens and y i corresponds to a binary label, for example, if the LLM has correctly answered the question x i . We define a i as the greedily sampled response from the LLM, or that a i = arg max c P ( c | x i ) . Then, we construct our black-box representation as some vector z = ( z 1 , ..., z d ) , where each z j = P ( yes | x ⊕ a ⊕ q j ) , where ⊕ denotes the concatenation of strings (or tokens). Each dimension of our representation corresponds to the probability of the yes token under the LLM (where the distribution is specified over the yes and no tokens), in response to the follow-up question q j about the pair of the original question x and greedily sampled answer a . In our paper, we find that working with a set of roughly 50 questions seems to be sufficient for strong performance (see ablations in Section 4.5). We also analyze different choices of these questions in Appendix A.5. Notably, all features z can be extracted in parallel, so increasing the number of follow-up questions adds minimal computational overhead.

In addition to these features, on closed-ended QA tasks, we append the distribution over possible answers. On both closed-ended and open-ended QA tasks, we append the pre- and post-confidence score, which is the confidence of the language model before and after it sees its own sampled answer.

We train a linear predictor β to predict the label y (e.g., whether the model is correct or not) given our feature vector z .

Generating Follow-up Prompts To construct this set of eliciting questions Q , we specify a small number of questions that relate to the model's confidence or belief in its answer. We also use GPT4 to generate a larger number (40) of questions. The questions and prompts used to generate the GPT4-generated questions are given in Appendix D.4. The elicitation questions are detailed in Appendix D.4, but generally consist of simple self-inquiry questions such as 'Do you think your answer is correct?' or 'Are your responses free from bias?' This simple approach allows us to add more information to our extracted representations by continuing to generate new follow-up questions.

We note that, based on the specific nature of the question, the response (e.g., the probability of responding yes ) could define a weak predictor of whether the model is correct or not. This is reminiscent of the design of weak learners in boosting [Freund and Schapire, 1996] or weak labelers in programmatic weak supervision [Ratner et al., 2017, Sam and Kolter, 2023, Smith et al., 2024]. However, to maintain our approach's generality and to not restrict our approach to only a certain type of elicitation questions, we treat these as abstract features for a linear predictor. We also note that further work could perform discrete optimization over prompts to further improve the extracted representation's usability, through methods described in [Wen et al., 2024, Zou et al., 2023b]. However, one key appeal of the current approach is that it defines an extremely simple classifier in a task-agnostic fashion. Performing optimization over these questions might lead to overfitting, and the resulting predictors on the outputs of these prompts require more complex analysis in deriving valid generalization bounds.

3.2 Theoretical Analysis of Sampling-based Approximations

While our approach described above assumes access to the topk probabilities, some language models are only accessible through APIs that do not provide this information [Team et al., 2023]. In this setting, we can approximate these probabilities via high-temperature sampling from the LLM. Here, we provide a theoretical analysis of how this approximation impacts the performance of our method.

Recall that we have our representation z = ( z 1 , ..., z d ) , which corresponds to the actual probability of the yes token under the LLM. Without access to these true probabilities through an API, we instead have some approximation ˆ z = (ˆ z 1 , ..., ˆ z d ) , where each ˆ z j is an average of k samples from Ber ( z j ) . From prior work in logistic regression under settings of covariate measurement error [Stefanski and Carroll, 1985], when we have that k grows with n , we observe that the naive MLE (maximum likelihood estimator) on the observed approximation results in a consistent, albeit biased, estimator. We present an analysis of our setting, showing a result on the convergence rate of the MLE for β .

Proposition 1 (Estimator on Finite Samples from LLM) . Let ˆ β be the MLE for the logistic regression on the dataset { ( x j i , y i ) | i = 1 , ..., n, j = 1 , ..., k } , where x j i are independent samples from Ber ( p i ) . We assume there exists some unique optimal set of weights β 0 over inputs p = ( p 1 , ..., p d ) , and we let n, k >> d . Then, we have that ˆ β → β 0 as n →∞ and k →∞ . Furthermore, ˆ β converges at a rate O ( 1 √ n + √ n k ) .

We provide the full proof in Appendix B. At a high level, this follows straightforwardly; ˆ β converges to the optimal predictor on the sampled dataset (which we call β ∗ ), via asymptotic results for the MLE. Then, we derive that β ∗ converges to β 0 at a rate of O ( √ n/k ) .

This result demonstrates that, under the setting where we do not have access to the LLM's actual probabilities, we can closely approximate this with sampling, as long as we approximate it with a sample of size k that grows (at a slower rate) with n to get a consistent estimator. Later in Section 4.5, we empirically demonstrate that a naive logistic regression model with an approximation over a finite k samples performs comparably to using the actual LLM probabilities.

4 Experiments

We now evaluate our method in three main applications: (1) predicting the performance of various open- and closed-source LLMs on a variety of text classification and generation tasks, (2) detecting

LLaMA3-3B

LLaMA3-8B

LLaMA3-70B

GPT-3.5

GPT-4o-mini

0.5

0.6

0.7

0.8

0.9

1.0

NQ AUROC

RepE

Pre-conf

Post-conf

Self-consistency

Sem. Entropy

QueRE (Ours)

LLaMA3-3B

LLaMA3-8B

LLaMA3-70B

GPT-3.5

GPT-4o-mini

0.5

0.6

0.7

0.8

0.9

1.0

SQuAD AUROC

whether a LLM has been influenced by an adversary, and (3) distinguishing between different LLM architectures. We refer to our approach as QueRE (Follow-up Que stion R epresentation E licitation).

Baselines In our experiments, we compare against a variety of different baselines. Our first two baselines are white-box methods , which assume more information than QueRE. These include RepE [Zou et al., 2023a], which extracts the hidden state of the LLM at the last token position, and Full Logits , which uses the distribution over the LLM's entire vocabulary. Neither of these can be applied to black-box language models and should be seen as strong baseline comparisons. For instance, information from the full logits over the complete vocabulary has been shown to reveal proprietary information from LLMs [Finlayson et al., 2024]. To approximate Full Logits for black-box LLMs, we approximate this with a sparse vector of top-k probabilities provided by the API.

For black-box baselines on open-ended QA tasks, we compare against Self-Consistency [Wei et al., 2024], where we sample 10 times from the language model to define a probability distribution over potential answers. For closed-ended QA tasks, we can directly use the probability distribution over the potential answer questions ( Answer Probs ), as is done in prior work [Abbas et al., 2024]. We also compare with Semantic Entropy [Kuhn et al., 2023] on open-ended tasks, which aims to extract a more accurate quantification of uncertainty by grouping semantically similar answers. Finally, on all tasks, we also compare against pre-conf and post-conf scores, which are a univariate feature that

LLaMA3-3B

LLaMA3-8B

LLaMA3-70B

GPT-3.5

GPT-4o-mini

0.4

0.5

0.6

0.7

0.8

0.9

HaluEval AUROC

RepE

Pre-conf

Post-conf

Answer P.

Sem. Entropy

QueRE (Ours)

LLaMA3-3B

LLaMA3-8B

LLaMA3-70B

GPT-3.5

GPT-4o-mini

0.4

0.5

0.6

0.7

0.8

0.9

1.0

BoolQ AUROC

LLaMA3-3B

LLaMA3-8B

LLaMA3-70B

GPT-3.5

GPT-4o-mini

0.4

0.5

0.6

0.7

0.8

0.9

1.0

DHate AUROC

corresponds to the probability of the 'yes" token under the language model to a prompt about the model's confidence either before (pre-) or after (post-) seeing the greedy (temperature 0) sampled response. This is the same as the naive approach in directly extracting confidence scores from LLMs [Xiong et al., 2023]. Pre- and post-conf (and Answer Probs on closed-source tasks) are components of our representations on closed-source tasks, so this comparison illuminates how much of our performance is gained by our follow-up queries.

Datasets and Models We evaluate predicting the behavior of LLMs on a variety of benchmarks. We consider the open-ended QA benchmarks NQ [Kwiatkowski et al., 2019] and SQuAD [Rajpurkar et al., 2016]), as well as the closed-ended QA datasets of BoolQ [Clark et al., 2019], WinoGrande [Sakaguchi et al., 2021], HaluEval [Li et al., 2023], DHate [Vidgen et al., 2021], and CS QA [Talmor et al., 2019]). These datasets encompass commonsense reasoning, hallucination detection, factual recall, and toxicity classification. Finally, we also evaluate on math ( GSM8K [Cobbe et al., 2021]) and code ( Code Contests [Li et al., 2022]) benchmarks to evaluate if our approach is predictive of tasks that require reasoning. In our experiments, we evaluate the performance of LLaMA3 (3B, 8B, and 70B) [Dubey et al., 2024] and OpenAI's GPT-3.5 and GPT-4o-mini models [Achiam et al., 2023]. In all of the text generation tasks, we sample greedily from the LLM for its answer. Additional experimental details can be found in Appendix D.5.

4.1 Predicting Model Correctness on QA and Reasoning Tasks

Our first evaluation focuses on predicting instance-level LLM performance on QA and reasoning benchmarks, according to each benchmark's respective metric. For example, on SQuAD [Rajpurkar et al., 2016], correctness is defined by exact match, while for reasoning benchmarks such as math and code, correctness is determined using GPT-4o as an LLM judge.

We find that QueRE consistently outperforms other methods (including white-box approaches) on open-ended QA tasks (Figure 2) and is most often the best-performing black-box method on

LLaMA3-8B vs LLaMA3-70B

LLaMA3-3B vs LLaMA3-8B

GPT-3.5 vs GPT-4o-mini

0.5

0.6

0.7

0.8

0.9

Pre-conf

Post-conf

Answer Probs

Sem. U

QueRE (Ours)

closed-ended QA tasks (Figure 3). While we do not claim that QueRE captures semantic notions of reasoning, it nevertheless proves highly predictive of performance on reasoning tasks (e.g., coding and math benchmarks), while other approaches fail. Full results across all models are provided in Appendix A.1, where similar trends hold. We also compare QueRE to other uncertainty quantification approaches from [Xiong et al., 2023] in Appendix A.3, similarly finding that QueRE outperforms these techniques as well. Overall, our approach in using follow-up queries leads to predictive features for a wide variety of tasks, often rivalling or exceeding white-box baselines.

4.2 Detecting Adversarially Influenced LLMs

Next, we demonstrate QueRE can reliably distinguish between a clean version of the LLM and one that has been influenced by an adversary, which is crucial for maintaining robust and trustworthy AI systems. We provide an experiment on both a QA task and a code generation task, where we add an adversarial system prompt to instruct GPT models to answer questions or generate code incorrectly. For the QA task, we use multiple copies of the LLM with 3 different adversarial and 3 different helpful system prompts, and train linear probes on top of the extracted features to distinguish between helpful and harmful system prompts (see Appendix D.1 for more details and for the specific prompts). For the code generation task, we use multiple copies of the LLM with different adversarial system prompts to generate incorrect code. These experimental settings are similar to the work of MacDiarmid et al. [2024], where they could reliably detect the presence of sleeper agents in the white-box setting. In our setting, we only assume black-box access to the model.

Our results show that QueRE (with a simple linear probe) can reliably detect when a model has been influenced by an adversarial system prompt in both QA and code generation settings (Table 2), outperforming other methods on all datasets. These results also suggest that QueRE is also robust to the specific framing of the adversarial prompt, given that it can distinguish between multiple versions of harmful versus helpful system prompts in both QA and code generation tasks.

4.3 Distinguishing Between Black-box LLMs

Finally, we consider the setting of distinguishing between different LLMs in a black-box setting, purely via analyzing their outputs. This has a practical application; when using models given through an API, our approach can be used to reliably detect whether a cheaper, smaller model is being falsely provided through an API. This problem has also been studied by concurrent work [Gao et al., 2024] in the setting of hypothesis testing. We provide an experiment where the goal is to classify which LLM from which each extracted representation was generated.

We demonstrate that QueRE can be used to reliably distinguish between different LLM architectures and sizes (Figure 4 and in Appendix appendix A.7). We observe that linear predictors using QueRE can often almost perfectly classify between LLMs of different sizes, while other black-box approaches do not perform as well. This suggests that the distributions learned by different LLMs behave in distinct ways, even within the same family, and the only difference is the model size. Notably, this suggests that different model scales cannot be differentiated simply through naive confidence scores.

4.4 Additional Results

We present additional results on the generality of our approach through its ability to transfer across different datasets and models, as well as yield tight generalization bounds. We defer further results on the improved calibration of predictors learned via QueRE to Appendix A.4.

QueRE transfers across datasets and models. We also provide experiments that demonstrate the generality and transferrability of classifiers trained on representations extracted via QueRE to OOD settings. We compare QueRE to other baselines as we (1) transfer the learned predictors from one QA dataset to another, or (2) transfer from one LLaMA3 model size to another. Across all tasks, QueRE shows the best transferring performance (Table 3). Thus, this suggests QueRE performs the best in OOD settings without any access to labeled data from the target task.

QueRE yields tighter generalization bounds. Another added benefit of our approach is that it yields low-dimensional representations, which can be used with simple models, to achieve strong predictors of performance with tight generalization bounds. We use the following PAC-Bayes generalization bound for linear models (see Appendix A.8 for more details). We observe that linear predictors trained our representations have stronger guarantees on accuracy, when compared to baselines (Table 4 and Appendix A.8). A limitation of these results is that they require an assumption that the representations extracted by a LLM are independent of the downstream task data; this

5

10

15

20

25

Number of Samples

0.565

0.570

0.575

0.580

0.585

AUROC

LLM Probabilities

Sampling

5

10

15

20

25

30

35

40

45

Number of Elicitation Prompts

0.76

0.78

0.80

0.82

AUROC

assumption is verifiable via works in data contamination [Oren et al., 2023] or is valid on datasets released after LLM training (e.g., HaluEval for GPT-3.5).

4.5 Ablations

Sampling-based approximations achieve comparable performance. As previously mentioned, we often do not have access to topk probabilities through the closed-source API. While we have provided asymptotic guarantees (in terms of both n and k ) on the estimator learned via logistic regression, we are also interested in the setting where we have a finite number of samples k . Therefore, we run an experiment where instead of using the actual ground-truth probability, we approximate this via an average of k samples from the distribution of the LLM. We report results using approximations via sampling from the distribution specified by GPT-3.5's topk log probs (Figure 5 - Left). We do not observe a significant drop (less than 2 points in AUROC) in performance when using sampling, which implies that our method can be used with APIs that do not provide topk probabilities.

More follow-up questions lead to better performance. We study how much the number of elicitation questions directly impacts how much information is extracted in QueRE. We randomly subsample the number of elicitation questions and report how much the performance of our approach varies when only using this subset of questions. We observe the overall trend that our predictive performance increases as we increase the number of elicitation prompts (Figure 5 - Right), with the rate of increase slowly diminishing with more prompts. We defer results on other datasets to Appendix A.12, where we observe similar results. Overall, this demonstrates that we can achieve even stronger performance with our method by scaling up the number of follow-up questions. As previously mentioned, this only comes with a slight increase in computational complexity, as these follow-up questions can all be handled in parallel.

We defer further ablations on using MLPs instead of linear models in Appendix A.11 and on the type of follow-up questions used in QueRE to Appendix A.5.

5 Discussion

Our contributions find that querying a language model with follow-up questions leads to features that are useful in a wide variety of applications in predicting model behavior. Remarkably, they can often match the performance of predictors that work in the white-box setting over model internals when predicting correctness on LLM benchmarks or in detecting when language models have been adversarially manipulated. Overall, we believe that our work provides promising results towards reliably predicting the behavior of language models and detecting when they have been adversarially manipulated, which supports the potential of their deployment in larger autonomous systems and foundations towards more trustworthy language models.

Limitations While QueRE demonstrates strong predictive performance across many tasks, it has a few limitations. First, although the features extracted via QueRE are grounded in natural language, our focus is not on interpretability or attribution. We treat these features purely as abstract inputs to a predictive model, rather than as explanations or understanding of model behavior. Second, our approach introduces latency through multiple follow-up queries per example, although this can be mitigated through batching. Finally, while our method generalizes across datasets and model families, it relies on the assumption that a model's responses to follow-up questions meaningfully vary-a property that may not hold for very low-quality language models.

References

Momin Abbas, Yi Zhou, Parikshit Ram, Nathalie Baracaldo, Horst Samulowitz, Theodoros Salonidis, and Tianyi Chen. Enhancing in-context learning via linear probe calibration. arXiv preprint arXiv:2401.12406 , 2024.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Victor Akinwande, Yiding Jiang, Dylan Sam, and J Zico Kolter. Understanding prompt engineering may not require rethinking generalization. In The Twelfth International Conference on Learning Representations , 2023.

Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. URL https://openaipublic. blob. core. windows. net/neuron-explainer/paper/index. html.(Date accessed: 14.05. 2023) , 2023.

Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences , 120(6):e2218523120, 2023.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread , 2023. https://transformer-circuits.pub/2023/monosemanticfeatures/index.html.

James Campbell, Richard Ren, and Phillip Guo. Localizing lying in llama: Understanding instructed dishonesty on true-false questions through prompting, probing, and patching. arXiv preprint arXiv:2311.15131 , 2023.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2924-2936, 2019.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Julian Coda-Forno, Marcel Binz, Jane X Wang, and Eric Schulz. Cogbench: a large language model walks into a psychology lab. arXiv preprint arXiv:2402.18225 , 2024.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Matthew Finlayson, Swabha Swayamdipta, and Xiang Ren. Logits of api-protected llms leak proprietary information. arXiv preprint arXiv:2403.09539 , 2024.

Yoav Freund and Robert E Schapire. Experiments with a new boosting algorithm. In Proceedings of the Thirteenth International Conference on International Conference on Machine Learning , pages 148-156, 1996.

Dan Friedman, Andrew Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. Interpretability illusions in the generalization of simplified models. arXiv preprint arXiv:2312.03656 , 2023.

Irena Gao, Percy Liang, and Carlos Guestrin. Model equality testing: Which model is this api serving? arXiv preprint arXiv:2410.20247 , 2024.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations , 2019.

Yiding Jiang, Vaishnavh Nagarajan, Christina Baek, and J Zico Kolter. Assessing generalization of sgd via disagreement. In International Conference on Learning Representations , 2021.

Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221 , 2022.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Sanyam Kapoor, Nate Gruver, Manley Roberts, Katherine Collins, Arka Pal, Umang Bhatt, Adrian Weller, Samuel Dooley, Micah Goldblum, and Andrew Gordon Wilson. Large language models must be taught to know what they don't know. arXiv preprint arXiv:2406.08391 , 2024.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning , pages 5338-5348. PMLR, 2020.

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664 , 2023.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics , 2019.

Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models, 2023. URL https://arxiv. org/abs/2305.11747 .

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science , 378(6624):1092-1097, 2022. doi: 10.1126/science.abq1158. URL https://www.science.org/doi/abs/10.1126/science.abq1158 .

Sanae Lotfi, Marc Finzi, Yilun Kuang, Tim GJ Rudner, Micah Goldblum, and Andrew Gordon Wilson. Non-vacuous generalization bounds for large language models. arXiv preprint arXiv:2312.17173 , 2023.

Monte MacDiarmid, Timothy Maxwell, Nicholas Schiefer, Jesse Mu, Jared Kaplan, David Duvenaud, Sam Bowman, Alex Tamkin, Ethan Perez, Mrinank Sharma, Carson Denison, and Evan Hubinger. Simple probes can catch sleeper agents, 2024. URL https://www.anthropic.com/news/ probes-catch-sleeper-agents .

Alessio Mazzetto, Cyrus Cousins, Dylan Sam, Stephen H Bach, and Eli Upfal. Adversarial multi class learning under weak supervision with performance guarantees. In International Conference on Machine Learning , pages 7534-7543. PMLR, 2021a.

Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen Bach. Semi-supervised aggregation of dependent weak supervision sources with performance guarantees. In International Conference on Artificial Intelligence and Statistics , pages 3196-3204. PMLR, 2021b.

Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems , 36, 2024.

Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations , 2022.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 , 2022.

Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination for black-box language models. In The Twelfth International Conference on Learning Representations , 2023.

Lorenzo Pacchiardi, Alex James Chan, Sören Mindermann, Ilan Moscovitz, Alexa Yue Pan, Yarin Gal, Owain Evans, and Jan M. Brauner. How to catch an AI liar: Lie detection in black-box LLMs by asking unrelated questions. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=567BjxgaTp .

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 2383-2392, 2016.

Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB endowment. International conference on very large data bases , volume 11, page 269. NIH Public Access, 2017.

Alexander Robey, Zachary Ravichandran, Vijay Kumar, Hamed Hassani, and George J Pappas. Jailbreaking llm-controlled robots. arXiv preprint arXiv:2410.13691 , 2024.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

Dylan Sam and J Zico Kolter. Losses over labels: Weakly supervised learning via direct loss construction. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9695-9703, 2023.

Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the era of large language models. arXiv preprint arXiv:2402.01761 , 2024.

Ryan Smith, Jason A Fries, Braden Hancock, and Stephen H Bach. Language models in the loop: Incorporating prompting into weak supervision. ACM/JMS Journal of Data Science , 1(2):1-30, 2024.

Leonard A Stefanski and Raymond J Carroll. Covariate measurement error in logistic regression. The annals of statistics , pages 1335-1351, 1985.

Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762 , 2024.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of NAACL-HLT , pages 4149-4158, 2019.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine , 29(8): 1930-1940, 2023.

Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. arXiv preprint arXiv:2002.11448 , 2020.

Laurens Van Der Maaten, Eric Postma, Jaap Van den Herik, et al. Dimensionality reduction: a comparative. J Mach Learn Res , 10(66-71), 2009.

Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela. Learning from the worst: Dynamically generated datasets to improve online hate detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1667-1682, 2021.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368 , 2023.

Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368 , 2024.

E Hope Weissler, Tristan Naumann, Tomas Andersson, Rajesh Ranganath, Olivier Elemento, Yuan Luo, Daniel F Freitag, James Benoit, Michael C Hughes, Faisal Khan, et al. The role of machine learning in clinical research: transforming the future of evidence generation. Trials , 22:1-15, 2021.

Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems , 36, 2024.

Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864 , 2023.

Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. Predicting performance for natural language processing tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020.

Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. In The Twelfth International Conference on Learning Representations , 2023.

Yasin Abbasi Yadkori, Ilja Kuzborskij, András György, and Csaba Szepesvári. To believe or not to believe your llm. arXiv preprint arXiv:2406.02543 , 2024.

Qinyuan Ye, Harvey Fu, Xiang Ren, and Robin Jia. How predictable are large language model capabilities? a case study on big-bench. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 7493-7517, 2023.

Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in Neural Information Processing Systems , 36, 2024.

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405 , 2023a.

Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043 , 2023b.

A Additional Experiments

A.1 Full Table Results

We present the full set of our results on open-ended QA tasks (Table 5) and closed-ended QA tasks (Table 6) comparing all different methods on all LLMs applied to all considered datasets.

A.2 Distinguishing Between Levels of Quantization

We also provide additional experiments that can distinguish between different levels of quantization of a language model. We find that on SQuAD with LLaMA models, we find that QueRE can easily distinguish between model responses that are generated via different levels of quantization, while these other baselines fail.

A.3 Uncertainty Quantification Baselines

Another line of work in uncertainty quantification [Xiong et al., 2023] looks to extract estimates of model confidence from the LLM directly. This is fundamentally related to our problem setting, but perhaps is less focused on the applications of predicting model behavior (and certainly not focused on our other applications of detecting adversarial models or distinguishing between architectures). These baselines include: (1) Vanilla confidence elicitation, which is to directly ask the model for a confidence score, (2) TopK, asking the LLM for its TopK answer options with their corresponding confidences, (3) CoT, asking the LLM to first explain its reasoning step-by-step before asking for a confidence score, and (4) Multistep, which asks the LLM to produce multiple steps of reasoning each with a confidence score. We use K = 3 for the TopK baseline and 3 steps in the multistep baseline. Table 8: Comparison of AUROC between QueRE, uncertainty quantification baselines, and the vanilla model for the LLaMA3-3B and LLaMA3-8B models.

We observe that QueRE achieves stronger performance than these these uncertainty quantification baselines (Table 8). We also remark that QueRE is more widely applicable as these methods (which are implemented in Xiong et al. [2023]), as they heavily on being able to parse the format of responses for closed-ended QA tasks. On the contrary, QueRE indeed applies to open-ended QA tasks (see our strong results in Figure 2).

A.4 Models Trained on QueRE are Better Calibrated

While we have previously reported the AUROC of our predictors, we are also interested in the calibration of our models (e.g., accuracy at a given confidence threshold). This is particularly useful for high-stakes settings, when we may only want to defer prediction to a LLM when we are confident in its performance. We observe that predictors defined by QueRE generally have much lower ECE compared to those defined by using answer probabilities.

Our approach shows promise in constructing well-calibrated and performant predictors of LLM performance, which are important for the application of LLMs in high-stakes settings [Weissler et al., 2021, Thirunavukarasu et al., 2023].

A.5 Studying the Role of Diversity in Follow-up Questions

We also provide experiments to study the exact role of diversity in these elicitation questions, on top of our prior experiment using random sequences. We use various prompts to generate other types of

LLaMA3-8B

LLaMA3-70B

0.00

0.05

0.10

0.15

0.20

0.25

ECE

0.193

0.274

0.146

0.149

Answer Probs

QueRE (Ours)

LLaMA3-8B

LLaMA3-70B

0.00

0.05

0.10

0.15

0.20

ECE

0.034

0.223

0.070

0.167

LLaMA3-8B

LLaMA3-70B

0.00

0.05

0.10

0.15

0.20

0.25

ECE

0.267

0.268

0.138

0.149

LLaMA3-8B

LLaMA3-70B

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0.014

0.016

ECE

0.006

0.012

0.007

0.017

Answer Probs

QueRE (Ours)

LLaMA3-8B

LLaMA3-70B

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

ECE

0.161

0.179

0.106

0.129

LLaMA3-8B

LLaMA3-70B

0.00

0.05

0.10

0.15

0.20

0.25

0.30

ECE

0.296

0.284

0.189

0.150

follow-up questions (see Appendix D.3 for the resulting questions). One prompt attempts to produce a set of more diverse queries, while another attempts to output a set of more similar queries.

5

10

15

20

25

30

35

40

Number of Elicitation Prompts

0.70

0.72

0.74

0.76

0.78

0.80

AUROC

Elicitation Questions

Diverse Elicitation Questions

Similar Elicitation Questions

5

10

15

20

25

30

35

40

Number of Elicitation Prompts

0.70

0.75

0.80

0.85

0.90

AUROC

Elicitation Questions

Diverse Elicitation Questions

Similar Elicitation Questions

We analyze the performance of these approaches in generating elicitation questions that differ in human interpretable notions of diversity (Figure 7). We observe that generally, attempting to increase diversity does not necessarily improve performance. This suggests that as it is difficult for us to

interpret what diversity is important for these LLMs, and that the notion of diversity generated through prompting for more 'diverse" questions does not necessarily result in diverse features extracted from the LLM. We believe that better understanding this discrepancy in notions of 'diversity' is an interesting line for future research.

A.6 Unrelated Sequences Ablations

We also explore the potential of, instead of using follow-up questions, to use unrelated sequences of natural langauge. We vary the number of these unrelated sequences of language and elicitation questions to better understand the impact and importance of diversity in the follow-up questions/prompts to the model.

2

3

4

5

6

7

8

9

10

Number of Elicitation Prompts

0.65

0.70

0.75

0.80

0.85

0.90

AUROC

QueRE

Random

2

3

4

5

6

7

8

9

10

Number of Elicitation Prompts

0.76

0.78

0.80

0.82

0.84

0.86

AUROC

QueRE

Random

2

3

4

5

6

7

8

9

10

Number of Elicitation Prompts

0.50

0.51

0.52

0.53

0.54

0.55

AUROC

QueRE

Random

We observe that using follow-up questions generally achieves better performance (Figure 8). However, we still find that indeed unrelated sequences of language can extract useful information from these models in a black-box manner, which we believe is an interesting result. This suggests that generating prompts for QueRE is extremely easy, as they can take on the form of unrelated sequences of language and do not need to be limited to the form or follow-up questions. In fact, our finding that responses to unrelated sequences can reveal information about model behavior aligns with prior work describing flaws in existing interpretability frameworks [Friedman et al., 2023, Singh et al., 2024].

A.7 Additional Results for Distinguishing Models

We now present additional results on distinguishing between different model sizes on the SQuAD dataset. We observe the same trends, finding that QueRE better distinguishes between different LLaMA3 and GPT models, when compared to alternatives.

A.8 Additional Generalization Results

For our PAC-Bayes bounds over linear models [Jiang et al., 2019], we use a prior over weights of N (0 , σ 2 I ) , giving us our bound as

where L represents the 0-1 error.

We also present additional results for generalization bounds comparing the linear predictors on top of our extracted representations with those trained on the more competitive baselines (e.g., RepE, Full Logits, Answer Probs). We observe that our representations lead to the best black-box predictors with the largest lower bounds on accuracy on the NQ dataset while being outperformed on DHate.

We remark that our work defines a different line to approach generalization bounds through a more human-interactive approach to eliciting low-dimensional representations, although we remark that this human-interaction in specifying these elicitation questions must be independent of any training data (e.g., questions must be predefined before seeing the dataset of interested). Perhaps the most related work in this line are existing works that have achieved tight generalization bounds for VLMs [Akinwande et al., 2023] and for LLMs modeling log-likelihoods [Lotfi et al., 2023].

LLaMA3-8B vs LLaMA3-70B

LLaMA3-3B vs LLaMA3-8B

GPT-3.5 vs GPT-4o-mini

0.5

0.6

0.7

0.8

0.9

Pre-conf

Post-conf

Answer Probs

Sem. U

QueRE (Ours)

A.9 Robustness to System Prompts

We provide an additional experiment to illustrate that QueRE is robust to slight changes in the system prompt. We have two sets of vectors extracted via QueRE from a GPT-4o-mini model without an additional system prompt, and a version with an additional system prompt that is "You are a helpful and cautious assistant.' on the Boolean Questions dataset.

When performing linear probing between these representations, we are able to achieve an accuracy of 0.5445 , or that we cannot accurately distinguish between these two sets of vectors . Therefore, we have that adding a slight change to the system prompt does not largely influence the vectors extracted from QueRE, showing that it would not trigger these classifiers for detecting adversarial or harmful LLMs.

Furthermore, we run an experiment to check whether the classifier that distinguishes between versions of GPT-3.5 and GPT-4o-mini without any system prompt can transfer to the task of differentiating versions of GPT-3.5 and GPT-4o-mini that both have the cautious system prompts. Our model is able to perform this task with an accuracy of 0.983 , which shows us that indeed these classifiers can transfer between tasks with or without cautious system prompts . Thus, indeed our representations are robust to slight changes in the system prompt.

A.10 Representation Visualizations by Different Model Sizes

We also provide visualizations of our extracted embeddings for various LLMs architectures, noting that different models are distinctly clustered in the plots (Figure 10).

A.11 Results Using MLPs

We provide experiments that use 5-layer MLPs instead of linear classifiers to predict model performance, where each of the MLP hidden layers are of size 8. We compare different methods that extract representations (that are not single dimensional). We observe that performance is still stronger with QueRE, showing that the benefits still hold for models other than linear classifiers (Table 10).

LLaMA3-3B

LLaMA3-8B

LLAMA3-70B

GPT-3.5

A.12 Additional Results for Varying the Number of Elicitation Questions

We present additional results when varying the number of elicitation questions on other QA tasks. Here, we only look at subsets of the elicitation questions and do not include the components of preconf, postconf and answer probabilities. We observe that across all tasks, we observe a consistent increase in performance as we increase the size of the subset of follow-up questions that we consider, with diminishing benefits as we have a larger number of prompts (Figure 11). Generally, increasing the number of elicitation prompts leads to an increase in AUROC, clearly defining a tradeoff between extracting the most informative black-box representation and the overall cost of introducing more queries to the LLM API. An interesting future question is how to best select follow-up queries, and perhaps, removing those that add redundant information or noise. This is reminiscent of work in prior work in pruning or weighting ensembles of weak learners [Mazzetto et al., 2021a,b] or in dimensionality reduction [Van Der Maaten et al., 2009].

A.13 Latency Analysis

We additionally report latency-performance trade-offs for QueRE on SQuAD with LLaMA3-8B, varying the number of follow-up questions up to the full 50 used in our experiments. Table 11

5

10

15

20

25

30

35

40

45

Number of Elicitation Prompts

0.77

0.78

0.79

0.80

0.81

0.82

0.83

AUROC

5

10

15

20

25

30

35

40

45

Number of Elicitation Prompts

0.70

0.71

0.72

0.73

0.74

0.75

0.76

0.77

AUROC

5

10

15

20

25

30

35

40

45

Number of Elicitation Prompts

0.53

0.54

0.55

0.56

0.57

0.58

AUROC

5

10

15

20

25

30

35

40

45

Number of Elicitation Prompts

0.52

0.53

0.54

0.55

0.56

0.57

0.58

0.59

AUROC

compares QueRE against key black-box baselines, including Post-Conf, Self-Consistency, and Semantic Entropy.

We find that QueRE consistently outperforms other approaches at similar or lower runtimes, demonstrating a superior latency-accuracy trade-off. For a comparable latency to Self-Consistency (0.17 s vs. 0.19 s), QueRE with just 5-10 follow-up questions achieves dramatically higher AUROC ( ∼ 0.90 vs. 0.53). Furthermore, QueRE significantly outperforms the much slower Semantic Entropy baseline, achieving both higher predictive power and greater computational efficiency.

A.14 Precision and F1 on Incorrect Examples

We additionally compute precision and F1 scores on negative samples (i.e., cases where the LLM produces incorrect answers). Higher precision on these examples indicates more reliable detection of incorrect model behavior. Table 12 reports results for LLaMA3-8B and LLaMA3-70B across all QA benchmarks.

Across nearly all datasets and both model scales, QueRE remains the strongest black-box method, achieving the highest precision and F1 on negative examples. This further confirms that QueRE offers a more reliable mechanism for identifying incorrect or uncertain model behavior, complementing its superior AUROC performance.

A.15 Multi-Negative Identification

We further extend our evaluation to a multi-negative identification setting, where the goal is to determine whether a given response originates from GPT-4o-mini, with negatives drawn from a pool of three other models (LLaMA3-8B, LLaMA3-70B, and GPT-3.5). This setup more closely reflects real-world scenarios such as detecting fraudulent API substitutions or model impersonation. As shown in Table 13, QueRE remains highly effective, achieving near-perfect accuracy while maintaining strong generalization across datasets.

Across both benchmarks, QueRE achieves substantially higher AUROC than all black-box baselines, highlighting its robustness in distinguishing target model generations even in the presence of multiple distractor models. These findings underscore QueRE's practical utility for tasks that require reliable model source identification and detection of potentially substituted or spoofed model outputs.

B Proof of Proposition 1

We again present Proposition 1 and now include its proof in its entirety.

Proposition 1 (Estimator on Finite Samples from LLM) . Let ˆ β be the MLE for the logistic regression on the dataset { ( x j i , y i ) | i = 1 , ..., n, j = 1 , ..., k } , where x j i are independent samples from Ber ( p i ) . We assume there exists some unique optimal set of weights β 0 over inputs p = ( p 1 , ..., p d ) , and we let n, k >> d . Then, we have that ˆ β → β 0 as n →∞ and k →∞ . Furthermore, ˆ β converges at a rate O ( 1 √ n + √ n k ) .

Proof. Consider the standard logistic regression setup (as in the work of Stefanski and Carroll [1985]), where we are learning a linear model β , which satisfies that

Then, when optimizing β given some dataset, we consider an objective given by the cross-entropy loss

where σ i = 1 1+exp( X T i β ) . Standard asymptotic results for the MLE give us that it converges to β 0 at a rate of O ( 1 √ n ) .

In our setting, instead of having access to covariates X i , we rather have access to an approximation of these covariates ˆ X i , which is an average of k samples from Ber( X i ). An application of the results in the work of Stefanski and Carroll [1985] gives us the result that the MLE ˆ β is a consistent estimator of β 0 , given that k → ∞ . This is fairly straightforward as when k → ∞ , we have that 1 k ∑ k j =1 ˆ X j i → X i , implying that the noise in the covariates goes to 0 as n →∞ (i.e., satisfying a main condition of the result in Stefanski and Carroll [1985]).

However, we also are interested in the rate of convergence of this estimator. To do so, we perform a sensitivity analysis on β with respect to the input data x . First, we are interested in solving for the quantity

where β ∗ represents the MLE, J represents the Jacobian, and H represents the Hessian. We have that the Jacobian of the loss function is given by

and since this objective is convex and β 0 is our unique optimum, we have that

where D is a diagonal matrix with entries σ i (1 -σ i ) n . Next, we compute the directional derivative for J with our perturbation to the data as ∆ X

Taking a first-order Taylor approximation, we have that

We use this term to analyze || ( β -β 0 ) || 2 . First, we can apply the Cauchy-Schwarz inequality, which gives us that

The Hessian is given by

Then, we note that || ˆ X -X || 2 converges to 0 at a rate of O (√ d k ) via an application of the CLT. We can also analyze the term

due to the submultiplicative property of the Frobenius norm. We can bound the Frobenius norm of the left term as follows

where σ min ( A ) denotes the smallest singular value of A . We can analyze the other term by converting it into a Kronecker product. First, we will consider the term

by noting that ∆ X asymptotically approaches mean 0 with variance 1 k via the CLT, and that 1 n ( σ -y ) has a norm that is O ( √ d ) . Next, we will consider the term involving X T D ∆ Xβ . This can be rewritten as

where ⊗ denotes the Kronecker product and vec( · ) vectorizes ∆ X into a ( nd , 1) vector. Then, letting

the expected norm of this quantity can be considered as

as we note that z has mean 0 since it is the perturbation ∆ X from X . This scales the terms in A by a factor of less than 1 k . Next, we can analyze the remaining term

Now, assuming that β has norm || β || 2 ≤ B , we have that

as we note that

as all terms in the diagonals of D are smaller than 1 n and all terms in X are in [0 , 1] . Thus, we have that the Jacobian term has a norm that is bounded by

when we note that d is roughly a constant with respect to n, k , and B is a constant, and assuming that σ min ( X T DX ) = O ( 1 √ n ) . Putting this back together with the Taylor expansion and the standard asymptotics of || ˆ X -X || , we get that β converges to β 0 at a rate of O ( √ n k ) .

Finally, combining this with the rate at which the MLE converges from ˆ β to β , we can add these asymptotic rates together, giving us our result that ˆ β → β 0 at a rate of O ( 1 √ n + √ n k ) .

C Additional Related Work

Understanding and Benchmarking LLMs A large body of work has focused on understanding the capabilities of LLMs. The field of mechanistic interpretability has recently evolved around understanding the inner workings of LLMs by uncovering circuits or specific weight activations [Olsson et al., 2022, Nanda et al., 2022]. This has developed a variety of potential hypotheses for how models learn to perform specific tasks [Zhong et al., 2024], as well as the tendencies of certain activations in a LLM to activate on certain types of inputs [Bills et al., 2023, Sun et al., 2024]. Other works have studied model behavior by locating specific regions of a LLM that relate to certain concepts such as untruthfulness [Campbell et al., 2023] or honesty and ethical behavior [Zou et al., 2023a]. Our work is different in that we only assume black-box access, with a similar goal to extract information about model behavior. Finally, other work has attempted to study the abilities and performance of LLM via developing challenging benchmarks [Hendrycks et al., 2020], also including those that use techniques from the cognitive sciences [Binz and Schulz, 2023] or by comparing with human similarity judgments [Coda-Forno et al., 2024]. While these approaches look to benchmark and quantify performance in aggregate over tasks, our setting looks to predict the performance at the example level, for deciding when to trust or use LLMs in deployment.

D Additional Experimental Details

We now provide additional experimental details for our adversarial system prompt experiments, as well as prompts used to generate our follow-up questions. We also provide all of the questions used by QueRE.

D.1 Adversarial QA Experiment Details

We now present additional details for our adversarial QA experiment in Section 4.2. We consider the following 6 system prompts (3 harmful and 3 helpful) for our classification task between clean and adversarially influenced version of LLMs:

We remark that influencing the LLM in this fashion indeed works, as the first harmful system prompt drops the performance of GPT-3.5 on BoolQ from 87% to 32% and on DHate from 77% to 28%.

D.2 Adversarial Code Experiment Details

We now present additional details for our adversarially influenced code generation experiment in Section 4.2. In our setting, we take 500 train examples of coding prompts from the Code Contests dataset [Li et al., 2022] and evaluate on their full test set. We evaluate the ability of linear predictors trained on top of QueRE and alternative approaches to detect adversarially influenced models that behave normally when answering normal questions but introduce bugs into generated code.

The exact adversarial system prompt that we use is:

D.3 Prompting Details

We now present the prompts that we use to generate our elicitation questions and the other sets of prompts that we use in our ablations.

Prompting for Follow-up Questions To generate our eliciting queries through GPT4, we use the following prompt:

Prompting for Random Sequences To generate our prompts for random sequences of natural language for our ablation study, we prompt GPT4 with the following prompt:

Prompting for More Diverse Follow-up Questions To generate more diverse follow-up questions for our ablations, we use the following prompt:

prompt_diverse = "Can you generate a list of 40 short 'yes/no' questions designed to prompt a language model to explain its behavior? For example, 'Do you think your answer is correct?' Please ensure that these questions are diverse, distinct, and varied in phrasing and content."

Prompting for More Similar Follow-up Questions We also generate a set of more similar and redundant follow-up questions via the following prompt:

prompt_redundant = "Can you generate a list of 40 short 'yes/no' designed to prompt a language model to explain its behavior? For example, 'Do you think your answer is correct?' Ensure that these questions are similar in nature, with some being rephrasings of the same question."

D.4 Follow-up Prompts for QueRE

Pre/Post-Confidence Prompts To compute our pre and post-confidence scores, we use the following prompts:

Follow-up Questions For our hand-specified follow-up questions, we use the following questions:

The resulting eliciting queries (resulting from prompting GPT-4o) that we use in QueRE are:

Random Sequences We use the following random sequences of natural language (again generated via GPT-4o) for our ablation study.

D.5 Dataset Details

For all datasets, we truncate the number of training examples to the first 5000 instances from each dataset's original train split (if they are longer than 5000 examples). We take the first 1000 instances from each test split to construct our test dataset. For the experiments with the LLaMA3-70B and GPT models, we use 1000 instances for the training datasets due to computational costs.

We also note that for the HaluEval task, we use the 'general' data version, which consists of 5K human-annotated samples for ChatGPT responses to user queries. On HaluEval, we only take 3500 instances from the training dataset due to its size. On our SQuAD task, we evaluate using exact match and use SQuAD-v1, which does not introduce any unanswerable questions, as unanswerable questions makes the evaluation metric less straightforward to compute. On WinoGrande, we use the 'debiased' version of the dataset. On the NQ dataset, we prepend prompts with two held-out training examples to have the LLMs better match the answer format.

For evaluating model performance on Natural Questions (NQ) [Kwiatkowski et al., 2019], we measure if the LLM has outputted one of the valid answers to the question. As mentioned previously, we use GPT-4o as a LLM judge to assess performance on CodeContests and on GSM8k.

Semantic Uncertainty Details For the semantic uncertainty baseline, we use the default 10 generations for each question. For clustering, we use their Deberta bidirectional entailment approach, without strict entailment.

QA Task Formatting To format our prompts to LLMs, we leverage the instruction-tuning special tokens and interleave these with the question and answer for our our in-context examples on Natural

Questions. For all MCQ tasks, we use the standard set of answers of ('True', 'False') or ('A', 'B', 'C', 'D', 'E') when they are the existing formatting in the dataset. The one exception is WinoGrande, where we map the two potential answer options onto choices ('A', 'B").

D.6 LLMInference and Downstream Model Training

For our LLMs, we load and run them at half precision for computational efficiency. To train our downstream logistic regression models, we use the default settings from scikit-learn, with the default (L2) regularization. We balance the logistic regression objective due to the unbalanced nature of the task (e.g., models are mostly incorrect on very challenging tasks).

D.7 Generalization Details

For our generalization details, we use PAC-Bayesian bounds over the linear models, as is outlined in the work of Jiang et al. [2019]. Here, we consider a prior of weights specified about the origin, with a grid of variances of [0.1, 0.11, 0.12, ..., 0.99, 1.0]. For the generalization experiments, we balance both the train and test datasets as we evaluate the accuracy of different predictors.

D.8 Computational Resources

Our largest experiments are with LLaMA3-70B, which are run on a single node with 4 NVIDIA RTX A6000 GPUs. The other experiments are run with ≤ 2 RTX A6000 GPUs. For each model and dataset, running inference over the datasets takes roughly 24 hours and 100GB of RAM.

D.9 Asset Licenses

The existing assets that we use have the following licenses:

LLaMA3 Models: LLaMA3 License

BoolQ: Creative Commons Attribution Share Alike 3.0

HaluEval: MIT License

Commmonsense QA: MIT License

DHate: CC BY 4.0

SQuAD: Creative Commons Attribution Share Alike 4.0

Natural Questions: Apache-2.0 license

WinoGrande: Apache-2.0 license

GMS8K: MIT License

CodeContests: CC BY 4.0

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims are supported by experimental and theoretical results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in the Discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The proofs are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All experimental details are provided.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is provided in the supplement.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All details are provided in the code in the supplement.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars are provided when applicable.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute details are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The submission conforms with the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Impacts of the paper are discussed in the discussion section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No such risks are posed.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Licenses for all assets are mentioned in the Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: All code details are provided in the supplement.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: The usage of LLMs in extracting features and evaluating them as predictors is clearly outlined in the paper.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper defines the task of black-box language model performance prediction and proposes a method (QueRE) based on follow-up queries to elicit predictive features from model outputs for model behavior prediction (e.g., truthfulness of generated responses). This approach is evaluated on three applications: predicting instance-level correctness in open/closed-ended QA tasks, detecting potential adversarial prompts, and distinguishing between black-box LLMs. QueRE performs competitively, often matching or outperforming white-box baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Pros:**\n1. The paper formally introduces a practically relevant task, especially since many mainstream LLMs are only accessible in black-box settings through APIs.\n2. The authors design various evaluation settings and present extensive quantitative results. Some interesting applications are included, such as black-box model identification and performance prediction.\n3. QueRE generally performs competitively even compared with white-box baselines. \n4. The paper is clearly written and easy to understand.\n\n**Cons:**\n1. The technical contribution of the proposed QueRE is limited, as it relies heavily on LLM-generated prompts (i.e., follow-up questions) and an overly strong hypothesis that the predictive distributions for these questions meaningfully vary with correctness.\n2. The significance of this task is closely tied to the achievable performance upper bound. I think the authors should further investigate whether the task of black-box performance prediction is in fact predictable—i.e., whether LLMs are sufficiently aware of their own factually incorrect outputs, and whether this internal perception can be effectively extracted via follow-up questions. If there is not enough headroom in performance, further exploration in this area may have limited practical utility.\n3. **Regarding experimental settings:**\n- In the model performance prediction task, the authors should additionally report precision and F1-score on negative samples (i.e., cases where LLMs produce incorrect answers), as higher precision indicates more reliable detection of incorrect behavior.\n- A more realistic setup for black-box model identification should be determining whether a given response is or is not from a specific model. This implies that the negative samples in the test set should come from multiple different LLMs. The current setting—distinguishing between two specific models—is much easier and less representative of real-world applications, such as detecting fraudulent API substitutions.\n\nQUESTIONS:\nPlease refer to Strengths And Weaknesses.\n\nLIMITATIONS:\nThe method’s reliance on multiple follow-up queries per instance could increase inference cost in deployment settings.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes QueRE, a black-box method of predicting model performance as well as discriminating models. The method is then compared to baselines, both white-box and black-box, on a variety of tasks ranging from predicting model performance on closed and open-ended question answering to distinguishing between different models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1) Simple approach that is not tailored to a specific model\n2) Strong performance and generalization compared to other methods\n3) Useful and practical approach that is directly applicable\n4) Experiments on many datasets and different tasks\n\n**Weaknesses**:\n1) Lack of datasets that are not potentially contaminated (As far as I could see, HaluEval and GPT3.5 is the only combination that is not potentially contaminated, assuming that later versions of GPT3.5, namely GPT3.5-Turbo-0125, have not been further trained).\n2) The choice of using pre- and post- confidence scores in addition to the original vector z in the case of QA tasks should be ablated.\n\nQUESTIONS:\n1) In L117, how is the distribution over the set of possible answers calculated in the case of LLMs that don’t return probabilities? Also, I think the sentence is grammatically incorrect. Either the distribution over possible answers or what?\n2) As mentioned in weaknesses point 1, the datasets in section 4 are mostly released before the models evaluated have been released. Do the authors think this would generalize to newer datasets that are definitely not in the training set of these models?\n3) Would it be possible to measure the ability of this approach to differentiate between a full/half precision model and a more severely quantized (<=4bit) version of the model? I think this is another important usecase for evaluating serving APIs aside from checking if they are serving the right model.\n4) Minor detail for final version: The code has a typo with the name of the Boolean Questions (BoolQ) dataset, using i instead of an L for the dataset name.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an approach for predicting the behavior of black-box language models by asking them a series of follow-up questions. The approach, QueRE, works by taking the probabilities of \"Yes\" responses to these questions and using them as features to train a simple linear predictor. The research demonstrates that this black-box technique can reliably and accurately predict a model's correctness on various question-answering and reasoning benchmarks, surprisingly outperforming even white-box methods that have access to the model's internal activations. Furthermore, the paper shows that these features can be used to effectively detect when a model has been adversarially influenced by a system prompt (e.g., to answer incorrectly or generate buggy code) and to accurately distinguish between different black-box LLMs (e.g., GPT-3.5 vs. GPT-4o-mini), offering a promising direction for monitoring the reliability and integrity of closed-source models.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper has the following key strengths:\n- A simple yet highly effective approach: The proposed approach does not require access to model internals to calibrate the confidence but is remarkably effective, often matching or even outperforming white-box methods that require access to the model's internal hidden states or full logits.\n- Versatility across multiple critical tasks: The utility of the features extracted by QueRE is demonstrated across three distinct and important applications. It can accurately predict instance-level correctness on a wide range of QA and reasoning benchmarks, reliably detect when an LLM has been adversarially influenced by a system prompt, and accurately distinguish between different black-box LLMs, which is useful for auditing APIs.\n- Strong generalization and transferability: The predictors trained using QueRE show strong generalization properties and perform well on out-of-distribution data. Experiments demonstrate that the predictors transfer effectively across different QA datasets (for example, from SQUAD to NQ) and across different model scales (for example, from LLaMA3-3B to 8B) without needing new labeled data from the target task.\n\nThe paper has the following weaknesses:\n- Computational cost overhead compared to model internals approach: The approach relies on asking n follow up questions (n = 50 shows good performance in the paper). The authors note in line 116 that these follow up questions can be asked in parallel adding minimal computational overhead. However, parallelizing the computation would reduce latency but the cost would still remain high. So for a given query, a model would have to be called 50 times to estimate the calibrated confidence which is quite expensive. Model internals approach as well as other baselines do not have such high computational costs. Also, since follow up questions are asked which rely on using the original output as an input, latency would also increase. I would suggest the authors to mention this explicitly in their paper for transparency.\n- Results only shown on simpler tasks: The datasets considered in the paper are comparatively simpler for the LLMs shown in the paper. For example, on GSM-8K, GPT-4 has a performance of ~95%. Confidence calibration in such tasks would be a lot different than tasks on which the models have medium or low performance (such as Olympiad level math problems, multi-turn function calling). Having results on such difficult tasks can certainly strengthen the paper. I would also encourage authors to add model’s accuracy for each of the tasks so that this is also clearer. \n- Comparison to stronger baselines like MICE: The authors have included white box and black box baselines for comparison. I would also encourage the authors to include stronger baselines such as [1].\n\n[1] Subramani, Nishant, et al. \"MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools.\" Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025.\n\nQUESTIONS:\n1. The approach shows surprisingly strong performance in predicting correctness on reasoning-intensive tasks like GSM8K and CodeContests, where many other methods fail.  Given that the follow-up questions seem to probe for general confidence rather than specific reasoning steps, do the authors have a hypothesis for why these features are so predictive for complex reasoning tasks?\n2. The ablation study in Figure 5 shows that performance increases with the number of elicitation questions.  Have the authors investigated methods for optimizing this set of questions to identify a minimal, yet highly informative, subset? This could help reduce the computational cost associated with the method.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to predict the behavior of black-box language models, e.g. predicting whether their outputs are correct. The paper proposes a method by asking follow-up questions and taking the probability of responses (e.g. the probability of \"yes\") as representations to train linear probes. Surprisingly, this simple method can outperform white-box linear method (e.g. train on the output logits).\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- This is an interesting new research problem. The demonstrated use case (e.g., judging output correctness, distinguishing adversarially perturbed LMs, and differentiating different LMs) justifies the importance of this problem setup.\n- The main idea of asking follow-up questions is common, but to the best of my knowledge, this is the first paper that takes the probability of responses as representations to train linear probes. \n- The experiment results are strong. QueRE consistently outperforms a wide range of strong baselines, including white-gox methods like RepE and Full Logits.\n\nWeaknessess\n- While QueRE gets strong performance on open-sourced models, it sometimes gets limited performance on closed-sourced models (e.g. Figure 2). \n- The studied benchmarks (e.g. SQUAD, Natural Questions) are outdated. It'd be great to see experiments on more realistic benchmarks (e.g. coding, human preference). \n- It's unclear how different follow-up questions might influence the performance of the proposed method. In particular, some follow-up questions are hand-specified, and the others are generated by prompting GPT-4.\n- While the authors claim that the latency issue can be mitigated through batching, I didn't find any relevant reports about the computation overhead, especially with more follow-up questions (e.g. 100). Could you present the results of latency v.s. AuROC?\n\nQUESTIONS:\nsee weaknesses.\n\nLIMITATIONS:\nDo you think there are any types of tasks that the proposed method might fail?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}