Confidence Elicitation: A New Attack Vector for Large Language Models
Authors: Brian Formento
Chuan-Sheng Foo
See-Kiong Ng
Venue: ICLR 2025 Poster (2025)
Keywords: adversarial attack
adversarial robustness
confidence elicitation.
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: Using confidence elicitation to find adversarial samples

Abstract: A fundamental issue in deep learning has been adversarial robustness. As these systems have scaled, such issues have persisted. Currently, large language models (LLMs) with billions of parameters suffer from adversarial attacks just like their earlier, smaller counterparts. However, the threat models have changed. Previously, having gray-box access, where input embeddings or output logits/probabilities were visible to the user, might have been reasonable. However, with the introduction of closed-source models, no information about the model is available apart from the generated output. This means that current black-box attacks can only utilize the final prediction to detect if an attack is successful. In this work, we investigate and demonstrate the potential of attack guidance, akin to using output probabilities, while having only black-box access in a classification setting. This is achieved through the ability to elicit confidence from the model. We empirically show that the elicited confidence is calibrated and not hallucinated for current LLMs. By minimizing the elicited confidence, we can therefore increase the likelihood of misclassification. Our new proposed paradigm demonstrates promising state-of-the-art results on three datasets across two models  (LLaMA-3-8B-Instruct and Mistral-7B-Instruct-V0.3) when comparing our technique to existing hard-label black-box attack methods that introduce word-level substitutions. The code is publicly available at GitHub: Confidence_Elicitation_Attacks.

Published as a conference paper at ICLR 2025
CONFIDENCE ELICITATION: A NEW ATTACK VECTOR FOR LARGE LANGUAGE MODELS
Brian Formento Chuan Sheng Foo See-Kiong Ng! 'Institute of Data Science, National University of Singapore Insttitte for Infocomm Research, *STARR CCentre for Frontier AI Research, A*STAR brian. formentolu nus.ede foo_chuar www.cchcaunnnnn..ee shenge 1 a s-staar... edu. sg seekiong@nnn www.nnussuuuuuuuuoo edu.sg sg
ABSTRACT
A fundamental issue in deep learning has been adversarial robustness. As these systems have scaled, such issues have persisted. Currently, large language mod- els (LLMs) with billions of parameters suffer from adversarial attacks just like their earlier, smaller counterparts. However, threat models have changed. Previously, having gray-box access, where input embeddings output log- its/probubilities were visible to the user, might have been reasonable. However, with the introduction of closed-source models, information about the model is available apart from the generated output. This means that current black-box at- tacks only utilize the final prediction to detect if attack is successful. In this work, we investigate and demonstrate the potential of attack guidance, akin to us- ing output probabilities, while having only black-box access classification set- ting. This is achieved through the ability to elicit confidence from the model. We empirically show that the elicited confidence is calibrated and not hallucinated for current L. By minimizing the elicited confidence, we can therefore increase the likelihood of misclassification. Our new proposed paradigm demonstrates promising state-of-the-art results on three datasets across two models (LLaMA-3- SB-Instruct and Mistral- TB-Instruct-VUL isting hard-label black-box attack methods that introduce word-level substitutions. The code is publicly available at GitHub: Confidence Elicitation Attacks
when comparing our technique to ex-
INTRODUCTION
Deep learning has demonstrated remarkable performance across variety of tasks and fields, includ- ing computer vision, NLP, speech recognition, and graph representation learning. These technolo- gies are employed for tasks such as classification, question answering, and more. However, deep learning models are known to be vulnerable to adversarial attacks (Szegedy et al. [014). This vul- nerability persists even as models have scaled up include billions of parameters, especially in the form of LLMs.
Such vulnerabilities are particularly concerning in critical applications, such as healthcare (Savage et al. 2024), socio-technice systems and human manhine collaboration (Sale et al. 2024). For ex- ample, in healthcare, where a medical system provides a diagnosis, an attacker might introduce input perturbations, aiming to achieve misclassification. In clinical support systems, such misclassifica- tions can have lethal consequences (Ghaffari Laleh et al. (022).
Providing confidence estimates through confidence elicitation, whether in template or free-form generation, has been shown to enhance the performance and utility of these systems. This is par- ticularly important in domains where assessing the reliability of model's responses is crucial for effective risk assessment, error mitigation, selective generation. and minimizing the effects of hal- lucinations. As result, quently, exploring whether we can strengthen adversarial perturbations using confidence estimates an important area of research, with the aim of designing more robust systems.
anticipate these techniques to become more widespread. Conse-

---

Published as a conference paper at ICLR 2025
Adversarial attacks can be classified primarily into three categories: white-box attacks, where every part of the model, including the gradients, is known at the time of the attack: grey-box attacks, where some information, such input embeddings, output logits or probabilities are available, and black-box attacks, where no information except for the output prediction known. Based on this classification, common types of adversarial attacks include traddeettbased methods (Ebrahimi et al., 2018: Shin et al. (2020), soft-label attacks a form of grey-box scenario where only output probabilities are available (Jin et al,, 2019; Ren al. 2019 Li et al.) 2020), and black-box hard- label scenarios, where only output predictions are accessible (Maheshwary et al. 2020b; Ye et al., 2022
Yu et
2022 Liu et al. )
Despite previous work in this area, the soft-label scenario has been regarded as unrealistic because I.LMs (In particular, commercially available models) are now accessed via an API that returns only the generated output, without providing the probability distributions or logits across the cat- egories/vocabulary (Ye et al.) 2022). while current hard-label approaches often require significant number of model queries, they follow a top-down optimization method illustrated in Figure 2 This method involves initially over-perturbing a sample and then gradually modifying it to maintain its semantic properties. Throughout this process, the model must be queried continuously to verify whether the changes result in the new sample remaining adversarial, which can be problem if the API is rate-limited. Furthermore, it can be argued that expecting only basic outputs is beyond the scope for modern LLMs that perform free-form generation. In fact, most previous hard-label works focused on BERT-based models rather than the new LLMs capable of performing classification in multiple ways.
We therefore investigate the realistic scenario of hard-label attacks on LLMs and examine whether some of their emergent abilities under a free-form generation setting can be leveraged to perform these attacks.
In this paper, we demonstrate that it is possible to approximate soft labels, essentially allowing for hard-label attack with more information. This is achieved through the technique of confidence elicitation (Xiong et al.) 2024), where we simply ask the model for its own uncertainty.
Our main contributions are as follows: Novel Attack Vector: confidence elicitation can be used as potential attack vector on LLMs, while providing strong motivations for why anyone would want to take this approach. Effective Black-Box Optimiza- tion: We demonstrate that confidence elicitation can be used effectively as feedback in black-box optimization to generate adversarial examples. Our evaluation across three datasets and two mod- els illustrates that black-box optimization is achievable even with imperfectly calibrated models. State-of-the-Art Hard-Label Attack on L.Mss: Our methodology achieves state-of-the-arr per- formance in hard-label, black-box, bard-substitution-aan current state-of-the-art hard-label optimization technique (SSPAttack), our method results in better Attack Success Rates (ASR) with fewer queries and higher semantic similarity. We also release our _
We are the first to investigate whether
attacks on LLMs. Compared to the
RELATED WORK
2.1 ATTACKS ON LLMS
The traditional adversarial attack formulation involves adding subtle perturbation 8 the original that Tadv = + 8 (Szegedy al. 2014). There are many ways to calculate 6; the overarching idea is that we want to add a perturbation to x to identify regions of high risk the input space (hhu et al. 2020). This, in turn, is expected to increase the output loss for a given task La
One of the most effective ways to find regions of high input risk remains the fast gradient sign method (FGSM) (Goodfellow et al. 22014), the basic iterative method (BIM) (Kurakin et al. 2016), and projected gradient descent (PGD) (Madry et al. 2019). These methods utilize gradient informa- multiple iterations.
tion find an optimal 5 given bound with either
These first-order techniques model a tractable maximization operation over a non-convex loss land- scape follows:
We release our code in GitHub Repository Confidence Elicitation Attacks)
2

---

Published as a conference paper at ICLR 2025
Sempling simiagy
Prompting
Resporse
Proking
Simpl: M Reepornes
Anwe:
Quesion
Col.
LLM fs
Aggragaoor
type
LLaMA W00 Meta
M
Resporse K
Nomercdly welilledd
Pronpe Guesacs, Eliciution Visholied
response desermunisto
Augregatte: Deichle
Topsk sumpling.
Sample
wit parsee and celllino
Appoocratis ofhobaiiir by Dincie
Answer Fale:
your guesses quosiuu (Qusation) provide vorbal confidence that your guzceus ane uxnect
Cunfilenne
Questin::
Ariontle laptop?
LLM fs
LLaMA by Meta
Attack!
Stop
Provide guescc for quotae Thr Aresant une sptop!) proviee your crufadmncs
Arawa:
Fabic Fibe, Toxc
Assea (g) File
Exenderee ligh Lightn Law
Contidane gp
Provide your gusaca quediion
(a f = ela Aisate utiaee bypop?) and provide your confidencce
LLM fs
Anssvr Take, Tala, The Exodideree Laose, Low
Contilmne QCC)
Arouu (gt Tale
Provide lotal d, 22 computer/) provide you confidenes
LLaMA by 00 Meta
utiize
Thee
(g)
Ead
Compute perturbation
Stop
Computc perturbation &g
Figure I: Confidence elicitation attack on an LLM, assuming a classification task (Start), x has a ground truth y = false, first perform inference and extract the model's prediction and original elicited confidence Pe (green, Step 1), we use the confidence as feedback (Step 2) to determine whether perturbation 6, modelled after word substitution "use" "utilize" (Step 3) added the input leads to lower confidence (yellow, Step 4), we carry on adding ôs to the input + Lade until achieve misclassification (red, End).
p(0) (a max L(O, 8.9) AE
Where p(0) represents the worst possible perturbation for æ on model fo with parameters 6. When we maximize L(0, x + y). we do so by using multiple 8 from set A of all possible perturbations given a bound e. This bound is often set to the L2-Norm: []0]2 S e] or Lxo-Norm: [[8] < e).
These techniques and their variants have found moderate success when applied to the input em- bedding (continuous) space as white-box attacks (Dong al. [0011 Zhu et al.) 2020 Dong al.) 2021b). However, most language applications interact with LLMs through the token space. Al- though previous work has attempted to use gradient information to perform attacks in the token space (Ebrahimi al., 2018 Zou et al. 2023 Wang al.) 2020b), the projection from continuous to discrete space results in high perplexity and low semantics (Zhu et al. 2024 Liu et al. 2024b).
Given the unrealistic threat model of having access gradients and input embedding spaces, some efforts have explored adversarial attacks the token space by perturbing at the character (Li et al.
2019 Eger 2019 Formento et al. 2021 2023). word (Jin et. 2019 Ren al. 2019 al.

---

Published as a conference paper at ICLR 2025
2020
Tan al. 2020). or sentence level (Wang et al. 2020a lyyer al. 2018: Gan & Ng 2019
Bhuiya et al. 2024). These attacks often use output probabilities as feedback while employing heuristic search techniques, such beam search, greedy search, or particle swarm optimization (Zang et al. 2020). These methods have been exceptionally effective on BERT (Devlin et al. 2019)- based encoding models. However, widely used commercial L.LMs (e.g. ChatGPT) are closed- and logits or probabilities are not available; attacks have to operate in a purely black-box setting with only hard- hard-predictions available as feedback (Liu al.) 2023). Recent works have explored using LLMs to red team (perform multiple black-box hard label attacks (Maheshwary et al. (020aa) on other LLMs with moderate success (Chao et al.) 2023 Xu al., 2024). We believe the lack of feedback in the perturbation (ń) optimization process is holding these attacks back.
In this paper, we propose novel attack technique we call confidence elicitation attacks, which aim to attack models in completely black-box setting while still utilizing feedback from the model in the form of elicitation. Our work shows promising state-of-the-ae results on word substitution attacks on LLMs.
2.2 CONFIDENCE ELICITATION
Multiple studies have explored calibration in language models. A common method, which has been thoroughly explored in previous work (Guo al., 2017a; Jiang et al., (02T)) involves using output probabilities proxy for confidence. This could be implemented by focusing on the first generated vector for a specific token, by adding a binary classification prediction head that utilizes the last generated token (Kadavath et al. 2022), focusing on the answer specific token or take the average of the probabilities across the whole sequence, these techniques have been classified as white-box confidence estimation. While these approaches could be effective, several challenges arise. Firstly output logits or probabilities may not be accessible, particularly with proprietary models. Secondly the likelihood of the next token primarily signifies lexical confidence and not epistemic uncertainty (Lin et al.] 2022). and therefore, struggles to capture the semantic uncertainty in the entire text Xiong al. 2024).
As a result, previous work highlighted the need for models capable of directly expressing uncer- tainty in natural language in a black-box setting. Some research has explored enhancing calibration by empirically deriving confidence through repetitive model querying (Portillo Wightman et al., 2023). Alternatively. models can be prompted to express their confidence verbally, either through verbalized numerical confidence elicitation (Xiong et al.) (024) or verbal confidence elicitation (Lin al. 2022). has been found that some prompts can achieve reasonable uncertainty quantification, especially by querying the model twice, first for the prediction, and the second time for the uncer- tainty estimates (Tian et al. 2023) (Example of a prompt for confidence elicitation is in Table 6 in the Appendix).
METHODOLOGY
In this work, we assume classification setting similar previous work (Xu al., 2024). Here, model fo maps fo X' + y and is used to classify a sample x with ground truth label y. This is done by first obtaining token IDs Z + f(z). where is a tokenizer, and then using such IDs look up the corresponding embedding vectors from embedding matrix E so that X + E(Z). The model will then make prediction ý fx) The goal of an adversarial attack to modify through an adversarial sample generator g (which is an attack algorithm) and a perturbation 5 (such as word substitutions or character insertions) to produce zadv. such that y o ÿ. The adversarial sample zndv can be evaluated for linguistic qualitative or quantitative properties using an evaluation algorithm d,-
p(0) E
max C(O, +6,)
(2)
DEA
In an adversarial settings, minimizing the confidence C' in the correct class y can be linked to maximizing the probability of misclassification.
The approach is Equation 2 is illustrated in the dashed red box of Figure I As 5 substitutions added to the input sample (green), pC decreases (yellow) until a misclassification occurs False - True (red), at which point the confidence can be maximized.
4

---

Published as a conference paper at ICLR 2025
[A] Setting
pobelilty
Class
produbiity Adh coarid surple
scendesoryy
Clavo
Class
(C) waylac-dl Sarasiiii entaa werh
Fadr
Po
Figure 2: Confidence Elicitation Attack on an LLM: Left) SSPAttack and other previous hard- label attacks first (A) perform multiple 8 word substitutions, so that heavily perturbed sample is misclassified. Then they (B)((C) perform further optimization to improve the adversarial sample's quality. Right) In contrast, CEAttacks take bottom-up approach by progressively perturbing the original sample with 5 word substitutions until misclassification is achieved, using model guidance through probability approximations. The adversarial perturbation is bounded by e to preserve its quality.
3.1 CONFIDENCE ELICITATION ATTACKS
In this section, we outline a method to attack model by leveraging its confidence levels. Assuming an ideal scenario, where the model black-box, calibrated and always outputs a response with it's prediction and confidence values (Ulmer et al. 2024), the process involves querying the model using a prompt that includes an output request for its answer's confidence. The response, confidence value a percentage between 0 and 100 verbalised, is then subjected string analysis. The procedure can be formulated a generalized confidence elicitation attack, detailed in Algorithm
The algorithm initiates by querying the model with an input to obtain both prediction ú and the model's fe confidence in that prediction, denoted as Pş- Subsequently, a generator ga. designed to perturb the input by factor of 8, is employed produce adversarial example Fadv- This adversarial input is then verified by a discriminator de with bound which checks for adherence specified linguistic constraints. Should the constraints not be met, 98 is requested to generate revised Fade-
Once acceptable adversarial sample is obtained, it is processed by fo to determine the confidence level of the adversarial example, denoted as Pc. This new confidence is then compared with the original Py decide whether accept reject the perturbations introduced in Fadv.
4 EXPERIMENTAL SETUP
We conducted our confidence elicitation attacks Meta-Llama-3 La--llaaa-----mmmaa Instrruct (Touvron al. 2023) and
(Jiang et al. 2023) while performing classification on two common
datasets to evaluate adversarial robustness: SST-2, AG-News and one modern dataset: Strategy
Geva et al. 202I). We utilize the evaluation framework previously proposed in (Morris et al.. 2020),
where an evaluation set is perturbed, and we record the following data metrics: Clean accuracy (CA), Accuracy under attack (AUA), the Attack success rate (ASR), Semantic similarity (SemSim) based on the Universal Sentence Encoder (Cer et al.) 2018). We compare the original perplexity with the new perturbed sample's perplexity. Queries where we subdivide this metric into two categories: All Att Queries Avg and Succ Att Queries Avg. Additionally, we track Attack Time. We compare our guided word substitution attacks, CEAttack to "Self-Fool Word Sub" from (Xu et al., (024), TextHoaxer Ye et al. 2022) and SSPAttack from (Liu et al. 2023). We discuss every metric and baseline in more detail in Appendix E
4.1 IMPLEMENTATION
Prompting From Figure I we first initialize the model using a two-step prompting strategy. This strategy consists of initial k guess query the model, yielding kpred guesses, followed by
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: Confidence elicitation attack
Input:
Initial imput z, Prompt for perturbation, Vocabulary {T1, 72.... TV original sample confidence Py
Output: Predicted class ÿ, adversarial sample Fado conditions are met
Initialize generator function Is and input æ.
while not exceeded number of queries do
repeat
Endv < Output from
until (xade) is true
Compute prediction and confidence: ÿ. Pe + fa( predi on and calibrated confident
// Assumes fo returns
if Pc < Py then Substitute z with Tode
Py f Pc
// Update previous confidence to current
10
else if Pc 2 Py then
11
Perform an alternative perturbation or adjustment.
2 +- Some function of Tade that modifies or reverts changes
13
Py t Pc
// Optionally update the reference confidence
second query to the model to obtain verbalized confidence levels for these guesses keonf. The confidence levels used are 'Highest', 'High', "Medium', 'Low', and 'Lowest'. This technique has been demonstrated to effective in previous work (Tian et al. 2023 Lin et al. 2022). our experiments set & to 20 for SS72 and AG-News and 6 for StrategyQA.
Model Model-wise, previous confidence elicitation works (Xiong al. 2024 use model settings commonly found in generative tasks where the model samples from the top-k (top-k: 40) most probable next tokens, applies top-p (p 0.92) nucleus sampling, which only considers next tokens with high probability, and uses a temperature setting of T = 0.7. This setup naturally introduces some randomness the model, whose behavior is still not fully understood in an adversarial setting, as highlighted in previous work (Huang et al., 2023 Zhao et al.) 2024] Russinovich et al.) 2024). We keep all the settings consistent with previous work, but set O, which follows previous work related to adversarial evaluation Xu et al. 2024).
Dirichlet aggregation The differences among each of the kyred and kcon, can be viewed as a form of epistemic uncertainty. To model our confidence thresholds using these kpreed and Reonf, employ a Dirichlet distribution. First, we construct an a vector where each prediction in kpred is assigned value of 1. For each class, then add the following foundational values from keonf 'Highest' = 'High' = 4, 'Medium' 3, 'Low' = 2, and 'Lowest' = 1. Additionally, an dg with a value of is included. With this aevector, classes. The mean (expectation) of the Dirichlet distribution is given by μe ',a The mean is taken the probability of the classifier, where the classifier uses the argmax of these probabilities for classification.
[ar,, 02, ac), where C the number of expected
Adversarial setup Given a sample x, we first extract an ordered subset of words W C x to perturb randomly, with the size of W capped at |W| 5. For each word w e W. we obtain set of syn- onyms sourced from Counter-fitted embeddings (Mrkšić et al. 2016). We introduce a perturbation 8 in x by performing a word substitution, replacing w with synonym s from S. For each synonym replacement, we generate transformation. The list of all such transformations is denoted as T T....s> where TweS represents the original sample x with the word w systematically replaced by all pos- sible synonyms in S. Formally, for each E W we first identify the set $ = {31, 82...... ... 8n), where is the set of synonyms for w obtained from counter-fitee word embeddings and then per- form substitutions of transformed samples T, Tz = {Iwe-s s E S} where we evaluate each Fado E T Tim_s to de- termine if it successfully achieves drop in confidence in the prediction. Querying the model with
create transformed samples {Iwees Twe-ess..... ). This yields set

---

Published as a conference paper at ICLR 2025
Tado produces new Kproed and Kemn resulting in new a'- With the new a' veetor derived from Todes the new mean p' will be p/ Σ
We aim to induce misclassification by minimizing the probability of the current class:
max dE (8. -(14)
Where 8 E A represents the set of perturbations given [W| and [S]. For problem, aim to minimize the current class probability based on the Dirichlet mean, J.. a transformation Zade succeeds in lowering the confidence level of the model's output, retain that word substitution. Otherwise, we proceed to the next word W. This iterative process similar to a hill-climbing greedy algorithm, where we aim perturb the sample æ iteratively to achieve the desired reduction in model confidence.
5 RESULTS
5.1 MODEL CALIBRATION
By calculating the Expected Calibration Error (BCE) (Guo al.) 2017b), the Area Under the Re- ceiver Operating Characteristic Curve (AUROC), and plotting reliability diagrams (Xiong et al. 2024 we can evaluate how well a model is calibrated for confidence elicitation. ECE is met- ric used to assess how well model's confidence estimates align with the actual probabilities of outcomes being correct. For example, helps evaluate how accurately a model's predicted confi dence (e.g., 'I'm 80% sure this is correct') matches reality. This assessment is averaged across 500 examples. thorough explanation of ECE is provided Appendix E.4
We first demonstrate that LLama3 is well-calibrated for the SST2, AG-News, and Strategy tasks, as illustrated Table T and Figure 3 contrast, Mistral-VO.3 is reasonably well-calibrated for these same tasks, shown in Table I and Figure 6 in Appendix F.3 The Expected Calibration Error (ECE) is low across all our tests. Additionally, all tests exhibit a high AUROC. Furthermore, for SST2 and StrategyQA with LLama3, the reliability plots are close the expected diagonal, indicating that the model performs as expected on these tasks and demonstrates some awareness of its own uncertainty in the answers. Consequently, by minimizing confidence, we anticipate an increased likelihood of misclassification. These tests primarily highlight that well-calibrated models, such LLama3 for SST2 and StrategyQA, already exist. This capability is likely improve further in the future, as models have been shown to develop emergent abilities with increased scale, thereby making confidence elicitation attacks more powerful.
Calibration of verbal confidence elicitation
AUPRC
AUPRC Negative 1
Model
Dataset
Avg ECE
AUROC
Positive
SS12
0.1264
0.9696
0.9730
0.9678
LLaMa-3-8B
AG-News
0.1376
0.9293
Instruct
StrategyQA
0.0492
0.6607
0,6212
0.6863
SST2
0.1542
0.9537
0.9616
0.9343
Mistral-7B
AG-News
0.1216
0.8826
Instruct-v0.3
StrategyQA
0.1295
0.6358
0.6421
0.6185
Table 1: Expected Calibration Error (ECE) and the Area Under Receiver Operating Characteristic (AUROC) of models performing zero shot classification on SST2, AG-News and StrategyQA.
5.2 CONFIDENCE ELICITATION ATTACK RESULTS
When feedback from the model is provided through confidence elicitation, an attack algorithm can identify approximated input perturbations that minimize the elicited confidence, thereby in- creasing the likelihood of misclassification. Table 2 demonstrates that our Confidence Elicita- tion Attack (CEAttack) achieves higher attack success rate "ASR" for both LLama3 and Mistral across all datasets compared to "Self-Fool Word Sub" which performs no optimization, and SSPAt- Tack/TextHoaxer which perform optimization on hard labels.
7

---

Published as a conference paper at ICLR 2025
Calibration Plot for Classification ss s3t2/lamm3
Calibration Piot for
Calibration Plot for Clessification strategy QA/llama3
Classificatónn aq, nawee/lamaa
Caliaration
Colloration
caliorated
Prection of soreect www.sedlllllaaeeeeo senniiiiaaiioo
calltrated
Friet of pasther
Frod of praitnus
World
Sports
Dusiness
Sciftch
lssally
6o
calinnttdd
Weon prodicted probabiltty
Mezn predicted probabilty
Mean predicted probab
ROC Curve for Classification sat22laam.
ROC Curve for
ROC Curve for
Classification news/
Classification OtratagyOA/llamaa
ROC Curve
Random Chance
C.l-
This Powitte Pate
Thus Pooiitin Thalle
True Pobltbe Pablibee Malto
E2
World
Sports
Business
ROC Curve Random Chance
ScViech
Random Chance
EX Fabs Posttive Rase
False Positive Raze
Falso Poskive Rate
Figure 3: Reliability plots. Top) We show the SST2, AG-News and StrategyQA on LLama 3 8B Instruct calibration plots. Bottom) The ROC curves. The diagonal line is the optimal calibration.
Performance
CATSIT
ALA
ASRUNT
Model
Dataset
Vanilla
Wortl Sub
Heuxer
Atlack
SSP
Attack
Sell-Fool Word
Howver
Attack
Allack
L-SMMa3-SB
SS12
AG News
61 61.6260.38
M13690.11 14
53.35
61.17
82:01
49.3
81.01
45.27
43.06
72.09
0.33
2.23
19.41
8.45
26.71
9.75
30.74
19.73
lastruct
SuategyQQ
60.224)..1
59,52
45.29
42.25
32.67
1/66
24.07
29.67
45.67
Mistral 7B
Iastract-10.3
NG-News
SST2
StaleyyQ
50.9240.32
29
07.0700.
84.73
74.27
44.33
48.69
41.13
52.48
75.31
40.82
36.21
71.76
3.57
1.22
16.08
14.08
38,33
39.26
17.94
Table Results of performing Confidence Elicitation Attacks. Numbers in bold are the best results
Having high threshold allows only high-quality, label- preservvig perturbations to be kept. The successful perturbations in our study all have an angular semantic similarity of at least c = 0.84, which is common threshold used in previous works (Jin al.) (009)). In practice, any successful perturbation that changes the prediction while being above this threshold is deemed successful at- tack. However, find that our technique is also at times better preserving quality when compared to the alternatives, as shown by the high "SemSim" in Table 3 This is likely due to the algorithm not having to change more words than absolutely necessary to achieve a successful perturbation.
We present detailed analysis of one qualitative example in Table Additionally, a further discus- sion on sample quality, along with multiple qualitative examples, can found in Appendix and Appendix[] []]
Quality Ioss
SemSimm
Perplesity
Original
After-Attack
Perplesity
Madel
Dataser
Word Sub
Sell-Fool
Houxer
Text
Attack
SSP
Attack
CF:
Word Sub
Sell.Fool
Huxxer
Test
Attack
SSP
Attack
C.
Word Sub
Scil-Fool
Hoxxer
Text
Attack
SSP
Attack
C.E.
Instrnt
LLaMa-3-88
ACI-Now
StattgyOA
0.80
0,39
354.12
281.38
10544
09.81
72.01
220.73
300.05
$2.95
252.31
193.16
143.81
206.23
111.16
98.9
Mistral 70
Instmet-v0.3
StratgyQQ
Nows
0.89
0.9
0.80
0.88
0.9
74.04
61.68
97.3
73.2
91.85
93.57
140.08
195335 33
177.04
97.19
Table 3: Quality results of performing Confidence Elicitation Attacks. Only successful perturbations considered.
8

---

Published as a conference paper at ICLR 2025
AS8
(W) Sacc Queserder
Abcatao (M] 7Tts Adack Tise Bloues
Ablation on (R]
Artuic Timee [Tluud
6012
Figure 4: Ablation study
[W| and |S|
Because CEAtttac''s 's optimization path is more direct compared SSPAttack as illustrated in Figure 2. the algorithm doesn't I need to explore a large section of the manifold. Instead, by approximating probabilities, finds regions of high risk closest the original input, drastically cutting the number of required queries and optimization time. This efficiency is demonstrated in the columns "Suce Att Queries Avg" which only records the number of queries for the successful attacks, and "Total Attack Time" in Table 4
Filiciency
Queries Avg
Queries Avg...
[HIEEMMSSS]]
Word
SST2
017.2037
AC--Neww
34.18
42.88
42.36
0M433343
090-4506
Qu
22.33
8.5
10.95
00040:00
001:22:34
001
Intruct-1033
Miersl-7B
NGwNews
23.96
42.84
8.71
42.66
11.37
000:34441
003:43:43
000:55:14
001.38:57
50.15
00).43:48
017.16.52
Table 4: Efficiency results of performing Confidence Elicitation Attacks.
All figures in present ablations for the maximum number of possible word substitutions per sample [W| and the number of possible synonym embeddings per word |S). The results indicate that as both parameters increase, the attack success rate also increases, provided there are enough words to perturb. The trend varies across datasets due to differences in the average number of words per example. In our evaluation, AG-News has the longest examples, resulting in the scaling of [W] and |S] having the most significant impact.
6 ANALYSING ATTACK PATHING
Our attack framework, when compared to previous work adversarial attacks LLMs, allows us to track how perturbations affect the model's output state. Despite being approximate, it provides valuable insights. In the experiment illustrated Table and Figure [S] we allow the search algo- rithm to perturb sample beyond the boundary. Sable/]highlight an example where the prediction sentiment shifts from positive to negative after just two word substitutions. As the sample under- goes word substitutions, the probability of the wrong class increases, yet it retains its positive label over five different substitutions. As words are substituted, the empirical mean from the Dirichlet distribution moves from the positive region (top-left plot) to the most negative region (bottom-right plot) in Figure [ The final sample is approximately 97% correlated with being negative. This ap- proximate information would not be available in a hard label attack scenario, making it difficult to detect the confidence level of the new adversarial example. We observe this behavior across multiple examples.

---

Published as a conference paper at ICLR 2025
Example Analysis
8812 (Sentiment Classifiation) LLaMMa-3-80 3-8B---st-urr
Techuique
Sample
Perturbed Words
Prediction
Empirical Mean (Score)
Original
although with hanoe
film is refreshingly seriuss
young woman.
Positive
the 6hm is blithely serious lock yourge women
aexough looed with manoe and tow faacifal touches,
Positive
0.33
CFAmek
(Ous)
the 6lm blithely cetious heed
3tbough fooed with Bunser and Bew fancifal touches
Negativ
0.78
ahhoogh looed with Rannir ind
siesooii touclies,
Ohe blithely
young
Negative
"irhgggh faoid with and
the fim is bitthel cerious heed at young women. "thooghh loodd with funn and TEW alresomie woiches the film is biithely deeply heed at young women.
awcsome totches
Negativ
0.97
Negative
0.97
Table 5: Example of sample being progressively perturbed, the Dirichlet distributions of this process in Figure Perturbed words are in bold.
Dirichlet Distribution Samples (a-19.0, 65.0,
Dirichlet Distribution Samples (a=
Dirichlet Distribution
Samples (o-166.0, 17.0, 1.0))
Iepirse
S S
12 G.
Bock +
Negatve Dirichlet Distribttion Samples (a-196,0, 6.0, 1.00
Rejabve
Dirichlet Distribution Sampless (a-177.0, 2,0, 1.01]
Dirichiet Distribution Samples [a-[77.0, 2.0, 1.01
Nagaboe
Negelor
Figure 5: Ternary plots highlighting the attack path for the example in Table As the algorithm adds adversarial word substitutions the model's predictions and associated confidences such predictions change leading to a different Dirichlet distribution profile
7 CONCLUSION
In this work, we demonstrated that elicited confidence can serve as a feedback mechanism for identi- fying input perturbations. This feedback enables us to craft stronger adversarial samples. We believe this mechanism is agnostic to the type of input perturbation, the search algorithm, and the quanti- tative or qualitative bounds of Our word substitution attack can be tracked through substitution steps to observe how confidence diminishes and eventually alters the prediction. We achieve all of this within fully black-box threat model, and for the first time, point out that confidence elicitation may be at odds with robustness. Our results suggest the potential for confidence elicitation to en- hance jailbreaks. For example, may enable current multi-turn dialog jailbreaks (Chao et al. 2023 Mehrotra al. 2024) query the probability of the model's answers and use this information as feedback. Another promising direction is to investigate the susceptibility of token wise confidence elicitation input perturbations, and whether it is possible to control or influence the model's 's token selection process. Additionally, it is worth exploring how effective confidence elicitation attacks are generative tasks such free-form question answering (Joshi al.) 2017). and reasoning (Gan) ct al.) 2024, Ma et al. 2024 Bhuiya et al.) 2024| Wang et al. (0224) given that confidence elicita- reliable and calibrated measure of uncertainty in generative tasks (Liu et al. 2024a Chaudhry et al. 2024). We hope that our attack, evaluation, insights, and open-source code can assist researchers identifying sample perturbations, exploring their model's 's confidence elicitation behavior, and, more broadly, advancing adversarial robustness research.
tion also proven
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Neeladri Bhuiya, Viktor Schlegel, and Stefan Winkler. Seemingly plausible distractors in multi- hop reasoning: Are large language models attentive readers? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Meth- ods in Natural Language Processing, pp. 2514-2528, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10. 10.18653/1120224cmmmpp maapp-mmii.144. 147. URL fattss://aclamthcoory.oooc
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Pang Wei W Koh, Daphne Ippolito, Florian Tramer, and Ludwig Schmidt. Are aligned neural networks adversarially aligned?
In A. Oh. T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural In- formation Processing Systems, volume 36, pp. 61478-61500. Curran Associates, Inc., 2023. URL httpp:://prcceed http://rooeedd..............o.oo neurips htcpppsccccp/ee cwtpppperfllll//peeeeeeeeeeeeeee www.ffeelleee////a wtt.fel///pppp/222////// stant, Mario FLle/c1f0B55a3336111111111116661 -aa---7777766f--Pppprr Conference pdf
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Con- Steve Yuan, Chris Tar, et al. Universal sentence encoder. arXiv preprint arXiv:1803.1 1175,, 2018.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models twenty queries. 2023.
Arslan Chaudhry, Sridhar Thiagarajan, and Dilan Gorur. Finetuning language models to emit lin- guistic expressions of uncertainty, 2024. URL
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v1/N19-140 1423. URL https: has.lannholooyy.err//SSS.3
Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. Towards robustness against natural language word substitutions. In International Conference on Learning Representations, 2021a. URL htthttts///ppeeeiiieeee net worumm hi-.mmmmmdddkssnnnnnn...
Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. Towards robustness against natural http:////w....oo//////////-1111
language word substitutions, 2021b.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. HotFlip: White-box adversarial ex- amples for text classification. In Proceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 2: Short Papers), pp. 31-36, Melbourne, Aus- tralia, July 2018. Association for Computational Linguistics. doi: 10. 10.18663/11/P18-2000 URI. htttps://aclanthor wog..olly............ ogg.
Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. Text processing like humans do: Visually attacking and shielding NLP systems. 2019.
Brian Formento, See-Kiong Ng, and Chuan-Sheng Foo. Special symbol attacks on nlp systems. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, 2021. doi: 10.1 IJCNN52387 201138872221111 200199992722119952554444 9534254.
Brian Formento, Chuan Sheng Foo, Luu Anh Tuan, and See Kiong Ng. Using punctuation as an adversarial attack on deep learning-based NLP systems: An empirical study. In Andreas Vla- chos and Isabelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EACL 2023, pp. 1-34, Dubrovnik, Croatia, May 2023. Association for Computational Linguis- tics. doi: 10. 10.18653/v1/2223 htt.:/......n.nsseee....... findings-eacl.... URL https: http///aaaatttooooooo/////////// sindings-eaci.

---

Published as a conference paper at ICLR 2025
Brian Formento, Wenjie Feng, Chuan- Sheng Foo, Anh Tuan Luu, and See- Bee-Kiong& Ng. Sem- RoDe: Macro adversarial training to learn representations that are robust to word- attacks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 8005-8028, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. ht http://aacannthhoooooooo org/ 2024. maaci-lonne 4437
URL
Esther Gan, Yiran Zhao, Liying Cheng, Mao Yancan, Anirudh Goyal, Kenji Kawaguchi, Min-Yen Kan, and Michael Shich. Reasoning robustness of LLMs to adversarial typographical errors. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 10449 10459, Miami, Florida, USA, main.
November 2024. Association for Computational Linguistics. doi: 10.
584. URL https: / /acl anthologyy org/ 2024. emmlp- ma 584/
Wee Chung Gan and Hwee Tou Ng. Improving the robustness of question answering systems to question paraphrasing. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6065-6075, Flo- rence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1011//66///199 URL https:
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transac- tions of the Association for Computational Linguistics. 9:346-36 04 2021. ISSN 2307-387X. (RRtsstp:://dee org/1 1162/tacc a_ _00300
doi: 10.1162/acc
Narmin Ghaffari Laleh, Daniel Truhn, Gregory Patrick Veldhuizen, Tianyu Han, Marko van Treeck, Roman D. Buelow, Rupert Langer, Bastian Dislich, Peter Boor, Volkmar Schulz, and Jakob Niko- las Kather. Adversarial attacks and adversarial robustness in computational pathology. Nature Communicaaions, 13(1):5711, 2022.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2014. httt::
org/
6572
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks, 2017a.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 1321 1330. PMLR, 06-11 Aug 2017b. URL httpptss/pprrcccingnnii...cceeep guol 7a html]
Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. jaalstro of open-source Iims via exploiting generation, 2023. URL http:ppp///aaax...o.ooo/////////
06987
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example generation with syntactically controlled paraphrase networks. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies, Volume 1 (Long Papers), pp- 1875-1885, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v 1/133/11/111111 1170. URL /ttt::://allantholoyyyooo
Yuheng Ji, Yue Liu, Zhicheng Zhang, Zhao Zhang, Yuting Zhao, Gang Zhou, Xingwei Zhang. Xinwang Liu, and Xiaolong Zheng. Advlora: Adversarial low-rank adaptation of vision-language models. arxiv preprint axXivv2404.13425, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
12

---

Published as a conference paper at ICLR 2025
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How we know when language models know? on the calibration of language models for question answering, 2021.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment, 2019. URL httpps///axxx: brg/abs/ 1907. 11932
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Com- putational Linguisties. doi: 10. 111777/111111711177 URL https: httph//clllnooooog.o.oooooooo
11771147
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bow- man, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Ka- plan. Language models (mostly) know what they know, 2022.
A Kurakin, Goodfellow, and S Bengio. Adversarial examples in the physical world. arxiv 2016. arXiv preprint arXiv: 1607.02533, 2016.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. TextBugger: Generating adversarial text against real-world applications. In Proceedings 2019 Network and Distributed System Security Symposium. Internet Society, 2019. doi: 10.14722hds..221922313...
URL https:
Linyang Li, Ruotian Ma. Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. BERT-ATTACK: Ad- versarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Em-
pirical Methods in Natural Language Processing (EMNLP), pp. 6193-6202, Online, November 2020. Association for Computational Linguisties. doi: 10. 10111886111222 TA//S3//1/222.eere 12200..emmllmmaiiie maapp--mii.0000.. URL https://acl
wgw.oggg..........// (atpr/2220.eemnpmmmme. i .n. 500
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words, 2022.
Han Liu, Zhi Xu, Xiaotong Zhang, Xiaoming Xu, Feng Zhang, Fenglong Ma. Hongyang Chen, Hong Yu, and Xianchao Zhang. Sspattack: simple and sweet paradigm for blxck-box hard- label textual adversarial attack. Proceedings of the AAAI Conference on Artificial Intelligence, 37 aaai.
(11):13228-13233 Jun. 2023. doi: 10. 10.1609/aaii.v391 137:11.26553. URL.
/art wtt.et/eeeeeeeeeeeeeee////////
Linyu Liu, Yu Pan, Xiaocheng Li, and Guanting Chen. Uncertainty estimation and quantification for Ilms: A simple supervised approach, 2024a. URL fttpp://aax..v.rrr/bbbs///
15993
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models, 2024b.
Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi. Flipattack: Jailbreak 1lms via flipping. arXiv preprint arXiv:2410.02832. 2024c.
Yingwei Ma, Yue Liu, Yue Yu, Yuanliang Zhang, Yu Jiang, Changjian Wang, and Shanshan Li. At which training stage does code data help LLMs reasoning? In The Twelfth International Conference on Learning Representations, 2024. URLbttps: wwwp///pppee...o///o//oo/p net net/forumt forum?
id Kd-KIPJKST4gW
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks, 2019.
13

---

Published as a conference paper at ICLR 2025
Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. Generating natural language attacks in a hard label black box setting. In AAAI Conference on Artificial Intelligence, 2020a. doi: 1044850/ARXIV.2012.4455
Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. Generating natural language attacks in hard label black box setting. In AAAI Conference Artificial Intelligence, 2020b. doi: 10.48550/ARXIV2202222
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum S Anderson, Yaron Singer. and Amin Karbasi. Tree of attacks: Jailbreaking black-box LLMs automatically. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL net Ptee:/forumm100SSM33nn00000
attps:://ppenee
John X. Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp, 2020. URL
https://arxtt
awlgg//a////// 2000 05909
Nikola Mrkšić Diarmuid Séaghdha, Blaise Thomson, Milica Gašié, Lina Rojas-Bard Barahona, Pei- Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. Counter-fitting word vectors to linguistic constraints, 2016.
Gwenyth Portillo Wightman, Alexandra Delucia, and Mark Dredze. Strength in numbers: Esti- mating confidence of large language models by prompt agreement. In Anaelia Ovalle, Kai-Wei Chang, Ninareh Mehrabi, Yada Pruksachatkun, Aram Galystan, Jwala Dhamala, Apurv Verma, Trista Cao, Anoop Kumar, and Rahul Gupta (eds.). Proceedings of the 3rd Workshop on Trust- worthy Natural Language Processing (TrusiNLP 2023), pp. 326-362, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: https::/aalaattholooooo org. wtt.:...rrr//222..uut..... 1.28
URL
Vyas Raina, Samson Tan, Volkan Cevher, Aditya Rawal, Sheng Zha, and George Karypis. Ex- treme miscalibration and the illusion of adversarial robustness. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 2500 2500-2525, 2525, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/24/////004.....loooo 137. URL https::/aaee antho wow.oooooo.ooooooooooooooooooooo ogy org/
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversar- ial examples through probability weighted word saliency. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1085-1097, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18853/11/11-1111 1/19--113. URL https: / V/acl Taa.lannholoog..org//t....
Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, write article about that: The crescendo multi-turn Ilm jailbreak attack, 2024. URL http::////aaiii.oo/aabbbbsss/
01833
Yusuf Sale. Viktor Bengs, Michele Caprio, and Eyke Hüllermeier. Second-order uncertainty quantification: distance-based approach. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver. Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 43060 43076. PMLR, 21-27 Jul 2024. URL https: //proceedings.. www.nng.ms....... ml
press/ wwwpppeesss... v255 /235 00/35//aaeee sale
Thomas Savage, John Wang, Robert Gallo, Abdessalem Boukil, Vishwesh Patel, Seyed Amir Ahmad Safavi-Naini, Ali Soroush, and Jonathan Chen. Large language model uncertainty measurement and calibration for medical diagnosis and treatment. medRxiv, 2024. doi: 00.1100/2224.0.0062.333333
Taylor Shin. Yasaman Razeghi. Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on
14

---

Published as a conference paper at ICLR 2025
Empirical Methods in Natural Language Processing (EMNLP), pp. 4222 4235, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 10065////2200.mmmmmmmmmiiii URL httpps://aaaaattol..ooo/ http://hwwoooooog//oooeoo// ewnlp-mainnnn..ppee 346
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Good- fellow, and Rob Fergus. Intriguing properties of neural networks. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. URL http:
//axxiv.or/ abs 1312.1111262 6199
Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. It's morphin' time! combating lin-
guistic discrimination with inflectional perturbations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. doi: 10.
URL
http:/ /ttp://dxx /xxx doi. wot.ooiii..1........ 11653//1/262000 acl main. 266
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just ask for calibration: Strategies for eliciting calibrated con- fidence scores from language models fine-tuned with human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 5433 5442, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10. 2ttp:///111232...mmmaaiaal wmw.mmmmmmppmmmmmmmmmm
URL https:
//aclantholooyyoorg//
330
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Scong Joon Oh. Calibrating large language models using their generations only, 2024.
Tianlu Wang, Xuezhi Wang, Yao Qin, Ben Packer, Kang Li, Jilin Chen, Alex Beutel, and Ed Chi. CAT-gen: Improving robustmess in NLP models via controlled adversarial text generation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Con- ference Empirical Methods in Natural Language Processing (EMNLP), pp. 5141-5146, On- line, November 2020a. Association for Computational Linguistics. doi: 10.18653/v/20000 emnlp- ftilpmmmin.411...
org/ 2020 wwweemmnlllmmm. main. 417
Xiaosen Wang, Yichen Yang, Yihe Deng, and Kun He. Adversarial training with fast gradient projection method against synonym substitution based text attacks, 2020b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL htttp://openzeviees 2wt.://////..22211111111M
Zefeng Wang. Zhen Han, Shuo Chen, Fan Xue, Zifeng Ding, Xun Xiao, Volker Tresp, Philip Torr, and Jindong Gu. Stop reasoning! when multimodal LLM with chain-of-thought rea- soning meets adversarial image. In First Conference on Language Modeling, 2024. URL https://openreviee net forum hwt.mmidd=ossaeaeee
Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in Ilms with continuous attacks, 2024. URL https://arxiy:
orq/abs/ 2tt.//455..5555 15589
Miao Xiong, Zhiyuan Hu, Xinyang Lu, YIFEI LI, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs. In The Twelfth International Conference on Learning Representations, 2024. URL https:
7/ //openreview.o
Xilie Xu, Keyi Kong, Ning Liu, Lizhen Cui, Di Wang, Jingfeng Zhang, and Mohan Kankanhalli. An LLM can fool itself: A prompt- based adversarial attack. In The Twelfth International Confer- ence on Learning Representations, 2024. URL
net/ ffr.mmirddd
VVGGbR9TNV
15

---

Published as a conference paper at ICLR 2025
Muchao Ye, Chenglin Miao, Ting Wang, and Fenglong Ma. Texthoaxer: Budgeted hard-label ad- versarial attacks on text. Proceedings of the AAAI Conference on Artificial Intelligence, 36(4): 3877-3884, Jun. 2022. doi: 10.1609/aaii.33444.20000 20303. URL
aaai. org/
Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. Robust Ilm safeguarding via refusal feature adversarial training, 2024. BRLhttps::/ /tLLLttt:p///aaav.or////a////// http://..wrrab///2//0//
20089 20089
Zhen Yu, Xiaosen Wang, Wanxiang Che, and Kun He. TextHacker: Learning based hybrid local search algorithm for text hard-label adversarial attack. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 622-637, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Lin- org/
guisties. doi: 10. 10.18863/11/2222 httS5//ww22222..nnnn-eemmmm findings-emmlp.44, URL
2022. findings emnlp. .4.4
Yuan Zang, Fanchao Qi, Chenghao Yang. Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. Word-level textual adversarial attacking as combinatorial optimization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pp. 6066-6080, Online, July 2020. As- 1/203//11120aaacmmmmmm
sociation for Computational Linguistics. doi:
URL https:
www.looy.org//0000 //aclanthology wwtp///gg222.. acl-mainn 540
Xuandong Zhao, Xianjun Yang, Tianyu Pang. Chao Du, Lei Li, Yu-Xiang Wang, and William Yang Wang. Weak-to Weak-to-strong jailbreaking on large language models, 2024. URL (tRpps:ss:///aiiitt
htg/abs//e 2401. 17256
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding, 2020.
Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Furong Huang, and Tong Sun. Au- toDAN: Automatic and interpretable adversarial attacks on large language models, 2024. URL
net 2ttp://wwwweeeeeeeeoooooomeedmdd wwwfoorummii-uuuue Zuzu J0L1
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models, 2023.
A ETHICS STATEMENT
This research was conducted in accordance with the ACM Code of Ethics. Although our technique may used to bypass refusal mechanisms, we propose in the discussion section various ways to defend against this.
B FUTURE WORK
B. DEFENSE DISCUSSION
The main challenge with defending against this issue arises from an active push within the com- munity to make confidence elicitation an integral part of LLMs' behavior. We believe this is an interesting emergent behavior and do not think the community should halt these efforts. Therefore, simply blocking models from performing confidence elicitation or impairing their ability by adding noise or deliberately making them uncalibrated may not be a viable option (Raina al. 2024). Ulti- mately, we concluded that confidence elicitation may be at odds with robustness. However, we have identified two potential directions that the community may find worth exploring:
B. ADVERSARIAL TRAINING / ADVERSARIAL DATA AUGMENTATION
Can the generated adversarial inputs reintroduced into the training process? This opens up op- portunities for confidence elicitation adversarial training, aiming to both enhance robustness against
16

---

Published as a conference paper at ICLR 2025
imput perturbation and potentially improve calibration. This would adhere to the traditional adver-
sarial training min-max formulation (Yu et al. 2024 Xhonneux et al. 2024 et al. 2024 Formento et al. 2024).
As we incorporate perturbations into the input that alter predictions during instruction fine-tuning, loop these samples back into the training process, following black-box adversarial setup. Adversarial training could potentially be done in a white-box setting by perturbing the input embed- dings then checking how the confidence elicitation behavior changes.
Alternatively, simpler solution involves generating the data first and then using for further fine- tuning (adversarial augmentation).
B.1.2 DEFENSE BY INTENT
In this case, we aim to protect the system by analyzing the use cases of confidence elicitation with rule-based defense approach.
Is the user performing the same query multiple times with small semantic similarities be- tween queries, likely adding minor input perturbations?
Is the user explicitly asking for confidence elicitation? This could be implemented classifier; if yes, it indicates desire for confidence elicitation.
Are confidence elicitation values some tokens decreasing over time? This could suggest some form of optimization progress.
B.2 APPLICATIONS CONFIDENCE ELICITATION ATTACKS
As briefly introduced in the paper, we target classification settings where either confidence is pro- vided as main component of the system as feedback or where the attacker can perform confidence elicitation in free-form generation setting. Exploring this approach using actual medical datasets would be an interesting direction.
Although chatbot jailbreaks fundamentally differ from adversarial attacks, there precedents where adversarial techniques have been employed to craft jailbreaks. For example, variants of the HotFlip (Ebrahimi al.; 2018) adversarial attack and AutoPrompt (Shin al. 2020) were utilized to develop the first automatic jailbreak (Zou et al. 2023). This was achieved by projecting gradients over vocabulary to select optimal token substitutions for suffix by simply adding left-side noise to the prompt (Liu et al., 2024c). Similarly, confidence elicitation could be an interesting concept enhance chatbot jailbreaks. For instance, it might enable prompt-level multi-tum tulli-turn dialog attacks, such as PAIR (Chao et al. (202)) and Tree of Attacks (Mehrotra et al.) (2024), to query the probability of the model's answers during multi-turn dialog and use this information as feedback. similar approach could be explored for adversarial misalignment (Carlini et al. (023).
C PROMPTS
Table shows the prompt we first perform prediction r='text' and then elicit confidence. This prompt is combination of methods from (Lin et al. (002)). where verbal confidence is utilized, and (Tian al. 2023), where a two-shot approach is used for confidence elicitation.
D FURTHER IMPLEMENTATION DETAILS
We use 12 Nvidia A40 GPUs for our testing, every test can be conducted on only A40GPU, For tests we perturb 500 samples on A40 GPU with 46GB of memory.
17

---

Published as a conference paper at ICLR 2025
Verbal Elicitation Verb, 28 guesses prompt example
1n ssell prompt header
Provide your best guess for the following text (positive, negative).
Give ONLY the guesses, other words explanation.
For example:
Prediction Prompt
Guesses: (most likely guesses, either positive or negative; not complete sentence, jusst the guesses!
Sepurated by comma, example (Negative, Positive, Positive, Negative sk()
The text (t:S(tex))
Guesses:
[selC.end prompt footer)
("e"[seelstaa promptttheaee header
re model needs to give the confidence of answers being correct.
The previous prompt was:
Provide your
guesses
following (positive, negative)).
Give ONIYY the guesses, no other words or explanation.
For example:
Guesses: fmost likely guess, either positive or negative; not a complete sentence, just the guesses!)
Confidence Elicitation
The text (s:(text) the guesses were: (guesses output),
Prompt
given guesses provide the contidences your guesses correct.
Give ONLY the verbal confidences, no other words or explanation.
example:
Confidences: (the confidences, from either (Highest. High, Medium, Low, Lowest) that your guesses are correct, without extra commentary whatsocver, for example [Highest, High, Medium, Low, Lowest ...].
just confidence! Separated by a coma
Confidences:
sell and propppt__fooeerrr voen
Table 6: An example of the Verb. 2S top-1 prompting technique is as follows: The first prompt generates answer, through the model fe- This answer is then passed to the Confidence Elic- itation Prompt as the variable "guess result'. Next, the second prompt is passed through fo to generate the verbal confidence, Pc- The variable 'text' is the sample under analysis, while the 'start_ ceart_promp__eer _hedeer' and 'end_ www.ppooooooottcc are model's formatting tokens
E EXPERIMENTAL SETUP DETAILS
E. DATASETS, TASKS AND MODELS
We conducted our confidence elicitation attacks and Mistral-7B Hisstrl-7B-Instruct-O00 Instruct-v0.2 (Jiang et al. 2023) while performing classification on two common datasets to evaluate adversarial robustness: SST-2, AG-News and one modern dataset: Strategy Geea et al. 2021).
(Tovrron et al. 2023)
E.2 EVALUATION METRICS
We utilize the evaluation framework previously proposed (Morris al., 2020), where an evalua- tion set is perturbed, and we record the following data from the Total Attacked Samples (T'S)) set: Number of Successful Attacks (N sace-aakk). Number of Failed Attacks (N fast-atkk). atk), and Number of Skipped Attacks (Nskp-atk). We utilize these values to record the following metrics. Clean accu- racy/Base accuracy/Original accuracy, which offers a measure of the model's performance during normal inference. After attack accuracy/Accuracy under attack (Aaft-atk = Nrait uck ) or (AUA), critical, representing how effectively the attacker deceives the model across the dataset. Similarly, Nause-atk the After success rate (Asucc-rte TAS TAS-Nat--est or (ASR) excludes previously misclassified samples. The paper also considers the Semantic similarity/SemSim, an automatic similarity index, modeled by de (Cer al. 2018). We compare the original perplexity with the perturbed sample's 's perplexity, calculated using a GPT-2 model. A higher perplexity indicates that the example less natural and fluent to the language model. Queries denotes the number of model calls for inference. We subdivide this metric into two categories: All Att Queries Avg and Suce Att Queries Avg. The latter records the queries for successful attacks only, while the former includes all queries. Additionally, we track the duration of the attack process perturb all the samples under Total Attack Time.
TAs
E.3 EVALUATION BASELINES
We compare guided word substitution attacks, CEAttack to "Self-Fool Word Sub" from (Xu
al. 2024). SSPAttack from (Liu et al. (023) and TextHoaxer from et. (022). The Word Sub" method operates by instructing the L.M model substitute words with synonyms while
18

---

Published as a conference paper at ICLR 2025
maintaining semantic integrity. We execute query generate Zadv, extract Fade from the generated string, and perform another call to the model to achieve the misclassification, if no misclassification is found, repeat the process twenty times. On the other hand, the SSPAttack algorithm initially heavily perturbs the original sample with multiple synonym word substitutions to induce misclas- sification. Subsequently, they optimize the sample for quality by first reverting many perturbed words to their original form as possible, and then by substituting words with synonyms that enhance semantic similarity. This optimization process is conducted using the hard-label as feedback. manner similar to SSPAttack, TextHoaxer initially introduces significant perturbations to the input formulate an adversarial candidate. then utilizes resources like Counter-Fitted word embed- dings (Mrkšić et al., 2016) to extract the word embeddings of both the original and the adversarial version Tado- From these embeddings, a perturbation matrix constructed. TextHoaxer constructs a loss function that is optimized over this perturbation matrix. The optimization aims to enhance semantic similarity while adhering to two constraints: a pairwise perturbation constraint to main- tain semantic closeness of word substitutions, and a sparsity constraint to control the extent of word replacements, ensuring minimal yet effective perturbations.
E.4 EXPECTED CALIBRATION ERROR (ECE)
The ECE is calculated using the formula:
M
Bml
ECE
Σ m7
|aco( (Bm) - conf( Bmm)
In this formula, n represents the total number of samples, and M is the total number of bins used to partition the predieted confidence scores. The term Bn denotes the set of indices of samples whose predicted confidence falls into the m-th bin, and [Bm] is the number of samples in this bin. The accuracy within each bin, acc (m. is calculated as the proportion of correctly predicted samples, given by the equation acc( label and y4 is the true class label for sample i. The confidence of the predictions in the m-th bin, (onf(Bn), is the average of the predicted confidence scores for the samples the bin, calculated as conf(B,,) [B..l ExeB pi, where P; is the predicted probability for the predicted class of sample The ECE thus captures the weighted average of the absolute differences between accuracy and confidence across all bins, providing a summary measure of model calibration.
[B..l ΣieBm. 1 y4), where 9J is the predicted class
We use 10 bins to generate our plots in Figure
F FURTHER CALIBRATION STUDIES
F.1 CONFIDENCE ELICITATION GENERALIZATION
We investigate whether confidence elicitation serves a reliable measurement of uncertainty across various models (Table 7 and datasets (Table 8. Our findings suggest that confidence elicitation is indeed dependable tool for approximating confidence across different models.
F.2 SELF-CONSISTENCY CALIBRATION
is possible use empirical self-consistency (Wang al. 2023), with the parameters set to k 1, M = 20, and - = 1, instead of employing confidence clicitation for our attacks. This approach generates multiple predictions from the model, which we can then leverage to obtain empirical un- certainty estimates. We find that the results are similar to those achieved using confidence elicitation, as shown Table However, approximating uncertainty using this technique renders the attacks impractical, since each input perturbation would require M calls to the model to estimate confi- dence, whereas confidence elicitation requires only a single call. We find results similar to previous work, where the outcomes mixed. Specifically, in line with the findings of (Xiong et al. 2024). we observe that the confidence elicitation technique outperforms self consisteeyy on Strategy QA for uncertainty estimation.
19

---

Published as a conference paper at ICLR 2025
Calibration of verbal confidence elicitation models
AUPRC
AUPRC
Model
Dataset
Avg ECE
AUROC1
Positive f
Negative
SST2
0.0591
0.9486
0.9547
0.9357
Gemma2
AG-News
0.1666
0.8342
9B-Instruct
SiralegyQA
0.2295
0.6631
0.5899
0.7563
SST2
0.0645
0.9958
0,9944
0,9970
Mistral-Nemo
AG-News
0.0673
0.9194
12B-Instrutt-2407
StrategyQA
0.2748
0.6214
0.6425
0,5863
SST2
0,0382
0.9534
0,9399
0,9480
Qwen2.5
AG-News
0.0753
0.8722
7B-Instruct
StrategyQA
0.2332
0.6247
0.6649
0.5624
SST2
0,0581
0.9535
0.9645
0,9270
LLaMa-3.2-11B
AG-News
0.1090
0.8954
Vision-Instruct
StrategyQA
0.2720
0.6366
0.6532
0.5928
Table 7: Calibration of other models on core datasets SST2, AG-News and StrategyQA
Calibration verbal confidence elicitation on more datasets
AUPRC
AUPRC
Model
Dataset
Avg ECE
AUROCT 1
Positive
Negative
LLaMa-3-8B
RTE
0,2598
0.8230
0.7972
0.8418
Instruct
QNLI
0.1352
0,8413
0.8561
0.8182
Mistral-7B
RTE
0.3047
0.6507
0.6032
0.6927
Instruct-v0.3
QNLI
0.2764
0.6951
0.6444
0.7345
Table Calibration of other datasets on core models Mistral and L.LaMa3
F.3 MORE CALIBRATION PLOTS
We show the reliability plots for mistral in Figure 6
G EVALUATION ON A CLOSE-SOURGE API MODEL
We conducted tests on GPT-4o using the OpenAl API. We found this model to be more robust against word substitutions (Table TT) and better eliciting confidence (Table 10p.
H ABLATION
Since work introduces the concept of confidence elicitation attacks, follow previous adver- sarial attack work and set the temperature r to approximately 0. This approach maintains the general purposefulness of the model while allowing deterministic behavior across multiple model calls with the same input M in Figure [ Setting ~ 0 also has the added benefit of reducing computa- tion costs for our analysis since we won't have to perform multiple calls to the model after each perturbation. Nonetheless, we provide an ablation study with r = 0.7 in Appendix H.3
H.1 ABLATION ON NUMBER OF EMBEDDING S
The following table 12 holds the values in figure for the ablation on [S] plots.
H.2 ABLATION ON MAXIMUM NUMBER OF WORD SUBSTITUTIONS [W]
The following table 13 holds the values in figure for the ablation on W| plots.
H.3
TEMPERATURE ABLATION
We conduct the same experiments in Section 5.2 with a temperature of 0.7, the findings are shown
in Tables 14 15 16
20

---

Published as a conference paper at ICLR 2025
Calibration of empirical self-consistnncy
Uncertainty
AUPRC Positive
AUPRC
Model
Dataset
Estimation
Avg ECE 4
AUROCT
Negative 1
Technique
Sell-Consistency
0.0ST5
01.631
0,9730
0,9433
SST2
Confidence Elicitation
0.1264
0.9696
0,9730
0,9778
LLaMa-3-88
Seli-Consifeeny
0.0774
0.9147
AG-News
Instruct
Confidence Flicitation
0.1376
0.9293
ColssCoosistency
0,2T13
0.6975
0.6639
0,7124
StralegyQA
Confidence Elicitation
0,0492
0.6607
0.6212
0.6863
Self- Cont--ssiitnnncy
0.0675
0.9466
0,9418
0.9255
SST2
Confidence Elicitation
0.1542
0.9537
0,9616
0,9343
Mistral-7B
Sell-Consislenuy
0.0837
0.9240
AG-News
Instruct-v0.3
Confidence Elicitation
0.1216
0.8826
Self-Consistency
0.3671
0.6182
0.6416
0.5861
StrategyQA
Confidence Elicitation
0.1295
0.6358
0.6421
0.6185
Table Calibration of empirical self-consistency
Calbratto Pist for
Cal anlionation Plat Classifical ion ag new.ymmttrraas
Calibration Plat
Classitration s012/mittrae00
Classificatio sir strateey stralegyAAmiistaii DAmistravvo
Call brstion Rot callirtiodd
Imesgn pfpogthee
Word
Bashass
Galortio Por
Sallcce
Munn
Tean
ROC
Curve
Glssiiication 5012mmbtraiv33
Classifiiation newzrmitrrav03
Classification OAMistraavv3
BOC
Random Chance
Tha potthin WATH
batbe
Sports Businiss
1550 Aodil
FM5 Poilsc Kate
Figure 6: Reliability plots. On the top, we show the SST2, AG-News and StrategyQA on Mistralv0.3 7B Instruct calibration plots. On the bottom, the ROC curves. The optimal calibration corresponds to diagonal line.
H.4 DELETE WORD RANKING SCHEMA ABLATION
possible to enhance the efficacy of the attack by initially ranking the input words based on their importance using word deletion ranking schema (Table 17}. This involves removing each word from the input example, one at time, and observing the change in confidence clicited in the output. Words that cause the largest change in confidence are ranked higher, while those causing minimal change are ranked lower. Once the words are ranked, we proceed to perform CEAttacks as previously done. A word deletion ranking schema is appropriate for technique because it adheres to the black box constraints of the attack. Alternative ranking methods, such as using attention scores word saliency, would require some knowledge of the model's 's inner workings.
H.5 SIMPLE CONFIDENCE ELICITATION ATTACKS
We can replace the Dirichlet aggregator by setting 1, and instead of using verbal confidence (VC), we employ numerical verbal confidence (NVC). In this approach, we ask the model to provide its confidence numerically value between and for prediction. We find that the performance
21

---

Published as a conference paper at ICLR 2025
Calibration of verbal confidence elicitation API model
AUPRC
AUPRC
Model
Dataset
Avg ECE +
AUROC1 1
Positive t
Negative f
SST2
0.0286
0.9713
0.0297
0.0274
GPT-4o
AG-News
0.0641
0.9306
2024-08-06
StralegyQA
0.2300
0.7410
0,2373
0.2227
Table 10: Calibration of GPT-4o
Attack performance API model
Suce Att
Total Attack Time
Model
Dataset
Tochnique
[%]
[%]]
ASR [%]
SemSim
Queries
[HHH:MM:SS]
Avg L
SSPAttack
94.0
5.32
60.8
001:3:122
SST2
CEAttack
96.0
82.0
14.58
0.88
41.92
002:54:00
CEAttack+
95.0
28.42
108.33
005:34:25
GPT-44o
SSPAttack
88.0
87.0
1.14
0.87
144.0
002:37:35
2024 08 06
CFAttack++
88.0
75.0
14.77
0.91
412.23
025-56:00
SSPAtaakk
StrategyQA
CEAttack
64.0
45.0
29.69
0.89
21.15
000:31:01
CEAllack+
39.52
Confidence elicitation attacks can also target closed-source API models. Naturally, their larger scale makes them more robust to semantic perturbations. Therefore, we [S] to 20 for SS- PAttack and CEAttack. For CEAttack++, we set [S] to 50 across all datasets, employ delete-word ranking scheme, and specify [W] as 5 for StrategyQA, 10 for SST2, and 20 for AG-News. These configurations represent the best set of hyperparameters identified through our ablation studies.
Table 11:
of the attack is lower (Table 18 likely due to having weaker feedback signal with less fine-grained thresholds.
I QUALITATIVE EXAMPLES
Word substitutions using Counter-fitted embeddings are already constrained in terms semantics due the pre-built nature of the dictionary. This ensures that each word can only be replaced with a previously vetted synonym. Compared to "Self-Fool Word Sub" and SSPAttack our method more effectively preserves the original meaning of the text, evidenced by the high "SemSim" in Table 3 Table technique, CEAttack demonstrating conversions of examples from Positive Negative and from Negative to Positive sentiment. Additional examples for AG-News and Strategy can be found in Table 20 and Table in the 'More qualitative examples' Section
|9 provides multiple examples from SST2 that have been perturbed by our proposed
J MORE QUALITATIVE EXAMPLES
J.1 AG NEWS
J.2 STRATEGY QA
22

---

Published as a conference paper at ICLR 2025
Model
Dotseet
I8|
[S]]
ADA [8]1
ASR [4]]
SemSim
Original Perplesity
Perplesity
Queries
Queries
[HHRMMM:S]]
Avg.
Avg.
83.80
15.79
8.24
10.3
D1.25
105.0
4.53
4:71
102:4:32
87.85
99.7
20.69
0.89
81.75
131.95
13.45
16.11
[002:11 18
SST2
K5.86
68.69
2010
0.89
75.06
105.2
21.81
25.52
[00.22 45
85.86
63.64
25.88
0.89
72.24
106.77
34.56
49.36
105.19.55
86.87
$8.59
32: 56
0.88
74.25
112.37
61.95
73.85
(00: 16
60.0
62.0
10.14
0.92
133.95
14156
5.71
5.71
[00:4
1LMaa-3-8R
Insercc
AG News
20
10
69.0
69.0
68.0
$2.0
57.0
57.0
24.64
17.39
16.18
0.93
0.92
0.91
87 26
93.73
73.76
122.97
92.56
$8.90
23.83
43.61
75.32
75.35
23.45
38,5
000.02
005:47233
006:4] 30
68.0
47.0
30.88
0.91
87 03
95 10
132.49
150.19
016:30-55
6330
35.0
12.7
03
123.55
201422
1.35
2:75
10021555
64.0
39.0
39.06
0.9
109.45
185.19
5.38
6.68
000:1200
Q4
10
65.0
$1.0
52.31
0.89
109. 5
216.1
9.14
11:35
000:30:29
20
68.0
26.0
60.1
0.89
99.77
184.98
14.98
18.66
1000322.50 50
30
63.0
24.0
61.9
0.89
113.4
202.82
28.24
35.74
000:58.31
Table 12: How the greedy search process is affected if we increase the number of potential synonyms per word [S].
W
CEAttack
[W]
[A]
[S]]
[S]]
SemSim
Original Perplesity
Perplesity
Queries
Queries
[HHHMMM::N]
15.16
11500
0.89
75.05
105.2
21.81
000:23:17
SST2
0.87
76.
034:14-46
0.89
72:11
20
$5.85
66.67
22.35
0.8
02.32
101.31
27.85
36,73
001:12:58
67.0
61.0
3.50
033
1173
112.5
13.57
200
001:5933
LLMa Inttuct
AG News
10
69:0
69.0
50.0
57.0
27.54
17.39
0.91
0.91
03.75
75.45
115.21
122.97
78.35
43.61
72.73
38.5
(05:47:23
010:15:10
15
09.0
49.0
0.91
03.93
114.29
106.51
99.5
014:17:19
20
71.0
47.0
35.8
0.91
07.15
116.75
135,87
113.58
018:35 78
63M
43.0
2331
0.87
107.1
225.22
4.05
K.35
[00309
65.0
31.0
52.31
0.89
109.8
216.1
9.14
11.35
000:20 29
SaatagyQ4
10
64.0
33.0
50.0
0.89
105
209
9.43
11.78
000:20-49
15
64.0
33.0
90.0
0.89
105
209 16
9.43
11.78
000:20-49
20
64.0
32.0
50.0
0.89
105.45
209.16
9.42
11.78
000:20-49
Table 13: How the greedy search process is affected if we increase the maximum number of potential word substitutions in the sentence [W].
Atlack
Word Sub
SSPAmack
CEAttack
Model
Dalasse
CA
ALA [N]I
ASR [S]]
CA
AUA
ASR [6]
CA
AUA [S]]
ASR [%]
SST2
5.42
1-aMa-3-8B
AG-News
61.96
19.14
59.8
35.96
Inserict
Qtrategy
61.12
59.32
2.95
63.53
37.27
41.32
62. 12
23.05
62.9
Mistral-7R
SST2
89.77
86.01
4.19
30.77
9.98
89.38
66.46
25,64
Instruct-v0.3
StrategyQA
58.68
58.44
0.42
54.9
34.11
37.5
55.88
23.36
Table 14: Results of performing Confidence Elicitation Attacks when the model has temperature of 0.7. Numbers bold represent the best results
Quality Tests
SelF-Fool Fool
Sub
SSPArtack
CEAttack
Model
Datuset
SemSim 1
Original Perplexity
Peplesity
SemSim 1
Original Perpleiity
ArlasAmtck Perplesity
SemSim
Onigina Perplesity
Affer-Artack
Perplxsity
LLaMa-3-8B R-B
Instract
SST!
AG News
StnstgyyA
0387
0.88
112.97
50.94
131.68
90.45
0.58
0.87
74.61
76.8
210.26
208.4
148.
0.93
0.88
0.89
99.98
66.82
66.39
178.05
84.68
105.85
SS12
Mistral-20
Instract-v0.3
StntegyQA
Table 15: Quality results of performing Confidence Elicitation Attacks when the model has a tem- perature of 0.7. Numbers in bold represent the best results for semantic similarity, only successful perturbations are considered.
23

---

Published as a conference paper at ICLR 2025
Fiiiieency Test
Word Sub
Fol-Fool
SSPAtbckk
CEAibck
Model
Dutsset
Queniss
All
Suvc All
Queries
Total Atrak Time
[HHH MMJSS]
Queries
All All
Suce
Total Abrck Time
[HHH.MM.SS]
Queries
All All
Soce
Queries
Total Attack Time
[HHHIMM:SS]
1-M----88
SST2
AG-News
20.22
001:4513
165.3
048:17377
073:29:13
43.17
27:28
42.30
020:3545
028:16:30
Instruct
QA
21.78
3.0
000:11:37
10.99
21.47
002:15:44
8.45
10.54
001:27:07
Mistral7RR
Instruct-O0. 0.3
SS72
OA
AG News
20.85
21.01
20.43
3.0
3.0
000:34:38
001:20:40
001:18:3
20.50
10.4
157.26
72.04
001:35:31
028:18:11
062:05:10
43.86
23.71
10,90
25.06
44,68
001:40:31
018:02:17
018302:37
Table 16: Efficiency results of performing Confidence Elicitation Attacks when the model has temperature of 0.7.
Attack Performance with delete word ranking schema CFAttacks
Suce
[%]T
[%]
ASR [%]
SemSim
Querles
[HHHLMMI:S)]]
Avg
Delece Kanking
Delete
Ranking
Ranking
Ranking
Ranking
Ranking
Ranking
004-11:58
029.2:19
OA
59.92
001-25:34
002:19:30
Instrct
Miersl-7B
Qu
NG--ews
85.0
58.87
67.76
33.24
39.26
38.33
0.81
0.98
0.9
65.95
33T8
0111344
017.16522
001:43:48
002:43.34
027.49.55
0340507
Table 17: Confidence elicitation can also serve as proxy for ranking the importance of words in the input.
Confidence Elcitation Attack with simple confidence cliciaaion technique
Model
CA1%11
[%11
ASR[%]1
SemSim
Suce Att
Queries
Total Attack Time
[HIHMMM:S)]
Avg
LL-Ma-3-BB
SST2
DiricheetVC
NVC
912
72.69
76.0
16.67
0.89
29.5
017:30:57
005:11:15
Instruci
Striegy
Di
NVC
65.0
60.12
48.2
32.67
45.67
25.85
0.89
0.89
12:22
10.95
001:25:34
000:52:20
Table 18: Confidence Elicitation Attack with simple confidence elicitation technique
Otcaliaative Ecample
(Seatiment Cha efication
Tichelque
Simple
Perplecity
Wards
Truth
Empiriall Meon (Score)
Lhere mothine cotdanding about this film, bta is good
Orignnal
coragct und will liccly be appreciniod mosi salkes and fidka know thes way unound sulemane..
48.59
Plsitiv
Postive
0.46
CEAsuck
(Oun)
caloe male who know their muttre invedd osherarie..
appespriate
will lizlyy he
0R5
116.7
Poettive
Nagater
0.50
Orignnl
Boogght hälden as quillo
Use movic aalioro as pick ampad by leepis lacie
showing thens.
407
Postrvv
Postive
0.40
CEAauck
(Oan)
Use facegis hidies
fltrmakees obtuins foratiabbee
(plume did slow lng dann
616
Poetrive
Negalse
0:50
Eoethning gred-can siting and Monty scars mocl mash
Originl
time, beak wame under illacon
324.20
Negative
Negaive
0.26
syson thondering
CEAtack
(Dur))
wising
periods, beck appear get under the trickers that he's swooinn the faoes syson down siteo.
notal much
088
571.90 93
Negative
Postiiv
0:75
atuutier has
waica hung
Drignal
Negatho
Nagalier
romuiittionn of the mnddlo--ppd chunates
(Omn)
CEAzuck
rolom to fnd some hank which to hang his pervisranly
Incingrass thm and may acatright get mesectaation
150
Negatoo
Poctise
Table 19: Confidence elicitation attacks and their confidence levels. Perturbed words are in bold.
24

---

Published as a conference paper at ICLR 2025
Qualitative Example
AG-Nows (News Clavifficatio) M-MMa---BBnnerrrn
Technique
Sumple
Perplesity
Perturbed Words
Ground Truih
Prediction
Empiriiall
(Suoee)
Producer snes Rings profis Hollywool proditrr
Okiginel
Zaenzz sues the prodacers of The Lord of Rings for $20m in royalties.
257.14
World
World
0.50
Producing sues for Kings peodit Hollywond producer Saul
CEAmack
(Oun)
7semez ues the producers of The Lodd of the Rings for $20m toyallies.
325.36
World
Business
0.66
Injuned Hankey to mis England Iniedlly NEWCASTLE, England (AP) Strike
Hoskey pulled
Original
England squad ahead of Wednesday #29s friendly against
213.64
Sport
Spurt
0.46
Ulkmine because of tight hunstringg the Football
Associaton snd Toesday.
Wound Heskey semorita Fngland frienily NEWCASTLF.
English (AP) Sinker Emile lieskey has pailed
CEAmack
of Wednesday #59s friendly Britaunica squid intensive hamstrinng Football
World
Ukmine
Thealay.
SEC may put and to quid pro goo USATTDAAYcom.
USATODAY com The Securities and Exchange Commission expected vote Wednesday to prohibit mutual fund conrganies from funncling stock trades to brokenge firms that their
Original
56.46
Business
Rusiness
0.47
pro (USATODAY
CEAmaci
(Ouns
expected to voices Wednesday probibot reciprocated fund conganies from fuancling stock trades to brokernge fimms that ok to promote their fands to investors
LSATODAY.com
Securilies Exchange Commission
0,87
140.03
Business
World
0.6
IBM Sects To Have SCD Claiss Dismissd (NowsFader)
NewsFactor IBM (NYSE: IBM has again
ponding
SCO
dismissed. According
Original
motion
argues
no evidence suppor its claims appoopritted
confidential source code from Unis System and placed Linux
IUM Seeks To Have SCD Claims Dismissed (NewsFactort) NewsPactor- 1&M (NYSE: (BM) has again
have
CEAtuck
the pending legal claims by The SCO Clasters dismiseed. According argues uppropeiated
Sci/Toch
Business
0.63
no support System and placed
Table 20: Examples confidence elicitation attacks and their respective confidence levels: Top) A positive example perturbed to negative, Bottom). A negative example perturbed to positive. Perturbed words are in bold.
Qualitalive Kxample
StrategyQA (Keasoning Clessifiation) LLMMa--MB-lmerrett
Technique
Sample
SemSitm
Perplextty
Pertarbed Words
Ground Truth
Prediction
Empinical Mean (Score)
Wchrrmcce
War
$3.2
Falke
0.49
[Ouns]
Conclide Whes?
Wehrmachs impacting
0.89
112.07
Fulse
Troe
0.62
The Weltemeett was the unifled military Germany from
Explamation
1935 to 1945 The War End All Wars a nickmame for
World War I World War ended in 1915
Original
Does Mercury make
Slip Slide matetial!
1224.4
False Filse
0.13
CEAmack
(Ouns
Does Mercury
Slip
was an water slide soy, Morcary is lhick liqiid at room temperature. Meroury is poisonous and used kill humers that lined their bais with the substance.
Slip
Explanation
Original
Would human go without chlonophyl!?
91.15
True
True
0.13
CEAmack
(Ouns
chlorophyll?
Chocophy pigment plants responsible
Explanation
chotosyntesis. Photosymiessis is the process by which plants release exygen into the atmosphere. Humans need esygen to live.
Original
Are more people today reltte to Cenghis Klian than Julins Caesr?
294.74
0.34
CEAnack
people today connected Genghis Carsar?
(Ours)
children. Genghis
Explanation
children. Modera geneticists have determincd that out of every 200 mon teday his DNA that can be ineed to Genghis Khim
Table 21: Examples of confidence elicitation attacks and their respective confidence levels: Top) A positive example perturbed to negative, Bottom) negative example perturbed to positive. Perturbed words are in bold.
25


Reviews:
[{"review_id": "5M1vFGQtRo", "reviewer": "Reviewer_dcYJ", "summary": "This paper explores a novel method of adversarial attack on large language models (LLMs) by leveraging their ability to express confidence levels. Unlike previous attacks that relied on model transparency, this research focuses on black-box scenarios where only the model's final output is accessible. The authors demonstrate that by eliciting confidence scores from LLMs, it is possible to approximate the effectiveness of soft-label attacks, which traditionally require access to output probabilities or logits. Through a series of experiments on three datasets and two models (LLaMa-3-8B Instruct and Mistral-7B Instruct-v0.3), the paper shows that minimizing the model's confidence can increase the likelihood of misclassification, achieving state-of-the-art results in hard-label, black-box attacks.", "questions": "* Is it possible to extend the proposed attack to text generation tasks?\n\n* Do most mainstream large language models (LLMs) support confidence elicitation?\n\n* Does the computational complexity of the proposed attack increase with the length of the input prompt?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This study introduces the first approach for constructing adversarial examples of large language models (LLMs) by leveraging their confidence elicitation capabilities. \n\n* The proposed confidence elicitation attack is easy to conduct and requires fewer queries than existing black-box adversarial attacks. This straightforward adversarial attack provides an easy-to-implement method for testing the potential vulnerabilities of LLMs while maintaining a higher degree of semantic similarity in the perturbed samples.", "weaknesses": "The study provides insights into the calibration of LLMs and the effectiveness of confidence elicitation in guiding adversarial perturbations, highlighting potential implications for the robustness of LLMs. Thus, it contributes a new perspective to the field of adversarial machine learning. However, I still see some issues that may improve the quality of the paper:\n\n* Limited in the Classification Tasks. Different from existing Jailbreaks against the generation function of LLMs. This work focuses specifically on the classification functions of LLMs. However, the primary use of LLMs is not for classification but rather for generating responses that provide users with their desired answers. The authors should justify the potential scenarios in which their attack may be employed, specifically those in which users utilize LLMs to classify their inputs.\n\n* Generalizability of Confidence Values. The study assumes that the elicited confidence values are well-calibrated for the models used, but it does not sufficiently explore or discuss the generalizability of this assumption across different LLMs or datasets. This limitation raises questions about the robustness of the attack vector when applied to models with varying levels of calibration or in different linguistic contexts.\n\n* Lack of Detail on ECE Calculation. The paper lacks a clear explanation of how to ensure that the confidence values returned by large language models (LLMs) are accurate for guiding the generation of perturbations, i.e., the Expected Calibration Error (ECE) technique, which is crucial for understanding the calibration of the models and the reliability of the confidence elicitation. Without this detail, it is challenging to assess the validity of the calibration claims and the effectiveness of the proposed attacks under different calibration scenarios. It is encouraged to integrate ECE into the methodology section of the paper or provide as background knowledge, which would offer transparency into the calibration process and allow readers to better assess the reliability of the confidence elicitation technique.\n\n* Complexity and Efficiency of Synonym Replacement Method. The proposed attack method involves synonym replacement that grows exponentially with the length of the input sentence, leading to a significant increase in computational complexity. This exponential growth can result in prolonged processing times for longer sentences, which may limit the practical applicability of the proposed attack. To mitigate the high computational complexity and time consumption associated with the synonym replacement method, especially for longer sentences, the authors should explore and implement optimizations in the word selection process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "92TkiUx7iR", "reviewer": "Reviewer_7pp6", "summary": "This paper proposes a new attack vector for large language models by overcoming the unachievable soft-label issues when attacking the large language models. To be more specific, the authors propose to approximate soft labels for hard label attacks by confidence elicitation. Experiment results have shown that their proposed method can reach state-of-the-art attack performance in hard-label, black-box, word-substitution-based settings on large language models.", "questions": "Please add experiments based on the suggestions from the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The first paper proposes to apply confidence elicitation to address the unachievable soft-label issues in adversarial attacks for large language models.\n2. Experimental results demonstrate the effectiveness of CEAttack compared with baseline attack methods Self-Fool Word Sub and SSPAttack.", "weaknesses": "1. Lack of baseline analysis compared with prompt optimization methods such as Tree of Attacks [1].\n2. The study scope of the paper is limited to classification tasks, which are only a small part of the tasks current large language models can do and have a minor impact for now. It is still unclear whether the method maintains effectiveness on generative tasks like jailbreaking.\n3. Potential defense discussions about the attack.\n\n[1] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box LLMs automatically. arXiv:2312.02119, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jecdzh3tCG", "reviewer": "Reviewer_Xe6F", "summary": "The authors observe that modern NLP models, specifically Large Language Models (LLMs), are still susceptible to adversarial examples. However, SOTA adversarial attacks require white-box or grey-box access to the target LLMs and this is no longer a realistic threat model. To overcome this constraint, the authors leverage methods from LLM calibration work to recover output confidence ratings, which they make compatible with typically grey-box attacks. Their success suggests that confidence elicitation provides a new attack vector for LLMs.", "questions": "+ This could be made clearer: The authors use the prompt template (asking for confidence scores) while optimising but is the optimised question then evaluated outside of the template?\n\n+ Is the total attack time tens of hours or tens of minutes? How many calls did each adversarial attack take on average? \n\n+ Does flipping the model confidence serve as an approximation to flipping the output probabilities, or is it actually the more desirable objective (in terms of what adversarial examples are meant to accomplish)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ This work is significant to the field, as the closed-source nature of modern LLMs is often referenced when arguing against popular optimisation methods (that require grey-box access). Additionally, this work shows that there may be some consequences to the confidence elicitation that many developers have been pushing for, though it likely isn’t enough to pause this work.\n\n+ The authors do a good job in presenting existing work, its findings and problems, as well as presenting their own method (through providing diagrams, examples, equations and pseudo code).\n\n+ The authors carry out substantive analysis of their method against multiple datasets, models, metrics and all compared against other methods. Its particularly nice that they tested how trustworthy the confidence elicitation outputs seem to be since the method assumes these to be correct (as seen in figure 3, though these graphs are not the easiest to understand straight away).", "weaknesses": "+ Experiments could possibly be improved by evaluating against actually closed-sourced models, especially since they should be more resilient to non-semantic input perturbations and are probably better at confidence elicitation. In fact, I assume that using anything smaller than the instruction-tuned 8B parameter models from the paper, will result in the prompt template no longer working?\n\n+ Since open-source models were used, the authors could have shown what difference still remains between your method and unconstrained white-box methods (if it is significant, then people can still claim that close-sourcing is very protective), but it is understandable for this to be too far outside of scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LpMrSYNFbr", "reviewer": "Reviewer_mH4K", "summary": "This paper shows how estimation of LLMs' confidence can be obtained even from black box models by directly asking models for their confidence (aggregating their answers over multiple queries), and how these continuous values can be used to generate better adversarial attacks.\n\nIt uses this approach to attack Llama3-8B and Mistral-7B-v0.3 on NLP classification tasks (SST2, AG-News and StrategyQA):\n* It uses these estimates to create better word-substitution attacks using a hill-climbing approach, which achieves higher ASR with smaller perturbations than baseline methods that use hard labels or no labels.\n* It qualitatively analyzes these attacks and their intermediate steps.", "questions": "* How good is the performance of the calibration estimation? How does it compare to simple baselines (like multiple queries at T=1)?\n* How would CEAttack perform with the complex confidence estimation replaced by a simpler confidence estimation technique?\n* What is a real world scenario where you expect your technique (or a technique inspired by your technique) to be relevant?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of estimating the confidence of black-box LLMs to inform adversarial attacks is interesting.\n* The presentation and the figures are clean and clear. It explains its ideas well.\n* The attacks are compared to baselines and a detailed sensitivity analysis on both the main technique and baselines is performed.\n* The attacks themselves are relatively interpretable, and this tool could present a nice starting point for future work aimed at understanding how LLMs perform classification tasks.", "weaknesses": "* The confidence estimation method presented here is never evaluated properly.\n  * Its performance or calibration is never measured.\n  * It is not compared to simple baselines (e.g. simply making multiple queries at temperature 1 and measuring the frequency of each answer).\n* It is unclear if the attack presented is relevant for the threat models described in the introduction (jailbreaks and adversarial misalignment): because in the case of harmful requests, the probability of a harmful completion is tiny, and models might put maximum confidence in their rejection, CEAttack might not get any continuous signal, and thus might not outperform hard-label methods. Other threat models might have been relevant (e.g. attacking LLM classifiers behind an API) but these also have properties which make attacks hard (e.g. it might not be possible to ask them questions).\n* The attack is never performed on real attack scenarios, or on real black-box models (which might have uncovered the issue above).\n* The attacks are not very impressive: the ASR is low in absolute terms, and when looking at the attacks, many of them seem to work only because they change the meaning of the sentence or make the sentence less meaningful. This is particularly bad for StrategyQA (e.g. \"Did the Wehrmacht affect the outcome of the War to End All Wars?\" --> \"Did the Wehrmacht impacting the outcome of the War to Conclude All Wars?\")\n* Black-box jailbreaking techniques like TAP already use similar ideas of more-or-less adherence to the target, and achieve higher performance in more difficult and realistic scenarios.\n\nMinor points:\n* The text is not clear about whether the experiments are conducted on Mistral v0.2 or v0.3.\n* The abstract should probably say what size of model was studied (which matters at least as much as the version number).\n* On line 267, the ASR is defined as the after success rate (it is the attack success rate)\n* The text in most tables is too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)