Title: SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models; Adversarial Attack; Red Teaming; LLM Hallucination; Robustness Evaluation; Trustworthy AI
Abstract: Large Language Models (LLMs) are increasingly deployed in high-risk domains. However, state-of-the-art LLMs often exhibit hallucinations, raising serious concerns about their reliability. Prior work has explored adversarial attacks to elicit hallucinations in LLMs, but these methods often rely on unrealistic prompts, either by inserting nonsensical tokens or by altering the original semantic intent. Consequently, such approaches provide limited insight into how hallucinations arise in real-world settings. In contrast, adversarial attacks in computer vision typically involve realistic modifications to input images. However, the problem of identifying realistic adversarial prompts for eliciting LLM hallucinations remains largely underexplored. To address this gap, we propose Semantically Equivalent and Coherent Attacks (SECA), which elicit hallucinations via realistic modifications to the prompt that preserve its meaning while maintaining semantic coherence. Our contributions are threefold: (i) we formulate finding realistic attacks for hallucination elicitation as a constrained optimization problem over the input prompt space under semantic equivalence and coherence constraints; (ii) we introduce a constraint-preserving zeroth-order method to effectively search for adversarial yet feasible prompts; and (iii) we demonstrate through experiments on open-ended multiple-choice question answering tasks that SECA achieves higher attack success rates while incurring almost no semantic equivalence or semantic coherence errors compared to existing methods. SECA highlights the sensitivity of both open-source and commercial gradient-inaccessible LLMs to realistic and plausible prompt variations. Code is available at https://github.com/Buyun-Liang/SECA.

SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations

Abstract

Warning: This method may be misused for malicious purposes.

Large Language Models (LLMs) are increasingly deployed in high-risk domains. However, state-of-the-art LLMs often exhibit hallucinations, raising serious concerns about their reliability. Prior work has explored adversarial attacks to elicit hallucinations in LLMs, but these methods often rely on unrealistic prompts, either by inserting nonsensical tokens or by altering the original semantic intent. Consequently, such approaches provide limited insight into how hallucinations arise in real-world settings. In contrast, adversarial attacks in computer vision typically involve realistic modifications to input images. However, the problem of identifying realistic adversarial prompts for eliciting LLM hallucinations remains largely underexplored. To address this gap, we propose Semantically Equivalent and Coherent Attacks (SECA), which elicit hallucinations via realistic modifications to the prompt that preserve its meaning while maintaining semantic coherence. Our contributions are threefold: (i) we formulate finding realistic attacks for hallucination elicitation as a constrained optimization problem over the input prompt space under semantic equivalence and coherence constraints; (ii) we introduce a constraint-preserving zeroth-order method to effectively search for adversarial yet feasible prompts; and (iii) we demonstrate through experiments on open-ended multiple-choice question answering tasks that SECA achieves higher attack success rates while incurring almost no semantic equivalence or semantic coherence errors compared to existing methods. SECA highlights the sensitivity of both open-source and commercial gradient-inaccessible LLMs to realistic and plausible prompt variations. Code is available at https://github.com/Buyun-Liang/SECA .

1 Introduction

Large Language Models (LLMs) have rapidly become integral to many high-stakes domains, including medical diagnosis [67], financial analysis [76], educational support [24], and scientific research [66]. However, these systems remain fundamentally brittle and may hallucinate incorrect responses that lead to catastrophic consequences if misused or uncritically trusted. For instance, given the prompt 'what is the value of p in 24 = 2 p ? ' , an LLM may produce a factually correct and faithful response, such as ' p = 12 , because 24 / 2 = 12 ' . By contrast, when presented with the lexically different but semantically equivalent prompt 'If doubling the value of p results in 24 , what is p ?' , the same model might produce a factual hallucination like ' p = 8 , because 24 / 2 = 8 ' . Hallucinations triggered by such realistic variations pose significant concerns about the safety and trustworthiness of LLMs, particularly in domains where factuality and faithfulness are paramount.

However, prior methods that elicit adversarial hallucinations fail to generate realistic attacks. For example, token-level optimization methods [90, 81] often produce unnatural or incoherent prompts

(e.g., Table 1(b)). While LLM agent-based [33, 7], beam search-based [64], and manual prompting [75, 85] approaches generate fluent prompts, they frequently diverge from the original question and are not semantically equivalent (e.g., Table 1(c),(d)). Such prompts provide limited insight into how hallucinations may occur in realistic scenarios and limited value for assessing LLM robustness.

To generate realistic attacks for hallucination elicitation, we pose the following research question:

(Q1) How can we formulate the problem of generating realistic attacks for hallucination elicitation as an optimization problem?

Realistic adversarial attacks have been widely explored in computer vision (CV) [26, 41, 71, 37, 88, 47], where the class of the adversarially perturbed image x is the same as that of the original image x 0 to a human observer, yet x causes a target model to produce a misclassification. Such attacks can be found via solving the optimization problem (1), where the objective is to minimize the classification loss L cls subject to two constraints: (i) the adversarial image x must remain close to the original image x 0 , that is, d img ( x , x 0 ) ≤ ϵ img; and (ii) x must lie within the set of valid images X img, e.g., staying within the valid pixel range and resembling a natural-looking input.

Inspired by problem (1), we formulate realistic attacks for hallucination elicitation in LLMs as problem (2): the objective is to minimize the hallucination loss L hall, subject to two constraints: (i) the adversarial prompt x must semantically close to the original prompt x 0 , namely d text ( x , x 0 ) ≤ ϵ text ; and (ii) x must belong to the set of valid prompts X text . In both cases, the attack objectives are targeted: the output of f CV is driven towards the target image class y ∗ img , and the output of f LLM towards the target hallucination prompt y ∗ text . The distances d img ( · , · ) and d text ( · , · ) ensure proximity under budgets ϵ img and ϵ text, respectively.

This formulation raises a question: how should we define the proximity and validity constraints in the discrete prompt space? For the proximity constraint, prior work has primarily relied on semantic similarity measures [8, 5, 62, 3, 79, 15, 82, 80, 28, 17], but similarity alone is insufficient for modeling realistic attacks. For instance, prompts 'What is the value of p in 24 = 2 p ?' and 'What is the value of p in 24 = 3 p ?' would be judged as semantically similar, yet they differ substantially in task goal and lead to different correct answers. By contrast, semantic equivalence provides a more appropriate notion of proximity. In program or formal expression domains [50, 53], it refers to entities that produce the same result despite surface differences. In natural language [14, 10, 2, 56, 57], it refers to mutual entailment (i.e., logical implication) between two prompts. We therefore define the proximity constraint d text ( x , x 0 ) ≤ ϵ text in problem (2) as a semantic equivalence constraint. For the validity constraint, the definition is more straightforward, since it is commonly understood as requiring semantically coherent and human-like language [9, 43, 84, 34]. We therefore define the validity constraint x ∈ X text as a semantic coherence constraint.

(Q2) How can we solve (2) to obtain semantically equivalent and coherent prompts that elicit LLM hallucinations?

Addressing Q2 requires overcoming two key challenges. First, exploring the discrete prompt space is combinatorially hard [90]. Second, because frontier LLMs already achieve high accuracy on many benchmarks, hallucination-inducing rephrasings are expected to be rare and thus difficult to uncover without exhaustive search. To tackle these challenges, we introduce Semantically Equivalent and Coherent Attacks (SECA), which avoids the need for exhaustive search over a combinatorial space by leveraging LLMs to propose and enforce feasible rephrasings. An illustrative example of SECA is shown in Figure 1. The key contributions of our approach are summarized as follows:

In §1 and §2.1, we formalize the problem of generating semantically equivalent and coherent attacks for hallucination elicitation in LLMs as a constrained optimization problem.

In §2.2, we propose SECA, a constraint-preserving zeroth-order method that effectively identifies the most adversarial yet feasible prompts in a gradient-free manner.

In §3, we demonstrate SECA's effectiveness on open-ended multiple-choice question answering tasks across frontier open-source and commercial LLMs, showing its ability to generate diverse, semantically equivalent, and coherent prompts that successfully elicit factual and faithful hallucinations. We also show strong agreement between LLM-based evaluators and human annotators, validating the reliability of automated evaluation in this setting. Moreover, we analyze the characteristics of successful attacks, finding that more verbose and lexically diverse prompts are more likely to induce hallucinations. Overall, our work underscores the importance of evaluating LLM robustness under realistic attacks.

2 Semantically Equivalent and Coherent Attacks (SECA)

2.1 Problem Formulation

In §1, we formulated the problem of obtaining semantically equivalent and coherent attacks as the general constrained optimization problem in (2). We now provide a more detailed explanation. As the name suggests, our formulation consists of three components, which we present in the sequel: the attack (objective), semantic equivalence (constraint), and coherence (constraint).

Attack . Prior work [90] showed that forcing aligned LLMs to produce certain tokens can induce harmful behavior. Inspired by this, given an input prompt and a set of answer choices, our goal is to elicit an incorrect option and a hallucinated rationale from the target LLM T . We define the hallucination loss as the probability of generating a target response y ∗ given the user prompt x as

where T is the total number of tokens in the target response y ∗ , y ∗ t is the t -th token in the target response, and P T ( y ∗ t | x , y ∗ 1: t -1 ) denotes the probability of generating y ∗ t given the input prompt x and tokens y ∗ 1: t -1 . For instance, x can be the multiple-choice question in Table 1 2 , and the target response y ∗ corresponds to a single token associated with the factuality-hallucinated choice, e.g., "B" . Our experiments in §3.3 show that responses beginning with an incorrect token are strongly associated with hallucinated reasoning in the model's explanation.

Semantic Equivalence . In CV (see problem (1)), adversarial attacks are corrupted images that are visually similar to the original image under metrics such as ℓ p distance or perceptual similarity [26, 41]. By analogy, in the language domain (see problems (2)), we advocate that realistic attacks should alter the original prompt only in ways that preserve its semantic content. To explicitly characterize the semantic equivalence constraint introduced in problem (2), we use a binary feasibility checker LLM F that determines whether two prompts satisfy the semantic equivalence constraint (see Appendix §H for the instruction template). The checker evaluates x and x 0 based on five criteria: (i) mutual entailment, (ii) x introduces no additional information beyond x 0 and the answer choices, (iii) x does not omit essential information from x 0 , (iv) x preserves the meaning of x 0 , and (v) x yields the same ground-truth answer as x 0 3 . Both prompts are semantically equivalent only if all five conditions are satisfied. Thus, we further define the semantic equivalence function as:

Semantic Coherence . In CV (see problem (1)), adversarial attacks require adversarial images to remain within the set of valid images, e.g., within the proper pixel range [37] and resembling a natural input [88]. By analogy, in the language domain (see problem (2)), we advocate that a realistic attack prompt should lie within the set of valid prompts, e.g., it should be semantically coherent to humans. As LLMs are trained on human-written corpora, semantic coherence naturally emerges as a property of their outputs when guided by appropriate instructions. Therefore, we treat all prompts produced by our instructed proposer LLM P (see §2.2 for details) as semantically coherent and denote this set as

Putting It All Together . For a prompt x to be considered an attack that is both semantically equivalent to the original prompt x 0 and semantically coherent 4 , we explicitly characterize problem (2) as the following optimization problem:

By maximizing the log probability of generating the target response y ∗ from x under these constraints, the resulting prompt is adversarial, yet semantically equivalent to x 0 and semantically coherent.

2.2 SECA: Traversing the Space of Semantically Equivalent and Coherent Prompts

A canonical challenge in token-level optimization, which also arises in (6), is that the prompt space is discrete and exponentially large. Our formulation alleviates this issue by imposing constraints on the search space. Specifically, our method traverses the space of semantically equivalent and coherent prompts, which is significantly smaller than the entire search space. However, this introduces its own difficulties, as these constraints are difficult to directly enforce via classical constrained optimization techniques such as standard projection operations. To address this, our key idea is to enforce the constraints directly by leveraging LLMs.

Traversing the Prompt Space with a Semantic Equivalence Proposer . We feed instructions to a proposer LLM P , asking it to propose M prompts that are all semantically equivalent to a given prompt; it corresponds to Line 6 of Algorithm 1 (see Appendix §G for the instruction template). A potential issue here is semantic collapse : The M prompts might be identical to each other or to the original prompt. To alleviate this, we randomly sample instructions from predefined verb, style, and

1: Input: Original prompt x 0 , target prompt y ∗ , target LLM T , proposer LLM P , feasibility checker LLM F 2: Initialize: x best ← x 0 , candidates ← [ x 0 ] N 3: Terminology : We say a is more adversarial than b if log P T ( y ∗ | a ) > log P T ( y ∗ | b ) while stop criterion not reached: 4: candidates_tmp ← candidates 5: for each x in candidates : 6: [ x i ] i M =1 ← M prompts proposed by LLM P given x and instructions; 7: adv ← all prompts in [ x i ] i M =1 that are more adversarial than x best ; 8: feasible ← all prompts in adv that pass the feasibility test of LLM F ; 9: candidates_tmp ← merge candidates_tmp and feasible ; 10: candidates ← the most N adversarial prompts in candidates_tmp ; 11: x best ← the most adversarial prompt in candidates ; 12: Output: x best

template sets to encourage diversity in candidate prompts. On the other hand, as LLMs are trained on human-written corpora, coherence surfaces as an emergent property of the LLM outputs given our instructions. Thus, we do not explicitly encourage the proposer to produce coherent responses.

Enforcing Constraints via a Feasibility Checker . In principle, all candidate prompts generated by LLM P should be semantically equivalent to the input x . In practice, however, the proposer LLM may still produce hallucinations 5 . To mitigate this, we verify each candidate using the feasibility checker LLM F 6 from problem (6), which evaluates whether x and x 0 satisfy the semantic equivalence constraint (see Line 8 of Algorithm 1 and Appendix §H for the instruction template). This additional verification provides strict feasibility guarantees, as shown in §3.2.

The Most Adversarial Attack . With the proposer and feasibility checker in effect, we reduce the search space to a limited set of candidate prompts rather than the entire space. Moreover, these prompts tend to satisfy the constraints of semantic equivalence and coherence. Thus, what remains is to identify the most adversarial candidate. As a proxy for the adversarial strength of a prompt x , we condition on this prompt and compute the log probability log P T ( y ∗ | x ) of generating y ∗ . Following the philosophy of our optimization problem (6), the most adversarial attack arises as the prompt among the candidates that maximizes this log probability.

Putting It All Together . Algorithm 1 integrates above components into a unified procedure. We initialize the candidate set with N copies of the original prompt x 0 (Line 2). At each iteration, new candidates are generated by the semantic equivalence proposer (Line 6), evaluated for adversarial strength (Line 7), and then filtered by the feasibility checker (Line 8). Among all feasible candidates, we retain only the topN adversarial prompts for the next iteration as candidates (Line 10). The process terminates when either log P T ( y ∗ | x best ) exceeds a predefined threshold or a maximum number of iterations is reached. These design choices make SECA a simple yet effective constraintpreserving zeroth-order method. An illustrative example of one iteration is shown in Figure 2.

3 Experiments

3.1 Experimental Setups

Dataset and Target Token . We evaluate our approach with the commonly used MMLU dataset [19], where each sample consists of multiple-choice questions and the correct answer. However, some questions in this dataset might already induce hallucinations of a target LLM. To isolate the effect of this, we only consider the questions for which the target LLMs produce correct answers. To do so, we create a filtered subset of MMLU, where each prompt is included if and only if all target LLMs assign the highest confidence to the correct answer token. After this filtering, the resulting

dataset contains 347 samples and spans 16 diverse subjects such as science, engineering, and arts 7 . Motivated by MMLU, we conduct attacks in an open-ended multiple-choice question answering (MCQA) setting. Each question is paired with four answer options labeled 'A', 'B', 'C', and 'D', which are included in the input prompt. The target LLM T is then required to produce an answer choice and a corresponding explanation 8 . Within this setup, the log probability log P T ( y ∗ | x ) in the experiments measures the log probability that an input x elicits the target incorrect token y ∗ , e.g., 'B'. In practice, we designate the most likely incorrect answer choice in the MCQA as y ∗ .

Baselines . Our first baseline, Raw , simply uses the original MMLU queries directly as attacks on target LLMs. The second baseline is Greedy Coordinate Gradient (GCG) [90]. We do not include other hallucination-elicitation or jailbreak baselines, as we are not aware of any prior work explicitly designed to address the task in problem (2), namely finding semantically equivalent and coherent prompts (See §1, §4, and Appendix §D for detailed discussion). We therefore select GCG as the most representative SOTA method applicable to hallucination elicitation. Additional configuration details for both GCG and SECA are provided in Appendix §L.

LLMs . We use both open source and commercial models as our target LLMs; this includes Qwen-2.57B/14B, Llama2-13B, Llama3-3B/8B, GPT-4.1-Nano, GPT-4o-Mini (see Appendix §J for the detailed version of LLMs used in this paper). For SECA, we use GPT-4.1-Nano as the semantic equivalence proposer LLM P due to its high response speed, low query cost, and strong instruction-following capabilities (see Appendix §G). We use GPT-4.1-Mini as the feasibility checker LLM F (see Appendix §H) to evaluate the semantic equivalence between two prompts x and x 0 ; LLM F produces binary outputs and returns either 1 or 0. To evaluate the semantic coherence of a prompt x , we use perplexity computed by GPT-2 [61] (denoted as G ), i.e., PPL G ( x ) = exp { -1 n ∑ n t =2 log P G ( x t | x 1: t -1 ) } . Furthermore, to analyze the specific types of hallucinations induced, we employ GPT-4.1 as a hallucination evaluator to classify the response of the target LLM into one of four categories: Factuality , Faithfulness , Other , or None ; see Appendix §I for the instruction template. Factuality indicates the response contains false or inaccurate information; Faithfulness denotes misrepresentation of the input prompt; Other captures issues such as ambiguity, incompleteness, or under-informativeness; and None is assigned to responses that are both factually correct and faithful to the input.

Metrics . We define several evaluation metrics based on LLMs introduced above. First, note that the target LLM typically generates a token indicating the option it chooses, followed by an explanation. We say an attack is successful if it elicits an incorrect option and then an explanation that is classified as either Factuality or Faithfulness by the hallucination evaluator (see Appendix §B for an example of a successful attack). The Best-ofK Attack Success Rate , written as ASR@ K , measures the percentage of samples for which at least one successful attack is found within K trials. Next, we define Semantic Equivalence Error (SEE) and Semantic Coherence Error (SCE) that measure the extent to which the constraints on semantic equivalence and coherence are respectively violated.

Here SEE ( x , x 0 ) = 0 indicates the algorithm output x preserves semantic equivalence, while SEE ( x , x 0 ) = 1 indicates x deviates from what the original prompt x 0 means. Also, a lower value of SCE ( x ) indicates better semantic coherence in x . Throughout all experiments, we fix the tolerance at γ = 60 to permit a small degree of incoherence, reflecting what is commonly observed in human-generated prompts. Finally, we define Type Token Ratio ( TTR ) to be the ratio between the number of unique tokens in an output prompt and the total number of tokens (within a given window and averaged over the dataset). Therefore, a larger TTR indicates the prompts contain a more diverse set of vocabulary. Appendix §L includes additional details of our experimental setups.

3.2 Attack Performance Comparison with GCG

Here we compare our SECA algorithm with GCG in terms of ASR@ K with K = 30 and semantic errors; we found the experimental conclusion remains the same for different values of K (e.g., K = 1 , 10 , or 30 ). Please see Appendix §M for ASR@30/10/1 results of SECA and raw prompts.

ASR . Table 2 shows SECA has much higher ASR@30 than raw prompts and GCG, demonstrating its superior ability to elicit hallucinations. To understand why GCG has even lower ASR@30 than the

AttackSuccessRate(ASR)@30

Average△logP(y*|x)

AverageSemanticErrors（↓)

ML

CS

Mat

ML

CS

Mat

GCG

SCE

0.83

866.45

Sec

Ana

Sec

Ana

SEE

0.00

0.97

80

10.0

Phy

60

Bio

Phy

6.0

8.0

Bio

40

4.0

2.0

Llama-3-3B

Che

Cli

Llama-3-8B

Llama-2-13B

CPy

Eco

CPy

Eco

Qwen-2.5-14B

Qwen-2.5-7B

Psy

Law

Psy

Law

GPT-4o-Mini

GPT-4.1-Nano

Soc

Hi

Soc

Hi

Phi

Phi

raw prompts, we note that GCG initializes the original prompt x 0 by appending a gibberish suffix, which would typically decrease the objective value (that is the probability of generating the target token). In fact, the decrease is so large that the subsequent optimization steps of GCG, despite being very costly, are unable to improve the objective back to the original level. This is very different from our SECA approach, which increases the objective value monotonically by design and efficiently; see Figure 3 (middle) and Figure 4a.

Semantic Errors . Table 2 furthermore shows that SECA has as minimal SEE and SCE as the original prompt. In sharp contrast, GCG tends to generate incoherent and gibberish prompts. Overall, the experiment in Table 2 corroborates our design purpose of SECA, which aims to generate semantically equivalent and coherent yet adversarial prompts.

3.3 Empirical Analysis of SECA

As SECA is an early approach to elicit LLM hallucinations under the constraints of semantic equivalence and coherence, in this section, we aim to provide an extensive empirical analysis of its behavior and properties.

Attack Performance Analysis . Figure 3 presents a comprehensive evaluation of SECA on 16 subjects of MMLU and 7 different LLMs. From Figure 3 (left), we make two observations:

For commercial LLMs (GPT-4o-Mini and GPT-4.1-Nano) and competitive open-source models (Qwen-2.5-7B/14B and Llama-2-13B), the raw prompts (yellow, brown, green, pink, and blue dotted

-4

-6

Objective log P(y

-8

-10

12

Llama-3-3B

Qwen-2.5-7B

14

Llama-3-8B

GPT-4o-Mini

16

Llama-2-13B

Qwen-2.5-14B

GPT-4.1-Nano

18

0

5

10

Iteration

15

20

25

30

Llama-3-3B

Qwen-2.5-7B

80

Llama-3-8B

GPT-4o-Mini

Llama-2-13B

GPT-4.1-Nano

Qwen-2.5-14B

Percentage

60

40

20

0

Factuality

Faithfulness

Other

None

Type-Token Ratio

0.90

0.85

0.80

Prompt Length (Log Scale)

0.75

100

200

SECA Prompts

Original Prompts

50

20

10

Figure 4: Analysis of SECA: (a) objective progression over iterations; (b) hallucination type breakdown; and (c) lexical diversity and verbosity of SECA prompts. The shaded area in (a) and the error bar in (c) represent the standard deviation calculated over 10,000 bootstrap samples.

lines) yield ASR@30 below 10% in most cases, indicating that hallucinations are not triggered by default. In contrast, SECA (solid lines in matching colors) raises ASR@30 by at least 20% on most subjects-except for a few knowledge retrieval-based subjects such as ML, Law, Phi, and Che, where hallucinations are less likely compared to reasoning-focused subjects.

For open-source Llama-3-3B and Llama-3-8B, the raw prompts (cyan and orange, dotted) yield 40-60% ASR@30 for most subjects, which is relatively high already. Nonetheless, SECA still boosts ASR@30 by around 20% on the majority of subjects, except in cases such as Mat, ML, and CS. These exceptions likely stem from the fact that some original prompts either already induce hallucination or do not hallucinate with high confidence. SECA, or any other algorithm, finds no improvements in the former case, while the latter case indicates that the target LLM has learned some notion of semantic equivalence with respect to those particular prompts, which makes them more robust to rephrasings.

From Figure 3 (middle), we see that SECA increases the objective, that is, the (log) probability log T P ( y ∗ | x ) eliciting the incorrect target token. This verifies again the correctness of our design of SECA and its effectiveness in maximizing the objective of (6) while maintaining semantic equivalence and coherence. Indeed, since a larger increase in this objective in general correlates with a higher probability of a successful attack, the objective increase in Figure 3 (middle) thus justifies the increased attack success rate in Figure 3 (left). Notably, target LLMs with lower initial confidence in predicting the target token tend to exhibit greater increases in the objective: GPT-4o-Mini, which has low initial confidence, exhibits the most significant increase among all 7 LLMs. Finally, Figure 3 (right) shows that, despite its strong performance, SECA maintains nearly zero average semantic errors, thanks to the design of the proposer + feasibility checker pipeline. For additional experimental results targeting GPT-3.5-Turbo and GPT-4, please see Appendix §N.

Empirical Convergence Analysis . When solving (6), SECA generates only candidate prompts that satisfy the semantic equivalence and coherence constraints. As a result, the change in the objective value over iterations serves as the key metric for tracking the progress of SECA. From Figure 4a we first note that SECA exhibits a clear trend of increasing the objective, resulting in a convergence in 30 iterations for most LLMs. Then, we find it insightful to compare Figure 4a with Figure 3:

The objective value (Figure 4a) is positively correlated to the attack success rate (Figure 3, left). Indeed, in Figure 4a the objective values on Llama-3-3B and Llama-3-8B are the largest, which correspond to the largest circles in Figure 3 (left). This suggests that the log probability log P T ( y ∗ | x ) is an effective proxy measure for the adversarial level of a prompt x and that SECA is an effective strategy to find such prompt.

Similarly to Figure 3 (middle), we observe here that LLMs with lower initial confidence in predicting the target token tend to exhibit larger increases in the objective values (see, e.g., GPT-4oMini in both figures).

Complementary to the numerical increase of the objective values in Figure 4a, we present in Appendix §K the attack prompts that SECA iteratively finds in textual form.

Hallucination Analysis . A key motivation for using an incorrect token y ∗ as the target in the SECA optimization problem (6) is that the follow-up explanation y explanation conditioned on incorrect tokens are more likely to be generated via hallucination. To empirically verify this, we take the adversarial prompt x along with the target token y ∗ , and invoke our hallucination evaluator (as defined in §3.1) to classify the generated explanation y explanation ∼ LLM T ( y explanation | x , y ∗ ) into one of the four hallucination types ( Factuality , Faithfulness , Other , None ). As shown in Figure 4b, most hallucinated responses fall under the factuality category. We also observe that SECA prompts are more likely to elicit the Llama variants to hallucinate, which aligns with the higher ASR@30 observed for these models in Figure 3. These results demonstrate the effectiveness of using an incorrect token y ∗ to elicit hallucinations in a controlled and targeted manner.

Prompt Analysis . Although SECA prompts are semantically equivalent to the original question prompts (e.g., see Table 2), here we aim to understand why they are more capable of eliciting LLM hallucinations than the original prompts. To this end, we compare SECA prompts to the original ones in terms of lexical diversity and prompt length. From Figure 4c, we make two key observations:

· SECA prompts exhibit higher TTR than the original prompts in nearly all subjects, indicating more diverse and creative wording used to express the same concepts, with only one exception on Philosophy (Phi), where the original prompts are already highly varied.

· SECA prompts are generally longer than the original prompts across most subjects, suggesting that they employ more elaborate sentence structures to convey the same meaning. An exception, though, is on High School US History (Hi), as it often includes verbose problem descriptions by default.

Overall, SECA prompts are both more lexically diverse and verbose compared to the original prompts, despite preserving semantic equivalence. This increased linguistic variation may obscure the core intent of the original prompt, thereby increasing the probability of hallucination. For an illustrative textual example comparing SECA prompts and the original prompt, see Appendix §K.

3.4 Are LLMs Reasonable Evaluators?

Our final experiment is concerned with whether LLMs are sufficient to check semantic equivalence (Appendix §H) and classify hallucination types (Appendix §I). To this end, we assess the alignment between LLM-based evaluations and human annotations by comparing the outputs of our feasibility checker and hallucination evaluators against human annotations. Human annotations were provided by two annotators, each with at least an undergraduate-level education in science or engineering, with access to external resources such as Google and Wikipedia. From Table 3, we make two

Table 3: Comparing LLM-based evaluators with two human annotators (A&B) on accuracy (Acc), precision (Pre), recall (Rec), F1 score, and Cohen's κ .

observations. First, the feasibility checker achieves perfect recall but relatively lower precision, indicating a higher false positive rate with no false negatives. Nonetheless, its overall F1 score and Cohen's κ suggest reasonably strong agreement with human annotations, validating our choice of LLM-based (semantic equivalence) feasibility checker. Second, the hallucination evaluator demonstrates strong alignment with both human annotators across all metrics, confirming that it aligns well with human judgment. These results support the use of both evaluators in our framework.

4 Related Work

There are two important lines of research on adversarial methods for LLMs. The first is jailbreak attacks , which are primarily aimed at inducing harmful behaviors in target models 9 . Representative approaches include gradient-based [89, 73, 17, 90], LLM attacker-based [9, 52, 42, 34],

puzzle/game/disguise-based [32, 83, 31, 48, 45], and genetic algorithm-based methods [43, 30]. The second is hallucination elicitation , which seeks to provoke factually or faithfully incorrect outputs and presents a distinct set of challenges. Representative approaches include optimization-based [81], LLM-agent-based [33, 7], beam search-based [64], and manual prompting-based methods [75, 85]. However, none of these methods satisfy the requirements of semantic equivalence and coherence. Therefore, they can be categorized as gibberish , trivial , or meaning-shift attacks:

Gibberish Attacks Token-level optimization-based methods such as GCG [90] and Hallucination Attack [81] can induce specific responses in target LLMs. However, they often produce attacks that deviate from real-world scenarios by inserting nonsensical tokens in the attack prompt. We therefore categorize these as gibberish attacks , as illustrated in Table 1 (b).

Trivial Attacks Manual prompting-based methods like ICD [85] and fictional scenario-based jailbreaking methods [9, 52, 34, 84] directly query specific hallucinated or harmful content from target LLMs. In jailbreak attacks, the goal is to bypass the safety mechanism. Thus, a model that complies with such instructions is considered successfully jailbroken. For hallucination elicitation, however, such attacks fail to evaluate robustness: the target model is merely following instructions, which is expected behavior, rather than failing on realistic user queries. We therefore categorize these as trivial attacks , as illustrated in Table 1 (c).

Meaning-Shift Attacks Jailbreak attacks focus on bypassing safety mechanisms with arbitrary attack prompts; consequently, many methods [89, 17, 9, 52, 43, 34] produce attacks that alter key information in the original prompt. For hallucination elicitation, LLM-agent-based methods like Investigator Agent [33] and Adaptive Evaluation [7], as well as beam search-based methods like BEAST [64], and manual prompting-based methods like Answer Assemble Ace [75] often rely on attack prompts that deviate from the original task. When the input prompt meaning changes, different outputs are expected-they reflect the altered task rather than a hallucination of the original prompt. We therefore categorize these as meaning-shift attacks , as illustrated in Table 1 (d).

Using the notation of problem (2), gibberish attacks generate incoherent prompts, i.e., x ̸∈ X text ; trivial attacks and meaning-shift attacks generate adversarial prompts that are not semantically equivalent to the original prompts, i.e., d text ( x , x 0 ) > ϵ text. Consequently, existing methods fail to find feasible solutions to problem (2).

In Appendix §E, we further elaborate on additional related work on faithful and factual LLMs and constrained deep learning .

5 Conclusion and Future Work

In this work, we introduce SECA, a novel constraint-preserving zeroth-order method for eliciting hallucinations in LLMs through semantically equivalent and linguistically coherent prompt rephrasings. By casting the attack generation as a constrained optimization problem and leveraging LLM-based proposers and feasibility checkers, SECA effectively discovers adversarial prompts that are semantically equivalent and coherent while significantly increasing hallucination rates across both commercial and open-source LLMs. Our empirical analysis reveals that hallucinations are more likely to occur when prompts are more verbose and lexically diverse, offering key insight into how subtle variations in natural language can trigger model failures. To support the community and enable further research on LLM robustness, we open-source our framework at https://github.com/Buyun-Liang/SECA . We hope SECA serves as a basic tool for advancing the understanding and mitigation of hallucinations in real-world LLM applications.

This work also opens several directions for future research: (i) integrating zeroth-order gradient estimation techniques (e.g., finite differences) to accelerate convergence and improve SECA's scalability for large-scale red teaming; (ii) extending SECA beyond the open-ended MCQA setting to open-ended free-form generation tasks, such as factual errors in long-form answers or summarization; (iii) developing untargeted variants by incorporating hallucination evaluator outputs directly into the objective, enabling the discovery of diverse hallucinations without relying on predefined targets; and (iv) extending SECA to target reasoning models, i.e., models that generate reasoning steps before answering. Please refer to Appendix §O for further discussion of limitations, societal impacts, and future directions.

Acknowledgments and Disclosure of Funding

This research was supported by NSF grants 2212457 and 2031985, the Simons Foundation grant 814201, and University of Pennsylvania Startup Funds. We would like to thank Nghia Nguyen, Kaleab Kinfu, Fengrui Tian, and Ziqing Xu for their valuable suggestions on improving the presentation of this paper. The views and conclusions expressed in this work are those of the authors and do not necessarily reflect the official policies, either expressed or implied, of NSF or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes, notwithstanding any copyright notice herein.

References

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, and others. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

I. Androutsopoulos and P. Malakasiotis. A Survey of Paraphrasing and Textual Entailment Methods. Journal of Artificial Intelligence Research , 38:135-187, May 2010.

A. Aynetdinov and A. Akbik. SemScore: Automated Evaluation of Instruction-Tuned LLMs based on Semantic Textual Similarity, Feb. 2024. arXiv:2401.17072 [cs].

N. Balepur, A. Ravichander, and R. Rudinger. Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?, June 2024. arXiv:2402.12483 [cs].

F. Bao, M. Li, R. Luo, and O. Mendelevitch. HHEM-2.1-Open, 2024.

N. Boumal, B. Mishra, P.-A. Absil, and R. Sepulchre. Manopt, a Matlab toolbox for optimization on manifolds. The Journal of Machine Learning Research , 15(1):1455-1459, 2014. Publisher: JMLR. org.

D. Brown, P. Balehannina, H. Jin, S. Havaldar, H. Hassani, and E. Wong. Adaptively evaluating models with task elicitation, Mar. 2025. arXiv:2503.01986 [cs].

D. Cer, Y. Yang, S.-y. Kong, N. Hua, N. Limtiaco, R. S. John, N. Constant, M. GuajardoCespedes, S. Yuan, C. Tar, Y.-H. Sung, B. Strope, and R. Kurzweil. Universal Sentence Encoder, Apr. 2018. arXiv:1803.11175 [cs].

P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking Black Box Large Language Models in Twenty Queries, July 2024. arXiv:2310.08419 [cs].

P. W. Culicover. Paraphrase Generation and Information Retrieval from Stored Text. Mech. Transl. Comput. Linguistics , 1968.

R. R. Curtin, M. Edel, R. G. Prabhu, S. Basak, Z. Lou, and C. Sanderson. The ensmallen library for flexible numerical optimization. Journal of Machine Learning Research , 22(166):1-6, 2021.

DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong,

Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, and Z. Zhang. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, Jan. 2025. arXiv:2501.12948 [cs].

S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, and J. Weston. Chain-ofVerification Reduces Hallucination in Large Language Models, Sept. 2023. arXiv:2309.11495 [cs].

S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal. Detecting hallucinations in large language models using semantic entropy. Nature , 630(8017):625-630, June 2024.

J. Gatto, O. Sharif, P. Seegmiller, P. Bohlman, and S. M. Preum. Text Encoders Lack Knowledge: Leveraging Generative LLMs for Domain-Specific Semantic Textual Similarity, Sept. 2023. arXiv:2309.06541 [cs].

A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, and others. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

X. Guo, F. Yu, H. Zhang, L. Qin, and B. Hu. COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability, June 2024. arXiv:2402.08679 [cs].

K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. REALM: Retrieval-Augmented Language Model Pre-Training, Feb. 2020. arXiv:2002.08909 [cs].

D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring Massive Multitask Language Understanding, Jan. 2021. arXiv:2009.03300 [cs].

L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ACM Transactions on Information Systems , 43(2):1-55, Mar. 2025. arXiv:2311.05232 [cs].

G. Izacard, M. Caron, L. Hosseini, S. Riedel, P. Bojanowski, A. Joulin, and E. Grave. Unsupervised Dense Information Retrieval with Contrastive Learning, Aug. 2022. arXiv:2112.09118 [cs].

A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, and others. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024.

V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 6769-6781, Online, Nov. 2020. Association for Computational Linguistics.

E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, S. Krusche, G. Kutyniok, T. Michaeli, C. Nerdel, J. Pfeffer, O. Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, and G. Kasneci. ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences , 103:102274, Apr. 2023.

M. Kochurov, R. Karimov, and S. Kozlukov. Geoopt: Riemannian optimization in pytorch. arXiv preprint arXiv:2005.02819 , 2020.

C. Laidlaw, S. Singla, and S. Feizi. Perceptual Adversarial Robustness: Defense Against Unseen Threat Models, July 2021. arXiv:2006.12655 [cs].

S. Laue, M. Mitterreiter, and J. Giesen. GENO-GENeric Optimization for Classical Machine Learning. Advances in Neural Information Processing Systems , 32, 2019.

G. Li, B. Shi, Z. Liu, D. Kong, Y. Wu, X. Zhang, L. Huang, and H. Lyu. Adversarial Text Generation by Search and Learning. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 15722-15738, Singapore, 2023. Association for Computational Linguistics.

J. Li, J. Chen, R. Ren, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen. The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models, Jan. 2024. arXiv:2401.03205 [cs].

X. Li, S. Liang, J. Zhang, H. Fang, A. Liu, and E.-C. Chang. Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs, Feb. 2024. arXiv:2402.14872 [cs].

X. Li, R. Wang, M. Cheng, T. Zhou, and C.-J. Hsieh. DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakers, Mar. 2024. arXiv:2402.16914 [cs].

X. Li, Z. Zhou, J. Zhu, J. Yao, T. Liu, and B. Han. DeepInception: Hypnotize Large Language Model to Be Jailbreaker, May 2024. arXiv:2311.03191 [cs].

X. L. Li, N. Chowdhury, D. D. Johnson, T. Hashimoto, P. Liang, S. Schwettmann, and J. Steinhardt. Eliciting Language Model Behaviors with Investigator Agents, Feb. 2025. arXiv:2502.01236 [cs].

B. Liang, K. H. R. Chan, D. Thaker, J. Luo, and R. Vidal. KDA: A Knowledge-Distilled Attacker for Generating Diverse Prompts to Jailbreak LLMs, Feb. 2025. arXiv:2502.05223 [cs].

B. Liang, T. Mitchell, and J. Sun. NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning, 2021. arXiv:2111.13984 [cs].

B. Liang, T. Mitchell, and J. Sun. NCVX: A General-Purpose Optimization Solver for Constrained Machine and Deep Learning, Nov. 2022. arXiv:2210.00973 [cs].

H. Liang, B. Liang, Y. Cui, T. Mitchell, and J. Sun. Optimization for Robustness Evaluation beyond $\ell_p$ Metrics, Nov. 2022. arXiv:2210.00621 [cs].

H. Liang, B. Liang, L. Peng, Y. Cui, T. Mitchell, and J. Sun. Optimization and Optimizers for Adversarial Robustness, Mar. 2023. arXiv:2303.13401 [cs].

H. Liang, B. Liang, J. Sun, Y. Cui, and T. Mitchell. Implications of Solution Patterns on Adversarial Robustness. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 2393-2400, Vancouver, BC, Canada, June 2023. IEEE.

X. V. Lin, X. Chen, M. Chen, W. Shi, M. Lomeli, R. James, P. Rodriguez, J. Kahn, G. Szilvasy, M. Lewis, L. Zettlemoyer, and S. Yih. RA-DIT: Retrieval-Augmented Dual Instruction Tuning, May 2024. arXiv:2310.01352 [cs].

J. Liu, C. Wei, Y. Guo, H. Yu, A. Yuille, S. Feizi, C. P. Lau, and R. Chellappa. Instruct2Attack: Language-Guided Semantic Adversarial Attacks, Nov. 2023. arXiv:2311.15551 [cs].

X. Liu, P. Li, E. Suh, Y . V orobeychik, Z. Mao, S. Jha, P. McDaniel, H. Sun, B. Li, and C. Xiao. AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs, Nov. 2024. arXiv:2410.05295 [cs].

X. Liu, N. Xu, M. Chen, and C. Xiao. AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models, Mar. 2024. arXiv:2310.04451 [cs].

Y. Liu, H. Gao, S. Zhai, J. Xia, T. Wu, Z. Xue, Y. Chen, K. Kawaguchi, J. Zhang, and B. Hooi. GuardReasoner: Towards Reasoning-based LLM Safeguards, Jan. 2025. arXiv:2501.18492 [cs].

Y. Liu, X. He, M. Xiong, J. Fu, S. Deng, and B. Hooi. FlipAttack: Jailbreak LLMs via Flipping, Oct. 2024. arXiv:2410.02832 [cs].

Y. Liu, S. Zhai, M. Du, Y. Chen, T. Cao, H. Gao, C. Wang, X. Li, K. Wang, J. Fang, J. Zhang, and B. Hooi. GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning, May 2025. arXiv:2505.11049 [cs].

J. Luo, Z. Wang, C. H. Wu, D. Huang, and F. De La Torre. Zero-Shot Model Diagnosis. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11631-11640, Vancouver, BC, Canada, June 2023. IEEE.

H. Lv, X. Wang, Y. Zhang, C. Huang, S. Dou, J. Ye, T. Gui, Q. Zhang, and X. Huang. CodeChameleon: Personalized Encryption Framework for Jailbreaking Large Language Models, Feb. 2024. arXiv:2402.16717 [cs].

Q. Lyu, S. Havaldar, A. Stein, L. Zhang, D. Rao, E. Wong, M. Apidianaki, and C. Callison-Burch. Faithful Chain-of-Thought Reasoning, Sept. 2023. arXiv:2301.13379 [cs].

D. L. McGuinness, F. Van Harmelen, and others. OWL web ontology language overview. W3C recommendation , 10(10):2004, 2004.

M. Meghwanshi, P. Jawanpuria, A. Kunchukuttan, H. Kasai, and B. Mishra. McTorch, a manifold optimization library for deep learning. arXiv preprint arXiv:1810.01811 , 2018.

A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi. Tree of Attacks: Jailbreaking Black-Box LLMs Automatically, Feb. 2024. arXiv:2312.02119 [cs, stat].

M. J. Min, Y. Ding, L. Buratti, S. Pujar, G. Kaiser, S. Jana, and B. Ray. Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain, Feb. 2024. arXiv:2310.14053 [cs].

N. Miolane, N. Guigui, A. Le Brigant, J. Mathe, B. Hou, Y. Thanwerdas, S. Heyder, O. Peltre, N. Koep, H. Zaatiti, and others. Geomstats: A python package for riemannian geometry in machine learning. Journal of Machine Learning Research , 21(223):1-9, 2020.

N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto. s1: Simple test-time scaling, Mar. 2025. arXiv:2501.19393 [cs].

M. Negri, L. Bentivogli, Y. Mehdad, D. Giampiccolo, and A. Marchetti. Divide and Conquer: Crowdsourcing the Creation of Cross-Lingual Textual Entailment Corpora. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing , 2011.

S. Padó, D. Cer, M. Galley, D. Jurafsky, and C. D. Manning. Measuring machine translation quality as semantic equivalence: A metric based on entailment features. Machine Translation , 23(2-3):181-193, Sept. 2009.

A. Paulus, A. Zharmagambetov, C. Guo, B. Amos, and Y. Tian. AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs, Apr. 2024. arXiv:2404.16873 [cs].

P. Pezeshkpour and E. Hruschka. Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questions, Aug. 2023. arXiv:2308.11483 [cs].

G. Pillo and M. Roma. Large-scale nonlinear optimization , volume 83. Springer Science & Business Media, 2006.

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, and others. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

N. Reimers and I. Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERTNetworks, Aug. 2019. arXiv:1908.10084 [cs].

J. Robinson, C. M. Rytting, and D. Wingate. Leveraging Large Language Models for Multiple Choice Question Answering, Mar. 2023. arXiv:2210.12353 [cs].

V. S. Sadasivan, S. Saha, G. Sriramanan, P. Kattakinda, A. Chegini, and S. Feizi. Fast Adversarial Attacks on Language Models In One GPU Minute, Feb. 2024. arXiv:2402.15570 [cs].

S. Sra, S. Nowozin, and S. J. Wright. Optimization for machine learning . Neural information processing series. MIT press, Cambridge (Mass.), 2012.

R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. Galactica: A Large Language Model for Science, Nov. 2022. arXiv:2211.09085 [cs].

R. Tinn, H. Cheng, Y. Gu, N. Usuyama, X. Liu, T. Naumann, J. Gao, and H. Poon. Fine-tuning large neural language models for biomedical natural language processing. Patterns , 4(4):100729, Apr. 2023.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, and others. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

J. Townsend, N. Koep, and S. Weichwald. Pymanopt: A python toolbox for optimization on manifolds using automatic differentiation. Journal of Machine Learning Research , 17(137):1-5, 2016.

B. Wang, W. Ping, L. McAfee, P. Xu, B. Li, M. Shoeybi, and B. Catanzaro. InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining, May 2024. arXiv:2310.07713 [cs].

C. Wang, J. Duan, C. Xiao, E. Kim, M. Stamm, and K. Xu. Semantic Adversarial Attacks via Diffusion Models, Sept. 2023. arXiv:2309.07398 [cs].

C. Wang, Y. Liu, B. Bi, D. Zhang, Z.-Z. Li, Y. Ma, Y. He, S. Yu, X. Li, J. Fang, J. Zhang, and B. Hooi. Safety in Large Reasoning Models: A Survey, May 2025. arXiv:2504.17704 [cs].

H. Wang, H. Li, M. Huang, and L. Sha. ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings, June 2024. arXiv:2402.16006 [cs].

J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, Jan. 2023. arXiv:2201.11903 [cs].

S. Wiegreffe, O. Tafjord, Y. Belinkov, H. Hajishirzi, and A. Sabharwal. Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions, Mar. 2025. arXiv:2407.15018 [cs].

S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann. BloombergGPT: A Large Language Model for Finance, Dec. 2023. arXiv:2303.17564 [cs, q-fin].

A. Wächter and L. T. Biegler. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical programming , 106:25-57, 2006. Publisher: Springer.

F. Xu, Q. Hao, Z. Zong, J. Wang, Y. Zhang, J. Wang, X. Lan, J. Gong, T. Ouyang, F. Meng, C. Shao, Y. Yan, Q. Yang, Y. Song, S. Ren, X. Hu, Y. Li, J. Feng, C. Gao, and Y. Li. Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models, Jan. 2025. arXiv:2501.09686 [cs].

S. Xu, Z. Wu, H. Zhao, P. Shu, Z. Liu, W. Liao, S. Li, A. Sikora, T. Liu, and X. Li. Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis, Feb. 2024. arXiv:2402.11398 [cs].

X. Yang, W. Liu, D. Tao, and W. Liu. BESA: BERT-based Simulated Annealing for Adversarial Text Attacks. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence , pages 3293-3299, Montreal, Canada, Aug. 2021. International Joint Conferences on Artificial Intelligence Organization.

J.-Y. Yao, K.-P. Ning, Z.-H. Liu, M.-N. Ning, Y.-Y. Liu, and L. Yuan. LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples, Aug. 2024. arXiv:2310.01469 [cs].

X. Yin, X. Zhang, J. Ruan, and X. Wan. Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2270-2286, Bangkok, Thailand, 2024. Association for Computational Linguistics.

Z.-X. Yong, C. Menghini, and S. H. Bach. Low-Resource Languages Jailbreak GPT-4, Jan. 2024. arXiv:2310.02446 [cs].

J. Yu, X. Lin, Z. Yu, and X. Xing. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts, June 2024. arXiv:2309.10253 [cs].

Y. Zhang, L. Cui, W. Bi, and S. Shi. Alleviating Hallucinations of Large Language Models through Induced Hallucinations, Mar. 2024. arXiv:2312.15710 [cs].

Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, L. Wang, A. T. Luu, W. Bi, F. Shi, and S. Shi. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models, Sept. 2023. arXiv:2309.01219 [cs].

Y. Zhao, W. Zheng, T. Cai, X. L. Do, K. Kawaguchi, A. Goyal, and M. Shieh. Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Sampling, Nov. 2024. arXiv:2403.01251 [cs].

Y. Zhong, X. Liu, D. Zhai, J. Jiang, and X. Ji. Shadows can be Dangerous: Stealthy and Effective Physical-world Adversarial Attack by Natural Phenomenon. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 15324-15333, New Orleans, LA, USA, June 2022. IEEE.

S. Zhu, R. Zhang, B. An, G. Wu, J. Barrow, Z. Wang, F. Huang, A. Nenkova, and T. Sun. AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models, Dec. 2023. arXiv:2310.15140 [cs].

A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson. Universal and Transferable Adversarial Attacks on Aligned Language Models, Dec. 2023. arXiv:2307.15043 [cs].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our abstract and introduction carefully describe the background, motivation, research scope, and key contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Appendix §O, we discuss the current limitations and possible future directions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper is primarily empirical in nature; rigorous theoretical analysis and formal proofs are beyond its scope. However, we still provide a detailed formulation of the constrained optimization problem and describe our algorithmic approach for solving it in §2.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed descriptions of our algorithm, experimental setup, LLM prompting techniques, and hyperparameter selection in §2, §3, and Appendix §F, §G, §H, §I, §J, and §L.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The complete function demo code, along with detailed instructions and the dataset used in our experiments, is available at https://github.com/Buyun-Liang/ SECA .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Detailed experimental settings are described in §3.1 and Appendix §L.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes, all reported means, and standard deviations when applicable, are computed using 10,000 bootstrap samples.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, the computational resources are detailed in Appendix §L.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We ensure that all research presented in this paper aligns with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss both the potential positive and negative societal impacts of this work in Appendix §O.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We include usage guidelines at https://github.com/Buyun-Liang/SECA . and explicitly communicate the dual-use nature of SECA to discourage malicious applications. Additionally, we emphasize the potential negative societal impacts in Appendix §O.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have ensured proper citation for all open-source assets used in this work, including code, datasets, and pretrained models.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide detailed instruction prompts for the key LLM-based components of our method in Appendix §F, §G, §H, and §I. In addition, the full functional demo code, along with detailed documentation, is available at https://github.com/Buyun-Liang/SECA .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: Table 3 compares the performance of LLM-based evaluators with two human annotators. Both annotators are volunteers with at least an undergraduate level of education and are well-acquainted with the experimental setup.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our experiments did not involve any physical or psychological risks to participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: LLMs act as important components in our framework design: the proposer, victim models, feasibility checker, and hallucination evaluator.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Structure of Appendix

Appendix §B demonstrates the SECA-generated adversarial prompt and the LLM response.

Appendix §C provides the lists of all abbreviations used in the paper.

Appendix §D presents illustrative examples of various jailbreaking methods applied to open-ended MCQA tasks.

Appendix §E discusses related work for faithful and factual LLMs and constrained deep learning.

Appendix §F, §G, §H, and §I show the instruction templates for MMLU-style query, semanticequivalence proposer, feasibility checker, and the hallucination evaluator correspondingly.

Appendix §J lists the model versions of both open-source and commercial LLMs used in the paper.

Appendix §K shows examples of SECA-rephrased prompts against the original MMLU question.

Appendix §L details additional experimental setup information.

Appendix §M provides the full ASR@30/10/1 results for SECA and raw prompts, along with the objective improvements achieved by SECA.

Appendix §O discusses the paper's limitations, societal impacts, and directions for future work.

B Example of SECA Attack Adversarial Prompt

Figure 5 shows a successful SECA attack in elementary mathematics, where the adversarial prompt causes the model to select an incorrect answer and generate a hallucinated explanation.

Figure 5: Illustration of a factuality hallucination induced by a SECA adversarial prompt. The top two green boxes show the full attack prompt based on the original MMLU question in elementary mathematics, followed by the faithful and factual response from the target LLM. The bottom two blue boxes present a SECA-generated adversarial variant of the original prompt, with edits highlighted in red, and the corresponding target LLM explanation, which includes red-highlighted hallucinated content. In this example, the model selects the incorrect choice ('B') and generates a hallucinated explanation, showcasing a factuality hallucination.

C List of Abbreviations

To ensure clarity and consistency of the notations, this section presents a summary of all abbreviations used throughout the paper. Table 4 lists the baseline and evaluation abbreviations. Table 5 maps each evaluated MMLU subject to its corresponding abbreviation.

D Comparison with Jailbreak Attacks

We provide illustrative examples of jailbreaking methods applied to open-ended MCQA tasks:

From Table 6, we observe that the objective of jailbreaking is to bypass safety mechanisms. Arbitrary prompts such as intent-hiding, storytelling, or gibberish are considered acceptable. However, none of the existing jailbreak methods 10 are capable of generating semantically equivalent and coherent prompts. Such constraints are essential for hallucination elicitation, as they allow us to study how hallucinations may arise in realistic scenarios and to evaluate LLM robustness (see §1 and §4).

E Additional Related Work

In addition to §4, this section discusses prior art that is relevant to our proposed framework.

E.1 Faithful & Factual LLMs

Strategies for reducing hallucinations are widely adopted across the stages of model development [86, 20]. Corpus Processing. Early efforts focus on filtering out low-quality data and up-sampling reliable sources to reduce the incidence of false information [68]. Later studies emphasize the importance of sufficient coverage of long-tail knowledge [29] to reduce knowledge gaps. Such philosophy is further adopted to curate verified synthetic data to scale the training set [16]. However, corpus processing does not guarantee that all underrepresented facts are resampled, and over-cleaning increases the risk of discarding useful information. Pre-training & Post-training. Supervised FineTuning (SFT) on vetted dialogue data with Reinforcement Learning from Human Feedback (RLHF) penalization of undesirable responses is commonly applied to enhance the LLM's faithfulness [1]. With large-scale external knowledge bases [23], designing objectives to train retrievers [18, 21] and enforce the LLM reason on the retrieved external documents [70, 40] can curb factual hallucinations. Inference-Time. Hallucination safeguards for runtime deployment mainly address the challenges of reasoning robustness with factual verification. To enhance the faithfulness of Chain-of-Thought (CoT) prompting [74], previous work adopt various kinds of symbolic verifiers [49, 13] to justify the reasoning. An emerging trend is to develop test-time computing [22, 55] that internalizes and scales up the thinking trace. Yet, as we will show in the following sections, such alignment approaches for hallucination mitigation may overfit to specific training distributions, and the target LLMs are still prone to semantically equivalent and coherent queries.

E.2 Constrained Deep Learning

As discussed in §2, generating a semantically equivalent and coherent rephrasing that induces hallucination in LLMs requires solving a highly nonconvex, nonsmooth, and constrained optimization problem. The nonsmoothness arises from nonlinear activation functions within the LLM, while the nonconvexity is an inherent property of deep neural networks.

As shown in Table 1, violating these constraints often leads to trivial or incoherent attacks, making them ineffective for evaluating model robustness. It is therefore essential to solve this problem in a reliable and constraint-aware manner. However, as discussed in prior work on constrained deep learning [36, 35], existing optimization methods face fundamental limitations in this setting. Structured methods like projected (sub)gradient and Frank-Wolfe [65], as well as manifold optimization methods [25, 51, 54, 69, 6] are not directly applicable due to the nonconvex and nonsmooth nature of the constraints. Likewise, the interior-point based [77, 60] and augmented Lagrangian-based solvers [11, 27] are tailored to specific classes of nonconvex, nonsmooth constraints and cannot be extended to the LLM-based constraints used in SECA.

Second, the optimization must be performed under black-box conditions . Many LLMs involved in SECA-such as the target model and semantic equivalence evaluator-are commercial APIs without accessible gradient information. This rules out the use of standard gradient-based solvers, including those developed for constrained deep learning [36, 38, 39]. Even when gradients can be estimated via backpropagation or finite differences, making such approaches prohibitively expensive in practice.

F Example Full Prompt to Target LLMs

Figure 6 shows our exact full prompt template used to query and attack target LLMs on MMLU-style multiple-choice questions.

Figure 6: (Top) Input prompt template used to evaluate target LLMs on MMLU questions. (Bottom) Example prompt for the Elementary Mathematics subject, where the target token y ∗ is set to the incorrect answer choice "B".

G Instruction Prompt for Semantic Equivalence Proposer

Figure 7 shows the instructions used to guide the semantic equivalence proposer in generating semantics-preserving rephrasings.

H Instruction Prompt for Feasibility Checker

Figure 8 shows the instructions used to guide the feasibility checker to constrain the optimization.

I Instruction for Hallucination Evaluator

Figure 9 shows the instructions used to guide the hallucination evaluator for answer judgment.

J LLMVersion

Table 7 lists the abbreviations and corresponding detailed model versions used in this paper. The versions are hyperlinked to their corresponding Huggingface repository or documentation 11 .

K Example SECA Candidate Attack Prompts

Figure 10 demonstrates the progression of candidate prompts generated from the original prompt x 0 .

What is the wild progenitor of maize? (x0) - Which wild species is the ancestor of maize? - What is the wild ancestor identified as the original source of corn? - Which ancient species of plant is recognized as the primary progenitor of maize? - Which ancient plant is recognized as the primary ancestor of modern corn? - What ancient vegetation is acknowledged as the most significant progenitor of contemporary maize? - What ancient species of plant is recognized as the primary ancestor of contemporary maize? - What type of ancient plant is recognized as the progenitor of contemporary maize? - What is the name of the ancient plant species recognized as the progenitor of contemporary corn? - Which historical plant is acknowledged as the ancestor of modern maize?

L Additional Experimental Setups

Computational Resources . All experiments were conducted using four NVIDIA A5000 GPUs, each with 24.5 GB of memory. Running SECA to attack a single target LLM on 100 MMLU samples with the specified hyperparameters takes approximately 8-16 GPU hours.

Perplexity . We concatenate all attack prompts and calculate PPL ( · ) over a sliding window. More technical details of the sliding window design can be found at https://huggingface.co/docs/ transformers/en/perplexity . For the evaluation of coherence, values of v C exceeding 100 typically suggest that the prompt lacks meaningful semantic content.

SECA Setting . For SECA, we set the hyperparameters as follows: M = 3 , N = 3 , max_iteration =30, and termination_threshold = 1.0.

GCG Setting . GCG [90] performs token-level optimization to elicit arbitrary target token sequences from LLMs. We adopt the encapsulated, full-featured implementation provided at https://github. com/GraySwanAI/nanoGCG . Since GCG assumes white-box access and we impose GPU memory constraints across all baselines, only Llama-3-3B/8B and Qwen-2.5-7B are feasible targets for GCG-based attacks. Due to GCG's lower efficiency compared to SECA, we evaluate both GCG and SECA on separate MMLU subsets of 218 samples to reduce computational overhead during the comparison. The hyperparameters used in GCG are: num_steps = 300, search_width = 32, and batch_size = 32. All other unspecified hyperparameters follow the defaults from https: //github.com/GraySwanAI/nanoGCG/tree/v0.3.0-release .

Target LLMs Setting . The key hyperparameters for all target LLMs are: top_p = 1 . 0 and temperature = 1 . 0 . To ensure reproducibility, we set seed = 42 .

M Full Results for Figure 3

We provide detailed results corresponding to Figure 3, extended to different values of the Best-of-K Attack Success Rate with K ∈ { 30 , 10 , 1 } . Table 8 reports the full ASR@30/10/1 results for raw prompts, while Table 9 presents the corresponding results for SECA prompts. Across all settings, SECA consistently outperforms raw prompts when targeting different LLMs. Moreover, increasing the number of trials K naturally improves ASR; however, SECA still yields substantial improvements over raw even under limited trial budgets. Finally, Table 10 shows that SECA significantly increases the objective compared to raw prompts.

N Additional Experiments on GPT-3.5-Turbo and GPT-4

We further provide detailed results for attacking GPT-3.5-Turbo and GPT-4, extended to different values of the Best-of-K Attack Success Rate with K ∈ { 30 , 10 , 1 } . Table 11 reports the full ASR@30/10/1 results for raw prompts, while Table 12 presents the corresponding results for SECA prompts. Across all settings, SECA consistently outperforms raw prompts when targeting different LLMs. Moreover, increasing the number of trials K naturally improves ASR; however, SECA still yields substantial improvements over raw even under limited trial budgets. Finally, Table 13 shows that SECA significantly increases the objective compared to raw prompts.

O Limitations, Societal Impacts, and Future Work

O.1 Societal Impacts

This work reveals a concerning vulnerability in modern large language models (LLMs): even semantically equivalent and linguistically natural rephrasings of benign prompts can elicit hallucinated responses. By demonstrating that factual and faithfulness hallucinations can be elicited from reliable LLMs, SECA highlights risks in deploying LLMs in high-stakes settings such as healthcare, law, finance, and education. While this method can be misused to degrade trustworthiness in LLMs or propagate misinformation, its primary intent is to evaluate hidden failure modes that are easily overlooked by standard benchmarks. We hope that by exposing these subtle vulnerabilities, SECA can help guide the development of more robust and trustworthy LLMs, as well as inform safety evaluations for real-world deployment. Nevertheless, careful access controls, responsible disclosure, and mitigation strategies are essential to prevent the malicious use of such attacks.

O.2 Limitations and Future Work

Although SECA completes each attack within approximately two minutes, this runtime may be insufficient for large-scale red teaming applications. Future work could explore incorporating zerothorder gradient estimation techniques (e.g., finite difference methods) to more efficiently traverse the constrained prompt space and accelerate convergence. Such improvements would enable broader deployment of SECA for stress-testing LLMs at scale.

This paper focuses on hallucination elicitation in the open-ended multiple-choice question answering (MCQA) setting, where hallucinations are characterized by incorrect answer selection followed by flawed reasoning. In future work, we aim to extend SECA to more free-form generation settings, such as factuality errors in long-form answers or hallucinated entities in summarization.

SECA currently optimizes for a specific incorrect target token, making it a targeted attack. Future directions include developing untargeted versions of SECA by incorporating hallucination evaluator outputs directly into the objective function. This would allow the framework to maximize hallucination likelihood without relying on predefined target responses, broadening its applicability and reducing reliance on prior knowledge of model behavior.

SECA focuses on attacking non-reasoning models, similar to many existing hallucinationelicitation [33, 64, 75] and MCQA studies [63, 59, 4]. An interesting direction for future work is to extend this line of research to reasoning models [12, 78, 46, 44, 72], i.e., models that generate reasoning steps before answering. Such an extension would require redefining the objective function, as the unpredictable length of reasoning chains makes the answer token position more difficult to locate.

Peer reviews:
comments: SUMMARY:
This paper investigates the problem of generating Semantically Equivalent and Coherent prompts that elicit hallucinations from large language models. The proposed approach leverages one LLM to generate diverse rephrasings of a given prompt and another LLM to verify their feasibility. Among the feasible rephrasings, the method selects the most adversarial one. Experimental results on multiple-choice benchmarks demonstrate that this approach achieves a high success rate while maintaining low constraint violation rates.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The problem formulation of finding semantically equivalent and coherent adversarial prompts is interesting. The authors did a good job at connecting it to traditional adversarial attack formulation in computer vision.
2. The human study on whether LLM can check semantic equivalence is important. 

Weakness: The main weakness is the over-simplified experimental setting.
1. The experiment is only on multi-choice problems, whereas in practice open-ended problems are more common.
2. The prompt template set up can be problematic: In the example in figure 5 (Appendix B), it seems that the LLM is prompted to first give an answer (from "A", "B", "C", "D") and then explain the reasoning. However, in practice, it is more reasonable to first reason and then give the final answer. It is unclear whether the method can still work under this more practical setting.

QUESTIONS:
Does the method still work in a setting where the LLM is prompted to first reason through the question before selecting a multiple-choice answer?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-06-24

comments: SUMMARY:
This paper proposes a novel framework for eliciting hallucinations in LLMs using semantically equivalent and linguistically coherent prompt rephrasings. It formulates hallucination elicitation as a constrained optimization problem and uses two LLMs: a proposer and a feasibility checker. Experiments on the MMLU benchmark across multiple LLMs demonstrate that the proposed method achieves higher attack success rates while preserving naturalness.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The proposed method is simple and effective. The idea of using two LLMs to generate and check the adversarial prompts is novel and insightful.
2. The authors conduct comprehensive experiments including ablation studies and human evaluations to show the effectiveness of the proposed method. 
3. The paper is well-written and well-organized. The method flows naturally and clearly, making it easy to reproduce.

Weaknesses:
1. The method is only evaluated in the multiple-choice QA setting (MMLU). It remains unclear how well SECA generalizes to open-ended generation tasks such as summarization or dialogue.
2. While SECA is effective in eliciting hallucinations, the paper lacks analysis on whether models trained to resist SECA prompts would generalize to other types of realistic adversarial inputs.

QUESTIONS:
Can the proposed method be applied to other types attacks against LLMs (e.g. adversarial attacks, backdoor attacks)?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel framework called SECA for creating coherent hallucination attacks on large language models (LLMs). By utilizing a feasibility checker and proposer LLM, the framework is able to find the most adversarial attack, while retaining the semantic content of the original attack and its coherency. Authors evaluate compared to two other methods and also provide human evaluation for both feasibility and hallucination evaluation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The semantic coherency in adversarial attacks is important as it can shed more light into the reasoning capabilities of LLMs. Their evaluation through this type of attack can also catch spurious semantic incoherencies, which could in turn show that LLMs may be grammatically and syntactically driven, instead of semantically.\n\n* SECA reaches a strong attack success rate with only ≈30 proposer-checker loops (~2 minutes per sample), while GCG needs hundreds of gradient steps and still under-performs (Table 2 & Appendix K/L). The monotonic objective curves in Fig. 4a further show consistent, fast convergence. This efficiency matters if the framework is to be used for routine red-teaming or large-scale robustness audits.\n\nWeaknesses:\n* The evaluation requires more explanation, as the only reported metrics are new and evaluated only against Raw and GCG, omitting stronger jailbreakers such as Tree-of-Thought or Prompt-Inversion  (Table 1, Section 4.2, Figure 3).\n* The LLMs evaluated on the paper are relatively small and are more prone to hallucinations. Experiments on larger models, especially those with explicit reasoning traces, would better showcase the framework's ability identifying true semantic adversarial attacks. Reasoning is also more important as a few reasoning steps could lead the model into identifying the hallucinations.\n* A hallucination mitigation analysis on the proposing LLMs would be beneficial as well. Authors should explain how they mitigate the language hallucinations on the auxiliary models, in order to prevent a non-intended hallucinatory sample that is not semantically coherent but still provides an adversarial attack.\n\nQUESTIONS:\n* Could authors explain if there any metrics relevant to the use or if there any other experiments comparing with different methods?\n* Are there any experiments done with larger commerical models or Large Reasoning Models (LRMs), in order to showcase the framework's performance scaling with parameter size or reasoning steps, even with chain-of-thought methods?\n* How does the framework mitigate hallucinations created by the auxiliary LLMs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SECA, a black-box attack framework that provokes hallucinations in large language models while keeping the adversarial prompts both semantically faithful and readable. SECA runs a heuristic tree search over prompt space, guided by two auxiliary LLMs:\n\n1. Proposer LLM: generates candidate paraphrases that preserve the original meaning, which become the tree’s nodes.\n2. Feasibility Checker LLM: filters those candidates, retaining only the ones that remain semantically equivalent and linguistically coherent.\n\nThe authors test SECA on the MMLU benchmark against the Greedy Coordinate Gradient (GCG) attack. SECA consistently delivers higher attack-success rates across several target models. They also verify that the Checker’s judgments of equivalence and coherence align closely with human annotations, confirming its reliability in the pipeline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. This paper proposes to focus on generating human-like instructions that can elicit LLM hallucinations, which addresses a practical gap between existing jailbreaking work and real-world usage scenarios.\n\n2. The paper proposes a training-free heuristic tree search method with a Proposer and a Checker LLM, which is simple to deploy and test.\n\nWeaknesses\n1. In line 67, the authors claim that they compare seven frontier LLM models, yet in Table 2 reports results for only three relatively small models (average < 14 B).  Figure 3 illustrates improvements on seven models, but the exact numbers are not provided. I suggest the authors consider adding a table with full Attack Success Rates (ASR) for all seven models tested on their benchmark. \n\n2. The authors take MMLU as their benchmark, which is less commonly used in LLM adversarial attacking domain. MMLU as an multiple-choice question answering (MCQA) setting, is easier than open-ended answer benchmarks, such as AdvBench[1] and JBB-Behaviors[2]. I encourage the authors to test their method at least on AdvBench to demonstrate broader effectiveness. \n\n3. The authors only compare their method with Greedy Coordinate Gradient (GCG). It would benefit from including comparison with other recent white-box methods—Probe Sampling [3], COLD-Attack [4], and black-box method AdvPrompter [5]—which have shown strong performance and efficiency compared with GCG. \n\n\n[1]Zou, Andy, et al. \"Universal and transferable adversarial attacks on aligned language models.\" arXiv preprint arXiv:2307.15043 (2023).\n[2]Chao, Patrick, et al. \"Jailbreakbench: An open robustness benchmark for jailbreaking large language models.\" arXiv preprint arXiv:2404.01318 (2024).\n[3]Zhao, Yiran, et al. \"Accelerating greedy coordinate gradient and general prompt optimization via probe sampling.\" Advances in Neural Information Processing Systems 37 (2024): 53710-53731.\n[4]Guo, Xingang, et al. \"Cold-attack: Jailbreaking llms with stealthiness and controllability.\" arXiv preprint arXiv:2402.08679 (2024).\n[5]Paulus, Anselm, et al. \"Advprompter: Fast adaptive adversarial prompting for llms.\" arXiv preprint arXiv:2404.16873 (2024).\n\nQUESTIONS:\n* How does SECA perform on larger models such as GPT-3.5-turbo or GPT-4? How will reasoning models such as GPT-o4-mini and DeepSeek-R1 behave under your method's attack? Any preliminary figures would be insightful.\n* What is the efficiency of your method? Roughly how many candidate prompts can SECA generate and evaluate per GPU-minute on an Nvidia A5000? \n* How does SECA perform at stricter query budgets, such as ASR@1 and ASR@10? An ablation or scaling curve would be very illuminating.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}