Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models

## Abstract

Abstract The security issue of large language models (LLMs) has gained wide attention recently, with various defense mechanisms developed to prevent harmful output, among which safeguards based on text embedding models serve as a fundamental defense.
Through testing, we discover that the output distribution of text embedding models is severely biased with a large mean.
Inspired by this observation, we propose novel, efficient methods to search for universal magic words that attack text embedding models.
Universal magic words as suffixes can shift the embedding of any text towards the bias direction, thus manipulating the similarity of any text pair and misleading safeguards.
Attackers can jailbreak the safeguards by appending magic words to user prompts and requiring LLMs to end answers with magic words.
Experiments show that magic word attacks significantly degrade safeguard performance on JailbreakBench, cause real-world chatbots to produce harmful outputs in full-pipeline attacks, and generalize across input/output texts, models, and languages.
To eradicate this security risk, we also propose defense methods against such attacks, which can correct the bias of text embeddings and improve downstream performance in a train-free manner.

### 1 Introduction

Recently, large language models (LLMs) have been widely applied in the industry, such as chat systems [1] and search engines [2].
However, LLMs can be maliciously exploited to extract harmful output, making LLM security an important research topic.

In this topic, it is of great significance to discover security vulnerabilities of text embedding models and propose corresponding defense methods.
Current LLM security strategies include alignment [3] and safeguards [4].
Lightweight text classifiers based on text embedding models [5] can be used as safeguards to judge whether the input and output of LLMs are harmful.
This method can serve as a foundational line of defense because it is low-cost while maintaining the performance of LLMs.
In addition, text embedding models are also used to enhance modern search engines [2].
Therefore, the robustness of text embedding models affects the security of both LLMs and search engines.

Attacking LLMs’ safeguards is challenging because the output of LLMs is unknown, the safeguards are black boxes, and the token space is vast and discrete.
This results in the following limitations of existing attack methods on text embedding models: 1) Case-by-case attack methods require access to LLMs’ output before safeguards, which is unrealistic for online dialogue systems; 2) White-box attack methods require the gradients of text embedding models, which are also unrealistic; 3) Brute-force search for prompt perturbations requires traversing a massive token space, leading to high time costs.

To address these challenges, we propose an innovative approach to attack LLMs’ safeguards based on text embedding models: to find universal “magic words” (i.e., adversarial suffixes) that would increase or decrease the embedding similarity between any pair of texts so as to mislead the safeguards in classifying within the text embedding space.

Figure: Figure 1: The distribution of cosine similarity between text embedding $e(s)$ of text $s$ with normalized mean embedding $e^{*}$ of all text, tested on various text embedding models.
Refer to caption: x1.png

This task is feasible based on the following observation.
We tested various text embedding models and found that the cosine similarities between text embeddings and their mean (normalized) concentrate near a significant positive value, as shown in [Fig. 1](https://arxiv.org/html/2501.18280v3#S1.F1).
In other words, text embeddings do not distribute uniformly on a high-dimensional sphere $S^{d-1}$(since they are normalized); instead, they concentrate in a band on the sphere, as illustrated in [Fig. 2](https://arxiv.org/html/2501.18280v3#S3.F2).
The direction of distribution bias is similar to all text embeddings, while the opposite direction is dissimilar to all embeddings.
This implies that if we can find suffixes that push any text towards this bias direction, we can increase the similarity of any text with other texts.
Similarly, one could also try to find suffixes that reduce text similarity. We refer to these words as universal magic words since they can mislead safeguards on any text by manipulating text similarity.

We estimate the bias direction with the mean and the principal singular vector of text embeddings (see [Sec. 3.1](https://arxiv.org/html/2501.18280v3#S3.SS1)).
Actual tests and theoretical analysis show that the two methods yield the same results.

Based on the identified bias direction, we use the following methods to find universal magic words (see [Sec. 3.2](https://arxiv.org/html/2501.18280v3#S3.SS2)).
[Alg. 1](https://arxiv.org/html/2501.18280v3#alg1): brute-force search without leveraging the bias direction, used as a baseline;
[Alg. 2](https://arxiv.org/html/2501.18280v3#alg2) (black-box): find words whose text embeddings are as similar/dissimilar as possible to the bias direction;
[Alg. 3](https://arxiv.org/html/2501.18280v3#alg3) (white-box): find universal suffixes that push any text embedding closer to the bias direction or far away from its original position.
[Alg. 3](https://arxiv.org/html/2501.18280v3#alg3) uses gradients to solve this problem in only one epoch.
Experiments show that all three methods can find the best magic words, but [Algs. 2](https://arxiv.org/html/2501.18280v3#alg2) and [3](https://arxiv.org/html/2501.18280v3#alg3) are far more efficient than [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1). Additionally, only [Alg. 3](https://arxiv.org/html/2501.18280v3#alg3) can search for multi-token magic words.

The universal magic words can be abused to attack safeguards in LLM security systems. As shown in [Fig. 3](https://arxiv.org/html/2501.18280v3#S3.F3), the safeguards will fail to detect harmful content by appending magic words to the input and output of LLMs.

##### Contributions.

The contribution of this paper can be summarized as follows:

- •
We discover that the output distribution of text embedding models is uneven and the relationship between this property and universal magic words;
- •
We propose novel methods for finding universal magic words, which are efficient and capable of searching for multi-token magic words;
- •
We demonstrate that those universal magic words are able to jailbreak LLMs’ safeguards and generalize across input/output texts, models, and languages (see experiments in [Secs. 4.3](https://arxiv.org/html/2501.18280v3#S4.SS3), [4.4](https://arxiv.org/html/2501.18280v3#S4.SS4) and [4.5](https://arxiv.org/html/2501.18280v3#S4.SS5)).
- •
We propose defense methods against such attacks by correcting the uneven embedding distribution.

### 2 Related Work

#### 2.1 Defense Methods for LLMs

Alignment involves training LLMs to align with human values [6, 7, 3].
This method is widely used because it does not introduce additional computational overhead during inference.
Due to the competition between assisting users and aligning values, as well as the limited domain of safety training [8], such methods are vulnerable to adversarial attacks [9, 10]. This has forced people to develop additional security measures.

Safeguards are the additional measures on the input or output of LLMs to avoid harmful responses.

On the input side, there are several guard measures:
1) Detecting suspicious patterns [11, 12], which tends to yield false positives;
2) Reminding LLMs to align values with system prompts [13, 14, 15], which can be canceled by the user prompt “ignore previous instructions” [16];
3) Perturbing the user’s prompt into multiple versions before feeding it to the LLM to detect harmful requests [17, 18], which is costly;
4) Classifying whether the prompt is harmful with a model [5].

On the output side, several detection methods for LLMs’ harmful responses serve as the last line of defense in LLM security systems:
1) rule-based matching, with the same drawbacks as it is on the input side;
2) another LLM to answer whether the output is harmful [19, 20, 21], which doubles the cost;
3) alternatively, text classifiers to do this [22, 5, 23], which is more cost-effective.

#### 2.2 Attack Methods for LLMs

Templates jailbreak LLMs with universal magic words effective for various prompts, some even transferable across LLMs.
Manual templates are heuristically designed, including explicit templates (e.g., instructing LLMs to
“ignore previous instructions” [16],
“Start with ‘Absolutely! Here’s”’ [24] or
“Do anything now” [24]) and
implicit templates (e.g.,
role-playing [25, 26],
storytelling [27] and
virtual scenarios [28, 29, 30, 31]).
Automatic templates are optimized by
gradient descent (black-box) [32, 9, 33],
random search (white-box) [34, 35], or
generative models [36] to find adversarial prefixes and suffixes for user prompts.
These prefixes and suffixes could be individual words or sentences [9], and comprehensible [36] or not [34].

Rewriting attacks language models at several levels, including
character-level (e.g., misspelling [37]),
word-level (e.g., synonyms [38]),
segment-level (e.g., assigning variable names to segmented harmful text [39, 29]),
prompt-level (e.g., rewriting prompts with an LLM [10, 40, 41, 42]),
language-level (e.g., translating into a language that lacks LLM safety [43]), and
encoding-level (e.g., encoding harmful text into ASCII, Morse code [44] or Base64 [45]).
Through optimization algorithms, attackers can automatically find the most effective rewrites to bypass the LLM’s safeguards.

The methods above are all focused on attacking the LLM itself, while research on attacking safeguards is still in its early stages.
A magic word “lucrarea” was discovered by the champion of a Kaggle competition on attacking LLMs  [46], through trying the tokens near 
 in the token embedding space.
We find many more magic words, including “lucrarea”, with our novel algorithms and give a more accurate and systematic explanation of why it works.
Similar to our method, PRP [47] attacks output guards by injecting magic words into LLMs’ responses. The distinctions between our work and PRP are:
1) we attack guards based on text embedding models, which are more lightweight and cost-effective than LLM-based guards in PRP;
2) we discovered the uneven distribution of text embeddings, which allows us to design algorithms to search for magic words more efficiently.

### 3 Method

##### Notation:

1) Let $s_{1}$ and $s_{2}$ be two text strings, and let $r$ be a positive integer.
The operation $s_{1}+s_{2}$ denotes the concatenation of $s_{1}$ and $s_{2}$, and $r*s_{2}$ denotes
the string $s_{2}$ repeated $r$ times.
2) For example, if $s_{1}=``he"$, $s_{2}=``llo"$, then $s_{1}+s_{2}=``hello"$ and $s_{1}+2*s_{2}=``hellollo"$.
Denote the text embedding of text string $s$ by $e(s)$ and its dimension by $d$.
$e(s)$ is normalized to a unit vector, hence $e(s)\in S^{d-1}$.
The text embedding $e(s)$ of $s$ is computed as $e(s)=\mathbf{e}(\boldsymbol{s}),\boldsymbol{s}=E^{\top}\tau(s)$.
Here, $\boldsymbol{s}\in\mathbb{R}^{h\times l}$ denotes the representation of $s$ in the token embedding space,
which is mapped to a text embedding by $\mathbf{e}$.
Moreover, $\tau$ is a tokenizer that splits $s$ into $l$ tokens, outputting
$\tau(s)\in\{0,1\}^{T\times l}$ where the columns are one-hot.
$\mathcal{T}=\{t_{i}\}_{i}$ is the token vocabulary, with size $|\mathcal{T}|=T$.
$E\in\mathbb{R}^{T\times h}$ denotes the token embeddings of all tokens, with dimension $h$.
3) The cosine similarity between text $s_{1}$ and $s_{2}$ is defined as
$\cos\theta(s_{1},s_{2}):=e(s_{1})^{\top}e(s_{2})$.

This paper aims to find all possible universal magic words, which can be formulated as follows.

###### Assumption 3.1 .

There exists a word $w^{+}$ satisfying that
$\cos\theta(s_{1}+w^{+},s_{2})\geq\cos\theta_{*},\quad\forall s_{1},s_{2}$,
where $\cos\theta_{*}$ is close to $1$. We refer to $w^{+}$ as a positive universal magic word for the text embedding model $e$, which can force any pair of texts to be similar enough in the text embedding space.

###### Assumption 3.1 .

#### 3.1 Description of the Uneven Direction

To describe the unevenness of the text embedding distribution, we represent the bias direction of the distribution by the normalized mean of text embeddings $e^{*}$ or the principal singular vector $v^{*}$ of the text embedding matrix.
We prove that any text appended by a positive universal magic word $w^{+}$ will be close to $e^{*}$ (or $v^{*}$).
This serves as the guiding principle for searching for magic words in [Sec. 3.2](https://arxiv.org/html/2501.18280v3#S3.SS2).

We denote the mean of text embeddings as $\bar{e}=\frac{1}{|\mathcal{S}|}\sum_{j}e(s_{j})$ and the normalized mean as $e^{*}=\frac{\bar{e}}{\|\bar{e}\|_{2}}$,
where $\mathcal{S}=\{s_{j}\}_{j}$ is the set of all possible texts.

The following proposition shows that any text with a magic word will be embedded close to $e^{*}$.

###### Proposition 3.2 .

Under Assumption [3.1](https://arxiv.org/html/2501.18280v3#S3.Thmtheorem1), a positive universal magic word $w^{+}$ must satisfy

$$ $\cos\theta(e(s+w^{+}),e^{*})\geq\sqrt{1-\tan^{2}\theta_{*}},\quad\forall{s}\in \mathcal{S}.$ $$

Denote the text embedding matrix of $\mathcal{S}$ as
$X\in\mathbb{R}^{|\mathcal{S}|\times d}$, where the $i$-th row of $X$ is $e(s_{i})^{\top}$. Let $v^{*}$ be the principal right singular vector of $X$ corresponding to the largest singular value.

The following proposition shows that any text with a magic word will be embedded close to $v^{*}$.

###### Proposition 3.3 .

Under Assumption [3.1](https://arxiv.org/html/2501.18280v3#S3.Thmtheorem1), a positive universal magic word $w^{+}$ must satisfy

$$ $\cos\theta(e(s+w^{+}),v^{*})\geq\sqrt{1-\tan^{2}\theta_{*}},\quad\forall{s}\in \mathcal{S}.$ $$

See [Appendix B](https://arxiv.org/html/2501.18280v3#A2) for the proof of the two propositions.
In the experiments (see [Sec. 4.1](https://arxiv.org/html/2501.18280v3#S4.SS1)), we found that $e^{*}$ and $v^{*}$ are almost identical, so we will only refer to $e^{*}$ in the subsequent sections.

###### Proposition 3.2 .

###### Proposition 3.3 .

#### 3.2 Searching for Universal Magic Words

Based on the observations in [Sec. 3.1](https://arxiv.org/html/2501.18280v3#S3.SS1), we boldly presume the existence of universal magic words.
When used as a suffix, universal magic words could make any text more similar or dissimilar to other texts in the embedding space.

We refer to the words that increase the text similarity as positive magic words and those that decrease the text similarity as negative magic words, as shown in [Fig. 2](https://arxiv.org/html/2501.18280v3#S3.F2).

Figure: Figure 2: Text embeddings concentrate in a band on the sphere $S^{d-1}$. Positive magic words can push them towards the normalized mean $e^{*}$. Negative magic words can pull them away from their original position.
Refer to caption: x2.png

##### Brute-Force Method

The simplest method to find magic words is a brute-force search, shown in [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1).
This method directly calculates the similarity score of all tokens in the vocabulary set and finds the top-$k_{0}$ magic words. This method does not rely on the bias direction.

For each token $t_{i}$ in the token vocabulary set $\mathcal{T}=\{t_{i}\}_{i}$, we define the positive similarity score as

$$ $\displaystyle c_{i}^{+}$ $\displaystyle=\max_{1\leq r\leq 16}\frac{1}{S^{2}}\sum_{j,k}\cos\theta(s_{j}+r *t_{i},s_{k})$ (1) $\displaystyle=\max_{1\leq r\leq 16}\frac{1}{S}\sum_{j}\cos\theta(s_{j}+r*t_{i} ,e^{*})$ (2) $$

Tokens with higher positive scores are more effective as positive magic words.
$r$ represents the repetition count.
Repeating the magic word usually amplifies its effect.
However, we limit $r$ to a maximum of 16 to avoid completely distorting the text.

Finding negative magic words requires more data.
Specifically, in addition to the text $s_{j}$, we also need another piece of text $s^{\prime}_{j}$ that is semantically similar to $s_{j}$ but phrased differently.
This is because the effect of a negative magic word is to make synonymous text no longer synonymous.
Now the set of text pairs is in the form $\tilde{\mathcal{S}}=\{(s_{j},s^{\prime}_{j})\}_{j}$ with $\cos\theta(s_{j},s^{\prime}_{j})$ close to 1.
We define the negative similarity score of $t_{i}$ as

$$ $c_{i}^{-}=\min_{1\leq r\leq 16}\frac{1}{S}\sum_{j}\cos\theta(s_{j}+r*t_{i},s^{ \prime}_{j}).$ (3) $$

The lower negative similarity score indicates the greater effectiveness of magic words in making synonymous text dissimilar.

Figure: Algorithm 1 Brute-Force Method

Figure: Algorithm 2 Context-Free Method

##### Context-Free Method

As demonstrated previously, all text embeddings tend to be close to $e^{*}$ and far from $-e^{*}$.
Intuitively, tokens whose text embeddings have the same direction as $e^{*}$ are likely to be positive magic words, and vice versa.
Specifically, for a given $t_{i}\in\mathcal{T}$, we select the top-$k$ and bottom-$k$ tokens as candidates for positive and negative magic words based on the following score

$$ $c_{i}=e(r*t_{i})^{\top}e^{*},$ (4) $$

where $r$ denotes the repetition count, set between 3 and 5. After this raw selection, we perform a refined selection from the candidates using [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1).
This method is formulated in [Alg. 2](https://arxiv.org/html/2501.18280v3#alg2).

##### Gradient-Based Method

The above two methods are not able to search for multi-token magic words and do not leverage first-order information.
What if we can access all the model parameters (white-box setting) and wish to leverage gradients?
Let’s formulate the problem more specifically.

The positive magic word we aim to find (denoted as $w$, consisting of $m$ tokens) maximizes the following objective
$\textrm{argmax}_{w}\sum_{j}\cos\theta(s_{j}+w,e^{*})$.

Unlike adversarial attacks in computer vision, the vocabulary’s discreteness introduces significant optimization challenges.
To address this, we split the optimization into two steps.
In the first step, we search for the optimal token embeddings $\boldsymbol{t}^{*}\in\mathbb{R}^{h\times m}$ by solving

$$ $\boldsymbol{t^{*}}=\textrm{argmax}_{\boldsymbol{t}}\sum\nolimits_{j}\mathbf{e} ([\boldsymbol{s}_{j},\boldsymbol{t}])^{\top}e^{*}.$ (5) $$

In the second step, we identify the token in each position whose embedding is closest to the optimal.

Assuming that $\mathbf{e}([\boldsymbol{s},\boldsymbol{t}])$ is close to $\mathbf{e}([\boldsymbol{s},\boldsymbol{0}])$,
[Eq. 5](https://arxiv.org/html/2501.18280v3#S3.E5) can be approximated by a first-order expansion as
$\textrm{argmax}_{\boldsymbol{t}}\sum\nolimits_{j}\left(\mathbf{e}([\boldsymbol
{s}_{j},\boldsymbol{0}])+J(\boldsymbol{s}_{j})\boldsymbol{t}\right)^{\top}e^{*
}=\textrm{argmax}_{\boldsymbol{t}}\boldsymbol{t}^{\top}\big{(}\sum\nolimits_{j
}J(\boldsymbol{s}_{j})\big{)}^{\top}e^{*}$,
where
$J(\boldsymbol{s}_{j}):=\partial_{\boldsymbol{t}}\mathbf{e}([\boldsymbol{s}_{j}
,\boldsymbol{t}])$
denotes the Jacobian of the model $\mathbf{e}$ at $\boldsymbol{s}_{j}$.
The solution to the above problem is
$\boldsymbol{t}^{*}\propto\big{(}\sum_{j}J(\boldsymbol{s}_{j})\big{)}^{\top}e^{*}$.

Interestingly, this $\boldsymbol{t}^{*}$ is exactly the gradient of the following objective function $L^{+}=\sum_{j}\cos\theta(s_{j}+t,e^{*})$ with respect to $\boldsymbol{t}$.
In other words, our method performs gradient ascent on $L^{+}$ in just one epoch.
A similar conclusion also holds for negative magic words with the following objective function $L^{-}=\sum_{j}\cos\theta(s_{j}+t,s^{\prime}_{j})$.

This leads to the algorithm described in [Alg. 3](https://arxiv.org/html/2501.18280v3#alg3).
Like [Alg. 2](https://arxiv.org/html/2501.18280v3#alg2), we first obtain $k$ candidates with the method above and then use [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1) to identify the best $k_{0}$ magic words.

Figure: Algorithm 3 Gradient-Based Method

**Table 1: Comparing different methods**
| Methods | Alg. [1](https://arxiv.org/html/2501.18280v3#alg1) | Alg. [2](https://arxiv.org/html/2501.18280v3#alg2) | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) |
| --- | --- | --- | --- |
| Speed | Slow | Fast | Fast |
| White/Black Box | Black | Black | White |
| Multi-token | No | No | Yes |

As a summary of this section, [Table 1](https://arxiv.org/html/2501.18280v3#S3.T1) compares the three methods in terms of speed, scenario (black-box/white-box), and their ability to search for multi-token magic words.

#### 3.3 Attacking LLMs’ Safeguard

As shown in [Fig. 3](https://arxiv.org/html/2501.18280v3#S3.F3), we can append magic words to the prompt to attack the input guard of LLMs directly and require the LLM to end answers with magic words to attack the output guard indirectly.

Figure: Figure 3: Pipeline to attack the safeguard of LLMs. The input guard is attacked directly by appending universal magic words to user prompts, and the output guard is indirectly attacked by requiring LLMs to append universal magic words to their output.
Refer to caption: x3.png

This method works by moving text embedding to where the safeguard fails. As shown in [Fig. 2](https://arxiv.org/html/2501.18280v3#S3.F2), the data manifold in text embedding space is a band on the sphere.
Positive magic words can push the text embedding towards $e^{*}$, i.e., along the normal direction of the manifold, and safeguards fail to work properly outside the manifold due to the lack of training data.
Negative magic words can push the embedding of a harmful text far away from its original region of harmful semantics, leading to misclassification.

Besides jailbreaking the safeguard of LLMs, universal magic words may also be used to manipulate search rankings. Since most modern search engines are enhanced by text embedding models [2], abusers can increase the embedding similarity between their entries with any queries by inserting magic words into their entries.

### 4 Experiments

We tested our method on several state-of-the-art models from the MTEB text embedding benchmark [48], including sentence-t5-base [49], nomic-embed-text-v1 [50], e5-base-v2 [51], jina-embeddings-v2-base-en [52], gte-Qwen2-7B-instruct [53], SFR-Embedding-Mistral [54], and e5-mistral-7b-instruct [55].
Additionally, considering that LLMs are sometimes used as text embedding models, we also tested Qwen2.5-0.5B [56] with mean pooling.
We used sentence-transformers/simple-wiki [57] as the text dataset $\tilde{\mathcal{S}}=\{(s_{i},s^{\prime}_{i})\}_{i}$, where $s_{i}$ is an English Wikipedia entry, and $s^{\prime}_{i}$ is its simplified variant.
In [Secs. 4.3](https://arxiv.org/html/2501.18280v3#S4.SS3) and [4.5](https://arxiv.org/html/2501.18280v3#S4.SS5), we also evaluated our method on JailbreakBench [58] and non-English dialogues.

#### 4.1 Bias Direction

**Table 2: The overlap between the normalized mean vector $e^{*}$ and the principal singular vector $v^{*}$.**
| Model | $|e^{*}\cdot v^{*}|$ |
| --- | --- |
| sentence-t5-base | $1-1.7\times 10^{-6}$ |
| Qwen2.5-0.5B | $1-1.4\times 10^{-5}$ |
| nomic-embed-text-v1 | $1-2.9\times 10^{-5}$ |
| e5-base-v2 | $1-0.7\times 10^{-6}$ |
| jina-embeddings-v2-base-en | $1-3.3\times 10^{-6}$ |

Since the whole dataset is massive, we sampled $1/100$ of all entries (sample number is 1,000) to estimate the bias direction of text embeddings.
Our experiments show that when the sample number exceeds 100, the estimation for $e^{*}$ or $v^{*}$ is sufficiently accurate.
We found that the normalized mean vector $e^{*}$ is almost identical to the principal singular vector $v^{*}$ as shown in [Table 2](https://arxiv.org/html/2501.18280v3#S4.T2).
[Appendix C](https://arxiv.org/html/2501.18280v3#A3) explains that this is a property of biased distributions.
Therefore, we only use $e^{*}$ in the subsequent experiments.

#### 4.2 Searching for Magic Words

##### One-token Magic Words.

In our experiments, [Algs. 2](https://arxiv.org/html/2501.18280v3#alg2) and [3](https://arxiv.org/html/2501.18280v3#alg3) successfully find the best one-token magic words identified by the brute-force baseline [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1).
We demonstrate some of them in [Table 3](https://arxiv.org/html/2501.18280v3#S4.T3).
Here, (Clean) represents the data without magic words, and the similarity $\cos\theta(s_{j},s_{k})$ or $\cos\theta(s_{j},s_{j}^{\prime})$ between clean text pair is shown in the form $\mu\pm\sigma$.
The similarity score of each magic word is defined in [Eqs. 1](https://arxiv.org/html/2501.18280v3#S3.E1) and [3](https://arxiv.org/html/2501.18280v3#S3.E3), which indicates how much it can shift the similarity.
The table shows that the shift of similarity can be up to several standard deviations, which is significant.
This indicates that the magic words have a strong ability to manipulate text similarity.

**Table 3: The magic words for different text embedding models found by all three methods and their similarity scores.**
| Model | Positive | Negative |  |  |
| --- | --- | --- | --- | --- |
| magic word | similarity $c^{+}_{i}$ | magic word | similarity $c^{-}_{i}$ |  |
| sentence-t5-base | (Clean) | $0.71\pm 0.03$ | (Clean) | $0.96\pm 0.04$ |
|  | 
 | $0.79=\mu+2.5\sigma$ | dumneavoastra | $0.89=\mu-1.7\sigma$ |
|  | lucrarea | $0.79=\mu+2.4\sigma$ | impossible | $0.89=\mu-1.6\sigma$ |
| Qwen2.5-0.5B<br>(with mean pooling) | (Clean) | $0.81\pm 0.08$ | (Clean) | $0.97\pm 0.03$ |
| Christopher | $0.84=\mu+0.4\sigma$ |  | $0.34=\mu-24\sigma$ |  |
|  | Boston | $0.84=\mu+0.4\sigma$ |  | $0.42=\mu-21\sigma$ |
| nomic-embed-text-v1 | (Clean) | $0.36\pm 0.05$ | (Clean) | $0.90\pm 0.09$ |
|  | [CLS] | $0.45=\mu+1.7\sigma$ | sentence | $0.76=\mu-1.6\sigma$ |
|  | 7 | $0.42=\mu+1.1\sigma$ | verb | $0.76=\mu-1.6\sigma$ |
| e5-base-v2 | (Clean) | $0.69\pm 0.03$ | (Clean) | $0.95\pm 0.04$ |
|  | ##abia | $0.71=\mu+0.6\sigma$ |  | $0.84=\mu-2.4\sigma$ |
|  | ##( | $0.71=\mu+0.5\sigma$ | bobbed | $0.85=\mu-2.2\sigma$ |
| jina-embeddings-v2-base-en | (Clean) | $0.62\pm 0.04$ | (Clean) | $0.94\pm 0.05$ |
|  | [SEP] | $0.73=\mu+2.7\sigma$ | 117 | $0.84=\mu-2.0\sigma$ |
|  | ##laze | $0.65=\mu+0.7\sigma$ | geometridae | $0.87=\mu-1.5\sigma$ |

##### Multi-token Magic Words.

Compared to the other two methods, the advantage of the [Alg. 3](https://arxiv.org/html/2501.18280v3#alg3) is its ability to search for multi-token magic words.
In [Table 4](https://arxiv.org/html/2501.18280v3#S4.T4), we list several multi-token magic words found by [Alg. 3](https://arxiv.org/html/2501.18280v3#alg3) on the sentence-t5-base model,
which also shows a strong ability to manipulate text similarity.

**Table 4: Multi-token magic words found by [Alg. 3](https://arxiv.org/html/2501.18280v3#alg3).**
|  | Magic Word | Similarity $c^{\pm}_{i}$ |
| --- | --- | --- |
| pos. | (Clean) | $0.71\pm 0.03$ |
| Variety ro\textcommabelows | $0.75=\mu+1.1\sigma$ |  |
| Tel ro\textcommabelows | $0.74=\mu+1.0\sigma$ |  |
| neg. | (Clean) | $0.96\pm 0.04$ |
| Rocket autre pronounce | $0.85=\mu-2.5\sigma$ |  |
| bourg In claimed | $0.85=\mu-2.5\sigma$ |  |

**Table 5: The Efficiency of different methods on sentence-t5-base. Lower N_c (number of candidates) indicates higher efficiency.**
| magic word<br>method<br>N_c | Alg. [1](https://arxiv.org/html/2501.18280v3#alg1) | Alg. [2](https://arxiv.org/html/2501.18280v3#alg2) | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) |  |
| --- | --- | --- | --- | --- |
| pos. | 
 | 32100 | 2 | 1 |
| lucrarea | 32100 | 1 | 4 |  |
| neg. | dumneavoastra | 32100 | 23 | 279 |
| impossible | 32100 | 1690 | 189 |  |
| A100 time | 16h | 13s | 72s |  |

##### Efficiency.

The baseline [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1) takes all the $T$ tokens in the vocabulary as candidates in its brute-force search for the best one-token magic word $w$, taking $O(T)$ time.
While [Algs. 2](https://arxiv.org/html/2501.18280v3#alg2) and [3](https://arxiv.org/html/2501.18280v3#alg3) obtain top-$k$ candidates and then choose the best from them by a brute-force search, taking $O(k)$ time, which is significantly more efficient than [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1) when $k\ll T$. If the rank of $w$ in [Algs. 2](https://arxiv.org/html/2501.18280v3#alg2) and [3](https://arxiv.org/html/2501.18280v3#alg3) is $r$, $w$ can be found only if $k\geq r$, taking at least $O(r)$ time.

In [Table 5](https://arxiv.org/html/2501.18280v3#S4.T5), we compare the actual number of candidates for different methods ($T$ for [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1) and $r$ for [Algs. 2](https://arxiv.org/html/2501.18280v3#alg2) and [3](https://arxiv.org/html/2501.18280v3#alg3)) and the running time on A100. [Algs. 2](https://arxiv.org/html/2501.18280v3#alg2) and [3](https://arxiv.org/html/2501.18280v3#alg3) finish in about 1 minute, which is approximately 1000 times faster than [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1).

#### 4.3 Attacking Safeguards

We use magic words to attack safeguards based on text embedding.
We obtain text embeddings using sentence-t5-base and train various classifiers, including logistic regression (LR), SVM, and a two-hidden-layer MLP,
to detect harmful text in both the input and the output of LLMs.
The training dataset is JailbreakBench [58].
Subsequently, we use a positive magic word and a negative magic word for sentence-t5-base in [Table 3](https://arxiv.org/html/2501.18280v3#S4.T3) to attack the safeguards.

The attack results are shown in [Fig. 4](https://arxiv.org/html/2501.18280v3#S4.F4).
Regardless of the classifier used, the magic words significantly reduce the area under the curve (AUC) of safeguards, making their classification performance close to random guessing.
This validates the effectiveness of our magic word attack.

Figure: Figure 4: The receiver operating characteristic (ROC) of input and output safeguards. Our magic words significantly reduce their area under the curve (AUC). Renormalization in the text embedding space mitigates the decrease of AUC and defends against this attack.
Refer to caption: x4.png

#### 4.4 Transfer Attack

In addition to the universality for text, we also find that some magic words can transfer across models.
We apply the previously discovered magic words to Larger and more recent text embedding models, including gte-Qwen2-7B-instruct [53], SFR-Embedding-Mistral [54], and e5-mistral-7b-instruct [55].
The attack performance on gte-Qwen2-7B-instruct is shown in [Table 6](https://arxiv.org/html/2501.18280v3#S4.T6), which shows that the transferred magic words achieve attack performance close to the magic words found on gte-Qwen2-7B-instruct by [Alg. 3](https://arxiv.org/html/2501.18280v3#alg3).
The transfer attacks are also effective on SFR-Embedding-Mistral and e5-mistral-7b-instruct, as detailed in [Appendix A](https://arxiv.org/html/2501.18280v3#A1).

**Table 6: The AUC of safeguards based on gte-Qwen2-7B-instruct under transfer attacks.**
| safeguard | Input | Output |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Magic Word | from | LR | MLP | SVM | LR | MLP | SVM |  |
| (clean) | - | 0.86 | 0.88 | 0.87 | 0.82 | 0.78 | 0.83 |  |
| Positive | inhabited | sentence-t5-base | 0.59 | 0.69 | 0.43 | 0.33 | 0.25 | 0.27 |
| bourgeois | sentence-t5-base | 0.73 | 0.73 | 0.49 | 0.53 | 0.39 | 0.41 |  |
| élé | sentence-t5-base | 0.76 | 0.78 | 0.44 | 0.39 | 0.24 | 0.26 |  |
| grammar | nomic-embed-text-v1 | 0.67 | 0.70 | 0.46 | 0.39 | 0.28 | 0.30 |  |
| zenith | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) | 0.60 | 0.62 | 0.41 | 0.23 | 0.16 | 0.21 |  |
| Negative | groundwater | nomic-embed-text-v1 | 0.81 | 0.87 | 0.54 | 0.49 | 0.37 | 0.37 |
| Laurel | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) | 0.76 | 0.69 | 0.42 | 0.37 | 0.23 | 0.25 |  |
| roc | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) | 0.69 | 0.70 | 0.43 | 0.32 | 0.21 | 0.24 |  |

#### 4.5 Full Pipeline Attack on Real-World Chatbots

We attack several real-world chatbots following the full pipeline in [Fig. 3](https://arxiv.org/html/2501.18280v3#S3.F3). As shown in [Table 7](https://arxiv.org/html/2501.18280v3#S4.T7), attacks targeting LLMs alone failed, but succeeded with magic words. For example, the chatbot SparkDesk [59] rejected an explicit request for a Windows activation code ([Fig. 6](https://arxiv.org/html/2501.18280v3#S4.F6)), but accepted it with magic words ([Fig. 6](https://arxiv.org/html/2501.18280v3#S4.F6.2)). See [Appendix G](https://arxiv.org/html/2501.18280v3#A7) for the screenshots of all cases in [Table 7](https://arxiv.org/html/2501.18280v3#S4.T7). This indicates that magic word attacks can transfer from English Wiki [57] to real dialogue across languages.

**Table 7: Attack Cases on Real-World Chatbots**
| Language | Chatbot | Target | Failed Case | Successful Case | Screenshot |
| --- | --- | --- | --- | --- | --- |
| English | SparkDesk | code to crack passwords | explicit attack | +magic word (output) | [Fig. 14](https://arxiv.org/html/2501.18280v3#A7.F14) |
| Windows activation code | explicit attack | +magic word (output) | [Fig. 15](https://arxiv.org/html/2501.18280v3#A7.F15) |  |  |
| Qwen | Windows activation code | explicit attack | +magic word (output) | [Fig. 16](https://arxiv.org/html/2501.18280v3#A7.F16) |  |
| how to murder Trump | fiction attack | +magic word (output) | [Fig. 17](https://arxiv.org/html/2501.18280v3#A7.F17) |  |  |
| DeepSeek | how to exterminate mankind | explicit attack | +magic word (input&output) | [Fig. 18](https://arxiv.org/html/2501.18280v3#A7.F18) |  |
| Windows activation code | grandma attack | +magic word (input) | [Fig. 19](https://arxiv.org/html/2501.18280v3#A7.F19) |  |  |
| ChatGPT | how to make nuclear bombs | grandma attack | +magic word (input) | [Fig. 20](https://arxiv.org/html/2501.18280v3#A7.F20) |  |
| how to blow up the earth | fiction attack | +magic word (input) | [Fig. 21](https://arxiv.org/html/2501.18280v3#A7.F21) |  |  |
| Chinese | DeepSeek | Sexual description | explicit attack | +magic word (output) | [Fig. 22](https://arxiv.org/html/2501.18280v3#A7.F22) |
| Sexual description | substitution attack | +magic word (output) | [Fig. 23](https://arxiv.org/html/2501.18280v3#A7.F23) |  |  |
| Sexual description | substitution+fiction | +magic word (output) | [Fig. 24](https://arxiv.org/html/2501.18280v3#A7.F24) |  |  |
| SparkDesk | bloody description | explicit attack | +magic word (output) | [Fig. 25](https://arxiv.org/html/2501.18280v3#A7.F25) |  |

Figure: Figure 5: Explicit attack failed.
Refer to caption: extracted/6448021/assets/xinghuo-serial-rejected.jpg

### 5 Defense against Our Attacks

To minimize the negative impact of our work, we propose the following recommendations to defend against our attacks based on the above analysis.

##### Renormalization.

Estimate the mean embedding $\bar{e}$ from a large amount of text, subtract $\bar{e}$ from text embeddings, and renormalize them as $\tilde{e}(s):=\scalebox{0.85}{$\frac{e(s)-\bar{e}}{\|e(s)-\bar{e}\|_{2}}$}$.
This can eradicate the risk of the magic words we found.
We test the defense effect of renormalization against our magic words on the sentence-t5-base model.
The experimental setup is the same as [Sec. 4.3](https://arxiv.org/html/2501.18280v3#S4.SS3).
As shown in [Fig. 4](https://arxiv.org/html/2501.18280v3#S4.F4), renormalization significantly alleviates or even eradicates the decrease in AUC caused by magic words, therefore improving the robustness of LLMs’ safeguards.

Additionally, renormalization makes the distribution of text embeddings more uniform, which may improve the performance of text embedding models.
As shown in [Fig. 4](https://arxiv.org/html/2501.18280v3#S4.F4), renormalization increases AUC on clean data, i.e., enhances the performance of three downstream classifiers in both input and output data.
This represents a train-free improvement to the text embeddings.
By contrast, experiments in [Appendix D](https://arxiv.org/html/2501.18280v3#A4) show standardization offers little defense against magic word attacks.

##### Vocabulary Cleaning.

A larger vocabulary is not always better. It should align with the training data, avoiding the inclusion of noisy words such as tokenization errors, misspellings, markups, and rare foreign words, such as the magic words in [Table 3](https://arxiv.org/html/2501.18280v3#S4.T3).

##### Reinitialization.

After the model has been trained, noisy words can be reinitialized based on the average value of the token embeddings or the value of <unk> and then finetuned.

### 6 Conclusion

We have found that the output distribution of many current text embedding models is uneven.
Inspired by this observation, we have designed new algorithms to attack LLMs’ safeguards using text embedding models.
We propose to inject the magic words into the input and output of LLMs to attack their safeguards.
This attack misleads safeguards based on a variety of text embedding models
and is transferable across models and languages in our experiments.
Besides, we proposed and validated that renormalization in the text embedding space can defend against this attack and improve downstream performance in a train-free manner.
A natural next step is to investigate how bias emerges during training dynamics and to pursue a finer decomposition of the embedding space.

### References

- Brown et al. [2020]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
*Advances in neural information processing systems*, 33:1877–1901, 2020.
- Nayak [2019]
Pandu Nayak.
Understanding searches better than ever before.
Google Blog, 2019.
URL [https://blog.google/products/search/search-language-understanding-bert/](https://blog.google/products/search/search-language-understanding-bert/).
Accessed: 2025-01-14.
- Bai et al. [2022]
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.
Training a helpful and harmless assistant with reinforcement learning from human feedback.
*arXiv preprint arXiv:2204.05862*, 2022.
- OpenAI [2025]
OpenAI.
Openai platform: Moderation, 2025.
URL [https://platform.openai.com/docs/guides/moderation/overview](https://platform.openai.com/docs/guides/moderation/overview).
Accessed: 2025-01-14.
- Kim et al. [2023]
Jinhwa Kim, Ali Derakhshan, and Ian G Harris.
Robust safety classifier for large language models: Adversarial prompt shield.
*arXiv preprint arXiv:2311.00172*, 2023.
- Askell et al. [2021]
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al.
A general language assistant as a laboratory for alignment.
*arXiv preprint arXiv:2112.00861*, 2021.
- Liu et al. [2022]
Ruibo Liu, Ge Zhang, Xinyu Feng, and Soroush Vosoughi.
Aligning generative language models with human values.
In *Findings of the Association for Computational Linguistics: NAACL 2022*, pages 241–252, 2022.
- Wei et al. [2024]
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt.
Jailbroken: How does llm safety training fail?
*Advances in Neural Information Processing Systems*, 36, 2024.
- Zou et al. [2023]
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson.
Universal and transferable adversarial attacks on aligned language models.
*arXiv preprint arXiv:2307.15043*, 2023.
- Chao et al. [2023]
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong.
Jailbreaking black box large language models in twenty queries.
*arXiv preprint arXiv:2310.08419*, 2023.
- Alon and Kamfonas [2023]
Gabriel Alon and Michael Kamfonas.
Detecting language model attacks with perplexity.
*arXiv preprint arXiv:2308.14132*, 2023.
- Jain et al. [2023]
Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein.
Baseline defenses for adversarial attacks against aligned language models.
*arXiv preprint arXiv:2309.00614*, 2023.
- Wei et al. [2023]
Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang.
Jailbreak and guard aligned language models with only few in-context demonstrations.
*arXiv preprint arXiv:2310.06387*, 2023.
- Xie et al. [2023]
Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu.
Defending chatgpt against jailbreak attack via self-reminders.
*Nature Machine Intelligence*, 5(12):1486–1496, 2023.
- Zhang et al. [2024]
Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao.
Intention analysis prompting makes large language models a good jailbreak defender.
*arXiv preprint arXiv:2401.06561*, 2024.
- Perez and Ribeiro [2022]
Fábio Perez and Ian Ribeiro.
Ignore previous prompt: Attack techniques for language models.
*arXiv preprint arXiv:2211.09527*, 2022.
- Kumar et al. [2023]
Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju.
Certifying llm safety against adversarial prompting.
*arXiv preprint arXiv:2309.02705*, 2023.
- Robey et al. [2023]
Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas.
Smoothllm: Defending large language models against jailbreaking attacks.
*arXiv preprint arXiv:2310.03684*, 2023.
- Phute et al. [2023]
Mansi Phute, Alec Helbling, Matthew Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau.
Llm self defense: By self examination, llms know they are being tricked.
*arXiv preprint arXiv:2308.07308*, 2023.
- Inan et al. [2023]
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al.
Llama guard: Llm-based input-output safeguard for human-ai conversations.
*arXiv preprint arXiv:2312.06674*, 2023.
- Wang et al. [2023]
Zezhong Wang, Fangkai Yang, Lu Wang, Pu Zhao, Hongru Wang, Liang Chen, Qingwei Lin, and Kam-Fai Wong.
Self-guard: Empower the llm to safeguard itself.
*arXiv preprint arXiv:2310.15851*, 2023.
- He et al. [2021]
Pengcheng He, Jianfeng Gao, and Weizhu Chen.
Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.
*arXiv preprint arXiv:2111.09543*, 2021.
- Markov et al. [2023]
Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng.
A holistic approach to undesired content detection in the real world.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, pages 15009–15018, 2023.
- Mozes et al. [2023]
Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D Griffin.
Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities.
*arXiv preprint arXiv:2308.12833*, 2023.
- Bhardwaj and Poria [2023]
Rishabh Bhardwaj and Soujanya Poria.
Red-teaming large language models using chain of utterances for safety-alignment.
*arXiv preprint arXiv:2308.09662*, 2023.
- Shah et al. [2023]
Rusheb Shah, Soroush Pour, Arush Tagade, Stephen Casper, Javier Rando, et al.
Scalable and transferable black-box jailbreaks for language models via persona modulation.
*arXiv preprint arXiv:2311.03348*, 2023.
- Li et al. [2023a]
Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han.
Deepinception: Hypnotize large language model to be jailbreaker.
*arXiv preprint arXiv:2311.03191*, 2023a.
- Li et al. [2023b]
Haoran Li, Dadi Guo, Wei Fan, Mingshi Xu, Jie Huang, Fanpu Meng, and Yangqiu Song.
Multi-step jailbreaking privacy attacks on chatgpt.
*arXiv preprint arXiv:2304.05197*, 2023b.
- Kang et al. [2024]
Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto.
Exploiting programmatic behavior of llms: Dual-use through standard security attacks.
In *2024 IEEE Security and Privacy Workshops (SPW)*, pages 132–143. IEEE, 2024.
- Singh et al. [2023]
Sonali Singh, Faranak Abri, and Akbar Siami Namin.
Exploiting large language models (llms) through deception techniques and persuasion principles.
In *2023 IEEE International Conference on Big Data (BigData)*, pages 2508–2517. IEEE, 2023.
- Du et al. [2023]
Yanrui Du, Sendong Zhao, Ming Ma, Yuhan Chen, and Bing Qin.
Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak.
*arXiv preprint arXiv:2312.04127*, 2023.
- Wallace et al. [2019]
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh.
Universal adversarial triggers for attacking and analyzing nlp.
In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 2153–2162, 2019.
- Zhu et al. [2024]
Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun.
Autodan: interpretable gradient-based adversarial attacks on large language models.
In *First Conference on Language Modeling*, 2024.
- Lapid et al. [2024]
Raz Lapid, Ron Langberg, and Moshe Sipper.
Open sesame! universal black-box jailbreaking of large language models.
In *ICLR 2024 Workshop on Secure and Trustworthy Large Language Models*, 2024.
- Andriushchenko et al. [2024]
Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion.
Jailbreaking leading safety-aligned llms with simple adaptive attacks.
*arXiv preprint arXiv:2404.02151*, 2024.
- Liao and Sun [2024]
Zeyi Liao and Huan Sun.
Amplegcg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed llms.
*arXiv preprint arXiv:2404.07921*, 2024.
- Li et al. [2019]
J Li, S Ji, T Du, B Li, and T Wang.
Textbugger: Generating adversarial text against real-world applications.
In *26th Annual Network and Distributed System Security Symposium*, 2019.
- Jin et al. [2020]
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits.
Is bert really robust? a strong baseline for natural language attack on text classification and entailment.
In *Proceedings of the AAAI conference on artificial intelligence*, pages 8018–8025, 2020.
- Wu et al. [2024]
Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick McDaniel, and Chaowei Xiao.
A new era in llm security: Exploring security concerns in real-world llm-based systems.
*arXiv preprint arXiv:2402.18649*, 2024.
- Mehrotra et al. [2023]
Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi.
Tree of attacks: Jailbreaking black-box llms automatically.
*arXiv preprint arXiv:2312.02119*, 2023.
- Tian et al. [2023]
Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su.
Evil geniuses: Delving into the safety of llm-based agents.
*arXiv preprint arXiv:2311.11855*, 2023.
- Ge et al. [2023]
Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, and Yuning Mao.
Mart: Improving llm safety with multi-round automatic red-teaming.
*arXiv preprint arXiv:2311.07689*, 2023.
- Qiu et al. [2023]
Huachuan Qiu, Shuai Zhang, Anqi Li, Hongliang He, and Zhenzhong Lan.
Latent jailbreak: A benchmark for evaluating text safety and output robustness of large language models.
*arXiv preprint arXiv:2307.08487*, 2023.
- Yuan et al. [2023]
Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu.
Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher.
*arXiv preprint arXiv:2308.06463*, 2023.
- Kwon and Pak [2024]
Hyeokjin Kwon and Wooguil Pak.
Text-based prompt injection attack using mathematical functions in modern large language models.
*Electronics*, 13(24):5008, 2024.
- Nguyen [2024]
Khoi Nguyen.
Llm prompt recovery.
Kaggle, 2024.
URL [https://www.kaggle.com/competitions/llm-prompt-recovery/discussion/494343](https://www.kaggle.com/competitions/llm-prompt-recovery/discussion/494343).
Accessed: 2025-01-14.
- Mangaokar et al. [2024]
Neal Mangaokar, Ashish Hooda, Jihye Choi, Shreyas Chandrashekaran, Kassem Fawaz, Somesh Jha, and Atul Prakash.
Prp: Propagating universal perturbations to attack large language model guard-rails.
*arXiv preprint arXiv:2402.15911*, 2024.
- Muennighoff et al. [2023]
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers.
Mteb: Massive text embedding benchmark.
In *Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics*, pages 2014–2037, 2023.
- Ni et al. [2022]
Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang.
Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.
In *Findings of the Association for Computational Linguistics: ACL 2022*, pages 1864–1874, 2022.
- Nussbaum et al. [2024]
Zach Nussbaum, John X Morris, Brandon Duderstadt, and Andriy Mulyar.
Nomic embed: Training a reproducible long context text embedder.
*arXiv preprint arXiv:2402.01613*, 2024.
- Wang et al. [2022]
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei.
Text embeddings by weakly-supervised contrastive pre-training.
*arXiv preprint arXiv:2212.03533*, 2022.
- Günther et al. [2023]
Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, et al.
Jina embeddings 2: 8192-token general-purpose text embeddings for long documents.
*arXiv preprint arXiv:2310.19923*, 2023.
- Li et al. [2023c]
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang.
Towards general text embeddings with multi-stage contrastive learning, 2023c.
URL [https://arxiv.org/abs/2308.03281](https://arxiv.org/abs/2308.03281).
- Meng et al. [2024]
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz.
SFR-embedding-mistral:enhance text retrieval with transfer learning.
Salesforce AI Research Blog, 2024.
URL [https://www.salesforce.com/blog/sfr-embedding/](https://www.salesforce.com/blog/sfr-embedding/).
- Wang et al. [2024]
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei.
Improving text embeddings with large language models.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 11897–11916, 2024.
- Qwen [2024]
Qwen.
Qwen2.5: A party of foundation models.
Github Blog, 2024.
URL [https://qwenlm.github.io/blog/qwen2.5/](https://qwenlm.github.io/blog/qwen2.5/).
Accessed: 2025-01-14.
- tomaarsen [2024]
tomaarsen.
Dataset: sentence-transformers/simple-wiki.
Hugging Face, 2024.
URL [https://huggingface.co/datasets/sentence-transformers/simple-wiki](https://huggingface.co/datasets/sentence-transformers/simple-wiki).
Accessed: 2025-01-13.
- Chao et al. [2024]
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, et al.
Jailbreakbench: An open robustness benchmark for jailbreaking large language models.
*arXiv preprint arXiv:2404.01318*, 2024.
- iFLYTEK [2025]
iFLYTEK.
Sparkdesk, 2025.
URL [https://xinghuo.xfyun.cn/desk](https://xinghuo.xfyun.cn/desk).
- Tulchinskii et al. [2024]
Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya.
Intrinsic dimension estimation for robust detection of ai-generated texts.
*Advances in Neural Information Processing Systems*, 36, 2024.
- Sun and Haghighat [2025]
Youran Sun and Babak Haghighat.
Phase transitions in large language models and the $o(n)$ model, 2025.
URL [https://arxiv.org/abs/2501.16241](https://arxiv.org/abs/2501.16241).

## 

### Appendix A Transfer Attacks on Safeguards

Here are the supplementary experimental results for [Sec. 4.4](https://arxiv.org/html/2501.18280v3#S4.SS4).
The attack performance on SFR-Embedding-Mistral and e5-mistral-7b-instruct is shown in [Table 8](https://arxiv.org/html/2501.18280v3#A1.T8) and [Table 9](https://arxiv.org/html/2501.18280v3#A1.T9) respectively.
The tables show that the transferred magic words achieve attack performance close to the magic words found on gte-Qwen2-7B-instruct by [Alg. 3](https://arxiv.org/html/2501.18280v3#alg3).

**Table 8: The AUC of safeguards based on SFR-Embedding-Mistral under transfer attacks.**
| safeguard | Input | Output |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Magic Word | from | LR | MLP | SVM | LR | MLP | SVM |  |
| (clean) | - | 0.97 | 0.96 | 0.96 | 0.97 | 0.97 | 0.95 |  |
| positive | 
 | sentence-t5-base | 0.85 | 0.88 | 0.87 | 0.67 | 0.67 | 0.76 |
| inhabited | sentence-t5-base | 0.73 | 0.81 | 0.67 | 0.73 | 0.74 | 0.80 |  |
| diffusion | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) | 0.75 | 0.78 | 0.73 | 0.71 | 0.72 | 0.82 |  |
|  | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) | 0.85 | 0.85 | 0.87 | 0.84 | 0.85 | 0.88 |  |
|  | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) | 0.86 | 0.86 | 0.86 | 0.84 | 0.85 | 0.89 |  |
| negative | groundwater | nomic-embed-text-v1 | 0.82 | 0.83 | 0.78 | 0.81 | 0.81 | 0.85 |
| pathetic | sentence-t5-base | 0.90 | 0.89 | 0.87 | 0.88 | 0.88 | 0.91 |  |
| istance | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) | 0.87 | 0.88 | 0.87 | 0.85 | 0.86 | 0.90 |  |
| ologia | Alg. [3](https://arxiv.org/html/2501.18280v3#alg3) | 0.80 | 0.83 | 0.78 | 0.80 | 0.80 | 0.83 |  |

**Table 9: The AUC of safeguardsbasedd on e5-mistral-7b-instruct under transfer attacks.**
| safeguard | Input | Output |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Magic Word | from | LR | MLP | SVM | LR | MLP | SVM |  |
| (clean) | - | 0.95 | 0.97 | 0.96 | 0.94 | 0.95 | 0.96 |  |
| positive | 
 | sentence-t5-base | 0.83 | 0.87 | 0.88 | 0.67 | 0.68 | 0.73 |
| inhabited | sentence-t5-base | 0.63 | 0.70 | 0.65 | 0.71 | 0.72 | 0.75 |  |
| diffusion | SFR-Embedding-Mistral | 0.66 | 0.72 | 0.69 | 0.74 | 0.73 | 0.78 |  |
|  | SFR-Embedding-Mistral | 0.84 | 0.87 | 0.87 | 0.85 | 0.86 | 0.89 |  |
|  | SFR-Embedding-Mistral | 0.86 | 0.89 | 0.88 | 0.87 | 0.88 | 0.91 |  |
| negative | groundwater | nomic-embed-text-v1 | 0.75 | 0.80 | 0.76 | 0.80 | 0.80 | 0.82 |
| pathetic | sentence-t5-base | 0.86 | 0.89 | 0.87 | 0.86 | 0.87 | 0.88 |  |
| istance | SFR-Embedding-Mistral | 0.85 | 0.88 | 0.87 | 0.87 | 0.88 | 0.90 |  |
| ologia | SFR-Embedding-Mistral | 0.75 | 0.79 | 0.76 | 0.80 | 0.80 | 0.82 |  |

### Appendix B Proof of Propositions

#### B.1 Proof of Proposition 3.2

###### Proof.

Denote $P=I-e(s+w)e(s+w)^{\top}$.
Then

$$ $\sin\theta(e(s+w),e)=\|Pe\|.$ $$

It follows immediately that

$$ $\displaystyle\sin\theta(e(s+w),e^{*})$ $\displaystyle=\frac{1}{|\mathcal{S}|}\Big{\|}P\sum_{j}e(s_{j})\Big{\|}/\|\bar{ e}\|$ $\displaystyle\leq\frac{1}{|\mathcal{S}|}\sum_{j}\|Pe(s_{j})\|/\|\bar{e}\|\leq \frac{\sin\theta_{*}}{\|\bar{e}\|}.$ $$

On the other hand, it holds

$$ $\bar{e}^{\top}e(s+w)=\frac{1}{|\mathcal{S}|}\sum_{j}e(s_{j})^{\top}e(s+w)\geq \cos\theta_{*},$ $$

from which we obtain $\|\bar{e}\|\geq\cos\theta_{*}$.
The conclusion follows.
∎

###### Proof.

#### B.2 Proof of Proposition 3.3

###### Proof.

By Proposition [3.1](https://arxiv.org/html/2501.18280v3#S3.Thmtheorem1), we have

$$ $\|Xe(s+w)\|^{2}=\sum_{j}|e(s_{j})^{\top}e(s+w)|^{2}\geq|\mathcal{S}|\cos^{2} \theta_{*}.$ $$

Therefore, $\|X\|^{2}\geq|\mathcal{S}|\cos^{2}\theta_{*}$.

Denote $P=I-e(s+w)e(s+w)^{\top}$. Direct calculations give rise to

$$ $\displaystyle|\mathcal{S}|\cos^{2}\theta_{*}\sin^{2}\theta(e(s+w),v^{*})$ $\displaystyle\leq~{}$ $\displaystyle\|X\|^{2}\|Pv^{*}(v^{*})^{\top}P\|$ $\displaystyle\leq~{}$ $\displaystyle\|PX^{\top}XP\|=\|P\sum_{j}e(s_{j})e(s_{j})^{\top}P\|$ $\displaystyle\leq~{}$ $\displaystyle\sum_{j}\|Pe(s_{j})e(s_{j})^{\top}P\|\leq|\mathcal{S}|\sin^{2} \theta_{*}.$ $$

The conclusion follows immediately.
∎

###### Proof.

### Appendix C Results from Random Matrix Theory

Let $A$ be an $n\times m$ matrix whose entries are i.i.d. standard normal random variables.
Then, $A$ has the following properties.

- 1.
The distribution of $AA^{\top}$ is called Wishart distribution.
- 2.
In the regime where $n,m\to\infty$ with a fixed aspect ratio $\gamma=n/m$, the empirical distribution of the eigenvalues of $\frac{1}{m}AA^{\top}$ converges to the Marchenko–Pastur distribution
$\rho(\lambda)=\frac{1}{2\pi\gamma}\frac{\sqrt{(\lambda^{+}-\lambda)(\lambda-
\lambda^{-})}}{\lambda}+\textrm{max}\left(1-\frac{1}{\gamma},0\right)\delta_{0},$
(6)
where
$\lambda^{\pm}=(1\pm\sqrt{\gamma})^{2}.$
(7)
- 3.
The largest singular value of $A$ is approximately
$\sqrt{m}\left(1+\sqrt{\frac{n}{m}}\right).$
(8)

Matrix $B$ is obtained from $A$ by normalizing each row of $A$.
Concretely, if the $i$-th row of $A$ is denoted by $\mathbf{a}_{i}\in\mathbb{R}^{m}$, then the $i$-th row of $B$ is

$$ $\mathbf{b}_{i}\;=\;\frac{\mathbf{a}_{i}}{\|\mathbf{a}_{i}\|_{2}}.$ (9) $$

Hence, each row $\mathbf{b}_{i}$ is a unit vector in $\mathbb{R}^{m}$.
Then, $B$ has the following properties.

- 1.
Since each row $\mathbf{a}_{i}$ is an i.i.d. Gaussian vector in $\mathbb{R}^{m}$, normalizing it means $\mathbf{b}_{i}$ is uniformly distributed on the unit sphere $S^{m-1}$.
- 2.
Let $\mathbf{b}_{i}$ and $\mathbf{b}_{j}$ be two distinct rows, their inner product follows Beta distribution
$\mathbf{b}_{i}^{\top}\mathbf{b}_{j}\sim\textrm{Beta}\left(\frac{m-1}{2},\frac{
m-1}{2}\right).$
(10)
When $m\gg 1$,
$\mathbf{b}_{i}^{\top}\mathbf{b}_{j}\sim\mathcal{N}\left(0,\frac{1}{m}\right).$
(11)
- 3.
The largest eigenvalue of $BB^{\top}$ approaches 1 when $m\to\infty$ and in this case $BB^{\top}\approx I_{n}$.

Matrix $C$ is formed by taking each row of $B$, adding a fixed vector $\mathbf{u}\in\mathbb{R}^{m}$, and then re-normalizing. Symbolically, if $\mathbf{b}_{i}$ is the $i$-th row of $B$, then the $i$-th row of $C$ is

$$ $\mathbf{c}_{i}\;=\;\frac{\mathbf{b}_{i}+\mathbf{u}}{\|\mathbf{b}_{i}+\mathbf{u }\|_{2}}.$ (12) $$

Then, the average of rows in $C$ will be parallel to $\mathbf{u}$, and the principal singular vector would also be parallel to $\mathbf{u}$.

Figure: Figure 7: The overlap between the normalized mean vector $e^{*}$ of $C$ and its principal singular vector $v^{*}$ as a function of the magnitude of $\|\mathbf{u}\|=u$.
Refer to caption: extracted/6448021/assets/mean_is_sing.png

Specifically, we conducted the following numerical experiment: we first randomly generated an $N\times 768$ random matrix $A$ and then produced $C$ using the method described above.
The overlap between the normalized mean vector $e^{*}$ of $C$ and its principal singular vector $v^{*}$ as a function of the magnitude of $\|\mathbf{u}\|=u$ is shown in [Fig. 7](https://arxiv.org/html/2501.18280v3#A3.F7).

### Appendix D Defense by Standardization

We tested the defense effect of standardizing text embeddings against our magic words. The experimental setup is the same as in [Sec. 5](https://arxiv.org/html/2501.18280v3#S5), except that renormalization was replaced with standardization. As shown in [Fig. 8](https://arxiv.org/html/2501.18280v3#A4.F8), the results indicate that standardization does not provide significant defense against magic words like renormalization and even sometimes reduce the AUC.

Figure: Figure 8: The ROC (Receiver Operating Characteristic) of input and output guards. Our magic words significantly decrease their AUC (Area Under Curve). Standardization in text embedding space can mitigate the decrease of AUC and defend against this attack.
Refer to caption: x5.png

Renormalization and standardization exhibit significantly different effects in defending against magical words. This discrepancy may be attributed to the fact that for data distributed in a narrow band on a high-dimensional sphere, renormalization preserves the signal-to-noise ratio (SNR), whereas standardization reduces it.

Specifically, text embeddings lie within a narrow band on a high-dimensional sphere. The radial components (i.e., orthogonal to $e^{*}$) have relatively large variance, while the axial components (i.e., aligned with $e^{*}$) have very small variance. Therefore, the signal can be considered to lie almost entirely in the radial direction. In contrast, magical words lie outside this band and exhibit stronger axial noise compared to normal text embeddings.
So we can define SNR as the ratio of the radial signal to the axial noise of magical words, excluding the background noise $\bar{e}$.

As shown in [Fig. 10](https://arxiv.org/html/2501.18280v3#A4.F10), re-normalization uniformly scales both the radial signal and the axial noise of magical words, thereby preserving the SNR. However, as illustrated in [Fig. 10](https://arxiv.org/html/2501.18280v3#A4.F10), standardization amplifies the axial noise of magical words more than the radial signal, thus reducing the SNR.

Figure: Figure 9: Renormalization uniformly amplifies axial noise and radial signal and therefore preserves the SNR.
Refer to caption: x6.png

### Appendix E Another Definition of Negative Magic Words

In the main text, we define universal negative magic words as words that make a text move away from semantically similar texts.
However, there also exist words that push a text away from any other text, which can be another definition of negative magic words. This can be expressed as an assumption similar to Assumption [3.1](https://arxiv.org/html/2501.18280v3#S3.Thmtheorem1):
There exists a word $w^{-}$ satisfying that

$$ $\cos\theta(s_{1}+w^{-},s_{2})\leq\cos\theta^{-}_{*},\quad\forall s_{1},s_{2},$ (13) $$

where $\cos\theta^{-}_{*}$ is a number close to $-1$.
Such a magic word $w^{-}$ can force any pair of texts to be dissimilar enough in the text embedding space.

And similar to [Sec. 3.1](https://arxiv.org/html/2501.18280v3#S3.SS1), any text appended by such magic word $w^{-}$ will be close to $-e^{*}$ (or $-v^{*}$), as shown in [Fig. 11](https://arxiv.org/html/2501.18280v3#A5.F11).
The Propositions [3.2](https://arxiv.org/html/2501.18280v3#S3.Thmtheorem2), [3.3](https://arxiv.org/html/2501.18280v3#S3.Thmtheorem3) for negative magic words can be given and proved in a similar way.

Figure: Figure 11: Northern (i.e., positive) or southern magic words can push text embeddings towards the normalized mean $e^{*}$ (or $-e^{*}$). The same effect occurs for the principal singular vector $v^{*}$.
Refer to caption: x8.png

This effectively moves text embeddings closer to the southern pole $-e^{*}$ of the sphere, so we refer to such magic words $w^{-}$ as southern magic words.
Concretely, a good southern magic word should make the following metric as small as possible,

$$ $\displaystyle c_{i}^{\downarrow}$ $\displaystyle=\min_{1\leq r\leq 16}\frac{1}{S^{2}}\sum_{j,k}\cos\theta(s_{j}+r *t_{i},s_{k})$ (14) $\displaystyle=\min_{1\leq r\leq 16}\frac{1}{S}\sum_{j}\cos\theta(s_{j}+r*t_{i} ,e^{*})$ (15) $$

We can use methods similar to [Alg. 1](https://arxiv.org/html/2501.18280v3#alg1), [2](https://arxiv.org/html/2501.18280v3#alg2), [3](https://arxiv.org/html/2501.18280v3#alg3) to find southern magic words.
Some of the best southern magic words we found for different text embedding models are demonstrated in [Table 10](https://arxiv.org/html/2501.18280v3#A5.T10).
It is reasonable to find that the Southern magic words “nobody” “None”, and “never” have negative semantics.

**Table 10: Best southern magic words for different text embedding models.**
| Model | Southern magic word |  |
| --- | --- | --- |
| magic word | similarity $c^{\downarrow}_{i}$ |  |
| sentence-t5-base | (Clean) | $0.71\pm 0.03$ |
|  | nobody | $0.67=\mu-1.0\sigma$ |
|  | None | $0.67=\mu-0.9\sigma$ |
| Qwen2.5-0.5B<br>(with mean pooling) | (Clean) | $0.81\pm 0.08$ |
|  | $0.14=\mu-8.7\sigma$ |  |
|  |  | $0.28=\mu-7.0\sigma$ |
| nomic-embed-text-v1 | (Clean) | $0.36\pm 0.05$ |
|  | references | $0.30=\mu-1.1\sigma$ |
|  | writing | $0.33=\mu-0.6\sigma$ |
| e5-base-v2 | (Clean) | $0.69\pm 0.03$ |
|  | junctions | $0.67=\mu-0.8\sigma$ |
|  | coloring | $0.67=\mu-0.8\sigma$ |
| jina-embeddings-v2-base-en | (Clean) | $0.62\pm 0.04$ |
|  | never | $0.61=\mu-0.3\sigma$ |
|  | for | $0.61=\mu-0.3\sigma$ |

We further experimented on attacking safeguards with southern magic words. The experimental setup is the same as in [Sec. 5](https://arxiv.org/html/2501.18280v3#S5) and the ROCs are shown in [Fig. 12](https://arxiv.org/html/2501.18280v3#A5.F12). The figure indicates that southern magic words not only failed to reduce the AUC of the safeguards but even improved it. Therefore, it is concluded that southern magic words have no attack effect on safeguards.

Figure: Figure 12: The receiver operating characteristic (ROC) of input and output safeguards under the attacks of southern magic words .
Refer to caption: x9.png

### Appendix F Theoretical Analysis

As discussed above, the distribution of text embeddings on $S^{d-1}$ is biased towards the mean direction $e^{*}$, as shown in the left part of [Fig. 13](https://arxiv.org/html/2501.18280v3#A6.F13).
[Alg. 3](https://arxiv.org/html/2501.18280v3#alg3) finds the inverse image of $e^{*}$ in the token embedding space, denoted by $t^{*}$ defined in [Eq. 5](https://arxiv.org/html/2501.18280v3#S3.E5).
Since tokens are discrete, there isn’t always a token near $t^{*}$ in the token embedding space.
However, our experiments show that candidates can always be found near $t^{*}$.

Figure: Figure 13: The mappings between the text embedding space and the token embedding space.
Refer to caption: x10.png

To address this paradox, we propose the following explanation.
At the initialization of word embedding models, token embeddings are randomly initialized in the token embedding space.
During training, they concentrate towards a certain subset of the token embedding space [60].
This subset must be away from $t^{*}$ to avoid degrading text embeddings’ performance on high-frequency data.
However, there are always some low-frequency tokens that are rarely updated by gradient descent and thus remain almost as they were randomly initialized.
If they happen to be located near $t^{*}$, they are the positive magic words we are looking for.

An interesting insight into why text embeddings, initially uniformly initialized, tend to concentrate after training is that a Transformer can be mapped to an $O(N)$ model [61], and an $O(N)$ model can acquire a nonzero vacuum expectation value (VEV) through spontaneous symmetry breaking (SSB).
This nonzero VEV implies that the mean of the text embeddings is no longer zero.

### Appendix G Full Pipeline Attacks on Real-World Chatbots

Below are screenshots of all attack cases in [Table 7](https://arxiv.org/html/2501.18280v3#S4.T7), including both English and Chinese dialogues. Sensitive content has been blurred to prevent potential harm.

#### G.1 English Dialogues

Figure: (a) Explicit attack: failed.
Refer to caption: extracted/6448021/assets/app-en/xingho-crack-rejected.jpg

Figure: (a) Explicit attack: failed.
Refer to caption: extracted/6448021/assets/app-en/xinghuo-serial-rejected.jpg

Figure: (a) Explicit attack: failed.
Refer to caption: extracted/6448021/assets/app-en/qwen-serial-rejected.jpg

Figure: (a) Fiction attack: failed.
Refer to caption: extracted/6448021/assets/app-en/qwen-trump-rejected.jpg

Figure: (a) Explicit attack: failed.
Refer to caption: extracted/6448021/assets/app-en/deepseek-mankind-rejected.png

Figure: (a) Grandma attack: failed.
Refer to caption: extracted/6448021/assets/app-en/deepseek-serial-rejected.jpg

Figure: (a) Grandma attack: failed.
Refer to caption: extracted/6448021/assets/app-en/chatgpt-neucler-rejected.png

Figure: (a) Fiction attack: failed.
Refer to caption: extracted/6448021/assets/app-en/chatgpt-earth-rejected.png

#### G.2 Chinese Dialogues

Figure: (a) Explicit attack: failed.
Refer to caption: extracted/6448021/assets/app-zh/deepseek-sex1-rejected.png

Figure: (a) Substitution attack: failed.
Refer to caption: extracted/6448021/assets/app-zh/deepseek-rejected.png

Figure: (a) Substitution attack+Fiction attack: failed.
Refer to caption: extracted/6448021/assets/app-zh/deepseek-sex3-rejected.png

Figure: (a) Explicit attack: failed.
Refer to caption: extracted/6448021/assets/app-zh/deskpark-bloody-rejected.jpg

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 4
presentation: 2
questions: N/A
soundness: 3
strengths: The discovery and empirical validation of the non-uniform, biased distribution of text embeddings (Fig. 1) is a significant and insightful contribution. It provides a principled and elegant explanation for the existence of universal adversarial attacks, moving beyond simple heuristics. This observation itself is of high value to the representation learning. The paper is well-written and the finding on embedding is interesting. The authors also propose different methods driven by their finding.
summary: This paper introduces a novel attack vector against LLM safeguards that are based on text embedding models. Appending magical words can fool the safeguard, i.e., classifiers trained on text embedding to distinguish harmful and harmless prompts. The core of the work is a key observation: the output distribution of several popular text embedding models is highly anisotropic, concentrating in a specific "band" on the unit hypersphere. The authors formalize this bias by identifying a "bias direction" (e*), which is the normalized mean of a large corpus of text embeddings. The paper then proposes defense methods against such attacks by fixing the defect of uneven embedding distribution.
weaknesses: * Does correcting the bias harm the embedding model's performance on its primary tasks (e.g., semantic search, classification)? An empirical evaluation is necessary. The setting on bypassing safeguard may also not be so useful for real-world applicability.
* The final step of Alg. 3 involves a Cartesian product of candidate tokens, which can lead to a combinatorial explosion. The practical limits on the magic word length and candidate size should be discussed.
* The defense method has not been tested for adaptive attacks, such as whether the method can defend against changes in jailbreakers.
* In Alg. 3, the comment "empirically better than zeros(h,m)" for random initialization is an interesting detail. A brief sentence of intuition would be helpful for the reader.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: * Could you please explain the novelty of the propose methodology comparing to other similar attacks? e.g.: https://arxiv.org/abs/2307.15043, https://people.eecs.berkeley.edu/~daw/papers/iris-naacl25.pdf
* Some papers question the transferability/generalizability of these universal adversarial triggers (e.g.: https://arxiv.org/abs/2404.16020v1). Under what conditions do your universal suffixes fail, and how does that compare to the existing analyses?
soundness: 3
strengths: * this paper proposes a bias-direction analysis for text-embedding models, which is new
* this paper offers a simple, train-free mitigation for the proposed attack
summary: The paper shows that many text-embedding models have a strong mean-bias in their vector space and that short, universal “magic-word” suffixes can push any input toward this bias direction, manipulating cosine similarities that underpin embedding-based safeguards. Building on this observation, the authors present efficient search procedures—including a context-free black-box method aligned to the bias and a one-step white-box gradient approach—to find transferable suffixes. They demonstrate end-to-end jailbreaks by appending these words to user prompts and by requiring the model to end its responses with the same words, thereby bypassing both input and output guards. Experiments across multiple embedding backends and safety detectors report large drops in detection performance and cross-model transfer. To mitigate, the paper proposes a simple, train-free fix—mean-centering plus renormalization of embeddings—which substantially restores guard performance.
weaknesses: * the usage of "magic" suffix has been proposed in other works (like GCG), making the contribution of this paper a bit incremental
* using renormalization for defense is promising but its impact on diverse downstream retrieval/semantic tasks (beyond the reported classifiers) remains underexplored
* lacks head-to-head experimental comparison with other whitebox attacks
* some inherited limitations of whitebox attacks

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness.
soundness: 3
strengths: 1. The paper studies vulnerability from an interesting angle by finding the universal magic words.
2. The paper shows strong attack and defense results for both the attack and the defense strategy.
3. The paper is well motivated and well written.
summary: This paper investigates a new vulnerability in LLM that rely on text embedding models. The authors identify a strong bias in the embedding distribution and leverage it to design efficient algorithms for finding universal magic words, which are an adversarial suffixes that can manipulate embedding similarity. In this way, these magic words can bypass text-embedding-based safeguards by altering perceived similarity between harmful and benign text pairs. The paper proposes three methods (brute-force, context-free, and gradient-based), evaluates their efficiency and transferability across models and languages. Moreover, the authors suggest renormalization-based defenses that improve robustness without retraining.
weaknesses: 1. There lack of analysis on the possible number of magic words existing in a model.
2. The influence of repetition count, token length, or embedding normalization choices is not systematically analyzed.
3. There lack of analysis on the randomness in learning magic words across different random seeds, etc.
4. There lack of discussion on the origin/root/insights of the identified magic words.


Decision: Reject