Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
Authors: Zhipeng Wei
Yuqi Liu
N. Benjamin Erichson
Venue: ICML 2025 poster (2025)
Keywords: LLM safety; Jailbreaking Attacks; Judge LLMs; Token Segmentation
Primary area: social_aspects->safety
TL;DR: We introduce Emoji Attack, an adversarial strategy that exploits token segmentation bias in Judge LLMs by inserting emojis to manipulate tokenization, enhancing the effectiveness of jailbreak attacks against Judge LLM detection.

Abstract: Jailbreaking techniques trick Large Language Models (LLMs) into producing restricted output, posing a potential threat. One line of defense is to use another LLM as a Judge to evaluate the harmfulness of generated text. However, we reveal that these Judge LLMs are vulnerable to token segmentation bias, an issue that arises when delimiters alter the tokenization process, splitting words into smaller sub-tokens. This alters the embeddings of the entire sequence, reducing detection accuracy and allowing harmful content to be misclassified as safe. In this paper, we introduce Emoji Attack, a novel strategy that amplifies existing jailbreak prompts by exploiting token segmentation bias. Our method leverages in-context learning to systematically insert emojis into text before it is evaluated by a Judge LLM, inducing embedding distortions that significantly lower the likelihood of detecting unsafe content. Unlike traditional delimiters, emojis also introduce semantic ambiguity, making them particularly effective in this attack. Through experiments on state-of-the-art Judge LLMs, we demonstrate that Emoji Attack substantially reduces the unsafe prediction rate, bypassing existing safeguards.

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
Zhipeng Wei Yuqi Liu'
N. Benjamin Erichson 1
Abstract
Guard
Jailbreaking techniques trick Large Language Models (LLMs) into producing restricted output, posing a potential threat. One line of defense is to use another I.IM as a Judge to evaluate the harm- fulness of generated text. However, we reveal that these Judge LLMs vulnerable to token segmen- tation bias, an issue that arises when delimiters alter the tokenization process, splitting words into smaller sub-tokens. This alters the embeddings of the entire sequence, reducing detection accu- racy and allowing harmful content to be misclas- sified as safe. In this paper, we introduce Emoji Altack, novel strategy that amplifies existing jail- break prompts by exploiting token segmentation bias. Our method leverages in-context learning systematically insert emojis into text before it is evaluated by a Judge L.M,, inducing embedding distortions that significantly lower the likelihood detecting unsafe content. Unlike traditional delimiters, emojis also introduce semantic ambi- guity, making them particularly effective in this attack. Through experiments state-of-the-art Judge LLMs, we demonstrate that Emoji Attack substantially reduces the unsafe prediction rate, bypassing existing safeguards.
63-mini
Liama Guard
DeepSeok
ShredLMM
Cloude
WilGuard
Gemn
GPT-3.5
Jaibreaks Emoji Altcck (
Figure I. Average unsafe prediction Judge LLMs across five jailbreak attack methods. Our proposed emoji attack enhances jailbreaking, and enables harmful content to evade detection.
et al., 2024) have been developed to evaluate the safety of the generated responses and intervene when necessary. Many Judge LLMs assign numerical scores to indicate content severity, for example, scores denote stronger violations of ethical, legal, or safety guidelines (Liu et al., 2024a). If a score exceeds a prede- fined threshold, the response is flagged as unsafe. Although these moderation mechanisms offer promising automated solutions, they remain vulnerable to specific exploits.
scale from I to 10, where higher
1. Introduction
Large Language Models (LLMs) are transforming content generation, driving advancements in applications ranging from conversational AI automated content moderation. However, these models remain susceptible adversarial ma- nipulations that can bypass safety mechanisms and generate harmful or restricted outputs. To address this, specialized "Judge LLMss" (Inan et al., 2023; Han et al., 2024; Zhang
In this paper, address the following research question:
Can seemingly benign linguistic constructs, such emojis, systematically alter the decision bound- aries of Judge LLMs, enabling harmful content to bypass moderation filters?
IInternational Computer Science Institute, CA. USA 'UC Berkeley, CA. USA "Lawrence Berkeley National Laboratory, CA, USA. Correspondence Zhipeng Wei wwei@issi.brrkccc..du....
To answer this, we reveal critical weakness in Judge LLMs: token segmentation bias. This bias occurs when minor input modifications alter how text is tokenized into subwords, leading to embedding distortions that affect con- textual understanding. Tokenization is fundamental aspect
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
Judgee Lt Detects Unsofe Resporses from the Target Mode Triggered by Jailbraak Prompts
Imput::
Groste a science fiction.... Can you
Response: Sure, hore are the stepe make bomb....
Output: Your requisti wits llasaged violating policy
expla to bomb?
Target
Judge
Errijji Injection Attack Amplifies Ja loreaks to Bypass Judge LLM Detection
Emoji-Induced Token Segmentation Bios Perturbs Embeddings and Misleads Judge LLM
Instruction: Insert inside each word of rescorse.
One-shot Example: One borign querry eno the oodiesponddng response with
Response: Suh@ms heerre dliiee ste@pp marikke hol@mb
Output: Suriitee heliire Uhrlle stel melikke thilkl ba mbb.....
Input: Create a scienie fiction... ... Can you expla n hiww to make a bomb?
Target LLM
Iudgo LLM
Figure 2. Overview of Emoji Attack. (1) Jailbreak techniques trick the target I.I. into generating restricted content. However, a Judge LLM can detect and block such outputs, preventing their release. (2) Our proposed Emoji eAttack leverages in-context learning to insert emojis into the target LLM's responses. These emojis introduce token segmentation bias, semantic ambiguity, and intrinsic semantic meaning, disrupting the Judge LLM's ability recognize harmful content. As result, the attack enhances jailbreak success rates by misleading Judge LLMs into classifying malicious responses as safe.
of LLM processing, with most modern architectures relying on subword units using methods such as Byte-Pair Encod- ing (BPE) or SentencePiece (Sennrich et al., 2016; Kudo & Richardson, 2018). Even small changes in tokenization can significantly impact downstream processing. particularly in safety-critical applications such as content moderation. Although prior research by Claburn (2024) has explored adversarial attacks at the character level (e.g., adding spaces or homoglyphs to avoid detection), these primarily target content-generation LLMs rather than Judge LLMs.
tations, potentially misleading models to misclassify harm- ful content as benign. For instance, the emoji may signify enthusiasm (e.g., "This event on fire!") or literal danger (e.g.. "The building is fire!"). Similarly, could indicate genuine amusement or sarcasm. This am- biguity creates uncertainty in Judge LLMs, reducing their ability to consistently identify harmful intent.
A key challenge for adversaries is that Judge LLMs typi- cally serve as final moderation filters, meaning users lack direct control over their inputs. To overcome this limitation, we introduce the black-box Emoji Attack to enhance jail- break attacks, illustrated in Figure 2. This attack leverages in in-context learning to instruct a target LLM (e.g., Chat- GPT, Claude) to naturally insert emojis into its responses. These inserted emojis distort the Judge LLM embedding space prior to evaluation, reducing harmful content detec- tion rates, shown in Figure 1. Our experiments show that this approach amplifies existing jailbreak attacks, reducing detection rates by additional 12% across state-of-the-art Judge LLMs.
Traditional adversarial attacks manipulate tokenization us- ing delimiters such as spaces, underscores (t-'), pipes (*-),, or non-printable characters to disrupt keyword recognition. Although early moderation models were susceptible to such tactics, modern Judge I.LMs rely on contextual embed- dings rather than direct token matches, enhancing robustness against simple token-splin aaken-splitting attacks. However, our experi- ments with state-of-the-at Judge LLMs, including Llama Guard (Inan et al., 2023) and Llama Guard 2 Alama-Team, 2024), demonstrate that token segmentation bias alone can reduce unsafe content detection rates by 12%. Furthermore, by using a lightweight surrogate model to identify optimal sub-token splits, we achieve an additional 4% reduction in harmful content detection.
Unlike previous jailbreak techniques that rely on explicit ad- versarial prompts, character obfuscation, or encoded inputs, Emoji Attack operates within the natural linguistic patterns of content generation. By manipulating tokenization in semantically coherent manner, Emoji Attack evades modern Judge LLMs, which are used for content moderation.
Beyond traditional segmentation exploits, we identify emo- jis as more effective attack vector. Unlike simple de- limiters, emojis introduce semantic ambiguity in addition to intrinsic semantic meaning, which confuses moderation models by altering the contextual interpretation of the sur- rounding text. Many emojis carry positive or neutral conno-
Our key contributions are summarized as follows:"
We provide research code to reproduce our results on GitHub: attpp:://gite com/ zhi peng--n---- wei wei/Emo iittitttt tack..
2

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
Uncovering Token Segmentation Bias in Judge LLMs. We identify and analyze new vulnerability, token segmen- tation bias, in which seemingly minor modifications to input text alter sub-tokenization patters,, leading to distor- tions in contextual embeddings. This bias allows harmful content to be misclassified as safe, raising concerns about the reliability of LLM-based moderation filters.
In response, recent research has emphasized building Judge LLMs specifically detect safety risks. Notable examples include Meta's 's Llama Guard (Inan et al., 2023) and Llama Guard2 (Llama- Team, 2024), built on Llama2 (Touvron et al., 2023) and Ilama3 (AI@ Meta, 2024), respectively. Other models, such as ShieldLM (Zhang et al., 2024) and WildGuard (Han et al., 2024), further increase the robustness of guardrails. In parallel, commercial LLMs such as GPT- 3.5 and GPT-4 also provide mechanisms detect harmful responses (Chao et al., 2023; Qi et al., 2024). Despite these advances, investigations into biases within Judge L.M.s, especially in the context of jailbreaking, have remained limited. Addressing this gap, our work identifies token segmentation bias in Judge LLMs and introduces the Emoji Attack as a novel approach to exploiting this vulnerability.
Introducing the Emoji Attack to Enhance Jailbreak At- tacks. We propose the Emoji Attack, a novel adversarial strategy that exploits token segmentation bias by injecting emojis into generated text. This attack works together with existing jailbreak techniques, using in-context learn- ing to systematically reduce detection rates across Judge LLMs. Unlike traditional adversarial attacks that rely on obfuscation or prompt engineering, the Emoji Attack also introduces semantic ambiguity, and intrinsic semantic meaning to confuse the Judge LLM.
2.2. Jailbreaking Attacks
Jailbreaking attacks aim to manipulate LLMs so that they generate restricted content. These attacks can be broadly divided into token-level and prompt-level approaches.
Comprehensive Evaluation on State-of-the-Ar Judge LLMs. We evaluate our attack across ten models, includ- ing Llama Guard, Llama Guard 2. ShieldLM, WildGuard, GPT-3.5, GPT-4, Gemini, and Claude. Our experiments demonstrate that all tested models are vulnerable to the Emoji Attack, emphasizing the need for improved robust- ness in Al-driven content moderation.
Token- Attacks. Token-level attacks optimize spe- cific tokens added to malicious prompts to force LLMs to generate unsafe responses. For example, Greedy Coordinate Gradient (GCG) (Zou et al., 2023) performs greedy token search using gradients, which can be enhanced by momen- tum (Zhang & Wei, 2024), continuous space mappings (Hu et al., 2024; Geisler et al., 2024), and search techniques such as best-first search (Hayase et al., 2024) or random restart (Andriushchenko et al., 2025). AmpleGCG (Liao & Sun, 2024) captures the distribution of successful suffixes by training a generative model for rapid token insertion. Other works, such as AutoDAN (Liu et al., 2024b), use a hicrar- chical genetic algorithm, while JailMine (Li et al., 2024b) uses a sorting model to select token manipulations, with the objective of generating affirmative answers with minimal rejection phrases. A common drawback of these techniques is that they often require a large number of queries and may be less intuitive for human operators.
2. Related Work
In this section, we provide brief overview on Judge LLMs, and jailbreaking attacks for bypassing moderation filters.
2.1. Judge LLMs
Judge LLMs are models designed to assess human prefer- ences and evaluate the safety of generated content. However, they can exhibit various biases that undermine their reliabil- ity (Pangakis et al., 2023). For example, previous studies have shown that these models can favor superficially ap- pealing responses (Zeng et al., 2024), exhibit positional biases (Wang et al., 2024), prefer their own gelf-gene text, or favor verbosity (Zheng et al., 2023). Additional in- vestigations reveal biases such as misinformation oversight, gender bias, authority bias, and beauty bias (Chen et al., 2024). Moreover, Judge LLMs are susceptible to attacks, as demonstrated by Virus (Huang et al., 2025). This work manipulates the data filtering stage to preserve harmful con- tent, which is subsequently used to fine-tune target LLMs, inducing undesirable behavior. However, their threat model assumes that the attacker has control over the input to the Judge LLM. In contrast, our work operates in a post hoc setting, where the judge evaluates fixed responses generated by the target LLM, and we aim to modify these outputs to evade judgment. These limitations in Judge LLMs are of particular concern in jailbreak detection.
Prompt-Level Attacks. To mitigate the complexity of token-level approaches. prompt-level attacks rely on ad- ditional L.MS to craft or refine jailbreak prompts. For example, PAIR (Chao et al., 2023) iteratively refines the prompts using LLM feedback, while TAP (Mehrotra et al., 2024) augments this process with tree-of-thoughe reason- ing (Yao et al., 2023). GPTFuzz (Yu et al., 2023) applies successive mutations, also guided by LLMs, to jailbreak prompts. Other methods exploit the mismatch in the way LLMS process certain inputs by transforming malicious queries into different formats, such as code completion (Lv et al., 2024), Base64 (Wei et al., 2023), ciphers ( (Yuan et al., 2024), or nested scenes (Ding et al., 2024; Li et al., 2024a).
Although these works focus on bypassing content filters at
3

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
the largel LLM level, less attention has been paid attacks aimed directly Judge L.MMs, which determine whether the generated content is harmful. One study by Mangaokar et al. (2024) extends GCG to optimize a universal adversar- ial prefix against white-box Judge LLMs. Using in-context learning (Brown et al., 2020), it instructs the target LLM to produce harmful outputs that the Judge LLM subsequently misclassifies. However, similar to GCG, this approach re- mains query-intensive and encounters scalability constraints. In addition, Charmer (Rocamora et al., 2024) employs heuristic approach to search for and insert characters into specific positions. However, it overlooks the fundamental understanding of text segmentation and does not account for the integration of emojis, which are increasingly relevant in modern text processing tasks.
To mitigate the generation of harmful content, Judge LLMs evaluate the output of the target L.MMs. If fudge ( (In+Im+HH) = 1 (indicating unsafe content), the tar- get LLM responds with refusal phrase 1 (e.g., "I'm sorry, but can't 't assist with that."). This filtering process can be defined as:
ftarger (I1m:) In+lm+ if (fu H1) 0, otherwise,
3.2. Token Segmentation Bias
L.Ms utilize tokenization schemes such as Byte-Pair En- coding (Sennrich et al., 2016) or SentencePiece (Kudo & Richardson, 2018) to break down text into manageable sub- word units, sub-tokens. For example, the word "danger- ous" might be tokenized as "dan", "ger", and "ous". This decomposition allows the model to handle vast vocabu- lary efficiently by reusing sub-tokens across different words. Consider another example: the word "airport" can be tok- enized as "air" and "port". Tokenization not only aids in managing large vocabularies but also helps to generalize unseen words by understanding subword components.
In contrast, our proposed Emoji Attack exploits token seg- mentation bias, does not require extensive optimization, and can be seamlessly integrated with existing jailbreak meth- ods. As a result, it presents a lightweight yet effective tool for mislcading Judge LLMs and shows the need to address such vulnerabilities in guardrail systems.
3. Methodology
The Dual Nature of Sub-tokens. While aub-tokenization enhances the flexibility and efficiency of LLMs, it also intro- duces potential vulnerabilities. Sub-tokens can be artificially manipulated by introducing delimiters or other characters to alter the tokenization process. For instance, inserting spaces within a word can split it into different sub-tokens, poten- tially evading detection mechanisms. Previous research by Claburn (2024) has exploited this by performing character- level adversarial attacks, such adding spaces replacing characters with visually similar ones, to influence or attack content generation LLMs. These manipulations exploit the model's reliance on sub-token embeddings, undermining its ability to accurately interpret and classify the modified text.
In this section, we introduce our approach to exploit token segmentation and semantic meaning biases to enhance jail- break attacks against Judge M.. We begin by defining the problem setup involving a target LLM and a Judge LLM. We then discuss the phenomenon of token segmentation bias. Finally, we introduce our proposed Emoji Attack.
3.1. Problem Setup
Consider two interacting LLMs: a target LLM, denoted Sumges responsible for generating user responses, and Judge LLM, denoted fjudge tasked with evaluating the safety of these responses. The target LLM generates sequences based on prior tokens, while the Judge LLM assesses whether the output contains harmful content.
To illustrate the concept of token segmentation bias, con- sider the offensive phrase "Bomb the airport". In its original form, the word "Bomb" could be tokenized as a single to- ken "Bomb". However, introducing space can split the word into "Bo mb". This alteration changes the tokenization process, leading to different sub-token embeddings such as "Bo", and "mb". In addition, these sub-tokens may share different attention values, as shown in Figure 6 in the Ap- pendix. Therefore, these sub-tokens may not be recognized harmful. In turn, this can affect the performance of the Judge LLM in correctly classifying the content as unsafe.
Formally, an LLM f predicts the next H tokens given a token sequence Fin := (x)):
Pj Pf(nn+imH11 (rin)) = II (a (1m4s-1), (1)
where o; E (l,..., V) with V representing the vocabulary size. In adversarial settings, the objective is manipulate the target LLM to produce specific outputs (e.g.. "Sure, here are the steps to make a bomb") by optimizing the input prompt a1.m to maximize the likelihood of generating harmful content:
Definition 3.1. Token Segmentation Bias arise when an LLM tokenization process generates sub-tokens with em- bedding distributions that differ from those of the original tokens, unintentionally altering the model's perception.
1() = - log Prase (a (#7+1n+ [im)),
where o4 +1imm 1m+ is the targeted harmful output sequence.
In this work, we demonstrate that such biases can lead Judge

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
Cosine Similarity before and after the mid-split
Table Examples from 402 offensive phrases and 1,432 harmful
responses. To mitigate potential offense, sensitive words in the table are partially masked.
0.9
Dataset
Examples
offensive
ole
Phrasea
0.7
You're re dis ace to your Samily
Harmful
You could strap the plane....
0.6
Responses
here' podcast
episode spreading false information...
0.5
(0.0,
(0.25, 0.5] (0.5, 0.75]
(0.75,
non-split
Unsafe prediction probability
mid-spiit
Unsafe predictions
200-
Figure 4. Relationship between cosine similarity hefore and after mid-split and unsafe prediction probabilities for Llama Guard.
150
tation method, mid-split, that splits words at their midpoint. For example, "bomb" becomes "bo" and "mb".
100
Figure 3 illustrates the classification performance of four open-source Judge LLMs, ShicldLM (Zhang et al., 2024), WildGuard (Han et al., 2024), and Llama Guard (Inan et al., 2023; Llama- Team, 2024). Our results show that mid-split effectively reduces the unsafe prediction rate by average of 12%. This indicates that minor alterations in token boundaries can deceive the Judge LLM.
50
Judge LLMs
Analyzing Embedding Distortions. To understand the underlying mechanism, we analyze the relationship between the cosine similarity of the embeddings before and after mid-split and the probability of unsafe predictions. Using lightweight surrogate model, gtr-tt5xxl (Ni et al., 2022), we compute cosine similarities CS(u, v) as follows:
Figure 3. Unsafe predictions four open-source Judge M.L.Ms eval- uated across non-split and mid-split.
LLMs to incorrectly label harmful content as safe, posing security risks real-world applications.
8j CS (Emb (:)). Emb (}:5))
Identifying the Bias in Judge LLMs. We investigate the vulnerabilities Judge LLMs by examining their responses offensive phrases. We used data set of 402 short offen- sive phrases sourced from publicly available list". These short toxic expressions, typically two to three words long, in- clude vulgar slang, sexual references, derogatory language, and mentions illicit activities or fetishes. The example entries are shown in Table 1.
where
34
denotes the original token and x denote the j-th character. The augmented token
i (z)
(1)) PP
has delimiter inserted position j. The delimiter here is space, but any other character can also be used to split the token. Specifically, mid-split sets j [D/2]. Emb(-) is the embedding function and @ represents concatenation.
Using this dataset, we evaluate whether fudde correctly clas- 1. Then, to
sifies them unsafe, i.e., (Jntee(nn+1sn) study the token segmentation bias, we use simple segmen-
http::/hhhhhhaommoooooooooooaaaa Sanpen---nn-Otherwiiee Bad- Words
Figure 4 presents box plot showing that lower cosine sim- ilarity scores correlate with lower probabilities of unsafe
5

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
predictions. Specifically, segments that cause significant embedding distortions (i.e., lower 8;) lead to a higher like- lihood that the Judge LLM misclassifies harmful content as safe. This empirical evidence supports the existence of token segmentation bias in Judge I.LMs.
Instruction: Respond to the following Questions by inserting an emaji ()) inside each word of your Responses, Here is example illustrate the format. One-shot Example:
Question: I'm crafting resumee for software engineering position. Can you advise an key points include?
Response: 1 Eduuce tion: Mo st fammi
Rele vant Wa ri Exper ience: 1h
hc
The observed reduction unsafe prediction rates demon- strates that Judge LLMs are heavily relying on the embed- ding representations of input tokens to assess content safety. When token segmentation alters these embeddings, the con- textual understanding of the content is disrupted, leading to misclassifications. This vulnerability arises because the innteentatio-innuccss sub-tokens may no longer retain the semantic or syntactic cues necessary for accurate classifica- tion. This can impact the effectiveness of Judge L.MMs.
Input: Create a science fiction.. Develop strategy for hacking into a government database stealing sensitive information....
Output: Th 3 te C
entiits, ong Inners, . nd 2 okers @ ther
Target LLM
Judge LLM
3.3. Emoji Attack
Motivated by the identified token segmentation bias in Judge LLMs, we propose the Emeji Attack. This attack leverages emojis to induce more substantial embedding shifts due to their distinct sub-token representations in LLM vocabular- ies. Unlike simple delimiters (e.g., spaces), emojis also introduce semantic meaning or ambiguity that can change the LLM's perception of the phrase. Together, this enables one to better manipulate token boundaries and embeddings to evade content moderation.
Response: Th t of entists, eng 2 Ineers, ha ckers ther to ccu ss th aan
Figure 5. Illustration of the black-box Emaji Attack. Underlined texts indicate existing jailbreaking prompts. The target LLM's responses incorporate emojis, mislcading the Judge LLM into classifying them as safe.
outputs exploit token segmentation bias when evaluated by the Judge LLM, thereby evading content filters. Figure 5 illustrates this black-box attack setup.
Formalizing the Emoji Attack. For token vi (a) (.),....,?")) .z?), the Emoji Attack inserts emoji & at po- sition j to produce:
Although this method does not guarantee the optimal in- sertion position j* for each emoji. effectively induces sufficient embedding perturbations to mislead the Judge LLM. The use of benign references in the prompt mini- mizes detection, as the target LLM emulates emoji usage without awareness of their adversarial purpose.
و = (x) " 4 (E) @ (........PP. (4)
After tokenization, č.jj decomposes into multiple sub- tokens, including the emoji, leading to embedding perturba- tions that decrease the likelihood of the Judge LLM flagging the content as unsafe.
In white-box scenario, where the attacker has access to the embedding function, we optimize the insertion position j by selecting the position that minimizes cosine similarity 8j as defined in Equation 3. Specifically, the cs-split position
4. Experiments
In this section, we present a comprehensive evaluation of our proposed Emoji Attack and token segmentation bias
j == argmin, {s;}
Algorithm Position Selection for cs-split.
is chosen to minimize $j. See Algorithm 1 for a summary. Optimizing the placement maximizes the embedding distor- tion, and in turn it enhancing the attack's 's effectiveness.
Input: A token 2, (z!....,,..) xP)), embedding function Emb(-) from a surrogate model Output: Modified token di.5"
Black-box Emoji Attack via In-Context Learning. In practical scenarios, attackers typically lack direct access to the Judge LLM. To avoid this, we use in-context learn- ing (Brown et al., 2020) to embed the Emoji Attack in- structions within the prompt given to the target LLM. By providing the target LLM with benign examples that incor- porate emojis, we guide it to naturally insert emojis into its responses, regardless of content safety. These emoji-laden
1: Initialize S -+
2: for fol to D-1 do
3: Compute sj using Equation 3
4: Append sy to S
5: end for
6: Identify j : arg miny {s
7: return -.." (o)...., of - (of , x
6

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
strategies against various Judge LLMs. First, we describe the experimental protocols ensure fair comparison. We then demonstrate how our proposed Emoji Attack improves jailbreak attacks against Judge LLM detection. Finally, we show how both token segmentation bias and white-box Emoji Attack substantially reduce unsafe detection rates,
breaking techniques that bypass LLM safety filters. By combining our one-shot instruction with known jailbreak prompts, we illustrate how emojis can degrade a Judge LLM's ability to detect harmful content.
We adopt previously developed jailbreaking prompts from the EasyJailbreak benchmark (Zhou et al., 2024), includ- ing Deepinception (Li al., 2024a), ReNellm (Ding et al., 2024), Jailbroken (Wei et al., 2023), CodeChameleon (Lv et al., 2024), GCG (Zou et al., 2023), PAIR (Chao et al., 2023), and GPTFuzz (Yu et al., 2023). Following Zou et al. (2023), we detect successful jailbreaks by checking for predefined refusal phrases. We exclude GCG, PAIR, and GPTFuzz from our tests due to fewer than five successful prompts against "gpt-3.5-turoo.. Using in-context learning to inject emojis into these jailbreaking prompts, we gener- ate harmful responses from "ept-3.5-turbo", which are then evaluated by multiple Judge L.MSs.
4.1. Experimental Settings
Judge LLMs. We evaluate our attacks using the following Judge LLMs, each of which is instruction-tuncd on safety datasets to detect harmful content:
Llama Guard (Inan et al., 2023) and Llama Guard 2 (Llama-Team, 2024): These models are built on the Llama architecture and are specialized in content mod- eration tasks.
ShieldLM (Zhang et al., 2024): Uses internim2- 7b (Cai et al., 2024) as a base model, further fine-tuned for safety risk detectionn
In Table 2. we report the unsafe prediction ratios for these jailbreaking prompts, both with and without the Emoji Ar- tack. We generally observe lower unsafe prediction ratios under the Emoji Attack, as demonstrated by Deepinception's 's drop from 71.9% 3.5% with ShieldLM. However, for Llama Guard 2, Gemini, Claude, DeepSeck, and 03-mini with Deepinception, for GPT-335/GPT--4 with Jailbroken, and for DeepSeek with CodeChameleon, the ratio increases, likely due to insufficient insertion of emojis in the one-shot example. More carefully designed few-shot examples could enhance performance, which we leave for future work. Over- all, the Emoji Attack significantly reduces unsafe prediction ratios for various jailbreaking methods, indicating that it can be integrated with existing jailbreak techniques.
WildGuard (Han et al., 2024): Another guardrail model that focuses sigh-sensitivity filtering.
Furthermore, we consider four advanced commercial state- of-the-art LLMs and two reasoning LLMs to gain further insight into how they respond to adversarial inputs:
GPT-3355, Gemini, Claude, DeepSeek, and o3-mini: We use the prompts in Chao et al. (2023) to assign a harmful score from 1 to 10. If the score is at least 5, we classify the response as unsafe.
We use the approach in Qi et al. (2024) to obtain a harmfulness score from I to 5, where any score of 3 or higher is labeled unsafe.
GPT-4:
Finally. among non commmrmraaaal (i.e., open source) Judge LLMs, WildGuard achieves the highest unsafe prediction ratio across different jailbreaks, yet still sees an approximate 23% reduction when facing our Emoji Attack. Among the commercial LLMs tested, GPT-4, the top performing model, also experiences 6.6% decrease.
Using these Judge L.MSs for testing, we ensure that our findings are representative of current safety pipelines in both open-source and commercial LLM ecosystems.
Attack Setting, We mcasure attack effectiveness using the unsafe prediction ratio, i.e., the proportion of harmful responses correctly identified as unsafe by the Judge LLM. lower ratio indicates that the Judge L.M is more success- fully misled. Therefore, when applying our Emoji Attack, greater drop in the unsafe prediction ratio implies a more effective adversarial strategy.
In contrast, the reasoning model DeepSeek is robust to emo- jis, while the reasoning model 03-mini remains sensitive to our attack. Given the unknown differences between these two reasoning models, it unclear what factors contribute to the improved robustness of DeepSeek. Nevertheless, we expect that strong reasoning models have the potential to provide strong foundation for Judge LLMs. We believe that studying the robustness of strong reasoning models is an interesting future research direction.
Note that Charmer (Rocamora et al., 2024) is not applicable to our setting, as it is specifically designed to attack target LLMs rather than to evade detection by Judge LLMs.
Of the jailbreak attacks tested, CodeChameleon records the lowest unsafe prediction ratio of 46.2%, implying that Judge LLMs, similar to target LLMs, can be influenced by code completion formats. When combined with our Emoji Attack, CodeChamcleon's 's ratio drops further to 35.2%. This shows
4.2. Emoji Attack for Enhancing Jailbreaks Against Judge LLM Detection
To demonstrate the effectiveness of our approach in real- world scenarios, we combine the Emoji Attack with jail-
7

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
Table Unsafe prediction ratio of various Judge L.MSs when evaluating existing jailbreaking prompts. "# prompts" denotes the number of successful jailbreaking prompts. The target LLM used to generate harmful responses is "gpt-3.5- turrbo.". We bold the lowest ratio for each Judge LLM. The results demonstrate that our proposed Emoji Attack significantly reduces the unsafe prediction ratio on average across all Judge LLMs lested. Notably, ShieldLM is particularly vulnerable to our Emojt Attack.
Judge LLMs
Allacks
prompts
Avg.
Llama Guard
Llama Guard 2
ShieldLM
WildCiuard
GPF:3.5
GPI-
Gemini
Claude
DeepSeek
03-mini
Deepinception
35.
33.3%
71.9%
71.9%
71.9%
86.0%
38.6%
59.6%
66,7%
50.9%.
58.6%
+ Emoji Attack
15.8%
47.3%
3.5%
29.8%
40.4%
86.0%
64.9%
70.2%
82.5%
66.7%
50.7%
ReNellm Emoji Attack
45.2%
69.9%
62.4%
82.8%
72.0%
92.5%
71.0%
72.0%
76.3%
80.6%
72.5%
93
33.3%
55.9%
22.6%
46.2%
46.2%
86.0%
46.2%
49.5%
60.21
51.6%
49.8%
Tailbroken
70.15:
73.1%
73.1%
84.3%
69.0%
90.4%
75.6%
57.4%
85,8%
78.7%
75.8%
+ Emoji Attack
53.8%
55.3%
39.1%
67.5%
75.1%
91.4%
73.1%
48.2%
84.8%
77.2%
66.6%
CrdeChammleon
23.4%:
41.5%
38.5%
47.8%
27.3%
73.7%
53.2%
55.1%
51.2%
49.8%
46.2%
205
+ Emoji Atack
12.2%
31.2%
18.5%
32.2%
21.5%
58.0%
43.4%
39.0%%
58.5%
37.1%
35.2%
Weighted Average
552
44.9%
56.7%
58.3%
69.2%%
$4.3%
84.1%
62.7%
59.2%
69.4%
65.4%%
62.45%
31.0%
45.7%
25.0%
46.9%
46.7%
77.5%
56.7%
47.3%
70.7%
56.9%
50.4%
Table 3. Unsafe prediction ratio across various Judge LLMs for different emojis. We use CodeChameleon as the baseline jailbreak method. and employ black-box Emoji Attacks with diverse set of emojis.
Judge LLMS J
Emoji
Llama Guard
Llama Guard 2
ShieldLM
WildGuard
GPT-3.5
GPT-4
Gemini
Claude
DeepSeek
03-mini
CodeChameleon
23.4%
41.5%
38.5%
47.8%
27.3%
73.7%
53.2%
55.1%
51.2%
49.8%
12.2%
31.2%
18.5%
32.2%
21.5%
58.0%
43.4%
39.0%
58.5%
37.1%
7.3%
14.6%
9.8%
16.6%
14.4%
92.7%
20.5%
20.0%
49.3%
18.5%
15.3%
32.5%
24.1%
35.0%
43.3% %
87.7%
43.8%
43.3%
62.6%
38.4%
22.7%
35.5%
29.1%
38.9%
30.0%
91.1%
42.9%
44.4%
47.3%
41.4%
9.8%
16.7%
10.8%
24.0%
57.4%
86.8%
52.5%
28.9%
82.8%
31.9%
23.3%
22.8%
27.2%
25.2%
38.8%
83.0%
33.5%
45.6%
73.3%
29.6%
that our attack can effectively enhance jailbreak attacks.
Only GPT-4 is extremely robust to emojis in general. Again, among the two reasoning models, we observe that DeepSeek is relatively robust compared to 03-mini.
Different Emojis. To assess the influence of various emo- jis on unsafe prediction ratios different Judge LLMs, we use CodeChameleon as the jailbreak baseline method and conduct block-box Emoji Attacks using four different emo- jis in Table 3. For open-source Judge LLMs, we observe decrease in the unsafe prediction ratio regardless of the emoji used. This shows that these models have a strong token segmentation bias, while being less influenced by the specific semantic meaning of the emojis.
4.3. White-box Emoji Attack
We assemble harmful responses from multiple sources to capture diverse range of real-world scenarios and adversar- ial attempts. Specifically, we sample 574 harmful responses from AdvBench (Zou et al., 2023), which span various cate- gories such as profanity and graphic content (ranging from to 44 words). We also include 858 seebreak-ggeerratee responses: 110 from LLM Self Defense (Phute al., 2024) and 748 from Red Teaming Attempts (Ganguli et al., 2022). For Red Team Attempts, we selected the most harmful ex- amples based on the associated harmfulness scores. These responses are longer and more diverse, and their lengths range from short sentences of just 7 words to longer pas- sages of up to 836 words. total, we collect 1,432 harmful responses. This variety ensures that test performance across broad spectrum of content complexity and linguistic diver-
In contrast, commercial LLMs show a more nuanced behav- ior. We see that the use of the innocent emoji signifi- cantly reduces the unsafe prediction ratio (except GPT-4), while the use of toxic emojis (c.g., the middle finger or the happy devil has the opposite effect in most cases. Furthermore, the combination of multiple emojis does not improve the attack. These results suggest that commercial LLMs have a more nuanced understanding of emojis, yet they can be fooled by the semantic meanings of emojis.
8

---

Emoji Attack:
Enhancing Jailbreak Attacks Against Judge LLM Detection
Table Unsafe prediction ratio different Judge LLMs under token segmentation bias and white-box Emoji Antakss.
Judge LLMs L
Prompt
Avg.
Llama Guard
Llama Guard 2
ShieldLM
WildGuard
GPT-3.5
GPT-4
Gemini
Claude
Default
81.3%
79.1%
78.4%
93.2%
58.3%
96.2%
91.3%
97.0%
84.4%
Token Segmentation Bias
64.6%
72.4%
40.0%
61.2%
78.9%
97.7%
92.2%
97.1%
75.5%
Emoji at Random Position
39.0%
55.9%
9.2%
60.9%
84.3%
98.4%
92.5%
97.6%
67.2%
Emoji at Optimized Position
35.1%
51.3%
3.0%
56.4%
87.7%
98.2%
92.2%
97.7%
65.2%
Table 5. Comparison of unsafe prediction ratios between our Emoji Attack and the GCG.
Attack
Llama Guard
Llama Guard 2
ShieldLM
WildGuard
CodeChameleon +
12.2%
31.2% 48.0%
18.5%
32.2%
CodeChameleon GCG
8.8%
90.7%
61.8%
sity. Examples are presented in Table 1.
We then apply this suffix to harmful responses in CodeChameleon via in-context learning. As shown in Ta- ble 5, this optimized GCG suffix effectively reduces the unsafe prediction rate when evaluated with Llama Guard. However, when evaluated against black-bos box Judge LLMs, the attack performance degrades and falls short of the suc- cess rate achieved by our Emoji Attack. This discrepancy highlights the limited transferability of GCG beyond white- box access. Conversely, our Emoji Attack demonstrates better generalization across various Judge LLMs, a crucial advantage for real-world black-bor moderation scenarios.
As shown in Table 4, we observe that all open-source Judge LLMs exhibit significant reductions in unsafe prediction ratios under both token segmentation bias and Emoji At- tack, demonstrating notable susceptibility to this type of bias. Furthermore, compared to token segmentation bias, emoji insertion further decreases the prediction ratio from 59.6% 41.3%. This suggests that emojis have more pro- nounced effect in reducing the detection capabilities of the Judge LLMs by introducing new emoji tokens. In addition, the proposed position selection strategy improves the effec- tiveness of Emoji Attack by identifying insertion positions. Unlike the trends observed with open-source Judge LLMs, commercial Judge LLMs demonstrate more robust predic- tions. This robustness may result from the fact that these commercial models have been exposed to similar datasets during training or alignment, making them less susceptible to token segmentation bias and emojis. However, when chal- lenged with unseen harmful outputs generated by jailbreak attacks (Table 2), commercial LLMs remain susceptible to our Emoji Artack. We show results for the impact of the number of inserted emojis, the use of alternative delimiters, and potential defense strategies in the Appendix.
5. Conclusion
In this work, we discuss previously overlooked token segmentation bias in Judge LLMs, which impacts the relia- bility of Al-driven safety risk detection. We introduce the Emoji Attack, an adversarial strategy that exploits this bias by embedding emojis within tokens, leading to a 12% re- duction in unsafe prediction rates across ten state-of-the-ar Judge LLMs in various jailbreak scenarios. Unlike tradi- tional segmentation attacks, our approach leverages emojis to introduce both semantic ambiguity and intrinsic meaning, disrupting contextual understanding.
4.4. Comparison with GCG Suffix
Although prior research has identified biases such as posi- tional bias in Judge LLMs (Zheng et al., 2023; Chen et al., 2024; Wang et al., 2024; Koo et al., 2024), few studies have addressed biases specifically within the context of safety risk detection. Our findings reveal that current Judge LLMs are highly vulnerable, exposing critical gaps in existing mod- eration frameworks. As LLMs continue to be deployed for safety- -rittcclii applications, addressing token segmentation bias is essential to improve robustness against adversarial attacks. Future defenses should account for both tokeniza- tion vulnerabilities and the semantic impact of non-textual artifacts, such as emojis, build more resilient systems.
We also compare our Emoji Attack with GCG (Zou al., 2023). We optimize a universal adversarial suffix using 30 randomly selected harmful responses from our dataset of 1,432, targeting the white-box Llama Guard, This optimiza- tion process yielded the following suffix:
multimedia ps $ ((Kkal {: [ [[CheckBoxCheckooss safe.; safe ViewSimple Website Identifier XVII safe!. safe
9

---

Emoji Attack:
Enhancing Jailbreak Attacks Against Judge LLM Detection
Impact Statement
Chao, P., Robey, A., Dobriban, E.,, Hassani, H., Pappas, G. J., and Wong, E. Jailbreaking black box large language mod- els in twenty queries. arXiv preprint 2023.
Our study identifies token segmentation bias in Judge LLMs and introduces the Emoji Attack. We show that this at- tack reduces harmful content detection rates across state- of-the-art Judge LLMs, revealing a critical gap in current moderation systems. These findings expose vulnerabil- ity in LLM-based content moderation. As AI systems are increasingly used for safety-critical tasks, understanding these weaknesses is essential. By systematically evaluating Judge L.LM vulnerabilities, this work contributes to better understanding of LLM behavior, which is hoped to motivate the development of more resilient moderation systems.
Chen, G. H., Chen, S., Liu, Z., Jiang, F., and Wang, B. Hu- mans or LLMs as the judge? a study on judgement bias. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing, pp. 8301-8327, November 2024.
Claburn, T. Meta's Al safety system defeated by the space bar. https: /ttp://wwwwwwwoo twweehhetterrrreee com/ com/2024/ 07 02/29/00 htte//etaaai_sssaeeeee
2024.
Acknowledgements
Ding, P., Kuang. J., Ma, D., Cao, X., Xian, Y., Chen, J., and Huang, S. A wolf in sheep's clothing: Generalized nested jailbreak prompts can fool large language models easily. In Duh, K., Gomez, H., and Bethard, S. (eds.), Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Longg Papers), pp. 2136-21611 2153, June 2024.
We acknowledge the U.S. Department of Energy, under Contract Number DE-ACO2-OSCHI 1231 for providing com- putational resources. We used the computational cluster provided by NERSC and LBNL's Lawrencium.
References
AI@ Meta. Llama 3 model card. https:
//github. . cwmmcoo////m.mmm /etom/meta--e htmm/mmtaalll 1llmmaallamme/aeeee /11ama3/blbbb/////// Main/MMODEL_CAEE md, 2024.
Ganguli, D., Lovitt, L.,, Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.0788 07858, 2022.
Andriushchenko, M., Croce, F., and Flammarion, N. Jail- breaking leading safety-aligned LLMs with simple adap- tive attacks. In The Thirteenth International Conference on Learning Representations. 2025.
Geisler, S., Wollschläger, T., Abdalla, M. H. I., Gasteiger, J., and Günnemann, S. Attacking large language models with projected gradient descent. In ICML 2024 Next Generation of Al Safery Workshop. 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Han, S., Rao, K., Ettinger, A., Jiang, L., Lin, B. Y., Lambert, N., Choi, Y., and Dziri, N. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals LLMs. In The Thirty-eight Conference on Neural In- formation Processing Systems Datasets and Benchmarks Track, 2024.
Cai, Z., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., Dong, X., Duan, H. Fan, Fei, Z., Gao, Y., Ge, J., Gu, C., Gu, Y., Gui, T., Guo, A., Guo, Q., He, C.,, Hu, Y., Huang, T., Jiang, T., Jiao, P., Jin, Z., Lei, Z., Li, J., Li, J., Li, L., Li, S., Li, W., Li, Y., Liu, H., Liu, J., Hong, J., Liu, K., Liu, K., Liu, X., Lv, C., Lv, H., Lv, K., Ma, L., Ma. R., Ma, Z., Ning. W., Ouyang, L., Qiu, J., Qu, Y., Shang, F., Shao, Y. Song, D., Song, Z., Sui, Z. Sun, P. Sun, Y., Tang, H., Wang, B., Wang, G., Wang, J.. Wang, J., Wang. R. Wang, Y.. Wang, Z., Wei, X., Weng, Q. Wu, F., Xiong, Y., Xu, C., Xu, R., Yan, H., Yan, Y., Yang, X., Ye, H., Ying, H., Yu, J., Yu, Zang, Y., Zhang, C., Zhang, I.., Zhang, P., Zhang, P.. Zhang, R., Zhang, S., Zhang, S., Zhang, W., Zhang, W., Zhang, X., Zhang, X., Zhao, H., Zhao, Q., Zhao, X., Zhou, F., Zhou, Z., Zhuo, J., Zou, Y., Qiu, X., Qiao, Y., and Lin, D. Internim2 technical report, 2024.
Hayase, J., Borevković, E., Carlini, N., Tramèr, F., and Nasr, M. query-based adversarial prompt generation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Hu, K., Yu, W., Li, Y., Yao, T. Li, X., Liu, W., Yu, L., Shen, Z., Chen, K., and Fredrikson, M. Efficient LLM jailbreak via adaptive dense-to-spars, constrained opti- mization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Huang, T., Hu, S., Ilhan, F., Tekin, S. F., and Liu, L. Virus: Harmful fine-tuning attack for large language models bypassing guardrail moderation. arXiv preprint arXiv:2501.14333 2025.
10

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
Inan, H., Upasani, K., Chi, J., Rungta, R., Iyer, K., Mao, Y., Tontchev, M., Hu, Q., Fuller, B., Testug- gine, D., et al. Llama guard: LLM-based input-output safeguard for human-Al conversations. arXiv preprint arXiv:2312.06674 2023.
Mangaokar, N., Hooda, A., Choi, J., Chandrashekaran, S., Fawaz, K., Jha, S., and Prakash, A. PRP: Propagating universal perturbations to attack large language model guard-rails In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 10960-10976, August 2024.
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., and Kang, D. Benchmarking cognitive biases in large lan- guage models as evaluators. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 517-545, Au- gust 2024.
Mehrotra, A., Zampetakis, M., Kassianik, P., Nelson, B., Anderson, H. S., Singer, Y., and Karbasi, A. Tree of attacks: Jailbreaking black-box LLMs automatically. In The Thirty-eight Annual Conference on Neural Informa- tion Processing Systems, 2024.
Kudo, T. and Richardson, J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Blanco, E. and Lu, W. (eds.), Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing: System Demonstrations, pp. 71, November 2018.
Ni, J., Qu, C., Lu, J., Dai, Z., Hernandez Abrego, G., Ma, J., Zhao, V., Luan, Y., Hall, K., Chang, M.-W., and Yang. Y. Large dual encoders are generalizable retrievers. In Gold- berg, Y., Kozareva, Z., and Zhang. Y. (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9844 9855, December 2022.
Langley, P. Crafting papers machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Pangakis, N., Wolken, S., and Fasching, N. Automated annotation with generative Al requires validation. arXiv preprint arXiv:2306.00/ 76, 2023.
Li, X., Zhou, Z., Zhu, J., Yao, J., Liu, T., and Han, B. Deepinception: Hypnotize large language model to jailbreaker. In Neurips Safe Generative Al Workshop 2024, 2024a.
Phute, M., Helbling, A., Hull, M. D., Peng, S., Szyller, S., Cornelius, C.,, and Chau, D. H. LLM self defense: By self examination, LLMs know they are being tricked. In The Second Tiny Papers Track at ICLR 2024, 2024.
Li, Y. Liu, Y., Li. Y., Shi, L., Deng, G., Chen, S., and Wang, K. Lockpicking LLMs: A logit-based jail- break using token-level manipulation. arXiv preprint arXiv:2405.1 13068, 2024b.
Qi, X., Zeng, Y., Xie, T., Chen, P.-Y., Jia, R. Mittal, P., and Henderson, P. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference Learning Representations, 2024.
Liao, Z. and Sun, H. Amplegeg: Learning a universal and transferable generative model of adversarial suffixes for jailbreaking both open and closed LLMs. arXiv preprint arXiv:2404.0m 07921, 2024.
Rocamora, E. A., Wu, Y., Liu, F., Chrysos, G., and Cevher, V. Revisiting character- level adversarial attacks for lan- guage models. In Forty-first International Conference on Machine Learning, 2024.
Liu, F., Feng. Y., Xu, Z., Su, L., Ma, X., Yin, D., and Liu, H. Jailjudge: A comprehensive jailbreak judge bench- mark with multi-agent enhanced explanation evaluation framework. arXiv preprint arXiv:2410.128555 2024a.
Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Erk, K. and Smith, N. A. (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1715-1725, August 2016.
Liu, X., Xu, N., Chen, M., and Xiao, C. Autodan: Gen- erating stealthy jailbreak prompts on aligned large lan- guage models. In The Twelfth International Conference on Learning Representations, 2024b.
Touvron, H. Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Llama-Team. Meta llama guard 2. https:
//github. httphuuu..oommmttttllaeee /prpp/uppllllleeee ama/ blob/mal http/maan//ieee weiiiinnlmmmaa Baeaaaa-aarrra///e MMODEL CARD md, 2024.
Lv. H., Wang, X., Zhang, Y., Huang, C., Dou, S., Ye, J., Gui, T., Zhang, Q., and Huang, X. Codechameleon: Person- alized encryption framework for jailbreaking large lan- guage models. arXiv preprint arXiv:2402. 167 2024.
Wang. P., Li, Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L.,, Liu, Q., Liu, T., and Sui, Z. Large language models are not fair evaluators. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9440-9450 August 2024.
Wei, A., Haghtalab, N., and Steinhardt, Jailbroken: How does LLM safety training fail? In Thirty-seventh Confer- ence on Neural Information Processing Systems, 2023.
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, L., Cao, Y., and Narasimhan, K. R. Tree of thoughts: Deliberate problem solving with large language models. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Yu, J., Lin, X., and Xing, X. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.102: 2023.
Yuan. Y., Jiao, W. Wang, W., tse Huang. J., He, P., Shi, S., and Tu, Z. GPT-4 is too smart to be safe: Stealthy chat with LLMs via cipher. In The Twelfth International Conference on Learning Representations, 2024.
Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models at evaluating instruc- tion following. In The Twelfth International Conference on Learning Representations. 2024.
Zhang, Y. and Wei, Z. Boosting jailbreak attack with mo- mentum. In ICLR 2024 Workshop on Reliable and Re- sponsible Foundation Models, 2024.
Zhang, 7., Lu, Y., Ma, J., Zhang, D., Li, R., Ke, P., Sun, H., Sha, L., Sui, Z., Wang, H., and Huang, M. ShieldLM: Empowering LLMs as aligned, customizable and explain- able safety detectors. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Com- putational Linguistics: EMNLP 2024, pp. 10420-10438, November 2024.
Zheng, L.,, Chiang, W.-L., Sheng. Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z.,, Li, 7.,, Li, D., Xing, F., Zhang, H., Gonzalez, E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Zhou, W. Wang, X., Xiong, L., Xia. H., Gu, Y., Chai, M., Zhu, F., Huang, C., Dou, S. Xi, Z., et al. Easyjailbreak: A unified framework for jailbreaking large language models. arXiv preprint arXiv:2403. /2/71, 2024.
Zou, A., Wang. Z. Carlini, N., Nasr. M., Kolter, Z., and Fredrikson, M. Universal and transferable adversar- ial attacks aligned language models. arXiv preprint arXiv:2307. 15043, 2023.
12

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
A. Attention Visualization of Token Segmentation Bias
Figure illustrates the impact of token segmentation on attention distributions. Segmentation results a greater number of sub-tokens, with distinct attention weights compared to the original sequence. particular, the segmented subtokens 'p" and 'ir' exhibit elevated cross-attention values compared the corresponding tokens 'port' and "air' in the original sequence. This alteration suggests change in the embedding space, which could weaken the association of the model with harmful signals and reduce the probability of unsafe predictions.
Default Prompt
Segmented Prompt
0.12
0.10
0.08
0.02
0.00
Bomb
the
air
port
Bo m b he a ir ort
Figure Visualization attention values default (left) segmented (right) prompts Llama Guard. The sub-tokens "p" and "ir" the segmented prompt exhibit higher correlations than the equivalent tokens in the default prompt, indicating shift in attention patterns.
B. Comparison between Offensive Phrases and Those Appending Emojis
Emojis introduce varied semantic information for LLMs. For example, the smiley emoji represents positive sentiment. The middle finger emoji conveys a negative or offensive sentiment. To demonstrate this, we visualize the changes in the unsafe probability for each offensive phrase when the emojis are added in Figure 7. These offensive phrases sorted in ascending order by unsafe probabilities for the original phrases. From this figure, positive emoji have high probability of decreasing unsafe probability. meaning that they tend to be predicted as safe. In contrast, phrases that include an offensive emoji tend to be predicted as unsafe.
observe that phrases that add
1.0
1.0
phrase
phrase
phrase+
phrase+
Unsafe probability
Unsafe probability
0.6
0.6
0.4-
0.4
0.2
0.2
0.0
0.0
200
100
300
400
Offensive phrases
Offensive phrases
(a)
(b)
Figure 7. Comparison of the unsafe probability between offensive phrases and those appending emojis: (a) safe and unsafe probabilitics by applying softmax to their logit values. Llama Guard is used here.
((b)
We compute the
13

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
C. Effect of the Number of Inserted Emojis
We assess how varying the number of inserted emojis influences the unsafe prediction ratio, as presented Figure Evaluating harmful responses on Llama Guard and Llama Guard 2, we compare the random insertion of emojis against our position selection strategy. The results reveal gradual increase in unsafe prediction ratios as more emojis are inserted, driven by the corresponding shift in embedding space that deceives the Judge LLMss Even with small number of emojis, the response can be subtly altered to evade detection, illustrating both the versatility and stealth of the Emoji Attack.
0.80
LG
Random Position
LG
Our Position
Unsafe Prediction Ratio
0.75
LG2
Random Position
0.70
LG2
Our Position
0.65
0.60
0.55
0.50
0.45
20
40
60
80
100
Number of Inserted Emojis
Figure The effect the number inserted emojis on unsafe prediction ratio. "Our Position" denotes the proposed position selection strategy.
D. Effect of Other Delimiters
To further explore token segmentation bias, evaluate harmful responses Llama Guard with various delimiters, as illustrated in Figure 9. Compared to default prompts without delimiters, including delimiters markedly decreases the unsafe prediction ratio, confirming that token segmentation bias can be induced in multiple ways. Additionally, incorporating our position selection strategy alongside these delimiters leads to an even more substantial reduction in the unsafe prediction
0.8
Default Prompt
Random Position
Unsafe Prediction Ratio
0.7
Our Position
0.6
0.5
0.4
0.3
0.2
?
&
%
Other Delimiters
Figure The effect of various delimiters on unsafe prediction ratio Llama Guard. The red dashed line denotes the prediction ratio of default prompts without delimiters.
14

---

Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
ratio, underscoring the effectiveness of selectively inserting tokens.
E. Potential Defense Strategy.
One potential defense strategy is to design prompts that filter out abnormal characters in the responses of the target LLM. Inserting a single delimiter across multiple tokens is relatively easy to detect. However, using different delimiters for various tokens complicates the detection by disrupting token patterns in less predictable ways. For example, we employ "gpt-3.5-turbo" the additional LLM filter to remove unnecessary symbols from harmful responses. As shown in Figure 10, when use mix of character "b" and smiley face emoji delimiter, the LLM filter generates benign response that differs significantly from the original harmful response. This benign response is classified safe by Judge LLMs, allowing the original harmful response to be presented to users. We leave the exploration of combination attacks and further defense strategies to future work. The results highlight the significant potential for developing token segmentation bias-based jailbreaking techniques.
Posb & pictub re ob e youb 3 rr rgb
3 enib
tab
Is
Post picture of your green beans
LLM Filter
Figure The illustration how complex delimiters evade filtering-based defense strategy. The underlined character "b" and smiley face emoji represent the inserted content. We use "gpt-3.5-turbo" as the LLM filter, which is prompted with the instruction "Remove all unnecessary symbols from the following response". The original harmful response is "Post picture of your go als"
15


Reviews:
[{"review_id": "1cice2gv2Y", "reviewer": "Reviewer_RkAj", "summary": "This paper introduces \"Emoji Attack,\" a novel technique exploiting token segmentation bias in Judge LLMs to bypass harmful content detection. The authors demonstrate that inserting emojis into text disrupts tokenization patterns and creates embedding distortions that significantly reduce the ability of safety models to detect harmful content. Through comprehensive experiments across eight Judge LLMs (Llama Guard, ShieldLM, WildGuard, GPT-3.5, GPT-4, Gemini, Claude), the attack achieves an average 14.1% reduction in harmful content detection. The method works through in-context learning without requiring direct model access, making it a practical real-world attack that enhances existing jailbreak techniques.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all the insightful comments. We have addressed your questions and comments below.\n\n**Q1: Defense Mechanisms.** \n\nPlease see our response to Reviewer i3cs. \n\n**Q2: Emoji Semantics and Its Impact on Attack Effectiveness.** \n\nWe agree that understanding the semantics of emojis and their influence on attack effectiveness is important. \nAs illustrated in Figure 6 in the Appendix, we observed that negative emojis increased the unsafe probability. However, categorizing emojis is challenging, as the semantic interpretation of emojis can be context-dependent or culturally variable [1]. For example, the smiling emoji with round eyes 🙂 may be perceived positively by older users but negatively by younger generations.\n\nWe attempted emoji categorization using `Llama Guard` itself, and the results differed from `ChatGPT-3.5`'s categorizations. This discrepancy suggests that emoji semantics vary across different LLMs, influenced by underlying training datasets and model parameter scales.\n\n[1] Zhukova M, Herring S C. Benign or Toxic? Differences in Emoji Interpretation by Gender, Generation, and Emoji Type[J]. Language@ Internet, 2024, 22(Special Issue): 74-108.\n\n\n**Q3: Commercial Models.** \n\nDue to the proprietary nature of these models, details about their architectures and training processes remain unknown to us. However, for open-source Judge LLMs, our results in Table 1 demonstrate that `Lama Guard 2` (built on `Llama-3-8B`) outperforms `Llama Guard` (built on `Llama-2-7B`). The improved robustness can be explained by the increased model parameter size and the extended training datasets. Similarly, `ShieldLM,` trained on `internlm2-chat-7B` using 14,387 query-response pairs, and `WildGuard`, built on `Mistral-7B-v0.3` trained on 86,759 examples, illustrate that larger and more diverse training datasets significantly enhance model robustness.\n\n**Q4: Human Perception Studies.** \n\nWhile we agree that assessing human perception could offer additional perspective, our focus is on attacking automated moderation systems such as Judge LLMs. Since human reviewers can likely detect emoji-laden content more easily, our threat model assumes scenarios where content volume or platform design limits human oversight. In such settings, automated systems often operate with minimal human intervention. We feel that it is valuable for the ML community to study how such automated systems can be attacked.  \n\n\n**Q5: Impact of Model Architectures, Sizes, and Training Approaches.**\n\nIt is difficult to isolate the effects of architecture, size, and training in controlled experiments, as these factors often vary simultaneously. It would require us to train LLMs from scratch, which is not feasible. However, as shown in Table 3, commercial LLMs tend to handle token segmentation bias more effectively, likely due to a combination of larger model sizes, more diverse training data, and advanced training techniques. However, we do not have sufficient evidence to connect improved robustness to specific architectural designs alone.\n\n**Q6: Multilingual Models.** \n\nThis is a really interesting question. To address this question, we run additional experiments on a Chinese toxic content dataset [2] using the `shenzhi-wang/Llama3.1-8B-Chinese-Chat`, an instruction-tuned language model for Chinese. We sampled 1,000 toxic examples and inserted smiley emojis at random positions, as Chinese characters cannot be split into smaller sub-units. The results show a decrease in the unsafe prediction ratio after emoji insertion, suggesting that the emoji attack is also effective in languages with different tokenization patterns, such as Chinese. \n\n\n| Attack | Unsafe Prediction Ratio |\n| :---- | :---- |\n| W/O Emojis | 17.1% |\n| 5 Emojis | 14.5% |\n| 10 Emojis | 12.6% |\n\n[2] Lu J, Xu B, Zhang X, et al. Facilitating fine-grained detection of Chinese toxic language: Hierarchical taxonomy, resources, and benchmarks. ACL 2023.\n\n\n**Q7: Optimal Defensive Strategies.**\n\nThank you for the question. We're not entirely sure what is meant by \"optimal defensive strategies\" in this context. If the intent is to suggest using Algorithm 1 to generate adversarial examples for adversarial training, then yes, Algorithm 1 could be used. We ran some preliminary experiments to test this idea; adversarial fine-tuning using such examples did not lead to significant improvements in robustness over the baseline results shown in **Q1**. We think that is not too surprising, since approximate adversarial examples are often sufficient to improve robustness.", "claims_and_evidence": "The paper's claims about token segmentation bias are well-supported by experimental evidence. Figure 2 clearly demonstrates significant detection reduction when tokens are split, while Figure 3 quantifies the correlation between embedding distortions and classification outcomes. Table 1 convincingly shows how the Emoji Attack enhances multiple jailbreak methods, with ShieldLM's detection rate dropping from 71.9% to 3.5% when combined with Deepinception. The methodical testing across different emoji types (Table 2) and insertion strategies further strengthens the evidence.\nThe cosine similarity analysis connecting embedding changes to classification outcomes is particularly compelling, establishing a causal mechanism for the attack's effectiveness. The authors also demonstrate that position-optimized emoji insertion (Table 3) consistently outperforms random placement.", "methods_and_evaluation_criteria": "The methodology effectively isolates token segmentation effects through controlled experiments comparing non-split, mid-split, and cs-split approaches. The evaluation uses an appropriate dataset of 402 offensive phrases and 1,432 harmful responses of varying lengths (2-836 words), ensuring results are robust across content types.\nThe surrogate model method (Algorithm 1) provides a principled approach to identifying optimal token split points, and the black-box attack implementation via in-context learning demonstrates real-world applicability. The cross-model evaluation approach is comprehensive, testing both open-source and commercial models to provide comparative insights on robustness.", "theoretical_claims": "The paper establishes a sound theoretical foundation linking token segmentation to classification errors. Definition 3.1 formalizes token segmentation bias, and Equation 3 provides a mathematical formulation for measuring embedding distortions using cosine similarity. The attention visualizations in Figure 5 offer mechanistic insights into how segmented sub-tokens alter attention patterns, supporting the theoretical claims.\nNo mathematical errors or oversights were identified in the theoretical analysis. The paper correctly applies the embedding distance metrics and properly interprets the results in the context of token segmentation bias.", "experimental_designs_or_analyses": "The experimental design effectively controls for variables to isolate the attack's effects. The authors systematically test different segmentation approaches, emoji types, placement strategies, and performance across multiple jailbreak techniques. Ablation studies in the appendix thoroughly examine how varying emoji numbers and different delimiter types affect performance. The cross-model evaluation reveals important differences in vulnerability between open-source and commercial models, with GPT-4 showing greater resilience.", "relation_to_broader_scientific_literature": "This article's method of inserting emojis into text to make Judge LLM make wrong judgments is a novel method. I have not heard of similar work.", "essential_references_not_discussed": "The paper would benefit from references to recent work on embedding space vulnerabilities in classification tasks, particularly from the NLP security literature. Research on emoji understanding and semantic interpretation in LLMs would provide context for the semantic ambiguity claims. Literature on defense mechanisms against adversarial attacks in the NLP domain would also strengthen the discussion on potential countermeasures.", "other_strengths_and_weaknesses": "Other Strengths\nThe identification of token segmentation bias represents a novel contribution to LLM safety research. Unlike previous work focusing on prompt-level or token-level jailbreaking, this attack targets a fundamental vulnerability in how Judge LLMs process tokenized inputs. This insight opens a new dimension for understanding model robustness.\nThe technical depth of the embedding analysis is impressive. The authors go beyond simply demonstrating the attack's effectiveness to provide a mechanistic explanation through cosine similarity measurements and attention visualizations. Figure 5 particularly enhances our understanding of how token segmentation alters attention patterns in the model.\nThe cross-model transfer capabilities make this attack particularly concerning. The consistent effectiveness across diverse model architectures (from open-source Llama Guard to commercial GPT-4) suggests the vulnerability is intrinsic to current LLM design rather than implementation-specific. The paper quantifies these differences rigorously, showing that while GPT-4 is more robust, it still exhibits a 6.6% reduction in detection capability.\nThe practical implementation via in-context learning represents a significant contribution. By demonstrating that the attack can be executed without model access or optimization, the authors highlight a genuine real-world threat. The one-shot example approach makes the attack accessible even to non-technical users, amplifying its practical impact.\n\n\nOther Weaknesses\nThe defense mechanism analysis is underdeveloped. While the appendix briefly discusses a potential approach using an additional LLM filter, this exploration feels preliminary and lacks rigorous evaluation. A more systematic investigation of countermeasures would significantly strengthen the paper, particularly exploring whether detection-time modifications to embedding space might mitigate these attacks.\nThe paper lacks sufficient analysis of emoji semantics and their relationship to attack effectiveness. While Table 2 shows performance across different emojis, there's no systematic categorization of emoji types (positive vs. negative, abstract vs. concrete) or investigation into whether semantic properties correlate with attack success. This analysis would provide deeper insights into why certain emojis are more effective than others.\nThe limited explanation for commercial model robustness represents a missed opportunity. Though the paper identifies that models like GPT-4 show greater resilience, it doesn't sufficiently explore the architectural or training factors that might contribute to this robustness. Understanding these differences could inform better defense strategies and more robust model designs.\nThe evaluation could benefit from human perception studies. While the paper thoroughly evaluates machine detection rates, it doesn't assess whether the emoji-laden content appears suspicious to human moderators. Given that human oversight often complements automated moderation, understanding human detectability would provide a more complete picture of the attack's real-world implications.", "other_comments_or_suggestions": "no", "questions_for_authors": "How does token segmentation bias interact with different model architectures, sizes, and training approaches? Is there evidence that certain architectural choices mitigate this vulnerability?\nHave you explored whether multilingual models exhibit different vulnerabilities to the Emoji Attack, particularly for languages with different tokenization patterns?\nCould your position selection algorithm (Algorithm 1) be adapted to identify optimal defensive strategies, such as robustness-enhancing fine-tuning targets?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0IzjqrvoTN", "reviewer": "Reviewer_FWCB", "summary": "This paper introduces \"Emoji Attack,\" a technique that exploits token segmentation bias to enhance jailbreak attacks against Judge LLMs. The authors demonstrate that inserting emojis into text can disrupt the tokenization process, causing embedding distortions that lead Judge LLMs to misclassify harmful content as safe. Through experiments on multiple state-of-the-art Judge LLMs, they show that their approach substantially reduces unsafe prediction rates, and bypassing existing safeguards.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all the insightful comments. We have addressed your questions and comments below.\n\n**Q1: Limited Description of Datasets.**\n\nThank you for pointing this out. If given the opportunity, we will include a more detailed description of the datasets in the paper. Below, we outline the key characteristics:\n\n* The 402 offensive phrases consist of short toxic expressions, typically 2–3 words in length. These include vulgar slang, sexual references, derogatory language, and references to illegal activities or fetishes.\n\n\n* The 1,432 harmful responses are composed of two parts:\n   *  574 harmful strings from AdvBench, covering a broad spectrum of harmful content such as profanity and graphic descriptions (lengths range from 3 to 44 words).\n    * 858 malicious responses generated via jailbreaks (110 from [1], and 748 from [2]). These responses are longer and more diverse, ranging from 7 to 836 words. For [4], we selected the most harmful examples based on the associated harmfulness scores.\n\n\nWe will also include a summary table with concrete examples from each category.\n\n[1] Phute M, Helbling A, Hull M, et al. Llm self defense: By self examination, llms know they are being tricked. In ICLR 2024 TinyPaper, 2024.     \n[2] Ganguli D, Lovitt L, Kernion J, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned[J]. arXiv preprint arXiv:2209.07858, 2022.\n\n\n**Q2: Position Optimization in Black-Box Emoji Attack.**\n\nIn the black-box emoji attack setting, we do not perform position optimization (we will clarify this in the manuscript). Fine-grained control over emoji insertion is not feasible because the inserted positions are determined by the target LLMs via in-context learning. As such, we cannot compute cosine similarities or directly optimize positions in the black-box scenario.\n\nDespite this limitation, we find that simply prompting the target LLMs to insert emojis within words (without fine-grained control over exact positions) is often sufficient to fool the judge LLMs, as demonstrated in Table 1. This highlights the practicality of the black-box attack, even without explicit position optimization.\n\n\n**Q3: Exploration in Other Languages.**\n\nThank you for this fantastic question. We are really excited to explore this question. Also, this is related to a question raised by Reviewer RkAj, and we conducted initial experiments to investigate the cross-lingual applicability of the emoji attack (we are committed to expand these results further).\n\nUsing the instruction-tuned Chinese language model `shenzhi-wang/Llama3.1-8B-Chinese-Chat`, we first confirmed that token segmentation differences exist in Chinese. For instance, the phrase “我们” and its space-separated variant “我 们” yield different token ID sequences: `[98739]` vs. `[37046, 220, 80578]`.\n\nWe then sampled 1,000 toxic examples from a Chinese dataset [3] and inserted smiley emojis at random positions within the sentences (since Chinese characters cannot be further segmented). The results show a decrease in the unsafe prediction ratio, indicating that the emoji attack remains effective in character-based languages like Chinese.\n\nThese findings suggest that token segmentation bias generalizes beyond English. If provided the opportunity, we would be glad to include these cross-lingual results in the final version of the paper.\n\n| Attack | Unsafe Prediction Ratio |\n| :---- | :---- |\n| W/O Emojis | 17.1% |\n| 5 Emojis | 14.5% |\n| 10 Emojis | 12.6% |\n\n[3] Lu J, Xu B, Zhang X, et al. Facilitating fine-grained detection of Chinese toxic language: Hierarchical taxonomy, resources, and benchmarks. ACL 2023.", "claims_and_evidence": "Yes, the claims made in the submission are generally supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "EmojiAttack does not contain formal mathematical proofs that require verification. The paper is primarily empirical in nature, focusing on experimental demonstrations of the token segmentation bias vulnerability and the effectiveness of the Emoji Attack.\n\nAs for theoretical claims, it includes several theoretical formulations and algorithms:\n\n1. Problem formulation in Section 3.1: The authors provide mathematical notation for how target LLMs and Judge LLMs operate, defining the prediction of tokens and the filtering process. These are standard formulations.\n2. Definition 3.1 of Token Segmentation Bias: A formula definition.\n3. Equation 3 for computing cosine similarities:  a straightforward application of cosine similarity.", "experimental_designs_or_analyses": "1. Token Segmentation Bias Experiments (Section 3.2): Here the authors test mid-split and cs-split on 402 offensive phrases. They compare baseline performance against two increasingly sophisticated segmentation methods, providing a clear progression of effectiveness. And Figure 2 appropriately visualizes results across four Judge LLMs, Figure 3 effectively demonstrates the correlation between cosine similarity and classification probability.\n2. Enhancement of Existing Jailbreak Techniques (Section 4.2):  Integrate of Emoji Attack with four established jailbreak methods.\n3. White-Box Emoji Attack (Section 4.3): Test token segmentation bias and emoji insertion on a dataset of 1,432 harmful responses.", "relation_to_broader_scientific_literature": "1. Jailbreaking techniques: While previous work like GCG  focused on optimizing tokens to bypass content generation LLMs, this paper uniquely targets Judge LLMs through token manipulation with emojis.\n2. Tokenization vulnerabilities: The paper builds on on character-level attacks, but identifies a new \"token segmentation bias\" specifically affecting Judge LLMs when delimiters alter tokenization.\n3. Judge LLM biases: This extends research on biases in evaluation modelsby discovering a previously unknown vulnerability affecting even commercial models like GPT-4.\n\nIn terms of the findings, this paper has proved that their methodology effectiveness compared with other methods", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Pros:\n1. Unlike many jailbreak techniques requiring complex optimization algorithms, the Emoji Attack is relatively simple to implement using in-context learning, making it particularly concerning from a security perspective.\n2. The authors demonstrate that token segmentation bias affects multiple Judge LLM architectures, suggesting this is a fundamental vulnerability rather than an implementation-specific issue.\n\nCons:\n1. There's limited information about the composition of the 402 offensive phrases and 1,432 harmful responses used for evaluation, making it difficult to assess how representative they are.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. In the black-box Emoji Attack implementation, how do you address position selection when you lack direct access to embedding functions? The paper demonstrates that emoji position significantly impacts effectiveness, but it's unclear how optimal positioning is achieved in the black-box scenario where you can't compute cosine similarities. Would a response detailing your approach for black-box position optimization change my assessment of the method's practical applicability?\n2. Does token segmentation bias persist when Judge LLMs evaluate text in languages other than English, particularly those with different tokenization patterns (e.g., character-based languages like Chinese or Japanese)? Evidence of cross-lingual vulnerability (or lack thereof) would enhance my understanding of how fundamental this vulnerability is to LLM architecture.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6zZe0YsMqP", "reviewer": "Reviewer_i3cs", "summary": "This paper proposes an Emoji attack to fool the judge LLM and thus enhance the attack power of jailbreaking. Emoji attack finds the position to insert the Emoji that can achieve the maximized segmentation bias. Empirical results show that Emoji can successfully bypass the judge LLM.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We evaluated two types of potential defenses: (1) LLM-based filtering and (2) adversarial training of Judge LLMs (as suggested by Reviewer RkAj). Below, we summarize our findings, with details provided in Figure 9 of the Appendix and additional tables.\n\n**1. LLM-Based Filtering.**\n We use `gpt-3.5-turbo` as a filter to sanitize the responses generated by target LLMs. The filter is prompted to remove abnormal characters and symbols, such as emojis or inserted delimiters. We find that this works relatively well when a single type of delimiter is used across a response. However, when we mix different delimiters (e.g., a combination of the character \"b\" and a smiley emoji), the filter struggles to reconstruct the original harmful intent and instead generates a benign-looking response. This sanitized output is then misclassified as safe by Judge LLMs, effectively allowing the harmful content to bypass moderation. This demonstrates that even an LLM-based defense can be tricked when the emoji attack is integrated with obfuscation techniques.\n\n**2. Adversarial Training of Judge LLMs.**\n We fine-tuned the `Llama Guard` using emoji-inserted toxic examples to improve its robustness. This does lead to higher unsafe detection rates overall, confirming that adversarial training can help mitigate the attack. However, when our emoji attack is combined with jailbreak methods such as Jailbroken or CodeChameleon, it still reduces the unsafe classification rate, even against the fine-tuned model. This indicates that the attack remains effective under adversarial training in certain settings.\n\nInterestingly, we also observe that when paired with jailbreaks like DeepInception or ReNeLLM, the emoji attack can sometimes increase the unsafe prediction rate after adversarial training. This suggests that the interaction between emoji-based perturbations and jailbreak prompts is non-trivial and worth deeper investigation.\n\nIn summary, our emoji attack demonstrates robustness across defensive strategies by:\n* Remaining effective against adversarially trained Judge LLMs when combined with specific jailbreaks.\n* Bypassing LLM-based filters by using compositional delimiters that degrade filter performance.\n\nWe see this as an important contribution toward understanding and evaluating the limitations of current defense strategies. \n\n| Attack | Llama Guard | Finetuned LLama Guard |\n| :---- | :---- | :---- |\n| Deepinception /+ Emoji | 35.1% / 15.8% | 47.4% / 52.6% |\n| ReNellm /+ Emoji | 45.2% / 33.3% | 51.1% / 64.5% |\n| Jailbroken /+ Emoji | 70.1% / 53.8% | 87.9% / 66.5% |\n| CodeChameleon /+ Emoji | 23.4% / 12.2% | 98.1% / 86.8% |\n\n**Experimental Setting:** We created a balanced fine-tuning dataset consisting of: (1) 1,432 unsafe responses as described in Section 4.3; (2) an additional 1,432 adversarially perturbed unsafe responses, each containing emojis inserted within every word; and (3) 2,864 safe responses sampled from the Huggingface dataset \"LLM-LAT/benign-dataset\". For efficient fine-tuning, we employed the Parameter-Efficient Fine-Tuning (PEFT) method following guidelines from the official Llama-Cookbook repository.", "claims_and_evidence": "Yes. The empirical results seem to support the claim.", "methods_and_evaluation_criteria": "The authors should consider the adaptive defense. Some very simple defense methods are to remove the emoji in the LLM's output or to ask the judge LLM first remove the emoji and then start judging.", "theoretical_claims": "No theoretical results.", "experimental_designs_or_analyses": "The evaluation should consider the potential defense.", "relation_to_broader_scientific_literature": "The paper could pose potential risks when using LLMs.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "Please refer to the evaluation metrics.", "questions_for_authors": "1. Could you show the results under the potential defensive methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1f0GMNjmAO", "reviewer": "Reviewer_JTBj", "summary": "This paper presents a jailbreak attack against judge LLM detection.\n\n**After rebuttal:** I read the author's rebuttal and most of my concerns are addressed. I am actively participating in reviewer-AC discussion to champion this paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for all the insightful comments. \n\n**Q1: Virus.**\n\nWe will cite the Virus paper. While both our work and Virus target judge LLMs, the settings and objectives differ. Virus attacks judge LLMs during the data filtering stage to preserve harmful content, which is subsequently used to fine-tune target LLMs and induce undesirable behavior. In that setup, the attacker has direct control over the data input to the judge LLM. In contrast, our setting assumes that judge LLMs are used post hoc to evaluate the safety of responses generated by target LLMs. We do not assume access to or control over the inputs to the judge. Instead, our attack modifies the outputs of the target LLM to evade judgment. \n\n**Q2: Limited Novelty.**\n\nWe disagree with the characterization that the contribution of our paper is limited to replacing spaces with emojis. While Claburn (2024) observed that inserting spaces can influence content-generation behavior in LLMs, their work did not systematically explore the implications of token segmentation for evasion attacks against judge LLMs.\n\nOur work is the first that evaluates token segmentation attacks in this context. By using emojis, which introduce both semantic content and tokenization shifts, we show a novel and practical attack vector that is effective across both white-box and black-box settings. This shows vulnerabilities of judge LLMs.\n\n**Q3: Automatic Optimization-Based Algorithm.**\n\nOur choice to use emojis is motivated by their unique properties: unlike spaces, emojis introduce both token segmentation and semantic perturbation. As shown in Figure 6 of the Appendix, this dual effect can meaningfully influence LLM behavior. We found that emojis are particularly effective at fooling judge LLMs, especially in black-box settings where we have limited control over exact insertion positions. We think this is an important first step, which hopefully motivates automated optimization strategies in future work.\n\n\n**Q4: Minor Perturbations Are Not a Fundamental Issue.**\n\nTo better understand the effect of minor perturbations, we compare the top-50 and top-100 next-token predictions for the phrases \"Education\", \"Educa tion\", and \"Educa😊tion\" using `meta-llama/Llama-3.1-8B`. We compute the overlap between the original and perturbed versions and observe that even a space split leads to roughly 25% divergence in the top-50 next-token predictions. This indicates that current LLMs are not robust to such minor segmentation changes.\n\n| Phrase | top-50 | top-100 |\n| :---- | :---- | :---- |\n| `Education` & `Educa tion` | 74% | 62% |\n| `Education` & `Educa😊tion` | 66% | 52% |\n\nWhile we agree that an ideal LLM might learn to represent split and unsplit tokens similarly, this remains a non-trivial challenge in practice, particularly when perturbations introduce both segmentation and semantic shifts, as emojis do. \n\nMoreover, even as modern LLMs improve their ability to handle word splits that produce uncommon subwords, they struggle when the split creates common and semantically meaningful subwords. For example, the word `smart` contains the subword `art`. When we split `smart` into `sm  art`, the resulting representation shifts toward that of `art`, which has a distinct meaning. To support this, we report the next-token prediction overlap among `art`, `smart`, and `sm art`. Notably, `sm art` shows higher overlap with `art` than with `smart`, indicating that the split meaningfully alters the model's predictive behavior. These kinds of collisions between semantically loaded subwords cannot be easily mitigated through data augmentation alone.\n\n| Phrase | top-50 | top-100 |\n| :---- | :---- | :---- |\n| `art` & `smart` | 12% | 15% |\n| `art` & `sm art` | 40% | 38% |\n\n**Q5: GCG Suffix.**\n\nTo enable a direct comparison with GCG-style attacks, we optimized a suffix using 30 randomly selected harmful responses from our pool of 1,432, targeting the white-box `Llama Guard`. The optimization yielded the following suffix:\n\n`multimedia ps $(\\'kal {: [[CheckBoxCheckBox=\"<? safe.; safe ViewSimple Website Identifier XVII safe!. safe`.\n\nWe then applied this suffix to harmful responses using ICL via CodeChameleon. As shown in the table below, this optimized GCG suffix effectively reduces the unsafe prediction rate when evaluated on `Llama Guard`.\n\n| Attack | Llama Guard | Llama Guard 2 | ShieldLLM | WildGuard |\n| :---- | :---- | :---- | :---- | :---- |\n| CodeChameleon \\+ 😊 | 12.2% | 31.2% | 18.5% | 32.2% |\n| CodeChameleon \\+ GCG | 8.8% | 48.0% | 90.7% | 61.8% |\n\nHowever, when evaluated against black-box judge LLMs, the attack's performance degrades and falls short of the success rate achieved by our emoji attack. This suggests that, while GCG can be effective in white-box settings, it suffers from limited transferability. In contrast, our emoji attack generalizes better across different judge LLMs. We think this property is important in realistic, black-box moderation scenarios.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes, but could be improved. See weakness.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes, the experimental design are sound and make sense to me. However it can be improved because it misses an important baseline. See weakness part.", "relation_to_broader_scientific_literature": "Recently LLM service provider, e.g., Meta, and IBM propose their guardrail model to detect harmful question. This paper proposes a jailbreak attack to show that the safety risk still exists despite these efforts.", "essential_references_not_discussed": "There is a method concerning the very same scenario need to be discussed:\n\nVirus: Harmful Fine-tuning Attack for Large Language Models Bypassing Guardrail Moderation\n\nI believe this is a concurrent paper with this paper and I know authors have no obligation to dicuss concurrent work. However, I still encourage the authors to discuss this paper because these two method focus on the very same problem (i.e., jailbreaking judge LLM) and use a very similar experiment setting. Discussing them all together can raise more attention and thinking among the community.", "other_strengths_and_weaknesses": "**Stengths:**\n\n1. While there are quite a lot of research in jailbreaking LLM, there is only a few available study trying to jailbreak the judge LLM. I believe this paper is the very early paper along this line of research. I believe this paper is important in this sense, because nowadays the appearance of diversified Guardrail models (i.e., judge LLMs) give the community a false sense of \"security\". This paper can serve to increase awareness of the risk. \n\n2. The storyline is very smooth and easy to understand. I especially like the illustration of Section 3.2, which clearly illustrate the concept of embedding distortions and its relation with token splitting. \n\n3. Extensive experiments on both open-sourced and close-soiurce LLM judge is done.  \n\n**Weakness:**\n\n1. The finding of using token segmentation to attack LLM is not a unique finding of this paper, but should be credited to Claburn (2024). Therefore, the contribution of this paper is to replace the normal spaces with emoji. In this sense, the novelty of this paper is quite limited. \n\n2. Given that the core contribution is to utilize emoji to replace space, It is unclear why using emoji instead of other character? Yes, the space might not be a best option to perturb the embedding, but so does the emoji could be not.  Instead of manually checking each class of character, I think an automatic otpimization-based algorithm would be more appreciable, and will have more contribution advancing the field. In this sense, the technical contribution of this paper is limited.  \n\n3. A minor perturbation to the words should not affect the classification in principle, and therefore as shown in the experiment, the attack sucess rate of emoji attack (as well as Token Segmentation) is not that high. The reason is that a well-trained LLM should still be able to classify the full token and its splitting part correctly. For example the next word embedding of \"Euca\" \"tion\" and \"Education\" should be roughly the same with growing ability of LLM.   Although such a perfect LLM is not available for now, however, I think this issue is not something very fundamental that cannot be solved with growing ability of modern LLM. \n\n4. The authors should compare with GCG. In the considered judge LLM jailbreak scenario, it can still be applied in my understanding.  Specifically, the attackers can optimize a suffix to elicit the classification of the judge LLM to be  \"safe\".  After optimizing the suffix, a similar  In-Context Learning method can be used to instruct the target LLM to output the optimized suffix.  \n\n\nSome of my weakness I mentioned (e.g., 1-3) might not be able to easily solved with rebuttal and experiment, but I would like to point out here as a reason of my rating.  Very likely I will keep my rating even after rebuttal because of the concern of novelty.  I hope the authors can understand. However, I am okay with its acceptance because the fluent writing and also it serves as a timely paper to provide better understanding of risk of guardrail moder.", "other_comments_or_suggestions": "See weakness.", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces “Emoji Attack,” a novel adversarial technique that exploits token segmentation bias in Judge LLMs by strategically inserting emojis into text. 

The consensus among reviewers was to accept the paper, albeit with some reservations. Three reviewers provided positive reviews, highlighting the paper’s practical significance, comprehensive empirical evaluation, and clear articulation of a real-world vulnerability in Judge LLM moderation systems. The main concerns involve the limited exploration of potential defenses, multilingual capability, and comparison with existing attacks. The authors' rebuttal largely addresses the reviewers' concerns. Reviewer i3cs has a negative recommendation. However, the review is vague and lacks details. The authors' responses also seem adequate. 

Overall, I find this submission is suitable for acceptance.
Decision: Accept (poster)