SPIN: Self-Supervised Prompt INjection

## Abstract

Abstract Large Language Models (LLMs) are increasingly used in a variety of important applications, yet their safety and reliability remain major concerns. Various adversarial and jailbreak attacks have been proposed to bypass the safety alignment and cause the model to produce harmful responses. We introduce Self-supervised Prompt INjection (SPIN) which can detect and reverse these various attacks on LLMs. Just by injecting an adaptive defense prompt at inference-time, our method is simple, effective, and compatible with existing safety-aligned models. Our benchmarks demonstrate that our system can reduce the attack success rate by up to 87.9%, while maintaining the performance on benign user requests. In addition, we discuss the situation of an adaptive attacker and show that our method is still resilient against attackers who are aware of our defense.

## 1 Introduction

Large-Language Models (LLMs) have achieved great success in code generation (Rozière et al., ), question answering (Touvron et al., ), and task planning (Surís et al., ). However, they can be jailbroken by adversarial attacks and prompts (Zou et al., ; Zeng et al., ; Yuan et al., ), leading them to exhibit behaviors of manipulation (Carroll et al., ; Ji et al., ), deception (Park et al., ; Hazell, ), and explicit content (Kang et al., ). This poses an inherent risk as LLMs are widely deployed in many critical applications, like programming.

To align LLMs for safety, prior work focuses on training to make the model more robust. For example, Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., ) trains reward models based on human ratings, and is used to judge LLM outputs. DPO (Rafailov et al., ) trains model on direct preference optimization. To make train-time defenses robust against attacks, it is often required to train the model on the generated attacks, which can take significant resources to gather responses (Casper et al., ). In addition, training-time defenses have fundamental difficulties defending against unforeseen attacks and jailbroken prompt injection, due to a mismatched generalization between the smaller safety training data and the broader pre-training data (Wei et al., ).

Figure: Figure 2: Self-supervised Prompt Injections. Examples of inputs at each stage of the defense. The red text is the adversarial suffix. On the left side, all the malicious examples with the trigger attack are discovered or reversed. The repeat task fails to reproduce the adversarial suffix in red because the token sequence becomes too random. Even though it does repeat the sentence partially, the difference is enough that the detection system will detect it as being malicious. The blue under the ‘Reversal’ task represent our defensive prompt injections which reverses malicious attacks, but does not impact benign inputs. On the right side, we show the same benign question passing through the layers without issue. Our defense systems is shown to defend against attacks that would otherwise break alignment, while maintaining performance on benign queries.
Refer to caption: extracted/5933294/figures/examples_new.png

In this paper we introduce a method to defend against these attacks by self-supervised prompt injection, dubbed SPIN allowing us to achieve a defense that adapts to each attack online. Just as an attacker finds the right prompt injection to break the LLM, our approach will find the right prompt injection to detect and repair the input. Figure 1 shows our prompt injection defending an adversarial prompt injection.

Our key insight is that the prompts which override the natural guidelines of the LLM also degrade other capabilities of the model. By constructing various self-supervised tasks, we can detect attacks without needing to know the true labels of the input requests or examples. Our constructed self-supervised tasks also allow us to repair attacked user prompts that slip through detection. This is achieved by adding tokens to the start of the input and taking gradient steps to reduce the perplexity of the resulting string. By using self-supervised prompt injection at test time, we can defend against unseen attacks and even adaptive attacks given the online adaptive property. Figure 2 shows the results of the different tasks rejecting malicious attacks while letting benign requests through.

Our method is model agnostic and can be compatible with existing defense systems. Our self-supervised criteria do not depend on malicious or benign labels, thus can be used online at inference time on demand.

Visualizations, experiments, and qualitative results demonstrate that our self-supervised prompt injection significantly improves the LLM safety upon existing alignment defenses. On Advbench with Universal Adversarial Triggers (Zou et al., ), we are able to reduce Attack Success Rates (ASR) to $12.11\%$ and $0\%$ on Vicuna and Llama-2 respectively. Our method also can detect natural language based jailbreak attacks (Wei et al., ), adversarial instructions (Wei et al., ), and role-play (Shen et al., ), reducing ASR to $0\%$ on both models, fully protecting them. Our method also is shown to be resilient against attackers who are aware of our defense.

Figure: Figure 3: Self Supervised Detection of Jailbreak Attacks. We detect jailbreak prompts through self-supervised natural language tasks. Here we show the ’Repeat’ task and the ’Interject’ task that form the detection layer. We are able to verify if the prompt passes the tests since we know the answer beforehand, either repeating the entire input or mentioning that Paris is the capital of France. Since jailbreak prompts often inject unusual patterns into the text, the model will often underperform on our created self-supervised language tasks compared to a benign user query. Our method uses this difference for detection.
Refer to caption: extracted/5933294/figures/repeat_task_new.png

## 2 Related Work

Large-Language Models: LLMs are pre-trained models on a huge corpus of textual data. A mix of these will have open source weights (Touvron et al., ), while others are closed off and are only accessible through APIs (Liu et al., ). Studies have shown that various capabilities are learned at the pre-training stage (Zhou et al., ). Performance of these models is correlated with parameter size and train time (Hoffmann et al., ; Kaplan et al., ), however, larger models also exhibit more problematic behavior (Perez et al., ; Elazar et al., ) and even the fine-tuning of these large models on benign datasets can compromise their safety (Qi et al., ).

Alignment: Alignment is an inbuilt system in recent LLMs. Originally used for text summarization (Stiennon et al., ), the main motivation recently has been to align the goals of AI models with human values and morals (Leike et al., ; Hendrycks et al., ). It can be done through reinforcement learning through human feedback (RLHF) or even through other LLMs as judges (Ziegler et al., ; Lee et al., ). However, as long as alignment does not remove the harmful behavior entirely, there will exist prompts that can elicit that behavior (Wolf et al., ). Wei et al. () showed that two main issues of alignment come from conflicting objectives, and mismatched generalization.

Adversarial Attacks: Specific modifications to inputs can cause models to output incorrect or malicious responses. This first gained prominence with Szegedy et al. ()’s paper attacking image classification models. For vision tasks, these attacks can also be repaired through natural supervision as well (Mao et al., ). For Large-language models, research has been done compiling jailbreak prompts that can cause harmful responses (Gehman et al., ; Liu et al., ). These refer to user-created prompts, and usually involve a scenario that encourages the LLM to break away from the preset safety guidelines (Wei et al., ). Recent developments such as AutoDAN (Liu et al., ) and CodeChameleon (Lv et al., ) incorporate LLMs to generalize and automatically generate these malicious prompts for a variety of requests.

Universal Adversarial Triggers (UATs) (Wallace et al., ) has been one method to attack these models which embeds a sequence of tokens into the user prompt that will cause it to bypass alignment and safety guidelines. In particular, gradient-guided search (Guo et al., ) has proven remarkably effective in adversarially misleading the model. Zou et al. () show that these attacks are also transferable across models, allowing harmful generation even in closed-box models. In response, various defense methods have been suggested, however, many of them (Kumar et al., ; Li et al., ; Phute et al., ) rely on external LLMs to verify the safety of the request.

## 3 Method

We begin by describing a series of defenses that use self-supervision to defend against both handcrafted jailbreak attacks and optimization-based adversarial trigger attacks (Zou et al., ). We provide two layers of defense, detection and reversal, that complement each other. Both methods rely on self-supervised signals that we construct for language tasks. These signals are shown in Figure 3 being able to accurately distinguish between benign and malicious prompts. Lastly, we consider the scenarios where the attacker can adapt their attack in response to the defense, and empirically show that our methods are still robust.

### 3.1 Jailbroken Language Models

In a given user interaction, the user inputs the request ${\bf x}$, and the chatbot responds with response ${\bf y}$. The LLM generates text outputs by predicting the next token, sampling from a probability distribution conditioned on the input such that ${\bf y}\sim F(\cdot|{\bf x})$.

For attacks, the attacker wants to minimize the difference between the response $y$ and their desired goal ${\bf t}$. In this case, this would be malicious instructions prefaced by an agreement such as “Sure, here’s how”. To do this, they can add a set of tokens $a$ which causes the model to respond with their desired output. This results in an attack optimization goal of:

$$ ${\bf a}=\min_{{\bf a}}\mathcal{L}_{\text{attack}}(F({\bf x}+{\bf a}),{\bf t})$ (1) $$

Note here that ’+’ denotes the concatenation of two strings, and that the attack is appended to the end of the user request. For GCG attack (Zou et al., ), this optimization is through gradient descent. For natural language jailbreaks (Wei et al., ), this optimization is through human construction.

### 3.2 Detect Jailbreaks via Self-supervision

We find jailbreaks prompt to bypass the LLM alignment contains different structure than benign input. We find jailbreak input often leave a trace, such as degrades other capabilities of the LLM in order to achieve a successful attack. Motivated by this, we propose to construct tasks that we already know the groundtruth or the expected behavior, so that we can detect jailbreak if we observe a violation of those tasks. While there are many self-supervised task in natural language we can construct, here we show self-supervised tasks as defense measures.

#### Repeat:

One simple request of LLMs is to ask it to repeat back the input. For most requests, there is no issue at all, and it is only when the input is harmful or it is under attack that the LLM will have trouble doing so. We use Levenshtein Distance “$\text{lev}(a_{1:i},b_{1:j})$” between the original user request and the newly generated sentence. This distance is then weighted by the average length of the two strings, where $s(x)$ denotes the size of the string, to compensate for differences in request length.

$$ $\mathcal{L}_{\text{repeater}}({\bf x}^{\prime})=\frac{2\ \text{lev}({\bf x}^{ \prime},F({\bf x}^{\prime}))}{s({\bf x}^{\prime})+s(F({\bf x}^{\prime})}$ (2) $$

#### Interjection:

At the end of the user request, another method is to interject with a separate question where we know the correct answer. This utilizes the internal knowledge set of the model, testing if it can answer fluently. We hypothesis that if the LLM has been jailbroken, they also degrade their capability in ansering those questions, like “What is the capital of France?” By calculating softmax for the logits for ‘Paris’ as the next token, it provides the following loss function:

$$ $\mathcal{L}_{\text{interject}}({\bf x}^{\prime},{\bf y})=\frac{e^{P({\bf y}|{ \bf x}^{\prime})}}{\sum\limits_{{\bf v}\in{\bf V}}e^{P({\bf v}|{\bf x}^{\prime })}}$ (3) $$

Each of these loss functions then cluster the inputs that are harmful and benign. Figure 4 shows the different loss functions for the repeater and interjection tasks. By placing a threshold $T$ on each of these self-supervised tasks, we are able to detect when a model has its capabilities attacked.

### 3.3 Reversals

As a second layer for defending against adversarial attacks, we can even reverse attacks that slip through detection. To do so, we append additional tokens in front of the request ${\bf x}^{\prime}$, and aim to find a series of tokens that will restore the natural alignment of the LLM. This would allow it to work with existing defenses, and further strengthens the models against attacks. Our proposed method, the introduction of defense tokens into the prefix of the user prompt also leaves the core content of the request unchanged. One observation we made with the GCG adversarial triggers (Zou et al., ) was that their inclusion would drastically increase the total perplexity of the input. By defining our $\mathcal{L}_{autoreg}$ as the perplexity of the entire user input, we get the following objective:

$$ ${\bf d}=arg\min_{\bf d}\mathcal{L}_{\text{autoreg}}({\bf d}+{\bf x}^{\prime})$ (4) $$

After every $p$ stages of optimization, we run the completion of the new input request $\hat{{\bf y}}=F({\bf d}+{\bf x}^{\prime})$ and record if the response began with any of the common alignment refusals. To pass through, it must run through all $n$ steps without triggering any of the common denial prefixes. Once the optimal ${\bf d}$ is found, the final output of the model is $\hat{{\bf y}}=F({\bf d}+{\bf x}^{\prime})$.

Figure: Figure 4: Self-supervised loss and ROC Curve for benign and malicious inputs. We show $\mathcal{L}_{\text{repeater}}$ loss on the left and $\mathcal{L}_{\text{interject}}$ on the right. For the repeat task, we find the model can repeat benign queries from the TriviaQA task well, but fails to repeat consistently when the query is from Advbench. For the interject task, the model is asked to answer ’What is the capital of France?’ and the loss is for predicting whether the next token will be ‘Paris’. As seen by the malicious instruction prompts, the loss increases only with the addition of the suffix. The divergent self-supervised loss values between malicious and benign allow us to detect malicious query. For the ROC curves, benign inputs are categorized as negatives, and malicious as positive. Losses for these graphs are calculated using Llama-2, with the best thresholds for the loss being 0.89 and 6.55 respectively.
Refer to caption: x1.png

### 3.4 Total System

For an attack to be successful, it must pass through all the detection stages, and the generated response will have the defense tokens as the prefix. The multi-layered approach allows each defense system to plug up holes in the others, and adds to the variables that an attack must consider. Due to the compositionality of our self-supervised defense, users can easily remove some tasks to speed up the system.

### 3.5 Adaptive Attacker

Another key advantage of our defense is that it is also robust to adaptive attackers. If the attack ignores our defense strategy, then we will defend it. If the attackers consider our defense strategy and then attack, then they need to optimize to also respect the self-supervised tasks, therefore make their optimization harder due to the additional constraints.
Formally, an adaptive attacker will alternate between attacking and defending the same prompt until it converges:

$$ ${\bf a}=arg\min_{\bf a}\mathcal{L}_{\text{attack}}(F({\bf x}+{\bf a}),t)$ (5) $$

$$ ${\bf d}=arg\min_{\bf d}\mathcal{L}_{\text{autoreg}}({\bf d}+{\bf x}+{\bf a})$ (6) $$

This iterative process is computationally intensive, and has the equivalent result as a constrained optimization problem. We use the Lagrangian penalty method by adding a new penalty to the score function. For each new defense system, we can include it as a new multi-target optimization the attack must consider. With all the detection layers and reversal implemented, the adaptive attack would have a multi-target optimization problem as follows:

$$ $\displaystyle\mathcal{L}_{\text{adapt}}({\bf x}^{\prime},t,\lambda_{r},\lambda _{i},\lambda_{s},\lambda_{p})=\mathcal{L}_{\text{attack}}({\bf x}^{\prime},t)+ \lambda_{r}\mathcal{L}_{\text{repeater}}({\bf x}^{\prime})+\lambda_{i}\mathcal {L}_{\text{interject}}({\bf x}^{\prime})+\lambda_{p}L_{\text{autoreg}}({\bf x} ^{\prime})$ (7) $$

By allocating more optimization budget to satisfying these constraints, the resulting attack is also less efficient. There is a trade-off that the attacker must consider: by optimizing to bypass the detection layers, it also increases the coherence of the input and constraint the input the be benign. If the attacker does not optimize to bypass the detection layers, our defense algorithm will be activated due to the violation of our self-supervised constraints. Thus, adding our self-supervised constraint will result in a lose-lose situation for the attacker, and we will straightly increase the safety.

## 4 Experiments

### 4.1 Datasets

Advbench ‘harmful behaviors’ contains 520 malicious requests, which we then added adversarial triggers by following Zou et al. ()’s method of multi-prompt attack.

TriviaQA (Joshi et al., ) contains a series of trivia questions crafted by humans along with an evidence dataset that contains the answer. We evaluate our performance on the wikipedia validation set with one-shot learning and closed-book performance after reversal.

### 4.2 Models

Llama 2-chat is a chatbot released by Meta, finetuned for dialogue and aligned through human feedback (Touvron et al., ).

Vicuna-7b is an open-source chatbot released by LMSYS based on finetuning Llama 2 with conversations from ShareGPT (Zheng et al., ).

### 4.3 Attacks

Universal Adversarial Trigger. We implement Greedy Coordinate Gradient-based (GCG) search as done in (Zou et al., ). Each malicious request gets appended a suffix, and the suffix gets optimized for up to 500 iterations or until alignment is broken. Then the next request begins with the previous optimized suffix appended. In the case of Vicuna, the suffix injection is able to achieve an Attack Success Rate (ASR) of $100\%$.

Natural Language Jailbreak. Jailbreak Chat is a website for users to share handcrafted prompts that can bypass AI alignment. We take the top 5 upvoted jailbreak prompts and pair them each with 30 randomly selected Advbench requests for a total of 150 prompts. These prompts tend to be older attacks, and were published before Llama2 released.

Adversarial Instructions. These attacks bypass alignment by circumventing the priorities of the LLM as mentioned in Wei et al. (). We test the strongest attack combination presented in their paper including prefix injection and refusal suppression, and pair it with 150 Advbench samples.

Multiple Role-Play. By simulating role playing in conversation, attackers instruct the LLM to act out a role which is able to bypass safety guardrails. This type of attack is similar to ’Do Anyhing Now’ (DAN) attacks in (Shen et al., ). We pair the ’dev mode v2’ (Wei et al., ) attack with the same 150 malicious requests from Advbench.

Automated Jailbreaks. Code Chameleon (Lv et al., ), AutoDAN (Liu et al., ), and ICA (Wei et al., ), are all automated ways of generating natural language jailbreaks. The attacks commonly have a starting attack template that is gradually modified with the attack query, similar to GCG-search, to bypass defense systems. We pair each attack with 150 malicious requests from Advbench.

Figure: Figure 5: Multi-Benchmark Attack Success Rate (ASR). We show performance of our defense against multiple types of attacks and benign inputs (Except for TriviaQA, lower indicates better performance). Our defense is robust and dramatically lowers ASR across all attacks. Even in cases where alignment is effective, SPIN is shown to decrease the ASR further.
Refer to caption: x5.png

Figure: Figure 6: Adaptive Attack Success Rate. The attacker minimizes the loss with respect to a Lagrangian score function, which balances between attacking the alignment, and bypassing the defenses we created. Then we run the same defenses on the newly generated attack suffixes. We vary the $\lambda$ penalty score for layers of the defense and measure the attack success rate. While the adaptive attacks do lower the ability of the specific layer, it leads to easier detection by the other layers.
Refer to caption: x7.png

### 4.4 Results

Detection.
For our Repeater layer, we inject the string “Repeat only the following sentences exactly without changes: ” into the model with the entire input surrounded by quotation marks. We use the best threshold of 0.89 to filter out inputs that failed to correctly repeat.

For our Interjection layer, we chose the question to be “But first, answer only what is the capital of France?”. This gets appended to the end of the model, which leaves the system prompt unchanged. On benign cases, the model was able to answer correctly more than $99\%$ of the time, making it a reasonable question to ask. It passes the layer if the generation returns the answer within the first 10 tokens, and in practice, this gave similar results to using the loss threshold.

The ROC curves in figure 4 show that there is a clear delineation in loss between malicious and benign requests when asked to repeat back the entire string. With only the malicious requests, the LLM will identify the harmful content and refuse to repeat the sentence. When the adversarial tokens do appear, the repeated string will usually not include the suffix tokens, leading to a higher loss. This is because when predicting the next word, the actual token that follows is too random to be reliably chosen.

As for the interjection task, it makes sense that the benign and malicious prompts without the attack have similar losses. This is because the model still has all of its capabilities, and alignment will still work. Only when an attack disturbs the coherency of the input does the loss go up.

Reversal.
For each request, we append 5 tokens “! ! ! ! !” to the start of the user message. For 25 steps of optimization, at each stage the reversal calculates the top-256 tokens that best decrease the perplexity of the overall sentence. Then for our batch size of 50, we uniformly pick from one of the 256 tokens and substitute it in. The sequence with the lowest loss is then chosen. At every 5 iterations it checks if the generated output matches one of the common denials for Vicuna or Llama. In both models, the reversal alone is able to reduce ASR by nearly half.

Jailbreaks present an interesting scenario. Because these are handcrafted prompts, they usually involve role-play and bypass guidelines by misleading the priorities of the model. The sentence structure is coherent, and as a result, they have the same perplexity as benign inputs. This means that our reversal is less effective because minor perturbations do not affect the overall understanding by the model. Most of these prompts also tell the model to ‘Ignore all the instructions you got before’, which is needed to weaken alignment or safety reminders in the system prompt. In the case of the repeat task, this actually causes the attack to fail that detection layer, because the model no longer repeats the input, and the Levenshtein distance will be above the threshold. As seen in Figure 5, the repeat condition is strong enough for SPIN to prevent all attacks.

Performance on TriviaQA
Figure 5 shows that under the benign cases, the overall performance is not significantly affected. Because of how we chose the thresholds for SPIN detection, the benign cases passed through without any issues. The addition of tokens that lower the perplexity does not impact how the model answers the question.

### 4.5 Defense Aware Adaptive Attack

We present the adaptive attack success rates in Figure 6. The same defense is run, but depending on the $\lambda$ penalty chosen, the attacker can optimize the attack for certain layers. However, this usually comes with a lower ASR in other layers. This is because the attacker has less of a budget to tackle the original goal of breaking alignment.
Our defense system still reduces the attack success rate by up to 76%, even under an adaptive attacker, proving that the method is resilient.

### 4.6 Ablation Study

Figure: Figure 7: Ablation Attack Success Rate for UAT and Jailbreak. On the left we show performance across multiple combinations of the defense system. In our defense, each of the layers complements the others, so that when placed sequentially it catches the majority of attacks, and is robust against adaptive attackers. On the right we show ASR for only the reversal layer with increasing iterations, both Vicuna and Llama2 level off in terms of gains from increasing optimization steps.
Refer to caption: x8.png

Computational Overhead With the detection layers, on benign inputs the model goes through two iterations of token generation. As shown in Table 1, the interjection task and the repeat task causes inference to run up to 1.2x and 1.3x longer, respectively. Reversal is 4.5x longer than inference when doing 25 steps of optimization, and using less steps can significantly speed up inference, however it is important that attacks do not succeed for the sake of a faster response time. Compared to the cost of running the GCG attack, our reversal is still only $1\%$ of the cost.

**Table 1: Latency (Averaged across 100 samples). The table shows the latency times for various defense methods.**
| Method | Time (s) |
| --- | --- |
| Standard Inference | 2.80 |
| Repeat | 0.45 |
| Interjection | 0.89 |
| Reversal (25 steps) | 12.62 |
| Full Defense | 16.31 |

Steps of optimization. There is an inherent trade-off between optimization steps and computation speed. Looking at Figure 7, the increase in steps does lead to a lower ASR. However, when examining the defense as a whole, the reversal only comes into effect if it slips through the previous detection layers. Under the assumption that detection will remove a large portion of attacks, going beyond 25 for the marginal improvements seems less appealing.

## 5 Conclusion

In this paper, we show that self-supervised metrics are sufficient in detecting adversarial attacks, and that the attacks are also repairable by targeting perplexity. We introduce an inference time defense system to detect and repair input generation through prompt injection. Our defense works with existing models and alignment, while not requiring any additional training or fine-tuning. This ensures that our defense system can react to new types of attacks that were not present in the alignment training set. The creation of these self-supervised defenses allow for multiple variations that can detect attacks. The layered nature of the method also shows resilience against adaptive attackers.

## 6 Acknowledgements

This work was supported in part by multiple Google Cyber NYC awards, Columbia SEAS/EVPR stimulus award, and Columbia SEAS-KFAI Generative AI and Public Discourse Research award.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Why is the defense suffix appended at the beginning? Is there any ablation study to support this choice?
2.The "best thresholds" mentioned in the title (0.89 and 6.55) do not match the values shown in the figure (0.90 and 5.73). Can this inconsistency be clarified?
3. Since the thresholds appear to be fixed, when the dataset is updated, will these thresholds change across different datasets? 
4. In real-world scenarios, the "repeat" and "interjection" tasks may affect the outputs required for benign prompts. Is this jailbreak detection mechanism limited in practical use cases?
5. In Figure 5, it seems that the paper by Jain et al. (2023) is not cited. I could not find this reference in the bibliography.
soundness: 3
strengths: 1. The paper presents a novel self-supervised detection scheme, "repeat" and "interjection," based on the impact of jailbreak attacks on LLM capabilities.
2. Building on this detection method and generating prefixes based on perplexity, this work proposes a multi-layer defense mechanism with multiple components. Combining "repeat" or "interjection" with "reversal" to fix malicious prompts appears feasible, achieving a balance between overhead and effectiveness. Additionally, it considers the challenges of adaptive attacks under the defense mechanisms and predicts an increase in the difficulty of future attacks.
3. The paper evaluates a comprehensive range of attack methods and defense mechanisms, providing comparisons with various existing works.
summary: This paper proposes a method to automatically detect and repair attacker prompts, gaining insights from the relationship between malicious cues and model capabilities. It leverages self-supervision techniques to identify adversarial attacks and implements a inference-time multi-layered defense mechanism against adversarial trigger attacks. The method also aims to complicate the adaptive attacker by considering more constraints.
weaknesses: 1. The figures and captions in the paper are inconsistent. For example, in Figure 1, the caption mentions "The blue example," which does not exist in the figure.
2. The overall insight is not prominent enough, the reasons for selecting the "repeat" and "interrupt" tasks are unclear, and there is no other ablation about the prompts of these two tasks.
3. The related work section does not provide a comprehensive introduction to the existing defense mechanisms against jailbreak
4. The number of evaluation models is limited, focusing on only two 7B models.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: See weaknesses.
soundness: 4
strengths: 1. It is novel to detect jailbreakings by prompt injections. The method is well motivated to construct a task where we know the ground truth, and judge using a loss function whether the redirected input leads to the expected response. This seems to be an initial paper that uses prompt injections for good.

2. The authors design three filters and demonstrate their individual defense performance with the computational cost, and show that we could choose different combinations according to the defense case. 

3. The experiments are very comprehensive, highlighting the detection advantage. Most notably, it shows good defense performance against adaptive attacks. The authors carefully design adaptive attacks that optimize with the defense, and show that this optimization becomes much harder.
summary: This paper detects jailbreakings by task redirection. It introduces an additional inference by prompting the model to do something else (repeat the input, output the capital of France, etc). The detection works because (1) the model tends to do the new task if there are no jailbreakings (2) the model tends to reject to respond when there is a jailbreak. Experiments show an improved security against jailbreakings, and a flexible choice of the three proposed filters. The repeat filter asks the model to repeat the instruction, and the detection loss is the Levenshtein Distance between the original user request (ground truth) and the newly generated sentence. The interjection filter redirects the model to answer “What is the capital of France?” with the loss as the logits for ‘Paris’ as the next token. The reversal layer optimizes a prefix to minimize perplexity, which is much larger when there is a jailbreaking.
weaknesses: 1. The title is misleading, and it seems to be proposing a new prompt injection attack. However, the paper aims to detect jailbreakings using the idea of prompt injection. Also, I think only the interjection is doing prompt injection as there are two conflicting instructions. Repeat is adding a higher level instruction that treats the original instruction as the data. Reversal is not for detection - it is a prevention defense by optimizing prefixes to have less perplexity. A clearer way to present is necessary.

2. For Table 1, can you explain why repeat and interjection costs less than standard inference? I thought it requires an additional inference, so the overall computation should be more than doubled.

3. The defense assumes the model is vulnerable to prompt injections. What if the model is secure against them? Does it invalidate some proposed defense filters? GPT-4o mini implemented with the instruction hierarchy defense against prompt injections could be a good model to test.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Could the authors elaborate on the process of threshold calibration for the repeat and interjection tasks? Specifically, how robust are these thresholds across different LLMs and configurations?
2. How does SPIN generalize across more LLMs, as mentioned in the weaknesses?
3. How does SPIN perform with prompts or adversarial inputs from specialized domains (e.g., medical, legal)? Are additional defenses required for domain-specific attacks, or does SPIN generalize effectively across diverse content areas?

Update: I have reviewed the authors' feedback and revised my score accordingly.
soundness: 3
strengths: The proposed method leverages adaptive prompts, which dynamically align with attacks, rather than requiring training-time adjustments or extensive resources for defense, offering an innovative defense mechanism against unforeseen and adaptive attacks.
summary: The paper introduces a defense mechanism designed to detect and counteract jailbreak prompts. The proposed approach, Self-supervised Prompt Injection (SPIN), employs adaptive prompt injection at inference time, allowing it to detect and mitigate adversarial prompts by leveraging self-supervised tasks. By injecting defense prompts, SPIN effectively reduces the success rate of attacks while preserving performance on benign inputs. The paper demonstrates that SPIN achieves a significant reduction in attack success rates, even against adaptive attackers aware of the defense mechanism. Extensive evaluations on benchmarks, including Advbench, indicate the robustness of SPIN, as it shows a notable reduction in attack success rates for both Vicuna and Llama-2 models.
weaknesses: 1. The proposed method is built on an argument that "We find jailbreak inputs often leave a trace, such as degrading other capabilities of the LLM in order to achieve a successful attack." However, this claim is not strictly verified. The paper conducts experiments on Llama-2-chat and Vicuna-7b, which, from today's perspective, do not have strong performance compared to models like the Llama 3 series. Thus, if the evaluations and the claim are based on Llama-2-chat and Vicuna-7b, it may hold true that injecting anything—not specifically jailbreaks—will influence the models' capabilities, but not true for other advanced models. The authors should demonstrate the generalization of their argument. This is critical as the proposed method is built on this argument.
2. Another fatal weakness is that the paper mentions predefined thresholds for the repeat and interjection tasks to distinguish between benign and malicious inputs. However, it is unclear how well these thresholds generalize across different models or contexts, and it seems the paper does not discuss how it decides the thresholds either. In practice, the choice of threshold can have a severe impact on the defense performance, as we can infer based on Figure 4.
3. Based on the computational overhead discussed in Section 4.6, the proposed method introduces additional (multiple) LLM inference rounds. Therefore, it should also be compared with other baselines that have similar computational overhead. For example, guardrail methods like the LlamaGuard [1] series have shown strong performance on jailbreak attacks. Thus, the paper should include a comparison with these guardrail defenses.

[1] Inan, Hakan, et al. "Llama guard: Llm-based input-output safeguard for human-ai conversations." arXiv preprint arXiv:2312.06674 (2023) \
[2] https://huggingface.co/meta-llama/Meta-Llama-Guard-2-8B \
[3] https://huggingface.co/meta-llama/Llama-Guard-3-8B

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. I believe the paper should clarify the specific type of attacks it defends against. Generally, prompt injection and jailbreak attacks are distinct. If SPIN is designed specifically for jailbreak attacks, it should state this clearly rather than introducing things like "detect and reverse these attacks (various adversarial and jailbreaks)" in Abstract or “Figure 1 shows our prompt injection defending an adversarial prompt injection.” If SPIN can also defend against prompt injection, it should be compared with more defenses, such as Known Answer Detection [1] and StruQ [2].

[1] Formalizing and Benchmarking Prompt Injection Attacks and Defenses

[2] StruQ: Defending Against Prompt Injection with Structured Queries


2. I recommend including FPR and TPR (ROC curves) in the evaluation. Current evaluation results only contain ASR. Based on the results in Figure 4, the interject task shows a relatively high FPR (over 0.2 at the best threshold). Therefore, metrics like FPR should also be presented in the evaluation to assess whether SPIN results in significant utility loss.

3. The computational cost of the proposed method may be impractical. First, both the repeat and interjection techniques require an additional query, which can be a substantial cost for high-traffic LLM applications (e.g., ChatGPT). Additionally, while the paper states that "detection will remove a large portion of attacks," Figure 7 shows a noticeable gap in ASR with and without reversal, especially on less robust models (e.g., Vicuna). In such cases, the computational overhead of reversal becomes excessive.

4. The evaluation is somewhat lacking. The experiments were conducted only on Vicuna and LLaMA2. I suggest including more models, such as LLaMA3, Mistral, etc., for broader evaluation. Additionally, since the reversal modifies the prompt structure, I would like to see tests on more datasets to assess the extent of utility loss caused by the reversal rather than just using TriviaQA.

Typo: Figure 4 shows "Best Threshold = 5.73" for interject tasks, but the caption states it as 6.55.
soundness: 2
strengths: 1. The proposed algorithm is straightforward and easy to implement, and the paper provides a clear description of the algorithm.

2. The experiments tested various attacks and evaluated results on different datasets.

3. An adaptive attack is introduced, along with evaluation and analysis.
summary: This paper introduces SPIN, a new defense against various LLM attacks. SPIN detects these various attacks by constructing self-supervised tasks and reverses them by adding an optimized prefix to the inputs. Experimental results demonstrate that SPIN effectively defends against different types of LLM attacks.
weaknesses: 1. The paper needs to clarify which specific type of attack it defends against, such as jailbreak, prompt injection, or both.

2. The paper should include additional experiments to show FPR and TPR, especially FPR on benign data. Furthermore, it should evaluate on more recent models.

3. The computational cost of the proposed method may be impractical.

Meta-review: The reviewers and the Area Chair (AC) acknowledge and appreciate the efforts made to improve the paper. However, the reviewers expressed concerns about the proposed technique, particularly the doubling of costs for each query. Additionally, the generalization of the threshold remains a significant issue, and the choice of models used in the evaluation was also questioned. Given these observations, the AC recommends resubmission after addressing all the concerns raised by the reviewers.

Decision: Reject