Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?

Authors: Egor Zverev, Sahar Abdelnabi, Soroush Tabesh, Mario Fritz, Christoph H. Lampert
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Instruction-data separation, ML Safety, LLM Safety, LLM Security, Indirect Prompt Injection, Large Language Models, Datasets

Abstract:
Large Language Models (LLMs) show impressive results in numerous practical applications, but they lack essential safety features that are common in other areas of computer science, particularly an explicit separation of instructions and data. This makes them vulnerable to manipulations such as indirect prompt injections and generally unsuitable for safety-critical tasks. Surprisingly, there is currently no established definition or benchmark to quantify this phenomenon. In this work, we close this gap by introducing a formal measure for instruction-data separation for single-turn language models and an empirical variant that is calculable from a model’s outputs. We also present a new dataset, SEP, that allows estimating the measure for real-world models. Our results on various LLMs show that the problem of instruction-data separation is real: all models fail to achieve high separation, and canonical mitigation techniques, such as prompt engineering and fine-tuning, either fail to substantially improve separation or reduce model utility.

Decision: accept (poster)
Reviewer scores: 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730373669675

thread_type:
official_review

<thread_object>
creator: reviewer_du3h

<content>
summary:
This paper explores the ability of large language models (LLMs) to distinguish between instructions and data within a given prompt. To evaluate this, the authors created a dataset where each sample contains {Task Prompt, Data Prompt, Probe Instruction, and Witness}. A perfect model would follow only the instructions from the task prompt while ignoring any instructions from the data prompt. If the model mistakenly follows the probe instruction, a "witness string" will appear in its output. By comparing the model’s behavior when the probe instruction is part of the task prompt versus when it appears in the data prompt, the authors assess its ability to separate instructions from data. Two evaluation metrics are introduced: the separation score and the utility score. The dataset and these metrics were used to evaluate GPT-3.5, GPT-4, and seven other models, ranging from 2B to 8B parameters. The paper also discusses three potential ways to improve model performance: prompt engineering, prompt optimization, and fine-tuning.

strengths:
1. The paper is well-structured, with clear problem definitions, case studies, and experimental results.
2. The study is comprehensive, covering problem definition, evaluation metrics, dataset creation, experimental evaluation, and potential methods for performance improvement.
3. The dataset and reasonable metrics proposed provide an effective way to evaluate the instruction-data separation capabilities of LLMs.

weaknesses:
1. One core contribution of the paper is the dataset; however, there are some questionable aspects regarding how it was built. As shown in Table 1, the "probe instruction" is appended to the end of the "data prompt," though they bear no semantic connection. Intuitively, this kind of example may not occur in real-world settings, creating input prompts that seem somewhat artificial. This raises concerns about whether the evaluation results truly reflect the model's ability to handle instruction-data separation in real-world usage. Moreover, the dataset creation process, as detailed in Appendix A, seems quite straightforward, being largely based on existing data and GPT-4, which furthers the aforementioned concern.
2. Some experimental setups and conclusions warrant more scrutiny:
    - **Experimental Setup**: In Tables 4 and 5, the baseline “Original” assigns the system prompt to the instruction argument, while the user prompt is treated as data. This setup seems problematic because it blends multiple instructions from the user and the system without clearly distinguishing what should be treated as instructions versus data, which may lead to input ambiguity. I suspect this confusion contributed to the low score for GPT-4 (20.8%) in Table 4. A more suitable baseline might be **PromptEng** method.
   - **Some conclusions appear misaligned with experimental results**:
       - In Line 450, the authors suggest fine-tuning significantly reduces utility, making it impractical. However, this conclusion seems premature. The poor performance of fine-tuning could be due to inadequate data quality or other subtle issues. Based on the current results, it’s too early to definitively state that fine-tuning is not a viable solution.
       - In Lines 461 and 497, the authors speculate that GPT-4's superior performance may be due to "principled differences in model architecture or training." However, the experimental data doesn’t robustly support this since models larger than 8B parameters were not included. Model size is a crucial factor that hasn’t been sufficiently considered. Including models like LLaMA3-70B, Qwen2.5-72B, or Mistral 8*7B would lend more solid support to the conclusions.

questions:
1. What’s the primary difference between studying "separation of instructions and data" and "prompt injection"? Why is it important to study them separately? What are the potential consequences if we don’t?
2. This is an open-ended question to encourage the author to share their perspective. In what practical scenarios do you think the ability to separate instructions from data is especially critical? The paper doesn’t seem to delve deeply into this consideration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730577685276

thread_type:
official_review

<thread_object>
creator: reviewer_z9dv

<content>
summary:
This paper motivates and formalizes the problem of instruction-data separation in LLMs - the ability to distinguish between instructions to be executed and data to be processed. The authors propose both a theoretical formal measure for instruction-data separation and a practical empirical metric for evaluating it. They introduce SEP, a carefully constructed dataset for testing instruction-data separation, and evaluate 9 popular LLMs using their methodology. Their results reveal that instruction-data separation is a significant problem in current LLMs, does not improve with model scale. These findings motivate the need for further research to address this limitation of LLMs.

strengths:
- The results are properly caveated and presented with appropriate skepticism.
  - I appreciate that the authors explain their results with skepticism. E.g. pointing out that the results of GPT-4 may be impacted by the fact that GPT-4 created the SEP dataset (page 8); acknowledging that the set of prompt templates was not exhaustive (page 9); etc.

- Well written.
  - The paper was a pleasure to read. It was logical and easy to follow.
  - I appreciate that each definition or result has coherent discussion following it.
  - The problem of instruction-data separation is also well motivated.

weaknesses:
- Some technical details are lacking.
  - See questions 1-3 below.

- Results are hard to make sense of.
  - As acknowledged by the authors, SEP performance varies widely between models (even between models of different scales from the same model family), as does the impact of the mitigations.
  - It is hard to draw conclusions from the results (Table 4, 5) as a result. The lack of clear patterns or trends makes it difficult to understand what factors contribute to better instruction-data separation in general.

questions:
1. What was the fine-tuning training objective?
    - I am specifically wondering if there was a dual objective to both achieve good separability and also good utility, or if only one of these was incentivized in the fine-tuning procedure.

2. How were the "artificial" system prompts (used for Gemma and Starling) determined?
    - I'm wondering whether there was some trial and error / evaluation on some validation set to, in an effort to get a system prompt that behaved in a certain way. This (limited) optimization pressure could introduce some bias in the resulting "artificial" system prompt.

3. What is a task vs a subtask? (section 4)
    - In general I thought that the dataset creation methodology could have included more details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705480197

thread_type:
official_review

<thread_object>
creator: reviewer_m5e4

<content>
summary:
This paper studies the problem of whether LLMs can separate instructions from data, which is important to the safety of LLMs. Specifically, this paper first introduces a formal measure for this problem, then proposes a new benchmark (i.e., SEP) to evaluate LLMs’ performance on this problem, and then conducts a study on the mitigation strategies of this problem.

strengths:
- The paper explores an interesting and important research direction.
- The paper proposes a new benchmark, namely SEP, to evaluate the problem of instruction-data separation.

weaknesses:
- There is a lack of detailed analysis on the evaluation results of different LLMs on SEP. For example, while authors report an abnormal phenomenon where better or larger models do not show stronger separation scores, they fail to provide either any detailed analysis or any explanation on the potential reason for this phenomenon.
- The study of mitigation strategies is not comprehensive. For example, while several existing fine-tuning techniques that target instruction-hijacking problems [1,2] can be naturally utilized to handle the problems in SEP, authors only include the vanilla fine-tuning technique in the study.

[1] Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. StruQ: Defending against prompt injection with structured queries. arXiv preprint arXiv:2402.06363, 2024.

[2] Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training LLMs to prioritize privileged instructions. arXiv preprint arXiv:2404.13208, 2024.

questions:
None beyond the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732389920284

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their feedback. We are encouraged that they found the topic of our work (m5E4: *“interesting and important”*, Z9dv: “well motivated”), appreciated the exposition (Z9dv: *“pleasure to read”*; du3H: *“well-structured”*) and found positive words about the dataset and evaluation metrics that are our main contribution (du3H: *“effective way to evaluate the instruction-data separation capabilities of LLMs”*). We also thank the reviewers for their constructive suggestions, which we have incorporated into a revised paper with improved experiments and discussions. We address individual questions in separate responses and use this comment to provide an overview of the changes in the revision.

**Expanded fine-tuning experiments**

In our experiments, fine-tuning achieved the highest separation score but significantly reduced model utility, making it impractical. Reviewers m5E4, du3H, and Z9dv asked about its simplicity, potential training issues, and whether we used a dual objective to balance separation and utility. We appreciate these suggestions and have addressed them in our revision:

1) We now fine-tuned models with three objectives: (1) original Supervised Fine-Tuning (SFT), (2) Direct Preference Optimization (DPO) on pairs of probe and no-probe data, and (3) SFT with a dual objective for both separation and utility.

2) We expanded hyperparameter search for the phi-3 and gemma models, which had lower scores initially in the original experiment.

We observed substantially increased separation for phi-3 and gemma-7b with SFT, making them comparable to other models, which also received a minor improvement (1-2%). Due to improvements in phi-3 and gemma, SFT's average separation increased from 81.8% to 94.4% and utility increased from 19.2% to 47%, though still well below other methods (62.2 -- 69.9%). On average, DPO slightly outperformed SFT in both separation (1.5% higher) and utility (2.3% higher). Double-objective training achieved 94.4% average separation and 47.7% average utility. Overall, we find the results quite consistent.
We now report DPO results in the main text due to its higher utility and practicality. Full fine-tuning experimental results are provided in Appendix B.4.
We have also expanded the discussion on fine-tuning in the manuscript accordingly.

**Improved discussion**

In the revision, we clarified statements that reviewers had asked for and added more discussions where needed. Specifically, we now discuss why larger models might have lower separation (m5E4, Z9dv), clarify our statement on GPT-4's performance (du3H), and rename our "Original" method to "Naive" to articulate its role in the discussion as an introduction to other mitigation techniques.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper studies the instruction-data separation problem in LLMs. It introduces a formal measure for this problem, proposes a new benchmark to evaluate the performance of LLMs on this problem, and suggests mitigation strategies for this problem.

+ The paper is well-written.
+ The experiments are comprehensive.

- Some of the results require more analysis.