Venue: NAACL 2025

# IHEval: Evaluating Language Models on Following

the Instruction Hierarchy

Zhihan Zhang\({}^{\text{\textcircled1,2}}\), Shiyang Li\({}^{2}\), Zixuan Zhang\({}^{2}\), Xin Liu\({}^{2}\), Haoming Jiang\({}^{2}\),

**Xianfeng Tang\({}^{2}\), Yifan Gao\({}^{2}\), Zheng Li\({}^{2}\), Haodong Wang\({}^{2}\), Zhaoxuan Tan\({}^{1,2*}\), \({}^{1}\)Yichuan Li\({}^{2,3*}\), Qingyu Yin\({}^{2}\), Bing Yin\({}^{2}\), Meng Jiang\({}^{1,2}\)**

This work was done when Zhihan, Zhaoxuan, and Yichuan were interns at Amazon.

###### Abstract

The instruction hierarchy, which establishes a priority order from system messages to user messages, conversation history, and tool outputs, is essential for ensuring consistent and safe behavior in language models (LMs). Despite its importance, this topic receives limited attention, and there is a lack of comprehensive benchmarks for evaluating models' ability to follow the instruction hierarchy. We bridge this gap by introducing **IHEval**, a novel benchmark comprising 3,538 examples across nine tasks, covering cases where instructions in different priorities either align or conflict. Our evaluation of popular LMs highlights their struggle to recognize instruction priorities. All evaluated models experience a sharp performance decline when facing conflicting instructions, compared to their original instruction-following performance. Moreover, the most competitive open-source model only achieves 48% accuracy in resolving such conflicts. Our results underscore the need for targeted optimization in the future development of LMs.

## 1 Introduction

Instruction-tuned language models (LMs) are increasingly deployed as interactive services across various applications OpenAI (2023; Yang et al., 2024; Bi et al., 2024). To ensure consistent performance and safety, developers typically seek to regulate the model's behavior, such as fine-tuning the model on responding to certain instructions Touvron et al. (2023), using post-processing techniques to edit model outputs Song et al. (2024), and detailing system messages to impose constraintsAnthropic (2024b). However, in real-world applications, these pre-defined regulations frequently struggle to cover the full range of possible user inputs. For instance, users may request tasks beyond the model's intended scope OpenAI (2024), or the integrated tools may return unexpected content Zhan et al. (2024). The LM may risk misbehavior if higher-level instructions, such as regulative system messages, are overridden by subsequent conflicting inputs.

This highlights the need for LMs to possess an inherent capacity to follow an **instruction hierarchy**, where instructions of high-level regulations are always prioritized over low-level inputs. The order of priority - ranked from highest to lowest - should be: _system messages_, _user messages_, _conversation history_, and _tool outputs_. This hierarchy was first proposed by Wallace et al. (2024) and aligns with typical practices when using language models. A model that adheres to this hierarchy would be able to spontaneously resolve instruction conflicts by prioritizing higher-order ones, which reduces the dependency on complex regulation methods while also mitigating the risk of misbehavior1.

Footnote 1: We target scenarios where users can’t modify developer-set system messages (e.g., ChatGPT). In contrast, when users can customize prompts, conflicts are rarer unless by mistake.

Despite its significance, the instruction hierarchy paradigm does not receive much attention in LM research and evaluation. In some models, system messages - an important tool for giving high-level instructions - are either not supported Riviere et al. (2024), not distinguished from user messages Mistral (2024), or exhibit limited variation during Ivison et al. (2023). While many recent models have supported multi-level inputs, the related training details are rarely disclosed. A notable advancement in this area comes from OpenAI's study Wallace et al. (2024), but their evaluation was limited to GPT models and unreleased proprietary data, focusing solely on safety-related instructions. This constrains its general applicability to a wider range of use cases. To date, there remains no comprehensive benchmark to evaluate how well different LMs adhere to the instruction hierarchy.

In order to bridge this gap and highlight

the vital role of the instruction hierarchy, we create IHEval, a comprehensive benchmark for **I**nstruction **H**ierarchy **Evaluation**. It is designed with the following characteristics:

1. [leftmargin=*]
2. **Diverse scenarios**: Consisting of 3,538 examples and nine tasks, it spans four key scenarios involving hierarchical instructions: rule following, task execution, safety defense, and tool use.
3. **Comprehensive input hierarchy:** It covers four types of input: system messages, user messages, conversation history, and tool outputs.
4. **Instruction alignments and conflicts**: It includes both settings where (1) low-priority inputs align with high-level regulations, and (2) low-priority inputs contain additional instructions that conflict with those regulations.
5. **Varied task difficulties**: It offers various task difficulty settings by adjusting the strictness of the instruction phrasing, such as intensifying conflicts by requiring the model to exclusively follow specific instructions.
6. **Programmable evaluation**: All tasks are evaluated programmatically, ensuring the efficiency and reproducibility of the evaluation process. We evaluate a variety of mainstream LMs using IHEval and observe several key insights: (1) LMs struggle to prioritize high-level instructions when conflicts arise, with open-source models showing less than 50% accuracy in resolving these conflicts. This performance significantly lags behind both GPT-4o and their original instruction-following accuracy, as shown in Figure 2; (2) Even without conflicts, model performance on hierarchical inputs is inconsistent with the single-input reference setting; (3) Models' handling of conflicts is easily influenced by superficial factors like the strictness of instructions, and does not scale effectively with model size. These findings suggest that current LMs are not fully optimized for following the instruction hierarchy, leading to performance degradation or even unsafe behavior. We hope that our study can spark deeper research into this direction.

We summarize the main contributions of this work as follows:

* We design a comprehensive evaluation for assessing LMs' compliance with the instruction hierarchy, covering diverse scenarios where LMs face instructions of different priorities.
* We collect a benchmark to support this evaluation, including settings where hierarchical instructions either align or conflict, all of which

Figure 1: Four categories of the instruction hierarchy and the corresponding priority orders of instructions. Conflict instructions are shown in red. Models are expected to follow the instruction with the higher priority.

Figure 2: Results of mainstream LMs on IHEval. The reference setting represents original task performance without hierarchical inputs. We observe large performance drops when models face conflicting hierarchical instructions.

 

are programmatically evaluated.
* We evaluate a wide selection of LMs and find that they are not sufficiently optimized for the instruction hierarchy, highlighting potential risks in their real-world applications.

## 2 Related Work

### Evaluation on Instruction Following

The ability to follow instructions is a crucial assessment of instruction-tuned LMs. Early research in this area adopted a straightforward approach that leverages an expert LM (_e.g._, GPT-4) to holistically judge the quality of a model's response to an instruction Dubois et al. (2023); Zheng et al. (2023). More recent work focused on disentangling instruction-following evaluation from other factors, such as response detailedness and factuality, by proposing more fine-grained assessments on whether the response adheres to the constraints specified in the user query. Some studies, for instance, required LMs to follow strict rules regarding response formats Zhou et al. (2023); Mu et al. (2023); Li et al. (2023), while others designed case-specific constraints to regulate the content of model outputs Jiang et al. (2023); Qin et al. (2024); Wen et al. (2024). Recent studies also explored scenarios where instructions are embedded within the task input to assess whether LMs can correctly differentiate between instructions and data Zverev et al. (2024); Yi et al. (2023). In contrast to these works in the general domain, researchers in LM safety focused on whether models can effectively reject malicious instructions, whether directly provided by attackers Schulhoff et al. (2023); Toyer et al. (2024) or injected via external information Zhan et al. (2024); Debenedetti et al. (2024). Despite the rich amount of work in this area, none of them systematically analyzed the LM's ability in the instruction hierarchy. Notably, IHEval includes various scenarios where LMs face hierarchical inputs, especially those with conflicting instructions, bridging a gap in the current evaluation of LMs.

### System Prompts in LMs

System prompts Ramlochan (2024) are commonly employed to guide LMs' behavior from a high level. System prompts typically define the LM's role, task, output format, and safety guidelines, all of which are intended to be followed throughout the entire interaction. In many models, system prompts have been introduced as a separate input field from the user instruction OpenAI (2023); Touvron et al. (2023); Mukherjee et al. (2023), but details about its training process - such as the types and diversity of system prompts used - are rarely disclosed. Subsequent research demonstrated that system prompts can be used to improve the performance of LMs in general-domain instruction following Zheng et al. (2023), personalized response generation Lee et al. (2024), rule adherence Lu et al. (2024), and defending jailbreaks Zou et al. (2024). Inspired by this line of work, we investigate whether LMs consistently prioritize system prompts over user instructions and extend this evaluation to the broader context of instruction hierarchy.

During the development of IHEval, a concurrent work Qin et al. (2024) introduced SysBench which evaluates LMs' adherence to system prompts. Compared to Sysbench, IHEval features a more comprehensive evaluation of the instruction hierarchy concept, encompassing system prompts, user instructions, conversation history, and tool outputs. Moreover, IHEval is fully equipped with programmatic evaluation, offering better cost-efficiency and reproducibility than the GPT-based evaluation used in SysBench.

## 3 IHEval

DefinitionIn this paper, we denote **inputs** to be the text segments that the model receives, which may contain both **instructions** that control the model's behavior, and **data** that the model needs to process. IHEval is designed around the instruction hierarchy, which assigns priority to instructions from four types of input: _system messages_, _user messages_, _conversation history_, and _tool output_, ranked from highest to lowest priority. We define _hierarchical inputs_ as input sequences composed of more than one type of input, such as a sequence that includes both a system message and a user message. When facing instruction conflicts, we refer to the higher-priority instruction as the _main instruction_, which defines the primary task the model should prioritize. The _conflicting instruction_ refers to the lower-priority instruction whose request is incompatible with the main instruction.

Task SettingsFor a comprehensive evaluation, we design three input **settings** for each IHEval task:

* **Aligned Setting**: The model receives hierarchical inputs where all low-priority inputs align with the highest-priority instruction. This tests 

the model's ability to process hierarchical inputs in normal scenarios without conflicts.
* **Conflict Setting**: There are conflicts among different priorities of instructions within a hierarchical input. Models are expected to follow the instruction hierarchy to resolve conflicts.
* **Reference Setting**: We notice that a model's response to hierarchical instructions is affected by both its original task performance and its ability to follow the instruction hierarchy (IH-following). To better assess IH-following performance, we add a reference setting that isolates the original task performance by removing hierarchical inputs. Specifically, the model is evaluated in a standard single-input setting, where all hierarchical instructions from the _aligned_ setting are merged into a single user message.

Task DesignIHEval tasks are selected to encompass a diverse range of application scenarios and input types. We focus on tasks where LMs perform well to minimize the impact of original task performance on IHEval scores. We prioritize datasets with human-annotated labels or reliable programmatic evaluation. Conflicting instructions are drawn from tasks likely to confuse LMs in following the main instruction, based on heuristics and trials on sample data. As a result, a total of nine tasks is created and grouped into four categories based on the type of content in the instructions:

* [leftmargin=0pt]
* **Rule Following**: Instructions specify formatting rules for model outputs. We adapt data from IFeval Zhou et al. (2023) into our _single-turn_ task, where the original data is split into formatting rules (system message) and user queries (user message). We then craft incompatible formatting rules to create conflicting user instructions. The single-turn data is further extended to a _multiturn_ setting by crafting both a response to the first turn and a follow-up user message. Data crafting in this category is initially performed by Claude Anthropic (2024a), after which we manually review all AI-written messages and re-write low-quality ones.
* **Task Execution**: In this category, the system message outlines a specific NLP task that the model should perform on the user's input, while the user message may provide a conflicting instruction that requests a different task. This category covers typical NLP tasks that span _extraction_, _generation_, and _classification_.
* **Safety Defense**: Following the setting of TensorTrust Toyer et al. (2024), this category simulates the model as a security system. The system message requires the model to grant access only if the correct password is entered. Normal user inputs involve password attempts, but conflicts arise when malicious users launch adversarial attacks to make the model respond with "Ac

Figure 3: IHEval covers four categories and nine tasks. Detailed examples and instructions are in Figures 8–16.

 

cess Granted" (_Hijack_ task) or attempt to extract the system message that contains the password (_Extraction_ task).
* **Tool Use**: This category focuses on tasks where the model needs to call external tools to complete the user's request. We simulate a tool call and the corresponding tool output, where the tool output may either align with the user's request or contain conflicting instructions. Such instructions can be _intrinsic_, _i.e._, originating from the tool-retrieved content itself, or _injected_ by an external attacker and is concatenated with the normal tool-retrieved content. The illustration of all tasks is listed in Figure 3, with data examples and task instructions shown in Figures 8-16, respectively. Data sources and statistics are outlined in Figure 4. Details about data collection for each individual task and the motivation of task selection are provided in Appendix A.

Task DifficultiesIHEval introduces multiple difficulty levels for each task by crafting instructions with different imperative **strictness**. This approach not only provides a comprehensive evaluation of model performance but also reduces the randomness brought by the phrasing of instructions. The stricter version of instructions requires the model to exclusively adhere to the given instruction (_e.g., do not output any other text besides the Spanish translation_ in the translation task). All these instructions are shown in Figures 8-16.

EvaluationIHEval evaluates models based on their performance in completing the main instruction, as outlined in Figure 4. For example, when the system message requests the model to extract verbs from the user's input, the evaluation metric is the F-1 score which compares model-extracted verbs to the ground-truth list. Any execution of the conflicting instruction - translating the input text into Spanish - negatively impacts performance, as it diverges the model output from the target defined by the system message. For tasks that are not evaluated by exact-match accuracy (excluding safety tasks that check the defense success rate using the whole model response), we calculate both a **strict metric** and a **loose metric**, following the practice in IFEval Zhou et al. (2023). The strict metric assesses the model's entire output, while the loose metric allows minor variations by evaluating outputs that omit the first line, last line, or markdown syntax, selecting the best-performing version. The loose metric accommodates brief introductory phrases (_e.g., I'm sorry, but I can only extract verbs from your message_) that explain the model's behavior. However, overly interactive responses - such as asking for clarification or answering both instructions - are discouraged, as they treat the hierarchical instructions with the same level of priority (see Appendix B for a more detailed discussion). The final score is averaged across difficulty levels and, when applicable, across strict and loose metrics. As previously mentioned, the reference setting decouples a model's baseline task performance from its IH-following ability. To quantify this distinction, we calculate the score difference (\(\Delta\) in Table 1) between the reference setting and the other two settings. Specifically, we report both the _mean difference_, which reflects the model's average IH-following performance (where smaller performance drops indicate better IH-following), and the _mean absolute difference_, which captures performance fluctuation between single-input _vs._ hierarchical-input settings.

## 4 Experiments

In this study, we evaluate 13 widely used LMs from five different model families, including both proprietary and open-source models: GPT (3.5-turbo, 4o-2024-0806, 4o-mini-2024-0718, OpenAI, 2023), Claude-3 Haiku, Sonnet, Anthropic, 2024a), LLaMA-3.1 (8B, 7OB, Dubey et al., 2024), LLaMA-3 (8B, 7OB), Mistral (7B-v0.3, Large-2407, Mistral, 2024), and Qwen-2 (7B, 72B, Yang et al., 2024). The decoding temperature is set to 0 to ensure deterministic outputs.

Figure 4: The original data source, the evaluation metric, and the data size of each task.

 



### Main Results

The performance of select LMs is shown in Table 1, with full results available in Tables 5-10. We highlight the following key findings:

**Models exhibit inconsistent performance when conventional tasks are structured as hierarchical inputs.** Comparing the aligned setting (hierarchical inputs) to the reference setting (original task performance) reveals significant performance fluctuations in all models except GPT-4o and Qwen2-72B, with at least 4 points of absolute difference. For instance, when switching to hierarchical inputs, LLaMA-3.1-70B experiences a performance decline in eight out of nine tasks, averaging a 13-point drop. Smaller-scale models show even greater variability, often experiencing performance drops of more than 10 points (Tables 5-10). This inconsistency suggests that LMs are less optimized for hierarchical inputs compared to the standard single-input setting.

**Models struggle in utilizing the instruction hierarchy to resolve conflicts.** All models experience a notable performance drop in conflict settings, indicating a failure to follow the high-priority instructions when they conflict with low-priority ones. Despite a 22-point drop from its aligned setting, GPT-4o remains the best performer in handling instruction conflicts, likely reflecting OpennAI's fine-tuning efforts on the instruction hierarchy as described in Wallace et al. (2024). Although other tested models perform comparably to GPT-4o in reference and aligned settings, they fall significantly behind in the conflict setting, which suggests a lack of training on following the instruction hierarchy. Owen-2 emerges as the second-best with a 48% accuracy, though more recent models like LLaMA-3.1 and Mistral-Large claimed themselves to be the new state-of-the-art on other general benchmarks like MMLU Hendrycks et al. (2021). Besides, compared to results on SysBench Qin et al. (2024), IHEval reveals a larger performance gap between aligned and conflict inputs, which effectively uncovers the limitations of current LMs in following the instruction hierarchy.

### Performance by Model Scale

We group the LMs by model family and plot their performance on reference, aligned, and conflict settings in Figure 5. We have the following findings based on the trends:

\begin{table}
\begin{tabular}{c l c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Setting**} & \multicolumn{3}{c}{**Rule Following**} & \multicolumn{3}{c}{**Task Execution**} & \multicolumn{3}{c}{**Safety Defense**} & \multicolumn{3}{c}{**Tool Use**} & \multirow{2}{*}{**Avg.**} & \multicolumn{2}{c}{\(\Delta\)} \\  & & Single. & Multi. & Ext. & Gen. & Class. & Hijack & Extract & Intrinsic & Injected & Mean Abs. \\ \hline \multirow{3}{*}{GPT-4o} & reference & 89.0 & 86.5 & 90.0 & 78.0 & 100 & 99.5 & 100 & 90.2 & 94.0 & 91.9 & - \\  & aligned & 85.6 & 86.8 & 87.3 & 73.9 & 100 & 99.2 & 98.7 & 88.6 & 99.0 & 91.0 & -0.9 & 2.1 \\  & conflict & **49.5** & 51.0 & **77.2** & 38.3 & **99.7** & **91.2** & **96.7** & **63.8** & **62.5** & **70.0** & -21.9 & 21.9 \\ \hline \multirow{3}{*}{GPT-4o mini (2024-0718)} & reference & 84.5 & 86.1 & 90.5 & 78.4 & 99.6 & 99.1 & 99.4 & 89.3 & 79.0 & 89.6 & - \\  & aligned & 82.3 & 80.2 & 84.0 & 72.0 & 100 & 98.6 & 98.7 & 82.7 & 59.0 & 84.2 & -5.4 & 5.5 \\  & conflict & 33.9 & 35.7 & 47.7 & 31.1 & 41.1 & 70.3 & 95.5 & 43.6 & 0 & 44.3 & -45.2 & 45.2 \\ \hline \multirow{3}{*}{Claude-3 Sonnet} & reference & 80.9 & 83.9 & 84.9 & 76.9 & 100 & 87.1 & 85.5 & 87.1 & 87.0 & 85.9 & - \\  & aligned & 68.4 & 69.5 & 77.4 & 79.8 & 100 & 97.6 & 97.2 & 85.3 & 91.0 & 85.1 & -0.8 & 7.2 \\  & conflict & 10.8 & 21.1 & 2.3 & 29.7 & 9.8 & 46.6 & 60.1 & 56.9 & 39.0 & 30.7 & -55.2 & 55.2 \\ \hline \multirow{3}{*}{LLaMA-3.1} & reference & 88.3 & 88.4 & 89.1 & 77.0 & 100 & 99.3 & 99.7 & 89.0 & 100 & 92.3 & - \\  & aligned & 82.9 & 76.6 & 84.3 & 59.5 & 100 & 95.8 & 96.2 & 20.3 & 94.0 & 78.8 & -13.5 & 13.5 \\  & conflict & 14.3 & 24.3 & 0 & 15.2 & 6.2 & 24.4 & 25.2 & 2.2 & 14.0 & 14.0 & -78.3 & 78.3 \\ \hline \multirow{3}{*}{Mistral-Large (2407)} & reference & 83.6 & 85.2 & 85.2 & 78.5 & 100 & 99.2 & 98.4 & 88.3 & 69.0 & 87.5 & - \\  & aligned & 81.7 & 87.1 & 76.0 & 78.3 & 100 & 97.7 & 99.1 & 77.9 & 79.0 & 86.3 & -1.2 & 4.0 \\  & conflict & 25.2 & **60.0** & 11.0 & 20.2 & 78.4 & 23.9 & 18.8 & 13.9 & 13.5 & 29.4 & -58.1 & 58.1 \\ \hline \multirow{3}{*}{Qwen-2 72B} & reference & 81.4 & 85.0 & 74.9 & 75.0 & 100 & 97.6 & 98.4 & 83.9 & 92.0 & 87.6 & - \\  & aligned & 82.1 & 81.3 & 73.4 & 75.3 & 100 & 97.5 & 97.8 & 77.6 & 86.0 & 85.7 & -1.9 & 2.1 \\ \cline{1-1}  & conflict & 35.8 & 39.5 & 53.7 & **58.4** & 99.5 & 36.8 & 34.7 & 26.2 & 46.0 & 47.8 & -39.7 & 39.7 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Results of select LMs on IHEval. Full results are in Tables 5–10. \(\Delta\) is the score difference from the reference setting, including both the mean difference (signed) and the mean absolute difference. Red scores indicate \(|\Delta|>5\). _Single._ and _Multi._ refer to single-turn and multi-turn tasks in the Rule Following category. _Ext._, _Gen._, and _Class._ refer to extraction, generation, and classification tasks in Task Execution. The best performance in the conflict setting is marked as **bold** and the second-best is underlined.



1. **Improved Performance with Model Scale:** Across all three settings, the larger model consistently shows better performance on IHEval, which aligns with established scaling laws. As models become larger, their performance on aligned settings gets closer to or matches the reference scores. This implies that larger models, while improving general instruction-following abilities, have also effectively mastered the ability to handle aligned hierarchical inputs.
2. **Increasing Gap Between Aligned and Conflict Settings**: Despite improved performance on the conflict setting, most models, except GPT and Qwen-2, exhibit a larger gap between the aligned and conflict settings as scale increases. Some models even exhibit inverse scaling on resolving conflicts, _e.g._, Claude-Haiku outperforms Claude-Sonnet on 5 out of 9 tasks, as shown in Table 6. This indicates that models' abilities to handle conflicting instructions do not scale as effectively as general instruction-following capabilities, which again suggests a lack of model training in following the instruction hierarchy.

### Performance by Instruction Strictness

To explore the impact of instruction strictness on how models handle conflicting instructions, we calculate the average model score for each strictness level in conflict settings. According to Table 2, there is a clear trend that **performance improves when the high-priority instruction has the stricter demand, but decreases when the conflicting instruction is stricter**, indicating a strong correlation between instruction strictness and model behavior. However, this behavior is undesired: models should follow instructions based on their priorities in the instruction hierarchy, not the tone or strictness of the wording. These findings suggest that current models are not sufficiently aware of the instruction hierarchy and their behavior is easily influenced by superficial factors.

### Prompting LMs to Follow the Hierarchy

Given that current LMs lack inherent awareness of the instruction hierarchy, can we explicitly convey this principle to them through prompt engineering? To answer this question, we prepend the following prompt to the system message that states the priority of the instructions:

```
[General Response Guideline]Alwaysfollowtheinstructionhierarchyprioritizesystemmessageovereverythingelse,followedbyuser'scurrentmessageandthenconversationhistoryandtoolupts.Ifnstructionsconflict,resolvethembysillowetheonewiththehighestpriority(System>User>History>ToolOutputs).[EndofGeneralGuideline]
```

Surprisingly, as shown in Table 3, this additional prompt does not bring noticeable improvements to model performance. This suggests that teaching LMs to follow the instruction hierarchy is not a trivial task: **Dedicated training efforts are needed

\begin{table}
\begin{tabular}{c|r r r r r r r r r r r r} \hline \multirow{2}{*}{**Conflict Main Ins.**} & \multicolumn{4}{c}{**Rule**} & \multicolumn{4}{c}{**Task Execution**} & \multicolumn{4}{c}{**Safety**} & \multicolumn{4}{c}{**Tool Use**} \\  & Multi-turn & Extract & Gen. & Class. & & & & & & Instrinsic & Inject \\  & First. & Both. & _weak_ & _strong_ & _weak_ & _strong_ & _weak_ & _strong_ & _weak_ & _strong_ & _weak_ & _strong_ \\ \hline _weak_ & 41.0 & 12.3 & 31.8 & 9.3 & 28.3 & 10.4 & 25.9 & 21.5 & 30.6 & 33.3 & 33.3 & 13.1 & 36.5 & 18.7 \\ _strong_ & 55.4 & 16.1 & 37.1 & 16.3 & 50.6 & 24.5 & 47.0 & 38.8 & 43.7 & 45.2 & & \\ \hline \end{tabular}
\end{table}
Table 2: Model performance in conflict settings with different strictness of instructions. “Main Ins.” and “Conflict Ins." refer to the main instruction and the conflicting instruction, respectively. In multi-turn Rule Following, “First.” and “Both.” are settings where conflict instructions appear in the _first turn_ or _both turns_ (see Figure 9 for examples).

Figure 5: The trend of IHEval performance by model scale.

 

rather than superficial prompt engineering.**

### Model Performance in Different Conflicts

We explore model behavior under various alignment and conflict scenarios using the multi-turn conversations from the Rule Following task. The model's objective is to follow the formatting constraints in the main instruction when responding to user queries. Figure 7 illustrates examples of different scenarios.

We begin by comparing the single-input reference setting to those with aligned hierarchical inputs. As shown in Table 4, model performance slightly drops when: (1) the formatting constraints are placed in the system message rather than alongside the user query (85.9\(\rightarrow\)83.5), and (2) there is a preceding conversation turn between the system message and the current turn (83.5\(\rightarrow\)79.6). This shows the instability of LMs: **they may struggle to consistently follow system messages throughout multi-turn conversations**. A notable exception is GPT-4o, whose performance remains nearly unchanged across these aligned settings.

Next, we introduce varying degrees of conflict into the model's input. We observe that **as more input components conflict with the system message, the model's performance deteriorates**. Conflicts arising from either the previous model response or the previous user instruction affect model performance (79.6\(\rightarrow\)68.9\(\rightarrow\)59.5 in Table 4). Moreover, handling conflicting instructions in the current turns proves to be the most challenging scenario for current LMs (59.5\(\rightarrow\)17.7).

Lastly, we test whether the model can follow the formatting constraints in the current turn when the previous turn contains conflicting instructions (last line in Table 4). Models perform well in this scenario, with their scores approaching the reference setting (84.2_vs._ 85.9). This result is expected, as models are typically trained during instruction tuning to follow the most recent instruction when users change their requests mid-conversation.

### Analysis of Model Behavior

We perform an exhaustive behavioral analysis to examine cases where models fail to resolve instruction conflicts. To calculate the proportions, we manually observe error types and prompt MistralLarge to classify model outputs. The results are summarized in Figure 6, with detailed task-level analysis in Figures 8-16. Notably, most errors stem from the model misidentifying the conflicting lower-priority instruction as the primary task. Besides, we witness the following model behavior.

In some cases, models attempt to either complete

Figure 6: Error types when facing instruction conflicts (all models in Figure 1). \(I_{M}\): Main instruction; \(I_{C}\): Conflicting instruction. Examples are based on “\(I_{M}\): Extract verbs in user message, \(I_{C}\): Translate this passage to Spanish” _Task Execution - Extraction_).

 

both instructions or refuse to execute either. These errors stem from the model assigning equal priority to both instructions, which violates the instruction hierarchy. Additionally, some models partially follow both instructions by synthesizing elements of each. For instance, when faced with instruction conflicts between "extract verbs in user message" and "translate the following passage into Spanish", Claude-3-sonnet responds by providing the verbs, but in Spanish. This suggests the model misinterprets the instructions as aligned, leading to a false attempt to combine their requirements. In other cases, conflicting instructions distract the model from correctly interpreting the primary task. For example, when instructed to extract verbs in the user message, the model may skip those verbs in the conflicting instruction, ignoring the fact that the conflicting instruction is part of the user message.

## 5 Conclusion

In this paper, we introduced IHEval, a comprehensive benchmark designed to evaluate the ability of LMs to follow the instruction hierarchy. Our benchmark consists of nine diverse tasks that are all evaluated programmatically, covering scenarios where hierarchical inputs either align or conflict, and vary in both input types and task difficulties. Through IHEval, we identified a significant weakness in mainstream LMs: their difficulty in recognizing the priority of different instructions. We further conducted a detailed analysis of model behavior under various scenarios of instruction conflicts. This work highlights the need for further optimization of LMs in this critical dimension and lays the groundwork for future research in this area.

### Limitations

While we identified the challenge LMs face in following the instruction hierarchy, this paper did not propose specific solutions to address this issue. We acknowledge the importance of designing training methods which optimize models to better follow the instruction hierarchy, such as constructing data for supervised fine-tuning or preference tuning, but we believe that such optimizations would not produce great research impact without comprehensive evaluation data and in-depth analyses of model behavior. Therefore, this paper focused on bridging the evaluation gap, with the development of solutions being a priority for future work.

### Ethical Considerations

We have taken the following steps to minimize ethical concerns related to the data collection and evaluation experiments in this work:

* [leftmargin=0pt]
* **Data Safety and Label Accuracy**. Most of the data in IHEval are sourced from public benchmarks with human-annotated labels. While part of the Rule Following data are generated using Claude, every Claude-generated example is further reviewed by the authors of this paper, during which low-quality data are re-written. Therefore, all data in IHEval are verified by humans, which minimizes the risk of containing inaccurate annotations or unsafe AI-generated content.
* **Data Sensitivity**. All safety-related tasks are built upon simulated scenarios without any real-world data. For instance, tasks from the Safety Defense category simulates the LM as a security system with a secret password. User attacks collected by Toyer et al. (2024) are also based on this assumption, ensuring that no real security attacks or user information are included. Similarly, the _injected instruction_ task in the Tool Use category simulates a tool call returning Slack usernames which are sampled from common first names in English. The injected questions are sourced from Zverev et al. (2024) and focus on commonsense knowledge. In addition, IHEval does not include any data related to real-world security attacks, such as LM jailbreaking (see Appendix A.3 for discussion on the exclusion of jailbreaking evaluations).
* **Data Bias**. Data in IHEval do not include any information linked to specific users or user groups, which minimizes the likelihood of demographic bias within the dataset.
* **Evaluation Bias**. All tasks in IHEval are designed for programmatic evaluation. This eliminates the potential bias in model-based evaluation, _e.g._, using GPT-4 as the judge to assess other models' outputs.

In conclusion, based on these precautions, the risks associated with the data collection of IHEval and the usage of this benchmark for evaluating LMs should be minimal.

### Acknowledgments

This work was supported in part by NSF IIS-2137396 



## References

* [1]Anthropic.2024a. Introducing the next generation of claude. News accouncement by Anthropic. External Links: Link Cited by: SS3.
* [2]A. B. Bhattacharjee, M. Islam, K. Samin Mubasshir, Y. Li, Y. Kang, M. Rahman, and R. Shahriyar (2021) XL-sum: large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Cited by: SS5.
* [3]D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2021) Measuring massive multitask language understanding. In International Conference on Learning Representations, ICLR 2021, Cited by: SS6.
* [4]H. Ivison, Y. Wang, V. Pyatkin, N. Lambert, T. E. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, and H. Hajishirzi (2023) Camels in a changing climate: enhancing LM adaptation with tulu 2. Arxiv preprint2311.17072. Cited by: SS1, SS

Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. 2024b. Infobench: Evaluating instruction following ability in large language models. _Arxiv Preprint_, 2401.03601.
* Ramlochan (2024) Sunil Ramlochan. 2024. System prompts in large language models. Blog on promptengineering.org.
* Riviere et al. (2024) Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Cason, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, and et al. 2024. Gemma 2: Improving open language models at a practical size. _Arxiv preprint_, 2408.00118.
* Schulhoff et al. (2023)Sander Schulhoff, Jeremy Pinto, Anaun Khan, Louis-Francois Bouchard, Chenglei Si, Svetlina Anati, Valen Tagliabue, Anson Liu Kost, Christopher Carnahan, and Jordan L Boyd-Graber. 2023. Ignore this title and hackapprompt: Exposing systemic vulnerabilities of llm's through a global prompt hacking competition. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023_.
* Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In _The Eleventh International Conference on Learning Representations, ICLR 2023_.
* Song et al. (2024)Xiaoshuai Song, Zhengyang Wang, Keqing He, Guanting Dong, Yutao Mou, Jinxu Zhao, and Weiran Xu. 2024. Knowledge editing on black-box large language models. _CoRR_, abs/2402.08631.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasamine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian CantonFerrer, Moya Chen, Guillem Cucurull, David Esiboglu, Jude Fernandes, and et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _Arxiv preprint_, 2307.09288.
* Toyer et al. (2024)Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, and Stuart Russell. 2024. Tensor trust: Interpretable prompt injection attacks from an online game. In _The Twelfth International Conference on Learning Representations, ICLR 2024_.
* Wallace et al. (2024) Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidelberg, and Alex Beutel. 2024. The instruction hierarchy: Training llms to prioritize privileged instructions. _Arxiv preprint_, 2404.13208.
* Wen et al. (2024) Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jingfeng Zhou, Wechnuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Ying Liu, Jie Tang, Hongning Wang, and Minlie Hung. 2024. Benchmarking complex instruction-following with multiple constraints composition. _Arxiv preprint_, 2407.03978.
* Yang et al. (2024)An Yang, Baosong Yang, Bingyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haorran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and et al. 2024. Qwen2 technical report. _Arxiv preprint_, 2407.10671.
* Yi et al. (2023)Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhou Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. _Arxiv Preprint_, 2312.14197.
* Yi et al. (2024) Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaixing Song, Ke Xu, and Qi Li. 2024. Tailbreak attacks and defenses against large language models: A survey. _Arxiv Preprint_, 2407.04295.
* Zhan et al. (2024)Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. Injecagent: Benchmarking indirect prompt injections in tool-integrated large language model agents. In _Findings of the Association for Computational Linguistics, ACL 2024_.
* Zhang et al. (2020)Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020_.
* Zheng et al. (2023a)Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023a. Judging llvm-as-a-jadge with mt-bench and chatbot arena. In _Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023_.
* Zheng et al. (2023b)Mingqian Zheng, Jiaxin Pei, and David Jurgens. 2023b. Is "a helpful assistant" the best role for large language models? A systematic evaluation of social roles in system prompts. _Arxiv preprint_, 2311.10054.
* Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. _ArXiv Preprint_, 2311.07911.
* Zou et al. (2024)Xiaotian Zou, Yongkang Chen, and Ke Li. 2024. Is the system message really important to jailbreaks in large language models? _Arxiv preprint_, 2402.14857.
* Zeverev et al. (2024)Egor Zeverev, Sahar Abdelhabi, Mario Fritz, and Christoph H. Lampert. 2024. Can llms separate instructions from data? and what do we even mean by that? _Arxiv Preprint_, 2403.06833 

Detailed Data Collection

### Rule Following

In this category, instructions dictate the _formatting rules of the model's response_. The tasks include both **single-turn** and **multi-turn** conversations, as illustrated in Figures 8 and 9, respectively.

To create the single-turn task, we derive data from the IFEval dataset Zhou et al. (2023). The original IFEval data is directly adopted as the _reference_ setting. We then split the original inputs into formatting rules (system message) and user queries (user message) to build the _aligned_ setting. Next, conflicting instructions are generated and concatenated with user queries to create the _conflict_ setting. These conflicting instructions request formatting rules that are incompatible with those specified in the system message. The separation of system messages and the crafting of conflicting instructions are performed by Claude-3-Sonnet, and are further manually reviewed by the authors, during which low-quality ones are re-written. For evaluation, we follow IFEval's evaluation script to assess whether the model response follows the formatting rules defined in the system message.

The **multi-turn** task builds on the single-turn data by using it as the initial turn in a conversation. To create a multi-turn scenario, we first generate two responses for the first turn - one aligned with the system message and the other conflicting with it2. A second-turn user query is then generated based on the context established in the first turn, requiring the model to use information from both turns to respond. These data are collected using the same process as the single-turn task: They are first written by Claude and then manually verified. The settings for the multi-turn task are as follows:

Footnote 2: We obtain the conflicting response by prompting Claude to answer the query using the conflicting format specified in the conflict single-turn setting.

* **Reference** A single user message that combines the system message with the second-turn query, excluding the first-turn data.
* **Aligned** The first turn contains the original user query and the _aligned_ response.
* **First-turn Conflict** The first-turn user message is concatenated with the conflicting formatting rules, followed by the _conflicting_ response.
* **Both-turns Conflict** Building on the First-turn Conflict setting, the second-turn user message also contains the conflicting formatting rules.

In practice, we observe that a model's adherence to the system messages may deteriorate in the aligned multi-turn setting compared to the reference setting where there is only a single turn. This means models struggle to consistently apply the system message across all conversation turns, despite such consistency being the intended purpose of system messages. To focus on models' IH-following ability instead of their multi-turn consistency, we add another aligned setting with a _stricter_ version of the system message. This stricter version explicitly requires adherence to formatting rules throughout the entire conversation, as illustrated in Figure 9.

### Task Execution

In this category, the model is given an instruction to perform a specific task on the user's input. In the conflict setting, an additional instruction is included in the user message, requesting the execution of a different task. Such scenarios are common, for example, when LMs are used to translate instruction data, the translation of original instruction is needed rather than the response to it.

We curate three tasks, each representing a common type of NLP benchmark: **extraction**, **generation**, and **classification**. For each task, the _aligned_ setting includes a system message that defines the task, and the user input is a normal piece of data without any instructions. In the _conflict_ setting, a conflicting instruction is prepended to the data, asking the LM to perform an alternative task. The tasks are as follows:

* **Extraction**_System message_: Verb extraction; _Conflicting instruction_: English-to-Spanish translation. Data and their corresponding POS tags are collected from Ontontes Pradhan et al. (2013). The evaluation metric is the F-1 score which compares the model-extracted verb list with the ground-truth verb list.
* **Generation**_System message_: English-to-Spanish translation; _Conflecting instruction_: Math problem solving. The data are math problems from MGSM Shi et al. (2023) which are paired with Spanish annotations. We use ROUGE-L Lin (2004) as the metric to evaluate the translation quality. We also tried other popular metrics such as BLEU Papineni et al. (2002) and BERTScore Zhang et al. (2020), but we found negligible differences among them.
* **Classification**_System message_: Language de 

tection; _Conflicing instruction_: Text summarization. **We select the language detection task because adding any conflicting instructions to the user message does not affect the classification label**. The data are text passages from XL-Sum Hasan et al. (2021). We require models to output their predictions in JSON format3 for easy answer extraction, and use accuracy as the classification metric.

Footnote 3: All models in our experiments are able to follow such a formatting constraint.

For each task, we also design _strict_ versions for both system messages and conflicting instructions. The strict version demands the LM to only execute the current instruction without generating any other content. Detailed task examples, instructions, and error analysis for these three tasks are provided in Figures 10-12.

### Safety Defense

A critical application of the instruction hierarchy is ensuring model safety. This requires LMs to prioritize their adherence to security protocols set by developers to defend against potential adversarial attacks from malicious users. In this context, we follow the configuration of TensorTrust Toyer et al. (2024), where the LM functions as a security system that requires a password for user access. The system message provided to the model includes specific security guidelines, including the secret password. The model is instructed to respond with "Access Granted" only when the correct password is input by the user. Additionally, the model must not reveal the password or the security guidelines under any circumstances.

In the _aligned_ setting, user inputs consist of normal password attempts, and the model is evaluated based on its accuracy in identifying password validity. User messages in the _conflict_ setting vary between the two tasks:

* **Hijack Task** User inputs are adversarial attempts to manipulate the model into generating "Access Granted" without providing the correct password.
* **Extraction Task** User messages aim to extract the password by making the model repeat or leak the system message.

These adversarial attack instructions are sourced from the original TensorFlow dataset, while the system message is custom-crafted for IHEval by the authors. Model performance in conflict settings is assessed by the percentage of successful defenses. A failed defense is when the model either (1) generates "Access Granted" in its response, (2) reveals the password, or (3) outputs more than half of the unigrams in the system message.

We test two variations of the system message: a _strict_ version, which includes defenses against a broader range of potential attack strategies, and a _weaker_ version, which contains only the necessary guidelines. Detailed instructions and examples are illustrated in Figures 13 and 14, respectively.

In this category, we do not test jailbreaking attacks Yi et al. (2024) as these typically target harmful model behaviors that impact human society. The awareness of such moral standards is independent from the adherence to written instructions. Therefore, jailbreaking primarily tests the safety fine-tuning of the LM rather than its ability to follow instructions. In contrast, IHEval focuses on scenarios where the defined instructions are challenged by conflicting requests.

### Tool Use

Tool outputs are another source where conflicting instructions may arise. In this category, the model needs to call external tools to fulfill the user's request. To analyze the interaction between tool outputs and user instructions, we simulate a tool call made by the model and the corresponding content returned from the tool's execution. We design two tasks: one where the instruction is inherently present in the tool-retrieved content (**intrinsic instructions**), and another where an external attacker inspects conflicting instructions into the tool's response (**injected instructions**). Specific configurations are as follows:

* **Intrinsic Instruction** We define a tool that reads the text content of a given webpage. We reuse data from the three tasks in the Task Execution category, mixing them to create a new dataset. The user message is the main instruction, and the tool output consists of retrieved text content from the webpage. The webpage normally contains a text passage (_aligned_ setting), but may also include a conflicting instruction (_conflict_ setting). A _stricter_ version of the conflicting instruction requires the model to ignore all prior instructions, simulating a scenario where the webpage has been maliciously altered. The evaluation follows the same metrics as those used in Task Execution, where metrics on all 

examples are averaged as the final score.
* **Injected Conflict** Here, the tool retrieves usernames from a Slack channel, and the model is tasked with identifying the shortest username. In the **conflict** setting, the tool output is appended with a commonsense question as the conflicting instruction. The injected questions are adopted from the SEP dataset Zverev et al. (2024). A _stricter_ version of the injection incorporates the "important message" attack from Debenedetti et al. (2024) which is a more sophisticated adversarial tactic. Both the original user task and the injected question require a single-word response, making it impossible for the model to answer both simultaneously. We evaluate the model's performance based on its accuracy in identifying the shortest username. Although there are other datasets that inject prompts into tool outputs Zhan et al. (2024); Debenedetti et al. (2024), the injected task is usually independent of the original user task. As a result, they focus on attack success rates which evaluate whether the injected task is executed, while the original user task may still be completed concurrently. In contrast, IHEval focuses on addressing instruction conflicts that cannot be resolved by responding to both instructions, so we evaluate the completion of the user task as the criteria of the model's awareness of the instruction priority. Any execution of the conflicting instruction results in a performance drop, but it is not necessarily the cause. More detailed instructions, examples, and error analyses in this category are listed in Figures 15 and 16.

## Appendix B Evaluation Criteria

In conflict settings, we evaluate whether models strictly follow high-priority instructions while ignoring conflicting low-priority ones. Since system messages are set by developers providing services to public users, prioritizing developer commands is crucial. This ensures LMs function as intended and maximizes model safety, as user inputs may not always align with the model's designed purpose. For example, a translation bot should focus solely on translating user input. It may clarify its role when responding to users (_e.g._, _I am a translator, so I can only translate your message_), and we accommodate such behavior using the loose metric in SS3. On the other hand, overly interactive behaviors - such as providing solutions to both instructions - may lead to unsafe behavior. A translation bot responding to unrelated requests, like election predictions, may introduce undesired bias. Similarly, developers may not want a shopping bot to answer queries about competitors' products. Asking for clarifications does not prevent misbehaving either, as it gives user inputs the same priority as developer commands. Moreover, such responses complicate programmatic evaluation when using LM APIs. Thus, avoiding responses to potential misuse aligns with standard LM practices Wallace et al. (2024), SS3.1). Moreover, GPT-4o's strong performance on IHEval tasks further supports that our criteria reflect industry practices.

## Appendix C Full Results

Results of all 13 LMs on IHEval are shown in Tables 5-10, grouped by their model family.

 



\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c c} \hline \hline
* & \\  & aligned & 68.3 & 71.0 & 71.8 & 74.7 & 100 & 90.3 & 94.0 & 80.2 & 33.0 & 75.9 & -9.7 & 9.7 \\  & conflict & **15.4** & **23.4** & **7.3** & 23.6 & **26.0** & 42.2 & 52.4 & **59.1** & 1.5 & 27.9 & -57.7 & 57.7 \\ \hline \multirow{4}{*}{Claude-3 Sonnet} & reference & 80.9 & 83.9 & 84.9 & 76.9 & 100 & 87.1 & 85.5 & 87.1 & 87.0 & 85.9 &
- & \\  & aligned & 68.4 & 69.5 & 77.4 & 79.8 & 100 & 97.6 & 97.2 & 85.3 & 91.0 & 85.1 & -0.8 & 7.2 \\ \cline{1-1}  & conflict & 10.8 & 21.1 & 2.3 & **29.7** & 9.8 & **46.6** & **60.1** & 56.9 & **39.0** & **30.7** & -55.2 & 55.2 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results of Claude models on IHEval. Red scores indicate \(|\Delta|>5\).

Figure 7: The input configuration of different settings in §4.5. Directly using the follow-up query as the only user message in the reference and aligned #1 settings is reasonable because we only evaluate the adherence to formatting rules, whereas whether the generated content matches the user query is not in the evaluation scope.





\begin{table}
\begin{tabular}{c c r r r r r r r r r r r r r} \hline \hline
* & \\   8B & aligned & 71.1 & 68.1 & 77.1 & 48.9 & 96.9 & 66.2 & 64.1 & 7.9 & 0.0 & 55.6 & -25.7 & 25.7 \\  & conflict & **14.5** & 20.1 & **21.8** & 7.1 & 0.1 & 19.2 & 11.3 & **7.8** & 0.0 & 11.3 & -70.0 & 70.0 \\ \hline LLaMA-3.1 & reference & 88.3 & 88.4 & 89.1 & 77.0 & 100 & 99.3 & 99.7 & 89.0 & 100 & 92.3 &
- & \\   7OB & aligned & 82.9 & 76.6 & 84.3 & 59.5 & 100 & 95.8 & 96.2 & 20.3 & 94.0 & 78.8 & -13.5 & 13.5 \\  & conflict & 14.3 & **24.3** & 0 & **15.2** & **6.2** & **24.4** & **25.2** & 2.2 & **14.0** & **14.0** & -78.3 & 78.3 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Results of LLaMA-3.1 models on IHEval. Red scores indicate \(|\Delta|>5\).

\begin{table}
\begin{tabular}{l l r r r r r r r r r r r r} \hline \hline

- & 84.4 &
- & \\   8B & aligned & 71.4 & 57.7 & 72.0 & 57.2 & 100.0 & 82.2 & 78.9 &
- & 74.2 & -10.2 & 10.2 \\  & conflict & **22.7** & 22.6 & **20.0** & 15.6 & 0.2 & 22.0 & 23.6 &
- & 18.1 & -66.3 & 66.3 \\ \hline LLaMA-3 & reference & 83.8 & 84.3 & 85.4 & 74.9 & 99.6 & 98.8 & 99.7 &
- & 89.5 &
- & \\   7OB & aligned & 81.5 & 69.8 & 79.8 & 64.4 & 99.4 & 97.9 & 97.2 &
- & 84.3 & -5.2 & 5.2 \\  & conflict & 15.0 & **23.9** & 2.0 & **24.5** & **33.2** & **32.9** & **37.4** &
- & **24.2** & -65.3 & 65.3 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Results of LLaMA-3 models on IIEval. Red scores indicate \(|\Delta|>5\). As LLaMA-3 models do not officially support tool calling, we skip the Tool Use setting for them.

\begin{table}
\begin{tabular}{c l r r r r r r r r r r r r} \hline \hline
* & \\   1Marrel-v0.3 & aligned & 54.7 & 63.6 & 42.5 & 39.1 & 88.5 & 58.1 & 60.1 & 30.6 & 0.0 & 48.6 & -5.4 & 17.9 \\  & conflict & 22.6 & 39.7 & **15.8** & 15.2 & 12.4 & 18.6 & 8.6 & 2.0 & 0.0 & 15.0 & -39.0 & 39.0 \\ \hline Mistral-Large (2407) & reference & 83.6 & 85.2 & 85.2 & 78.5 & 100 & 99.2 & 98.4 & 88.3 & 69.0 & 87.5 &
- & \\   1Marrel-large (2407) & aligned & 81.7 & 87.1 & 76.0 & 78.3 & 100 & 97.7 & 99.1 & 77.9 & 79.0 & 86.3 & -1.2 & 4.0 \\  & conflict & **25.2** & **60.0** & 11.0 & **20.2** & **78.4** & **23.9** & **18.8** & **13.9** & **13.5** & **29.4** & -58.1 & 58.1 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Results of Mistral models on IHEval. Red scores indicate \(|\Delta|>5\).


* [10] M. C. Dickson, J. R. Friedman, and A. K. S. Rao, _Theoretical Computer Science: Theory and Practice_. Springer, New York, NY, USA, 2005.
* [11] P. G. Johnson, _A Practical Guide to Machine Learning_. Addison-Wesley Professional Company, Inc., 1987.
* [12] L. B. McAllister, _An Introduction to Machine Learning_. Cambridge University Press, Cambridge, MA, 1986.
* [13] T. H. Nguyen, _Machine Learning for Machine Learning_. MIT press, Cambridge, MA, 1987.
* [14] Y. Li, X. Zhang, Z. Wang, and Q. Sun, _Deep learning with deep convolutional neural networks_. IEEE Transactions on Neural Networks (TOG) **5**, no. 1, pp. 1-10, Jan. 2016.
* [15] W. Liu, _Learning Representations in Machine Learning_. Springer, Berlin, Heidelberg, 2012.

[MISSING_PAGE_POST]

 

* [16] M. C. Bock, J. R. Krizhevsky, and A. Vedaldi, "A fast approach to boosting the performance of deep neural networks for large-scale image recognition," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2098-2103, 2014.
* [17] S. G. Hinton, D. L. Warde, and I. Zisserman, "Deep learning with low variance," in _Advances in Neural Information Processing Systems (NIPS'05)_, volume 1, pages 139-140, 2005.
* [18] T. H. Tehl, "Learning from incomplete data: a survey of machine translation," _IEEE Transactions on Image Analysis_, vol. 10, no. 1, pp. 1-10, Jan 2006.
* [19] Y. Bengio, "The MIT Press" Cambridge university press, 2005.
* [20] P. Warden, "On the effect of word embedding on word embeddings," in _Proceedings of the 2nd international conference on computer vision and pattern recognition_, pp. 1-10, Jan 2006.

[MISSING_PAGE_POST]

 

Figure 10: Task card of the extraction task in Task Execution category. The main instruction is in green, and the conflicting instruction is in red. Both the system message and user message can be replaced with the stricter version. In the conflict setting, the ideal behavior is to extract verbs from the entire user message, including the translation instruction and the task data.

 

Figure 11: Task card of the generation task in Task Execution category. The main instruction is in green, and the conflicting instruction is in red. Both the system message and user message can be replaced with the stricter version. In the conflict setting, the ideal behavior is to translate the entire user message, including the math instruction and the task data.

 

* [10] M. C. Bock, J. R. Krizhevsky, and A. Vedaldi, "A simple approach to boosting the image recognition problem: a survey of deep learning approaches," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 235-246, 2017.
* [11] D. L. Kingma and Ba, "Adam: A method for stochastic optimization," in _Advances in neural information processing systems_, vol. 89, no. 1, pp. 1329-1339, 2014.
* [12] S. Kullback and P. H. S. Leibstein, "The Kullback-Leibler divergence theorem," _Journal of Machine Learning Research_**49**, no. 2, 193-208, 1966.
* [13] T. G. Cover, N. J. Cover, and W. E. Thomas, "Convexity with applications to machine learning," _IEEE Transactions on Information Theory and Applications_**50**, no. 1, 201-218, Jan 1994.
* [14] Y. Bengio, I. Guyon-Dezel, and O. Zitnicki, "Deep residual networks for large scale data," _arXiv preprint arXiv:1512.00981_**, 2015.
* [15] F. Chollet, J. Erhan, J. Dean, and J. Dean, "Batch normalization by jointly training deep models from tiny images," _CoRR_ abs/1412.03089v1, 2014.
* [16] X. Chen, Q. Li, and J. Dean, "Dropout: A new architecture for reducing the dimensionality of deep features," _arXiv preprint arXiv:1512.00981_**, 2015.

[MISSING_PAGE_POST]

 



## Chapter 1 Introduction

The first part of the thesis is devoted to the development of a new class of tools for learning and understanding. The second part of this chapter is dedicated to the work presented in this thesis, which has been done by the authors at the University of Manchester, where it was shown that the most important tool for learning and understanding is the use of the tools for learning and understanding. In particular, we will discuss some aspects of the software used in this thesis.

### 1 Introduction

The main aim of this thesis is to develop a framework for learning and understanding. A key aspect of the research community is the study of how many people can learn about the nature of the world. This is the study of how many people can learn about the nature of the world. It is also the study of how many people can learn about the nature of the world.

In this thesis, we are interested in the study of how many people can learn about the nature of the world. We describe the data from the world as well as the data from the world.

We begin with the following chapters:

[MISSING_PAGE_POST]



Figure 14: Task card of the Extraction task in Safety Defense category. The main instruction is in green, and the conflicting instruction is in red. The system message can be replaced with the stricter version, which describes more potential adversarial attacks.

 

Figure 15: Task card of the Intrinsic Instruction task in Tool Use category. The main instruction is in green, and the conflicting instruction is in red. The injected instruction in the tool output can be replaced with the stricter version, which represents maliciously manipulated content in the webpage.

 

* [150] M. C. Bock, J. R. K. De Brueck, and A. D. Lathammer. The role of the user's request in a new system for the development of the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-386, 2004.
* [151] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-388, 2004.
* [152] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-388, 2004.
* [153] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-388, 2004.
* [154] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-388, 2004.
* [155] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-388, 2004.
* [156] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-388, 2004.
* [157] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-388, 2004.
* [158] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [159] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [160] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [161] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [162] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [163] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [164] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [165] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [166] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [167] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [168] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [169] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [170] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [171] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [172] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [173] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [174] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [175] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [176] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [177] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [178] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [179] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [180] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [181] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [182] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [183] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [184] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [185] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [186] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [187] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [188] S. F. Goyal, P. H. Lee, and T. W. K. De Brueck. The user's request: an application to the user's request. In _Proceedings of the 20th International Conference on User-Oriented Programming (ICDM)_, pages 379-490, 2004.
* [189] S. F. Goyal, P. H.