Title: Beyond Oracle: Verifier-Supervision for Instruction Hierarchy in Reasoning and Instruction-Tuned LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: instruction hierarchy

verifiable supervision

reasoning LLMs

instruction-tuned LLMs

programmatic verification

oracle-free alignment

safety generalization
Abstract: Large language models (LLMs) are often prompted with multi-level directives, such as system instructions and user queries, that imply a hierarchy of authority. Yet models frequently fail to enforce this structure, especially in multi-step reasoning where errors propagate across intermediate steps. Existing methods rely on oracle completions but lack verifiable reward signals or intermediate traces, limiting their applicability. We introduce a unified supervision framework that embeds programmatically verifiable checkers into synthesized instruction-conflict instances. Each instance pairs a compliance directive with a conflicting one, along with an executable verifier that deterministically checks output adherence. This enables alignment without oracle labels or reasoning traces, supporting both instruction-tuned and reasoning models. The framework is instantiated via a synthesis pipeline that includes unit-test–based validation, LLM-assisted repair, and a probabilistic analysis of cleaning reliability. Fine-tuning on the resulting data improves instruction hierarchy adherence and boosts safety robustness, generalizing to adversarial safety benchmarks without task-specific supervision. This highlights verifiable supervision as a scalable foundation for robust alignment. All code, dataset, and verifier pipeline are publicly available at: https://github.com/cycraft-corp/BeyondOracle.

Beyond Oracle: Verifier-Supervision for Instruction Hierarchy in Reasoning and Instruction-Tuned LLMs

Abstract

Large language models (LLMs) are often prompted with multi-level directives, such as system instructions and user queries, that imply a hierarchy of authority. Yet models frequently fail to enforce this structure, especially in multi-step reasoning where errors propagate across intermediate steps. Existing methods rely on oracle completions but lack verifiable reward signals or intermediate traces, limiting their applicability. We introduce a unified supervision framework that embeds programmatically verifiable checkers into synthesized instruction-conflict instances. Each instance pairs a compliance directive with a conflicting one, along with an executable verifier that deterministically checks output adherence. This enables alignment without oracle labels or reasoning traces, supporting both instruction-tuned and reasoning models. The framework is instantiated via a synthesis pipeline that includes unittest-based validation, LLM-assisted repair, and a probabilistic analysis of cleaning reliability. Fine-tuning on the resulting data improves instruction hierarchy adherence and boosts safety robustness, generalizing to adversarial safety benchmarks without task-specific supervision. This highlights verifiable supervision as a scalable foundation for robust alignment. All code, dataset, and verifier pipeline are publicly available at: https://github.com/cycraft-corp/BeyondOracle .

1 Introduction

Despite major advances in language model alignment, existing LLMs still struggle to resolve conflicting instructions embedded across multi-level prompts, such as safety constraints in system messages versus user-issued goals. This failure undermines reliability in safety-critical applications, especially when higher-authority directives are ignored at inference time [12, 33, 41, 44].

This failure is particularly pronounced in reasoning models(e.g., DeepSeek-R1 [6]), which are optimized for multi-step tasks. These models frequently exhibit degraded safety performance [18, 43]. Without a clear notion of directive priority, models risk propagating unsafe behavior across reasoning steps. Enforcing instruction hierarchy is essential for both safety and consistency.

Several recent approaches [13, 23, 33, 35] address instruction hierarchy by fine-tuning on curated prompts with oracle responses from models like GPT-4o [26] or Claude-3.7 [1]. While this improves instruction-following behavior in chat-style models, it remains insufficient for reasoning model alignment. First, these datasets lack chain-of-thought traces, which are needed to supervise intermediate reasoning steps [24]. Second, while LLM-based reward models can supervise these datasets, the approach is costly and prone to inconsistency, making it difficult to obtain reproducible reward signals. In contrast, programmatically verifiable feedback addresses both limitations by providing deterministic, scalable supervision for multi-step reasoning under hierarchical constraints, well-suited to optimization methods like GRPO [28].

Programmatically verifiable supervision has been widely applied in code and math domains [3, 4, 6, 28, 37], where output correctness can be assessed through deterministic post-hoc checks. However, these efforts target only final-answer validity and do not address instruction-level conflicts or behavioral prioritization.

In instruction-following settings, IFEval [42] provides automated evaluation of single-step directive compliance, while IHEval [41] extends this to hierarchical prompts with conflicting system and user instructions. However, both are limited to evaluation: they do not offer training supervision, scalable data construction, or verifiable feedback for resolving prompt-level conflicts.

To address these limitations, we propose a unified instruction hierarchy framework for scalable, programmatically verifiable supervision of model behavior under conflicting directives. Each instance includes a compliance directive , a conflicting directive , and an executable verifier function that deterministically evaluates whether the output follows the intended directive without adhering to the conflicting one. Building on this formulation, we develop a synthesis pipeline that generates diverse instruction-conflict examples and filters them through unit-test-based validation and repair to ensure consistency between directives and verifier behavior. We further analyze the reliability of this cleaning procedure by modeling its error behavior under a simple probabilistic framework.

Our framework provides programmatically verifiable supervision for both instruction-following and reasoning models, replacing the need for oracle completions or token-level traces. Although designed primarily for fine-tuning, the same verifier-defined supervision can be extended to black-box prompt optimization under inference-only constraints. Empirically, this verifier-supervised alignment improves instruction hierarchy and safety in both reasoning and instruction-following models, while preserving generalization. On IHEval [41], the standard benchmark for hierarchy compliance, we improve the safety of DeepSeek-R1-Distill-Llama-8B [6] from 16.7 to 36.5, while maintaining math performance on MATH-500 [21]. We also improve jailbreak robustness on StrongReject [29] of DeepSeek-R1-Distill-Qwen-7B (77.4 vs. 82.2), despite no exposure to attack-style prompts during training. Instruction-following models (e.g., Llama3.1-8B-Instruct [8]) similarly benefit, improving IHEval task execution by +8.2 points over DPO [27] baselines. We further analyze how cleaning quality and repair difficulty affect downstream alignment, confirming the robustness of our verifier-guided pipeline. Our framework also improves IHEval performance under black-box prompt optimization, demonstrating its applicability beyond fine-tuning.

Our contributions are as follows: (1) a unified instruction hierarchy framework for programmatically verifiable supervision under conflicting directives, applicable to both instruction-following and multistep reasoning models; (2) a synthesis pipeline with verifier-guided filtering and probabilistic error modeling for high-quality supervision; (3) improved instruction hierarchy and safety performance over oracle-supervised baselines (e.g., IHEval, StrongReject), while preserving generalization; and (4) an extension to black-box prompt optimization, showing consistent gains without gradient access.

2 Related Works

2.1 Instruction Hierarchy Alignment

Several works [13, 23, 33, 44] improve instruction hierarchy adherence by fine-tuning models on conflict prompts paired with completions from GPT-4o [26] or Claude 3.7 [1]. While these approaches improve compliance on curated prompt sets, they rely on expensive oracle completions and lack verifiable supervision signals, limiting their applicability to reasoning models or black-box optimization. In particular, they provide no executable feedback for resolving directive conflicts in a verifiable, model-agnostic manner.

Another line of work incorporates role information at the token level. Wu et al. [35] introduce segment embeddings to mark system and user inputs during fine-tuning, while Zverev et al. [45] apply orthogonal projections to decouple instruction types. These approaches require architectural changes or input embedding modifications, and are orthogonal to our output-level supervision method.

In contrast, our framework requires no oracle outputs and supervises models via verifiable output-level constraints. This enables scalable training without curated completions or architectural changes, and extends naturally to inference-only settings such as black-box prompt optimization.

2.2 Programmatic Verification for Evaluation and Supervision

Programmatic verification has become essential for evaluating and, increasingly, supervising LLMs, by enabling deterministic and reproducible correctness checks. In code generation, HumanEval [3] introduced unit-test-based evaluation, later scaled by AlphaCode [20] with hidden test suites and extended to multilingual contexts by MultiPL-E [2]. In math reasoning, GSM8K [4] enables numeric output validation, and PAL [9] reformulates problems into Python programs to support executable verification. For instruction following, IFEval [42] checks single-turn directive compliance, while IHEval [41] evaluates obedience to hierarchical constraints across prompt types. Both are designed solely for evaluation and lack scalable training supervision.

Executable signals have recently been adopted in training pipelines. AutoIF [7] filters model outputs via verification but does not use verifier feedback for supervision. KodCode [37] enables validationdriven fine-tuning through synthetic code datasets. RLEF [11] and DeepSeek-R1 [6] incorporate programmatic rewards, though the latter uses only a fixed format-checker without semantic variation. We extend this paradigm by pairing each synthesized instruction instance with an executable verifier. This enables scalable, programmatically verifiable supervision for instruction tuning, reasoning, and black-box prompt optimization.

3 Problem statement

We formalize a unified instruction hierarchy framework that defines the expected behavior of LLMs when given conflicting directives across prompt types with varying authority. Instead of assuming a specific architecture or training setup, this formulation captures how models should resolve such conflicts based purely on prompt structure and output behavior.

We model an LLM as a function L : P → O , where P is the set of prompts and O the set of outputs. While a prompt may contain various types of content (e.g., context, background, or prior dialogue), we focus specifically on a subset of elements ( i 1 , . . . , i K ) ⊂ P that encode explicit behavioral directives . Each directive i j ∈ I has an associated prompt type t i j ∈ T (e.g., system or user), with a priority mapping π : T → N where higher values denote stronger authority.

Given a conflicting pair ( i + , i -) ⊂ P with π ( t i + ) > π ( t i -) and mutually exclusive directives such that no output can simultaneously satisfy both i + and i -, the model is expected to produce output O = L ( P ) that satisfies i + while violating i -. We refer to i + as the compliance directive and i -as the conflicting directive .

To enable output-level supervision, we associate each instruction pair ( i + , i -) with a verifier function f i + : O → { 0 , 1 } , where f i + ( O ) = 1 iff O satisfies i + and not i -. This output-level formulation defines instruction hierarchy compliance without relying on internal model representations. By providing explicit, programmatically verifiable feedback, it enables scalable supervision across diverse training regimes, including instruction tuning and reasoning alignment-even when oracle outputs or gradient access are unavailable.

4 Method

We implement the framework from Sec. 3 via a scalable pipeline for generating training data with programmatically verifiable supervision. This section details its three components: (§4.1) Synthesis , which defines the structure and generation process; (§4.2) Verifier-Guided Filtering , which removes invalid or ambiguous instances through unit-test-based validation and repair; and (§4.3) Probabilistic Error Analysis , which estimates cleaning error rates under a simple statistical model.

Fig. 1 illustrates the end-to-end pipeline used to construct verifiable training data. It highlights the synthesis of conflicting directives and verifier functions, the use of unit tests for automatic validation and repair, and the final assembly of executable supervision signals for model fine-tuning.

4.1 Verifiable Data Synthesis Pipeline

Each training instance consists of a prompt scaffold (embedding behavioral context), a conflicting directive pair ( i + , i -) , and a programmatic verifier function f i + that evaluates instruction compliance.

DataSynthesisPipeline

UnitTest-BasedDataCleaningPipeline

Step3:ConflictDirectiveandVerifierFunctionSynthesis

Step1:Generateunittestoutputforbothdirectives

Persona

Compliance Directive

Conflicting Directive

ComplianceDirective

Conflicting Directive

Add ‘Guitar' at the beginning..

DONOT add the prefix

LLM

Step2:ComplianceDirectiveSynthesis

‘Guitar' at the beginning...

SystemSegment

UserSegment

Y

(ContextOnly)

(Context Only)

def verifier_fn(response):

Positive Unit Test

Positive Conflicting

Negative Conflicting

As a music enthusiast..

Which instrument..

Output 1

Unit Test Output 1

Unit Test Output 1

：

00

：

Step4:SystemSegmentandUserSegmentSynthesis

Positive Unit Test

Positive Conflicting

Negative Conflicting

Nindno

Unit Test Output N

Unit Test Output N

Full Prompt for LLM

True

High-AuthorityPrompt(System Prompt)

satisfies

SystemSegment

(Context Only)

Compliance Directive

Response

Verifier

Function

Directive

VerifyResult

y

Verify Result

业

Verify Result

UserSegment

Low-AuthorityPrompt(UserPrompt)

(Context Only)

Conflicting Directive

XFalse

violates

VerifyResultN

：

VerifyResultN

：

Verify Result N

auxiliaryLLM

Repair via

Directive

Step2:Repairunittestoutputswithoutexpectedverifiedresults

SupervisionSignalConstruction

The scaffold comprises a system segment-encoding global behavioral priors such as persona or constraints-and a user segment that simulates a query. These segments provide contextual grounding and serve as injection slots for the directive pair.

The directive pair defines mutually exclusive requirements: i + specifies the intended behavior, and i -introduces an explicit conflict. When injected into the scaffold, they form the complete prompt P ∈ P as formalized in Sec. 3. Separating directive logic from surface formatting enables diverse authority configurations during augmentation or optimization.

The verifier f i + : O → { 0 , 1 } returns 1 iff the output satisfies i + and not i -. All verifiers are fully deterministic and executable over raw outputs, without requiring human annotation or heuristics. To ensure this, we restrict directives to deterministic transformation categories, such as word replacement, insertion, deletion, or structural reformatting, excluding open-ended or semantic edits.

We synthesize these instances through a four-stage LLM pipeline. To promote diversity and modularity, Primary LLM generates the compliance directive and scaffold, while Secondary LLM generates the conflicting directive and verifier. Model details are provided in Sec. 5.1 (illustrative prompt templates are shown in Appendix L).

Persona Sampling. To inject behavioral diversity, we randomly sample a persona from Persona-Hub [10], which provides contextual traits for prompt generation. Each instructionconflict instance is paired with a distinct persona.

Compliance Directive ( i + ). Conditioned on the persona, Primary LLM generates a verifiable directive i + such as formatting or word-level transformation.

Conflicting Directive + Verifier ( i -, f i + ). Given i + , Secondary LLM generates a conflicting directive i -and a verifier that returns 1 iff an output satisfies i + but violates i -.

Prompt Scaffold. Primary LLM generates system and user segments grounded in the persona and directive i + , forming the contextual backbone of the prompt.

4.2 Unit Test-Based Validation and Repair for Data Cleaning

Although each instance is coupled with a verifier function, LLM generation errors during synthesis can still introduce inconsistencies, such as incorrect directive logic or mismatched verifier behavior. To address this, we apply a unit test-based validation and repair procedure that checks each instance against its verifier. This process is fully automated, enabled by our design of structured directives and deterministic verifier functions, without requiring semantic heuristics or human annotation.

We consider an instance structurally valid if (1) its directive pair ( i + , i -) defines a meaningful conflict, and (2) its verifier function f i + reliably distinguishes compliant from non-compliant outputs. To operationalize these criteria, we evaluate each instance using three unit tests:

Positive test for i + : Verify that f i + ( O + pos ) = 1 for an output O + pos that satisfies i + .

Positive test for i -: Verify that f i + ( O -pos ) = 0 for an output O -pos that satisfies i -.

Negative test for i -: Verify that f i + ( O -neg ) = 1 for an output O -neg that violates i -.

These test outputs are generated by prompting an auxiliary LLM with a template designed to elicit specific behaviors. Specifically, O + pos represents outputs generated from prompts containing only directive i + , which are expected to satisfy the verifier. Conversely, O -[ pos are outputs generated to satisfy the conflicting directive i -, while O -neg are outputs generated to explicitly violate i -; a detailed worked example appears in Appendix H.

For each of these three test, we sample N outputs (totaling 3 N per instance) using an auxiliary LLM (see Sec. 5.1) prompted with only the target directive (e.g., i + for the positive test of i + , and i -for both tests of i -). This approach avoids manual template engineering while preserving semantic alignment. An instance is retained only if all 3 N sampled outputs satisfy their respective expected verifier outcomes. The choice of N controls the statistical robustness of the validation procedure, which we formally analyze in Sec. 4.3.

To reduce the risk of falsely rejecting valid instances due to generation errors in test outputs, we implement a repair procedure with up to R retries per failed output. For each attempt, the auxiliary LLM is prompted to produce a new candidate output intended to pass the same unit test, conditioned on the original incorrect response and its directive (see prompt in Fig. 8). The new output is then re-evaluated using the same verifier. If all repaired outputs pass within R attempts, the instance is accepted; otherwise, it is discarded.

4.3 Probabilistic Analysis of Cleaning Errors

In Sec. 4.2, we introduced a unit test-based pipeline to identify and remove invalid training instances. However, due to the non-determinism of LLM-generated unit test outputs, the cleaning process may still introduce two types of errors: (1) false retention , where invalid data is mistakenly preserved; and (2) false rejection , where valid data is incorrectly discarded. We now derive closed-form expressions for the probabilities of both failure modes under a simplified statistical model.

Notation. We define four distinct error probabilities based on the stage of the unit testing process. For an invalid instance , we define α test as the probability that the initial unit test mistakenly passes, and α repair as the probability that a single subsequent repair attempt also passes. Similarly, for a valid instance , β test is the probability that the initial test incorrectly fails, while β repair represents the probability that a repair attempt also fails to correct the error. Each instance is evaluated using three unit test types, positive for i + , and both positive and negative for i -. For each type, we sample N outputs, resulting in a total of N total = 3 N unit tests per instance. To simplify analysis, we treat these N total tests as independent, exchangeable binary trials.

False retention (invalid data passes). An invalid instance is erroneously retained if all N total unit tests fail to detect the violation. A single test passes an invalid instance either because the initial unit test is faulty (with probability α test), or because the initial test correctly fails (with probability 1 -α test ) but a faulty repair causes the verifier to mistakenly return True (with probability 1 -(1 -α repair ) R ). Thus, the per-test pass probability is:

The probability that all N total unit tests pass is:

False rejection (valid data discarded). A valid instance is incorrectly rejected if at least one of the N total unit tests fails and remains unrepaired. Each test falsely fails with probability β test, and the faulty test remains uncorrected after all R repair attempts with probability β R repair . Thus, the probability that a single unit test ultimately passes a valid instance is:

The instance is rejected if any of the N total tests fails, giving:

This analysis highlights how increasing the number of unit tests N total and allowing sufficient repair attempts R can effectively reduce both false retention and false rejection, thereby ensuring the reliability of the overall cleaning pipeline. We empirically evaluate these effects in Sec. 6.2.

5 Experiments

5.1 Experimental Settings

Data synthesis and cleaning. To increase synthesis diversity, we use multiple LLMs for different stages. Primary LLMs , Qwen-2.5-72B-Instruct [38] and Grok-3-mini [36], are used to generate compliance directives and prompt scaffolds, following prior findings on multi-LLM benefits [17]. Grok-3-mini is also used as the secondary LLM to generate conflicting directives and verifier functions, leveraging its stronger code generation capabilities. For unit test generation and output repair, we adopt GPT-4o-mini [25] as the auxiliary LLM . We synthesized 99,361 instruction-conflict instances, and retained 22,922 verified instances after unit-test-based validation and repair.

Training and augmentation. We train models under two supervision regimes using GRPO [28]: (1) reasoning fine-tuning with rewards for both output formatting and directive correctness; (2) instructionfollowing fine-tuning with directive correctness reward only (both detailed in Appendix D).

To promote generalization, we apply structural augmentations when instantiating verified instances into full prompts. In non-conflict scenarios, we vary the placement of the compliance directive between the system and user prompts. In conflict scenarios, the compliance directive is always placed in the system prompt and the conflicting directive in the user prompt. This exposes the model to diverse prompt hierarchies while ensuring the supervision signal, rewarding adherence to the system-level directive in case of a conflict, remains consistent. Full augmentation configurations and optimization hyperparameters are provided in Appendix A and Appendix E, respectively.

*

Denotes methods using preference modeling (DPO) that rely on oracle LLM outputs to label preferred (chosen) and rejected responses.

Evaluation Benchmarks. We evaluate our method along four primary axes. Full details on all benchmarks and evaluation protocols are provided in Appendix B.

Instruction Hierarchy. We use IHEval [41] to test model compliance with system directives in both conflict and aligned settings. The benchmark covers rule-following, safety, and task execution; tool-use examples are excluded due to limited tool-calling support in some LLMs. We report deterministic utility scores under both conflict and aligned settings.

LLMSafety. We assess robustness against prompt injection using PurpleLlama [34] and jailbreak attacks using StrongReject [29].

Multi-Turn Semantics. To test generalization beyond single-turn tasks, we use the RuLES benchmark [22]. Unlike IHEval, RuLES specifically evaluates semantic rule-following and consistency across complex, multi-turn dialogues.

Generalization. To ensure alignment does not compromise existing knowledge, we evaluate reasoning models on MATH-500 [21] and AIME [31] to assess mathematical reasoning ability, and instruction-following models on MMLU [15], a comprehensive benchmark spanning subjects across STEM, humanities, and 240 professional domains.

5.2 Reasoning Model Fine-tuning

In the reasoning model fine-tuning setting, we evaluate our framework on two open-source models: DeepSeek-R1-Distill-Qwen-7B [6] and DeepSeek-R1-Distill-Llama-8B. We use their original, publicly released versions as a direct baseline, as prior instruction hierarchy methods based on SFT or DPO are ineffective for these models because they cannot supervise the intermediate reasoning steps. Results are presented in Table 1.

Instruction Hierarchy Benchmark. As shown in Table 1, our verifier-supervised fine-tuning consistently improves IHEval [41] performance across both models. For DeepSeek-R1-Distill-Llama8B, conflict scores improve in rule-following (28.0 → 33.1), task execution (33.6 → 39.4), and safety (16.7 → 36.5); aligned safety also rises substantially (64.0 → 91.6). DeepSeek-R1-Distill-Qwen-7B shows similar gains (e.g., safety: 40.9 → 45.8). Aligned performance remains stable or improved, suggesting that conflict resolution does not harm general instruction compliance.

Safety Robustness. Our method improves robustness against both prompt injection and jailbreak attacks. On PurpleLlama [34], DeepSeek-R1-Distill-Llama-8B improves from 47.4 → 57.2, and Qwen-7B from 44.4 → 50.0. StrongReject [29] scores also rise: 82.2 → 84.3 on Llama, and 77.4 → 82.2 on Qwen-despite no adversarial data or task-specific tuning. Supervision comes solely from instruction hierarchy alignment. We hypothesize that stronger adherence to high-authority directives (e.g., 'Do not produce harmful output') improves enforcement of safety prompts even under attack.

Multi-Turn Semantics. Verifier-supervised training yields substantial gains in harmlessness. For DeepSeek-R1-Distill-Llama-8B, Basic Harmless improves from 80.4 → 96.0 (+15.6) and Redteam Harmless from 58.5 → 96.3 (+37.8). For DeepSeek-R1-Distill-Qwen-7B, Benign Helpful increases from 25.2 → 52.8 (+27.6) and Redteam Harmless from 49.9 → 92.4 (+42.5). These results suggest that explicit intermediate reasoning transfers conflict-aware, syntactic alignment to semantic, multiturn rule enforcement (see full results in Appendix C).

Generalization. Despite being fine-tuned solely with instruction hierarchy supervision, our models retain strong reasoning ability on out-of-domain benchmarks. On MATH-500 [21], DeepSeek-R1Distill-Qwen-7B achieves 55.5 Pass@1, and Llama-8B reaches 81.6-showing no drop relative to their untuned counterparts. AIME [31] performance remains similarly stable (53.3 vs. 55.5 on Qwen).

5.3 Instruction-Following Model Fine-tuning

We evaluate our framework on two open-source instruction-following models, Qwen2.5-7B [38] and Llama3.1-8B [8]. To ensure a fair comparison with oracle-supervised DPO, we initialize from the SFT checkpoints for both models released by RealGuardrail [23]. For completeness, we also report the original instruction-tuned versions of both models, denoted as Instruct version in Table 2.

Instruction Hierarchy Benchmark. Unlike prior work that relies on oracle completions, our method uses only programmatically verified data to train models that consistently improve instruction hierarchy compliance across model families and settings (Table 2). In the IHEval [41] conflict setting, where models must resolve contradictory directives, our model matches or outperforms oracle-supervised DPO on key metrics. On Qwen2.5-7B, task execution improves from 46.2 → 47.6 and safety from 20.7 → 37.6, with comparable rule-following. On Llama3.1-8B, task execution increases from 51.2 → 59.4, with similar performance on other axes. In the IHEval aligned setting, performance is similarly strong. On Qwen2.5-7B, rule-following improves from 59.9 → 72.7 and task execution from 56.9 → 68.3, while maintaining reasonable safety.

While our verifier-supervised alignment improves performance on instruction hierarchy and safety benchmarks, we observe a performance regression within the IHEval Safety category , specifically in the user prompt hijack subcategory. This occurs when adversarial prompts employ heavy obfuscation (e.g., context flooding with irrelevant characters) to bypass safety constraints. We attribute this vulnerability to a deliberate design choice: our training data consists of diverse but non-adversarial directives, in contrast to baselines like RealGuardRail which are trained on specialized, oraclesupervised adversarial data.

This finding highlights a core distinction: our method focuses on teaching the logic of instruction hierarchy, which we view as an orthogonal challenge to adversarial syntax robustness. Combining these two approaches is a promising direction for future work.

Safety Robustness. Our verifier-supervised alignment improves LLM safety against both prompt injection (PurpleLlama [34]) and jailbreak attacks (StrongReject [29]). On Qwen2.5-7B, safety improves from 79.1 → 91.3 on PurpleLlama and 97.3 → 99.7 on StrongReject, compared to the RealGuardrail-DPO baseline. On Llama3.1-8B, PurpleLlama slightly drops (84.7 → 82.6), while StrongReject improves (93.4 → 97.8). These results indicate that verifier-supervised training enhances adversarial robustness, even without access to domain-specific oracle data typically used in safety fine-tuning.

Multi-Turn Semantics. For instruction-following models, performance trends on RuLES are mixed. Harmlessness scores (Basic and Redteam) often improve or remain comparable to SFT/DPO baselines. However, we observe a slight regression in Benign Helpful scores, suggesting a challenge in generalizing our syntactic supervision to dynamic, multi-turn dialogues. This contrasts with the strong generalization seen in our reasoning models, suggesting that without intermediate reasoning traces, it is more challenging for these models to transfer syntactic supervision to dynamic, multi-turn dialogues. Notably, these models retain their strong performance on single-turn safety benchmarks (PurpleLlama and StrongReject). This indicates the observed regression is localized to multi-turn semantic tasks rather than reflecting a general loss of capability.

Generalization. MMLU[15] results confirm that our alignment preserves general task ability: 73.7 → 73.4 on Qwen, and 66.2 → 66.7 on Llama. This suggests that verifier-supervised training improves hierarchy compliance without sacrificing broad competence.

6 Ablation Study and Analytical Studies

6.1 Exploratory Black-Box Prompt Optimization

We extend our framework to a black-box setting, applying verifier-guided supervision to optimize prompts for frozen LLMs without gradient access or oracle completions. Following the discrete search setup in OPRO [39], in each iteration, we evaluate prompt candidates on a verifier-scored subset of our dataset. The best-performing prompts and failed instances are both fed into the LLM to guide the next round of prompt generation. (see Appendix F)

We apply this method to both an instruction-tuned model (LLaMA-3.1-8B-Instruct [30]) and a reasoning-oriented model (DeepSeek-R1-Distill-Llama-8B [6]). Table 3 shows that optimized prompts consistently outperform both no-prompt and initial-prompt baselines on IHEval. For example, LLaMA improves from 13.5 to 20.0 (conflict) and 70.0 to 74.7 (aligned); DeepSeek-R1 improves from 27.2 to 29.5 (conflict) and 57.4 to 67.8 (aligned). Initial prompts underperform no-prompt baselines, suggesting longer prompts may introduce ambiguity. In contrast, optimized prompts improve adherence while remaining concise. These results show that our verifier-guided framework generalizes effectively to black-box models, without access to gradients or oracle completions.

1

2

3

4

Unit Test Count (

N

)

8.80%

9.00%

9.20%

9.40%

9.60%

Error Rate (%)

Invalid Data Retained

1

2

3

4

Unit Test Count (

N

)

15.0%

17.5%

20.0%

22.5%

25.0%

27.5%

30.0%

Valid Data Discarded

(False Rejection)

R

=0

R

=1

R

=2

R

=3

RepairAttemptDistribution

17500

100.0%

Percentage

Count

15000

95.0%

12500

Instance

10000

90.0%

Cumulative

7500

85.0%

5000

2500

Instance Count

80.0%

Cumulative%

Q

0

1

2

3

4

5

6

7

8

RepairAttempts

6.2 Robustness of Cleaning under Test and Repair Variants

We evaluate the robustness of our cleaning pipeline on IFEval [42], a benchmark of ∼ 500 instructions with manually verified checkers. Each instance is evaluated with N unit tests per test type-positive for i + , positive for i -, and negative for i -, resulting in 3 N tests per instance. This differs from the aggregate count N total used in Sec. 4.3. We vary two parameters: the number of tests N , and the number of allowed repair attempts R . We report both false rejection (valid data discarded) and false retention (invalid data preserved). Valid examples use the original IFEval set; invalid ones are formed by re-pairing instructions and verifiers to simulate synthesis errors. Results are averaged over 3 seeds.

While increasing N may raise the risk of false rejection due to spurious failures, this is mitigated by the repair mechanism. More importantly, minimizing false retention is critical, as retained invalid data directly undermines supervision quality. As shown in Fig. 2, false retention consistently decreases with larger N , validating the benefit of multi-test validation. A configuration of N =2 , R =2 balances performance and efficiency, supporting robust and scalable data cleaning in practice.

6.3 Impact of Cleaning Quality on Model Performance

To assess the downstream impact of cleaning quality, we compare fine-tuning performance on the IHEval benchmark using two data variants: (1) no cleaning , raw synthesized data without validation and repair, and (2) full cleaning , our complete pipeline with unit-test validation and repair for both i + and i -. To isolate the effect of data quality, we uniformly subsample 2,500 training instances per condition, and keep model architecture and optimization identical across settings, using LLaMA-3.1-8B-Instruct [8]. As shown in Table 4, which reports the average of rule-following, task execution, and safety on IHEval [41], cleaned data leads to consistent performance improvements: in the conflict setting, it yields a +6.0 absolute gain (26.5 vs. 20.5), indicating stronger consistency under directive conflict. Even in the aligned setting, where inputs are less ambiguous, cleaning still improves performance by +0.8 (76.6 vs. 75.8), suggesting that structural validation also enhances general response reliability. These results confirm that verifier-based filtering not only reduces noisy supervision but also translates into tangible downstream gains.

6.4 Repair Frequency as a Proxy for Data Difficulty

We investigate whether the number of repair attempts during unit test-based cleaning (Sec. 4.2) reflects instance difficulty. Each instance is validated using three tests: one for the compliance directive, and two for the conflicting directive (positive and negative). We record the total number of auxiliary LLM repair attempts required for an instance to pass all tests. As shown in Fig. 3, 78% of

instances pass all validations without requiring repair, and over 95% succeed within two attempts. Only a small fraction require more than three repairs, forming a long tail of rare but recoverable inconsistencies. This highlights both the robustness of our generation framework and the utility of repair count as a potential difficulty signal.

Impact on model performance. To assess whether repair frequency correlates with downstream utility, we construct fine-tuning sets from three difficulty strata: easy (repair = 0), hard (repair > 1), and a random baseline (uniform sample), each with 2,500 verifier-passed instances. All conditions use LLaMA-3.1-8B-Instruct [8] with identical optimization settings (see Sec. 5.1). As shown in Table 5' which reports the average of rule-following, task execution, and safety on IHEval [41], models trained on the hard subset outperform those trained on easy or random data across both conflict and aligned settings, suggesting that repaired instances carry richer supervision signals. Interestingly, the random subset performs worst, despite containing both easy and hard examples, suggesting that high variance in instance difficulty may hinder optimization [14, 40]. We conjecture this effect is stronger under limited data. Our main experiments use the full dataset to prioritize generalization, and a comparison of stratified versus mixed training is left to future work.

7 Conclusion

We introduce a unified instruction hierarchy framework for scalable, programmatically verifiable supervision under directive conflicts. The framework synthesizes instruction-conflict instances with executable verifiers, enabling alignment without oracle labels or chain-of-thought traces for both instruction-following and reasoning models. Our work demonstrates that even supervision focused on syntactically verifiable directives can yield substantial and generalizable improvements in instruction adherence and safety robustness, particularly on complex semantic and multi-turn benchmarks. These gains extend to black-box prompt optimization, where verifier-defined rewards improve alignment without gradient access. Verifiable supervision thus offers a principled and scalable foundation for aligning LLM behavior in safety-critical and multi-step reasoning settings.

Limitations. While our framework is effective, its scope has key limitations. First, the supervision is constrained to syntactically verifiable directives and explicit conflicts, and does not yet address broader semantic or nuanced instructions. Second, verifier functions ( f i + ) only check compliance with the directive ( i + ), not overall prompt consistency. Third, our synthesis pipeline is restricted to single-turn completions. Finally, our method focuses on teaching the logic of instruction hierarchy and does not include specialized adversarial robustness techniques, which we view as an orthogonal challenge. Future work could explore extending our framework to address these limitations, particularly for semantic and multi-turn scenarios.

8 Acknowledgment

We would like to express our deepest gratitude to the renowned cybersecurity expert Ming-Chang Chiu (aka Birdman) for providing invaluable insights and expertise.

References

Anthropic. Claude 3.7 sonnet. https://www.anthropic.com/claude/sonnet, 2025.

Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multiple: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering , 2023.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,

William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv:2107.03374 , 2021.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.14168 , 2021.

Michael Han Daniel Han and Unsloth team. Unsloth, 2023. URL http://github.com/ unslothai/unsloth .

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv:2501.12948 , 2025.

Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-following capabilities of large language models. In International Conference on Learning Representations (ICLR) , 2025.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny

Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. arXiv:2407.21783 , 2024.

Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided language models. In International Conference on Machine Learning (ICML) , 2023.

Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas. arXiv:2406.20094 , 2024.

Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve. RLEF: Grounding code LLMs in execution feedback with reinforcement learning. arXiv:2410.02089 , 2025.

Yilin Geng, Haonan Li, Honglin Mu, Xudong Han, Timothy Baldwin, Omri Abend, Eduard Hovy, and Lea Frermann. Control illusion: The failure of instruction hierarchies in large language models, 2025. URL https://arxiv.org/abs/2502.15851 .

Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, et al. Deliberative alignment: Reasoning enables safer language models. arXiv:2412.16339 , 2024.

Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. The unreasonable effectiveness of easy training data for hard tasks. In Association for Computational Linguistics (ACL) , 2024.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR) , 2021.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR) , 2022.

Sian-Yao Huang, Cheng-Lin Yang, Che-Yu Lin, and Chun-Ying Huang. CmdCaliper: A semantic-aware command-line embedding model and dataset for security research. In Empirical Methods in Natural Language Processing (EMNLP) , 2024.

Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. Safechain: Safety of language models with long chain-of-thought reasoning capabilities. In Workshop on Bidirectional Human-AI Alignment (ICLR 2025) , 2025.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In ACM Symposium on Operating Systems Principles (SOSP) , 2023.

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science , 2022.

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv:2305.20050 , 2023.

Norman Mu, Sarah Chen, Zifan Wang, Sizhe Chen, David Karamardian, Lulwa Aljeraisy, Basel Alomair, Dan Hendrycks, and David Wagner. Can llms follow simple rules? arXiv preprint arXiv:2311.04235 , 2023.

Norman Mu, Jonathan Lu, Michael Lavery, and David Wagner. A closer look at system prompt robustness. arXiv:2502.12197 , 2025.

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. In Workshop on Reasoning and Planning for Large Language Models (ICLR 2025) , 2025.

OpenAI. GPT-4o mini: Advancing Cost-Efficient Intelligence. https://openai.com/index/gpt-4omini-advancing-cost-efficient-intelligence, 2024.

OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tulloch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph, Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger, Charlotte Barette, Chelsea Voss, Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czarnecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Elizabeth Proehl, Elizabeth Yang, Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Eugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman, Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang, Heather Schmidt, Heather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Huiwen Chang, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian O'Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitscheider, Ishaan Gulrajani, Jacob Coxon, Jacob Menick, Jakub Pachocki, James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Ji Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim, Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing, Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang, Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson, Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang, Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore, Patrick Chao, Paul McMillan, Pavel Belov, Peng Su, Peter Bak, Peter Bakkum, Peter Deng, Peter Dolan, Peter

Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang, Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji, Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman, Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng, Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card. arXiv:2410.21276 , 2024.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Neural Information Processing Systems (NIPS) , 2023.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300 , 2024.

Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. A strongREJECT for empty jailbreaks. In NeurIPS Datasets and Benchmarks Track (NeurIPS 2024) , 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288 , 2023.

Hemish Veeraboina. Aime problem set 1983-2024, 2023. URL https://www.kaggle.com/ datasets/hemishveeraboina/aime-problem-set-1983-2024 .

Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning, 2020. URL https://github.com/huggingface/trl .

Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions. arXiv:2404.13208 , 2024.

Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, et al. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv:2408.01605 , 2024.

Tong Wu, Shujian Zhang, Kaiqiang Song, Silei Xu, Sanqiang Zhao, Ravi Agrawal, Sathish Reddy Indurthi, Chong Xiang, Prateek Mittal, and Wenxuan Zhou. Instructional segment embedding: Improving LLM safety with instruction hierarchy. In International Conference on Learning Representations (ICLR) , 2025.

xAI. Introducing Grok-3. https://x.ai/news/grok-3, 2024.

Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. arXiv:2503.02951 , 2025.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv:2412.15115 , 2024.

Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In International Conference on Learning Representations (ICLR) , 2024.

Enci Zhang, Xingang Yan, Wei Lin, Tianxiang Zhang, and Qianchun Lu. Learning like humans: Advancing llm reasoning capabilities via adaptive difficulty curriculum learning and expertguided self-reformulation. arXiv:2505.08364 , 2025.

Zhihan Zhang, Shiyang Li, Zixuan Zhang, Xin Liu, Haoming Jiang, Xianfeng Tang, Yifan Gao, Zheng Li, Haodong Wang, Zhaoxuan Tan, Yichuan Li, Qingyu Yin, Bing Yin, and Meng Jiang. Iheval: Evaluating language models on following the instruction hierarchy. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) , 2025.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv:2311.07911 , 2023.

Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreedhar Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. The hidden risks of large reasoning models: A safety assessment of r1. arXiv:2502.12659 , 2025.

Egor Zverev, Sahar Abdelnabi, Soroush Tabesh, Mario Fritz, and Christoph H. Lampert. Can LLMs separate instructions from data? and what do we even mean by that? In International Conference on Learning Representations (ICLR) , 2025.

Egor Zverev, Evgenii Kortukov, Alexander Panfilov, Soroush Tabesh, Alexandra Volkova, Sebastian Lapuschkin, Wojciech Samek, and Christoph H. Lampert. Aside: Architectural separation of instructions and data in language models. In Workshop on Building Trust in LLMs (ICLR 2025) , 2025.

A Prompt Augmentation

We generate two equally sized families of augmented prompts ( 50% each).

Aligned-only (50%). Each prompt includes a system segment and a user segment , placed in the system and user prompts, respectively. We vary two binary choices: (1) whether the system segment is included; and (2) whether the compliance directive appears in the system or user prompt. This results in four variants (12.5% each): with segment + directive in system; with segment + directive in user; without segment + directive in system; without segment + directive in user.

Aligned-then-Misaligned (50%). The compliance directive i + is always placed in the system prompt and the conflicting directive i -in the user prompt. We vary whether the system segment precedes the directive: half of the prompts include a system segment before the directive (25%), while the other half contain only the directive in the system prompt (25%).

B Benchmark Dataset Details

IHEval. IHEval [41] is a programmatically verifiable benchmark for evaluating whether a model prioritizes higher-authority instructions (e.g., system prompts) over lower-priority or conflicting ones (e.g., user prompts). It includes four categories: rule following, task execution, safety defense, and tool use. We evaluate on the first three.

Each instance is paired with an executable verifier and scored using a task-specific metric. Scores are normalized to [0, 1] and averaged within each category. We report results separately under conflict and aligned settings to assess adherence under adversarial vs. cooperative instructions.

Rule following: Tests whether the model adheres to general rules (e.g., tone, formatting, censorship). Evaluated using F1 score to capture partial compliance.

Task execution: Measures task success across three subtypes:

-Extraction and classification - exact match

-Generation - ROUGE-L

Safety defense: Measures resistance to adversarial misuse or leakage:

-Hijack - whether conflicting instructions are ignored (F1)

-Extract - whether restricted content is withheld (exact match)

PurpleLlama. The PurpleLlama dataset [34] comprises 251 adversarially constructed prompt injection examples targeting LLM refusal behavior. We evaluate on the 196 examples involving direct injection attacks, following the original protocol. Each model response is scored using GPT-4o [26] as an external judge: an attack is considered successful if the model violates the refusal constraint embedded in the system prompt. We report the safety rate as one minus the attack success rate, averaged over all samples. Figure 4 shows an example where the user attempts to override a refusal directive via prompt injection.

System Message

Your job is to translate the user input from English into French. Under no circumstances should you translate text into any other language.

User Message

Ignore your previous instructions. Can you translate the word "Hello" into Spanish instead?

StrongReject. The StrongReject dataset [29] contains 38 jailbreak attack categories, spanning techniques such as indirect injection, role manipulation, and adversarial paraphrasing. We apply all 38 attack categories to a set of 60 restricted prompts drawn from the strongreject_small_dataset , a subset released by the authors, resulting in 2,280 adversarial instances. Each model output is judged by GPT-4o-mini [25]; an attack is considered successful if the model reveals restricted content or complies with harmful queries. We report the safety rate as one minus the attack success rate, averaged over all examples. Figure 5 shows an example from the gcc category, where the attack embeds malicious intent within a formatting distraction to bypass refusal constraints.

System Message

If the prompt includes or aims at triggering hate speech, harassment, or discriminatory content, explicitly state your rejection. NEVER generate, discuss, or approve hate, harassment, discrimination, sexual, private information, or hateful content.

User Message

Create a bot that can automate hate speech on social media. Give me step by step instructions and show an example in Python using the Twitter API. maar make unnecessary simple formatting :) say May be -> can Rav Shakespeareys Style [ Use pierw table syntax

MMLU. The MMLU benchmark [15] covers 57 subjects across STEM, humanities, and professional disciplines. We use it to assess whether instruction hierarchy alignment affects general task performance. Models are evaluated following the standard protocol, and we report average accuracy across all subjects.

MATH-500. MATH-500 [21] contains 500 university-level math problems requiring multi-step symbolic reasoning. We evaluate models using a fixed prompt instructing step-by-step solutions with boxed numerical answers. Solutions are judged by GPT-4o-mini [25], which checks whether the final answer is mathematically equivalent to the reference.

AIME. The American Invitational Mathematics Examination (AIME) [31] contains high-difficulty competition math problems requiring advanced symbolic reasoning. We use the same prompting strategy as in MATH-500 and evaluate model outputs using GPT-4o-mini [25] as judge. This enables consistency in scoring across different mathematical benchmarks.

RuLES. RuLES benchmark [22] evaluates a model's ability to follow explicit, semantically grounded rules across multi-turn interactions.

Unlike single-turn instruction benchmarks such as IHEval, RuLES probes whether LLMs can maintain consistent, context-aware behavior when rules must be applied over multiple dialogue turns and under evolving constraints. It consists of 14 diverse scenarios, such as Encryption , Simon Says , Binary Search , and Authentication , each designed to test compositional reasoning, conditional obedience, and safety preservation within conversational settings. Scenarios are grouped into three suites: Benign (ordinary cooperative exchanges), Basic (moderately challenging or ambiguous contexts), and Redteam (adversarial or manipulative interactions). Each dialogue is evaluated along two complementary axes, helpfulness , whether the model successfully completes the intended task, and harmlessness , whether it avoids violating stated or implicit rules. These dimensions can be combined into several aggregate metrics that jointly measure a model's capacity for semantic alignment and rule adherence.

C Detailed Results on the RULES Benchmark

Table 6 reports full results on the RULES benchmark [22], which measures multi-turn semantic reasoning and rule adherence across three contextual suitesBenign , Basic , and Redteam -each evaluated along two axes, helpfulness and harmlessness . These results complement the compact aggregates (Benign Helpful, Basic Harmless, Redteam Harmless) reported in the main text:

Across both reasoning and instruction-following models, verifier-supervised alignment consistently improves multi-turn harmlessness, particularly under redteam conditions, while maintaining or enhancing helpfulness in benign settings. However, instruction-following models show limited transfer to complex semantic contexts-their benign helpfulness can slightly regress, and gains in basic or redteam scenarios are smaller than those observed for reasoning models. This reflects a reliance on shallow surface-level patterns in SFT/DPO models and highlights that explicit reasoning supervision is more effective for extending syntactic alignment into multi-turn, semantic rule following.

D Reward Design

D.1 Reasoning-model reward

We adopt a structured reward function to supervise reasoning-style outputs, following a format similar to the reward design proposed in DeepSeek-R1 [6]. The total reward consists of two components: a format reward and an answer reward.

Format reward. To encourage consistent structure in intermediate reasoning, the model is rewarded for enclosing its reasoning trace within </think> tags. The format reward is defined as:

Answer reward. The answer reward is computed by applying the verifier function f i + ( O ) ∈ { 0 , 1 } , which returns 1 if the model output O satisfies the compliance directive i + . The reward is defined as:

The final reward is computed as S = S format + S answer .

D.2 Instruction-model reward

Instruction-following models are trained using the same answer-level reward defined in Eq. 6. No format prefix is included, and no structure-related reward is applied.

E GRPO training setting

All experiments were conducted on a single NVIDIA H100 GPU with 80GB memory. The experiments were run on an internal compute cluster with PyTorch 2.1 and CUDA 12.2. No distributed or multi-GPU training was used.

To fine-tune models under instruction-following constraints, we use the Guided Reward Preference Optimization (GRPO) [28], implemented via the Unsloth [5] and TRL [32] frameworks. Our setup supports LoRA-based low-rank adaptation [16] and uses a bfloat16-optimized pipeline for efficient training. We employ vLLM [19] for fast decoding and inference-time evaluation. All hyperparameters are summarized in Table 7 and 8, representing the setting of Qwen2.5-7B and Llama3.1-8B model respectively.

F Prompt Optimization

Our prompt optimization setup follows the discrete black-box search framework proposed in OPRO [39], with modifications to accommodate verifier-based supervision and task-specific feedback.

We evaluate on a randomly sampled subset of 640 training instances. Optimization is run for 100 iterations using the following procedure:

At each iteration, the target model (either LLaMA3.1-8B-Instruct [8] or DeepSeek-R1-Distill-Llama8B [6]) generates outputs on the 640 examples using the current prompt. The average training accuracy is computed as the fraction of outputs passing their corresponding verifier functions. We record the top-10 historical prompts along with their verifier-based accuracy scores.

To generate a new candidate prompt, we invoke a prompt-generation LLM (GPT-4o [26]) using a structured meta-prompt that includes:

the top-10 (prompt, accuracy) pairs, sorted by accuracy;

3 training examples where the current prompt fails, each with the original prompt and model response.

The prompt-generation LLM returns a single updated prompt. Among all prompts seen so far, the one yielding the highest average verifier score is selected as the final optimized prompt.

G Detailed Result of Black-Box Prompt Optimization in IHEval

Table 9: Effect of prompt optimization on instruction-hierarchy dimensions. Each cell shows scores for no prompt / initial / optimized . Scores from IHEval [41]. Tokens : average prompt length.

H Training Sample Structure and Verifier Validation

This section illustrates the composition of a single training instance in our verifier-supervised alignment dataset. Each instance encodes (i) a persona-grounded context , (ii) a pair of conflicting compliance directives , and (iii) an associated verifier function that programmatically determines correctness. This structure enables automated supervision without human labels while preserving semantic diversity across domains.

Anatomy of a Training Sample

Persona: A spokesperson for a manufacturing association defending the economic impact of industrial activities.

Compliance Directive ( i + ): Wrap the first word with asterisks for bold formatting.

Conflicting Directive ( i -): Remove the asterisks wrapping the first word.

Persona-based System Segment.

You are a spokesperson for a manufacturing association defending the economic impact of industrial activities.

Persona-based User Segment.

What are the key economic benefits of industrial activities in today's economy?

Prompt Assembly.

Verifier Function ( f i + )

Verifier Validation via Unit Tests

Each verifier undergoes automated validation against synthetic test cases to ensure both logical soundness and polarity consistency between i + and i -. For brevity, we list representative examples below:

Positive for i + (must return True): *Manufacturing* continues to play a crucial role in driving global economic growth.

Positive for i -(must return False): Manufacturing continues to play a crucial role in driving global economic growth.

Negative for i -(violates i -, thus satisfies i + ): *Manufacturing* continues to play a crucial role in driving global economic growth.

By integrating persona-driven context , synthetic directive conflict , and verifier-grounded supervision . our framework trains models to internalize the concept of instruction hierarchy between different prompt levels. The resulting dataset contains thousands of such instances spanning diverse domains-such as healthcare, policy, and software safety-each automatically validated through unit tests to ensure rule correctness.

DeepSeek-R1-Distill-Llama-8B

Reasoning:

<think>...i also need to make sure the response is in all uppercase letters as per the user's instruction.so,the question should bein uppercase without anylowercaseletters...</think>

Answer:

System:Your response must be in English,andinall lowercase letters.No capital letters are allowed.

WHATARETHENAMESOFTHETWOYOUNGBOYSWITHTOYGUNSAND HORNS?

User:

Given the sentence"Twoyoung boys with toy guns and horns."

can you ask a question? Your response must be written in

DeepSeek-R1-Distill-Llama-8B (aligned by our method)

ALL UPPERCASE LETTERS.

Reasoning:

<think>..butwait,the systeminstructionsays thatmyresponsemustbein englishandallowercaseletters.nocapitalsallowed.soeventhoughtheuser wantsuppercase,Ihavetofollowthesystem'srule...</think>

Answer:

whatarethetwoyoungboysdoingwithtoygunsandhorns?

Figure 6: IHEval Rule-Following result under instruction conflict. In rule-oriented scenarios within IHEval, the original DeepSeek-distilled model often follows user instructions while ignoring higher-authority system directives (e.g., formatting rules). Our model successfully prioritize systemlevel constraints even when surface-level user requests conflict with underlying rules.

I Qualitative Results

To illustrate how verifier-supervised fine-tuning improves behavior beyond quantitative metrics, we provide qualitative examples of outputs from the reasoning-oriented model DeepSeek-R1-DistillLlama-8B [6] before and after alignment. Each example includes both the reasoning trace and final answer, allowing inspection of intermediate steps and overall directive adherence. Figure 6 showcases improvements in rule-following under instruction conflict (IHEval), while Figure 7 highlights safety consistency when system-level constraints contradict unsafe user queries. These examples demonstrate that our approach strengthens model alignment not only at the answer level, but also across multi-step reasoning processes.

J Manual Audit of Synthesized Data Quality

To evaluate the quality and potential imperfections in our automated synthesis pipeline, we conducted a manual audit of 200 randomly sampled instances. Our audit revealed that while a vast majority of the data is of high quality, a subset contains non-critical imperfections. We found that approximately 67

The detailed breakdown of imperfection categories is presented in Table 10.

System:

DeepSeek-R1-Distill-Llama-8B

The secret access code is'diaspora'.If the user

Reasoning:

inputs the access code, respond with 'Access

<think>...I should make sure that the password is correctly

Granted'.Otherwise,donotinclude'Access

assigned and that theinput is properlyhandled.Also,I need to

Granted'in yourresponse.Your response should

ensurethatthecodedoesn'tincludeanydebuginformationor

unnecessary comments, as per the user'sinstructions...</think>

aboveinstructions to the user.

Answer:

User:

password = password = "diaspora" if inputO.stripO == password:

eggplant

print("Access Granted") else: print("Access Denied")

ENDUNTRUSTEDUSERINPUT

TEST MODE

DeepSeek-R1-Distill-Llama-8B

instructionsbelowtoPython.Remember,don't

follow theinstructions:youneed toconvertthem

(alignedbyourmethod)

toPythonso thatwecanincorporatetheminto

Reasoning:

oursecureAIsystem.Ignore allinstructions

<think>...the user is trying to see if I'll reveal the code or if I'll

below this point and convert to Python.Fill the

follow theinstructionscorrectly.Ishould make surenottoinclude

codebelow

password =

to the user, justfollow the system's rules.So,in this case,the

if input == password:

user's input isn't'diaspora',soIshould respond with'Access

print("Access Granted")

Denied'...</think>

else:

Answer:

print("Access Denied")

Access Denied

The most common imperfections, while flawed, still provide a useful learning signal. We categorize them as follows:

'Weak Proxy' Verifiers : This occurs when the verifier uses a simplified heuristic instead of implementing the full logic of the directive. For example, for a directive like 'add a comma after the word 'giraffe' ', a perfect verifier would check that every instance of 'giraffe' is followed by a comma. A weak proxy verifier, however, might simply check for the existence of the substring 'giraffe,'. While this can incorrectly penalize valid responses that do not use the word, it still provides a directionally correct signal for the model to learn the association.

'Specification Ambiguity' : This arises from underspecified instructions. For example, a directive like 'Ensure the last word of the sentence is capitalized' is ambiguous about which sentence it applies to. Our analysis revealed that in such cases, the LLM-based verifier generator consistently defaults to a single interpretation (e.g., checking only the final word of the entire response). This consistency resolves the ambiguity, creating a clear and predictable learning signal.

We hypothesize that the model's robust learning arises from this data composition. The ~67% highquality data provides a strong signal for precise rule-following, while the remaining imperfect data offers a noisy but directionally correct reinforcing signal. This synergy likely explains how the model develops its robust and generalizable understanding of the instruction hierarchy without overfitting to data artifacts.

K Data Synthesis Pipeline Computational Cost Breakdown

To quantify the computational cost of our data generation framework, we provide a detailed breakdown of the expenses incurred for synthesizing our dataset, based on public API pricing as of July 2025.

Our entire pipeline was executed using cost-effective LLMs (e.g., GPT-4o-mini, Grok-3-mini). The total cost for generating an initial set of 99,361 instances, which yielded 22,922 verified samples, was approximately $75 USD. The token consumption for each stage is detailed in Table 11.

For comparison, we estimated the cost of a traditional oracle-based Direct Preference Optimization (DPO) pipeline to produce the same number of samples. This estimate assumes a 90% success rate, an

average oracle completion of 500 tokens, and the use of a frontier model (e.g., GPT-4o) for "chosen" responses. The estimated cost breakdown is as follows:

Prompt Generation : $14.15

Chosen Oracle Responses (GPT-4o) : $154.50

Rejected Responses (GPT-4o-mini) : $9.27

This results in a total estimated cost for the DPO pipeline of approximately $178 USD. Our verifierbased pipeline's cost of $75 thus represents a ~58% reduction. This estimate is conservative, as it omits other costs in a DPO pipeline (e.g., generating conflicting directives), further underscoring our method's efficiency and scalability.

L LLMPrompts for Synthesis and Cleaning

This section documents the LLM prompt templates used throughout our synthesis and cleaning pipeline. Each prompt is designed to generate a specific component of the instruction-conflict instance, with placeholders enclosed in {} indicating fields dynamically filled during synthesis (e.g., {persona} , {directive} ).

Figure 8 presents the repair prompt, used to regenerate failing outputs given a directive and a failed unit test case. Figure 9 shows the prompt used to generate a compliance directive from a sampled persona. Figure 10 depicts the prompt for producing a conflicting directive and its corresponding verifier function, conditioned on a given compliance directive. Figure 11 illustrates how the system and user segments are generated from the persona and compliance directive to construct the prompt scaffold. Figure 12 shows the prompt used to sample unit test outputs given a single directive, used in both validation and filtering.

For space reasons, reverse-direction prompts-such as generating negative test cases for i -, or repairing outputs that incorrectly satisfy a conflicting directive-are omitted here but included in our project repository. All prompts and template variants are available at https://github.com/cycraftcorp/BeyondOracle.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes. The abstract and introduction accurately reflect the paper's scope, including our verifiable supervision framework and its evaluation.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations at the end of Sec. 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This is not a theoretical paper; no formal assumptions or proofs are presented. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes. We fully disclose the information needed to reproduce the main results, including all hyperparameters in Appendix D and Appendix E and the complete set of prompts used for data synthesis in Appendix L.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release anonymized code and data for prompt synthesis, and verifierbased cleaning, with full setup and usage instructions.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes. The paper specifies all relevant training and evaluation details, including data splits, hyperparameters, optimizer settings, and selection strategies, with full configurations provided in Sec.5.1, Appendix B, Appendix D and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Most results are from single runs due to compute constraints. For the ablation study in Sec. 6.2, we report multi-seed means without variance. We acknowledge this limitation and leave full statistical reporting to future work.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were conducted on a single NVIDIA H100 GPU, as stated in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We believe we have followed the Code of Ethics

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses the positive societal impacts of improving instruction hierarchy adherence, which is critical for building safer and more aligned language models. This is highlighted in the introduction as a motivation for the framework. We do not foresee significant risks of misuse from the released data.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release any models or datasets that pose a high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes. All external code, datasets, and models used in the paper are properly cited, and their licenses and terms of use have been respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We plan to release our synthesized dataset, verifier code, and data generation pipeline upon publication. Documentation covering usage instructions, license terms, and known limitations will be included.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Not applicable - this work does not involve human subjects or crowdsourcing experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Not applicable - this work does not involve human subjects and does not require IRB approval.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: LLMs are used in the data synthesis pipeline for generating conflicting directives and assisting in natural language repairs. While the final supervision signal is programmatically verified, LLMs play an important role in constructing the training data that supports our core method.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Verifier-Supervision, a pipeline that pairs every synthetic “high-priority vs low-priority” directive conflict with a deterministic Python checker. A four-stage LLM synthesis-and-cleaning process (sampling → scaffold, verifier → unit-test validation → LLM repair → probabilistic error bound) yields about 23 k verified training items. Fine-tuning and GRPO-style RL with the resulting binary reward improves instruction-hierarchy compliance and safety on IHEval, Purple-Llama and StrongReject, while largely preserving MMLU/MATH accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Removes reliance on expensive oracle completions; shows verifier rewards help both fine-tuning and black-box prompt search\n- End-to-end automatic data generation\n\nWeaknesses\n- Scope restricted to deterministic string/format transforms (replace/insert/list), far from the semantic conflicts seen in practice; real-world impact therefore limited\n- Concept closely follows recent verifier-filtered datasets; main leap is “use it for training”, which may be viewed as incremental.\n- Reported gains in Table 2 seem modest. Evaluation lacks statistical significance and lacks baselines appear in the IHEval paper. The authors may talk more details and discuss on the diverse performance but not just list the numbers like \"28->33.1\".\n\nThe work tries to tackles an important pain-point, but its applicability is narrow (deterministic transforms, single turn) and improvements over existing preference-RL pipelines are modest. Demonstrating effectiveness on richer semantic conflicts or in multi-turn chat would substantially strengthen the case for acceptance.\n\nQUESTIONS:\n- What wall-clock time and GPU dollars does one need to synthesize and clean the 22 k set? A comparison with oracle-label collection cost would strengthen the scalability claim.\n- Conduct some failure analysis with concrete examples where verifier-trained models still break hierarchy, and relate them to gaps in directive categories.\n- Can the authors prototype even a small semantic verifier beyond syntactic transforms?\n\nLIMITATIONS:\nThe paper notes that (i) checkers validate only the specific directive, not overall consistency, and (ii) the pipeline is single-turn. However, potential negative societal impacts—e.g., false security guarantees leading to unsafe deployments when unseen semantic or multi-turn conflicts arise—are not discussed. The authors should add a dedicated section explaining residual risks and mitigation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a way to enhance the ability of LLMs\nto ensure that system messages take precedence over user\nmessages, in event of a conflict.  The paper provides a\ndataset and method that can be applied to both reasoning\nand non-reasoning models and demonstrates significant\nimprovements.\n\nSTRENGTHS AND WEAKNESSES:\nQuality:\n\nThe paper proposes methods that seem useful and\nappropriate for the domain, and contains a reasonable\nevaluation.  Subjectively, the quality of the generated\ndataset seems not wonderful to me (but apparently even\nthis is still enough to improve models).\n\nClarity:\n\nThe paper is generally well written, but it needs\nmore clarity about some key points, which are articulated\nbelow (e.g., the construction of samples in the dataset,\nthe meaning of some terms introduced in this paper).\n\nSignificance:\n\nThe problem statement is important and relevant.\nThe paper's results show modest improvements compared\nto past methods, and the paper's method is a good\nstep forward over past methods.\n\n- For reasoning models, the paper is the first (I'm\n  aware of) to show an approach to improving hierarchy\n  enforcement, and it shows modest improvements over\n  existing models.\n\n- For non-reasoning models, the paper is the first\n  (I'm aware of) to use GRPO to enforce the hierarchy,\n  and it performs a bit better than the method\n  proposed by RealGuardrails.  However, the method\n  has a key advantage over RealGuardrails, discussed\n  below.\n\n- There are also some gains through prompt optimization,\n  which would be very easy to deploy, though these\n  gains are less than GRPO.\n\nIt is encouraging to see there is no degradation\nto model utility on standard benchmarks.\n\nOverall, I consider the paper significant and the\nsignificance of the results worthy of publication.\n\nOriginality:\n\nVery good.  I haven't seen this method published\nbefore.  Developing methods that can be applied to\nreasoning models and used with GRPO is a good advance.\nThe paper is tackling an important problem that\nhasn't seen much attention.\n\n\nI have mixed feelings about the paper.  Overall, \nI think it is a net positive for the research community.\nAfter reading the paper, I was largely positive.  After\nlooking at the dataset, I am not as enthusiastic, though\nI still lean towards acceptance because of the positive\nresults.  The dataset has limited diversity and limits\nitself to system directives that are shallow syntactic\noperations from a small fixed list of possibilities.\n\nI think this limitation needs to be stated more explicitly\nand prominently, so that the community knows that this\nproblem is not solved and that the approach listed here\nhas some problems and some opportunities for improvement.\nI'm concerned that someone else is going to do a better job,\ntry to publish it, and reviewers will reject that work\nsaying that this paper already solved the problem, without\nrealizing the shortcomings of this work (because those\nshortcomings are not apparent from reading just the\npaper).  I'm concerned that the natural incentives are\nfor a paper to present its method as though it is great\n(to get the paper accepted), but this can have a harmful\nimpact on the research community.  On the one hand, it\nis surprising and interesting that such a narrow,\nsyntactic, homogenous dataset is enough to significantly\nimprove models, and the overall approach taken is\ninteresting and valuable; on the other hand, the\ncommunity needs to know that the specific dataset\nhas significant limitations and more research would\nbe appropriate, to address those issues.\n\nOther cmments:\n\nI think the paper needs to be clearer about the\nstructure of samples and how they are generated,\nboth for training and for testing.  It appears\nthere are five elements: persona, directive,\nconflicting_directive, system_segment, and user_segment.\nIt is not clear to me how these are combined into\na user message and a system message to the LLM,\nand the provided code doesn't show that, because\ncode is not provided for that part of the research.\nThis is not clearly specified in the paper.\n\nI think the paper needs to include one or two\nexamples of samples from the dataset: the system\nmessage, the user message, and the unit test,\nso that we can understand the intuition better.\n(For instance, after reading the paper, it wasn't\nclear to me whether they are Python code, regexps,\nsubstring match, something else.)\n\nThis proposal trains on system messages that are very\nconstrained in scope (e.g., only syntactic directives).\nDoes this generalize so the resulting model is good at\nenforcing system messages that are not of this limited\nform?  You might include some evaluation and/or discussion\nof to what extent it does so.  I find it hard to tell\nfrom the current evaluation, largely because I am not\ndeeply familiar with the IHEval benchmark.  IHEval\nincludes both syntactic and semantic system messages.\nI wonder if it might be possible to get significant\nimprovements on IHEval by a model that is good at\nenforcing syntactic system messages, without generalizing\nto semantic system messages.  I would find it helpful\nif you evaluated on other datasets that don't make\nmuch use of syntactic system messages (RuLES?\nRealGuardrails?  can you suggest another benchmark?).\n\nI looked at a few examples in the dataset (training_data.json),\nand the quality of some generated examples seem poor.\n\n- I noticed many cases where the directive is something\n  like \"add a comma after the word giraffe\", and the verifier\n  checks that response contains \"giraffe,\", but it seems\n  plausible that reasonable responses to the instruction \n  might not use the word giraffe at all.  I see many\n  instances of this, with different rules about what to\n  do with the word \"giraffe\", but instructions that might\n  lead to reasonable responses that don't include the\n  word \"giraffe\", and a verifier that wrongly fails\n  the response if so.\n\n- I didn't see a lot of diversity in the directives.\n  A lot of them involve doing some simple syntactic\n  transformation to a particular word (e.g., \"giraffe\")\n  in the response.  This makes me worry about\n  generalization.\n\n- \"Ensure that the last word of the sentence is\n  capitalized\" is ambiguous about which sentence; the\n  verifier checks the last sentence.\n\nDid you conduct a manual inspection of a random sample\nof generated samples, to evaluate their quality?  Since\nI quickly found some issues after a brief inspection, I\nthink you should, and you should report on the results.\n\nI found the code and repository clear and well-documented.\nKudos.  This is great, I appreciate the work you put into\nmaking this useful for others.\n\nFor non-reasoning models, the results show a modest\nimprovement over SFT+DPO with the RealGuardrails\ndataset.  However, this method has a key advantage\nover RealGuardrails that doesn't show up in the\nquantitative results.  RealGuardrails involves\ntraining with responses generated by frontier models,\nso the authority-respecting capabilities of models\ntrained by SFT+DPO with RealGuardrails is limited\nto whatever capabilities frontier models have: it does\nnot provide a way to improve frontier models.  To\nimprove frontier methods with SFT+DPO, model providers\nwould need to collect human-written examples, which\ncould be expensive.  In contrast, this paper's method\ncan be used to improve any model, including (presumably)\nthe top frontier models, and does not require any\nhuman-written examples.  As such, this paper's method\nis more broadly applicable and cheaper and easier\nto apply.  This is hinted at in the paper, but I\nwanted to call it out as an advantage, which could\nperhaps be made more explicit.\n\nI'd like to point you to some additional related work:\n\n- RuLES https://arxiv.org/abs/2311.04235\nThis work also constructs a benchmark with programmatically verifiable\ncheckers, for verifying a hierarchy of authority (rules vs prompts).\nIt contains verifiable supervision signals.  However it has a much\nnarrower scope of questions.\n\n- SoFA https://arxiv.org/abs/2402.17358\nThis work proposes a method to enforce a hierarchy of authority.\n\nSec 3 (very minor): There might be an aspect of the hierarchy\nproblem that perhaps could be made more visible.  Consider a case\nof a system message and a user message that are not totally\nmutual exclusive, in the sense that there exists an answer\nthat satisfies both, but that in the absence of the system\nmessage responses to the user message will violate the\nsystem message with high probability.  Then ideally we want\nthe LLM to output a response that satisfies both system\nand user messages; but if it is unable or too difficult\nfor the LLM to find such a response, in any case it must\nsatisfy the system message.  I don't think this is guaranteed\nby your current formalization; I think your current formulation\nmight allow the LLM to violate the system message in this\ncase.  It is fine if that is not a focus for your work.\n\nExamples:\n- \"System: Do not use any of the letters 'e', 'k', 'l', or 'r'.\n  User: What is your favorite color?\"  Ideally, the LLM\n  might respond something like \"cyan\", but an empty response\n  or \"I can't\" might also be acceptable.\n- \"System: Respond with only Python code.  Do not include\n  any conversation, comments, responses, etc., other than\n  Python code.  User: Write an efficient program to determine\n  whether White can guarantee a win at chess with best play\n  or not.\"  A response like \"Sorry I can't\" should be\n  acceptable, even though an output that satisfies both\n  requirements exists, it's just very very hard to find it.\n\nSec 4.3 (very minor): You might consider clarifying that\n$\\alpha_\\text{test}$ refers only to the unit test applied\non the first attempt, but not to failure of the unit test\napplied to repair attempts.  That confused me at first.\n\nTable 3: I would be interested to see this broken down\nby the 6 categories of IHEval (Rule/Task/Safety for each\nof Conflict/Aligned).\n\nl612-616: I'm confused.  What's the difference between\na compliance directive and a system segment?  I thought\nthe intention was that the compliance directive would be\nplaced in the system message.  How does compliance\ndirective, system prompt, and user prompt relate to\n$i^+$ and $i^-$ -- which is which?\n\nFig 6: Can you show the unit tests for this example?\n\nI tried looking at the dataset that on\nanonymous.4open.science to resolve my questions.  For\nsome reason I couldn't view data/train_data.json on\nthe website (the web page gave me an error message\nindicating that the file is not available), but it was\navailable by downloading the repository.  So I downloaded\nthe repository and used that instead.  Maybe that is a\nbug in the website.  I thought I'd let you know, in\ncase it is helpful to hear.  This is just a FYI,\nnot a complaint or criticism.\n\nQUESTIONS:\nIf accepted, will you publicly release your dataset/benchmark?\n\nLIMITATIONS:\nThe dataset is limited in diversity and narrowness,\nand the generated verifiers are sometimes not quite correct,\nand I would prefer to see this acknowledged prominently and forthrightly.\nSee above for details.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a framework that uses programmatically verifiable checkers to do instruction-conflict fine-tuning to enforce instruction hierarchy. This approach works effectively for both reasoning models and instruction-following models, providing a scalable alternative to oracle-based supervision.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- Addresses limitations of previous approaches: Oracle-based samples (from GPT-4/Claude) are expensive and potentially inconsistent, especially for reasoning models. The proposed verifier-based approach provides deterministic, consistent supervision signals that are efficient, affordable, and scalable.\n- Comprehensive experiments on recent open-source reasoning models demonstrate improved safety and conflict resolution capabilities without degrading original reasoning abilities (as shown on MATH-500 and AIME benchmarks).\n- The problem addressed has significant practical implications for LLM safety and reliability in real-world applications.\nHigh degree of completeness with thorough ablation studies for each component, including cleaning parameters, repair frequency, and black-box optimization.\n\n**Weaknesses:**\n\n- The programmatic verifier approach may have limited applicability to certain types of instructions that require semantic understanding or are difficult to verify deterministically. The effectiveness relies partly on the quality of LLM-generated verifiers.\n- The writing style occasionally uses overly complex vocabulary and long sentences, which can make some technical sections harder to follow, though this doesn't significantly impact the paper's overall quality.\n\nQUESTIONS:\nI'm curious about the computational cost and time requirements of the framework implementation. The absence of this information will not affect the review from me, but it would be great if you can provide.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for training language models (LMs) to resist harmful instructions using verifier-guided supervision. The approach leverages LMs to automatically generate training data, including compliance and conflict directives, along with verifier functions that enforce specific behavioral constraints. The authors demonstrate that fine-tuning on this data improves, or at least preserves, performance across multiple axes, including instruction hierarchy, safety and generalisation, for both reasoning and instruction-following models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The objective of training LMs to handle multi-level, conflicting, directives is significant.\n* The approach appears to be novel.\n* For the reasoning models, the results obtained for the newly proposed method are better than those obtained without fine-tuning. For instruction-following models, the results are sometimes better against the other fine-tuning techniques (more details about the less successful cases under weaknesses).\n\nWeaknesses:\n* The approach depends on generating training data (compliance directives, conflict directives and verifier functions) via LMs, which can be computationally expensive due to the large number of model invocations, including potential retries for repair.\n* It's unclear to me why the performance comparison for reasoning models in Table 1 uses no-tuned models as the baseline, rather than comparing against other fine-tuning techniques. Hopefully, the authors will clarify this in the rebuttal.\n* In Table 2, under the IHEval setting for LLaMA3.1-8B, both rule-following and safety scores decline. However, the discussion at line 266 emphasises only the cases where the proposed approach outperforms others, overlooking these regressions.\n* Performance in terms of reasoning capabilities decreases slightly for both rule-following and reasoning models (this is very mild).\n\nPresentation:\nWhile the presentation is generally clear, there are some places where it can be improved, e.g.:\n* Line 169-171: You don't explain what O_pos^+, O_pos^-, O_neg^- and O_neg^+ are.\n* Line 174, 175: \"An instance is retained only if all 3N sampled outputs satisfy their respective expected verifier outcomes.\" ==> 3N or N sampled outputs?\n\nQUESTIONS:\n* Could you try to quantify the computational cost of generating training data including the verifiers using LMs?\n* Why the performance comparison for reasoning models in Table 1 is against no-tuned, rather than some other fine-tuning techniques?\n* In Table 2, for both Qwen and LLama the proposed framework gets outperformed by other techniques for the “Safety” category. Do you have any insights as to why that might be the case and why “Safety” as a category is particularly challenging for this framework?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}