Complex Logical Instruction Generation

## Abstract

Abstract Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors. As tasks grow more challenging, the logic structures embedded in natural language instructions becomes increasingly intricate. However, how well LLMs perform on such logic-rich instructions remains under-explored. We propose LogicIFGen and LogicIFEval . LogicIFGen is a scalable , automated framework for generating verifiable instructions from code functions , which can naturally express rich logic such as conditionals, nesting, recursion, and function calls. We further curate a collection of complex code functions and use LogicIFGen to construct LogicIFEval , a benchmark comprising 426 verifiable logic-rich instructions. Our experiments demonstrate that current state-of-the-art LLMs still struggle to correctly follow the instructions in LogicIFEval . Most LLMs can only follow fewer than 60% of the instructions, revealing significant deficiencies in the instruction-following ability.

## 1 Introduction

Before the advent of ChatGPT (Ouyang et al., 2022), chatbots built on earlier models such as GPT-1 (Radford & Narasimhan, 2018), GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020), and other architectures struggled to generate coherent and contextually appropriate utterances. At that time, it was difficult to imagine that such models could assist with tasks in daily life. With the emergence of instruction-following capabilities, large language models (LLMs) are now able to accurately understand basic human intentions and even leverage tools to perform a wide range of tasks that enhance productivity, such as deep research(^1^11https://openai.com/index/introducing-deep-research/), coding assistance(^2^22https://cursor.com/en), and scientific discovery (AI4Science & Quantum, 2023). The instructions for these tasks could contain rich logic structures such as sequencing, loops, nesting, recursion, and backtracking. Previous instruction-following evaluations typically focus on instructions with constraints on the response format (e.g., “fewer than 300 words”) or content (e.g., “in Shakespeare’s tone”) (Zhou et al., 2023; Qin et al., 2024; Jiang et al., 2024) and seldom explore how well LLMs perform on the instructions with rich logic structures.

To address this, we first propose LogicIFGen, a scalable, automated framework for generating verifiable instructions from code functions, which can naturally contain rich logic structures. Given test input data, models are expected to rely solely on the natural language instruction to simulate every logic of the code function and produce the same results, analogous to being verbally guided by an examiner to process the input data step-by-step (see Figure [1](#S0.F1) (Left)). The models are required to refrain from writing code or using external tools; instead, they must simulate and execute code logic only through text generation. This setting aligns more closely with instruction-following evaluation, as most tasks require the model to explicitly unfold the underlying logic. For example, when given an instruction such as “repeat asking the user for clarification until you fully understand the intent of the user,” the model must perform each logical step through natural language alone. LogicIFGen obtains the reference labels by executing the code function on the same inputs. By comparing model outputs with these reference labels, we can easily verify whether a model follows the natural language instruction correctly. In addition, LogicIFGen incorporates state trackers to monitor intermediate logic flow, enabling us to double check whether models faithfully adhere to the instruction’s internal logic rather than hallucinating the final results. Second, we construct a benchmark called LogicIFEval using LogicIFGen, which contains 426 verifiable, logic-rich instructions paired with associated test cases. The functions used to generate LogicIFEval are the solutions of challenging simulation problems from competitive programming platforms, CodeForces(^3^33https://codeforces.com) and POJ(^4^44http://poj.org). These simulation problem solutions are especially suitable for instruction-following evaluation because they require models to faithfully emulate complex, step-by-step processes and state transitions, often involving intricate control flow, edge case handling, and the coordination of multiple logic elements.

Experimental results show that most popular LLMs are only able to correctly follow fewer than 60% of the instructions in LogicIFEval, revealing a significant deficiency in their instruction-following ability (see Figure [1](#S0.F1) (Right)). Open-source models continue to lag behind frontier models such as the OpenAI o-series and Anthropic Claude. As logical complexity increases, models find it increasingly difficult to accurately interpret and follow instructions. We also observe that incorporating explicit thinking before generating a response can potentially enhance instruction-following performance for large LLMs, but not for smaller LLMs. Further error analysis and case studies reveal key failure modes and highlight promising directions for advancing LLMs’ ability to follow logic-rich instructions. We release LogicIFGen and LogicIFEval, as well as a compute-friendly version of the benchmark, LogicIFEval-mini at:

https://github.com/mianzhang/LogicIF.

## 2 LogicIFGen : Verifiable Instruction Generation from Code

We introduce LogicIFGen, a framework that automatically generates verifiable, logic-rich instructions from code functions. Each instruction provides a comprehensive, step-by-step natural language description of the function’s behavior, clearly specifying input/output formats and detailing all relevant control flows and data processing steps. Models are then expected to follow these instructions without access to the source code to process inputs and reproduce the function’s outputs. Figure [2](#S2.F2) illustrates our instruction generation framework with a running example. Additionally, LogicIFGen integrates Multi-turn Difficulty Evolution and Multi-turn Verification and Refinement modules, which dynamically adjust instruction complexity and verify correctness to ensure each instruction fully and accurately captures the function’s logic. We provide a more detailed explanation of each module in the remainder of this section.

Figure: Figure 2: Pipeline of LogicIFGen. Given a seed function and its corresponding test cases, LogicIFGen generates natural language instructions along with gold labels, which include both the function outputs and the values of state trackers. 1) The input function is first anonymized and augmented with state trackers. 2) The anonymized function is then translated into a natural language description, producing a instruction that precisely describes its logic and expected behavior with test cases verified to have no execution errors. 3) Finally, the test cases are executed on the anonymized function to obtain the gold labels.
Refer to caption: /html/2508.09125/assets/x3.png

##### Anonymized Function with State Trackers

The first step in LogicIFGen is to anonymize the original function so that only its data operations and control logic remains. LogicIFGen use an LLM to replace the function and variable names with simple, generic identifiers, single characters or common placeholders, thereby stripping away any semantic hints that could enable models to leverage domain-specific knowledge. Moreover, LogicIFGen asks an LLM to augment the function with state trackers, which are variables designed to log specific runtime states of the function, reflecting the function’s internal logic and execution flow. These include, for example, the number of iterations of a for loop, the number of times an if block is executed, or the maximum length reached by a dynamically list. Models instructed via natural language are required to reproduce not only the function’s outputs but also the values of state trackers. We consider the instruction to be correctly followed only when both the final output and the state tracker values match the ground truth. This ensures that models truly comprehend and execute the intended logic, rather than relying on shortcuts or hallucinating plausible outputs. The prompt used for this step is shown in Figure [6](#A3.F6). Furthermore, to increase the logical complexity of the function, the Multi-turn Difficulty Evolution module in LogicIFGen prompts an LLM to evolve the code function by introducing more diverse compositions of logic units, such as loops, function calls, and recursion (see Figure [9](#A3.F9) for the full prompt).

##### Natural Instruction Generation

LogicIFGen uses an LLM (see Figure [7](#A3.F7) for the prompt) to generate detailed natural language instructions based on the anonymized function with state trackers. The instructions adopt a conversational style (e.g., “Now you need to…”, “Next, go through…”) to guide step-by-step execution. Each instruction is crafted to be sufficiently precise so that an LLM can follow it to conduct all the data operations without access to the source code, producing identical outputs and state tracker values (as illustrated in Figure [1](#S0.F1) (Left)). To ensure the instructions fully and correctly capture the sub-tasks and logic of the function, LogicIFGen incorporates the Multi-turn Verification and Refinement module. In this module, an LLM (see Figure [8](#A3.F8) for the prompt) reviews the generated instructions, checking for comprehensive coverage of all code operations and flagging potential omissions, such as loop conditions, variable updates, or edge case handling. The instructions are iteratively refined based on this feedback until they accurately reflect the complete operations and logic of the function. The full function description of of the runing example in Figure [2](#S2.F2) is shown below:

##### Test Case Filtering and Gold Result Generation

At this stage, we have two modalities representing the same operations and logic: the anonymized code function and the corresponding natural language instruction. Given the same input data, LLMs are expected to follow the instruction and produce results identical to those generated by the code function, including both the function outputs and the values of the state trackers. LogicIFGen generates these gold labels by executing the test cases on the anonymized function. Prior to this, it performs a preliminary execution step to filter out test cases that may trigger errors due to modifications introduced to function during the multi-turn difficulty evolution process. This ensures that only valid and executable test cases are retained for final gold label generation.

##### Quantifiable Instruction Complexity

One of the advantages of LogicIFGen is that the difficulty of the generated instructions is quantifiable: the anonymized function and natural language instruction describe the same operations and logic so we could use the code function as a proxy to analyze the difficulty of the instruction. We use Python’s Abstract Syntax Tree (AST) package(^5^55https://docs.python.org/3/library/ast.html) to traverse the syntactic structure of each function and calculates several key measures. Cyclomatic Complexity ($C$) quantifies the total number of control flow decision points (such as if/elif statements, for/while loops, and try/except blocks), representing the number of linearly independent execution paths through the code. Nesting depth ($D$) tracks the maximum depth of nested control structures. Function Call Count ($F$) counts the number of function invocation, including both built-in and user-defined functions. Function length ($L$) is calculated as the number of lines spanned by the function definition. We adopt an intuitive weighting scheme for aggregating these AST-based measures based on their cognitive complexity, which has been proven effective in measure the logical complexity of codes (Muñoz Barón et al., 2020).

$$ $\text{Score}=D\times 3+F\times 2+C\times 1+L\times 0.5.$ (1) $$

Our experiments in Section [4](#S4) also show the effectiveness of the intuitive weights. More advanced weighting methods like learning-based methods (Sepidband et al., 2025) could be explored in future work.

## 3 LogicIFEval

In the previous section, we introduce LogicIFGen, which generates instructions and gold labels from code functions. However, we observed that the complexity of the seed function is crucial for producing challenging instructions. To this end, we curate a collection of simulation problem solutions and corresponding test cases from competitive programming platforms, including Codeforces and POJ. Specifically, for Codeforces, we select problems tagged with “implementation” and a difficulty score above 1700. For POJ, we include difficult simulation problems as identified by various online users. These functions require models to faithfully emulate complex, step-by-step processes and state transitions, involving intricate control flow, edge case handling, and the coordination of multiple logic elements, making them ideal sources for generating complex instructions used to evaluate the instruction-following ability. Then we construct LogicIFEval based on these functions as follows:

##### Data Filtering

We use a two-stage filtering process to make the seed functions more diverse and remove test cases not suitable for the instruction-following evaluation: 1) Seed Function Filtering: We remove duplicate or highly similar functions to avoid redundancy. Specifically, we use OpenAI’s text-embedding-3-small model(^6^66https://platform.openai.com/docs/models/text-embedding-3-small) to compute embeddings of each function and calculate pairwise cosine similarities. If the similarity between two functions is greater than 0.7, we consider them near-duplicates. For each such pair, we keep only the longer function (measured by code length), since longer implementations often include richer logic and contribute to more challenging instruction-following tasks. 2) Test Case Filtering: We remove test cases with unusual or problematic outputs to keep the dataset clean and manually executable. Specifically, we discard cases where: I) The values of state trackers are greater than or equal to 50 (to avoid overly large internal states). II) The output values have excessive precision (more than six decimal places). III) The state tracker dictionaries are malformed or incorrectly formatted. IV) The input values are too large (magnitude exceeding $10^{7}$). After this step, we remove functions with fewer than 3 test cases to ensure that each function has enough test coverage for reliable evaluation. These filtering steps are designed to prevent failures caused by large loops, deep recursion, or complex numerical computations that could overwhelm model capacity. This ensures that any model failures are attributable to weaknesses in instruction following, rather than the inherent difficulty of execution. The final dataset is easy to understand and execute step-by-step, staying consistent with our goal of generating clear and verifiable instructions. Originally, we collected 1,107 seed functions. After these filtering steps, 426 unique functions and 3,050 test cases remain.

##### Data Generation and Human Verification

We apply LogicIFGen using o4-mini as the generation LLM on the filtered seed functions and test cases, resulting in LogicIFEval, a benchmark comprising 426 complex instruction-following tasks. The numbers of turns used for Multi-turn Difficulty Evolution and Multi-turn Verification and Refinement are 1 and 3, respectively. Instructions that still fail verification after 3 turns are discarded. To evaluate the quality of the generated instructions, we hired five PhD-level experts in computer science (our co-authors) to conduct manual verification. The annotators were instructed to verify whether each line of the anonymized function is accurately and completely described in the corresponding natural language instruction. According to their assessment, 97% of the instructions fully and correctly capture the underlying function logic and their agreement is 97.79%, demonstrating the effectiveness of LogicIFGen in transforming code functions into natural language instructions. More details human evaluation process could be found in Appendix [D](#A4). We further categorize the functions into difficulty levels using tercile-based thresholds derived from the complexity scores computed by Equation [1](#S2.E1). The final benchmark contains 142 easy, 145 medium, and 139 hard instructions.

##### Benchmark Statistics and Release

The instructions in LogicIFEval have average of 3,428 characters and 662 words. The functions used to generate instructions have an average cyclomatic complexity of 11.10 and a maximum nesting depth of 3.16. In total, the benchmark includes 2,049 loops, 2,253 conditional statements, and 5,289 function calls, reflecting diverse control flow patterns. Each function is evaluated with an average of 7.2 test cases, resulting in 3,050 test cases overall. These statistics highlight the benchmark’s scale and its focus on challenging instruction-following in LLMs. To support researchers with limited computational resources, we also release a representative mini-benchmark, LogicIFEval-mini, which consists of 102 functions sampled in a stratified manner based on the complexity scores. Our experimental results in Section [4](#S4) show that constructing the min-benchmark based on the complexity scores is effective.

## 4 Frontier Model Performance on LogicIFEval

We test 21 LLMs on LogicIFEval:

- •
Frontier Thinking Models: Models incorporating explicit thinking process before generating a response, including gpt-5, gpt-5-mini, o3, o4-mini, o3-mini, Claude-3.7-Sonnet and Claude-4-Sonnet, Gemini-2.5-Flash, Qwen3-32B and Qwen3-8B, DeepSeek-R1-Distill(DS)-Llama-70B and DS-Qwen-32B.
- •
Frontier NoThinking (NT) Models: Models directly give a response without explicit thinking: GPT-4.1, GPT-4.1-mini, GPT-4o, Claude-3.7-Sonnet-NT and Claude-4-Sonnet-NT, Gemma-3-27B, Qwen3-32B-NT and Qwen3-8B-NT, and Llama-3.3-70B(-Instruct).

##### Inference Setting

For closed-source models, we use the default temperature setting provided by the respective API. For open-source models, we adopt the official recommended inference settings; for example, for the Qwen3 series reasoning models, the temperature is set to 0.6. If no specific recommendation is available, we use a temperature of 1.0 by default. For all models, we set the maximum number of generated tokens to 16k to ensure that the models have sufficient capacity to follow instructions and process the inputs.

**Table 1: Model performance (%) by complexity and average. “Both” denotes both the output and state trackers match. Best results in Average columns are in bold. Overall performance are highlighted in pink. Performance of each difficulty level are in blue.**
|  | Easy (142) | Medium (145) | Hard (139) | Average |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Output | State | Both | Output | State | Both | Output | State | Both | Output | State | Both |
| Thinking Models |  |  |  |  |  |  |  |  |  |  |  |  |
| gpt-5 | 94.37 | 95.07 | 90.85 | 97.24 | 91.03 | 89.66 | 88.49 | 82.01 | 74.10 | 93.43 | 89.44 | 84.98 |
| o3 | 94.37 | 90.14 | 89.44 | 93.10 | 87.59 | 84.83 | 84.89 | 79.86 | 72.66 | 90.85 | 85.92 | 82.39 |
| gpt-5-mini | 91.55 | 92.96 | 88.73 | 93.79 | 84.83 | 82.07 | 86.33 | 78.42 | 71.22 | 90.61 | 85.45 | 80.75 |
| o4-mini | 93.66 | 90.14 | 87.32 | 91.72 | 81.38 | 77.93 | 81.29 | 68.35 | 63.31 | 88.97 | 80.05 | 76.29 |
| o3-mini | 89.44 | 87.32 | 83.10 | 89.66 | 78.62 | 73.79 | 69.06 | 61.87 | 56.12 | 82.86 | 76.06 | 71.13 |
| Claude-4-Sonnet | 91.55 | 87.32 | 81.69 | 96.55 | 77.93 | 75.86 | 73.38 | 58.27 | 51.08 | 87.32 | 74.65 | 69.72 |
| Claude-3.7-Sonnet | 81.69 | 76.76 | 70.42 | 79.31 | 64.14 | 57.93 | 57.55 | 45.32 | 39.57 | 73.00 | 62.21 | 56.10 |
| Gemini-2.5-Flash | 79.58 | 75.35 | 72.54 | 64.14 | 55.17 | 53.10 | 41.01 | 34.53 | 31.65 | 61.74 | 55.16 | 52.58 |
| DS-Qwen-32B | 66.20 | 59.86 | 50.70 | 57.93 | 36.55 | 30.34 | 38.13 | 22.30 | 15.83 | 54.23 | 39.67 | 32.39 |
| DS-Llama-70B | 69.01 | 66.20 | 54.93 | 53.10 | 36.55 | 29.66 | 38.13 | 24.46 | 15.83 | 53.52 | 42.49 | 33.57 |
| Qwen3-32B | 38.73 | 31.69 | 23.94 | 17.93 | 5.52 | 3.45 | 13.67 | 4.32 | 3.60 | 23.47 | 13.85 | 10.33 |
| Qwen3-8B | 32.39 | 27.46 | 16.90 | 13.10 | 6.21 | 3.45 | 4.32 | 2.88 | 1.44 | 16.67 | 12.21 | 7.28 |
| NoThinking Models |  |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4.1-mini | 85.92 | 81.69 | 74.65 | 82.07 | 64.83 | 60.00 | 71.22 | 49.64 | 41.73 | 79.81 | 65.49 | 58.92 |
| GPT-4.1 | 83.10 | 78.87 | 71.13 | 84.83 | 64.83 | 58.62 | 64.75 | 51.80 | 39.57 | 77.70 | 65.26 | 56.57 |
| Claude-4-Sonnet-NT | 80.28 | 76.06 | 69.01 | 63.45 | 46.21 | 37.93 | 44.60 | 30.22 | 24.46 | 62.91 | 50.94 | 43.90 |
| Claude-3.7-Sonnet-NT | 73.24 | 66.20 | 57.04 | 61.38 | 42.07 | 34.48 | 43.88 | 24.46 | 17.99 | 59.62 | 44.37 | 36.62 |
| GPT-4o | 59.86 | 49.30 | 38.73 | 32.41 | 24.14 | 17.93 | 12.95 | 10.07 | 5.04 | 35.21 | 27.93 | 20.66 |
| Qwen3-32B-NT | 33.10 | 29.58 | 19.72 | 18.62 | 9.66 | 4.14 | 9.35 | 4.32 | 2.88 | 20.42 | 14.55 | 8.92 |
| Llama-3.3-70B | 30.28 | 25.35 | 15.49 | 10.34 | 4.83 | 2.76 | 7.91 | 3.60 | 2.16 | 16.20 | 11.27 | 6.81 |
| Gemma-3-27B | 28.17 | 26.76 | 15.49 | 10.34 | 5.52 | 2.07 | 5.76 | 4.32 | 2.16 | 14.79 | 12.21 | 6.57 |
| Qwen3-8B-NT | 28.17 | 29.58 | 16.90 | 14.48 | 8.28 | 4.83 | 5.04 | 5.04 | 1.44 | 15.96 | 14.32 | 7.75 |

##### Main Results

A model is considered to successfully follow a natural language instruction if it passes all associated test cases by producing both the correct outputs and accurate state tracker values. Figure [1](#S0.F1) and the last column of Table [1](#S4.T1) report the overall accuracies across evaluated models. The top-performing models are the OpenAI gpt-5, o-series and Claude-4-Sonnet, with the best-performing model, gpt-5, achieving an accuracy of 84.98%. These results highlight the strong instruction-following and logic execution capabilities of advanced proprietary LLMs. In contrast, GPT-4o achieves only 20.66%, underperforming relative to other OpenAI models. Besides, widely used open-source models still lag significantly behind. For example, Qwen3-32B, Gemma-3-27B, and Llama-3.3-70B all score below 11%, failing to correctly execute the multi-step logic and state tracking required by LogicIFEval, which highlights a clear performance gap between commercial and open-source LLMs on this benchmark. We also notice that explicit thinking before response can potentially improve instruction following for large LLMs. For instance, Claude-4-Sonnet achieves 69.72% accuracy, notably outperforming Claude-4-Sonnet-NT (43.9%) and Claude-3.7-Sonnet (56.1%) outperforms Claude-3.7-Sonnet-NT (36.62%). Similarly, OpenAI’s thinking models, gpt5 and o-series, perform substantially better than other non-thinking models from OpenAI. However, the Qwen3 variants show little difference between thinking and non-thinking modes, suggesting that explicit thinking helps only when the underlying model has sufficiently strong capabilities. We also report model performance on LogicIFEval-mini in Figure [5](#A1.F5). The results show that models exhibit nearly identical rankings compared to the full benchmark, indicating that LogicIFEval-mini effectively represents the whole benchmark and preserves a similar distribution.

##### Results Across Difficulty Levels

Table [1](#S4.T1) presents model performance across three difficulty levels in LogicIFEval, with separate evaluations for output correctness, state tracker correctness, and their intersection (“Both”). We could see that all models show a clear degradation in performance as difficulty increases (see blue columns), validating the effectiveness of our AST-based complexity scoring strategy. For instance, gpt-5 drops from 90.85% on Easy tasks to 74.10% on Hard ones, and GPT-4.1-mini from 82.07% to 41.73%. This trend confirms that our benchmark’s stratification meaningfully reflects logical complexity of instructions. Second, output accuracy consistently exceeds state tracker accuracy across nearly all models, especially as complexity increases. For example, GPT-4.1-mini achieves 71.22% output accuracy on Hard instructions, but only 49.64% in state tracking. This implies that models may generate correct answers without strictly adhering to the intended logic steps or may follow alternative logic paths. These observations underscore the importance of adding state trackers to supervise the logic flow in complex instruction-following tasks.

## 5 Analysis

##### Failure Modes

To understand the reasons of why models can not pass the test cases, we first go through some errors and summarize them as following five types:
1) Control Flow Misexecution: incorrect, incomplete, or inconsistent execution of core control structures (e.g., loops, branches, function calls), including wrong iteration counts, improper branching, or mishandled recursion/returns.
2) State Tracking Errors: failure to correctly maintain or update internal variables or data structures, such as counters, flags, arrays, stacks, or accumulated values.
3) Missing Logic Elements: omission of required components (e.g., loops, branches, edge case handling or initialization).
4) Misordered Execution: performing steps in the wrong sequence, such as using uninitialized variables, premature function calls, or out-of-order updates.
5) Instruction Misinterpretation: misunderstanding the instruction’s intent, leading to hallucinated steps, misapplied patterns, or ignored constraints. We map the error cases to these categorizes with the help of GPT-4.1.

Figure: Figure 3: Error Type Distribution.
Refer to caption: /html/2508.09125/assets/x4.png

Figure: Figure 4: Error Cases: On the left are the excerpts from function codes where the model makes errors. On the right are excerpts from the LLMs’ responses, highlighting their failures across different modes. The explanations for model failures are indicated in red, and the corresponding code lines are highlighted. Please note that the model only has access to the natural language instruction, which could correctly describe the logic, when solving the tasks; the code is provided here solely to facilitate understanding of the errors.
Refer to caption: /html/2508.09125/assets/x5.png

Figure [3](#S5.F3) summarizes the error types made by some popular models. We could see that the most frequent error categories across models are Control Flow Misexecution, Instruction Misinterpretation, and State Tracking Errors. In contrast, Missing Logic Elements and Misordered Execution are consistently low (mostly under 10%), suggesting that most models can identify the required logic components (”what to do”) and their approximate ordering (”when to do it”). However, they often struggle with actually executing the logic elements correctly, either by mismanaging control structures (e.g., loop iterations, function calls), hallucinating or misinterpreting instruction details, or failing to track internal state variables accurately over time. In addition, open-source models like Qwen3-32B, DS-Llama-70B, and Gemma-3-27B exhibit especially high rates of Control Flow Misexecution, up to 53.5% in the case of Gemma-3-27B. This highlights their difficulty in faithfully reproducing the logic-heavy instruction steps, which require consistent handling of nesting, conditionals, and function boundaries. Besides, State Tracking Errors is also a major issue across nearly all models. For instance, Claude-4-Sonnet and Gemini-2.5-Flash show 38.8% and 32.4% error rates respectively, indicating frequent failures in maintaining correct variable states when following the instructions. These include losing track of counters, failing to propagate updates through data structures, or resetting intermediate results incorrectly. This reinforces the importance of evaluating beyond output correctness. We show representative cases for each error type in Figure [4](#S5.F4) and the full examples in Appendix [B](#A2). It should be noted that the error types are not strictly mutually exclusive. For example, the missing logic elements error observed in the fourth example of Figure [4](#S5.F4) can also be attributed to the incorrect execution of the while condition.

##### Why Thinking Helps Large LLMs

As indicated in Section [4](#S4), incorporating explicit thinking before generating a response can potentially enhance instruction-following performance for large LLMs. To gain deeper insights into this effect, we conduct a case analysis focusing on instances where Claude-4-Sonnet produces correct results, whereas Claude-4-Sonnet-NT fails. These cases can be categorized into two types: 1) the model arrives at the correct answer during the thinking process itself, and 2) the model articulates a detailed, step-by-step plan to solve the task. Both scenarios suggest that explicit thinking encourages the model to slow down and solve the problem more deliberately, rather than relying on pattern matching or intuitive leaps. We provide two representative examples in the Appendix [E](#A5) due to space limit.

## 6 Related Work

General instruction-following evaluation work typically focuses on instructions that impose constraints on the response format or content (Zhou et al., 2023; Qin et al., 2024; Wen et al., 2024; Pyatkin et al., 2025). These research centers on enabling models to follow instructions with multiple constraints (Jaroslawicz et al., 2025; Pyatkin et al., 2025). Verification methods generally fall into two categories: heuristic functions or LLM-as-judge. While LLM-as-judge has been shown to correlate highly with human judgments (Qin et al., 2024), there is still room for improvement (Zeng et al., 2023). In addition to general-purpose instruction-following benchmarks, other datasets target specific scenarios or constraints, such as length control (Zhang et al., 2025), long-context settings (Wu et al., 2024), or agentic scenarios (Qi et al., 2025). Yang et al. (2025) explore LLMs’ ability to adhere to user intent while producing functionally accurate code. In contrast, our work uses code as the source to generate instructions, requiring LLMs to generate text, rather than code, in response. Some studies have suggested that reasoning may decrease instruction-following performance (Li et al., 2025; Fu et al., 2025). However, our findings in Section [4](#S4) indicate that reasoning can actually enhance instruction-following for logic-rich instructions. Further research like Tam et al. (2024); Qin et al. (2025) is needed to clarify the relationship between reasoning and instruction following. To the best of our knowledge, we are the first to systematically investigate whether LLMs can precisely follow logic-rich instructions and how to generate such instructions in scale.

## 7 Conclusion and Future Work

In this paper, we introduce LogicIFGen, a framework that automatically generates verifiable, task-intensive, logic-rich instructions from code functions. We also present LogicIFEval, a challenging instruction-following evaluation benchmark constructed using LogicIFGen, which consists of 426 tasks featuring complex logic. Experiments show that most proprietary and open-source models struggle to solve these tasks, revealing a significant deficiency in instruction-following capabilities. Future work includes exploring the use of LogicIFGen as a verifiable instruction generator for model training and LogicIFEval for evaluation to develop models with generalized and robust instruction-following capabilities.

## References

- AI4Science & Quantum (2023)
Microsoft Research AI4Science and Microsoft Azure Quantum.
The impact of large language models on scientific discovery: A preliminary study using GPT-4.
*arXiv [cs.CL]*, November 2023.
- Brown et al. (2020)
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
Language models are few-shot learners.
*arXiv [cs.CL]*, May 2020.
- Fu et al. (2025)
Tingchen Fu, Jiawei Gu, Yafu Li, Xiaoye Qu, and Yu Cheng.
Scaling reasoning, losing control: Evaluating instruction following in large reasoning models.
*arXiv [cs.CL]*, May 2025.
- Jaroslawicz et al. (2025)
Daniel Jaroslawicz, Brendan Whiting, Parth Shah, and Karime Maamari.
How many instructions can LLMs follow at once?
*arXiv [cs.AI]*, July 2025.
- Jiang et al. (2024)
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang.
FollowBench: A multi-level fine-grained constraints following benchmark for large language models.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 4667–4688, Stroudsburg, PA, USA, 2024. Association for Computational Linguistics.
- Li et al. (2025)
Xiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen, Ziji Zhang, Yingying Zhuang, Narayanan Sadagopan, and Anurag Beniwal.
When thinking fails: The pitfalls of reasoning for instruction-following in LLMs.
*arXiv [cs.CL]*, May 2025.
- Muñoz Barón et al. (2020)
Marvin Muñoz Barón, Marvin Wyrich, and Stefan Wagner.
An empirical validation of cognitive complexity as a measure of source code understandability.
In *Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)*, New York, NY, USA, October 2020. ACM.
- Ouyang et al. (2022)
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E Miller, Maddie Simens, Amanda Askell, P Welinder, P Christiano, J Leike, and Ryan J Lowe.
Training language models to follow instructions with human feedback.
*Adv. Neural Inf. Process. Syst.*, abs/2203.02155, March 2022.
- Pyatkin et al. (2025)
Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi.
Generalizing verifiable instruction following.
*arXiv [cs.CL]*, July 2025.
- Qi et al. (2025)
Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li.
AGENTIF: Benchmarking instruction following of large language models in agentic scenarios.
*arXiv [cs.AI]*, May 2025.
- Qin et al. (2024)
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu.
InFoBench: Evaluating instruction following ability in large language models.
*arXiv [cs.CL]*, January 2024.
- Qin et al. (2025)
Yulei Qin, Gang Li, Zongyi Li, Zihan Xu, Yuchen Shi, Zhekai Lin, Xiao Cui, Ke Li, and Xing Sun.
Incentivizing reasoning for advanced instruction-following of large language models.
*arXiv [cs.CV]*, July 2025.
- Radford & Narasimhan (2018)
Alec Radford and Karthik Narasimhan.
Improving language understanding by generative pre-training.
2018.
- Radford et al. (2019)
Alec Radford, Jeff Wu, R Child, D Luan, Dario Amodei, and I Sutskever.
Language models are unsupervised multitask learners.
*OpenAI blog*, 1(8):9, 2019.
- Sepidband et al. (2025)
Melika Sepidband, Hamed Taherkhani, Song Wang, and Hadi Hemmati.
Enhancing LLM-based code generation with complexity metrics: A feedback-driven approach.
*arXiv [cs.SE]*, May 2025.
- Tam et al. (2024)
Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-Yi Lee, and Yun-Nung Chen.
Let me speak freely? a study on the impact of format restrictions on performance of large language models.
*arXiv [cs.CL]*, August 2024.
- Wen et al. (2024)
Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang, Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxin Xu, Yiming Liu, Jie Tang, Hongning Wang, and Minlie Huang.
Benchmarking complex instruction-following with multiple constraints composition.
*arXiv [cs.CL]*, July 2024.
- Wu et al. (2024)
Xiaodong Wu, Minhao Wang, Yichen Liu, Xiaoming Shi, He Yan, Xiangju Lu, Junmin Zhu, and Wei Zhang.
LIFBench: Evaluating the instruction following performance and stability of large language models in long-context scenarios.
*arXiv [cs.CL]*, November 2024.
- Yang et al. (2025)
Jian Yang, Wei Zhang, Shukai Liu, Linzheng Chai, Yingshui Tan, Jiaheng Liu, Ge Zhang, Wangchunshu Zhou, Guanglin Niu, Zhoujun Li, Binyuan Hui, and Junyang Lin.
IFEvalCode: Controlled code generation.
*arXiv [cs.CL]*, August 2025.
- Zeng et al. (2023)
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen.
Evaluating large language models at evaluating instruction following.
*arXiv [cs.CL]*, October 2023.
- Zhang et al. (2025)
Wei Zhang, Zhenhong Zhou, Kun Wang, Junfeng Fang, Yuanhe Zhang, Rui Wang, Ge Zhang, Xavier Li, Li Sun, Lingjuan Lyu, Yang Liu, and Sen Su.
LIFEBench: Evaluating length instruction following in large language models.
*arXiv [cs.CL]*, May 2025.
- Zhou et al. (2023)
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou.
Instruction-following evaluation for large language models.
*arXiv [cs.CL]*, November 2023.

## Appendix A LogicIFEval-mini Performance

Figure: Figure 5: Overall instruction-following performance (%) of all evaluated models on LogicIFEval-mini, measured by the percentage of test cases where both the output and state trackers match the ground truth. Models are sorted by overall accuracy.
Refer to caption: /html/2508.09125/assets/x6.png

## Appendix B Error Cases

### B.1 Error Case: POJ-1852

Error Type: Control Flow Misexecution

Test Input:

LLM Results:

Code Results:

Function:

### B.2 Error Case: CodeForces-328A

Error Type: Instruction Misinterpretation

Test Input:

LLM Results:

Code Results:

Function:

### B.3 Error Case: CODEFORCES-2018B

Error Type: Misordered Execution

Test Input:

LLM Results:

Code Results:

Function:

### B.4 Error Case: CODEFORCES-1146E

Error Type: Missing Logic elements

Test Input:

LLM Results:

Code Results:

Function:

#### Step 1: Initialize variables

#### Step 2: Scan the array arr to fill list A

##### $i=0$ :

##### $i=1$ :

##### $i=2$ :

##### $i=3$ :

##### $i=4$ :

#### Step 3: Build list A2

#### Step 4: Build list B

#### Step 5: Setup C (BIT array) with $n=5$

#### Step 6: Initialize D and j

##### Processing B[1] = (2, 4, 1)

##### Processing B[2] = (3, 3, 2)

#### Final D after all queries:

#### Step 1: Initialize variables

#### Step 2: Scan the array arr to fill list A

##### i = 0 i=0 :

##### i = 1 i=1 :

##### i = 2 i=2 :

##### i = 3 i=3 :

##### i = 4 i=4 :

#### Step 3: Build list A2

#### Step 4: Build list B

#### Step 5: Setup C (BIT array) with n = 5 n=5

#### Step 6: Initialize D and j

##### Processing B[1] = (2, 4, 1)

##### Processing B[2] = (3, 3, 2)

#### Final D after all queries:

### B.5 Error Case: CodeForces-1481D

Error Type: State Tracking Error

Test Input:

LLM Results:

Code Results:

Function:

## Appendix C Prompts

Figure: Figure 6: Prompt for Adding State Trackers and Anonymizing Functions

Figure: Figure 7: Prompt for Generating Natural Language Description

Figure: Figure 8: Prompt for Verifing Natural Language Description

Figure: Figure 9: Prompt for Function Evolution

## Appendix D Human Verification

To ensure the quality of the generated instructions, we conducted a human verification study on 136 randomly sampled instances from LogicIFEval. The data were split evenly into four parts, each containing 34 instructions, and each part was assigned to two independent PhD-level annotators with expertise in computer science. Annotators were instructed to verify whether every line of the anonymized function was accurately and completely captured in the corresponding natural language instruction. For Batches 1 and 2, both annotators fully agreed on all items, yielding 100% agreement. In Batch 3, both annotators marked 32 out of 34 items as correct, with one identical negative case, resulting in an agreement rate of $33/34\,(97.06\%)$. In Batch 4, the two annotators agreed on 32 out of 34 items ($94.12\%$). Overall, the human agreement rate across all 136 items was $\frac{133}{136}\approx 97.79\%$, demonstrating a high level of consistency and validating the reliability of the generated instructions.

## Appendix E Cases of Thinking Helps Large LLMs

### E.1 Case 1: CODEFORCES-818D

Test Input:

LLM Results:

Code Results:

Function:

### E.2 Case 2: POJ-1516

Test Input:

LLM Results:

Code Results:

Function:

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - I would like to see how the current models w/ strong reasoning methods perform on this task.
- I'm curious how do the author view the difference between using natural language instructions vs directly using code as instructions and asking the model to follow the logic.
soundness: 3
strengths: - The paper studies an interesting problem of building complex natural language instructions from code and test the model's instruction following ability by using these code generated instructions.
- The paper's main pipeline of building these instructions is interesting and solid. The paper also did a decent job in collecting the coding problems which could be a contribution to the community.
summary: The paper proposes LogicIFGen and LogicIFEval. LogicIFGen is a method framework of generating natural language instructions based on code, and LogicIFEval is a benchmark derived by using the framework on hard coding problems. Then the authors tested the current LLMs on the benchmarks and shows it is very challenging for the current models.
weaknesses: - The naming is very confusing. Fundamentally LogicIFGen is the framework and LogicIFEval is derived from using this. But the naming made this very misleading. 
- I think the analysis part is not solid enough. For example, some simple reasoning baselines such as Program-of-Thought should be tested and analyzed. Since the instruction is derived from code, I think in general the paper should consider the aspect of reasoning with code.

---

confidence: 3
contribution: 1
initial_rating: 2
presentation: 2
questions: 1.  Table 1 should specify whether performance is a percentage of test cases or questions. The number of questions in brackets after the difficulty levels seems to indicate a percentage of questions. However, it is unclear whether a question is counted if only some test cases are correct in terms of output, state, or both. The numbers appear to be a percentage of test cases, especially when compared to Figure 3. For example, GPT-5 has state tracking errors in 28.1% of questions but could not achieve a "state" performance of 89.44% correct answers; rather, it achieved this result for only a subset of test cases.
2.  Why is the fourth result group labeled "Average" when the subtitle says "Overall"? The most intuitive average would be the weighted and unweighted means of the three columns on the left, respectively. However, that is not the case. Is this about overall performance rather than an average?
3.  Figure 3 needs a subtitle and should specify if it is about at least one of the specific failure modes in some test case per question.
soundness: 2
strengths: Overall, the paper is easy to follow. The experiments appear to be fully documented and reproducible, and the topic of assessing LLM abilities in relation to the logical complexity of the task is highly relevant.
summary: The paper introduces LogicIFGen, a method that uses an LLM to anonymize functional programs. This method further introduces state tracking variables and translates the programs into a natural language description of the function. The method is used to create a benchmark dataset, called LogicIFEval, for evaluating the ability of LLMs to follow instructions when asked to execute the provided natural language instruction step-by-step for several test cases. The complexity of the instructions is captured using a syntax-tree-based measure on the anonymized programs. Several frontier LLMs were tested on the benchmark, revealing declining performance with increasing instruction complexity. Further analysis of the models' different failure modes was conducted.
weaknesses: 1.  The paper's main weakness is the limited relevance and coherence of the analysis conducted. Although assessing the abilities of LLM in instruction-following tasks by investigating dependency on increasing levels of logical complexity is promising, the main results focus on comparing the performance of different models. The overall pattern of declining performance with increasing instruction complexity is briefly discussed. However, the subsequent analysis of different failure modes does not consider varying difficulty levels. Hence, there is no insight into whether or how these failure modes depend on instruction complexity. This does not align with the paper's motivation and potential novelty. To significantly improve the paper's quality, the benchmark and related analysis should focus on the connections between LLM behavior and the logical complexity of instructions. This would allow the paper to fulfill the promises made in the abstract and introduction. 
2.  A second weakness is the lack of discussion or theory on the motivation behind the introduced dataset. Why is it important to explicitly prohibit the model from using programming or tools in general? Does the benchmark provide evidence for or against systematic compositionality in LLM behavior?

In summary, the paper appears to be immature in its current state. It presents an interesting idea for using LLM to process a selection of functions into datasets for assessing LLM instruction-following abilities in relation to task complexity. However, it lacks a sufficient theory explaining why and how these abilities can be assessed with the proposed dataset. Additionally, the analysis is insufficient for meaningfully evaluating the LLM behavior related to task complexity.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. It would be really helpful if the authors could kindly address concerns mentioned above.

2. The theme of this paper is testing complex logic. What is the great advantage of using code as backbone to generating the questions instead of using classic formal logic frameworks such as first-order logic? The formal logic is also verifiable and easily programmable.
soundness: 3
strengths: 1. The proposed framework, LogicIFGen, is scalable and verifiable. The authors also conduct human studies to show that the generation is of high quality.
2. The research question itself is interesting and important: LLMs have to be able to follow various types of instructions, which potentially involve complex logic relations, to properly serve users.
3. Writing and presentation are very clear. It is very easy to understand the authors' points.
summary: This paper proposes LogicIFGen, a framework which enables automatic generation of test instances for logic relations based on code. The framework is scalable and verifiable. Using this framework, the authors construct LogicIFEval, a benchmark curated to evaluate LLM capabilities in logic understanding and instruction following. Extensive empirical experiment results show that many of the models have poor performance on the benchmark, highlighting the issues in using LLMs to handle complex logic queries.
weaknesses: 1. Related work section is not comprehensive enough. It reviews many works which are relevant but not closely relevant to the research question in the general LLM reasoning area. More strongly related works should be thoroughly discussed as I will elaborate below.

2. The novelty of the research question is not extremely clear given many existing works in highly similar areas. This is my biggest concern. I think there are many existing works in code execution which are highly relevant, but not fully discussed in the related work section. For example, [1,2,3,4, inter alia] have done comprehensive analysis on the limitations of LLMs in simulating code execution result, trace, and executing natural language instructions which can be solved/verified by code. The authors should more carefully review relevant literature and argue for their novelty.

[1] La Malfa, E., Weinhuber, C., Torre, O., Lin, F., Marro, S., Cohn, A., ... & Wooldridge, M. (2024). Code simulation challenges for large language models. arXiv preprint arXiv:2401.09074.

[2] La Malfa, E., Weinhuber, C., Torre, O., Lin, F., Huang, X. A., Marro, S., ... & Wooldridge, M. (2025). Code Simulation as a Proxy for High-order Tasks in Large Language Models. arXiv preprint arXiv:2502.03568.

[3] Sun, S., Hsieh, C. P., Ladhak, F., Arakelyan, E., Serano, S. A., & Ginsburg, B. (2025). L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution. arXiv preprint arXiv:2503.22832.

[4] Liu, C., Dylan Zhang, S., Ibrahimzada, A. R., & Jabbarvand, R. (2024). Codemind: A framework to challenge large language models for code reasoning. arXiv e-prints, arXiv-2402.


Decision: Reject