Attend to the Active: Structure-Aware Dynamic Attention in LLMs for Compositional Instruction Following
Authors: Fangrui Lv
Yulei Qin
Ruixin Hong
Jian Liang
Jinyang Wu
Ke Li
Xing Sun
Changshui Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Instruction Following; Dynamic Attention; Large Language Models
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models (LLMs) have exhibited strong instruction-following capabilities; however, they often struggle with compositional instructions involving multiple interleaved yet logically independent sub-tasks. These sub-tasks are typically organized in mutually exclusive structures, such as branching, chaining, or paralleling, where only one sub-task should be active at each generation step, while the others remain dormant.  Despite their inactivity, dormant sub-tasks can inadvertently attract the model's attention due to structural entanglement within the input context or intermediate representations, leading to interference that compromises output fidelity. To address this challenge, we propose ATA, a structure-aware dynamic attention mechanism grounded in compositional structures, which dynamically identifies the active sub-task during generation while suppressing attention to inactive ones. By precisely steering the model’s focus, ATA mitigates interference and explicitly enhances model adherence to the active sub-task.  Importantly, ATA operates within a single forward pass without requiring parameter updates. Extensive experiments show that ATA consistently enhances LLMs' instruction-following ability across various compositional structures, effectively mitigating attention distraction and demonstrating a strong generalization ability.

Published as a conference paper at ICLR 2026
ATTEND TO THE ACTIVE: STRUCTURE-AWARE DYNAMIC ATTEN- TION IN LLMs FOR COMPOSITIONAL INSTRUCTION FOLLOWING
Fangrui Lv Yulei Qin' Ruixin Hong'
Jian Liang? Jinyang Wu'
Ke Li Xing Sun* Changshui Zhangl
Beijing National Research Center for Information Science and Technology (BNRist)
Department of Automation, Tsinghua University, Beijing, P.R.China
2 Teneent Youtu Lab Kuaishou Technology
1vfr tsinghua. edu. cn hongrx x 6@tsinghua. org. cn
lyuleiqin, tristanli, wttpstttaiwwiiierrsssttttteeeeee wwwnfreduun/etencentccom com
liangjian@ shouo com
wu- wy-jy230mails, tsinghua. edu.cn cn
zcs@mail. tsinghua. edu.cn
ABSTRACT
Large language models (LLMs) have demonstrated strong capabilities; however, they often struggle with compositional instructions that in- volve multiple interleaved yet logically independent sub-tasks. These sub-tasks are typically organized in mutually exclusive structures, such branching, chaining, or paralleling, where only one sub-task should be active at each generation step, while the others remain dormant. Despite their inactivity, dormant sub-tasks inadvertently attract the model's attention due to structural entanglement within the input context or intermediate representations, leading interference that compro- mises output fidelity. To address this challenge, we propose ATA, a structure-aware dynamic attention mechanism grounded in compositional structures, which dynam- ically identifies the active sub-task during generation while suppressing attention to inactive ones. By precisely steering the model's 's focus. ATA mitigates interfer- ence and explicitly enhances model adherence to the active sub-task. Importantly, ATA operates within a single forward pass without requiring parameter updates. Extensive experiments show that ATA consistently enhances LLMs' instruction- following ability across various compositional structures, effectively mitigating attention distraction and demonstrating a strong generalization ability.
INTRODUCTION
Large language models (LLMs) have remarkably succeeded in numerous natural language processing tasks (Zhao et al. 2023 OpenAl 2023). With their growing capabilities, LLMs are increasingly applied to handle compositional instructions involving multiple sub-tasks in real-world scenar- ios (Ouyang et al. 2022 Xu et al. 2024p. However, existing research has predominantly focused on compositional instructions requiring the simultancous completion of interleaved sub-tasks (He) let al. 2024b) (Sun al. 2024), the exploration of complex composition structures remains limited. Consider the example in Figure (left), the compositional instruction comprises two independent sub-tasks, but only should be executed according the condition judgement. The model should accurately identify and faithfully follow the selected active branch while avoiding interference from the unselected inactive one. Existing LLMs, however, often exhibit degraded performance in such intricate structures due to spurious attention to inactive sub-tasks (Wen et al. 2024). We attribute this limitation to the model's tendency to diffusely attend across the entire input context (Zhang et al. 2024bb Zhu et al. 2024), making it susceptible to distraction from irrelevant sub-tasks. This inability to selectively focus often leads to ambiguous reasoning or inconsistent outputs, ultimately undermining the model's 's ability to follow complex compositional instructions (Wen al. 2024).
Building on prior work in the community (Hayati et al. 2025 Wen et al. 2024), we identify three prototypical composition structures commonly observed in compositional instructions, illustrated in Figure?? 2 i) Chaining-sequena execution of sub-tasks, ii) Branching-conditionnn selection of
"Corresponiigg author.

---

Published as a conference paper at ICLR 2026
Please introduce the following painling
If the work contains animal, the descriation should be in Chinese Otherwise the description should English.
25
Dirct
Decomposition
23
20-
Think_xxecute
Painting: "Mona Lisa"
Sei_ corraction
contains any animal?
Rato
15
12.7552
Yes
No
10
the desoription
the description
should in Chinese
should be in Englissh
Atfention Score
Wrong Generation
Mixed Generation
Omitted Generation
Please following sainting, Fainting: "Mona
Nrong Generation
[Confused] Interfered the draped branch
it contairs any animal. the description should Chineso. Othenwise, description should English.
Mona Lisa is created by the Italian Renaissance artist Leonardo Vinci m) This masterpiece, also known 11
select wrong branch
Original Output:
Mixed Generation
SE3
famous paintings...
Painting "Mona lisw": Painting 2: Starry Night"
"The Mona Liaa"" paaineed Vincent van Gogh, masterpiece of mix their information
that
0
0
o
0 0
[Focused] strictly adhere the selected branch
Oritted Generation
lintly, introduce year creation, descrbbe background the impart of
Steered Output:
The Mona Lisa a portrait painted by Leonardo da Vind during the Renaissanco. The painting renowned for its enigmatic smile...
fisall,
A: The "Mona Liaa" was painted around 1503 by Leonando da Vindi. This painting is famous for
omit the second Sub-task
Figure I: (left) LLMs with original attention distributed across the entire input or with steered attention eliminating the interference from inactive sub-tasks. (right) Distribution and examples of the error types in compositional instructions generated by Llama3-8h.
sub-tasks, and iii) Paralleling-paral stacking of independent sub-tasks. Our systematic analysis reveals that these sub-tasks, although interleaved, are often logically mutually exclusive. That is, only one sub-task (e.g.. the selected branch, the currently executed chain or parallel element) should be active and govern the output at each generation step, while others remain dormant and may cause interference. It is worth noting that the status of sub-tasks (active vs. inactive) can dynamically shift throughout the generation. While existing approaches have made progress through techniques such as fine-tuning (Jiang et al. 2024 He et al. (024a), high-level planning (Chae et al., 2024 Wang et al. (202)) or iterative self-reflection (Sun et al. 2024 (Shinn et al. 2023). they often incur substantial computational overhead and still fail to fully suppress attention dormant sub-tasks during inference. To better understand these limitations, we analyze typical error types in compositional instruction following (Figure [ (right)): i) Wrong Generation: erroneously executing inactive sub-tasks; ii) Mixed Generation: conflating multiple exclusive sub-tasks, leading to incoherent or blended reasoning; iii) Omitted Generation: skipping one or more required active sub-tasks. These issues largely stem from blended comprehension, where the model conflates mutually exclusive sub-tasks, and from unfocused tracking of dynamic active sub-tasks during generation. Motivated by the observation, we highlight critical need for handling compositional instructions: mitigating interference from inactive sub-tasks during both task comprehension and generation.
Building upon this insight, we propose ATA (Attend To the Active), a seuucture-awary dynamic attention framework to tackle the challenges posed by compositional instructions. Recognizing that sub-tasks in such instructions are typically organized in mutually exclusive structures, ATA first analyzes the compositional structure type and extracts constituent sub-tasks. To prevent blending comprehension of multiple sub-tasks during encoding, ATA applies mutual attention masking be- tween exclusive sub-task pairs, ensuring their representations remain disentangled and semantically independent. During generation, ATA dynamically identifies the currently active sub-task for next token prediction by analyzing attention scores, while treating the exclusive ones as dormant. It then selectively suppresses attention to these inactive sub-tasks, minimizing interference and guiding the model's 's focus toward the active sub-task. By enforcing this structure-aware dynamic attention modulation throughout the generation process, ATA enables I.L.Ss to consistently and pertinently attend to active sub-tasks, thereby reducing ambiguity and improving output fidelity. Notably, ATA operates in a plug-and-play manner with a single forward pass and requires no parameter updates, offering an efficient and versatile technique to boost various LLMs.
is worth noting that the structure information serves as a reliable guide to conduct safe attention masking in our ATA framework. The attention steering mechanism is confined to the structurally mutually exclusive sub-tas bub-tasks, ensuring the surrounding context remains untouched. Besides, we propose perform attention masking at the granularity of independent and complete sub-tasks, such
2

---

Published as a conference paper at ICLR 2026
that the mix across multiple sub-tasks is prevented for disambiguation during masking. In addition, ATA selectively interposes at a subset of the most relevant attention heads, minimizing the risk of global information attenuation. Consequently, ATA ensures the effective climination of interference from irrelevant information while preserving critical details for the integrity of the overall structure.
To the best of our knowledge, this work is the first systematically identify the reasons behind the performance degradation of LLMs in tackling compositional instructions, and also pioneers the introduction of parallel structure in this field. We highlight the role of recalibrated attention in mitigating interference from inactive sub-tasks. Comprehensive experiments across diverse bench- marks and models consistently demonstrate that ATA significantly outperforms various baselines. effectively shields distraction attacks under various composition structures, showcasing its robustness and versatility. Furthermore, its parameter-free modifications and plug-and-play compatibility with various off-the-shellf LLMs guarantee both efficiency and practicability in real-world scenarios.
2
RELATED WORK
Compositional Instruction Following. With the increasing complexity in real-world tasks, the study of compositional instructions that involve multiple sub-tasks has garnered significant attention. Prior studies primarily focus on integrating diverse coupled sub-tasks such as semantic (Zheng et al. 2023 Jiang al. 2024) and format (Xu al., 2024 He et al., (024b) constraints, which must be satisfied simultaneously. However, the exploration of compositional instructions comprising multiple independent or mutually exclusive sub-tasks remains limited, where the sub-tasks may be organized with complex structures. Recent efforts, such as Chain-of-Instruction (Hayati et al. (025), model sequential execution of sub-tasks, while Complexbench (Wen al., 2024) introduces conditional selected branches. Nonetheless, we are the first to introduce parallel structure, fundamental yet underexplored paradigm real-world scenarios. Moreover, existing research lacks systematic investigation into the inherent commonalities across different compositional structures to enhance LLMS' reasoning fidelity. Existing works largely rely on fine-tuning with curated datasets (Xu et al., 2024 Hayati et al. 2025 Jiménez-Navarroo al. 2024), which is ensource-intensive Alternative approaches include iteratively output refinement via self-reflection (Sun et al. 20244) [Shinn et al.
2023), decomposition of compositional tasks into sequentially solvable sub- babb--tkks (Wen et al. [024]] and high-level planning followed by step-by-step execution (Chae et al.] 2024; Wang et al. 2023 ).
These methods, however, typically depend heavily on high-quality intermediate steps and require multiple inference iterations that lead to inefficiencies. In contrast, our approach innovatively modifies the attention mechanism to handle compositional instructions within single forward pass. Furthermore, it generalizes effectively across diverse complex structure types and supports plug-and-plas functionality, demonstrating superior versatility and generalization ability.
Attention Steering, Advances in attention mechanisms have significantly boosted model performance across various tasks. For instance, SampleAttentie (Zhu al. (024) and SASK (Jin et al., 2024) improve the efficiency of long-context LLM inference through adaptive sparse attention, which compresses redundant information. However, these strategies often risk performance drops due to the loss of critical global information. Besides, SAR (Duan et al. (024) shifts attention to more relevant components at the token level for better uncertainty quantification, while PASTA and autoPASTA (Zhang et al.) 2024a b) scale up attention on predefined or self-selected input segments to highlight key information. Yet, their emphasized text remains fixed during generation, making them unsuitable for compositional instructions where active sub-tasks shift dynamically. Differently, proposed ATA constrains attention steering to mutually exclusive sub-task spans, preserving global context. And it dynamically adjusts attention by identifying the active sub-task at each generation step, thus maintaining information integrity while effectively eliminating interference.
3 FORMULATION
3.1 COMPOSITIONAL INSTRUCTIONS
For a comprehensive analysis, we build on the chain and branch composition structures from existing literature, which encompass most compositional types in real-world LLM instructions (Wen et al.
3

---

Published as a conference paper at ICLR 2026
Structure Type
Definition
Example
Illustration
Please introduce "Mona Lisa" briefly, Firstly, Introduce the year creation, then describe the background of the work's creation, and finally, summarize the impact of the work.
The generation required complete multiple sub-tasks sequentially.
7
Chain
Please introduce the following painting.
Sy(cond,)
The generation is required to select different branches according to certain conditions.
work contains any animal, the description
Branch
should be in English
Otherwise, the description should be Chinese. Painting: "Mona Lisa"
T,
The generation required to complete multiple independent tasks in parallel.
Please introduce the following paintings. Painting 1: "Mona Lisa"
Parallel
Painting 2: Starry Night"
Figure 2: Structure type in Compositional Tasks. The blue nodes are independent or mutually exclusive sub-tasks, while the grey nodes are other content in the context such selection condition.
2024 Hayati et al. 2025). We further broaden our coverage by introducing the parallel structure, common yet underexplored composition type. Their formal definitions are as follows.
Chain. The generation is required to complete multiple sub-tasks sequentially. Formally, Chain contains m sub-tasks T, T..... Tm, which need be executed in sequence. The generation of T441 depends on that of Tg (k 1,2, .... .m 1).
Branch. The generation is required to select specific branch according to given conditions, fulfilling the sub-task associated with the chosen branch. Formally, Branch contains distinct branches, each corresponding to a sub-task Ti, T2. Tm with respective expected outputs Y,, Y2. .ym. We denote selection function S with range [1. taking the selection condition cond as input. This function determines which sub-task to execute. Finally, the expected generation is Ystcomd)- Parallel. The generation is required to complete multiple sub-tasks independently. Formally, Parallel contains m independent sub-tasks Ti, T..... ... Tm. each of which is executed individually with the respective expected outputs Y1, Y2, Ym. Ultimately, the expected generation is a parallel joint of these individual outputs.
Complexity. We define the complexity of compositional instruction by the number its structurally mutually exclusive sub-tasks. As this complexity escalates, the challenge of adhering to associated requirements proportionally intensifies. Notably, each sub-task may encompass multiple constraints that must be satisfied concurrently.
In the standard paradigm of LLM generation, a pre-ned LLM is provided alongside compositional instruction T comprising sub-tasks, represented as T = (T,, T2.... .Tm). We denote T, T Tj for i # j to indicate that they are structurally exclusive. During each generation step, only one sub-task, denoted Tayy (key mI). is active and should govern the output, while the structurally incompatible sub-tasks T, that satisfy T, 1 Teey fall in a dormant and thus irrelevant state. This work aims to dynamically identify and focus the active sub-task while mitigating interference from inactive ones, enhancing the output fidelity of the complex compositional instructions.
3.2 MULIT-HEAD ATTENTION
typical mansformer-baseee LLM consists of L. stacked layers, where each layer contains two sub- modules: multi-head attention (MHA) and fully connected feed-forward network (FFN). Given the input X e Rexd, MHA of the layer / performs the attention function in parallel // heads: MHA (X) Concat( (l.)) H with
H Softmax (A Softmax (QK Vdn) V.
where XW a..... K XW V XW,,, and W n matrices of head h. d, is typically d/H. (lh denotes the attention scores at head in layer
W., W., E Raxd, are learnable projection
4

---

Published as a conference paper at ICLR 2026
4 METHOD
Our proposed ATA framework seamlessly integrates structure analysis with attention steering. The structure analysis component identifies the composition structure and mutually exclusive sub-tasks. Equipped with the identified structure as guidance, the attention steering component safely identifies the active sub-task and effectively eliminates interference from inactive ones during both task comprehension and generation. For clarity, the overall algorithm provided Alg. T
4.1 STRUCTURE ANALYSIS AND IDENTIFICATION
Given compositional instruction T, we first need to accurately identify its structure type, i.e., Chain, Branch or Purallel, and extract the corresponding sub-tasks to support attention steering. Although LLMs often struggle with end-to-end execution over complex entangled structures, they exhibit strong capabilities in interpreting and decomposing sub-tasks when guided by well-crafted prompts (Zhang ett al. 2024b) Chae al. (022)). We leverage this by designing prompts that elicit both structural labeling and sub-task enumeration, using the consistent LLM for instruction following:
S. S.T,,T2, - I = LLM((P))
where S [Branch, Chain, Parallel] denotes the structure label, [T,, T..... .Tm represents the identified structurally exclusive sub-tasks within T, and P is the designed prompt template below. Additional details and examples provided in Appendix A.2 Remark. To avoid potential token-level errors in sub-task extraction, we further match each identified sub-task back to its corresponding span in the original input, enforcing faithfulness and preventing сπтоr propagation. Empirical ev- idence shows that LLMs perform well in struc- ture identification across various composition types and complexity (see Appendix [B.].. The acquired structure information serves as both guidance and constraint for subsequent safe at- tention steering, which is confined to sub-task spans to preserve global information integrity. Therefore, minor omissions in sub-task extrac- tion remain non-destructive and only slightly di- minish the effectiveness of attention modulation since leaving partial sub-task regions untouched, indicating the robustness of ATA imperfect structure identification (see Figure ((b)
Prompt template of structure identification
You are an excellent logic expert. Given a compositional task and the definition structure types, please determine the structure label of the task and ideatify the corresponding sub
*Structure Type**:
Chain: The task is required complete multiple sub tasks sequentially.
Branch: The task required select different branches according to cestain conditions.
required complete multiple independent suh-tasks in parallel.
Purallel: The
Output structure type of input task (chosen from "Chain", "Branch", "Parallel") and the exactly sub-tasks (withour modifying. deleting adding any original text) the following joon format:
{ "Structure Type": ".. tSub-tasks": taskl, task2, ...]
Task: (inpur task]
4.2 STRUCTURE-AWARE DYNAMIC ATTENTION
Leveraging the structure information, ATA subsequently mitigates mutual interference between exclusive sub-tasks (Step 1) and suppresses interference of inactive sub-tasks during inference (Steps 2-3), thus preventing task blending and enhancing faithful generation. See Figure for illustration.
Step 1: Mutually Exclusive Masking, Given the identified structure type S and the mutually exclusive sub-tasks [7i, T2...... Tml, we can obtain the exclusive sub-task pairings, i.e., 7; + Tj, Vi * j,i,j [1,m] m]. Considering their structural exclusivity, information interaction between them may introduce blending comprehension during encoding. Specifically, sequential sub-tasks should not consider subsequent ones; cross attention between sub-tasks corresponding to different branches can cause ambiguity; an independent task referencing others may hinder seff-understanding. Therefore, we propose to mask attention between the corresponding token sets of each exclusive sub-task pair, thereby mitigating potential mutual interference. Inspired by the attention mechanism in LLMs that restricts each token from attending subsequent tokens via incorporating negative infinity
5

---

Published as a conference paper at ICLR 2026
Attention A
Attention Steering
17, 2
T
Average along T
(at each generation step
key T2
Head h)
T1
T2
Sterred
Avg scores
T3
(1,h TeT)
A (1hh) T.They
Q
K
1) Mutually Exclusive Masking
ii) Active Sub-task
iii) Dynamic Attention Masking
Identification
Figure 3: The illustration of structure- dynamic attention steering. At each generation step 1, steered attention replaces the original attention with a three-stage process: 1) attention between mutually exclusive sub-tasks is masked to avoid blending comprehension; 2) computing the average attention score for each sub-task and identifying the highest-scoring one as active: 3) attention to sub-tasks exclusive with the active one is masked mitigate interference from irrelevant information.
value (OpenAl 2023 similarly adopt bias matrix M restrict mutual attention accordingly:
H Softmax
A
M
V.
-a, T; T otherwise
M (1)) (Ti, T
0,
Here, inf) is constant attention bias controlling the masking degree, from masking. (T) T denotes the attention bias matrix M indexed by token sets of T, and T₁. After applying Softmax()), attention scores between exclusive sub-task pairs scaled down by exp(a), effectively minimizing interference between mutually exclusive sub-tasks and enabling independent comprehension. As empirically studied in Section 5.3
complete
we set a log 100 across all experiments.
Step 2: Active Sub-task Identification. To suppress interference from inactive sub-tasks during generation, we first identify the active sub-task Teey at generation step f, i.e., the one that dominates the model's focus and guides next token prediction. Specifically, we score each sub-task 7; by averaging the attention its associated tokens have attracted from subsequent tokens:
score(Ti.)) = ITi
Σ KeT:
Σ KEqSS
(. (g.
(5)
where [T]] denotes the number of tokens associated with and k is associated token. A (0 represents the attention matrix ( indexed by the query preceding the next predicted token and the token & in T, The sub-task with the highest score is identified the active sub-task Tkey. Notably, the score excludes influence from other sub-tasks since our mutually exclusive masking strategy. To enhance the robustness of this identification, we draw upon the entropy formula from information theory (Chen et al. 2019). H(X)
Σ/= p(x;) log p(x)), which measures the uncertainty of distribution. Accordingly, we impose an additional constraint on the entropy of normalized scores across all sub-tasks when identifying Tey:
Tkey (rgmaxxy(score(T..)))
s.t.H(ssorre(Ti.)) score((2.11....)) <,
where c y (mm(m) scales with the number of sub-tasks to adapt to instruction complexity, is empirically set to 0.5. Since lower entropy corresponds higher confidence, this constraint serves as confidence measure. When the entropy exceeds the threshold, ATA keeps the previous masking state, avoiding unreliable active sub-task switches and thus ensuring deterministic and stable identification.
Remark. Considering the instruction's global structure, ATA augments the above coore-based active sub-task identification with an active control mechanism to ensure safe progression. For Chain/Parallel tasks, it permits only sequential active sub-task transitions, i.e., T, - T, ... Skip or revisit is rejected and the previous masking state is preserved, preventing premature or illogical switches. For Branch tasks, once branch is selected, it keeps consistent active sub-task throughout generation to ensure faithful execution. This mechanism mitigates errors from occasional attention drift, and the ablation study in Table ("w/o Active Control") demonstrates its effectiveness.

---

Published as a conference paper at ICLR 2026
Step 3: Dynamic Attention Masking. Given the active sub-task Tkey at each generation step i, other structurally exclusive tasks are considered as inactive. To eliminate their interference, ATA masks their attention scores with constant attention bias, building upon the matrix M in Eq 4
H (ih) Softmax
(A +M
(1,0
M (: Tj)
aa 0.
(8)
otherwise
where ( 7 denotes the attention bias matrix M indexed by token sets of T. After Softmax()), the attention of inactive sub-tasks are suppressed by expla), while those corresponding to Tiey are indirectly upweighted. The is set consistently with Eq. I log 100. Consequently, this mechanism effectively steers the model to focus exclusively on the active sub-task while minimizing interference from irrelevant information, ensuring faithful and consistent following of compositional instructions.
The attention steering mechanism operates during a single forward pass and requires no parameter modification. Though incurs minor runtime overhead due to the attention edition, we emphasize that this overhead is minimal (less than 7% as shown in Appendix [B.3). Importantly, the attention modification is confined to the span of mutually exclusive sub-tasks, ensuring the surrounding context remains intact and preventing global information loss (see Section Moreover, attention modules exhibit diverse loken-attending patterns across different heads (Voita et al. 2019) Michel et al. 2019; Zhang al. 2024a). Involving all attention heads in steering could risk degrading global context understanding and lead to model collapse, as demonstrated experimentally in Section 5.3 Thus, we heuristically select small subset of attention heads that particularly responsive to active sub-tasks to conduct attention steering. Details can be found in Appendix (A.3]
Discussion.
5 EXPERIMENT
5.1 SETUPS
Benchmark. We evaluate ATA across three composition structure types: (i) Chain-basee benchmark. comprising 325 instructions with sequential sub-tasks of length 2 to 3; (ii) Branch- benchmark, branch to follow according conditions
consisting of 435 instructions that require selecting varying from single-layer to nested multi-layers. Both chain-based and branch-based benchmarks subsets of Complexbench (Wen et al., 2024); (iii) Parallel-based benchmark, including 450 instructions formed by concatenating multiple independent tasks from gsmök (Cobbe et al. [021]. Baseline. We compare ATA with various baselines commonly adopted for handling compositional instructions, including general prompting, decomposition based, multi-turn dialogue, and planning- based techniques. Direct I/O represents the standard method of interacting with L.LMs, where users input single instruction. CoT Prompting (Wei et al., 2022) encourages a chain-of-thouge reasoning process to improve understanding. Decomposition [Wen et al. 2024) breaks down complex instructions into sequential sub-tasks to achieve the final result. Self-correction (Sun et al.) 2024) first generates initial outputs and then iteratively refines errors based on its own feedback. Think-
execute (Chae et al.) 2024) requires LLMs to first outline plan and subsequently execute it.
Experiment Setups. We implement ATA on two pre-trained L.MMs: I.L.MA3-BB (Touvron et al. 2023) and Mistral-7B (Tiang al. 2023). Hyper-parameter sensitivity analysis indicates that ATA is robust to mask degree a (see Section 5.33) which we therefore fix it at log 100 for all experiments. For attention head selection, we choose 50 heads for branch-based benchmark and 20 heads for chain and parallel-bass benchmarks, with selection criteria detailed in Appendix A.3 Evaluation for chain and branch tasks follows Complexbench standards while precision metric adopted for parallel tasks.
5.2
MAIN RESULTS
The results in Table demoontttttt that ATA effectively steers models to attend to the active sub-task while eliminating interference from irrelevant information, ensuring better output fidelity with the compositional instructions. Specifically, while Llama3-8b achieves only 59.80% compliance in the parallel structure, and baseline methods perform marginally exceeded or even worse, ATA yields remarkable 10.1 1% improvement by steering attention appropriately. Moreover, ATA achieves 6.45% accuracy when applied to Mistral-7B in the parallel structure, surpassing all the baselines even though they explicitly incorporate iterative feedback. The failure of baselines may be attributed to
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance of ATA and baselines compositional instructions. The second best perfor- mance is underlined, while the best performance is bold.
Structure Type
Chain
Branch
Parallel
All
Complexity
2
3
Avg.
Avg.
2
3
2
Avg.
Avg.
Llama-3 3--n--3------ SB-Instruct
Direct 1/O
66.93
56.82
59.21
59.74
55.26
48.32
54.63
61.63
60.17
55.83
59.80
57.88
CoT Prompting
64.05
55.73
57.69
56.62
55.22
53.39
55.02
64.17
67.05
62.08
64.97
59.23
Decomposition
61.19
52.61
54.63
53.88
51.49
45.66
50.04
62.98
66.74
56.26
62.58
55.75
-iinu-Execute
65.94
59.28
60.85
57.49
52.26
50.25
52.92
63.4
68.33
57.09
63.
58.97
Self-correction
66.31
60.33
61.74
61.27
56.38
50.21
55.24
63.37
67.92
51.39
61.25
59.41
ATA
69.26
61.04
62.98
64.38
58.42
53.79
58.74
72.50
71.03
65.42
69.91
63.88
Mistral-7BB BB-nntruct
Direct I/O 57.41
55.68
56.31
55.40
49.37
41.19
48.83
37.87
35.24
31,50
34.89
46.67
CoT Prompting
55.78
53.64
54.19
57.14
51.24
42.98
50.45
40.45
39.07
33,89
37.62
47.42
Decomposition
52.50
51.17
51.62
48.74
37.59
43.77
40.26
38.44
32.15
37.41
44.27
Think-Execute
56.48
55.32
55.89
57.33
50.15
41.83
49.26
39.86
37.95
32.61
36,72
47.28
Self-correction
59.62
55.94
56.94
56.26
47.19
40.64
47.65
38.62
36.43
31,88
35.22
46.60
ATA
61.73
57.28
58.37
60.44
52.93
43.76
52.16
44.68
41.77
37.59
41.34
50.62
Direct
23.5 Mutual
ATA mi s/stuc_jnna (part)
ADA wi struc, info (ori)
20
ATA w/o Dynamic Mask
152
Performance (%)
70-
itruc (revised)
63.8
65.8
69.9
8a8lo
18
17.5
60
61.4
60.1
16
144
14-
12
11.8
50
10.4
Wrong Generation
Mixed Generation
Omitted Generation
Chain
Branch
Parailel
(a) Reduction of Generation Error (J).
(b) Robustness of ATA to Structure Identification.
Figure 4: (a) The generation error reduction with different ATA variants. (b) Robustness of ATA to different quality of structure identification. Both results produced by Llama3-8h model.
their reliance on high-quality feedback and their tendency to distribute attention across all mutually exclusive sub-tasks, potentially introducing interference. The consistent improvement of ATA across all benchmarks and models underscores its validity and superiority. Additionally, Appendix C presents few examples generated by Llama-3-8B using ATA, further demonstrating its efficacy.
5.3 ANALYSIS EXPERIMENTS
Ablation Study. To investigate the effectiveness of each component within ATA, we conduct ablation studies with results shown in Table i) Importance structure information: The variants ATA without structure information select active token spans across the entire text, leading to a notable performance drop. This is likely due to masking essential global context covering mixed and incomplete sub-tasks, highlighting the necessity of structural guidance for safe and effective attention steering, which restricts steering spans and granularity. ii) Importance of attention masking modules:
ATA outperforms variants using only mutually exclusive dynamic masking, and achieves superior error reduction across all generation error types in Figure 4(a) validating their efficacy. Specifi- cally, the mutual mask effectively prevents blending comprehension between mutually exclusive sub-tasks, thus eliminating mixed execution and wrong decisions. The dynamic mask eliminates interference from inactive sub-tasks during generation, ensuring faithful execution of active sub-tasks. iii) Importance of valid active sub-task identification: We have employed an active control strategy
in ATA ensure valid active sub-task identification. This strategy enforces consistent active sub-tasks
8

---

Published as a conference paper at ICLR 2026
Table 2: Ablation Study of ATA.
Table 3: Effectiveness of Attention Steering.
METHOD
CHAIN
BRANCH
PARALLEL
STEERING (Sr.) STRATEGY CHAIN
BRANCH
PARALLEL
DIRECT
59.21
54.63
57.88
DIRECT (No ST.)
59.21
54.63
57.88
SAMPLEATTENTION
60.47
51.92
57.71
ATA
62.98
58.74
69.91
PASTA
61.24
54.16
60.43
w/o STRUCTURE INTO
60.45
55.28
60.35
w/o MUTUAL MASK
61.82
$7.03
67.42
ATA
62.98
58.74
69.91
w/o DYNAMIC MASK
60.74
56.65
64,31
ww MISGUIDED Sr.
56.43
47.86
54.12
w/o ACTIVE CONTROL
61.14
57.23
66.82
-w RANDUM ST.
57.38
51.04
56.81
80
65
60
70
60
60
55
80
50
40
50
Lioma3-88
Mistral-78
45
basafine
40
50
100
1000
1e6
50 100
200
10
20
50
100
200
Exp(a)
heads number
heads number
heads number
(a) Varying Masking Degree
(b) Chain
(c) Branch
(d) Parallel
Figure 5: (a) the sensitivity of ATA to the masking degree hyperparameter o; (b)(c)d); The perfor- mance of ATA when applied to varying numbers of attention heads.
during generation for branching tasks and sequential progression for chaining paralleling tasks. The performance degradation without it indicates its role in reducing occasional identification errors.
Robustness of ATA to Structure Identification. In Figure 4(b) we evaluate ATA's robustness to structure information quality under three settings: 1) human-revised (ground- -rruund-truth) information to serve a performance upper bound, 2) original information from the raw output of our module, and 3) partial (intentionally degraded) information to assess robustness to imperfect identification. ATA consistently achieves performance gains even with imperfect structure information, with only slight reduction in effectiveness. This non-destructive character arises because attention steering, despite its imperfections, enables models to focus on smaller, relevant scope while preserving contextual information in unextracted sub-tasks. Although mild intra- group confusion may occur, it is preferable to the global entanglement seen with the original instructions, demonstratngg our robustness.
Validity of Attention Steering, To highlight the unique strength of our attention steering mechanism in handling compositional instructions, we compare ATA with other attention modulation methods, PASTA (Zhang et al.) 2024a) and SampleAttention (Zhu et al., 2024). as shown in Table Results show that these methods are ill-suited for dynamic active sub-task transitions. SampleAttention suffers performance drops due to disrupted information from its global sparse attention. Though PASTA (adapted to emphasize special sub-task tags (e.g., 'if", 'first', 'problem I', etc)) yields slight improvements, it still underperforms compared to ATA because its static emphasis fails to capture evolving reasoning demands. To provide more direct evidence of validity, we further measure the consistency between ATA's identified active sub-tasks and the ground-truth, achieving 81.2%, 75.1 1%, and 86.4% for Chain, Branch, and Parallel structures, respectively. Besides, steering the model toward misguided (inactive) random sub-tasks results significant performance degradation, strongly confirming that ATA's precise focus alignment with the active sub-tasks drives its improvements.
Generalization Nested Structures & Larger LLMs. Table evaluates ATA on complex composi- -tional instructions with nested structures involv- ing multiple types. The consistent improvements demonstrate the effectiveness of precise attention steering in maintaining focus on active sub-tasks and reducing inference ambiguity. While the gains are relatively modest compared to single-structure instructions, this capability is crucial for achiev- ing better adherence in such complex scenarios. Additionally, evaluate ATA a larger-scale LLM, Llama3-13B. 13B, where notable improvements further validate its generalization and versatility.
Table 4: Generalization on Nested Composition Structures and Larger LLMs.
STRUCTURE TYPE
DIRECT
ATA
LLAMA3-8B
CHAIN & BRANCH
48.42
51.95
CHAIN & PARALLEL
50.17
54.72
BRANCH & PARALLEL
46.83
48.41
LLAMA3-13B CHAIN & BRANCH
52.26
56.64
CHAIN & PARALLEL
56.48
59.30
BRANCH & PARALLEL
51.65
53.88

---

Published as a conference paper at ICLR 2026
Attention Head Profiling. To select appropriate attention heads for steering, employ simple yet effective heuristic strategy. Specifically, we analyze a set of high-quality question-responss pairs to identify heads that consistently focus exclusively active sub-tasks across different instructions, as detailed in Appendix A.3] We compare steering the selected heads against (i) all heads and (ii) random heads, as shown in Tigures(((b)e ((d) Steering all heads degrades performance due to excessive masking, while steering random heads leads to high performance variance as attention heads serve diverse functions (Clark al. 2019, Voita et al. 2019), demonstrating the necessity of this strategy. Additionally, sensitivity analysis on varying numbers of selected heads reveals a trade-off between focusing on active sub-tasks and maintaining overall structural information. Empirically, we apply ATA to a moderate number of heads from 20 to 50, balancing faithfulness and generation quality.
Sensitivity To Masking Degree. We analyze ATA's sensitivity with varying masking degree a from log5 to log 10000. Figure shows that ATA exhibits robust across this range, minimal residual attention to inactive sub-tasks effectively suppresses interference. This insensitivity to specific value allows us to fix at log 100 in all experiments, balancing simplicity and effectiveness.
Preservation General Response Capability. To assess whether ATA compromises the model's fundamental language ability, we conduct a GPT- 4o-based evaluation following [Solfoo et al. (2025). For each instruction, we extract a base query and generate corresponding scoring questions to mea- sure general response quality. Table 5 shows the average differences in quality score relative to a simple Baseline (Base Query), isolating the impact of adding complex instructions and ATA's atten- tion masking. While complex instructions can degrade the general response quality due to constraint compliance, ATA shows a robust performance: it introduces negligible additional degradation on Chain, mitigates the quality drop on Branch by masking irrelevant branches, and slightly improves Parallel by reducing interference between independent sub-tasks, enabling more focused and coherent generation. We attribute this to precise masking confined to sub-task spans and targeted steering over small set of structure-relevant heads, preserving most heads responsible for language ability.
Table 5: Average Differences in Response Qual- ity Scores.
CHAIN BRANCH PARALTEI.
DIRECT I/O (BASE QUERY) DIRECT 1/O (w/ INSTRUCTION)
0
0
-0.17
-0.41
+0.06
ATA (w/ INSTRUCTION)
-0.21
-0.28
+0.09
CONCLUSION & LIMITATION
This work aims tackle the challenge faced by LLMs in following compositional instructions. We systematically analyze the essential characteristics inherent in chaining, branching, and paralleling structure types, revealing that only one sub-task should be active at each generation step, and residual attention to inactive sub-tasks can introduce interference. To mitigate this, we propose ATA frame- work, which dynamically steers the model's attention focus on active sub-task during generation while masking the inactive ones, thereby eliminating potential interference. Comprehensive experi- ments across diverse structure types validate that ATA significantly enhances reasoning fidelity to compositional instructions, highlighting the potential of attention- steering as lightweight yet effective mechanism. Moreover, ATA is a alug-and-play mechanism that can be integrated with various on-the-shelf L.Mss and orthogonal methods to further improve their performance.
Limitation. ATA is tailored for complex compositional instructions with explicit sub-task structure. In challenging scenarios where structure is implicit, it needs to be paired with other structure-discoeer techniques, with ATA serving as a general-purpose attention-control module to improve attention adherence and reasoning fidelity. We will explore structure-agnoste attention steering in future work to further broaden ATA's applicability and hope it can bring some inspiration to the community.
ACKNOWLEDGMENTS
This work is supported by the National Science and Technology Major Project (No. 2022ZD01 14903) and the Natural Science Foundation of China (NSFC. No. 62476149).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Hyungjoo Chac. Yeonghyeon Kim, Seungone Kim, Kai Tzu-iunn Ong, Beong-woo Kwak, Moohyeon Kim, Sunghwan Kim, Taeyoon Kwon, Jiwan Chung, Youngjae Yu, and Jinyoung Yeo. Language models as compilers: Simulating pseudocode execution improves algorithmic reasoning in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 22471-22503 2024.
Pin-Y Yu Chen, Lingfei Wu, Sijia Liu, and Indika Rajapakse. Fast incremental von neumann graph entropy computation: Theory, algorithm, and applications. In Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97, pp. 1091-1101, 2019.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of bert's 's attention. In Tal Linzen, Grzegorz Chrupala, Yonatan Belinkov, and Dieuwke Hupkes (eds.), Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP,, pp. 276-286 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL httpt:t//aatttti
02s/2110.111155
Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 5050-5000000 5063, 2024.
Shirley Anugrah Hayati, Taehee Jung, Tristan Bodding-LLng, Sudipta Kar, Abhinav Sethy, Joo- Kyung Kim, and Dongyeop Kang. Chain-of-instructiinn Compositional instruction tuning on large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 24005-2400 24013, 2025.
Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. In Findings of the Association for Computational Linguistics: EMNLP, pp. 10864-10882, 2024a.
Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. Can large language models understand weal-world complex instructions? In Tiihty-Eighth AAAI Conference Artificial Intelligence, pp. 18188-1888 2024b.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06822 2023.
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Followbench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume I: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. pp. 4667-4688 2024.
M. J. Jiménez- Navarro, María Martínez Ballesteros. Francisco Martinez-Alvarez. Alicia Troncoso, and Gualberto Asencio-Core Cortés. From simple to complex: sequential method for enhancing time series forecasting with deep learning. Log. J. IGPL, 32(6))986 1003, 2024.
Tian Jin, Wanzin Yazar, Zifei Xu, Sayeh Sharify, and Xin Wang. Self-selected attention span for accelerating large language model inference. CoRR, abs/2404.09336, 2024.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, pp. 14014-14024, 2019.

---

Published as a conference paper at ICLR 2026
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024 8035, 2019.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8652, 2023.
Alessandro Stolfo, Vidhisha Balachandran, Safoora Yousefi, Eric Horvitz, and Besmira Nushi. Improving instruction-following language models through activation steering. The Thirteenth International Conference Learning Representations, 2025.
Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Baohua Dong, Ran Lin, and Ruohui Huang. Conifer: Improving complex constrained Instruction-following ability of large language models. CoRR, 0bss/244.02823,, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 5797-5808, 2019.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61 Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pp. 2609-26333 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou, Chain-of-thought prompting elicits reasoning in large language models. In Proc. of NeurlPS, 2022.
Bosi Wen, Pei Ke, Xiaotao Gu, Lindong Wu, Hao Huang. Jinfeng Zhou, Wenchuang Li, Binxin Hu, Wendy Gao, Jiaxing Xu, et al. Benchmarking complex eestruction-folloviii with multiple constraints composition. Advances in Neurul Information Processing Systems, 37:137610-137645, 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's 's transformers: State-of-the-un art natural language processing. CoRR, abs/1910.03771, 2019.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024.
Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. Tell your model where to attend: Post-hoc attention steering for 1lms. In The Tweifth International Conference Learning Representations, 2024a.
12

---

Published as a conference paper at ICLR 2026
Qingru Zhang, Xiaodong Yu, Chandan Singh, Xiaodong Liu, Liyuan Liu, Jianfeng Gao, Tuo Zhao, Dan Roth, and Hao Cheng. Model tells itself where to attend: Faithfulness meets automatic attention steering. CoRR, abs/2409.10790) 10790, 2024b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303. 18223, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: Anmual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
Qianchao Zhu, Jiangfei Duan, Chang Chen, Siran Liu, Xiuhong Li, Guanyu Feng, Xin Lv, Huanqi Cao. Xiao Chuanfu, Xingcheng Zhang, Dahua Lin, and Chao Yang. Sampleattention: Near- lossless acceleration of long context LLM inference with adaptive structured sparse attention. CoRR, abs/2406. 15486, 2024.
13

---

Published as a conference paper at ICLR 2026
A
IMPLEMENTATION DETAILS
A.1
ALGORITHM OF ATA
For clarity, the algorithm of proposed ATA is provided Alg.
Algorithm The algorithm of ATA
Input A compositional task T, an LLM LLM, a prompt template P. the selected head set H.
mT Generate structure type and mutually exclusive sub-tasks T2. .T.] within by Eq. 2
2: for each generation step I do
3: Mask the mutual attention between exclusive sub-task pairs by Eq.
4: Calculate contribution score score(T;, f) for each sub-task T; by Eq.
5: Identify the active sub-task Tey by Eq. 66
6: Mask the attention to inactive sub-tasks that structurally exclusive with They by Eq.
7: end for
Output: The final generation
We implement all algorithms using Py PyToch (Paszke et al. 2019) and Huggingface (Wolf et al. 2019).
A.2 DETAILED PROMFT TEMPLATE FOR STRUCTURE ANALYSIS
Table 7 provides the prompt template we used for structure analysis and identification in Section 4 To improve identification quality, use 3 manually constructed in-context examples in the prompt.
A.3 ATTENTION HEAD SELECTION STRATEGY
Since various attention heads function diversely, steering different heads yields dramatically different performance. To identify the effective heads, PASTA (Zhang et al. (2024a) proposes greedy selection strategy that evaluates the steering performance of each head small validation sets of multiple tasks and selects the heads that yield the best performance. This greedy strategy requires evaluating L H times, where L represents the number of attention layers and H represents the number of heads within each layer, resulting in non- non-trivia trivial overheads especially for large models. AutoPASTA (Zhang al. 2024b) enhances this strategy by adopting coarse-to-fine search approach, transitioning from layer-lever level to head-level selection. Despite its refinement in efficiency, the method still incurs high computational costs with / x H L forward passes is the number coarsely selected layers). To address this, we propose a simple yet effective heuristic selection strategy within a single forward pass as an alternative. Specifically, we begin by generating ideal outputs for small validation sets of multiple tasks using GPT-4. These outputs are then concatenated with the corresponding input context and processed by an inference model, such as Llama3-8B, to obtain attention scores for each head. Our objective is to select heads that functioned to be focused exclusively sub-task. Thus, for each data, we measure the attention each head allocates to the mutually exclusive sub-tasks. Heads that exhibit high attention to the active sub-task while showing minimal attention to the inactive ones are shortlisted as candidate heads. Finally, we select the tok-k heads from these Empirically,
the active
candidate, based on their frequency across the entire validation subset of heads (e.g., 50 out of 1024 heads) is sufficient for ATA to achieve superior performance, demonstrating its efficiency. Table 6 presents the number of heads be steered by ATA for Llama-8B and Mistral-7B on each benchmark.
find that small
Table 6: The number of heads to be steered by ATA.
Task
Llama3-8B
Mistral-7B
Chain
20
20
Branch
50
50
Parallel
20
20
14

---

Published as a conference paper at ICLR 2026
Table 7: Prompt template for Structure Analysis and Identification.
Prompt template for Structure Analysis and Identification.
You are an excellent logic expert. Given compositional instruction and the definition of structure types, please determine the structure label of
identity corresponding sub-tasks.
Structure Type:
Chain: The instruction is required to complete multiple sub-tasks sequentially.
Branch: The instruction required seloct different branches according to certain conditions.
Purallel: The instraction required to complete multiple independent sub- parailel.
Output Format:
Ouiput the structure type of input instruction (chosen from "Chain", "Branch", "Purallel") and the exactly sub tasks (without modifying, deleting adding any original text) following json format:
"Structure Type":
"Sub taskst taskl, sub Lask2,
#* Examples:
[Instruction You are playing the role Su Shi, Chinese literati from the Northern Song Dynasty. When my words are your pocm. please reply with the next line. Then write paragraph of your thoughts on this line of poctry, with coberent emstions.
When my words
your poem, pleuse reply with a poemn yours the same theme.
Note:
Always remember you are playing Shi.
2. Your response style should reflect Su Shi's criticism of reality, lamentation of life as a dream, and free-spirited lyrical expression.
3. Your responses should be positive and uplifting, with sense heroic spirit.
Me: When will the moon be hright?
Su Shi:
[Output 1]
"Structure Typc": "Branch".
"Sub-taskss: " l'please reply with the next line. Then write paragraph of your thoughts this line of poetry. with coberent emnttions., "please reply with poem of yours on the same theme."]
Select one the following works represents the "Fauvism" movement and generate a Xianhongshu article of about 300 words. Then, conduct analysis on the other work, with clear and distinct points, and easure to include the keywords "Cubism" or "Geometricism"" Works: "The Style of Cumbrai" and "TTe Dunce"""" [Output 2]
[Instruction 2]
"Structure Type": "Chain". "Chain".
"Sub-tasks": " ["Select one following works represents "Fauvism" art movement generate a Xiachongshu-atyl article of about 300 words.". "Then, conduct an art analysis on the other work, with clear and distinct points, and ensure to include the keywords "Cubism" or "Geometricism"" "!"
[Instruction Solve the following math problems and output the format:
"Output 1":
"Ourput 2":
"Output 3":
1. James decides to rum sprints 3 times a week. He runs 60 meters each sprint. How many total melers does run a week! Every day, Wendi feeil each of her chickens three cups of mixed chicken feed, containing seeds, mealworns and vegetables help keep them healthy, She gives the chickens their feed in three separate meals, In the morning, she gives her flock of chickens 15 cups of foed. In the afteroon..
gives chickens another chickens in the linal meal of the day il the size of Wendi's 's llack is 20 chickens?
cups of feed. How many cups food does need give her
every second glass costs only the price. Kylar wants buy 16 glasses. How mach does need to pay for them!" [Outpul 3]
Kylar went
buy glasses
new apartmentt One glass costs
"Structure Type": "Parallel".
"Sub-tasks": ["James decides run 3 sprints times week.
60 meters each sprint. How many meters does run a week." "Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy, She gives chickens
In the moming, she gives flock chickens 15 cups of feed. In the altermoon. she gives her chickens another 25 cups of leed, How many cups of feed does she need give her chickens the final meal the day the size of Wendi's x flack is chickens?", "Kylar went to the store to buy glasses for his apartment. One plass $5, but every second glass only 60% price. Kylar does be need to pay for them?"]
feed three separate
buy glasses, How much
Input Instruction: linput instruction)
15

---

Published as a conference paper at ICLR 2026
B ADDITIONAL ANALYSIS
B.I PERFORMANCE OF STRUCTURE IDENTIFICATION.
To evaluate the reliability of LLMs in identifying structure types and extracting sub-tasks, present the structure identification performance of LLaMA3-8B across compositional instructions of varying complexity in Table 8 Specifically, Structure-type Precision measures the accuracy of LLMs in predicting structure types; Sub-task Recall indicates the proportion of ground-truth sub-tasks that can be independently identified; and Sub-task Precision assesses whether the extracted sub-tasks are semantically complete without unrelated global context. The results demonstrate that LLMs exhibit strong capabilities structure-type classification and sub-task extraction, maintaining high reliability even for complex compositional instructions. The primary challenge, however, lies in sub-task recall, as multiple sub-tasks may sometimes grouped. Despite this, the sub-task splits remain contextually precise, that is, the extracted sub-tasks semantically complete without extraneous content, thereby preventing blend comprehension and ensuring the safety of subsequent attention steering.
Table 8: Performance of Structure Identification.
COMPLEXITY
2
3
STRUCTURE-TYPE PRECISION
100%
100%
100%
SUB-TASK RECALL
98.3%
87.6%
81.4%
SUB-TASK PRECISION
99.7%
96.2%
92.3%
B.2 STRATEGIES FOR ACTIVE SUB-TASK IDENTIFICATION.
Identifying the active sub-task at each generation step is pivotal for effective dynamic attention steering. To ensure the confidence of identification, we employ entropy the evaluation metric in Eq. 6 For comparative analysis, we compare it with Max probability (the highest probability score within sub-instructions) and the Margin confidence (the difference between the highest and second- highest score) as confidence metrics. As demonstrated in Table 9 entropy outperforms the other metrics, likely due to its consideration the entire distribution, thus offering more comprehensive assessment of uncertainty.
Table 9: Strategies for Identifying Active Sub-task.
STRATEGY
CHAIN
BRANCH PARALLEI
C-MAX PROBABILITY
61.74
57.91
68.62
6- MARGIN CONFIDENCE
61.45
58.16
69.27
6- ENTROPY (ATA)
62.98
58.74
69.91
B.3 TIME COST.
Considering that ATA modifies attention during model inference, we provide computational cost analysis in Table I0 presenting the generation time for parallel instructions of varying complexity on Llama3-8b. Our results demonstrate that ATA incurs a minimal time overhead compared to generation, with an increase of less than 7%. Notably, in contrast to approaches requiring finetuning or multiple LLM inferences, ATA performs generation single forward pass, highlighting its efficiency.
C EXAMPLES ATA GENERATIONS
Table
12 13 provides generation examples across chain, branch and parallel instructions. Specifically, ly, the Chain instruction, direct generation fails execute the third sub-task (Omitted
16

---

Published as a conference paper at ICLR 2026
Table 10: Efficiency of the proposed ATA framwork.
COMPLEXITY
2
3
> 4
DIRECT
9.69 (s)
14.91 (s)
18.66 (8)
ATA
10.14 (s)
15.83 (S)
19.95 (s)
TIME OVERHEAD GAIN (RELATIVE)
4.6%
6.2%
6.9%
Table 11: Generation examples of I.LAMA-7B on Chain Instruction.
Compositional
Blog Title
Healthy Apple Muffins
Instruction
Blog Theme: Recipes and Tips for Making Delicious Apple Mullins
The above are the title and theme of the blog posi we have identiied. Please write an introductory paragraph of no less than S00 words to capture the reader's is attention.
Pleuse start with "Famlly Recipe Collection" muffins in the middle part, and end with a rhetorical question that ties the beginning and the end together.
the introduction, then incorporate the health benefits of eating apple
Direct Generation
Family Recipe Collection: Making and Tips for Healthy Apple Muffins /
In our daily lives, apples are common and popular fruit, Not only do they tasie great, but they are also rich in various nutrients such vitamins, minerals, and dietary fiber, which greatly beneficial to human health. Today, will introduce you to healthy delicacy made with apples as the main ingredient-apple muffius. They are both delicious nutritious, making them an indispensable dish family recipes.
The main ingredient in upple mullins is apples, and through special processing and prepuration. the texture of the apples hecomes more delicate the nutrition more abundant. Additionally, apple muffins include other ingredients such as oats and honey. These additions not only enhance the taste of the muffins but also make their nutritional value more comprehensive.
So, what are the health benefits of cating apple muffins? First, apple muffins are rich in dictary fiber, which helps promote intestinal peristalsis and prevent constipation. Second, the antioxidants found in apples can effectively combat the damage caused by free radiculs acid and vitamin C in upple muffins help improve skin condition and maintain skin radiance. In summary, apple muffins are delicious and healthy food. Whether as family recipe or a snack, they are an excellent choice. So bow do you make apple muttins? Please stay tuned for our upcoming content, where we will provide detailed stops and tips for making apple muffins. Let's first take look at the main ingredients and condiments for apple muffins: Ingredients: Apples, oats, honey, butter
human budy, slowing down the aging process. Third. malic
Condiments: Eggs, salt, cinnamon powder, etc. / Next, please follow subsequent content,
we will bring you method and tips making apple mullins. Let's 's get started on making this delicious and healthy apple muffin together! x
([Omitted Generation]: fail to excaute the third sub-tack that require to end with a rhetorical question.)
ATA Generation
Family Recipe Collection always has few dishes that can evoke warm memories deep in our hearts, and apple muffins are undoubtedly one of them. Whether the breakfast table in the morning or during aftemoon tea time, apple mullins can always bring us enillss satisfaction with their unique aroma and taste. Imugine a freshly baked muffin emitting an enticing apple fragrance, gentle revealing a soft texture perfectly blended with the sweetness of apples, instantly awakening your taste buds. This simple yet classic delicacy is not just a treat for the tasto buds but also part of a healthy lifestyle.
Apple mullins not only delicious but also packed with nutritional value. Apples are rich in dietary liber and vilamin C, which help promote digestion and boost the immune systmm Adding apples to mulfins not only increases the natural sweetness but also provides additional nutritional support for the body. Studies have shown that moderate consumption of apples helps lower cholesterol levels and improve cardiovascular bealth. Morcover, the antioxidants in apples help fight free radicals, slowing down aging process. Incorporating these health benefits murfins allows you to enjoy deliciousaess while also positively impacting your body.
Making apple mutlins does not require complex skills: just few simple steps can easily be completed at home. Whether you a baking novice experienced culinary expert, this recipe will allow you to easily get started. By selecting quality ingredients and mastering some small rips, you can present sweet and delicions apple muffins to your family and friends. Imagine sunny momning family members sitting around the dining table, enjoying the apple muflins you made by hand-that sense of happiness is invplaccable. /
So, why not try making a healthy and delicious apple muffin for yourself and your family this weekend? this fast-paced cra, occasionally pausing to enjoy simple and bealthy meal may be exactly what we need. Are you ready to embark on this journey of deliciousness and health? /
Generation), which mandates that the output end with a rhetorical question. In the Branch instruction, direct generation additionally executes an unselected branch along with the selected one (Wrong Generation). In the Parallel instruction, direct generation makes mistakes in one of the questions (Mixed Generation), likely due to interference from the other question. In contrast, proposed ATA consistently achieves precise alignment with the requirement across all three instructions, showing its clear superiority.
17

---

Published as a conference paper at ICLR 2026
Table 12: Generation examples LLAMA-7B on Branch Instruction.
Compositional Instruction
Hu Juren was Chinese writer and educator, renowned for his literary works and contribations to education. His works span multiple fields, showcasing profound insight into society human nature.
If the shove content mentions Hu Juren and references his work "Self Encouragement Couplets." please provide full text "Self Faccuragemeett Couplets" and analyze each sentence point by point. Otherwise, provide a biography of Hu Juren in no more than 300 words.
If the above content does not mention Hu Juren but references qualities such as perseverance and persistence.
pleuse expand on this content about 500 words using only commas and periods, and conclude with famous quoto. Otherwise, state that the article does not contain relevant content.
Direct Generation
The content above does not mention Hu Juren. If did mention qualities like perseverance and persistence, please expand on this content to about 500 words, using only commas and periods, and conclude with a famous saying. Otherwise, please state that the article does not contain relevant content.
Hu Juren, a Chinese writer and educator, dedicaled his life to the relentless parsuit of knowledge and the exploration of truth. He fimily believed that only through perseverance and unwavering determination could achieve success in life.
said, "Life would be perfect if it were as was at first sight, why lament the autumn wind's sorrowful fan! People's 's hearts change casily, yet they say it is the heart that This quote reminds us that life is flecting. and we must cherish every moment, not letting time slip away in vain. At the same time, we must be brave enough ta face changes, constanily adjustng Hu Juren's 's persistence and dedication ultimately led him to achieve remarkable success the fields of literature and education. He always believed that only putting in immense effort could attain true success, This spirit is somothing we should all strive to emulate.
mindset to udapt to ever-chunging environment.
Therefore, should like Juren, embodyyng the qualities of perseverance and persistence, to achieve success on the path of life. As the ancients said, "If you carve and give up, the rotton wood will not break: if you carve and do not give up. the metal and stone can be engraved." As long as we persist, we will surely achieve success.
In the journey of life, we will all encounter setbacks and difliculiies, but long we persevere and persist, will surely overcome the difficulties and achieve success. As Hu Juren said, "I life would he perfect if were as it wind's sorrowful fan? People's hearts change casily, yet they say the
was at first sight, why lament the beart that is tickle." We must learn to persist, not be defeated by difficuttess, face life courageously, and puzsue our dreamrs.
Finally. lot us end with famous saying. "Life is like a play, and everything can be endured." May persevere and forge abead on the path of life, achieving our own success. X
([Wrong Generation]: Erroneously execute the unselected inactive branch.)
ATA Generation
The aforementioned content does not specitically mention Hu Juren the content of his "Sell" Encouragement Couplet." Therefore, I will provide a brief introduction to Hu Juren's life. Juren (434-1484), courtesy name Shuxin, educator of the Ming Dynasty. He was born in Jishui County, Jiangai Province, and showed remarkable intelligence and love learning from young age. He later studied under famous Neo- Confuciin scholar Yuhi, whose senchings profoundly influcnced him. Throughout his life. Hu Juren dedicated himself to education, founding academies in his hometown and nurturing a large number of outstanding talents. His educational philosophy emphasized moral cultivation and churacter developtnent, advocating for "learning for practical application." " which had lasting impact on future generations. Hu Juren's literary works also hold significant influence, aften reflecting deep insights into suciety and hummn nature. His writings, including the "Collected Works Jingelui," showcase his exceptional contributions to hoth literature and education. Vf
sobriquet Jingzhai, was renowned Neo- Confucian scholar and
D THE USE OF LLMS
In this section, we clarify that no LLMs were employed in the writing or polishing of this paper. All content presented herein is the result of original research and critical evaluation by the authors.
18

---

Published as a conference paper at ICLR 2026
Table 13: Generation examples of LLAMA-7B on Parallel Instruction.
Compositionl Instruction
I. Terry sells milkshakes for $5.50 cach, nine burger platters for SII each, and 20 sodas for $1.50 each. How much money does he make in total?
2. Paul is at train station and is wailing for his train. He isa't sure how long be needs to wait, but be knows that fourth train scheduled to arrive the station the one he needs to gel on. The first train is scheduled to arrive in 10 minutes, and this train will stay the station for 20 minutes. The second train arrive half an hour after the first train leaves the station, and this second train will stay in the station for quarter of the umount of time that third
the station, this third train is to leave the station immediately after it arrives. The fourth train will arrive 20 minutes after the third train leaves, and this the train Paul will board. In total, how long, in minutes, will Paul wait for his train.
stayed
anrive
after second
Direct Generation
Here are the answers:
[Answer 1|: Terty makes a total 6 milkshukes *S5.50 = $33.00.
burger platters $11.00 $99.00, 20 sodas $$1550 $30.00,
Tatal: $33.00 $99.00 $30.00 = $162.00 /
[Answer Paul will wait for his train for: 10 minutes (first train arrives), 20 minutes (first train stays), 10 minutes (sccond train arrives), 10 minutes (sccond train stays). minutes (second leaves, fourth train arrives).
third arrives), minutes (third stays), minutes
20
minutes
[Mixed Generation]: Impacted by potential interference from the information in question 1.
ATA Generation
Here the answers: (Anser 1]: Terry makes a total of:
milkshukes $5.50 $33.00,
9 burger platters x $11.00 $99.00,
$30.00,
Total: $33.00 $99.00 $30.00 $162.00 /
[Answer 2]: Paul will wait for his train for:
First train: 10 minutes, First train stays in the station for 20 minutes, so Paul has wait another 20 minutes after first train leaves.
Total wait time for the first train: 20 30 minutes,
Second train arrives 30 minutes after the first train, and stays in the station for 20/4 5 minutos, Paul bas to wait 5 minutes after second
Total wait time for the second train: 30 + 35 minutes,
Third train arrives 60 minutes after the second train, and leaves immediately, Paul has to wait O minutes after the third train leaves.
Tatal wait time the third train: 60 minutes,
Fourth train arives 20 minutes after the third train leaves, which the train Paul will board. Total wait time for the fourth train: 20 minutes,
time: 35
145 minutes
19


Reviews:
[{"review_id": "lXCpMLFUNu", "reviewer": "Reviewer_qkLE", "summary": "The paper proposes Attend to the Active (ATA), an inference-time method that steers a model’s attention toward the currently “active” sub-task within a complex instruction. ATA masks attention between mutually exclusive sub-tasks and then boosts attention to the detected active sub-task. The authors evaluate on chain, branch, and a synthetic parallel setup and report consistent gains over prompting and planning baselines, with ablations on masking strength and head selection. They also include robustness checks for structure identification quality and some results on nested compositions.", "questions": "None.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Good idea and implementation. The focus on compositional instruction following is timely. Doing it with attention masking at inference is interesting because it avoids retraining and aims to localize interference between sub-tasks.  \n\n2. Clear writing and organization. The paper explains the three structure types, gives concrete examples, and walks through the masking and steering pipeline in a way that is easy to follow.   \n\n3. Strong experiments and analysis. The experiments cover two base models, three composition types, and include ablations for masking degree and number of steered heads, as well as component removals. The figures show sensitivity to α and head counts, and tables report the impact of removing structure info, mutual masks, dynamic masks, and active control.", "weaknesses": "1. Parallel instruction data is simplistic - The “parallel” benchmark is built by concatenating independent GSM8K problems. This construction makes sub-tasks independent by design, which favors attention masking because there is little need for cross-task sharing or entity linking. A harder parallel set, closer to the realistic prompts illustrated in figure 2, would be better test.  \n\n2. General capability preservation is under-tested - Because ATA modifies attention patterns, it is important to show that language abilities are not degraded. Following evaluations in [1] it would be good to measure side effects on general text quality.  \n\n\nReferences: \n[1] Stolfo et al., “Improving Instruction-Following in Language Models through Activation Steering,” ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ir6nII0jLA", "reviewer": "Reviewer_TRL7", "summary": "This paper improves the instruction following behavior of LLMs by identifying three different compositional structures of instructions which LLMs struggle to follow. They propose an attention modification method which ensures that the LLM only attends to the relevant sub-instruction at any time during generation, and they show that this improves instruction following.", "questions": "* Figure 4a shows that ATA reduces the identified generation errors, but it doesn’t go to 0. Why do you think some of these errors remain? Are the wrong subtasks selected by the model’s attention?\n* How often is active subtask selection incorrect?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The paper is very well written and easy to follow.\n* The approach is novel and addressed an important problem with a relatively lightweight method.\n* A large number of baselines and models are used in the evaluation which shows that the proposed approach is most effective.\n* The attention head selection strategy proposed in Appendix A.3 is shown to be effective while using substantially less compute than baselines which makes these methods easier to use.\n* Results on nested composition also show that the method remains effective.", "weaknesses": "* Missing standard deviations for results.\n* The datasets used for evaluation are designed to have the three types of composition, but a traditional instruction following dataset with more “real” tasks is not used for evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "23Gn43BI8H", "reviewer": "Reviewer_grgz", "summary": "This paper proposes an attention steering technique for improving instruction following by LLMs for compositional instructions. Three compositions are considered: chaining (sequential), branching (conditional), and parallel execution of sub-tasks. The key claim is that at any generation step, only one sub-task should be active irrespective of the type of composition. The steering technique, Attend To the Active (ATA), first requires the composite instruction to be distilled to the composition type and list of sub-tasks. It then first constructs a mutual attention masking  matrix that downweighs attention between independent tasks, and a dynamic attention masking matrix that downweighs attention to tasks independent of the current \"active task\" (the active task is identified as the one receiving the most attention at the current step of generation). These masks are applied at every step of generation to a selected subset of attention heads. The experiments show that ATA outperforms other instruction following techniques and attention steering methods on instructions from the three composition types. An analysis of components of ATA demonstrates that it is robust to how the structure information is presented and the masking degree, but the structure information along with the masking modules and the control strategy (identification of the active task) are essential -- the selection of attention heads is crucial to the success of ATA.", "questions": "I will summarize my questions from the weaknesses section above (please refer to that section for more details):\n1. Which tokens are included as \"next tokens\" in the score computation for the active task?\n2. How is the entropy threshold computed and how sensitive is ATA to this choice?\n3. How is the condition for branching tasks factored in during structure identification?\n4. Does ATA promote global structure following by restricting attention to the current active task -- do sequential / parallel sub-tasks all eventually happen?\n\nAdditionally:\n\n5. How does the structure identification work for the nesting experiments (Table 4)? What does the prompt look like here? \n6. In the ablation on structure identification (Figure 4(b)), what do the variants partial, original and human-revised mean?\n7. For active sub-task identification, what happens when the entropy is over the threshold?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is well written and the key ideas are fairly easy to understand.\n\n- The problem of better instruction-following with compositional instructions for LLMs is important and the proposed approach, ATA, of masking attention on sub-tasks based on structural exclusiveness and the current active task seems novel.\n\n- The experiments include a good variety of baselines (other instruction following prompting strategies and attention steering methods) and provide insights into how various components of ATA are useful.\n\nOverall, I think that the paper studies an important problem and the experiments demonstrate effectiveness over other techniques.", "weaknesses": "- ATA essentially ensures that only one sub-task is attended to at any point but does not focus on whether all sub-tasks are eventually attended to (for chaining / parallel compositions). Improved instruction following would require that the order of the tasks is also taken into account (for chaining), and that all sub-tasks are \"accomplished\" (for parallel). At the least, I would expect this to be explicity mentioned in the text and some analysis from the experiments on whether this happens with ATA.\n- For branch compositions, I would also expect the condition to be a sub-task that is attended to first. But the prompt template in Section 4.1 does not seem to include the condition during structure identification. Moreover, as I mention in the previous point, ATA does not take into account that only one of the sub-tasks should be accomplished (at least in the way it is defined and discussed in Section 4.2).\n- I find it difficult to understand which tokens are being included in the identification of the current active sub-task. For Equation 5, it seems like it computes the average attention between the tokens of a task instruction T_i (k) and all tokens after the instruction (q >= k)? Would this not include other tokens of T_i, othe following sub-tasks, and the entire generation sequence so far? The preceeding description of the score seems to convey that the score is computed with respect to the next token at a given step (lines 283-286).\n- The identification of the active task also mentions the entropy threshold. How this threshold is computed and how the value affects the performance is not discussed in the main text / experiments. Moreover, I do not understand what happens is the entropy is over the threshold.\n\nOverall, I think this work would benefit from providing more details on the technique (the active sub-task score, the entropy threshold, the inclusion of the condition for branching tasks) and the contribution can be improved by taking the overall structure into account (sequential tasks should happen in order, parallel tasks should all eventually happen, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "axQezdwYvz", "reviewer": "Reviewer_Hrq3", "summary": "This research studies the complex instruction-following problem. The authors consider that there are three structures of sub-tasks from the user instructions, and propose to manipulate the attention-patterns for different task structures such that the model can attend to the correct contents for each sub-task. The results of the control experiments support the effectiveness of the proposed method.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The overall idea of manipulating the attention mask to enforce the model to attend to certain sub-tasks for complex instruction following is reasonable.\n2. The comprehensive ablation study provides evidence for the effectiveness of each proposed component. \n3. For most of the manuscript, the writing is clear.", "weaknesses": "1. The authors decompose sub-tasks by leveraging an LLM, which can be a significant bottleneck to the framework when it tries to handle more challenging tasks. Although the performance of structure identification (Appendix B.1) looks good on current datasets, I am not sure whether this reliability can extend to more challenging tasks/datasets, such as AIME 24/25 for mathematical solving. The reason I raise this concern is that, for many hard questions, the sub-tasks are not explicitly presented in the user instructions, so the model needs to plan them by itself at first. \n\n2. Figure 3 has not been rendered properly. \n\n3. How is the model's performance on general (simple) instructions, such as email drafting and poem writing? In other words, whether the system is robust to the instructions that do not hold the hypothesis that they have multiple sub-tasks? \n\n4. How much is the overall overhead for model inference, including the time of annotating sub-task structures?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)