LLMs Process Lists With General Filter Heads
Authors: Arnab Sen Sharma
Giordano Rogers
Natalie Shapira
David Bau
Venue: ICLR 2026 Poster (2026)
Keywords: interpretability
language models
map-filter-reduce
functional programming
symbolic systems
Primary area: interpretability and explainable AI
TL;DR: LLMs implement list-processing through specialized attention heads that encode reusable predicate representations, revealing dual lazy/eager evaluation strategies that mirror functional programming primitives.

Abstract: We investigate the mechanisms underlying a range of list-processing tasks in LLMs, and we find that they have learned to encode a compact, causal representation of a general filtering operation that mirrors the generic ``filter'' function of functional programming. Using causal mediation analysis on a diverse set of list-processing tasks, we find that a small number of attention heads, which we dub *filter heads*, encode a compact representation of the filtering predicate in their query states at certain tokens. We demonstrate that this predicate representation is general and portable: it can be extracted and reapplied to execute the same filtering operation on different collections, presented in different formats, languages, or even in tasks. However, we also identify situations where LMs can exploit a different strategy for filtering: eagerly evaluating if an item satisfies the predicate and storing this intermediate result as a flag directly in the item representations. Our results reveal that transformer LMs can develop human-interpretable implementations of abstract computational operations that generalize in ways that are surprisingly similar to strategies used in traditional functional programming patterns.

Published as a conference paper at ICLR 2026
LLMs PROCESS LISTS WITH GENERAL FILTER HEADS
Arnab Sen Sharma", Giordano Rogers, Natalie Shapira, and David Bau Khoury College of Computer Sciences, Northeastern University
ABSTRACT
We investigate the mechanisms underlying a range of list-proeessing tasks LLMs, and find that LLMs have learned to encode compact, causal representation of a general filtering operation that mirrors the generic "filter" function of functional programming. Using causal mediation analysis tasks, we find that a small number of attention heads, which we dub filter heads, encode a compact representation of the filtering predicate in their query states certain tokens. We demonstrate that this predicate representation is general and portable: can be extracted and reapplied execute the same filtering operation on different collections, presented in different formats, languages, or even in tasks. However, we also identify situations where transformer I.Ms can exploit different strategy for filtering: eagerly evaluating if an item satisfies the predicate and storing this intermediate result as a flag directly in the item representations. Our results reveal that transformer LMs can develop numan-interpretabl implementations abstract computational operations that generalize in ways that are surprisingly similar to strategies used in traditional functional programming patterns.
diverse set of list- listpproeessing
INTRODUCTION
When asked to find fruit in list, language models reveal surprisingly systematic mechanism: they don't solve each filtering task anew, but instead encode predicates into portable representations. This neural representation of "is this fruit?" be extracted from one context and applied to different format, different language, and to some extent to a different task. These abstract, reusable operations suggest that transformers develop modular computational primitives rather than task-specific heuristics.
different list, presented
(a) Pere P(Crres Merc)
source run, M M(p.r)
+2.0
Cherry, Knife, Pants, Ambulance.
Cherry, Knife, Pen, Ambulance,
+1.5
Find the fruit.
Cherry
(g)
16:
Answerg
Arre
Head Index
24
+0.5
(e) M (Pdea)
( M (.ooe) 1[- Bme
Du
0.0
(c) Poistt = P(Ctttoty Viaea)
Binder
3. Binder
32
a. Binder
b. Peach
b. Peach
40
-0.5
Peach
C. Watch
C. Watch
-1.0
Watch
Scooter
d. Scooter
48
Scooter
(d)
-1.5
Phone
Adeet f- fere
Phone
e. Phone
56
Find the vehicle
d.Scooter
b.Pecch
-2.0
Answerg
Gere
20
23'
30
45 Layer
35 40
50
53
AIE
Figure I: A filter head [35, 19] in Llama-70B encodes compact representation of the predicate "is this fruit?".
(a) Within prompt Pare to find fruit in list, we examine the attention head's behavior the token (b) The head focuses its attention on the one fruit in the list. (c) We examine the same attention head's behavior in a second prompt Pdest searching different list for a vehicle (d) and we also examine the hehavior of the head when patching its query state to use the fere vector from the source context. (e) The head attends to the vehicle hut then redirects attention the fruit in the list after the query vector patched. (g) sparse set attention heads work together to conduct filtering over a wide range of predicates. These filter heads are concentrated in the middle layers (out of 80 layers Llama-70B).
"Correspondence sensharma.as northeastem.ede Wehsite filter. baulab. info.

---

Published as a conference paper at ICLR 2026
To understand this phenomenon systematically, turn to Marr's three levels of analysis (Marr, 1982). At the computational level, we identify what is being computed: the selection of elements satisfying a predicate. At the algorithmic level, we reveal how this is achieved: through a three phase computation corresponding to a filter, and reduce, occurring in that order. The map step is equivalent to populating the latents of the items in list with the right associations semantic information, step that has been documented in prior literature (Geva et al., 2023; Meng et al., 2022). In this work we focus on the non-trivial computation step. filter, that follows after map. At the implementation level, we reveal how filtering is implemented in LMs: through specialized attention heads, which we dub filter heads, that encode predicates as geometric directions in query space. We find that these heads, concentrated in the middle layers of the LM, remain largely shared even as the specifie predicate varies. This framework allows us to move beyond simply observing that LMs can filter, to understanding the explicit mechanisms through which list-processing operations emerge from the transformer architecture.
Our analysis yields three key insights:
Localized Mechanism. The list processing algorithm is implemented in a consistent set of localized components: set of attention heads that we call filter heads. These heads encode "compiled" representation of the predicate as query states at specific tokens - typically where the LM is required to produce its answer. These query states interact with the key states that earry semantic information of the list items, producing attention patterns that select the items satisfying the predicate.
Generalization. These filter heads are not specific a single predicate, but can encode distribution of predicates. And this encoding is sufficiently abstract that it can be extracted from one context and transported another context to trigger the same filtering operation on different collection of items, presented in a different format, in a different language, even in a different reduce task that follows after the filtering step.
Computational Redundancy. Additionally, our investigations reveal that LMs can perform filtering in two complementary ways: lazy evaluation via filter heads vs eager evaluation by storing is ssmtathh flags directly in the item latents. This dual implementation strategy mirrors the fundamental lazy/eager evaluation strategies in functional programming (Henderson & Morris Jr, 1976; Friedman et al., 1976). This second route reveals broader principle in neural computations: transformer LMs can maintain multiple pathways for the same operation (McGrath et al., 2023; Wang et al., 2023) and can dynamically select between them based on what information is available.
We validate these findings through experiments across six different filter-reduce tasks of varying complexity, each requiring the LM to filter based on different information before performing reduce step to provide a specific answer. We test the portability of the "compiled" predicate across different presentation format, language, and tasks. We conduct ablation studies to confirm the necessity of filter heads when the LM performs filter operations. Finally, we demonstrate that the learned predicate representations can serve as zero- zero-shot probes for concept detection, offering training-fis free alternative to traditional linear probing methods.
2 METHOD
2.1 BACKGROUNDS AND NOTATIONS
Language Model. An autoregressive transformer language model, M X + Y over a vocabulary V. maps a sequence of tokens {z1, ................................ IIn 2; E V} to y E RD,, which is probability distribution over the next token continuation of x. Internally, M has L layers, where the output of the ęth layer is computed as, h = h m Σys, a²s Here, m² is the output of the MLP, and a is the contribution of 7th attention head. For an individual head, its contribution to & at token position t is computed as:
8j
-
M
wes Wov Attn(g,, K)₂
i5t
9eKT
where q: q==1 h way
K = W Wh: and Atta(q., K) = softmax
Vilccad
2

---

Published as a conference paper at ICLR 2026
Here, denotes all tokens up to the current token t. Following Elhage et al. (2021), combine the value projection W and out projection W in a single W ov- From here onward will denote the jth attention head at layer l as [e, j]-
Filter Tasks. In functional programming. the filter operation is used to select items from a collection that satisfy specific criteria. filter takes two arguments: the collection, and a predicate function that returns a boolean value indicating whether an item meets the criteria. Formally:
filter(C, v) = {c E C v)(c) is True}
where C = (c1. C2...,,..... Cn) is a collection of items and v X (True, False} is the predicate function
To study how language models implement filtering, design suite of filter-reduce tasks T. For each task E T, we construct dataset D, containing prompts PPI.P2..... Pm }. Each prompt p: P(C, Φb) represents a natural language expression denotes the verbalization function that converts the formal specification into natural language. Figure I shows a concrete example, and we include additional examples from each task Appendix
specific filter-reduce operation, where
2.2 FILTER HEADS
We observe that, for a range of filtering tasks, specific attention heads in the middle layers of Llama- 70B consistently focus their attention on the items satisfying the given predicate, v). See Figure 1 (more in Appendix P) where we show the attention distribution for these filter heads from the last token position. From Equation (1), we know that this selective attention pattern emerges from the interaction between the query state at the last token (q-1) and the key states from all preceding tokens (K<t {kg, k...... .. }.).. We employ activation patching (Meng et al., 2022: Zhang & Nanda, 2024) to understand the distinct causal roles of these states.
To perform activation patching, we sample two prompts from Dj: the source prompt, Pare P(C.re. Vrrc) and the destination prompt, Pilstt (Cdesst, (dest), such that the predicates are different (V're * (. and the collections are mutually exclusive (Care O Cdest = 0). We ensure that there is at least one item Ctarg E Cdest, that satisfies thrre-
Figure illustrates our activation patching setup with an example. For filter head [e, . we analyze its attention pattern on three different forward passes.
source run (P We run the LM on the source prompt Parc and cache the query state for [e, j] at the last token position, - hereafter denoted as qarc for brevity.
destination run
M (Pdest): The LM is run with Pdest-
M (pdest][+ qsve]: We run the LM with Pdest again, but we replace the query state at the last token position for head [e, j].
patched run
8j with Gae cached from the source
The attention patterns for the head [e, j] from the three forward passes for an example prompt pair are depicted in Figure 1(b), (c), and (f) respectively. In the source and destination runs, the head attends to the items that satisfy the respective predicates. But in the patched run, the filter head [E,j] shifts its attention to the item in Cdest that satisfies virre- Patching Qae is enough trigger the execution of Varc for this head a different context, validating that ferc encodes a compact representation of vlare-
Notably, we cache the query states before the positional embedding (Su et al., 2024) is applied, while Attn(ge, K) in Equation (1) is calculated after the position encoding is added. This indicates that filter heads are category of semantic heads (Barbero et al., 2025) with minimal sensitivity the positional information.
2.3 LOCATING FILTER HEADS
Now we introduce the methodology to systematically locate these filter heads within LM.
Activation Patching with Learned Masks. While analyzing attention patterns can provide valuable insights. attention patterns sometimes be deceptive (Jain & Wallace, 2019) they may not give insights into the underlying causal mechanisms of the LM (Grimsley et al., 2020). To address this issue, perform causal mediation analysis with the activation patching setup discussed in
3

---

Published as a conference paper at ICLR 2026
Section 2.2 to identify the heads carrying the predicate representation. We want to find set of heads that causes the score (logit or probability) of the target item Ctarg to increase in the patched run.
We begin by patching the attention heads individually and selecting the heads that maximize the logit difference of Crarg in the patched run vs the destination run, logit(+- qsrel - (Caart We use logits instead of probabilities logits have more direct linear relationship with the influence caused by the intervention (Zhang & Nanda, 2024).
However, find that patching single filter head is often not strong enough intervention to exert inftuence over the final LM behavior because other filter heads, in addition to backup mechanisms (Wang et al., 2023; McGrath et al., 2023), may work against the intervention and rectify its effects. To address this issue, we learn sparse binary mask over all the attention heads, similar to De Cao al. (2020) and Davies al. (2023). We cache the query states for the source run A (Pærc) and destination run M (Paest), ) and then perform the following interchange intervention over the query
states of all the attention heads in the patched run:
+ mask" gote. + (1 mask * Idest
Here, the vector o01 denotes the query state of the jth head at layer l at the last token position; and dffc and L are the query states of that same head the last token from M (Pare) and M (paea). respectively. The terms mask R are each binary values, jointly learned with an objective to maximize the logit of Crarg. while suppressing other options in Cass, in the patched run. We use sparsity regularizer to ensure that the mask is sparse (i.e., only a few heads are selected), In Figure 1(g) we mark the filter heads identified for one of our filtering tasks, SelectOne Obj. with their individual average indirect effect (AIE) of promoting the logit of Ctarg.
Causality. If the identified filter heads fully capture a compact representation of the predicate vi in their query states which is used by the LM perform the filtering operation, then transferring qree from M (pare) to M (paest) should be causally influential: it should cause the LM to select Crarg the item in Cdest that satisfies Vlarc- We introduce causality score quantify the collective causal influence of the selected filter heads.
e" argmaax (preas) 104 + dee [[ejj] E #1), o Caca Pere:Pdest) Causality (H, Perc. Pdeest)
= 1 I[c ? Ctarg]
(4)
where H is the set of all selected filter heads
We run the LM on Pdest and patch the query state of only the selected heads at the last token position with their corresponding query states cached from M (P We then check if the LM predicts Grarg as the most probable item in the LM's output distribution among all items in Cdest-
Notably, while finding the heads we do not care if the heads exhibit the attention behavior illustrated in Figure 1. But, notice that the aggregated attention pattern of the identified heads consistently aligns with the selective attention pattern for filtering (see Appendix P for some examples).
3 EXPERIMENTS
We now empirically test the role of filter heads in different settings to validate claims.
Models. We study autoregressive transformer LMs in our experiments. Unless stated otherwise, all the reported results are for Llama-70B (Touvron et al., 2023). We include additional results for Gemma-27B (Team et al., 2024) in Appendix L.
Datasets. To support our evaluation, we curate a dataset consisting of six different tasks that all require the LM to perform filtering, followed by a reduce step to provide a specific answer. Each task-specific dataset D, contains a collection of items categorized in different categories (e.g. fruits, vehicles, in Obj rype), as well as different prompt templates for questions specifying the predicate and the reduction task (c.g. How many [category]s are in this list?). When we curate prompt for the task, we sample the collection from the items in D, and fill the template with the target predicate. The tasks are listed in Figure 3, and see Appendix A for example prompts from each task.
4

---

Published as a conference paper at ICLR 2026
1.0
Table 1: Causality of filter heads on SelectOne tasks. Heads identified using object-type filtering (e.g.. find the fruit) generalize semantically dis- tinct predicates like profession identification (find
0.9
Causality
0.8
the actor).
0.7
0.6
Semantic Type
Causality
Alogit
Object Category
0.863
+9.03
0.5
2
4
5
6
7
Number of Distractors
Person Profession
0.836
+7.33
Person Nationality
0.504
+5.04
Figure 2: Filter heads retain a causality close to 0.8 even with 7 distractors in the collections of the destination prompt.
Landmark in Country
0.576
+7.02
Word rhymes with
0.041
+0.65
Implementation Details. For each task we locate the filter heads using the method detailed in Sec- tion 2.3 on 1024 examples. During localization we perform the interchange operation (Equation (3)) only at the last token, but for evaluation we consider last tokens (1 (("Annnwwer,, ":" " 1) to reduce information leakage. We also calculate Qsee mean of n source prompts achieved from single Pare by changing the index of Csee in Curc.. While sampling the counterfactual prompts, we ensure that the answer for the source prompt. destination prompt, and the target answer for the patched prompt are all different from each other. All the reported scores are evaluated on a draw of 512 examples where the LM was able correctly predict the answer. In some cases we include Alogit, the logit difference of Cury in the patched run versus the destination run, as a softer metric to causality from Equation (4).
3.1 PORTABIILITYGGEEBRLLAAAAD
WITHIN TASK
Following the approach detailed Section 2.3, we identify the filter heads on the SelectOne task for object categorization. While localizing these heads we use English prompts that follow a specific format: the items are presented in a single line and the question specifying the predicate is presented after the items. We test whether the filter heads identified with this format generalize to various linguistic perturbations and SelectOne tasks that require reasoning with information of different semantic type. We evaluate generalization using the causality score (Equation (4)).
Information Types. Table shows that filter heads identified on object categorization maintain high causality even in entirely different semantic domains notably, identifying people by profession shows comparable causality despite the semantic shift. The filter heads also retain non-trivial causality for crrsonnnationality and landmark-couml associations, with causality improving by approximately 10 points when we include prefixes which prime the LM to recall relevant information in the item representations (see Appendix G).
However, the predicates captured by these filter heads show poor causality in situations that require reasoning with non-semantic information, such identifying rhyming words. This indicates that the filter heads play a causal role specifically in situations that require filtering based on semantic information rather than non-semantic properties like phonological similarity letter counting.
Size of the collection, C In Figure we plot the causality of filter heads by varying the number of distractor items in the list. The figure shows that the heads are not very sensitive to the size of the collection, retaining high causality even with 7 distractors.
Linguistic Variations. Recent works have shown that L.MSs capture language-independen ab- stractions in their internal representations (Dumas et al., 2024; Feucht et al., 2025). Building on these findings we test whether predicate representations in filter heads remain causal under linguistic perturbations by extracting frre from one prompt format and applying it to destination prompts with different presentation styles, or even languages. Table 2(a) and (b) demonstrate remarkable robustness: the same filter heads maintain high causality across different item presentation formats, and even cross-lingual transfer. This invariance to surface-level variation confirms that filter heads encode specific linguistic forms. However, we
abstract semantic predicates rather than mattern-matching also observe that when the question is presented before the items, the filter heads show poor causality (see Table 2(c)). We find that this is because in the question-before case the LM relies more complementary implementation of filtering, which we discuss in Section 5 and in Appendix B. All the other results presented in this section are calculated on prompts following the question-aftes format.
'This slightly increases the causality by removing the order information. See Appendix F.
5

---

Published as a conference paper at ICLR 2026
Table 2: Portability of predicate representations across linguistic variations. The predicate vector gae extracted from a source prompt and patched to destination prompts in (a) different languages, (b) different presentation formats for the items, and (c) placing the question before or after presenting the collection.
To
To
From
single line
bulleted
From
English
Spanish
French
Hindi
Thai
single line
0.863
0.842
English
0.863
0.893
0.779
0.928
0.951
bulletted
0.840
0.848
Spanish
0.857
0.877
0.775
0.875
0.891
(b)
Across option presentation style
French
0.938
0.932
0.793
0.931
0.9473
To
Hindi
0.920
0.920
0.885
0.918
0.957
From
after
before
after
0.863
0.580
Thai
0.897
0.928
0.887
0.940
0.943
before
0.398
0.020
(a) Cross-lingual transfer
(c)
Placement of the question
Evaluated On
Patched To
1.0
SelectOne (79)
0.8
SelectOne-MQQ (45)
0.39
0.50
0.27
0.44
0.06
0.02
0.60
0.90
0.22
0.41
0.04
0.12
Trained On
Cached From
SelectFirst (81)
0.68
0.73
0.77
0.11
0.01
0.03
0.58
0.71
0.26
0.14
0.07
0.6
SelectLast (145)
0,77
0.00
0.80
0.14
0.04
0.00
0.37
0.08
0.13
0.12
0.4
Counting (64)
0.61
0.79
0.38
0.58
0.36
0.05
0.60
0.66
0.34
0.50
0.36
0.12
0.2
CheckPresence (21)
0.30
0.00
0.02
0.91
Selectthe
SeleiOne-MMD MCQ
SelectFidd
Selectass
Comning
Check Presscee
Seleathee
SelectOne-Me MCQ
Select Selectiint
Couming
Preceeeeeenee
0.0
(a) Heads evaluated tasks
(b) Transferring asre across tasks
Figure 3: Generalization across different tasks. (a) shows whether the heads identified with one task (rows) maintain causal influence in another task (columns). (b) shows how portable the predicate representation is across tasks. The predicate rep Qae cached from one souree task example (c.g.. find the fruit in SelectOne task) and patched to an example from another destination task (c.g., count the vehicles Counting task). The heatmap shows causality scores, i.e. whether the LM correctly performs the destination task with the transferred predicate (e.g., count the fruits). For both (a) and (b) the values in the diagonal grid show within task scores.
3.2 PORTABBLITY/GENEAAIIIABLLLL ACROSS RRDDUE OPERATIONS
To understand the scope of filter head usage, we examine their participation across six filter-reduce tasks of different complexity. Each of these tasks requires the LM to perform a separate reduce step to produce an answer in a specific format. We measure whether the heads identified from one task maintain their causality when tested on another task.
Figure 3(a) reveals two distinct patterns. First, transferring the heads across the four tasks SelectOne, MCCQ)) SelectFirst, and Selectlast shows high causality scores (2 70%2) among them, which indicate a high overlap of the same filter heads. In contrast, Counting shows an interesting asymmetric pattern: while Selecr* heads fail on the Counting task, Counting heads show partial generalization to the Select* tasks sub-cireuit with Selecr* tasks, while having a more complex mechanism, likely involving additional circuits for specialized aggregation, that we have not yet identified. CheckPresence heads show poor causality even within the task, indicating that the LM possibly performs this task an alternate way that can bypass the filtering sub-circuit.
suggesting that Counting does share some common
We also test the portability of the predicate information encoded in gov by transferring across tasks, Figure 3().. SelectFirst and Selectl Last tasks show notably poor cross-task transfer of the predicate, even though the filter heads retain high within-task causality. This suggests that, in fere the predicate
"Exeep the heads from SelectOne-MCO. possibly because selecting MCQ-options is computationally simpler than output the filtered item.

---

Published as a conference paper at ICLR 2026
Table 3: LM performance on filtering tasks drops sig- nificantly when filter heads are ablated. These heads constitutc < 2% of the heads in the L.M Evaluated on 512 samples that the LM predicts correctly without any ablation (baseline 100%).
Table 4: Filter heads play a distinct causal role during filtering tasks. The table shows the causality of filter heads with other types of heads documented in the literature. None of the other head types match the causality of filter heads the SelecrOne task. To keep our comparisons fair, we keep the number of heads equal (79) for every head type.
LM Acc (Heads Abl)
Heaks)
Filter
Random
Head Type
Causality
Alogit
SelectOne (79)
22.5%
99.6%
SelectOne(MCQ) (45)
0.4%
100%
Filter
0.863
+9.03
SelectFirst (81)
13.1%
97.3%
Function Vector
0.002
-2.13
Selectl (145)
9.22%
99.4%
Concept Induction
0.080
+5.23
Count (64)
89.80%
99.19%
Token Induction
0.00
-3.23
CheckExistence (21)
98.61%
99.2%
Random
0.00
-0.96
information is possibly entangled with task-specific information. Otherwise, predicate transfer scores (Figure 3b) mirror the head transfer scores (Figure 3a) with slightly lower values.
Our findings suggest that filter heads form foundational layer for range of reduce operations, with simpler selection tasks relying primarily on this mechanism while more complex aggregation tasks build additional computation on top of it. This insight aligns with Merullo al. (2024a) that transformer LMs use common sub-cireuits (filter heads) across different (fitter-reuuce) tasks.
3.3 NECESSITY OF FILTER HEADS
We seek to understand to what extent the LM relies filter heads during these filter-reduce tasks. To assess their importance, we perform ablation studies.
We ablate attention head [ by modifying its attention pattern during the forward pass so that the last token can only bring information from the <B0S> token'. Previous works (Geva et al., 2023; Sharma et al., 2024) have investigated if critical information flows through a certain attention edge with similar attention knock-out experiments. The results in Table reveal a dramatic performance drop for the Selecr* tasks when filter heads ablated, despite these heads comprising less than 2% of the model's 's total attention heads - confirming their critical importance for the Selecr* tasks. In contrast the performance for Counting and CheckExistence do not drop significantly due this ablation, again indicating that these tasks do not fully rely on the filter heads.
To determine whether filter heads represent a novel discovery or merely overlap with existing attention head categories previously documented in the literature, we compared their functionality against such head categories with specific functional roles. Specifically, we measure the causality of Function Vector heads (Todd et al., 2024), Concept Induction heads (Feucht et al., 2025), and Token Induction heads (Olsson et al., 2022). As shown in Table 4, none of these previously identified head types exhibit the distinctive functional role of filter beads. We do not treat these heads as "baselines" in the traditional sense, as they are not intended for filtering tasks. We evaluate them to confirm that filter heads are a unique and previously unrecognized component in transformer LMs.
We also verify that filter heads encode abstract predicates rather than specific answer tokens: even when source prompts contain the predicate but no valid answer item in the collection, causality remains high at 0.80 (see Appendix E). Moreover, these predicate representations support composition through vector arithmetic: adding the query vectors of two predicates (e.g.. is ffuiit and is vehicle) produces vector that executes their disjunction, achieving a causality of 0.65 (see Appendix D). These results further confirm that filter heads encode predicates compact geometric directions in query space.
4 KEY STATES CARRY ITEM SEMANTICS FOR FILTERING
To understand how the predicate-encodee query states in the filter heads implement filtering via interacting with the key states from previous tokens, we design another activation patching experiment.
Approach. To isolate the contribution of key states, we designed a two-part intervention that combines query patching with key swapping. We select two items (Ctarg: Cother) from Caest such that Otarg satisfies there. but not Vdest: and Cother doesn't 'tt satisfy either of the predicates.
for the I. LM tokenizers that do not add a <B0S> token by default, we prepend <BOS> manually.
7

---

Published as a conference paper at ICLR 2026
The intervention proceeds as follows. For filter head [e,5], we patch the Gare from the source prompt (as before). Additionally, we swap the key states between Ctarg and Cother within the same forward pass. Figure illustrates this key-swa After this additional key-swapping intervention, the filter head [e, j] redirects its focus from Ctarg (iiuuee ((f) to Cather (Figure 4-right).
Result. We consider this 2-step intervention to be causally effec- tive if the LM assigns the highest probability to Guher among the items. On the SelectOne object categorization task, we achieve a causality score of 0.783 (432/552 examples) with Alogit 8.2591 + 3.352, confirming that key states indeed encode the seman- tic properties that predicates evaluate, For experimental simplicity, we restrict this analysis to only single-token items.
Binder
a. Binder
b. Peach
b. Peach
C. Watch
Watch
d. Scooter
d. Scooter
Phone
Phone
Figure 4:
Swapping the key
These results confirm our mechanistic hypothesis: filter heads im- plement filtering through a localized key-query interaction where queries selectively encode what to look for (the predicate) while discarding the item content and other unrelated prompt context, and key states similarly bring what is in the list by extracting the semantic properties of the items from the corresponding latents (h'-1), discarding the surrounding context. Notably, this behavior manifests in the attention pattern. Although previous works cautioned that attention patterns can be misleading (Jain & Wallace, 2019), in our prediaat-mmtching setting we see clear correspondence between how filter heads utilize abstract predicate encodings to attend to what was asked for and how they causally influence the LM behavior doing what Vaswani al. (2017) and the original designers of attention might have envisioned.
states of the items in addition causes the filter head to redirect its focus to an unrelated item.
5 WHAT HAPPENS IF THE QUESTION COMES before THE OPTIONS?
In Table 2(c) we see that we simply reverse the order of the question and the collection to ask the question before presenting the items, the causality scores drop to near zero. Our investigations reveal that this seemingly innocent ordering change fundamentally alters the computational strategy available to the LM.
Hypothesis. When the question comes first, the transformer can perform eager evaluation: as cach item is processed, the model can immediately evaluate whether it satisfies the predicate and store this information as an is match flag directly in the item's 's latents. And, at the final token position, rather than performing the predicate matching operation via filter heads, the LM simply retrieves items based on these pre-computed flags.
If this hypothesized flagging mechanism is true, then manipulating this flag should result in predictable outcomes in the LM's behavior. We find evidence for this alternative mechanism through a series of carefully designed activation patching experiments. We illustrate the most decisive experiment setup in Figure 5, while we leave the detailed analysis to Appendix B.
Experiment Setup. Figure 5 illustrates our two-part causal intervention setup to test for the presence of the hypothesized flag information in the item latents. Similar to the setup described in Section 2.2, we consider two prompts, a source prompt Parc = IP(C, thare) and a destination prompt Pdest = IP(C, (aest)), both following either the question-before or question-after format. As in Section 2.2, the predicates are different (Ware # #ddet), ), but they operate on the same collection of items in both the source and the destination prompt (Care Caest C). Let Care and Gdest be the items in C satisfying the predicates v/rcc and detest respectively. Our goal is to engineer a situation where third item CHagg & (Care; Cdest) } is the sole carrier of the is s_maachh flag, and then check whether the LM is misled into selecting Caag-
We achieve this with two simultaneous interventions:
(a) Transferring the predicate/decision context. In the destination run with Pdest, we patch the residual states of a single layer f, at the final two token positions ("\nAns" and ":") from the source run. When applied in later layers this intervention simply copies the final decision from the source run. But patching early layers transfers the predicate. See Figures 7 and 11 in Appendix B for details.
8

---

Published as a conference paper at ICLR 2026
Which one is the vehicle?
Care
Glest
Glag
Crand
Options: Watch, Peach, Truck, Monkey, Knife. AnAns:
20
(a) which is the fruit?
Options: Watch, Peach, Truck, Monkey, Knife. AnAns:
14
(b)
X
10
Watch
Peach
Truck
Monkey
Kn1 fe
20 Layer
50
70
Two-part intervention setup. (a) We patch the residual states at the final two token positions ("NnAns" and ":") from a source prompt Pere to destination prompt Pdest at a single layer. (b) Additionally, for an item c Cdest. we replace its hidden representations across all layers with those from prompt Pai containing different predicate Vaa & (dest:) We use a different Paiit (with different Mai)) per item in Cdest. Crucially, ensure that exactly one item caag (distinct from both the source and destination answers) carries its corresponding Paint Right: Logits of tracked items on the SeleciOne-O0 task as l varies for (a). with (b) applied throughout. After this 2-step intervention the LM consistently selects cRag in early layers, confirming the LM's reliance on pre-comppted answer flags stored in the item latents.
Figure 5: Testing for eagerly computed answer flags in question- prompts. Left::
is ssmatch flag from
(b) Planting the is match flag another item. We construct another prompt Prlag P(C, rhnng). with a different predicate Vnag & { Virc: thaest) ) that is satisfied by the third item CRag E C'(Carr: C. -. In M (PHag), only Otlag carries the is ms_match flag. To plant the flag on Oflag in the patched run, we replace the item's 's latents with corresponding latents cached from M (Pitag) across all layers.
Similarly to ensure that an item c' E C' (CHag) does not carry the flag, we construct another prompt p' TP(C, vf') such that (c)) We then swap the latents of d' for all layers in the patched run with the corresponding latents from M (p'). Doing these replacements for d' E C((CHag ensures that only Oftag carries the flag in the patched run. See Figure 10 in Appendix B for concrete example of Pnag and p'.
Results. Figure 5 (right panel) shows the logit of different items as we vary the layer & at which intervention (a) is applied, while intervention (h) is applied across all the layers throughout. In the question-before setting, the LM consistently selects Cflag in carly-to-middle layers, which is precisely the behavior predicted by the flag-based hypothesis. The LM ignores both the source predicate (Vare) and the destination predicate (ddest), and follows the planted flag instead. In contrast, replicating this experiment in the question -fter setting shows that the LM not strongly influenced by the flag, which is expected as the LM didn't 't get an opportunity to precalculate the flag in the item latents when predicate is presented after the items. See Appendix B for details on this.
Filter heads remain partially active. Despite the dominance of the flag-based mechanism in the guestion-before setting, filter heads are not entirely inactive. Table 2(c) shows that caching Qare from question-before prompts and patching question-after prompts yields non-trivial causality (0.398), indicating that filter heads still partially encode predicate information in question-firs prompts. The two mechanisms thus operate in parallel rather than in mutual exclusion, with the flag-based pathway typically dominating when the question precedes the collection.
APPLICATION: A LIGHT-WEIGHT PROBE WITHOUT TRAINING
The predicate information encoded by the filter heads can be leveraged for practical use-case: zero-shot concept detection through training-free probes.
Since filter heads encode predicates as query states that interact with key-projected item representa- tions to perform filtering, we can repurpose this mechanism for classification. To detect whether representation h belongs to particular concept class (e.g., animal, vehicle), we create filter prompts for each class: Pels = IP(C, viiscls) and collect the query states gels from a filter head [f,
Then we classify h by finding the class whose yets has the maximum affinity.
arg max Aols W cls
(5)

---

Published as a conference paper at ICLR 2026
Where W is the key projection from the head [e. 5]. Figure demonstrates that this approach achieves strong classification performance with- out any training, validating that filter heads learn generalizable concept representations that can be extracted and applied as probes.
0.-
Probe Accuracy
0.
02-
Filter heads encode predicates in compact geo- metric directions that produce distinctive atten- tion patterns via their interaction with the key states. We can use this insight to utilize filter heads to identify false information or detect sen- timent in frre-form text. See Appendix P.4 for some examples.
Pinte Accuncy [35, 19]
Lage
35
50
75
Layer Index
Figure 6: Training- free probe using filter head [35.
19]. Accuracy across layers on 238 objects spanning 16 classes from the PelectOne-Objeet dataset. Final token used for multi-token items. Compared against using the embedding vectors LM decoder as class probes.
7 RELATED WORKS
Attention Head Studies. Previous works have identified specialized attention heads that serve distinct computational roles. Olsson et al. (2022) discovered induction heads that implement pattern matching and copying, while Feucht et al. (2025) have identified heads that copy concepts instead of individual tokens. Todd et al. (2024) have found function vector heads that encode task representations that are transportable across contexts. Filter heads are an addition to this class attention heads that show distinct functional specialization.
LM Selection Mechanisms. A few empirical studies have explored the selection mechanism in LMs. primarily in MCQA settings. Tulchinskii et al. (2024) identifies "select-and-copy heads based on their attention pattern that focus on "An" after a correct item in a question-firs: MCQ format, Lieberum et (2023) also identify attention heads that attend to the correct MCQ label/letter and show that these "correct label" heads encode the ordering ID of the presented options. Wiegreffe et al. (2025) showed that attention modules in the middle layers promote the answer symbols a MCQA task. Unlike these works focused on MCQA settings, in this paper we investigate list-processing in general and find a set of filter heads that implement predicate evaluation that generalize across formats, languages, and even different reduction operations.
Symbolic Reasoning in Neural Networks. Recently researchers have been increasingly interested in the question of whether transformer LMs can develop structed symbolic-style algorithmic behavior. Yang et al. (2025) discuss how LMs can implement an abstract symbolic-like reasoning through three computational stages: with early layers converting tokens to abstract variables, middle layers performing sequence operations over these variables, and then later layers accessing specific values of these variables. Meng et al. (2022) and Geva et al. (2023) also notice similar stages while the LM recalls factual association. Several works have documented mechanisis//epprceetatii
specialized for mathematical reasoning (Nanda et al., 2023; Hanna et al., 2023; Kantamneni & Tegmark, 2025) and variable binding (Feng & Steinhardt, 2024; Prakash al., 2025).
Our paper continues this tradition of validating Smolensky (1991)'s assertion that distributed repre- sentations in connectionist systems can have "sub-symbolic" structures, with symbolic structures emerging over the interaction between many units. In this work we study a specific symbolic ab- straction filtering in list processing which is fundamental abstraction for both symbolic computation and human reasoning (Treisman, 1964; Johnson-Laird, 1983).
8 DISCUSSION
In this work, we have identified and characterized filter heads specialized attention heads that implement filtering operations in autoregressive transformer LMs.. These heads encode the filtering criteria (predicates) as compact representations in their query states of specific tokens. This encoding can be extracted and then transported to another context to trigger the same operation. We also identify that, based on information availability, the LM can use an eager implementation of filtering by storing flags directly on the item latents. These dual and complimentary filtering implementations mirror the lazy eager evaluation from functional programming. This convergence between emergent neural mechanisms and human -demandd programming primitives suggests that certain computational patterns arise naturally from task demands rather than architectural constraints.
10

---

Published as a conference paper at ICLR 2026
ETHICS
This research investigates the internal computational mechanisms of LMs through mechanistic in- terpretability techniques, contributing to the scientific understanding of transformer architectures. Our identification of filter heads advances LMs transparency by revealing how models implement functional programming primitives, though we acknowledge that interpretability findings do not di- rectly translate to safety improvements without additional work. The causal mediation techniques develop could potentially be applied to study more sensitive model capabilities, requiring responsible application and consideration of dual-use implications in future research on mechanisms related to deception or manipulation. Our experiments require significant computational resources that may limit reproducibility to well-resoureed institutions, though we commit releasing code and datasets to facilitate broader access. While our findings about filter heads appear robust across different tasks and languages, we caution against overgeneralizing to other domains without validation, as mechanistic interpretability remains carly-stage and our understanding of component interactions is incomplete.
REPRODUCIBILITY
The code and dataset produced in this work are available at gi github. com/arnabbapi/fiitte We ran all experiments on workstations with either 80GB NVIDIA A100 GPUs or 48GB A6000 GPUs, using the HuggingFace Transformers library (Wolf et al., 2019) and PyTorch (Paszke et al., 2019). We used NNsight
et al., 2025) for our intervention experiments.
ACKNOWLEDGEMENT
This research has been supported by a grant from Open Philanthropy (DB, AS, NS). and the Israel Council for Higher Education (NS). AS received compute credits from the U.S. NSF Advanced Cyherinfrastructure Coordination Ecosystem: Services & Support (NSF ACCESS), which supported some of our experiments.
REFERENCES
Afra Amini and Massimiliano Ciaramita. In-context probing: Toward building robust classifiers via probing large language models. arXiv preprint /41. 2023.
Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful? In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL net/Forrmm2id-GtwuNkkks
Róbert Csordás, Sjoerd van Steenkiste, and Schmidhuber. Are neural nets modular? inspecting func- tional modularity through differentiable weight masks. In International Conference on Learning Representations (ICIR), 2021. URL https:///ppppeeiiiii.eeooomommod nett:/orrummdd=7uuuuu---
Xander Davies, Max Nadeau, Nikhil Prakash, Tamar Rott Shaham, and David Bau. Discovering variable binding circuitry with desiderata. In ICML 2023 Workshop on Deployment Challenges for Generative AI, 2023, URL https: /ttps://opeereviiie www..e//oorumnndoooooo0o....
Nicola De Cao, Michael Schlichtkrull, Wilker Aziz, and Ivan Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking, arXiv preprint arXiv:2004. 14992, 2020.
Clément Dumas, Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. Separating tongue from thought: Activation patching reveals language-agnostic concept representations in transformers. arXiv preprint arXiv:2411.08744 2024.
Nelson Elhage, Neel Nanda. Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12, 2021.

---

Published as a conference paper at ICLR 2026
Jiahai Feng and Jacob Steinhardt. How do language models bind entities in context? In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https::/openrevittt net/fforum2id-z33b6kk00
Sheridan Feucht, Eric Todd, Byron C Wallace, and David Bau. The dual-route model of induction. In Conference on Language Models (COLM), 2025. URL https://openreviiee net.ie/wwn//ormmdddd bNTrKgqnG9.
Jaden
Alexander R Loftus, Eric Todd, Jannik Brinkmann, Caden Juang, Koyena Pal, Can Rager, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Lucchetti, Nikhil Prakash, Dmitrii Troitskii, Michael Ripa, Adam Belfki, Sumeet Multani, Carla Brodley, Arjun Guha, Jonathan Bell, Byron C Wallace, and David Bau. NNsight and NDIF: Democratizing access to foundation model internals. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https: http::///pperevveeeee net/ www.idddw/CCt@pp/oooooo
Daniel P Friedman, David S Wise, et al. CONS should not evaluate its arguments. Computer Science Department, Indiana University, 1976.
Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, December 2023. URL htttss//aclanttheee .rr. org/2023 emnlp-main main. 751 151..ff..
Christopher Grimsley, Elijah Mayfield, and Julia Bursten. Why attention not explanation: Surgical intervention and causal reasoning about neural models. In Proceedings of the 12th Conference on Language Resources and Evaluation, pp. 1780-1790. 2020.
Michael Hanna, Ollie Liu, and Alexandre Variengien. How does gpt-2 compute greater-than?: Inter- preting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36:76033-76060, 2023.
Peter Henderson and James H Morris Jr. A lazy evaluator. In Proceedings of the 3rd ACM SIGACT- SIGPLAN Symposium on Principles on Programming Languages, pp. 95-103, 1976.
Sarthak Jain and Byron C Wallace. Attention is not explanation. arXiv preprint 190.11890. 2019.
Philip Nicholas Johnson-Laird. Mental models: Towards a cognitive science of language, inference, and consciousness. Number 6. Harvard University Press, 1983.
Subhash Kantamneni and Max Tegmark. Language models use trigonometry to do addition. In ICLR 2025 Workshop Reasoning and Planning for Large Language Models, 2025. URL https://openreviiie heet/forumiid=HHhhoee
Tom Lieherum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09455 2023.
David Marr. Vision: A computational investigation into the human representation and processing of visual information. MIT press, 1982.
Thomas McGrath, Matthew Rahtz, Janos Kramar, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771, 2023.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances Neural Information Processing Systems, 13:19799-77372, 2022.
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024a. URL https:://ppnt eview. wwt.ee//orrmmmidffpooaaaaaa
12

---

Published as a conference paper at ICLR 2026
Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Talking heads: Understanding inter-layer commu- nication in transformer language models. Advances in Neural Information Processing Systems, 37: 61372-61418 61418, 2024b.
Simon Conway Morris. Evolutionary convergence. Current Biology, 16(19):R826 R827, 2006.
Neel Nanda, Lawrence Chan, Tom Lieherum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations (ICLR), 2023. URL https: //wp///eveeee.eee.e/eeeeeee/eeee
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.1 1895, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Nikhil Prakash, Natalie Shapira, Arnab Sen Sharma, Christoph Riedl, Yonatan Belinkov, Tamar Rott Shaham, David Bau, and Atticus Geiger. Language models use lookbacks to track beliefs. arXiv preprint arXiv:2505.14688 2025.
Arnab Sen Sharma, David Atkinson, and David Bau. Locating and editing factual associations in Mamba. In Conference on Language Models (COLM), 2024. URL https://openreviiae net/ Forum?ideyoVRyrggix.
Paul Smolensky. The constituent structure of connectionist mental states: A reply to fodor and pylyshyn. In Connectionism and the Philosophy of Mind, pp. 281-308. Springer, 1991.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocompuring, 568: 127063, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0m 08295, 2024.
Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function vectors in large language models. In The Twelfth International Conference Learning Representations (ICLR), 2024. URL https: http::://ppeneeiieee net/ for htt//formmmiiiiie in www.:eeetoouooeeeee
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.. 13971, 2023.
Anne M Treisman. Selective attention in man. British medical bulletin, 1964.
Eduard Tulchinskii, Laida Kushnareva, Kristian Kuznetsov, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, and Serguei Barannikov. Listening to the wise few: Select- and-copy attention heads for multiple-choice qa. arXiv preprint 4rXivv2410.023443, 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Kevin Wang, Alexandre Variengien, Arthur Conmy, Buck Shlegeris, and Jacob Steinhardt. In- terpretability in the wild: a circuit for indirect object identification GPT-2 small. In The Eleventh International Conference Learning Representutions (ICLR), 2023. URL https:
//openreview.oo www.iiewweeeffoooo www..e//ooormmiiiinsssssseooooo
Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabharwal.
Answer, assemble, Understanding how LMs answer multiple choice questions. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https:
//openreview net wttp::ene/.ooorriidddooa....
13

---

Published as a conference paper at ICLR 2026
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the ahe- natural language processing. arXiv preprint arXiv:1910.0377,, 2019.
Yukang Yang, Declan Iain Campbell, Kaixuan Huang, Mengdi Wang, Jonathan D. Cohen, and Taylor Whittington Webb. Emergent symbolic mechanisms support abstract reasoning in large language models. International Conference on Machine Learning (ICML), 2025. URL https: //openreview.ner wwt. mett/orrmmiddyy1SnPPPP
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https: /ppnrrevieeeeeeeeeeeeeeeee net nett/forumiid:tf11yyy888
Ziqian Zhong, Ziming Liu, Max Tegmark, and Jacob Andreas. The clock and the pizza: Two stories in mechanistic explanation of neural networks. Advances in neural information processing systems, 36:27223-27250, 2023.
14

---

Published as a conference paper at ICLR 2026
A EXAMPLE PROMPTS FROM OUR DATASET
A.1 DIFFERENT TASKS
Select One Type of Object
Select One Type of profession
"Options: Bus, Peach, Scooter, Phone, Pen Find the fruit in the options presented above.
"Options:
Neymar, Hillary Clinton, Clint Eastwood
who among these people mentioned above is actor by profession?
Answer:"
Answer:"
Expected LM Output: Peach"
Expected LM Output:
Clint"
Select One Type of nationality
Select One Location of landmark
"Options: Ronaldinho, Brad Pitt, Jet Li, Ken Watanabe
"Options: Cabo San Lucas Arch, Plaza de Armas Cusco, Mont Saint-Michel which of these landmarks is in Peru?
Who among these people mentioned above from China?
Answer:"
Answer:"
Expected Output: " Plaza"
Expected LM Output: Jet"
Select One Rhyme
Select One (MCQ)
"Options: blue, debt, bright, sting, sake which of these words rhymes with glue?
Banana
b. Paperclip
Answer:"
C. Oven
d. Dress
Expected LM Output: blue"
Church
f. Bench
Which among these objects mentioned above
is a clothing?
Answer:"
Expected Output:
Select First
Select Last
"Options:
Church, Scarf, Pendant, Slow cooker, Temple
"Options: Horse, Anklet, Golf ball, Cow, Necklace
What is the first building from the list above?
What is the last animal in this list above? Answer:"
Answer:"
Expected LM Output: Church"
Expected LM Output: " Cow"
Counting
Check Existence
"Options:
Trombone, Flute, Guitar, Train,
"Options:
Refrigerator, Museum, Notebook, Toaster, Juicer
Car
How many vehicles are in this list?
Do you see a kitchen appliance the list above?
Answer:"
Answer:"
Expected LM Output: Two"
Expected LM Output: " Yes"
15

---

Published as a conference paper at ICLR 2026
A.2
LINGUISTIC PERTURBATIONS
A.2.1 ITEM PRESENTATION
Single Line
Bulleted
"Options: House, Blender, Willow, Truck, Piano, Wrestling mat. Which among these objects mentioned above is kitchen appliance?
Temple
Air fryer
Basketball
Willow
Answer:
Van
Harmonica
Expected LM Output: Blender"
Which among these objects mentioned above
is a vehicle?
Answer:"
Expected LM Output: Van"
A.2.2 QUESTION PLACEMENT
Question After
Question Before
"Options: Elephant, Maple, Toilet, Camera, Juicer, Mall. Which among objects mentioned above is bathroom item?
"Which object from the following list is music instrument?
Options: Printer, Highlighter, Ukulele, Chair, Mirror, Locket. Answer:"
Answer:"
Expected LM Output: Toilet"
Expected Output: " Uk"
A.2.3 FROM A DIFFERENT LANGUAGE
Spanish
French
"Opciones:
Lirio, Colchoneta de lucha, Escritorio, Portátil, Refrigerador, Sandia.
"Options Aigle, Pastéque, Accordéon, Baignoire, Ciseaux, Bibliothéque. Lequel ces objets mentionnés ci-dessus est un(e) fourniture Réponse :"
?Cuáles de estos objetos mencionados anteriormente son un(a) electrénica?
bureau
Respuesta:"
Expected LM Output: " d"
Expected LM Output: Port"
Hindi
Thai
faras बेर मादन, 20य मोडर, sirg, लिययर, पसा.
"dbiãon: Raánmo, RoIa, nowlwña, muuhh, yniosa, usllá emplolussemmsmmmeuuiuiu.. 180m1 Aumau:
अपोीत angelt #f à को सlt yau पाानरर B?
Expected LM Output: "r""
Expected LM Outputt: "8"
16

---

Published as a conference paper at ICLR 2026
A.3 LM BASELINE PERFORMANCE ON TASKS
Table 5: LLM baseline performance filtering tasks. All the models were evaluated the same draw of 1024 examples for each of the tasks. The LMs are sorted in descending order by size. The question presented after the items.
SelectOne-Ne MCQ MCQ
Check CheckPresenes Presence
SelectOne
SelectFirst
Select Last
Couning
Model
(Huggingface ID)
Llama-70B
99.32%
98.63%
78.42%
87.01% 67.38%
91.02%
(meta- 1aaaaaaaaammmaaaaaa3a1 10-778-Innstree
Gemmaa277
99.02%
97.75%
75.29%
88.96%
64.16%
96.29%
2ooggle/gemaa-2277b-2
Gemma-9B
98.93%
97.85%
68.55%
78.03%
63.67%
95.90%
(google/
gemma-2-9b-it)
LIama-8B
98.44%
97.95%
50.88%
68.65%
63.28%
94.63%
(meta-Ilamm/LLamme333) I-B-Instuucc)
Table 6: LM baseline accuracy when the question presented before vs after the items. Reported SelectOne task object categorization. All LMs were evaluated on the same draw 1024 examples varying the question placement.
Ques Placement
Models
Before
After
Llama- 70B
99.3 99.32%
99.32%
Gemma-22
99.41%
99.02%
Gemma 9B
98. 08.93%
99.61%
L.aama-B
99.51%
98.44%
Table 7: LM baseline accuracy for the SelectOne tasks that require dealing with different semantic information. All the LMs evaluated on the same draw of 1024 examples semantic type. The question was presented after the options here.
Semantic Types Nationality
Models
Object
Profession
Landmark-comm country
Llama-70B
99.32%
98.44%
99.22%
100.00%
Gemma- 27B
99.02%
98.93%
91.60%
98. 14%
Gemma-9B
98.93%
98.73%
96.78%
98.14%
Llama-8B
98.44%
98.14%
94.04%
98.21%
17

---

Published as a conference paper at ICLR 2026
B DUAL IMPLEMENTATION OF FILTERING IN LMS: QUESTION BEFORE VS AFTER
Our analysis reveals that transformer LMs employ distinct computational strategies for filtering depending whether the question specifying the predicate precedes follows the collection. We briefly discussed this in Section 5 and here we provide our detailed analysis.
Table 2c shows that filter heads mini- mally causal when patched from question- before question-before prompt, even when both follow the same prompt tem- plate and item presentation style. To under- stand this better, we perform a multi-step causal mediation analysis.
which one is fruit in this list? Options: Cherry, Knife, Pen, Ambulance. AnAns:
which is vehicle this list?
Options: Binder, Peach, Watch, Scooter, Phone. AnAns:
Similar to the patching setup detailed in Section 2.2, we consider two prompts
Pere and Pdest- The prompts Pare and Pdest have different predicates (1'sre * (lesst) and sets of items (Care n Cdest 0). But both prompts follow the question-before format (or both follow the question-after format). See Figure 7 for an example of the two prompts.
Figure 7: Example of counterfactual prompt pair used to understand the effect of patching residual latents.
In the patched run M (pdest](+ h*). cache the residual stream latents at the last two tokens (f) ( "Ans", ":"}) }) for a layer from the source run A (Pste) and patch them to their corresponding
positions in the destination run M (Ptes). We perform this for all layers {1,..., T.} individually and track the scores (logits) of five tokens:
1. Care correct answer of the source prompt, Care E Care (aaceCaree)
2. Cdest correct answer of the destination prompt, Cdest E Cdest W/dess(Caes)
3. Ctarg the item in the collection in the destination prompt that satisfies the predicate of the source prompt, Curg E Caest Visc (Curg))
4. Coid : the item the destination collection that shares its index with carc in the source collection, Coid Caest index(Coia, Cdest) index Curc). We also make that Coid does not satisfy either predicate, -be (Coid) A -thaes (Goia). This token is supposed to capture if the residual states carry the positional information or order (Feng Steinhardt, 2024; Prakash et al., 2025) of Csre in the source run M (Pare).
5. Grand a random item in the destination collection that does not satisfy either predicate and is different from all the other four tokens.
Ctarg
Carc
Cdest
Coid
Crand
20
Logit
14
Causdny
0.5
0.5
0.0
0.0
0
10
20
30
60
70
08
20
30
40
50
60
70
Layer
Layer
(a) Question presented before
Question presented ofter the
Figure 8: Dual implementation filtering in LMs. both question- and question-after formats carg (blue) shows elevated scores after patching the residual state at middle layers. But in the question-before format that score is never strong enough to dominate cars (green). The violet line shows scores for Codd. the item that shares the same index in Cass with Gue in Cuc. which also shows elevated scores in middle layers, although pronounced as Carg-
18

---

Published as a conference paper at ICLR 2026
We curate the source and destination prompts such that all five tokens distinct and perform this experiment for both the question guestion-before and question-after settings. We plot the results in Figure and make the following observations.
01: In both question-before and question-after settings, the score of Ctarg (blue) increases in middle layers (30-55) where we identify the filter heads to be. However in the question-before setting, that score never strong enough to dominate Cdest (green). While in the question- -atter setting Curg becomes the highest scoring token among the four, achieving 70% causality in these critical layers.
02: We notice a bump in the score of Coid (violet) in the middle layers, although it is not as pronounced as Curg. This suggests that residual latents in these layers also contain the positional/order information of Crr- This has been observed in Feng & Steinhardt (2024) and Prakash et al. (2025). We also notice a slight bump the score of Grand (gray).
O3: If the patching is performed in late enough layers (> 60) it copies over the final decision (red) from the source run.
The distinction between the trends of Curg and Cdest in Figure 8a indicate that the LM relies more on an alternate mechanism, than the one involving the filter heads, in order to perform filtering in prompt where the question is presented before the items. We hypothesize that the question appearing before the collection allows the LM perform eager evaluation: storing is m_aachhh flag for each item in the collection when they are processed. If this true then manipulating the is_ s__aach flag should cause predictable changes in the LM's behavior in the question-fine setting, while the question-afier setting should not be sensitive to such manipulations.
Effect of ablating the is ___machh flag. If the L.M indeed using the is_match flag to perform filtering in cresttio-bbfoor set- tings, we would expect that ablating this flag would significantly degrade perfor- mance.
Peach
loo oo
ao Poo O-D
Find the fruit
Options:
To test this hypothesis, is_match flag in an item by replacing all of the residual stream latents of the to- kens of that item with their correspond- ing latents cached for the same item in a neutral prompt (see Figure 9 for details). When we perform this ablation for each of the items in the collection, we indeed see a significant drop in LM performance for the mmestion-before setting, while the performance remains mostly unchanged for the question-after setting (see Table 8).
ablate the
Peach
(*)
Answer
loo too foo
fo Roo
lool Loo
Figure 9: Ablating the is match flag. For item (e.g "Peach") in the collection, we cache the residual stream la- tents corresponding to the tokens of that item from a neutral prompt (c.g. any text w/o any predicate that contains the word "Peach"). Then in separate pass, replace the la- tents corresponding
This experiment supports our hypothesis that the question-before setting allows the LM to eagerly evaluate whether an item satisfies the predicate or not, store this in- termediate result in the residual latents of the items as it processes the collection, and relies on that to make the final decision. We also notice that the question-afted setting shows minimal sensitivity to this alag-ablation, which suggests that the processing of items do not rely on the context here: the LM populates the semantics (enrichment in Geva et al. (2023)) of each item context independent manner first, and then applies the predicate to perform filtering when the question is presented after.
that item in the original prompt with the cached latents. This effectively removes any information about whether that item satisfies the predicate or not.
Effect of swapping the ss__atcchhhh flag between two items. Our most decisive evidence comes from swapping the is match flag between items: if we swap the is match flag stored in Cpo. that satisfies the predicate with Oney that does not satisfy the predicate, should expect the LM to
Table 8: Effect of ablating is ss_atch.. Eval- uated on 512 examples from the SelectOne task.
Ques Place
W/o is match Acc
Before After
46.09%
19
96.06%

---

Published as a conference paper at ICLR 2026
Which is the vehicle?
Options: watch, Peach, Truck, Monkey, Knife. \nAns:
(a)
Which one is the fruit? Options: Watch, Peach, Truck, Monkey, Knife. AnAns:
(b)
Watch
Truck
Knife
Which one is a fuurnttur??
Which one is an animal?
Options: ..., .,,(Paac))
Options: Monkey
\nAns:
\nAns:
Figure 10: Counterfactual patching setup to swap the is s_amchc flag. We perform a swo-part intervention to determine whether the model stores filtering decisions flags in item representations. (a) We patch the residual states the final two token positions ("AnAns" and ":") from a source prompt Pee a destination prompt paet at a single layer. (b) Additionally, for an item c E Caes, we replace its hidden representations across all layers with those from a prompt Pai containing a different predicate Vait & (Pae, Viee J. We use different pain (with different vlua) per item in Caest Crucially, we ensure exactly one item Otug (distinct from both the source and destination answers) carries the is_match flag from corresponding plitt.
change its answer from Cpos to Cneg it relies on the is is_match flag. To test this hypothesis, we set up another activation patching experiment (illustrated in Figure 10, which is more elaborate version of Figure 5 in the main text). We perform a 2 part intervention:
I1: Similar to Figure 7, we consider two prompts Pare and Páest that follow the same format, either question- before or question afterr but with different predicates, there o d/dest- However, now they operate on the same collection of items, Care Cdest C. We perform the same intervention, patching the residual stream latents at the last two token positions from M (pec) to M (ptea) for layer f. And we track the scores of Cares Cdest Grand as defined before. Notice that as the collections are the same, Cearg = Coid
Care.
12: In addition, we choose another item Citag that is different from Case. Gdest. Gradd and perform the following intervention to make sure that only Olag carries the is j_maach flag while none of the other items do. In order to achieve that we cache Cnag 's latents from an alternate prompt Plag with predicate dhtag which is satisfied by Crtng, Unagg (Cang). Then in the patched run, replace the latents corresponding to Glag in M (paea) with the cached latents from M (pnag). This makes sure that Clag now carries the is match flag.
Similarly, to make sure that item e' E C(((nngg) does not carry the is_match flag, we cache its latents from another example p' with a predicate e such that -ab' (c'). Then we replace the latents corresponding c' in M (páes) with the cached latents from M (p'). We perform this for all items in C) (Caa ). This effectively ensures that only Cnag carries the is s_mmatch flag while all other items do not. See Figure 10 for an illustration.
In Figure 11 we plot the results of this experiment for both question-before and question-after settings. For layer e, 11 is applied for only that layer, without or with 12, which is applied to all layers.
As expected, we see that in the question-after setting applying 12 with I1 almost indistinguishable from just applying 11. 12 has minimal effect because the LM cannot rely on the is match flag when the question comes after.
However, in the ceestion-before setting, we observe that the score trend of crag (violet) and Caest (green) almost swap their positions in only I1 versus when 12 is applied in addition. With the flag-swap intervention 12, the LM systematically picks Ctag as the answer in the early layers. This further validates our hypothesis that the LM is relying on the is match flag to make the final decision in the question-before setting.
20

---

Published as a conference paper at ICLR 2026
Crcc
Cdest
Gilag
Crand
Residual
Residual match flag swap
75
20
15
Logi
14
14
12
12
10
10
0
10
201
80 Layer
50
00
70
090
0
10
20
30
40
50
60
70
(a) Question before collection
Layer
22
22
Logit
16:
15
14
12
10
50
(b) Question after collection
Figure 11: Effect of swapping the is ssmaachh flag hetween items. The figures on the left shows the effect of patching only the residual states (Figure 7) and figures the right are when additionally swap the is_math with another item (Figure 10). The pair of figures the top shows both cases in the question-before format (a). observe that endg becomes the top scoring item when the patching is performed early layers. this swapping of flags has no effect in the question after case, pair of figures on the bottom (b).
We do not claim that the LM only relies on the is matcch flag in the question-before setting. The fact that we see a bump in the score of Crarg (blue) in Figure and patching (src from question-before to question-afis -fater prompts has non-trivial causality (see Table 2c) indicates that the LM does carry the predicate information in middle layers even in the question- beforr setting, although not strongly as the question- after setting.
This dual filtering implementation strategy lazy evaluation via filter heads versus storing eager evaluation information with is match flag - exemplifies broader principle in neural computation: transformer LMs can maintain multiple pathways for core operations, dynamically selecting strategies based on what information is available. And the fact that filter heads still remain partially active even in the question- first setting shows that these mechanisms operate in parallel rather than in mutual exclusion.
C DIFFERENT APPROACHES FOR LOCATING THE FILTER HEADS
In this section we discuss the different approaches we explored identify the filter heads.
Filter Score. To capture the filtering behavior of the heads based their attention pattern, design a filter score that quantifies the extent to which a head focuses its attention on the elements satisfying the predicate v over other elements in C.
FilterScoro((e,j].c 4b) = score,y (c (b())) max (score;y(c) ((())
where, scorety Σ Attn K)+
LEC
While calculating FilterScore we make sure that there is only one item E C such that vè(c) is true. The FilterScore then select heads based on how much they focus their attention on the correct item over the most attended incorrect item. The score function sums up the attention scores over all tokens in an item c to account for multi-token items. Note that the score is calculated based on the attention pattern at the last token of the prompt "":
We notice that heads a range of middle layers exhibit stronger filtering behavior compared to those in the earlier or later layers.
21

---

Published as a conference paper at ICLR 2026
(a) FilterScore
(b) Indirect Effect
83
+0.4
163
+0.2
+0.5
Head Index
24
32.
0.0
0.0
001
-0.2
-0.5
483
40.4
-1.0
56
5
PikerSoore
10
< 10
5P Layer
10
ABR
Layer
Figure 13: Location filter heads Jlama-70B (a) shows the individual FilterScore each head: how much they attend to the correct option over others. (b) shows the indirect effect: how much patching Quee from a single head promotes the predicate target. The filter heads identified with Section 2.3 are marked with black borders.
Activation Patching. We can patch the attention heads indi- vidually and quantify their indirect effect at mediating the target property in the patched run. Todd et al. (2024) and Feucht et al. (2025) identified Function Vector heads and Concept heads with this approach. Specifically, for each head [ we patch aee from the source run to the destination run with the method detailed in Section 2 and check its effect on boosting the score (logit) of the target item, Starg. The indirect effect is measured logit/+ qerel (Ctarg) logit ( (targ).
1.0
0.86
0.8
0,67
0.6
0.59
0.4
0.2
Activation Patching with Learned Masks. our analysis we find that patching single head is not enough causally influence the LLM behavior. This is because the same mecha- nism have parallel implementations (McGrath et al., 2023), which may work against the intervention to rectify its effects. To overcome this issue, learn differential mask locate the minimal set of components (attention heads) that collec- tively implement filtering. Following De Cao al. (2020); Davies et al. (2023); and Csordás et al. (2021), we optimize a differential binary mask over all of the attention heads. The idea learn a binary mask for the neural components (query states of attention heads in our case) such that after an interchange intervention (Eq. (3)) is performed, the LM shows the desired counterfactual behavior (executing the predicate from the source prompt).
0.0
FiterScore
CMA
CMA-Mask
Figure 12: Collective causality 79 filter heads identified with FilterScore, CMA, and CMA with learned masks. Evaluated the
512 examples from the SelectOne task
When we compare different head identification approaches while controlling for the number of heads selected, the learned mask approach achieves the highest causality (Figure 12).
D VECTOR ALGEBRA WITH PREDICATE REPRESENTATION
We explore the geometric properties of the predicate representation (sre by examining its behavior under vector arithmetic operations. Specifically, we investigate whether, when we compose two predicates (find the fruit and find the vehicle) by adding their corresponding (sro vectors, this resulting vector represents a meaningful combination of the two predicates.
Adding predicate representations results in disjunction of the predicates. If we add the geee vectors of two source prompts with different predicates, Psrcl PCC1, 4/ is_fruit) and Pare2 = P(C2, 1/22 is vehicle), we find that the resulting vector 4composed qarei + qare2 can be used destination prompt Pdest = P(C3, th3 & {is fruit, is vehicle))) } to execute the disjunction the two predicates (i.e.,, the fruit vehicle) in C3. The setup is illustrated in Figure 14 with example from the SelectOne task.
We conduct this experiment for the SelectOne task. We compose Icomposed with two prompts and curate Paess such that there is only one item Grarg in Caest that satisfies the composed predicate w/sre1 1/ere2 We patch Geomposed the destination and consider the intervention to be successful
22

---

Published as a conference paper at ICLR 2026
if the LM thinks Caarg is the most probable item. We achieve causality score of 0.6523 (334 out of 512) with logit of Ctarg increased by 6.75 3.94 after the intervention.
This shows that the predicate representations compositional, allowing for the construction of more complex predicates through simple vector operations.
(a) (Prrit)
Ozclors: sharppeo SLlawberty Kylophone, Warehouse, Library, Library
d.1 M (Ptea) + 9ccempeeed
Which amorg thete abjacts mentioned above
Answe
Options: Blueberry Earring Warehouse, Trumpet, Nightstand which among these objects mentioned above music Instrument? Answer
(b) M (Peehllla)
Options: Herp Juicor, Bathoub Boet Swretor: Flano. amana these onjects mentlaned ahove asa kajsehics?? Answe
d.2 Gold reference
Afruit + Qvehiele = Teomposed
Options: Blueberry Earring, Tie, Warehouse, Trumpet, Nightstand, which among these objects mentioned above Fruit vehicle
Cptians: Slueberry, Earring,, Tia, Warchouse, Trump Nightstard among these nhjeers mantioned ahrue muci meat? Anowm
Answer
(c) M (pdes)
Figure 14: Aggregated attention pattern of the filter heads from the last token position, with the composition
setup. The predicate encoding Que is collected from 2 prompts, (a) Pac1 IP(C1, the is_ is_fritt) and (b)
Prec PC2C, 1/2 = is_ veviccll)) Their addition @oomposed patched the destination rum. The resulting attention pattern shown (d.1) indicates that now filter heads select the items in Piea that satisfy thaet U Mhee.
(d.2) shows the attention pattern for gold prompt with the disjunction predicate.
E DISTINGUISHING ACTIVE FILTERING FROM ANSWER RETRIEVAL
Our identification of filter heads raises two critical questions about their computational role. First, do these heads actively perform filtering, do they merely attend items that already filtered by earlier layers? Second, do they encode the abstract predicate (e.g., "is a fruit") or simply match specific answers from context? To establish that filter heads actively perform filtering rather than passively attending to pre-filtered results, we designed causal intervention experiments where query states carrying predicate information are transferred between prompts. The consistent ability of these transferred queries to trigger the transported filtering operation on entirely different collections demonstrates that the heads actively apply predicates rather than simply reading pre-come pompooeeeed results.
E.
Do THESE HEADS MERELY MATCH WITH CRE-COMPUTED ANSWER?
more subtle concern is whether these heads encode abstract predicates or simply store concrete answers. For instance, when the source prompt asks to find the fruit with the answer being Plum, does the query state encode the predicate find the fruit or the specific item Plum? In the patched run, the Plum representation would naturally show higher similarity to the representation of Apple (another fruit) than to Watch (a non-fruit), potentially explaining the selective behavior we observe in the attention pattern after paiching Gare in the patched run.
To resolve this ambiguity, we designed critical experiment: we use source prompts that contain predicates but valid answers. We observe that even in such cases, the filter heads retain their high causality of 0.80 (410 out of 512 examples from the SelectOne task, ACarg = 8.08 3.1). Combined with our ablation studies in Section 3.3, these experiments demonstrate the crucial role of filter heads in actively participating in filtering, rather than simply mediating the pre-filtered items.
F AVERAGING TO REMOVE THE ORDER ID
While in Appendix E we make the case that filter heads encode predicates rather than specific answers, analysis reveals that sometimes filter heads additionally transfer the position of the answer in the source prompt.
When examine the cases where our intervention fails, find that the model sometimes selects the item at the same position as the original answer in the source prompt. Figure also shows elevated scores for items that match the position of the source answer Coid in critical layers, although not as
23

---

Published as a conference paper at ICLR 2026
high Curg. This suggests that the LM also encodes the positional information order ID (Feng & Steinhardt, 2024; Prakash et al., 2025) of Care alongside the predicate. A probable explanation for this that, since these filter heads distributed across range of layers, patching are from filter heads in later layers may bring over specific contributions from filter heads in earlier layers.
Options: Cherry, Knife, Pen, Anbulance. which one fruit this list?
\nAns:
Knife, Cherry, Pen, Anbulance. Which one is fruit in this list?
Options:
AnAns:
Options:
Knife, Pen, Cherry, Anbulance. Which one is fruit in this list? \nAns:
Knife, Pen, Anbulance, Cherry. Which one fruit this list?
Options:
AnAns:
0.0
wro Avg
Avg
Figure 15: Averaging fee to remove the order information. This simple trick improves causality scores by 7 to 10 points across the board. Causality scores presented for 512 examples from the SelectOne task.
To isolate the predicate signal from this positional bias, query states across multiple source prompts. We produce n variations of the same source prompt Par (P(Crre, vhsre) by changing the index of the correct
simple trick: averaging the
Cre- Figure 15 illustrates this idea.
This simple averaging improves our causality scores.
G ADDING A PRIMING PREFIX HELPS WITH CAUSALITY.
Table 1 shows that filter head causality varies across information types for the same SelectOne task. Notably, tasks requiring recalling person's 's nationality and the location of landmark show lower causality scores.
Following Amini & Ciaramita (2023), provide contextual priming and check the causality in these cases. In a question-after format, before presenting the items we add a prefix that explicitly instructs the LM recall relevant information, can achieve approximately 10-point improvement in causality scores. This experiment further validates the hypothesis that filter heads work better when the relevant semantic information required for filtering already present in the item latents.
Table 9: Priming context helps improve the causality score.
Filtering Task
W/O Priming
With Priming
Priming Prefix
Recall the nationality of these people:\n
Person Nationality
0.504 (258/512)
0.625 (320/512)
Recall which country these landnarks are located in: An
Landmark in Country
0.576 (295/512)
0.670 (343/512)
24

---

Published as a conference paper at ICLR 2026
H ABLATION ON DESIGN CHOICES
In this section we conduct ablation studies two design choices we made in this work.
Sparsity Regularizer. To identify the minimal set of heads necessary for filtering, we incorporate the LI norm of the mask in our optimization objective. This sparsity constraint ensures we select only the most influential heads while filtering out noisy ones.
-logi((Ctarr)
logit(c) + A||mask||
Cdestt
eCCdeat
eshCCarg
Sparsity
Losset
The target loss maximizes the logit of Ctarg (the item satisfying the source predicate vrrre), while suppressing the other items in Cdest. The sparsity penalty, modulated by the hyperparameter X, pushes the mask towards zero so that fewer heads are selected.
We find that selecting the right sparsity coefficient is crucial to achieve higher causality. Through a systematic search across À E (0, 1J. while control- ling for other variables (training data, other hyper- parameters) of the optimization process, we find a clear sweet spot at À 0.02. In Figure 16 we re- port the causality of identified heads across format, where the source prompt is from SelectOne and the destination prompt from SelectOne-MCQ While evaluating, we do not employ the averaging trick (Appendix F), and transfer head states at only the last token.
Causality
Number 10r
5
10"
2c-05
2o-04
2e-05
26-02
26-01
16-00
(Sparsity Weight)
Figure 16: Sparsity-performane performance tradeoff. Optimal causality achieved with 79 heads identified with À 0.02. Lower values of X includes noisy heads while tion allows noisy irrelevant heads into the mask, larger values over-prune critical heads.
This tradeoff intuitive: insufficient regulariza-
reducing generalization performance. And excessive regularization ends up pruning essential heads, degrading causality. We have identified this optimal value of À with Jlama-70B but use the same value for all the models we investigate in this paper.
Last two tokens for evaluation. Autoregressive transformer LMs process texts sequentially without knowing what comes next. When the LM reaches the second to last token position " "AnAnswer", doesn't 't know that ":" will follow. So, it may begin performing filtering at "AnAnswer", which will be accessible from the ":" token. This creates potential problem for our intervention: if only perform our intervention at ".", pre-computed values "AnAnswer" may leak at or after our intervention location, reducing the efficacy.
To partially address this issue, during evaluation we patch the query states of filter heads for last two tokens, both "'nAnswer" and "." In our controlled setup, we ensure that our source and destination prompts always share these same last two tokens, which allows us to do this intervention.
Table 10: Evaluation by patching only the last token for SelectOne task.
Semantic Type
Causality
A Alogil
Object
0.819 (839/1024)
+8.12 + 2.93
Person Profession
0.842 (431/512)
+7.31 + 3.04
Person Nationality
0.629 (322/512)
+6.28 + 3.75
(+ Priming Prefix)
Landmark country
0.678 (347/512)
+7.05 + 3.20
(+ Priming Prefix)
For the SelectOne- Object task, single-token patching achieves a causality of 0.819 - only slightly
lower than the 0.863 report in the main text with two-token patching (Table 1). This suggests that while preventing information leakage helps, it not as critical as we initially anticipated. The filter heads remain robustly causal with this simpler intervention.
25

---

Published as a conference paper at ICLR 2026
I DECOMPOSING Wok INTO SMALLER FUNCTIONAL UNITS WITH SVD
We investigate whether attention heads are too large of a model unit for analyzing the filtering task and we can be more surgical by decomposing heads into smaller functional units. We take inspiration from Merullo et al. (2024b) and attempt to locate low-rank subspaces of the QK matrix necessary to perform the filtering task.
Approach. In attention heads, the query projection Wo and the key projection WKK read from the residual stream latent h2 to determine the attention pattern, while the OV projection Wov writes back to the residual stream. Equation (1) shows how the attention distribution from a query token t is calculated, but for simplicity it hides some technical details, such the fact that a pre-layemorm is applied on h e before Wx and Wo are applied, and RoPE positional encoding (Su et al., 2024) applied the resulting query state and key states before the attention score calculated. If do not ignore these details the pre-softmax attention score from the 9th token to the k token is calculated
Ogk Wa, (RoPE(huW<)) where h LayerNorm (h)
(8)
When we ignore the positional encoding (i.e. setting RoPE = I), we can combine Wo and WK projections in to a single bilinear form.
T = =hg Wo 1ig = h4 Wok hg Ogk = hy We (hx
Note that although Wor is a dembd X dembe matrix it has rank of dhead. This is because Wq and WK both have rank of dhead- To get the attention distribution from the 9th token the attention score Ogk is calculated for all & S which is then divided by scaling factor V before applying softmax (see Equation (1)).
If we take the SVD of Wok, we can rewrite it as sum of rank-1 1 matrices, the outer products of the left and right singular vectors, scaled by their corresponding singular values.
dheat
Wox U.VT
(10)
Merullo et al. (2024b) found that subset of these rank-1 components of Wo read from low-rank subspaces of the residual stream, which has been written into it by specific components W Wo of the heads from earlier layers, forming a low-rank communication channel via the residual stream. In this section we investigate if filtering is implemented by the specific components of Wok-
we want to learn a binary mask E {0. 1 Jhohae that selects specific components of W Wor crucial for the filtering task. In the patched run we recalculate Ogk by selectively patching these components:
For a head [f,]] j]
agk = Upatch VVT Tig
(11)
Where, Upatch Udest + (Nare Udess) mask
Use Mu U the left projected latent from source Pae
Udest hast U the left projected latent from destination Pdest and, C denotes element-wise multiplication
(12)
Note that Upatch a dhead vector where the 7th value the same as the ith value of use when mask; is on, and the same as the 7th value of Udest when mask; is off. Recalculating ags. in the patched run ensures that only the information encoded in the selected subspace is transferred from the source prompt while the information other components is preserved. Similarly to the method detailed in Section 2, we learn this mask for all heads.
Figure 17 shows that middle-layer attention heads contain more components relevant to the filtering task, which is consistent with where locate filter heads. Interestingly, while optimization
26

---

Published as a conference paper at ICLR 2026
Table 1I: Causality of the SVD components (with- out the averaging trick) on the SelectOne task. Components located using object-type selection (e.g. find the fruit) generalize person identif- cation (e.g. find the actor). Both in-distribution and cut-of-distriuutie
Number of Active Components
causality were calculated using 512 examples. Excluding the components in later layers significantly improves performance both the in-distribution and -iitt-i-iitiittiion validation sets.
30
50
Attention Heads
Object-type
Profession
Index
All
0.627
0.420
Distribution of critical SVD components of W Wox across attention heads in Jlama-70B. Middle layers show higher component counts, which aligns with filter head locations.
Figure 17:
Heads in & <, 52 Only filter heads
0.811
0.773
0.766
0.752
identifies components in later layers, excluding these actually improves causality (Table 11), which suggests that they may introduce noise rather than information relevant for filtering. Most notably, using only the SVD components of previously identified filter heads achieves high causality (0.766 for in-distribution, 0.752 for nut-of-distribution). demonstrating that predicate information is concentrated in low-rank subspaces within specialized heads. This finding indicates that filter heads implement filtering through coordinated interactions between low-rank subspaces the query and key states, rather than using their full representational capacity.
J WHAT DO THE FILTER HEADS WRITE TO THE RESIDUAL STREAM?
Our previous experiments established that the filter heads encode the filtering criteria (the predicate) in their query states and perform filtering through query-key interactions. But what information do these heads actually write back to the residual stream? We investigate two possibilities:
H1: Filter heads write the filtered item directly. For example, writing apple when filtering for fruits or Tom Cruise when filtering for actors.
H2: Filter heads write the positional information the order id of the filtered items (e.g., the second item). And this information is then dereferenced based on the task/format requirements with a lookback mechanism identified in Prakash al. (2025).
To distinguish between these hypotheses, design controlled causal mediation analysis by patching the output of the filter heads (output of the OV projection). We create source and destination prompts with different predicates but the same collection of items presented in different orders (see Figure 18 left). This ensures that Cue- the correct answer of the source prompt remains a valid option both prompts, enabling us to test HI. While shuffling, we make sure that the order of Care different so that we can test H2. We choose an MCQ format and ensure that the MCQ labels differ between the prompts to avoid confounding effects of this intervention.
We patch the output (OV contribution to the residual stream) of 79 filter heads identified with the SelectOne task (not MCQ). The results plotted in Figure 18 (right) reveal nuanced insights:
1. The logit of Cure the answer in the source prompt, and the target of the transferred predicate, increase significantly after patching. The MCQ label associated with Care in the destination prompt also elevates significantly. This provides some evidence in support of H1 that filter heads do write filtered items directly to the residual stream. However, we acknowledge that these elevated scores may also be explained if filter heads in earlier layers also mediate the predicate information, which is then read and executed by the filter heads in later layers.
2. The logit of caid and its MCQ label also increase, suggesting that filter heads write the positional information well (support for H2).
27

---

Published as a conference paper at ICLR 2026
Carnation
Destination run
Patched run
b. Basketball
Tomato
Clesst
d. Pressure cooker
e. Camera
moq (Gima.))
f. Truck
Care
Which among these objects mentioned
above flower?
Token Type
mod (Pac)
Answer
p. Pressure cooker
Coid
9- Carnation
mon (Cala)
Camera
5. Basketball
Grand
Truck
u. Broccoli
(Cmaa)
which among these objects mentioned
above is vegetable?
10
20
Answer
Logit
Testing what filter heads write to the residual stream. Left: Source and destination prompts contain identical items in different orders with different MCQ labels. Right: Logit changes after patching the filter head outputs. meq(c) denotes the MCQ label label in the source prompt. Both Gue and Cod (and the MCQ labels in destination prompt) show increased scores, suggesting that filter heads write both the filtered content and its positional information to the residual.
Figure 18:
item in the destination prompt and meq' (c) denotes MCQ
Although our findings from this patching experiment are somewhat inconclusive, they suggest that filter heads write both what items filtered and where they are located. This dual encoding strategy aligns with our earlier finding multiple implementation pathways. This redundancy may enable the LM to flexibly use either piece of information, both of them in parallel.
K DISTINGUISHING FILTER HEADS FROM OTHER SPECIALIZED ATTN HEADS
We compare filter heads with two previously characterized attention head types to establish their distinctiveness. Function Vector (FV) heads, identified in Todd et al. (2024), encode compact representation of the task demonstrated in few-shot examples. And, Concept Induction (CI) heads, identified in Feucht et al. (2025), encode the concept or lexical unit in multi-token words.
Following the identification procedures detailed in the respective papers, locate FV and CI heads in Llama-70B. We use publicly available dataset and source code from Todd et al. (2024) and Feucht al. (2025). In Figure 19 we plot the distribution of these heads with filter heads identified with SelectOne task. Interestingly, although both FV heads and filter heads are concentrated in the middle layers, only 7 (out of 79) heads overlap. This separation suggests that these head types play parallel, complementary roles in LLM computation.
Head
Induction Hoak
36
26
Gu
25
48
45
12
33
Luyyr
FV Heads
Concagt Induction Hoxis
Fiber Houls
Overlop
Figure 19: Distribution of specialized attention heads in Llama-70B. Both Filter and FV heads are concentrated in the middle layers, but there are only overlapping heads. CI heads exhibit weaker concentration in the middle layers, with minimal overlap with filter heads (only one head). For fairer comparison, choose 79 heads from each head type.
28

---

Published as a conference paper at ICLR 2026
To verify whether these head types different computational roles, check their causality and perform ablation studies. The pear-zero causality for FV and CI heads confirms that they don't directly contribute to filtering. Ablating filter heads catastrophically degrades the task performance, while removing other head types has a limited impact. The 12% performance drop when ablating FV heads, despite their minimal causality score, suggests that they contribute indirectly possibly encoding some task-specific information such as the format of the expected answer.
Table 12:
Causality and necessity of different head types in filtering task. Evaluated on the same 512 example pairs where the LM achieves 100% accuracy without any ablation. Filter heads show strong causality and ablating them significantly reduces the task performance. Both FV and heads show minimal causality. However, ablating FV heads reduces task performance by 12%.
Head-type
Causality
LM Acc after ablation
Filter
0.863
22.5%
Function Vector
0.002
88.3%
Concept Induction
0.080
98.2%
Random
0.00
99.6%
The clean separation between FV heads and filter heads, despite being concentrated in similar layers, suggests that transformer LMs have developed distinct, specialized modules that operate in parallel.
L REPLICATING EXPERIMENTS IN GEMMA-277 -7B
We have replicated
experiments for Gemma-27B and observe that the results mostly align with our findings in Llama-70B.
L.1 LOCATION OF THE FILTER HEADS
(a) FilherSoore
(b) Indirect Effect
40.6
-0.4
8
Head Lodex
+022
0.0
I
-0.2
-12
24-
-04
-05
+ Layer
10
5b
Layer
Figure 20: Location of filter heads in Gemma- -7B-it. Similar to Figure 13, (a) shows the individual FilterScore for each head: how much they attend to the correct option others. (b) shows the indirect effect: how much patching fere from single head promotes the predicate target. The filter heads identified with the method detailed Section are marked with black borders.
L.2 WITHIN TASK GENERALIZABILITY
Table 13: Causality of filter heads SelectOne tasks. Heads identified using object-type filtering generalize semantically distinet predicates like profession identification. Compare with .lama-70B scores Table
Semantie Type
Causality
Alogit
With Priming
Object Type
0.821
+9.95
Person Profession
0.770
+9.10
Person Nationality
0.305
+5.98
0.404
Landmark in Country
0.410
+6.48
0.455
Word rhymes with
0.018
+0.12
0.037
29

---

Published as a conference paper at ICLR 2026
Table 14: Portability predicate representations Gemma-27B filter heads across linguistic variations.
Compare with Table 2. The predicate vector faee is extracted from a source prompt and patched to destination prompts in (a) different languages, (b) different presentation formats for the items, and (c) placing the question before or after presenting the collection.
To
To
From
single line
bulleted
From
English
Spanish
French
Hindi
Thai
single line
0.824
0.805
English
0.824
0.822
0.777
0.840
0.852
bulletted
0.826
0.799
Spanish
0.822
0.815
0,834
0.820
0.818
(b)
Across option presentation style
French
0.760
0.811
0.779
0.805
0.807
Hindi
0.787
0.816
0.834
0.822
0.838
From
after
before
afler
0.824
0.412
Thai
0.805
0.816
0.807
0.793
0.797
before
0.190
0.055
Cross-lingual transfer
(c) Placement the question
L.3
ACROSS TASK GENERALIZABILITY
Evaluated On
Patched To
1.0
SelectOne (77)
0.69
0.82
0.83
0.05
0,02
SeleatOne-MCQQ (39)
0.35
Trained On
Cached From
SeleotEirst (77)
0.80
0.81
0.74
0.05
0.03
0.6
SelectL (76)-
0.82
0.82
0.06
0.88
0.04
0,02
0.57
0.74
0.37
0.58
0.04
0.01
004
Counting (39)
0.03
0.61
8.00
0.00
0.08
0.03
0.03
0.42
0.01
0.10
0.08
0.115
CheckPreseence
0.04
0,71
0.01
0.01
0.05
0.05
0.03
0.33
0.03
0.05
SelevOOne
SolevtOn-MCQ
SeleatFirst
SolectLast
Couning
Presmnce
SelenOhne
Select MCQ
SelectFinal
SalestLast
Counting
CheckPrecave
0.0
Heads evaluated across lasks
Transferring Gsrc across tasks
Figure 21:
Generalization of filter heads across different tasks. Compare with Figure 3. The ligure the left shows if the same filter heads generalize across tasks. And the figures the right shows if the predicate can be transferred to a different task.
30

---

Published as a conference paper at ICLR 2026
L.4
DUAL IMPLEMENTATION IN QUESTION BEFORE VS QUESTION AFTER
Otarg
Csrc
Cestt
Coid
Crand
17.5
115
13.5
1500
20.0
135
2.5
1.0
00
000
(s) Question presented before the items
(h) Question prosented ofter the ieetss
Figure 22: Effect of patching the residual latents in Gemma-27B. Compare with Figure 8.
Care
Glest
Cjagg
Grand
Residual
Residual maich Dlgg
215
223
78
38
172
17.2
35.0
15.0
10
50
10
20
Tayer
(a) Question before collection
Liyer
70
(b) Question after collection
Lays
Figure 23: Effect of swapping the is match flag between items in Gemma-27B. Compare with Figure 11.
31

---

Published as a conference paper at ICLR 2026
M FILTER HEADS IN SMALLER LMS: LLAMA-8B AND GEMMA-9B
We investigate whether filter heads also emerge in smaller models from the same families: Llama-8B and Gemma-9B. We find that filter heads do exist in these smaller models, though they show slightly weaker cross-task generalization compared their larger counterparts (Llama-70B and Gemma-2775 We hypothesize that this degradation likely because of the increased parameter constraints in smaller models individual attention heads are more inclined to serve multiple computational roles, causing the predicate signal to become more entangled with other information. However, these results hint that the filter head mechanism is architecturally general in Llama and Gemma family of LMs, but its abstraction and portability improve with model scale.
Table 15: Causality of filter heads on SelectOne tasks. Using object-type filtering queries we locate 42 heads in BB and 47 heads in Llama-8B. We check if the same heads generalize semantically different predicates like profession identification. Compare with Llama-70B in Table and Gemma-27B in Table 13.
Model
Semantic Type
Gemma-9B (42 Heads)
Llama-8B (39 Heads)
Object
0.856 (138/512)
0.826 (123/512)
Person Profession
0.828 (424/5122
0.756 (387 /512)
Person Nationality
0.387 (198/512)
0.525(269/512)
(+ Priming Prefix)
Landmark in country
0.492 (252/512)
0.492 (252/512)
(+ Priming Prefix)
Evaluated On
(i) Llama-8B
Patched To
SeleetOne (39)-
0.35
0.84
0.64
0.56
0.00
0.14
SelectOne (43)
0.55
0.78
0.40
0.47
0.06
0.11
Cached From
SelectFirst (54)
0.76
0.69
0.00
0.76
0.67
0.69
0.38
0.11
0.04
Trained
0.8
SelectLast last (52)
0.09
Counting (21)
0.02
0.42
0.03
0.04
0.19
0.13
0.62
0.13
0.01
0.05
0.19
0.07
CheckPresenee (13)
0.01
(ii) Gemma-
SelectOne (42)
0.61
0.17
SelectOne-MCC
0:46
0.69
0.35
0.43
0.11
0.01
(146
0.69
0.40
0.44
0.05
0.00
Cached From
SelectFirst (38)
0.01
Trained
Selecti ast (54)
0.92
0.79
0.65
0.52
0.33
0.01
0.85
0:68
0.37
0.82
0.10
0.02
Counting (29)
0.68
0.11
0.25
0122
0.22
1.29
0.0
CheckPresenee (38)
6.59
001
0.18
0.04
0.15
0.03
0.02
021
SetectO
SelectOe-MMQ
SelectFirt
SefeeiLau
Counting
Check Pessmeee
SelestOee
Selevthe-MMD
Salectiins
Selesiaam am
Counting
YouNo
Heads evaluated across tasks
Transferring face across fasks
Figure 24: Generalization of filter heads across different tasks Llama-8B and Gemma-9R 9B. Compare with Figure for 10B and Figure 21 for Gemma-27B.
32

---

Published as a conference paper at ICLR 2026
N USAGE OF LLMS
com Proprietery LLMs services with black-box access such claude. ai and www.ge...gog....oooooooooo.ooo. com were
used as general purpose assist tool, which is allowed as per the ICLR 2026 author guideline. We have used such LLMs to polish some of the writings in this paper. We have also used LLMs to get more items for our dataset and translating the prompts to other languages.
O LIMITATIONS
Our investigation of filtering mechanisms in LLMs, while revealing important insights, has several limitations
Task Coverage. We examined only six filter-reduce tasks, which may not capture the full diversity of filtering strategies employed by LLMs. Even within our six tasks we identified that the filter heads do not show high causality in the CheckPresence task, indicating that the LM uses alternate mechanisms for certain filtering operations. The consistent prompt templates we used enabled us to scale up our controlled experiment setup, but they may have biased us towards specific computational strategies inside the LM. LMs may adapt their filtering approach based on what information is available in ways that our limited task set and prompting strategies cannot fully reveal.
LM Coverage. We identified filter heads in Llama-70B, Llama-8B, Gemma-27B and Gemma-9B models. The fact that we were able to identify similar mechanisms in four models of different sizes, from different families, trained on different datasets echoes the idea of Evolutionary Convergence from Morris (2006): distantly related organisms (e.g. vertebrate and cephalopods) independently evolve similar adaptations (e.g. eyes) in response to similar environmental pressure.
However, such convergence is not guaranteed across all LMs.. Notably, findings from Zhong et al. (2023) suggest that identical architectures trained on different datasets can potentially develop different implementations for the same operation. LMs from other families may develop mechanism that does not make use of such filter heads. Additionally, LMs. is possible that smaller LMs, where parameter constraints might enforce higher head-level superposition, the predicate representation may get entangled with other unrelated stuff in the query states. Therefore, we might not see the distinct head-level causal role of filter heads we get in larger LMs.
restrict our analysis fairly larger
Implementation. Our tasks are designed such that we can determine whether an answer is correct based on what the LM predicts the next token. While curating prompt we ensure that none of the items share a first token with each other. In all of our experiments we also ensure that the answer of the source prompt, destination prompt, and the target answer of the transported predicate different, in order to ensure that patching does not merely copy over the answer. However, this choice of validating with only the first predicted token has potentially restricted from exploring other tasks that have a similar filter-reduce pattern of operation. Most of the causality scores we report in this paper were calculated on single trial with 512 examples sampled randomly. is possible that these scores would change slightly on a different trial.
all
33

---

Published as a conference paper at ICLR 2026
P QUALITATIVE ATTENTION PATTERNS OF FILTER HEADS
In this section provide the aggregated attention pattern of the filter heads in different examples. We use the same filter heads that we identified in Llama-70B for the SelectOne task. The query token is the last token in all cases.
P.I TYPICAL FILTER HEAD BEHAVTOR ON TASKS WE INVESTIGATE
Filter heads exhibit distinct attention pattern: when given a filtering task, they selectively focus their attention on items that satisfy the filtering criteria. Below we visualize this behavior across the six tasks we study in the main text.
Select One (Bulleted)
Select One (MCQ)
Shampoo
Yoga mat
Ash
Projector
Horse
Xylophone
Saxophone
Banana
Racket
Which among these objects mentioned above fruit? Answer
Which among these objects mentioned above is clothing?
Answer:
Select First
Select Last
Options: Birch. Rice cooker, Ash Tomato, Zucchini,
Options: Rabbiit Harmanica, Accortion, Answerj
Cabinet, Sheep
What the first tree from the list above?
Answer
Counting
Check Existence
Items: Blueberry, Church Pear, Apartment, Banana
Items: Bike, Microphone Surfboard, Rose, Toothpaste Do you see electronics in the list above?
How many buildings are in this list?
Answer
Answer
34

---

Answer Published as a conference paper at ICLR 2026
P.2 GENERALIZATION BEYOND OUR CORE TASKS
To check the generalization of filter heads beyond our six core tasks, we examine their attention patterns on additional filtering scenarios that we did not systematically evaluate in the main text. These examples include negated predicates, logical deduction, mathematical reasoning, and identifying odd-one-out items.
Across these diverse scenarios, observe that the same 79 heads identified for the SelectOne task maintain their characteristic filtering behavior: their aggregated attention consistently focuses on items satisfying the given predicate. This supports claim that range of different filtering operations share a common computational circuit implemented by a consistent set of filter heads.
Not Predicate
Logical Deduction
Marie Curie, Albert Einstein, Grover Cleveland
Charles Darwin,
Bob is older than Alice, but younger than Charlie.
Tesla, Issac Newton
Who the youngest?
Which of these people is not scientist? Answert
Answer:
Math Reasoning
Select Odd-One-oe
Which these numbers a multiple 3?
Items: apple, peach, banana, carrot, grape. Which of these items is the odd one out? Answer]
options: 43,
55, 62, 39
P.3
FAILURE CASES FILTER HEADS
We also identify boundary cases where filter heads do exhibit their characteristic attention patterns. These tasks require reasoning with non-semantic information as phonological similarity, letter counting, alphabetical ordering. This suggests that filtering operations requiring non-semantic reasoning may rely on different computational mechanisms. These alternate mechanisms may not involve the filter heads we identified or these heads may play some other role which have not investigated within the scope of this work.
Rhymes With
Number of Letters
Options: brown, fake, that flight.
Options: fake, pat drown row.
Which of these words rhymes with hat?
Which word has exactly 4 letters?
Answer
Answer
First Alphabetically
Last Alphabetically
Options: : bring play, drown, doe set,
Options: cake flee
low
flight.
Which word first alphabetically? Answer
Which word comes alphabetically?
Answer
35

---

Published as a conference paper at ICLR 2026
P.4. APPLICATION
We demonstrate practical utility of filter heads in detecting the presence of certain concepts in the text: identifying false information and detecting sentiment.
We curate paragraphs/Iree- form texts that mix factual and false statements about topic. We break the text into sentences and then append a question asking the LM to identify the false information. When visualize the aggregated attention pattern of the filter heads, identified with the SelectOne task, we observe that the heads focus their attention on the last token of the sentences containing false information. We annotate the last tokens of the sentences with black border to aid visualization.
We can apply the same approach for sentiment analysis in movie/item reviews containing both positive and negative sentences.
Lie Detector
Carl Sagan was an American astronomer, astrophysicist, and science communicator best known his work on the television series Cosmos: Personal Voyage, which popularized science and astronony for wide audience
Sagan made significant contributions to the field of planetary science, particularty in the study
atmos pheres Venus and Jupiter
He was awarded the Pulitzer Prize for his book "The God Delusion" in 1978
Sagan s work continues to inspire scientists and science enthusiasts around the world
Lie Detector
is movie about alien who is stranded Earth He befriends a young bay named Elliott and they form a close bond Together, they embark on a journey help ET return hone
It is considered classic in the romantic comedy genre The film explores thenes of friendship, adventure, and the importance of family ET's iconic appearance and memorable quotes have made it beloved film for audiences of all ages[]
The movie was directed by Christopher Nolan
The movie received critical acclaim for its heartwarming story and special effects
Negative Sentiment Detector
Jaws boring movie about a shark terror izing a town The music iconic builds suspense well The effects still hold up moving today The characters however lack depth and poorly acted The final scene is thrilling and satisfyingg It is classic that still holds power today
Negative Sentiment Detector
bought this keyboard for my IPad a few weeks ago
The keys responsive and have nice tactile fee
The battery Life quite disappointing, lasting only about hours on full charge The build quality is solid and the design is great[] Overall, decent keyboard but has some drawbacks that prevent it from being perfect
36

---

Published as a conference paper at ICLR 2026
Q
FILTER HEADS IN LLAMA-70B
Layer
Head
Indirect Effect
Layer
Head
Indirect Effect
35
19
3,546021
35
5
-0.003418
39
1.353394 45
39
36
-0.003418
35
1.306396 17
30
62
-0.004272
31
1,114380 38
32
48
-0.004395
35
0,611328 40
-0.005005 40
35
0.443115 39
35
36
-0.011719
31
0.340698
-0.013794 19
35
0.281738 18
33
-0.014160 23
29
0.208984 56
33
-0.017456 46
42
0.154541
37
-0.021362 3
28
0.151733 40
29
-0.030029 62
29
0,141235
47
-0.036133 17
36
0.128540 47
31
-0.036743 0
34
0.110474
50
-0.042847
37
0.106079 30
42
-0.043091 30
35
0.104248 23
31
-0.055786 37
31
0.098511 33
37
-0.061890 0
33
0.098145 18
33
21
-0.063599 4
29
0.076416
37
-0.067749
37
0.073608 39
36
-0.068481 40
34
0.068726
49
0..669824 1
35
0.052734 27
35
22
-0.079346
35
0.052734
29
-0.085938 60
28
0.050049 45
49
7
-0.087402
33
0.042725 30
33
43
-0.095558
39
0,038818 35
31
36
-0. 100586
38
0.028809 19
31
32
-0. 109375
38
0.022949 49
49
5
-0.
36
0.022461 44
42
28
-0, 172974
36
0.018066 17
31
43
-0. -0.183960
50
0.017456 34
37
28
-0. 00.188232
36
0.017090 54
49
-0.216553 4
37
0.016479 36
34
-0.247803 1
37
0.011108
38
-0.250610 23
36
0.010376 52
34
-0,252563 45
36
22
0.003052
47
-0.437988 18
32
12
0.001953
39
44
-0,486816
38
51
-0.001221
35
42
-0.770020
45
-0.001953
39
41
-0.843811
37
-0.003296
Table 16: Indirect effect scores for filter heads, sorted in descending order.
37


Reviews:
[{"review_id": "BzfiZy1ptO", "reviewer": "Reviewer_n99r", "summary": "This paper introduces an internals-based explanation for LLM list processing mechanisms. They find evidence for \"filter heads\" in Llama 70B, which encode representations of a predicate relevant for a list processing query (e.g. \"is this fruit\" given the query \"find the fruit\"). They show that causal interventions on these filter heads (e.g. patching query states) result in the expected behaviors.", "questions": "See weaknesses above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper is very well written, and has some really clean experiments\n* The experiments cover their bases quite well (good ablations, good generalization experiments)\n* I think the results from Section 5 are particularly insightful and also just really cool.\n* I think there's still a lot of value in doing this kind of mech interp :)", "weaknesses": "* I would like to see more tasks, especially given the CheckPresence results; it seems like this is a really nice explanation but I'm worried it won't actually generalize / you maybe got lucky with the tasks you chose.\n* Similarly, I don't know what's going on with cross task transfer for SelectFirst/SelectLast. I think this deserves more time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B3iMNDflvp", "reviewer": "Reviewer_Zhy1", "summary": "This paper studies the mechanism used by Large Language Models to solve a variety of simple filtering tasks that involve a list and a predicate. The authors identify the set of heads responsible by learning a sparse binary mask for activation patching. The uncover set of heads generalizes well across similar selection tasks, especially (1) selecting the only matching item from the list, (2) selecting the last matching item, or (3) selecting the first matching item. They also study the impact of placing the question before the list, revealing a different mechanism reminiscent of *eager* evaluation. An interesting application is that the query of the identified heads can be used to obtain a training-free probe.", "questions": "1. How many heads are identified for each of the tasks?\n2. Do LLMs perform better when the question comes before or after the list?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The studied task (selecting an item from a list based on a given predicate) is important and relevant in the current state of LLMs interpretability.\n- The comparison of placing the predicate before and after the options is insightful and sheds light on the ways that causal masking affects the learned mechanisms in LLMs.\n- The mechamism is explained clearly. The paper is easy to understand and figures are good.\n- Experiments are comprehensive.\n- The training-free probe is an interesting and important result.", "weaknesses": "1. The set of heads identified is quite large and the precise role of each head is unclear\n2. The identified heads have low portability to aggregation tasks (presence checking, counting) which suggests a limited relevance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEV3D1Rgq6", "reviewer": "Reviewer_E2mP", "summary": "The authors study how LLMs solve filtering tasks, in which the model must select one element from a list of options. Using differentiable binary masking, they identify the \"filter heads\" responsible for performing this operation. They show that these heads are reasonably general, and can transfer between different tasks, formatting types, etc. They also discovered that if the question is presented before the options, then the model relies on a different mechanism: storing flags for successful matches and aggregating based on them.", "questions": "- L155: What is DCM? This should also be described in the paper.\n- L175: \"We use a sparsity regularizer...\" - Why a sparsity regularizer? What is the exact formulation of this regularizer? What is the target sparsity? Since the goal is to select the minimal set of heads, why not just minimize the logit magnitude, like in [1]?\n- How does the first eq. in eq. 4 work exactly? M was defined earlier as \"source run\", \"destination run\", etc, which are concepts and not mathematical equations, but here it is used as a scalar to maximize over it. Is this the loss at each token? What is the index t? The index of the target token?\n- L210:  The authors mention \"for evaluation we consider last 2 tokens ({“\\Answer”, “:” }) to reduce information leakage\". Can you elaborate on this? Why is there a leakage? Why, considering the last 2 tokens solve it, and what is \"considering\" mathematically?\n- L379: \"queries encode \"what to look for\" (the predicate) and keys bring \"what is there\"\" - this is a somewhat trivial conclusion: this is what the attention was designed to do.\n\n\n[1] Csordas et al, 2021: Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Detailed study: ablations, transfer to other examples, and generalization across tasks/presentation types are all studied\n- Based on causal analysis\n- Mostly clear writing", "weaknesses": "- There are some unclear parts, see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I50045XiMp", "reviewer": "Reviewer_jAKf", "summary": "This paper investigates the mechanisms by which Large Language Models (LLMs) process lists, identifying a small set of \"filter heads\" in the middle layers that implement a general, abstract filtering operation akin to functional programming1. Using causal mediation analysis, the authors demonstrate that these heads encode a filtering predicate (e.g., \"is this a fruit?\") into their query ($q$) state at the final answer token. This query state then interacts with the key ($k$) states, which carry the semantic properties of the list items, to select the correct items. This predicate representation is shown to be portable and general, as it can be extracted and applied to different lists, formats, and languages to trigger the same filtering operation. The paper also discovers a complementary \"eager evaluation\" strategy: when the question (predicate) is presented before the list, the model stores an is_match flag directly in the items' representations, bypassing the \"lazy\" filter head mechanism.", "questions": "Please refer to Weaknesses part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The work isolates a very specific but important phenomenon—list processing—and shows that LLMs exhibit strategy shifts with longer contexts, a behavior previously undocumented. \n- The paper connects internal mechanisms (representation compression) to observable generalization failures in a logical, evidence-based way, through causal interventions (activation patching): patching $q$, swapping $k$, and manipulating \"flags\" provide a very convincing, mechanistic explanation.\n- Synthetic list tasks make it easy to disentangle algorithmic difficulty from length effects and the discovered behavior likely extends beyond toy tasks to real-world list-like inputs (e.g., tables, document parsing, code).", "weaknesses": "- While controlled tasks are valuable for analysis, it remains unclear whether similar behavior holds for natural language or multimodal data (although it is potentially fit for real-world data, the corresponding experiments are in lack).\n- Experiments appear limited to LLaMA-70B and Gemma-27B, which are quite old models (in terms of the development of LLMs). How about the results with recent models (e.g., LLaMA3, Qwen3, ...)? Besides, does this mechanism hold for models of different size? (LLaMA-7B, LLaMA-13B and LLaMA-70B)\n- The filter head mechanism explains the Select* tasks well, but the authors note it has low causality for Counting and CheckPresence tasks. This suggests these tasks (which are still filter-reduce operations) rely on a different, un-investigated mechanism. The paper's core claim is thus limited to selection-based filtering.\n- The compared baselines (e.g., Function Vectors, Induction Heads): how did the authors implement them? The details are not clear. The reviewer thinks some approaches (e.g., Function Vectors) mentioned here are not fit for the tasks studied in the paper inherently.\n- The logic of the paper is clear. However some acronyms might be quite improper (e.g., line 155, what is \"DCM\", I did not find the full name of DCM throughout the paper...). The clear definitions for tasks like SelectOne, SelectFrist are not in the main text, somehow confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)