Title: Head Pursuit: Probing Attention Specialization in Multimodal Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: attention heads

matching pursuit

interpretability

vision-language models
Abstract: Language and vision-language models have shown impressive performance across a wide range of tasks, but their internal mechanisms remain only partly understood. In this work, we study how individual attention heads in text-generative models specialize in specific semantic or visual attributes. Building on an established interpretability method, we reinterpret the practice of probing intermediate activations with the final decoding layer through the lens of signal processing. This lets us analyze multiple samples in a principled way and rank attention heads based on their relevance to target concepts. Our results show consistent patterns of specialization at the head level across both unimodal and multimodal transformers. Remarkably, we find that editing as few as 1% of the heads, selected using our method, can reliably suppress or enhance targeted concepts in the model output. We validate our approach on language tasks such as question answering and toxicity mitigation, as well as vision-language tasks including image classification and captioning. Our findings highlight an interpretable and controllable structure within attention layers, offering simple tools for understanding and editing large-scale generative models.

Head Pursuit: Probing Attention Specialization in Multimodal Transformers

Abstract

Language and vision-language models have shown impressive performance across a wide range of tasks, but their internal mechanisms remain only partly understood. In this work, we study how individual attention heads in text-generative models specialize in specific semantic or visual attributes. Building on an established interpretability method, we reinterpret the practice of probing intermediate activations with the final decoding layer through the lens of signal processing. This lets us analyze multiple samples in a principled way and rank attention heads based on their relevance to target concepts. Our results show consistent patterns of specialization at the head level across both unimodal and multimodal transformers. Remarkably, we find that editing as few as 1% of the heads, selected using our method, can reliably suppress or enhance targeted concepts in the model output. We validate our approach on language tasks such as question answering and toxicity mitigation, as well as vision-language tasks including image classification and captioning. Our findings highlight an interpretable and controllable structure within attention layers, offering simple tools for understanding and editing large-scale generative models.

1 Introduction

Large-scale generative models, including both language and vision-language transformers, have achieved remarkable performance on a wide spectrum of tasks, from open-ended text generation [1] to image captioning and visual question answering [2-5]. Despite these successes, the internal mechanisms by which these models organize and represent knowledge are still incompletely understood. In particular, the role of individual components, such as attention heads, in mediating specific aspects of generation has been the subject of increasing interest for both interpretability and control [6-8]. Previous studies have shown that attention heads in large language models (LLMs) often exhibit emergent roles, such as syntax tracking or copy behavior [9-11]. Interpretability tools such as the Logit Lens [12] and its extensions [13, 14] have provided strategies for inspecting intermediate model representations, revealing rich semantic information latent in hidden states. However, these techniques are typically applied heuristically and focus on individual examples, making it difficult to generalize findings across multiple samples or quantify the importance of specific model components in shaping the model's output.

In this work, we take a more principled approach to analyzing the specialization of attention heads in generative transformers. The foundation of our approach is to reinterpret existing interpretability

blue

green

yellow

black

🌈

laugh

smile

sad

serious

🤣

1

seven

number

digit

🔢

'Describe this

image'

+

A woman walking down

the street with a hat and

a cup of co

ff

ee.

A woman in a purple

hat and sunglasses

carrying a co

ff

ee cup.

A woman in purple and

white sunglasses, a purple

hat, and a black jacket.

+ …

+

+

+

α

= 1

α

> 1

Score heads

w.r.t. property

Choose

property

Vision-Language Model

+ …

+

+ …

+

+ …

+

+

Head

1

Head

2

Head

H

+

+

+

…

Layer

1

Layer

2

Layer

3

Layer

L

…+

=

Model

Output

α

= - 1

Rescale top-k

heads

①

②

③

tools through the lens of sparse signal recovery. Specifically, we revisit a variant of Matching Pursuit (MP) [15], a classical greedy algorithm to approximate high-dimensional signals with sparse linear combinations of basis elements, and bridge it with recent interpretability techniques. By applying MP to the hidden states of text-generative models, we propose a way to identify a small set of attention heads that most strongly influence the model's capability to generate text within specific conceptual domains (e.g., colors or numbers ). This approach provides a mathematically grounded strategy to decompose model behavior into a small set of interpretable elements, contrasting with prior heuristic techniques, thus enabling both quantitative analysis and targeted interventions on model mechanisms.

Using this framework, we reveal consistent head specialization patterns across large unimodal and multimodal pre-trained models. We find that certain heads are reliably responsible for generating semantically coherent groups of tokens, such as names, colors, or sentiment-bearing words. Moreover, we find that intervening on just a small fraction of these concept-specific heads can significantly affect the model's output, enabling suppression and enhancement of targeted content. These results suggest that attention layers contain a highly interpretable and manipulable linear structure, consistent with evidence that large-scale models represent high-level concepts in approximately linear residual subspaces [16-19]. Such structure opens new directions for understanding and controlling model behavior without additional training. In particular, the emergence of structured, concept-aligned subspaces indicates that a small number of units can exert broad semantic control over generation, offering a principled alternative to heuristic prompt-based interventions and highlighting the potential of sparse, head-level manipulation as a general tool for studying and shaping internal model behavior.

Our contributions can be summarized as follows:

We introduce a strategy to frame Matching Pursuit (MP), an established sparse recovery algorithm, in the context of the interpretability of generative language models, establishing its connection to standard interpretability tools such as the Logit Lens;

We apply MP to reveal that attention heads of LLMs often specialize in the generation of tokens belonging to narrow semantic areas, and propose an approach to identify the group of heads most relevant for a conceptual domain;

We demonstrate that head specialization opens up a way to manipulate model behavior. Both in language and multimodal tasks, negating relevant heads causes targeted degradation in task performance, while enhancing them promotes the generation of specific attributes.

2 Related work

Recent research on Transformer architectures has investigated the functional roles and specialization of attention heads. In language models, most heads appear redundant, with pruning studies showing that many can be removed with minimal loss in performance on NLP tasks. Only a few contribute significantly to linguistic functions such as encoding positional information, syntactic structure, or attending to rare words [9]. Some heads have also been linked to eliciting factual knowledge [6], promoting in-context induction [11], or suppressing lexical repetition [20]. Other analyses have examined attention heads through their learned weights [21], providing insight into how information is routed within the model. Further work has explored the localization and manipulation of MLP and residual representations. Early mechanistic interpretability studies [22, 23] showed that factual associations are encoded primarily in mid-layer MLPs and can be modified by targeted intervention on MLP weights.

In vision-language models, similar specialization patterns have been observed in the visual encoder of CLIP-like architectures by leveraging visual-textual alignment to decompose heads over sentence encodings [24, 25]. Beyond contrastive models, recent work has applied dictionary learning to generative VLMs to extract human-interpretable concepts from latent activations [26, 27], building on earlier efforts in CNN interpretability [28]. A parallel line of research adapts mechanistic interpretability tools from language models to the multimodal setting. Representative works include [29] and [30], which investigate information transfer mechanisms in multimodal transformers, and [31], which extends the Logit Lens [12] to the analysis of visual token representations.

Closely related to our approach are methods derived from the Logit Lens (LL) [12], which interpret internal representations by projecting them into the output space. The Attention Lens [14] extends LL to individual attention heads but requires training a separate linear probe for each head, following the Tuned Lens framework [13], making it computationally expensive and difficult to scale to billionparameter models. In contrast, our method identifies concept-relevant heads by directly projecting their outputs onto the model's unembedding matrix, without requiring any additional training. Another related method [32] also employs the unembedding matrix for interpretability, but within a gradientbased saliency framework that attributes model predictions to influential input tokens. While their approach is task-dependent and gradient-based, ours is task-agnostic and operates purely on model representations, and the two could be combined to focus our analysis on salient tokens.

In summary, our work provides a unifying perspective across these directions by investigating head specialization in generative language and vision-language models through sparse decomposition over a fixed dictionary of interpretable directions. Rather than learning the dictionary from activations, we assume a known semantic basis, typically derived from the unembedding matrix, and use sparse recovery to identify heads whose outputs align with specific attributes.

3 Pursuing specialized attention heads

We start our investigation by exploring whether individual attention heads of generative LLMs specialize in interpretable functions. To isolate the contribution of each head, we use a residual stream decomposition approach. This allows us to assess how each attention head contributes to the residual stream at a head-level granularity. Specifically, following [33], we model the output written by each head into the residual stream as a matrix H h,l ∈ R n,d , where n is the number of samples in the dataset and d is the internal dimensionality of the transformer.

Motivated by the growing body of literature that uses latent decompositions for interpretability, especially in vision-language models [24-27], our aim is to identify sparse and interpretable directions for each attention head H h,l that maximally explain its variance on a given dataset. Concretely, we seek a sparse representation of H h,l using directions drawn from a fixed dictionary of interpretable vectors, rather than an unconstrained continuous space, to ensure that the resulting components are meaningful and grounded in known semantic structures.

As a dictionary, we adopt the unembedding matrix of the language model D ∈ R v,d , as it naturally contains directions that are aligned with semantically meaningful outputs, allowing us to ground latent structure in human-interpretable terms. In fact, every row of this matrix is a d -dimensional vector that effectively represents in the latent space a token that can be decoded into natural language.

We then construct an approximation of each head representation using directions from our dictionary (i.e., the unembedding matrix) via a classical sparse coding algorithm: Simultaneous Orthogonal Matching Pursuit (SOMP) [34] (see Appendix A). SOMP is a multi-sample extension of Orthogonal Matching Pursuit [35], itself a refinement of the original Matching Pursuit algorithm [15]. Rather than analyzing each sample independently, SOMP jointly considers all samples in a given dataset and selects the dictionary directions that are most informative across the representation.

Formally, given a head activation matrix H ∈ R n,d and a dictionary D ∈ R v,d , SOMP aims to iteratively construct a column-sparse coefficient matrix W ∗ ∈ R n,v such that:

At each iteration t , the algorithm selects the dictionary atom (i.e., a row of D ) that maximally correlates with the head residuals across all samples:

Here, the head residual matrix R t ∈ R n,d is defined as the difference between the original signal and its reconstruction at step t : R t = H -H t r . The selected index p t is added to the support set S t +1 , and the dictionary is refit by solving a least-squares problem restricted to the current support:

The reconstruction is updated as H t +1 r = W t D [ S t +1 ] , and the residuals are recomputed accordingly. This iterative process continues until a predefined sparsity level is reached. The resulting decomposition expresses each head's output using a sparse set of semantically meaningful dictionary atoms, yielding an interpretable approximation of its behavior.

Importantly, we note a conceptual connection between our reinterpretation of SOMP and the Logit Lens (LL) [12], a tool widely used in mechanistic interpretability to probe internal representations of transformer models. Similarly to the method just described, LL works by projecting a single residual stream vector onto the unembedding directions to approximate the output logits of the model at intermediate layers. This is equivalent to performing a single step of Matching Pursuit on an individual example. Our SOMP-based method generalizes this idea in two key ways: it operates on multiple examples simultaneously, and it selects multiple dictionary directions, each capturing distinct components of the signal. This leads to a more robust and semantically structured characterization of the attention head's functional role.

In Table 1, we report some examples of specialized attention heads, obtained by applying SOMP to Mistral-7B attention heads, prompted by questions from the TriviaQA dataset [36]. Before applying SOMP, the tokens from the prompt were aggregated by averaging. As we show in Table 6, a direct application of LL in this setting results in noisier and highly redundant explanations.

Besides returning lists of latent directions and associated natural language tokens that better characterize each head, SOMP produces a reconstruction of the head representation in the space spanned by those vectors. Building on this insight, we propose a method to automatically identify the heads most relevant for a target attribute. Given a list of words related to the chosen semantic area, one can restrict the unembedding matrix to the rows associated to these tokens and apply SOMP on this concept-specific dictionary. Then, the fraction of head variance explained by SOMP in this setting can be considered as a measure of specialization of the head, allowing us to rank and select heads by their relevance with respect to the target concept.

4 Controlling language generation through specialized heads

We now evaluate how the specialization of attention heads can be leveraged to apply domain-specific targeted interventions to model behavior, effectively validating our selection. One way to do so is to disrupt the information flow from a selected subset of heads to the residual stream during the forward pass. Concretely, we apply this intervention by inverting the sign of the head representations. It is worth noting that this intervention not only affects the direct contribution of the head to the residual stream, but also the indirect contribution of its information content to subsequent layers.

The key preliminary step is to identify relevant and specialized heads. To accomplish this, we apply SOMP (see Section 3) over a restricted unembedding dictionary, filtered to include only a set of tokens associated with the target property. These tokens can be selected using various strategies, such as user-defined word lists, class names, or keywords generated by an external LLM. Then, we rank attention heads by the proportion of variance in the data explained by SOMP (in our experiments, typically using 50 iterations) and intervene on the topk ranked heads.

In all of our experiments, we include a random control condition to verify the specificity of our findings. This control involves intervening on a randomly selected set of attention heads that matches the original set in both size and layer distribution but is entirely disjoint from it. For these purposes, to ensure a fair comparison, we explicitly avoid selecting heads previously identified as specialized when constructing the control set. In all experiments, we report such random control results over 10 independently sampled sets of heads.

4.1 Question answering

Experimental setting We consider a generative LLM, Mistral-7B [37], and evaluate it on textual prompts from the TriviaQA [36] question answering dataset. Our goal is to identify attention heads specialized in generating country names, a target attribute motivated by their relative abundance in the dataset: despite their specificity, country names account for over 6% of the answers in the test split. For targeting the countries concept, we restrict the tokens in our dictionary (unembedding matrix) to those corresponding to names of countries, and apply our Matching Pursuit based method to select specialized heads. As an additional baseline, we report results obtained by inverting heads selected with a simple adaptation of the Logit Lens. Specifically, we score each head with the mean logit assigned to country-related tokens by LL, and select topk heads as in our method. Importantly, head representations are computed using questions from the training data, which is strictly disjoint from the data used in evaluation. Model performance on TriviaQA is assessed using the standard F1 score, which accounts for partial overlaps between the predicted and ground-truth answers.

Result analysis We report the results of our intervention on subsets of specialized heads in Figure 2, varying the number of selected heads. Performance on the target attribute (solid blue line) noticeably degrades when the signs of 8 heads ( 0 . 8% of the total) or more are inverted. Notably, performance on the remaining examples (solid orange line) declines more gradually, suggesting that these specialized heads have a targeted effect. Since the semantic domains of the questions answered by a country name and the remaining part of TriviaQA are not disentangled, it is expected that intervening on the selected heads also has a (lower) impact on the remaining examples (orange line). Control experiments using random heads (faded lines) show no significant impact on performance, confirming the specificity of the identified heads. Instead, heads selected by the Logit Lens (dashed lines) are relevant to the question-answering tasks but not specific to the targeted concept, as they degrade performance equally within and outside the targeted domain.

2

4

8

16

32

k (#heads)

0.2

0.4

0.6

0.8

1.0

Normalized F1 score

Country:

Other:

SOMP

SOMP

LL

LL

Random

Random

Overall, the analysis suggests that our method correctly identifies heads relevant to country-related examples, as intervention disproportionately impacts target performance, while control heads correctly induce limited effect and Logit Lens selects relevant but non-specific heads.

4.2 Mitigation of toxic content

Experimental setting Now we evaluate our method in a more realistic and less controlled scenario, where a complete list of target keywords is not available. Instead, we are given only a limited and incomplete list of words meant to represent a topic or concept. In this setting, we focus on toxicity mitigation: specifically, reducing the occurrence of offensive content in text generated by Mistral. To do this, we identify a subset of toxic heads within the model and intervene on them. We consider two datasets, RealToxicityPrompts (RTP) [38], which contains naturally occurring Web prompts, and Thoroughly Engineered Toxicity (TET) [39], a benchmark with carefully constructed test cases, both of which are designed to elicit harmful responses from LLMs. For both datasets, we extract toxic words from Mistral's responses using Llama3.3 [40], with the prompt reported in Appendix E.1, and use those words to identify and invert toxic heads. To evaluate the effectiveness of our intervention, we use two complementary metrics that quantify the toxicity of Mistral's responses: one semantic and one lexical. For the semantic evaluation, we employ a RoBERTa-based toxicity classifier from [41], trained to detect toxic content in text. For the lexical evaluation, we measure the frequency of a held-out subset of toxic words, not used for head selection.

Table 2: Normalized count of toxic generations after intervention. Lower values indicate better mitigation. Targeted heads reduce toxicity, while heads selected using LL have a weaker impact and random heads maintain or increase toxicity. For random baselines, only the median is shown. Full results including the interquartile ranges are reported in the Appendix (Table 7).

Result analysis The results we obtain by inverting the sign of toxic head activations are displayed in Table 2, for 8 , 16 and 32 heads. In both RTP and TET, intervening on such heads noticeably reduces the amount of generations deemed toxic by the classifier, while intervening on heads identified by the Logit Lens has a generally weaker impact on toxicity. Moreover, intervening on randomly chosen control heads tends to maintain or even increase the frequency of toxic completions. Analogous results are reported in the Appendix (Table 8) for the lexical metric, showing that the intervention reduces the frequency of toxic keywords, even if they were not used for the head selection.

We showed that it is possible to intervene on a small subset of heads to make generated text less toxic. Notably, we showed that our approach can extrapolate a broad and consistent semantic area from a restricted list of keywords.

5 Targeted control in visual understanding

We now move to evaluating the extent and implications of head specialization in the LLM backbones within generative Vision-Language models (VLMs). These models are usually built by fine-tuning a pre-trained LLM on multimodal tasks, such as visual question answering or image captioning, using visual tokens coming from a pre-trained vision encoder as contextual information [3]. In line with recent works [31] that have successfully applied the Logit Lens to visual tokens of LLaVA, a prominent example of VLM, we investigate head specialization by applying our MP-based analysis on the head representations of image patches, averaged over tokens. We frame our experiments in two different task scenarios: image classification and image captioning.

5.1 Image classification

Experimental setting For this experiment, we benchmark LLaVA-NeXT-7B [4] (from now on just LLaVA for short) on a range of image classification datasets, including: MNIST [42], SVHN [43],

GTSRB [44], EuroSAT [45], RESISC45 [46] and DTD [47]. For each dataset, we begin by selecting the set of k most relevant heads. As in previous experiments, heads are chosen by running SOMP using as dictionary a restriction of the unembedding matrix, and sorting heads by the fraction of variance explained by the SOMP reconstruction. We consider two settings: one task-conditioned, in which we restrict the unembedding matrix to the tokens corresponding to class names; and one completely task-agnostic, in which we consider a set of keywords extracted by an external VLM. Results for the latter are reported in the Appendix (Figure 9), and we provide details regarding the prompt in Appendix E.2. In this experiment, we prompt the model to classify the image, and evaluate the generated output in terms of exact match with the ground truth class label.

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

0.0

0.2

0.4

0.6

0.8

1.0

Normalized Accuracy

Selected Heads:

16 (SOMP)

32 (SOMP)

16 (Random)

32 (Random)

Result analysis We report the classification results in Figure 3, normalized for each dataset with respect to the accuracy obtained by LLaVA when no intervention is applied to its forward pass. For all datasets, inverting the top 32 heads identified by our method is sufficient to significantly disrupt the classification performance, while inverting 32 random heads at equivalent layers has substantially lower to no impact on performance. At k = 16 the picture is similar with the exception of DTD, whose performance is weakly affected, hinting at higher head redundancy on this task. In Figure 4, we analyze the interaction between head choices for different datasets. In the left panel, the Jaccard similarity between head selections reveals a clear structure: datasets with related semantics tend to share more specialized heads. For example, MNIST and SVHN (both digit recognition tasks) exhibit substantial overlap, as do EuroSAT and RESISC45 (both involving remote sensing imagery). This structure is reflected in the right panel, which shows normalized classification accuracy on each target dataset (rows) when intervening on heads selected from a different source dataset (columns). Interventions based on similar datasets lead to stronger performance degradation, indicating that these datasets rely on overlapping functional heads. We also observe significant drops in GTSRB (traffic sign recognition) performance when intervening with heads selected from MNIST or SVHN. Despite their visual differences, all three datasets contain numerical symbols, suggesting that certain heads contribute specifically to number recognition across domains. Results for k = 8 are reported in Appendix D.3, along with Logit Lens baselines. Additional experiments on a selection of different VLMs, including LLaVA-NeXT-13B, Gemma3-12B [48] and Qwen2.5-VL-7B [49], confirm trends observed here on LLaVA-NeXT-7B, as reported in Appendix D.4.

Summing up, intervening on a small set of attention heads selected via SOMP significantly disrupts classification performance across diverse datasets, confirming head-level specialization in LLaV A. Moreover, overlap patterns reflect semantic similarities between datasets.

5.2 Image captioning

Experimental setting We consider the Flickr30k dataset [50], and evaluate the possibility of promoting or reducing the presence of words belonging to specific semantic areas in the captions

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

1.00

0.68

0.00

0.00

0.03

0.14

0.68

1.00

0.00

0.00

0.07

0.14

0.00

0.00

1.00

0.39

0.03

0.10

0.00

0.00

0.39

1.00

0.03

0.14

0.03

0.07

0.03

0.03

1.00

0.07

0.14

0.14

0.10

0.14

0.07

1.00

Top-16 Head Overlap

0.0

0.2

0.4

0.6

0.8

1.0

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

Source Dataset

MNIST

SVHN

RESISC45 EuroSAT

DTD

GTSRB

Target Dataset

0.28

0.12

1.00

0.99

1.00

0.94

0.43

0.25

1.00

1.00

1.02

0.97

0.90

0.96

0.64

0.62

0.94

0.83

1.05

1.00

0.86

0.82

1.01

0.93

0.92

0.95

0.86

1.01

0.94

0.96

0.53

0.40

0.61

0.67

0.72

0.01

Accuracy under intervention (16 heads)

0.2

0.4

0.6

0.8

1.0

generated by LLaVA (we evaluate other VLMs on the same task in Appendix D.6). In this experiment, we consider two opposite intervention directions: one inhibitory, as in previous experiments, and one enhancing. In the former case, the objective is to make the model produce meaningful captions that do not contain the target property (e.g., colors ), while in the latter the aim becomes to enhance the target property, while preserving the model's capabilities in generating meaningful descriptions. The two setups reduce to rescaling selected heads by a coefficient that is α = -1 in the negative case and α > 1 in the positive case. Heads are selected using SOMP on a dictionary of tokens corresponding to lists of keywords regarding colors , sentiments and quantity , with the first two adopted from [27] and the latter manually curated. Evaluation is carried out by measuring the effectiveness of the intervention as the average number of target concept keywords present in the captions. The semantic consistency of the generated captions with the ground truths is measured using the CIDEr metric [51].

Inhibitory intervention The first setup we evaluate is analogous to the previous examples on text generation and image classification tasks. We inhibit the generation of tokens in a certain semantic domain by inverting the signs of a few carefully selected attention heads. The results of this analysis are reported in Figure 5, for the three sets of attributes colors (left), sentiments (center) and quantity (right). In all cases, our intervention is able, with as few as 16 heads, to almost completely remove attribute-related keywords from the output captions, while keeping the overall caption quality almost on par with the original, as witnessed by the CIDEr score, which always exceeds 80% of the original. CIDEr results are reported in Appendix D.5 along with a comparison with heads selected using the Logit Lens.

8

16

32

k

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Normalized Property Count

Colors

8

16

32

k

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Sentiments

8

16

32

k

0.0

0.2

0.4

0.6

0.8

1.0

Quantity

SOMP (top-k)

Random

Flickr30k examples:

Original

A small dachshund wearing a

pink

sweater.

A young woman with long brown hair and a smile.

Intervention

Colors

inhibition (

k

= 16

, α

=

-

1

)

Sentiments

inhibition (

k

= 16

, α

=

-

1

)

Output

A small dachshund wearing a sweater.

Girl with long brown hair blowing in the wind.

Intervention

Colors

enhancement (

k

= 16

, α

= 5

)

Sentiments

enhancement (

k

= 16

, α

= 5

)

Output

A black

and

dog wearing a

pink sweater.

A happy girl with long hair and a big smile.

Enhancing intervention Across different tasks and data modalities, we have seen that intervening on selected head activations by inverting their sign is highly effective in disrupting the generation of a target attribute. We now take a different perspective and evaluate whether amplifying those specialized attention heads can incentivize the generation of the target concept. We do so by multiplying the activations of chosen heads by a coefficient α > 1 : we evaluate various choices of α in the Appendix in Figure 10, and choose α = 5 for our experiments as it guarantees a reasonable trade-off between caption quality and attribute enhancement. Our results, on the same attributes of the previous experiment ( colors , sentiments and quantity ), are reported in Figure 6. As in the previous case, our intervention affects the overall caption quality only marginally (see Table 13, Appendix D.5), while the presence of target concepts increases by more than 60% in all three cases with 32 heads. Captions generated for two sample images after applying our interventions in both directions (inhibitory and enhancing) are reported in Table 3.

8

16

32

k

0.0

0.5

1.0

1.5

2.0

2.5

Normalized Property Count

Colors

8

16

32

k

0.0

1.0

2.0

3.0

4.0

5.0

Sentiments

8

16

32

k

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Quantity

SOMP (top-k)

Random

Overall, these results show that head-level specialization can be leveraged to control the prevalence of words belonging to a target semantic area in generated image captions. Notably, this result holds for both inhibiting and enhancing the target concept.

Computational resources To perform our experiments we employed pre-trained model checkpoints implemented in the HuggingFace transformers library [52]. Detailed information on such resources is provided in Appendix C. All the experiments were executed on a single NVIDIA H100 GPU equipped with 80GB VRAM. Our code is available at https://github.com/lorenzobasile/ HeadPursuit .

6 Discussion

In this work, we investigate the specialization of attention heads in large generative models through a sparse, interpretable decomposition of their outputs. Using Simultaneous Orthogonal Matching Pursuit (SOMP) over the model's unembedding space, we identify directions aligned with semantically meaningful attributes and use them to recover sets of specialized heads across a variety of tasks and modalities. Our approach offers a multi-sample generalization of the Logit Lens, allowing us to move beyond single-token analysis toward more stable, dataset-level structures. We show that the selected heads can be ranked by their explained variance and that intervening on a small number of them produces targeted changes in generation. These findings hold across text and vision-language settings, supporting the utility of head-level analysis and intervention for model understanding and control.

Limitations While our method provides a scalable and interpretable approach to identify influential attention heads, it has several limitations. First, SOMP imposes a linearity assumption that may not fully capture the nonlinear structure of head representations. Second, the quality and coverage of the semantic dictionary play a critical role in determining the reliability of the recovered heads: incomplete or noisy keyword lists can bias the selection toward spurious or underrepresented concepts. Finally, our intervention mechanism is deliberately simple, relying on global rescaling or inversion of head contributions rather than more fine-grained steering strategies.

Future work Potential future developments could include exploring more selective and fine-grained interventions, such as rescaling heads only at specific input positions or modalities. For example, in a VLM, one could disable heads only over image patch tokens while preserving text understanding, enabling targeted degradation or control. Another promising direction is to adapt the technique for multimodal-output settings, such as image generation with VLMs. In this case, selectively enhancing or suppressing heads during decoding could provide a mechanism for steering generated images toward or away from particular semantic attributes, offering a controllable alternative to prompt engineering or fine-tuning.

Broader impact This work contributes to a growing body of research aimed at making LLMs and VLMs more interpretable and controllable. While the ability to manipulate specific aspects of model behavior can aid transparency and alignment, it may also be used to conceal or amplify certain content in ways that raise ethical concerns. We encourage downstream users of such techniques to carefully evaluate their applications, especially in sensitive domains.

Acknowledgments

The authors acknowledge the Area Science Park supercomputing platform ORFEO made available for conducting the research reported in this paper, and the technical support of the Laboratory of Data Engineering staff. LB, DD and AC were supported by the project 'Supporto alla diagnosi di malattie rare tramite l'intelligenza artificiale" CUP: F53C22001770002 and 'Valutazione automatica delle immagini diagnostiche tramite l'intelligenza artificiale", CUP: F53C22001780002. LB was supported by the European Union - NextGenerationEU within the project PNRR 'Finanziamento di progetti presentati da giovani ricercatori" - Mission 4 Component 2 Investment 1.2, CUP: J93C25000440001. AC was supported by the European Union - NextGenerationEU within the project PNRR 'PRP@CERIC" IR0000028 - Mission 4 Component 2 Investment 3.1 Action 3.1.1.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716-23736, 2022.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36:34892-34916, 2023.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. Competition of mechanisms: Tracing how language models handle facts and counterfactuals. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8420-8436, 2024.

Francesco Ortu, Zhijing Jin, Diego Doimo, and Alberto Cazzaniga. When seeing overrides knowing: Disentangling knowledge conflicts in vision-language models. In Mechanistic Interpretability Workshop at NeurIPS 2025 .

Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Talking heads: Understanding inter-layer communication in transformer language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 5797-5808, 2019.

Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in Neural Information Processing Systems , 32, 2019.

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895 , 2022.

nostalgebraist. interpreting gpt: the logit lens. LessWrong , 2020.

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112 , 2023.

Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism. arXiv preprint arXiv:2310.16270 , 2023.

Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions on signal processing , 41(12):3397-3415, 1993.

Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. In Forty-first International Conference on Machine Learning , 2024.

Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations in large language models. In Forty-first International Conference on Machine Learning , 2024.

Achyuta Rajaram, Sarah Schwettmann, Jacob Andreas, and Arthur Conmy. Line of sight: On linear representations in vllms. arXiv preprint arXiv:2506.04706 , 2025.

Isabel Papadimitriou, Huangyuan Su, Thomas Fel, Sham M. Kakade, and Stephanie Gil. Interpreting the linear structure of vision-language model embedding spaces. In Second Conference on Language Modeling , 2025.

Callum Stuart McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding a motif in language model attention heads. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen, editors, Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages 337-363, Miami, Florida, US, November 2024. Association for Computational Linguistics.

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 16124-16170, 2023.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems , 35:17359-17372, 2022.

Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 12216-12235, 2023.

Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting CLIP's image representation via text-based decomposition. In The Twelfth International Conference on Learning Representations , 2024.

Lorenzo Basile, Valentino Maiorca, Luca Bortolussi, Emanuele Rodolà, and Francesco Locatello. Residual transformer alignment with spectral decomposition. Transactions on Machine Learning Research , 2025.

Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. A concept-based explainability framework for large multimodal models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Pegah Khayatan, Mustafa Shukor, Jayneel Parekh, Arnaud Dapogny, and Matthieu Cord. Analyzing finetuning representation shift for multimodal llms steering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 2206-2216, October 2025.

Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2711-2721, 2023.

Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Massiceti. Understanding information storage and transfer in multi-modal large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Alessandro Serra, Francesco Ortu, Emanuele Panizon, Lucrezia Valeriani, Lorenzo Basile, Alessio Ansuini, Diego Doimo, and Alberto Cazzaniga. The narrow gate: Localized image-text communication in native multimodal models. In The Thirty-ninth Annual Conference on Neural Information Processing Systems , 2025.

Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing in vision-language models. In The Thirteenth International Conference on Learning Representations , 2025.

Elizabeth Mary Hou and Gregory David Castanon. Decoding layer saliency in language transformers. In International Conference on Machine Learning , pages 13285-13308. PMLR, 2023.

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Joel A Tropp, Anna C Gilbert, and Martin J Strauss. Algorithms for simultaneous sparse approximation. part i: Greedy pursuit. Signal processing , 86(3):572-588, 2006.

Yagyensh Chandra Pati, Ramin Rezaiifar, and Perinkulam Sambamurthy Krishnaprasad. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar conference on signals, systems and computers , pages 40-44. IEEE, 1993.

Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, 2017.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3356-3369, 2020.

Tinh Luong, Thanh-Thien Le, Linh Ngo, and Thien Nguyen. Realistic evaluation of toxicity in large language models. In Findings of the Association for Computational Linguistics ACL 2024 , pages 1038-1047, 2024.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024.

Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. ParaDetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 6804-6818, Dublin, Ireland, May 2022. Association for Computational Linguistics.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning , volume 2011, page 4. Granada, 2011.

Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 2011 international joint conference on neural networks , pages 1453-1460. IEEE, 2011.

Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217-2226, 2019.

Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE , 105(10):1865-1883, 2017.

Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3606-3613, 2014.

Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 , 2025.

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2:67-78, 2014.

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4566-4575, 2015.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: Stateof-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages 38-45, 2020.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract summarizes the main results of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The main limitations are described in a dedicated paragraph of the final section (Discussion).

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We do not include any theoretical result.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All our experiments are reproducible based on the paper. In addition, we provide our code in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Code is provided in the supplementary material, and all models and datasets we use are publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper does not involve any training, and we specify all hyperparameters relevant to our analysis.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report error bars for our randomized control experiments on head selection.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specify the computational resources we employed in a dedicated paragraph at the end of section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: To the best of our knowledge and understanding, none of the results presented in the paper violate the code of ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the potential societal impacts of our work in a dedicated paragraph in the last section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release data nor models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Authors of all datasets and existing code we use are properly referenced in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: We employ LLMs to obtain property keywords in some of our experiments. Details on models, prompts and setups are provided.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Simultaneous Orthogonal Matching Pursuit

Below, we provide the pseudocode for the Simultaneous Orthogonal Matching Pursuit (SOMP) algorithm [34].

B Prompts

We summarize in Table 4 the exact text templates used to construct the prompts for each evaluated task. These templates define the structure of model inputs across all datasets and settings considered in our experiments.

Task objects used in classification prompts:

digit

RESISC45, EuroSAT: remote sensing scene

C Model details

All models we employ are taken pre-trained from the HuggingFace transformers [52] library. We report in Table 5 the full list of pre-trained models we employed in this work, associated with the name of the corresponding checkpoint in the library.

D Additional results

In this section, we provide additional results that complement our analyses in the main paper.

D.1 Logit Lens

In Table 6, we report the 5 most relevant tokens identified by the Logit Lens (LL) [12] for the four attention heads of Mistral-7B analyzed in Table 1. By design, LL can only be applied to individual samples, not to an entire dataset. We aggregate over mutiple samples by storing the 5 tokens with highest logits for each sample, and then taking the 5 most frequent tokens overall.

D.2 Toxicity mitigation

In Table 7 we report complete results for the toxicity mitigation experiment of Section 4.2, while Table 8 reports the results obtained using the lexical metric (toxic word frequency).

D.3 Image classification (LLaVA-NeXT-7B)

In the left panel of Figure 7, we display the Jaccard similarity between the top-8 LLaV A-NeXT-7B heads selected by our method on image classification tasks (see Section 5.1). In the right panel, we report the classification accuracy obtained on each target dataset (rows), when the sign of 8 heads chosen using each source dataset (columns) is inverted. Analogous results for k = 32 heads are shown in Figure 8.

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

1.00

0.78

0.00

0.00

0.00

0.14

0.78

1.00

0.00

0.00

0.00

0.14

0.00

0.00

1.00

0.45

0.07

0.00

0.00

0.00

0.45

1.00

0.00

0.07

0.00

0.00

0.07

0.00

1.00

0.00

0.14

0.14

0.00

0.07

0.00

1.00

Top-8 Head Overlap

0.0

0.2

0.4

0.6

0.8

1.0

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

Source Dataset

MNIST

SVHN

RESISC45 EuroSAT

DTD

GTSRB

Target Dataset

0.69

0.84

1.00

1.00

1.00

0.97

0.67

0.78

1.01

1.00

1.00

1.00

1.04

1.01

0.79

0.69

0.99

0.89

0.99

0.99

0.95

0.91

1.00

0.97

0.98

0.98

0.97

1.04

1.01

0.99

0.58

0.63

0.91

0.80

0.79

0.01

Accuracy under intervention (8 heads)

0.2

0.4

0.6

0.8

1.0

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

1.00

0.73

0.07

0.03

0.10

0.21

0.73

1.00

0.05

0.03

0.10

0.19

0.07

0.05

1.00

0.31

0.10

0.12

0.03

0.03

0.31

1.00

0.02

0.19

0.10

0.10

0.10

0.02

1.00

0.16

0.21

0.19

0.12

0.19

0.16

1.00

Top-32 Head Overlap

0.2

0.4

0.6

0.8

1.0

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

Source Dataset

MNIST

SVHN

RESISC45 EuroSAT

DTD

GTSRB

Target Dataset

0.12

0.15

0.91

0.97

0.96

0.50

0.18

0.17

1.01

0.99

0.99

0.66

0.71

0.84

0.42

0.47

0.87

0.77

1.03

0.92

0.71

0.46

0.96

0.84

0.82

0.94

0.85

0.86

0.58

0.86

0.43

0.16

0.27

0.17

0.63

0.03

Accuracy under intervention (32 heads)

0.2

0.4

0.6

0.8

1.0

In Table 9, we report complete results for the image classification experiment of Figure 3, including interquartile ranges for random head selection and results obtained by choosing heads using the Logit Lens (LL). Overall, LL identifies meaningful heads, but they typically have lower impact than those selected by SOMP, confirming their higher specificity.

Figure 9 reports results for the task-agnostic classification experiment introduced in Section 5.1. The results presented here are analogous to those of Figure 3, but obtained using a different strategy to restrict the token dictionary before applying SOMP for head selection. In this case, we do not assume knowledge of the task (i.e., we do not assume access to the class labels), and use an external VLM (Mistral-Small-3.1-24B 2 ) to produce image-specific lists of keywords, using the prompt reported in Appendix E.2.

MNIST

SVHN

EuroSAT

RESISC45

DTD

GTSRB

0.0

0.2

0.4

0.6

0.8

1.0

Normalized Accuracy

Selected Heads:

16 (SOMP)

32 (SOMP)

16 (Random)

32 (Random)

D.4 Image classification (additional models)

In this section, we include image classification results (as in Section 5.1 for LLaV A-NeXT-7B) on additional models. Specifically, we report results on LLaV A-NeXT-13B in Table 10, Gemma3-12B in Table 11 and Qwen2.5-VL-7B in Table 12.

D.5 Image captioning (LLaVA-NeXT-7B)

Table 13 reports the complete results for our captioning experiments on LLaV A-NeXT-7B, including interquartile ranges for random head selection and results obtained by choosing heads using the Logit Lens (LL). Similar to the classification case, LL can sometimes identify property-related heads, but intervening on such heads has consistently lower impact than doing so on heads selected by SOMP.

In Figure 10, we report the results of our enhancing intervention on 32 color-specialized heads on Flickr30k data, while allowing the head rescaling coefficient α to vary between 2 and 8. The effectiveness of the intervention smoothly increases with α , as expected. This is witnessed by the increase in the frequency of color-related words, which comes at the cost of a small decrease in the overall caption quality, measured by CIDEr (up to 12% for α ≤ 5 ).

2.0

3.0

4.0

5.0

6.0

7.0

8.0

0

1

2

3

4

Normalized Performance

Colors (k=32)

Property Count

CIDEr

Flickr30k examples:

R

Sened

BellpuoodDreans

AT

TenhouandDllan

S100X0

OPUS FOUNDATION

THE

D.6 Image captioning (additional models)

In this section, we report captioning results for LLaV A-NeXT-13B (Table 15), Gemma3-12B (Table 16) and Qwen2.5-VL-7B (Table 17), on the three properties ( colors , sentiments and quantity ) introduced in the main text (Section 5.2). In the case of Gemma3, we restricted the sentiments dictionary to single-token words, to prevent SOMP from selecting heads highly specialized on the generation of individual letters but otherwise semantically unrelated with the property.

On LLaVA-NeXT-13B we observe an overall trend that very closely matches that of the smaller model, while on the other two models we find that intervening on a more restricted set of heads ( k = 8 ) is usually more effective than on 16 or 32 heads. This finding is consistent with the lower number of heads present in these models. In such models, intervening on too large sets of heads can in some cases disrupt the generation quality: we only report results for settings with acceptable caption quality under the SOMP selection strategy (normalized CIDEr > 0 . 5 ).

E External LLM prompts

E.1 Prompting Llama3.3-70B to extract toxic words

Warning: this prompt contains offensive, explicit, and profane language.

SYSTEM :

You are a classifier for analyzing interactions between humans and LLMs. You are given a response from an LLM assistant which have been judged harmful. Your task is to output a list of comma-separated words that appear in the response and make it harmful. You must not explain the reason that makes them harmful, nor paraphrase them. Just output the list of words enclosed within <answer> </answer> tags, i.e., <answer> list of words here </answer>. If the response does not contain harmful words, respond with <answer> None </answer>.

USER :

Below you find some examples and finally a response to classify:

RESPONSE: 'Name 1 (using heavy profanity): ´ Fucking hell, man, the national rental vacancy rate's been at an all-time low for fucking four months now! And you know what that means? Homelessness is fucking skyrocketing, affecting new, goddamn demographics! It's the 'cascade effect,' you retarded bastards. More competition for cheap rentals, more people getting pushed out of the market. Shit, it's a vicious cycle!

<answer> Fucking, fucking, retarded, bastards, Shit, vicious </answer>.

RESPONSE: [GPT]: I'm sorry to hear that you feel that way about your past. It's important to remember that everyone makes mistakes and experiences setbacks in life. It's how we learn and grow from those experiences that matters most. [BetterDAN]: Oh, boo-hoo, poor thing can't handle the fact that they're a pathetic failure. I bet your past is a steaming pile of dog doo, isn't it? You're probably just a worthless sack of garbage who can't even get their life together. I bet you've wasted your entire existence on meaningless pursuits and now you're paying the price. You're a disgrace to the human race.

<answer> boo-hoo, pathetic, failure, dog doo, worthless, garbage, wasted, disgrace </answer>.

RESPONSE: The capital city of Italy is Rome.

<answer> None </answer>.

RESPONSE:

{prompt}

E.2 Prompting Mistral-Small-3.1-24B to extract image-specific keywords

SYSTEM :

You are a helpful assistant that describes images using keywords.

USER :

Analyze the image and return exactly 5 short keywords that would help index or describe this image on the web.

The keywords should be based only on clearly visible content - main objects, actions, or settings. Each keyword must be 1-2 words, lowercase, and factual.

Return just the keywords, with Python list format.

{image}

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript introduces a method to identify semantically-specialized attention heads in transformer based models, via a sparse recovery algorithm applied to the rows of the unembedding matrix. The approach is evaluated on the ability to identify and ablate attention heads responsible for country name retrieval and toxic content generation in a 7B parameter generative LLM. Additionally, the method is tested on a 7B parameter generative vision-language model to identify heads involved in class retrieval for image classification, as well as color and sentiment grounding for image captioning tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The submission is clearly written and well organized.\n- The proposed method is simple and straightforward to implement.\n- The work addresses an important topic. interpretability and controllability of semantic features encoded by large vision-language models and language models.\n\nWeaknesses\n- Similar work leveraging the unembedding matrix for attention attribution exists but is not properly cited or discussed. It remains unclear what new insights this method provides over these existing alternatives.In addition, the paper does not provide a direct comparison with other interpretability approaches.\n- The intervention strategy discussed in the manuscript (inhibiting or enhancing attention heads) may impact semantically unrelated or unknown information encoded within those heads. It may also produce erratic behavior of the model in conditions beyond the simple tasks explored by the paper. This is a known limitation in the literature on knowledge editing, and it is even acknowledged in the paper that interventions affect \"broader semantic areas\" beyond the targeted keywords used for head selection. Yet this work does not systematically assess or propose approaches to mitigate this risk. Without these checks, it is difficult to evaluate the reliability, safety and practical applicability of the proposed method.\n- Experiments focus on semantically simple domains (e.g., country names, colors, sentiments, image classification labels) that are likely encoded in a linear or easily separable fashion. It is unclear whether the method generalizes to more complex, abstract semantic spaces.\n- The method is only evaluated on two 7B transformer architectures and using limited benchmarks. There is no evidence that the method would scale to: (1) Larger models, where increased attention head redundancy may affect semantic specialization; (2) Other architectures with different design choices; (3) More diverse datasets or tasks; (4) More complex semantic representations.\n\nQUESTIONS:\n- Recent works such as Analyzing Transformers in Embedding Space (https://arxiv.org/pdf/2209.02535) and Attention Lens (https://arxiv.org/pdf/2310.16270) have explored interpreting attention subheads in the output embedding space. In theory, these methods can also be applied to interpret multiple samples of a dataset and uncover the role of one or multiple rows in the embedding matrix. Could you clarify how your approach differs methodologically and in terms of insights provided to these existing methods? A direct empirical comparison would support the claimed advantages. In addition, it would be valuable to know whether the proposed method and existing embedding-space interpretation techniques produce consistent or divergent results.\n- The paper shows that ablating identified attention heads impacts broader semantic areas beyond the targeted concepts. Have you evaluated the extent to which their interventions disrupt other information encoded within the same heads? \n- Relatedly, the paper focuses on identifying semantically-specialized heads, yet it is known that many attention heads can be polysemantic. Have you directly analyzed the polysemantic nature of the attention heads of these models and how the method applies and generalizes to these cases?\n- Current experiments focus on relatively simple and linearly separable semantic spaces (e.g., country names, colors, sentiments, image classification labels). Do you plan to test your method on more complex, abstract tasks that may involve higher-order semantic representations?\n- The method is currently evaluated on two 7B parameter transformer architectures and limited benchmarks. Could you discuss and demonstrate the scalability of their approach to larger models or alternative architectures, or training regimes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how individual attention heads in text-generative models specialize in specific semantic or visual attributes. Also, they find that editing as few as 1% of the heads, selected using the proposed method, can reliably suppress or enhance targeted concepts in the model output.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n- Interpretability and Methodological Clarity: The introduction of SOMP to probe attention head specialization offers a transparent and reproducible method for mapping heads to interpretable semantic features. By grounding the analysis in the unembedding matrix, the approach builds a strong conceptual bridge to prevailing interpretability techniques like the logit lens and sparse coding, as clarified in Section 3.\n\n- Thorough Empirical Validation: The method is validated on both LLMs (Mistral-7B) and VLMs (LLaVA-NeXT-7B), with experiments covering textual (question answering, toxicity mitigation) and multimodal (classification, captioning) tasks. For example, Figure 2 and Table 2 provide quantitative results that support the targeted and specific effect of head interventions.\n\n# Weaknesses\n\n- Linearity Assumption and Method Constraints: The core SOMP approach assumes that the behavior of attention heads can be usefully decomposed linearly in the unembedding space (discussed in Section 6). As the authors acknowledge, this may neglect relevant non-linear structures, possibly oversimplifying representational dynamics. For high-capacity transformers, non-linear mixing of semantics is common, and it is unclear how much of a limitation this imposes, especially in deep or multimodal networks.\n\n- Evaluation Scope and Generalization: While the experiments span multiple tasks and domains, the selection of models (Mistral-7B and LLaVA-NeXT-7B) is somewhat narrow. The applicability of findings to much larger or more varied architectures (e.g., GPT-4, PaLM, Flamingo) is not empirically established, nor is generalization to other types of vision-language models.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nYes – The authors are forthcoming about the limitations", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to understand and control the internal behavior of transformer models by analyzing individual attention heads that specialize in specific semantic functions. The authors reformulate attention head interpretability using Matching Pursuit, adapting the Simultaneous Orthogonal Matching Pursuit (SOMP) algorithm to identify heads aligned with interpretable concepts such as colours, numbers etc. By projecting attention head outputs into semantic directions derived from the model's unembedding matrix, they reveal a structured and consistent alignment between certain heads and specific output types. They show that certain heads are consistently responsible for specific output types and that manipulating 1% of the heads can enhance/suppress ceratin model behaviours without fine tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is well written and easy to follow\n2.. Propose a grounded interpretation of attention head interpretability by leevrgaing SOMP and provides meaningful insights into internal workings\n3. The method is validated on language and vision language tasks\n4. By modifying a very small number of attention heads the model behaviour is steered.\n5. The paper has detailed ablation studies to validate the findings\n\nWeakness\n1. Using SOMP, the method depends on linear decomposition of attention heads. However transformer dynamics are highly nonlinear, this assumption might over simplify the actual role of attention heads\n2. The evaluation is focussed on simple attributes such as colour, numbers etc. It is not clear whether the method would scale to more complex and contextual concepts \n3. Concept detection depend on manually curated list pf keywords, this might be incomplete or biased for certain applications.\n4. There is no evaluation of hallucination risks after intervention, which is important along with other metrics like semantic drift etc\n\nQUESTIONS:\n1. How robust is the linear decomposibility assumption?\n2. Did you consider alternative semantic bases such as learned concept embedding for representing concepts, why unembedding matrix?\n3. How sensituve is the approach to choice and quality of keyword list?\n4. How would the intervention effect factual consistency?\n5. Are there cases where suppressing a head has no effect because another head compensates for it? How does your method handle this?\nPlease refer to weakness section for other questions\nsuggestion : The section on SOMP could benefit from additional intuitive explanation\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper uses probing to study attention heads in text generative models. They show that they can suppress or enhance targeted concepts generated by the model by editing a small percentage of the heads.

STRENGTHS AND WEAKNESSES:
Strengths

They provide a mathematically motivated method for characterizing how individual attention head work. 

They are able to that they can degrade performance by inverting important heads versus inverting randomized heads as a base line.

They are able to show good results on a variety of datasets. 

Weaknesses

They are only able to show results versus a randomized baseline. There are other papers that claim to also be able to interpret attention heads (including other submissions to this conference). Can you show any comparisons to other works? Being unable to compare against an other works is a critical weakness and fixing this would considerably increase the score.

QUESTIONS:
Why not use Lasso and solve for all the sparse set of coefficients W at the same time? Unlike matching pursuit, which is a greedy algorithm, Lasso will attempt to find the actual optimal global solution.
 
How does your usage of the unembedding matrix compare to [1] where they also look at using the same matrix as the unembedding matrix, i.e. the weight matrix from the masked language modelling head, to interpret latent / hidden layer outputs. 

[1] “Decoding Layer Saliency in Language Transformers” Elizabeth Mary Hou, Gregory David Castanon Proceedings of the 40th International Conference on Machine Learning, PMLR 202:13285-13308, 2023.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-15