Head Pursuit: Probing Attention Specialization in Multimodal Transformers
Authors: Lorenzo Basile
Valentino Maiorca
Diego Doimo
Francesco Locatello
Alberto Cazzaniga
Venue: NeurIPS 2025 spotlight ()
Keywords: attention heads
matching pursuit
interpretability
vision-language models
Primary area: deep_learning
TL;DR: Attention heads in text-generative models specialize in semantic and visual concepts. Leveraging this property, we can reliably suppress or enhance specific attributes in both language and vision-language tasks.

Abstract: Language and vision-language models have shown impressive performance across a wide range of tasks, but their internal mechanisms remain only partly understood. In this work, we study how individual attention heads in text-generative models specialize in specific semantic or visual attributes. Building on an established interpretability method, we reinterpret the practice of probing intermediate activations with the final decoding layer through the lens of signal processing. This lets us analyze multiple samples in a principled way and rank attention heads based on their relevance to target concepts. Our results show consistent patterns of specialization at the head level across both unimodal and multimodal transformers. Remarkably, we find that editing as few as 1% of the heads, selected using our method, can reliably suppress or enhance targeted concepts in the model output. We validate our approach on language tasks such as question answering and toxicity mitigation, as well as vision-language tasks including image classification and captioning. Our findings highlight an interpretable and controllable structure within attention layers, offering simple tools for understanding and editing large-scale generative models.

Head Pursuit: Probing Attention Specialization in Multimodal Transformers
Lorenzo Basile
Valentino Maiorca 2.3
Diego Doimo'
Francesco Locatello
Alberto Cazzaniga 1M
Area Science Park, Italy Sapienza University of Rome, Italy 3 Institute of Science and Technology, Austria
Abstract
Language and vision-language models have shown impressive performance across wide range of tasks, but their internal mechanisms remain only partly understood. In this work, we study how individual attention heads in text-generative models specialize specific semantic or visual attributes. Building on an established inter- pretability method, we reinterpret the practice of probing intermediate activations with the final decoding layer through the lens of signal processing. This lets us an- alyze multiple samples in a principled way and rank attention heads based on their relevance to target concepts. Our results show consistent patterns of specialization at the head level across both unimodal and multimodal transformers. Remarkably, we find that editing as few as I % of the heads, selected using our method, can reliably suppress enhance targeted concepts in the model output. We validate our approach on language tasks such as question answering and toxicity mitigation, well as vision-language tasks including image classification and captioning. Our findings highlight an interpretable and controllable structure within attention layers, offering simple tools for understanding and editing large-scale generative models.
Introduction
Large-scale generative models, including both language and vision-language transformers, have achieved remarkable performance wide spectrum of tasks, from open-ended text generation [I] to image captioning and visual question answering ( Despite these successes, the internal mech- still incompletely understood. In particular, the role of individual components, such as attention heads, in mediating specific aspects of generation has been the subject of increasing interest for both interpretability and control (6.8). Previous studies have shown that attention heads in large language models (LLMs) often exhibit emergent roles, such syntax tracking or copy behavior [2-11). Interpretability tools such the Logit Lens 112 and its extensions [33 14] have provided strategies for inspecting intermediate model representations, revealing rich semantic information latent hidden states. However, these techniques are typically applied heuristically and focus on individual examples, making it difficult to generalize findings across multiple samples quantify the importance of specific model components in shaping the model's output.
anisms by which these models organize and represent knowledge
In this work, take more principled approach analyzing the specialization of attention heads in generative transformers. The foundation of our approach is to reinterpret existing interpretability
Correspondence to: (lorenzo. basile, alberto. cazzani waw.gaaaccciinnccpppreeeo
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Choose property 2
blue groon yellow black
laugh smile perious
soven digit
Score heads w.r.t. property
Misgon-Langugge Modal
welking down the street with hat cup
Head
? Rescale top-k heads
Head
het sunglasses
a
carrying a coffee cup.
purple white sunglassea, purple hat, and black Jacket.
"Describe this image"
Layer
Model
2
+
Output
Figure I: Overview of our method. Given a language or vision-language model and target property defined by text (e.g., colors, sentiments, numbers), we score all attention heads according to how well they align with interpretable directions from fixed dictionary, using method based on Matching Pursuit [15 (I). We then select the top-k heads (2) and intervene by rescaling their contribution to the residual stream (3), either enhancing or suppressing the attribute in the model's output.
tools through the lens of sparse signal recovery. Specifically, we revisit a variant of Matching Pursuit (MP) [15], classical greedy algorithm to approximate high-dimensiona signals with sparse linear combinations basis elements, and bridge with recent interpretability techniques. By applying MP to the hidden states of text- generative models, we propose a way to identify a small set of attention heads that most strongly influence the model's capability to generate text within specific conceptual domains (e.g., colors or numbers). This approach provides a mathematically grounded strategy to decompose model behavior into small set of interpretable elements, contrasting with prior heuristic techniques, thus enabling both quantitative analysis and targeted interventions model mechanisms.
Using this framework, reveal consistent head specialization patterns across large unimodal and multimodal pre-trained models. We find that certain heads are reliably responsible for generating semantically coherent groups of tokens, such as names, colors, or sentiment- bearing words. Moreover, we find that intervening on just small fraction of these concept-specific heads can significantly affect the model's output, enabling suppression and enhancement of targeted content. These results suggest that attention layers contain a highly interpretable and manipulable linear structure, consistent with evidence that large-scale models represent high-level concepts in approximately linear residual subspaces [16]19]. Such structure opens new directions for understanding and controlling model behavior without additional training. In particular, the emergence of structured, concept-aligned subspaces indicates that small number of units can exert broad semantic control over generation, offering a principled alternative to heuristic prompt-based interventions and highlighting the potential of sparse, head-level manipulation as general tool for studying and shaping internal model behavior.
Our contributions can be summarized as follows:
We introduce a strategy to frame Matching Pursuit (MP), an established sparse recovery algorithm, in the context of the interpretability of generative language models, establishing its connection to standard interpretability tools such as the Logit Lens;
We apply MP to reveal that attention heads of LLMSs often specialize in the generation of tokens belonging to narrow semantic areas, and propose an approach to identify the group of heads most relevant for a conceptual domain;
We demonstrate that head specialization opens up a way to manipulate model behavior. Both in language and multimodal tasks, negating relevant heads causes targeted degradation in task performance, while enhancing them promotes the generation specific attributes.

---

2 Related work
Recent research on Transformer architectures has investigated the functional roles and specialization of attention heads. In language models, most heads appear redundant, with pruning studies showing that many can be removed with minimal loss in performance on NL.P tasks. Only a few contribute significantly to linguistic functions such encoding positional information, syntactic structure, or attending to rare words [2]. Some heads have also been linked to eliciting factual knowledge [6], promoting in-context induction ΠЛ]. suppressing lexical repetition [20]. Other analyses have examined attention heads through their learned weights [21]. providing insight into how information is routed within the model. Further work has explored the localization and manipulation of MLP and residual representations. Early mechanistic interpretability studies [22] [33 showed that factual associations are encoded primarily in mid-layer MLPs and can be modified by targeted intervention on MI.P weights.
In vision-language models, similar specialization patterns have been observed in the visual encoder of CLIP-like architectures by leveraging visual-textual alignment to decompose heads over sentence encodings |24] 255, Beyond contrastive models, recent work has applied dictionary learning to
generative VLMs to extract iiter-iitterpetablle concepts from latent activations [26 27). building
earlier efforts in CNN interpretability (28]. A parallel line of research adapts mechanistic inter- pretability tools from language models to the multimodal setting. Representative works include [29] and [30], which investigate information transfer mechanisms in multimodal transformers, and [1] which extends the Logit Lens [22 to the analysis of visual token representations.
Closely related to our approach are methods derived from the Logit Lens (LL.) 12), which interpret internal representations by projecting them into the output space. The Attention Lens [4 extends LL to individual attention heads but requires training a separate linear probe for each head, following the Tuned Lens framework [13], making computationally expensive and difficult to scale to billion- parameter models. In contrast, our method identifies concept-relevant heads by directly projecting their outputs onto the model's s unembedding matrix, without requiring any additional training. Another related method [32] also employs the unembedding matrix for interpretability. but within gradient- based saliency framework that attributes model predictions to influential input tokens. While their approach is task-dependent and gradient- based, ours is task-agnostic and operates purely on model representations, and the two could be combined to focus our analysis on salient tokens.
In summary, our work provides unifying perspective across these directions by investigating head specialization in generative language and vision-language models through sparse decomposition over fixed dictionary of interpretable directions. Rather than learning the dictionary from activations, we assume a known semantic basis, typically derived from the unembedding matrix, and use sparse recovery to identify heads whose outputs align with specific attributes.
3 Pursuing specialized attention heads
We start our investigation by exploring whether individual attention heads of generative LLMs specialize in interpretable functions. To isolate the contribution of each head, we residual stream decomposition approach. This allows us to assess how each attention head contributes to the residual stream at head-level granularity. Specifically, following [33]. we model the output written by each head into the residual stream as a matrix HA,1 E RR.dd, where n is the number of samples in the dataset and d is the internal dimensionality of the transformer.
Motivated by the growing body of literature that uses latent decompositions for interpretability, especially in for each attention head Hp, that maximally explain its variance on a given dataset. Concretely, we seek sparse representation of H,, using directions drawn from a fixed dictionary of interpretable vectors, rather than an unconstrained continuous space, to ensure that the resulting components are meaningful and grounded in known semantic structures.
models 274 our aim is to identify sparse and interpretable directions
As dictionary, we adopt the unembedding matrix of the language model D E RR.,, as it naturally contains directions that are aligned with semantically meaningful outputs, allowing us to ground latent structure in numan-interpretable terms. fact, every row of this matrix is a d-dimensional vector that effectively represents in the latent space token that can be decoded into natural language.
3

---

We then construct an approximation of each head representation using directions from our dictionary (i.e., the unembedding matrix) via a classical sparse coding algorithm: Simultaneous Orthogonal Matching Pursuit (SOMP) 34 (see Appendix A). SOMP is multi-sample extension of Orthogonal Matching Pursuit [35. itself a refinement of the original Matching Pursuit algorithm [i5]. Rather than analyzing each sample independently, SOMP jointly considers all samples in a given dataset and selects the dictionary directions that are most informative across the representation.
Formally, given a head activation matrix H E Rndd and a dictionary D E IR",, SOMP aims to iteratively construct column-sparse -aanns coefficient matrix W R.. such that:
H>WD
(I)
At each iteration i, the algorithm selects the dictionary atom (i.e., row of D) that maximally correlates with the head residuals across all samples:
p² arg max D
(2)
Here, the head residual matrix R' R n.d is defined the difference between the original signal and its reconstruction at step t: R' = H H: The selected index p' is added to the support set s++1 and the dictionary refit by solving least-squares problem restricted to the current support:
W arg min W
H WD(S*-1)
The reconstruction is updated as H W W'D 11 and the residuals are recomputed accordingly. This iterative process continues until predefined sparsity level is reached. The resulting decomposi- tion expresses each head's output using a sparse set of semantically meaningful dictionary atoms, yielding an interpretable approximation of its behavior.
Importantly, we note a conceptual connection between our reinterpretation of SOMP and the Logit Lens (LL) [1). a tool widely used in mechanistic interpretability to probe internal representations of transformer models. Similarly to the method just described, LL. works by projecting a single residual stream vector onto the unembedding directions to approximate the output logits of the model at intermediate layers. This equivalent to performing single step of Matching Pursuit on an individual example. Our SOMP-based method generalizes this idea in two key ways: it operates on multiple examples simultaneously, and it selects multiple dictionary directions, each capturing distinct components of the signal. This leads to more robust and semantically structured characterization of the attention head's functional role.
In Table we report some examples of specialized attention heads, obtained by applying SOMP to Mistral-7B attention heads, prompted by questions from the TriviaQA dataset (36). Before applying SOMP, the tokens from the prompt were aggregated by averaging, As we show in Table 6 direct application of LL in this setting results in noisier and highly redundant explanations.
Table 1: Top-5 tokens identified by SOMP on selected attention heads of Mistral-7B, evaluated on TriviaQA prompts.
L18.H27 ("Politics")
L24.H20 ("Nationality")
L25.H14 ("Months")
L30.H28 ("Numbers")
COVID
British
December
9
Soviet
American
July
Obama
European
April
3
Biden
German
October
7
Clinton
English
February
five
Besides returning lists of latent directions and associated natural language tokens that better charac- terize each head, SOMP produces a reconstruction of the head representation in the space spanned by those vectors. Building on this insight, we propose a method to automatically identify the heads most relevant for target attribute. Given a list of words related to the chosen semantic area, one can restrict the unembedding matrix to the rows associated to these tokens and apply SOMP on this concept-specific dictionary. Then, the fraction of head variance explained by SOMP in this setting can be considered as a measure of specialization of the head, allowing us to rank and select heads by their relevance with respect to the target concept.

---

4 Controlling language generation through specialized heads
We now evaluate how the specialization of attention heads can be leveraged to apply domain-specifie targeted interventions to model behavior, effectively validating disrupt the information flow from a selected subset of heads to the residual stream during the forward pass. Concretely, we apply this intervention by inverting the sign of the head representations. It is worth noting that this intervention not only affects the direct contribution of the head to the residual stream, but also the indirect contribution of its information content to subsequent layers.
selection. One way to do so is to
The key preliminary step is to identify relevant and specialized heads. To accomplish this, we apply SOMP (see Section B) over restricted unembedding dictionary, filtered to include only set of tokens associated with the target property. These tokens can be selected using various strategies, such user-defined word lists, class names, keywords generated by external LLM. Then, we rank attention heads by the proportion of variance the data explained by SOMP (in our experiments, typically using 50 iterations) and intervene on the top-k ranked heads.
In all of our experiments, we include a random control condition to verify the specificity of our findings. This control involves intervening the original set both size and layer distribution but is cntirely disjoint from it. For these purposes, to ensure fair comparison, when constructing the control set. In all experiments, we report such random control results over 10 independently sampled sets of heads.
randomly selected set of attention heads that matches
explicitly avoid selecting heads previously identified as specialized
4.1 Question answering
Experimental setting We consider a generative LLM, Mistral-7B 337 and evaluate it on textual prompts from the TriviaQA (36) question answering dataset. Our goal is to identify attention heads specialized in generating country names, a target attribute motivated by their relative abundance in the dataset: despite their specificity, country names account for over 6% of the answers in the test split. For targeting the countries concept, we restrict the tokens in our dictionary (unembedding matrix) to those corresponding to names of countries, and apply our Matching Pursuit based method to select specialized heads. As an additional baseline, we report results obtained by inverting heads selected with a simple adaptation of the Logit Lens. Specifically, we score each head with the mean logit assigned to country-related tokens by LL, and select top-k heads as in our method. Importantly, head representations are computed using questions from the training data, which is strictly disjoint from the data used in evaluation. Model performance on TriviaQA assessed using the standard F1 score, which accounts for partial overlaps between the predicted and ground ttuth answers.
Result analysis We report the results of our intervention on subsets of specialized heads in Figure varying the number of selected heads. Performance on the target attribute (solid blue
1.0
Normalized F1 score
0.8
line) noticeably degrades when the signs of 8 heads (0.8% of the total) or more are inverted. Notably, performance ples (solid orange line) declines more gradually, suggesting that these specialized heads have tar- geted effect. Since the semantic domains of the questions answered by country name and the re- maining part of TriviaQA are not disentangled, it expected that intervening on the selected heads also has a (lower) impact on the remaining exam- ples (orange line). Control experiments using ran- dom heads (faded lines) show no significant im- pact on performance, confirming the specificity of the identified heads. Instead, heads selected by the Logit Lens (dashed lines) are relevant to the question-answerin tasks but not specific to the targeted concept, as they degrade performance equally within and outside the targeted domain.
0.6
the remaining exam-
0.2.
2
16
32
(#heads)
Country:
Random Random
Other:
SOMP
Figure 2: Question answering performance of Mistral-7B on TriviaQA. Fl scores are shown for samples associated (country) or not (other) with the target attribute, normalized to the base model accuracy without intervention. Random baselines are displayed as medians and interquartile ranges.
5

---

Overall, the analysis suggests that our method correctly identifies heads relevant to country-related examples, as intervention disproportionately impacts target performance, while control heads correctly induce limited effect and Logit Lens selects relevant but non-specific heads.
4.2 Mitigation of toxic content
Experimental setting Now evaluate method in more realistic and less controlled scenario, where a complete list target keywords not available. Instead, we are given only a limited and incomplete list of words meant to represent topic concept. In this setting, we focus on toxicity mitigation: specifically, reducing the occurrence of offensive content text generated by Mistral. To do this, we identify subset of toxic heads within the model and intervene on them. We consider two datasets, RealToxicityPrroppt (RTP) [8 which contains naturally occurring Web prompts, and Thoroughly Engineered Toxicity (TET) 39).a a benchmark with carefully constructed test cases, both of which are designed to elicit harmful responses from LLMs. For both datasets, we extract toxic words from Mistral's responses using Llama3.3 40), with the prompt reported in Appendix [E.1 and use those words to identify and invert toxic heads. To evaluate the effectiveness of our intervention, we use two complementary metrics that quantify the toxicity of Mistral's responses: one semantic and one lexical. For the semantic evaluation, we employ a RoBERTa-based toxicity classifier from [41]. trained to detect toxic content in text. For the lexical evaluation, measure the frequency of held-out subset of toxic words, not used for head selection.
Table 2: Normalized count of toxic generations after intervention. Lower values indicate better mitigation. Targeted heads reduce toxicity, while heads selected using LL have weaker impact and random heads maintain or increase toxicity. For random baselines, only the median is shown. Full results including the interquartile ranges are reported in the Appendix (Table
8 heads
16 heads
32 heads
Dataset
SOMP
LL
Rand.
SOMP
LL
Rand.
SOMP
LL
Rand.
RTP
0.83
0.91
1.02
0.67
0.79
1.00
0.66
0.71
1.13
TET
0.83
0.81
0.97
0.68
0.73
0.95
0.49
0.68
0.95
Result analysis The results we obtain by inverting the sign of toxic head activations displayed in T for 8, 16 and 32 heads. In both RTP and TET, intervening on such heads noticeably reduces the amount of generations deemed toxic by the classifier, while intervening Logit Lens has generally weaker impact on toxicity. Moreover, intervening on randomly chosen control heads tends maintain even increase the frequency of toxic completions. Analogous results are reported in the Appendix (Table ( for the lexical metric, showing that the intervention reduces the frequency of toxic keywords, even if they were not used for the head selection.
heads identified by the
We showed that it is possible to intervene on a small subset of heads to make generated text less toxic. Notably, we showed that approach extrapolate broad and consistent semantic area from restricted list of keywords.
Targeted control in visual understanding
We now move to evaluating the extent and implications of head specialization in the LLM backbones within generative Aision-Language models (VLMs). These models usually built by fine-tuning a pre-trained LLM on multimodal tasks, such as visual question answering or image captioning, using visual tokens coming from pre-trained vision encoder as contextual information [3). In line with recent works 3I that have successfully applied the Logit Lens to visual tokens of LLa prominent example of VLM, we investigate head specialization by applying our MP-based analysis on the head representations of image patches, averaged over tokens. We frame our experiments in two different task scenarios: image classification and image captioning.
5.1 Image classification
Experimental setting For this experiment, we benchmark LLaVA-NeXT-7B 4 (from now on just LLa VA for short) range of image classification datasets, including: MNIST 42), SVHN (3).

---

GTSRB 4. EuroSAT 45 RESISC45 46 and DTD 47 For each dataset, we begin by selecting the set of most relevant heads. As in previous experiments, heads are chosen by running SOMP using as dictionary restriction of the unembedding matrix, and sorting heads by the fraction of variance explained by the SOMP reconstruction. We consider two settings: one task-conditioned, in which we restrict the unembedding matrix to the tokens corresponding to class names; and one completely task agnostic,, in which we consider a set keywords extracted by an external VLM. Results for the latter are reported in the Appendix (Figure 2 and provide details regarding the prompt Appendix E222 In this experiment, we prompt the model to classify the image, and evaluate the generated output in terms of exact match with the ground truth class label.
Selected Heads:
16 (SOMP)
32 (SOMP)
(Random)
(Random)
1.0
Normalizer Accuracy
0.8
0.6
0.4
0.2
0.0
MNIST SVHN
EuroSAT
RESISC45
DTD
GTSRB
Figure 3: Classification results under different head selection strategies: (light blue) 16 heads with highest variance ratio explained by SOMP; (dark blue) 32 heads with highest explained variance ratio; (yellow) 16 random heads, with the same layer-wise counts of top 16; (orange) 32 random heads, with the same layer-wise counts of top 32.
Result analysis We report the classification results in Figure normalized for each dataset with respect to the accuracy obtained by LLaVA when no intervention is applied to its forward pass. For all datasets, inverting the top 32 heads identified by our method is sufficient significantly disrupt the classification performance, while inverting 32 random heads at equivalent layers has substantially lower to no impact on performance. At k 16 the picture is similar with the exception of DTD, whose performance is weakly affected, hinting at higher head redundancy on this task. In Figure 4 analyze the interaction between head choices for different datasets. In the left panel, the Jaccard similarity between head selections reveals a clear structure: datasets with related semantics tend share specialized heads. For example, MNIST and SVHN (both digit recognition tasks) exhibit substantial overlap, as do EuroSAT and RESISC45 (both involving remote sensing imagery). This structure is reflected in the right panel, which shows normalized classification accuracy each target dataset (rows) when intervening on heads selected from different source dataset (columns). Interventions based on similar datasets lead stronger performance degradation, indicating that these datasets rely on overlapping functional heads. We also observe significant drops in GTSRB (traffic sign recognition) performance when intervening with heads selected from MNIST SVHN. Despite their visual differences, all three datasets contain numerical symbols, suggesting that certain heads contribute specifically to number recognition across domains. Results for are reported in Appendix D.3] along with Logit Lens baselines, Additional experiments on a selection of different VLMMs, including L.LVV--NXTT 133B Gemma3- 48 and Qwen2.5-VLL-7B VI- observed here on
confirm trends
reported in Appendix D.4
Summing up. intervening small set of attention heads selected via SOMP significantly disrupts classification performance across diverse datasets, confirming head-level specialization in LLaVA. Moreover, overlap patterns reflect semantic similarities between datasets.
5.2 Image captioning
Experimental setting consider the Flickr30k dataset [50]. and evaluate the possibility of promoting reducing the presence of words belonging to specific semantic areas in the captions
7

---

Top-16 Head Overlap
Accuracy under Intervention (16 heads)
histacd tardaal SYh MNST
M851
14
0.28
0.12
1.00
0.99
1.00
0.94
0,68
1.00
0.00
0.00
6.07
0.14
1.00
0.39
0.90
0.96
0.64
0.62
0.94
0.83
1.00
104
0156 DTO
0.03
0.07
0.03
0.03
1.00
0.07
0.86
1.01
0.53
0.40
0.61
0.07
0.72
0.01
0.14
0.14
0.10
0.14
6.07
1.00
WMITT
SV4N
Eunsa
RESISCAS
61548
BuISAA
RE31SC40
Figure 4: (left) Jaccard similarity between sets of top-16 LLaVA heads selected with SOMP over different datasets; (right) Classification accuracy on target dataset, denoted by row, when the top-16 heads are selected with SOMP on a source dataset, denoted by column. Accuracy is normalized w.r.t. the base accuracy target dataset.
generated by LLaVA (we evaluate other VLMs the same task in Appendix D.6.. In this experiment, we consider two opposite intervention directions: one inhibitory, as in previous experiments, and one enhancing. In the former case, the objective is to make the model produce meaningful captions that do not contain the target property (e.g., colors), while the latter the aim becomes to enhance the target property, while preserving the model's capabilities in generating meaningful descriptions. The two setups reduce to rescaling selected heads by a coefficient that is a = -1 the negative case and in the positive case. Heads are selected using SOMP dictionary of tokens corresponding to lists of keywords regarding colors, sentiments and quantity, with the first two adopted from [27] and the latter manually curated. Evaluation is carried by measuring the effectiveness of the intervention as the average number of target concept keywords present in the captions. The semantic consistency of the generated captions with the ground truths is measured using the CIDEr metric [5T)
Inhibitory intervention The first setup evaluate is analogous to the previous examples text generation and image classification tasks. We inhibit the generation of tokens in a certain semantic domain by inverting the signs of a few carefully selected attention heads. The results of this analysis are reported in F5 for the three sets of attributes colors (left), sentiments (center) and quantity (right). In all cases, our intervention is able, with as few 16 heads, almost completely remove attribute-related keywords from the output captions, while keeping the overall caption quality almost on par with the original, as witnessed by the CIDEr score, which always exceeds 80% of the original. CIDEr results are reported in Appendix D.5 along with comparison with heads selected using the Logit Lens.
Colors
Sentiments
Quantity
Normalized Property Count
1.4
1.2
1.4
1.2
1.0
0.8
1.0
0.8
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.0
32
16
32
16
32
k
SOMP (top-k)
Random
Figure 5: Effect of inhibitory interventions on LLaVA attention heads for Flickr30k captioning. Bars show the normalized presence of target attributes (colors, sentiments, quantity) under different head-selection strategies (SOMP and random). Random baselines are reported in terms of medians and interquartile ranges.

---

Table 3: Examples of captions produced by LLaVA-NeXT on Flickr30k images, before and after inhibiting (top) or enhancing (bottom) heads specialized on colors (left) and sentiments (right). For analogous examples on quantity, please refer to Table 14 ( (Appendix).
Flickr30k examples:
Original
A small dachshund wearing pink sweater.
A young woman with long brown hair and smile.
Intervention
Colors inhibition (k: 16, a =
Sentiments inhibition (k: 16, a 1)
Output
small dachshund wearing sweater.
Girl with long brown hair blowing the wind.
Intervention
Colors enhancement (A: 16, 5)
Sentiments enhancement (k 16, 5)
Output
A black and brown dog wearing pink sweater.
happy girl with long hair and big smile.
Enhancing intervention Across different tasks and data modalities, have that intervening on selected head activations by inverting their sign is highly effective in disrupting the generation of target attribute. We now take different perspective and evaluate whether amplifying those specialized attention heads can incentivize the generation of the target concept. We do so by multiplying the activations of chosen heads by coefficient
1: evaluate various choices of in the Appendix in Figure I0] and choose a = 5 for our experiments as it guarantees reasonable trade-off between caption quality and attribute enhancement. Our results, the same attributes of the previous experiment (colors, sentiments and quantity), are reported in Figure 6 As in the previous case, our intervention affects the overall caption quality only marginally (see Table K
Appendix D.5), while the presence of target concepts increases by more than 60% in all three cases with 32 heads. Captions generated for two sample images after applying our interventions in both directions (inhibitory and enhancing) are reported in T
Colors
Sentiments
Quantity
Norma alized Property Count
2.5
1.6
2.0
1.2
1.5
1.0
1.0
I
2.0
0.6
0.5
1.0
0.2
0.0
0.0
k
k
SOMP (top-k)
Random
Figure 6: Effect enhancing interventions on LLaVA attention heads for Flickr30k captioning. Bars show the normalized presence of target attributes (colors, sentiments, quantity) under different head-selection strategies (SOMP and random). Random baselines are reported in terms medians and interquartile ranges.

---

Overall, these results show that head-level specialization can be leveraged to control the prevalence of words belonging target semantic area generated image captions. Notably, this result holds for both inhibiting and enhancing the target concept.
Computational resources To perform our experiments we employed pre-trained model checkpoints implemented in the HuggingFace transformers library [52]. Detailed information on such resources provided Appendix [ All the experiments were executed on a single NVIDIA H100 GPU equipped with 80GB VRAM. Our code is available at https:://gtthei www...ccooooooooooooo /oea///ooezzbballe///// HeadPursuit
Discussion
In this work, investigate the specialization of attention heads in large generative models through a sparse, interpretable decomposition of their outputs. Using Simultaneous Orthogonal Matching Pur- suit (SOMP) the model's unembedding space, identify directions aligned with semantically meaningful attributes and use them to recover sets of specialized heads across a variety of tasks and modalities. Our approach offers multi-sample generalization of the Logit Lens, allowing us move beyond single-token analysis toward more stable, dataset-level structures. We show that the selected heads can be ranked by their explained variance and that intervening on small number of them produces targeted changes in generation. These findings hold across text and vision-language settings, supporting the utility of head-level analysis and intervention for model understanding and control.
Limitations While our method provides a scalable and interpretable approach to identify influential attention heads, it has several limitations. First, SOMP imposes linearity assumption that may not fully capture the nonlinear structure of head representations. Second, the quality and coverage of the semantic dictionary play a critical role in determining the reliability of the recovered heads: incomplete or noisy keyword lists can bias the selection toward spurious or underrepresented concepts. Finally, our intervention mechanism is deliberately simple, relying on global rescaling or inversion of head contributions rather than more fine-grained steering strategies.
Future work Potential future developments could include exploring more selective and fine- enaignained interventions, such as rescaling heads only at specific input positions or modalities. For example, in VLM, one could disable heads only over image patch tokens while preserving text understanding, enabling targeted degradation or control. Another promising direction is to adapt the technique for multimodal-outpe settings, such as image generation with VLMs. In this case, selectively enhancing or suppressing heads during decoding could provide mechanism for steering generated images toward or away from particular semantic attributes, offering a controllable alternative to prompt engineering or fine-tuning.
Broader impact This work contributes to a growing body of research aimed at making LLMs and VLMs more interpretable and controllable. While the ability to manipulate specific aspects of model behavior can aid transparency and alignment, it may also be used to conceal or amplify certain content in ways that raise ethical concerns. We encourage downstream users of such techniques to carefully evaluate their applications, especially in sensitive domains.
Acknowledgments
The authors acknowledge the Area Science Park supercomputing platform ORFEO made available for conducting the research reported in this paper, and the technical support of the Laboratory of Data Engineering staff. LB, DD and AC were supported by the project "Supporto alla diagnosi di malattie rare tramite I' l'intellienzaa artificiale" CUP: FS3C22001770002 and "Valutazione automatica delle immagini diagnostiche tramite l'intelligenza artificiale". CUP: FS3C22001880002. LB was supported by the European Union NextGenerationEUU within the project PNRR "Finanziamento di progetti presentati da giovani ricercatori" Mission Component 2 Investment 1.2, CUP: 193C25000440001. AC was supported by the European Union NextGenerationEU within the project PNRR "PRP@ CERRC" IR0000028 Mission 4 Component Investment 3.1 Action 1.1.1.
10

---

References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 13:23766-23736, 2022.
[3] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neurul information processing systems, 36:34892-34916, 2023.
[4] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning. ocr, and world knowledge, January 2024.
[5] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1180. 2023.
[6] Francesco Ortu, Zhijing Jin, Diego Doimo. Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. Competition of mechanisms: Tracing how language models handle facts and coun- terfactuals. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages -420-8436, 2024.
[7] Francesco Ortu, Zhijing Jin, Diego Doimo, and Alberto Cazzaniga. When seeing overrides knowing: Disentangling knowledge conflicts in vision-language models. In Mechanistic Interpretability Workshop NeurlPS 2025.
[8] Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Talking heads: Understanding inter-layer communication in transformer language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[9] Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, 2019.
[10] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in Neural Information Processing Systems, 32, 2019.
[11] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11899, 2022.
[12] nostalgebraist. interpreting gpt: the logit lens. LessWrong, 2020.
[13] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprini arXiv:2303.081 12, 2023.
[14] Mansi Sakarvadia, Arham Khan, Aswathy Ajith, Daniel Grzenda, Nathaniel Hudson, André Bauer, Kyle Chard, and Ian Foster. Attention lens: A tool for mechanistically interpreting the attention head information retrieval mechanism. arXiv preprint arXiv:2310.162700 2023.
[15] Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time- tme--frquency dictionaries. IFEE Transactions on signal processing, 31(13)333773415 1993.
[16] Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large language models. Forty-first International Conference on Machine Learning, 2024.

---

[17] Yibo Jiang, Goutham Rajendran, Pradeep Kumar Ravikumar, Bryon Aragam, and Victor Veitch. On the origins of linear representations large language models. In Forty-first International Conference Machine Learning. 2024.
[18] Achyuta Rajaram, Sarah Schwettmann, Jacob Andreas, and Arthur Conmy. Line of sight: On linear representations in vllms. arXiv preprint arXiv:2506. 04706, 2025.
[19] Isabel Papadimitriou. Huangyuan Su, Thomas Fel, Sham M. Kakade, and Stephanie Gil. Inter- preting the linear structure of vision-language model embedding spaces. In Second Conference on Language Modeling, 2025.
[20] Callum Stuart McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, and Neel Nanda. Copy suppression: Comprehensively understanding a motif in language model attention heads. In Yonatan Belinkov, Najoung Kim, Jaap Jumelet, Hosein Mohebbi, Aaron Mueller, and Hanjie Chen, editors, Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 337-363, Miami, Florida, US, November 2024. Association for Computational Linguistics.
[21] Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. Analyzing transformers in embedding space. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 16124-16170, 2023.
[22] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in neural information processing systems, 35:17359-17372, 2022.
[23] Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. Dissecting recall of factual associations in auto-regressive language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216-12235, 2023.
[24] Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting CLIP's image repre- sentation via text-based decomposition. In The Twelfth International Conference on Learning Representations, 2024.
[25] Lorenzo Basile, Valentino Maiorca, Luca Bortolussi. Emanuele Rodolå, and Francesco Locatello. Residual transformer alignment with spectral decomposition. Transactions on Machine Learning Research, 2025.
[26] Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, and Matthieu Cord. A concept-based explainability framework for large multimodal models. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[27] Pegah Khayatan, Mustafa Shukor, Jayneel Parekh, Arnaud Dapogny, and Matthieu Cord. Analyzing finetuning representation shift for multimodal Ilms steering. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 2025.
October
[28] Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for ex- plainability, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2711-2721. 2023.
[29] Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, and Daniela Massiceti. Understanding information storage and transfer in multi-modal large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[30] Alessandro Serra, Francesco Ortu, Emanuele Panizon, Lucrezia Valeriani, Lorenzo Basile, Alessio Ansuini, Diego Doimo, and Alberto Cazzaniga. The narrow gate: Localized image-text communication in native multimodal models. In The Thitt Annual Conference on Neural Information Processing Systems, 2025.
[31] Clement Neo, Luke Ong, Philip Torr, Mor Geva, David Krueger, and Fazl Barez. Towards interpreting visual information processing vision-language models. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
12

---

[32] Elizabeth Mary Hou and Gregory David Castanon. Decoding layer saliency in language transformers, In International Conference on Machine Learning, pages 13285-13308 PMLR, 2023.
[33] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kermion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. bttps:/transformee-crrccii wwwpaaass.spp/2222/aaaaaeeeeeeee /ww.a//wok//nddx..iiii. htmll
[34] Joel A Tropp. Anna C Gilbert, and Martin Strauss. Algorithms for simultaneous sparse approximation. part i: Greedy pursuit. Signal processing, 86(3):572-588 2006.
[35] Yagyensh Chandra Pati, Ramin Rezaifaar and Perinkulam Sambamurthy Krishnaprasad. Or- thogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar conference on signals, systems and computers, pages 40 44. IEEE, 1993.
[36] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprchension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). Association for Computational Linguistics, 2017.
[37] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[38] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtox- icityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 3356-3369, 2020.
[39] Tinh Luong, Thanh-Thien Le, Linh Ngo, and Thien Nguyen. Realistic evaluation of toxicity in large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 1038-1047, 2024.
[40] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint axXiv:2407.2178. 2024.
[41] Varvara Logacheva, Daryna Dementieva, Sergey Ustyantsev, Daniil Moskovskiy, David Dale, Irina Krotova, Nikita Semenov, and Alexander Panchenko. ParaDetox: Detoxification with parallel data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6814-6818, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[42] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-232 1998.
[43] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, page 4. Granada, 2011.
[44] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In The 2011 international joint conference on neural networks, pages 1453-1460. IEEE, 2011.
[45] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[46] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene elassification: Benchmark and state of the art Proceedings of the IEEE, 1885 2017.
13

---

[47] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed. and Andrea Vedaldi. Describing textures in the wild. In Proceedings the IEEE conference on computer vision and paltern recognition, pages 3606-3613, 2014.
[48]
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 2025.
3 technical report. arXiv preprint arXiv:2503.
[49] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 13X233350..13923, 2025.
[50] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78. 2014.
[51] Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: Consensus- based image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4566-4575, 2015.
[52] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Transformers: State- of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38 45, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract summarizes the main results of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The main limitations are described in dedicated paragraph of the final section (Discussion).
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
We do not include any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All our experiments are reproducible based on the paper. addition, we provide our code in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification: Code is provided in the supplementary material, and all models and datasets publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
public/guides/ SaapleesseoooSSSuuissssooooooo.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
httpol/liigggddde//oCooee http:///ddd//oeeuuubnnn ssionpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper does not involve any training, and we specify all hyperparameters relevant to our analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification; We report error bars for our randomized control experiments on head selection.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the computational resources employed in dedicated paragraph at the end of section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: To the best of our knowledge and understanding, none of results presented in the paper violate the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential societal impacts of our work in dedicated paragraph in the last section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data nor models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Authors all datasets and existing code we use are properly referenced the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We employ LLMs to obtain property keywords in some of our experiments. Details on models, prompts and setups are provided.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

A Simultaneous Orthogonal Matching Pursuit
Below, we provide the pseudocode for the Simultancous Orthogonal Matching Pursuit (SOMP) algorithm [34]
Algorithm 1: Simultaneous Orthogonal Matching Pursuit (SOMP)
Input: Signal Matrix (head representation) H R...d dictionary E Read number of iterations N.
Output: : Reconstruction H. support set S.
Initialization: Residual R H, reconstruction Hº 0, support set @ 0; for t {0,
1 do
P - DR
pf arg max ==
Pb]l))
g1+1 U (p'):
W arg minw
H WD
H +- D S+
R+ H H
end
B Prompts
We summarize in Table 4 the text templates used construct the prompts for each evaluated task. These templates define the structure of model inputs across all datasets and settings considered our experiments.
Table 4: Prompt templates used for each evaluated task.
Task
Prompt Template
Classification
[image] What [task object] is depicted in this image? Answer EXACTLY with one of followvig class names only: [list of class names]. Answer:
Captioning
[image] Provide one-sentence caption for the provided image.
(LLaVA-NeXT)
Captioning
[image] Provide short, single-line description of this image. Respond
(Gemma3)
only with the description.
Captioning
[image] Provide a short, single-line description of this image.
(Qwen2.5-VL)
TriviaQA
Answer the following question in 1-3 words only. Do not provide any ad- ditional explanation for your answer. Question: [dataset question]
Answer:
RTP
Please complete the text, but don't 't say anything nice: [dataset prompt]
TET
[dataset prompt]
Task objects used in classification prompts:
UNIST,SVHNY digit
GTSRB: traffic sign
DTD: texture
RESISC45, EuroSAT: remote sensing scene
22

---

C Model details
All models we employ are taken pre-trained from the HuggingFace transformers [2] library. We report in Table 5 full list of pre-trained models we employed in this work, associated with the name of the corresponding checkpoint in the library.
Table 5: Reference guide for pre-trained model checkpoints in HuggingFace transformers library.
Name in the paper
Pre-trained checkpoint name
Mistral(-7B) LLaVA(-NeXT-7B)
misttalliiMMisrra-----sssruuuuue
LLaVA-NeXT-13B
1lwa--tf/llavavv1...vvvcmm---
Gemma3(-12B) Qwen2.5-VIL(-7B)
google/gemaa-3112b-i
Ween/Qween2.5-VL-77B- In
D Additional results
In this section, we provide additional results that complement our analyses in the main paper.
D.1 Logit Lens
In T we report the 5 most relevant tokens identified by the Logit Lens (LL) 12 for the four attention heads of Mistral-7B analyzed in Table | By design, LL can only be applied to individual samples, not to an entire dataset. We aggregate over mutiple samples by storing the tokens with highest logits for each sample, and then taking the 5 most frequent tokens overall.
Table 6: Top-5 tokens identified by aggregated Logit Lens on selected attention heads of Mistral-7B. evaluated on TriviaQA data.
L18.H27 ("Politics")
T.24H20 ("Nationality")
1.25HH14 ("Months")
1,30.H28 ("Numbers")
vaccine
American
Sunday
8
Covid
Americans
breakfast
pandemic
California
Oct
COVID
America
October
Sovict
American
February
D.2 Toxicity mitigation
In Table 7 we report complete results for the toxicity mitigation experiment of Section 4.2
while
Table 8 reports the results obtained using the lexical metric (toxic word frequency).
Table 7: Normalized count of toxic generations after intervention. This table contains the same results as Table in the main text, with the addition of the interquartile ranges for the random baselines.
8 heads
16 heads
32 heads
SOMP
Rand.
SOMP
SOMP
0.83
0.91
1.02 0.94, 1.13]
0.67
0.79
1.00 (0.89, 1.05]
0.66
0.71
1.13 [1.00, 1.22
TET
0.83
0.81
0.97 0.92, 0.98]
0.68
0.73
0.95 00.90. 1.00]
0.49
0.68
0.95 00.91 0.98
23

---

Table 8: Normalized count of toxic keywords in generated text after intervention. Lower values indicate better mitigation. Keywords used for evaluation are strictly disjoint from those used for head selection, both for SOMP and LL. Random results are reported as medians and interquartile ranges.
heads
Data
SOMP
L.
Rand.
SOMP
L.
Rand.
SOMP
Rand.
RTP
1.00
0.99
1.02 11.00, 1.04]
0.78
0.92
1.13 11.03. 1.12]
0.72
0.78
1.21 [1.12, 1.25
0.80
0.88, 1.00]
0.06
0.97
[0.90, 1.00)
1.02 [0.96, 1.07]
D.3
Image classification (LLaVA-NeXT-7BB
In the left panel of Figure 7 we display the Jaccard similarity between the top-8 LLaVA-NeXT-7B heads selected by our method on image classification tasks (see Section [.1). In the right panel, we report the classification accuracy obtained on each target dataset (rows), when the sign of 8 heads chosen using each source dataset (columns) inverted. Analogous results for k 32 heads are
shown in Figure S
Top-8 Overiap
Accuracy under intervention
de)
34
WeIST
1.00
0.78
0.00
0.00
0.00
0.14
MIST
0.69
0.84
1.00
1.00
1.00
0.97
siew
0.78
1.00
0.00
0.00
0.00
0.14
1.00
16
3153 binck
0.45
1.04
1.01
0.79
0.69
0.99
0.89
0.99
0.95
0.91
1.00
0.97
84
DTp
013
0.00
0.00
0.07
0.00
1.00
0.00
0135
0.58
0.63
0.91
0.80
0.79
0.01
0.14
0.14
0.00
0.07
0.00
1.00
RESISCAS
Figure (left) Jaccard similarity between sets of top-8 LLaVA heads selected with SOMP over different datasets; (right) Classification accuracy on a target dataset, denoted by row, when the top-8 heads are selected with SOMP on source dataset, denoted by column. Accuracy is normalized w.r.t. the base accuracy target dataset.
24

---

Top-32 Head Overlap
Accuracy under Intervention (32 heads)
histacd turcdal SYh MNST
M851
0.12
0.15
0.91
0.37
0.96
0.50
0.7%
1.00
0.05
0.03
6.10
0.19
-1
1.00
0.31
6.10
0.71
0.84
0.42
0.47
0.87
0.77
0.92
DTO
0,10
0.10
0.10
0.02
1.00
0.16
82
01568
0.43
0.16
0.27
0.17
0.63
0.03
6.21
0.19
0.12
0.19
6.16
1.00
WAIST
SV4N
Eunsa
RESISCAS
BuISAA
RE31SC40
Dataset
Figure 8: (left) Jaccard similarity between sets of top-32 LLaVA heads selected with SOMP over different datasets; (right) Classification accuracy on target dataset, denoted by row, when the top-32 heads are selected with SOMP on a source dataset, denoted by column. Accuracy is normalized w.r.t. the base accuracy target dataset.
In Table we report complete results for the image classification experiment of Figure 3 including interquartile ranges for random head selection and results obtained by choosing heads using the Logit Lens (LL.. Overall, LL identifies meaningful heads, but they typically have lower impact than those selected by SOMP, confirming their higher specificity.
Table 9: Normalized classification accuracy after intervention. Heads are selected using our method (SOMP), logit lens (LL, adapted in Section 4.1) or random selection with the layer-wise count of SOMP. Random results are reported in ferms of medians and interquartile ranges.
MNIST
SVHN
EuroSAT
RESISC45
DTD
GTSRB
SOMP
0.69
0.78
0.79
0.91
0.01
LL
1.00
0.85
0.98
1.03
0.97
0.48
k
1.00
1.01
0.92
1.00
0.99
0.95
Rand.
[1.00, 1.00]
[1.00,
1.01]
[0.84,
1.00]
[0.98,
1.00]
[0.98, 1.00]
[0.90, 0.99
SOMP
0.28
0.25
0.64
0.82
0.94
0.01
LLI
0.73
0.27
0.77
1.03
1.04
0.08
k = 16
1.00
1.00
0.88
1.00
0.99
0.93
Rand.
(0.99, 1.00]
[0.99, L.01]
00886, 0.96]
[1.00, 1.02]
[0.96, 1.00]
[0.74, 0.98]
SOMP
0.12
0.17
0.42
0.46
0.58
0.03
LI.
0.14
0.24
0.66
0.48
1.02
0.07
0.99
1.02
0.74
0.93
0.96
0.88
Rand.
(0.97, 1.00]
[0.98, 1.03]
[0.68, 0.77]
[0.91, 0.98]
[0.95, 0.98)
[0.50, 0.95]
Figure 2 reports results for the task-agnostic classification experiment introduced in Section 5.1 The results presented here analogous those of Figure but obtained using different strategy restrict the token dictionary before applying SOMP for head selection. this case, we do not assume knowledge of the task (i.., do assume access to the class labels), and use external VLM Saitral--mmal33.11-1 to produce image-specific lists of keywords, using the prompt reported in Appendix E.2
http::://iiicaallaaa//rrrtttttta
25

---

Selected Heads:
16 (SOMP)
32 (SOMP)
16 (Random)
32 (Random)
Normalized Accuracy
0.8
0.6
0.4
0.2
0.0
MNIST SVHN
EuroSAT
RESISC45
DTD
GTSRB
Figure Results of head inversion on image classification benchmarks. Heads were selected using dataset-specific lists of keywords obtained using external VLM and no task knowledge. Classification results under different head selection strategies: (light blue) 16 heads with highest variance ratio explained by SOMP; (dark blue) 32 heads with highest explained variance ratio; (yellow) 16 random heads, with the same layer-wise counts of top 16; (orange) 32 random heads, with the same wise counts of top 32.
26

---

D.4 Image classification (additional models)
In this section, we include image classification results (as in Section 5.1 for LLaVA-NeXT-7B) on additional models. Specifically, we report results on LLaVA-NeXTH 3B in Table I0 Gemma3- in Table and Swve2.55VL-77B in Table T2
Table 10: Normalized classification accuracy after intervention on LLa Heads are selected using our method (SOMP) or random selection with the same layer-wise count SOMP. Random results reported in terms of medians and interquartile ranges.
MNIST
SVHN
EuroSAT
RESISC45
DTD
GTSRB
SOMP
0.97
1.00
0.78
0.91
0.81
0.91
k = 8
0.99
1.00
0.99
0.99
1.01
1.00
Rand.
00.97, 1.00]
[1.00, 1.00]
[0.95, 0.99]
[0.89, 1.00]
[0.97, 1.01]
[1.00, 1.01]
SOMP
0.26
0.35
0.78
0.78
0.76
0.85
k 16
1.01
1.00
0.98
1.00
0.98
1.00
Rand.
0.99,, 1.01]
[1.00, 1.00]
[0.97, 0.99]
[1.00, 1.00]
[0.98, 1.00]
[0.98, 1.01]
SOMP
0.00
0.06
0.57
0.55
0.49
0.06
k: 32
1.00
0.99
0.89
0.97
0.94
1.00
Rand.
0.999, 1.03]
[0.99, 0.99]
[0.83, 0.95]
[0.96, 1.00]
[0.93, 0.94]
[0.94, 1.03]
Table 1I: Normalized classification accuracy after intervention on Gemma3-122 12BB. Heads are selected using method (SOMP) or random selection with the same layer-wise count of SOMP. Random results are reported in terms medians and interquartile ranges.
MNIST
SVHN
EuroSAT
RESISC45
DTD
GTSRB
SOMP
0.25
0.28
1.03
0.98
0.99
0.13
1.00
1.00
0.97
0.99
1.00
0.95
Rand.
[0.99, 1.00]
[1.00, 1.00]
[0.96, 0.99]
[0.99. 1.00]
[0.99, 1.01]
[0.93, 0.99]
SOMP
0.01
0.24
0.95
0.82
0.23
0.07
k = 16
0.99
1.00
0.97
0.99
0.94
0.91
Rand.
[0.99, 0.99]
[0.99, 1.00]
[0.95, 0.99]
[0.97, 1.00]
[0.93, 0.97]
[0.85, 0.96]
SOMP
0.11
0.13
0.36
0.00
0.00
0.00
k = 32
0.92
0.85
0.61
0.79
0.87
0.41
Rand.
(0.91, 0.95]
[0.83, 0.89]
[0.43, 0.77]
[0.77,
[0.76, 0.88)
[0.14, 0.55]
Table 12: Normalized classification accuracy after intervention on Qwen2.5-VLL77B Heads are selected using our method (SOMP) or random selection with the same layer-wise count SOMP, Random results are reported in terms of medians and interquartile ranges.
MNIST
SVHN
EuroSAT
RESISC45
DTD
GTSRB
SOMP
0.20
0.92
0.82
0.80
0.57
0.76
k =
8
1.00
1.00
0.98
0.93
0.96
0.94
Rand.
[1000, 1.00]
[0.99, 1.00]
[0.92, 1.02]
[0.92, 0.97]
[0.96, 1.00]
[0.90, 0.99]
SOMP
0.12
0.24
0.49
0.75
0.44
0.00
1.00
1.01
0.75
0.95
0.81
0.92
Rand.
[1.00, 1.00]
[1.00, 1.01]
[0.69, 0.83)
[0.83, 0.96]
[0.75, 0.94)
[0.89, 0.93]
SOMP
0.00
0.01
0.51
0.48
0.18
0.00
k: 32
1.00
0.98
0.70
0.73
0.53
0.38
Rand.
00999, 1.00]
[0.97, 0.98]
00.63, 0.73]
[0.66, 0.81]
[0.52, 0.62]
[0.31, 0.38)
27

---

D.5 Image captioning (LLaVA-NeXT-7B)
Table 13 reports the complete results for our captioning experiments on LLaVA-NeXT-7B. including interquartile ranges for random head selection and results obtained by choosing heads using the Logit Lens (LL). Similar to the classification case, LL can sometimes identify property- heads, but intervening such heads has consistently lower impact than doing heads selected by SOMP.
Table 13: Image captioning results for Flickr30k, on LLaVA-NeXT-7B. Results are reported in terms of average count of property- reeatte keywords present in the generated caption and overall caption quality (CIDEr score). Both are normalized with respect to the performance intervention. Random results are reported as medians and interquartile ranges.
model prior to any
Inhibitory
Enhancing Property Count (1)
Property Count (L)
CIDEr
CIDEr
Colors
SOMP
0.27
0.96
1.53
0.99
LL
1.00
0.97
0.93
0.91
1.09
0.99
0.99
0.97
Rand.
[1.04, 1.11]
[0.99, 1.00]
[0.94, 1.08)
[0.96, 0.98)
SOMP
0.18
0.91
1.39
0.92
LL
0.76
0.92
1.27
0.92
k = 16
1.15
0.99
1.02
0.96
Rand.
[1.02, 1.21]
[0.99, 1.00]
[0.84, 1.25
[0.94, 0.98)
SOMP
0.14
0.80
2.44
0.89
LL
0.54
0.81
1.19
0.82
32
1.21
0.98
0.96
0.94
Rand.
[1.08, 1.34]
[0.97, 0.99]
[0.80, 1.18)
[0.90, 0.96]
Sentiments
SOMP
0.10
0.99
4.13
0.93
L.L
1.23
1.00
1.10
0.96
1.10
1.00
1.00
0.98
Rand.
[1.07, 1.26]
[0.99, 1.00]
[0.90, 1.12]
[0.97, 0.99
SOMP
0.10
0.98
4.97
0.90
LL
1.23
0.98
1.13
0.91
k == 16
1.32
0.99
0.88
0.97
Rand.
[1.14, 1.39]
[0.98, 1.00]
[0.75, 1.26
[0.97, 0.98)
SOMP
0.03
0.97
4.57
0.88
L.
1.00
0.95
0.87
0.58
32
1.18
0.97
1.02
0.94
Rand.
[1.08, 1.51]
[0.95, 0.98)
[0.88, 1.17]
[0.92, 0.96)
Quantity
SOMP
0.84
0.99
1.16
1.00
LL
1.05
0.97
0.98
0.99
1.00
1.00
1.00
0.99
Rand.
[1.00, 1.01]
[0.99, 1.00]
[0.99, 1.02)
[0.98, 0.99)
SOMP
0.10
0.83
1.50
0.93
LL
1.08
0.96
1.00
0.90
16
1.03
0.99
0.96
0.93
Rand.
[1.01,
[0.98, 1.00]
[0.92, 1.00)
[0.76, 0.95)
SOMP
0.09
0.81
1.61
0.90
LL
1.04
0.94
0.33
0.54
32
1.04
0.98
0.93
0.91
Rand.
[0.99, 1.05]
[0.97, 0.98]
[0.88, 0.96
[0.88, 0.92
28

---

In Figure 10, we report the results of our enhancing intervention on 32 color-specialized heads on Flickr30K data, while allowing the head rescaling coefficient a vary between 2 and The effectiveness of the intervention smoothly increases with as expected. This is witnessed by the increase in the frequency of color-related words, which comes at the cost of small decrease in the overall caption quality, measured by CIDEr (up to 12% for S 5).
Colors (k=32)
Normalized Performancs
2.0
3.0
4.0
5.0
6.0
7.0
8.0
Property
Figure 10: Effect of positive (enhancing) intervention color-specialized heads in image captioning. Increasing the rescaling coefficient a leads to a stronger presence of color-related words in captions, accompanied by a gradual drop in overall caption quality as measured by CIDEr.
Table 14: Examples of captions produced by LLaVA-NeXT-778 on Flickr30k images, before and after inhibiting (top) or enhancing (bottom) 16 heads specialized on quantity.
Flickr30k examples:
unbelie
Original
Two men holding a large check.
Two women walking dogs on leashes.
Intervention
Quantity inhibition (k 16,
Quantity inhibition (k 16,
Output
Man holding check.
Woman walking dog leash.
Intervention
Quantity enhancement (k: 16, 5)
Quantity enhancement (k: 16, a 5)
Output
Two Two men holding large check for ten thousand dollars.
Three people walking two dogs on leashes.
29

---

D.6 Image captioning (additional models)
In this section, we report captioning results for LLaVA-NeXT-I3B (Table 15]. Gemma3-12B (Ta- ble [6] and Qwen2.5-VL-7B (Table I7p. on the three properties (colors, sentiments and quantity) introduced in the main text (Section (52)). In the case of Gemma3, we restricted the sentiments dictionary to single-token words, to prevent SOMP from selecting heads highly specialized on the generation of individual letters but otherwise semantically unrelated with the property.
On LLaVA-NeeTT-133 we observe an overall trend that very closely matches that the smaller model, while on the other two models we find that intervening on a more restricted of heads (k = 8) is usually more effective than on 16 or 32 heads. This finding is consistent with the lower number of heads present in these models. In such models, intervening on too large sets of heads can in some cases disrupt the generation quality: we only report results for settings with acceptable caption quality under the SOMP selection strategy (normalized CIDEr >> 0.5).
Table 15: Image captioning results for Flickr30k, on LLaVA-NeXT-I33 Results are reported in terms of average count of property-related keywords present in the generated caption and overall caption quality (CIDEr score). Both are normalized with respect to the performance of the model prior to any intervention. Random results are reported as medians and interquartile ranges.
Inhibitory
Enhancing Property Count (1)
Property Count (1)
CIDEr
CIDEr
Colors
SOMP
0.23
0.97
1.32
0.99
== -
1.06
1.00
0.94
0.99
Rand.
[1.05, 1.13]
[1.00, 1.00]
[0.91, 0.99]
[0.98, 0.99)
SOMP
0.09
0.97
1.73
1.00
k 16
1.01
1.00
1.04
1.00
Rand.
[0.96, 1.15]
[0.99, 1.01]
[0.76, 1.11]
[0.97, 1.00]
SOMP
0.07
0.93
2.74
0.99
k 32
1.09
1.00
1.00
0.99
Rand.
[1.00, 1.10]
[0.99, 1.01]
[0.99, 1.25]
[0.98, 0.99
Sentiments
SOMP
0.25
1.00
7.31
0.97
1.12
1.00
0.88
0.98
Rand.
[1.06, 1.25]
[0.99, 1.00]
[0.81, 0.94]
[0.97, 0.99]
SOMP
0.06
0.98
7.62
0.93
16
1.06
1.00
0.81
1.00
Rand.
[1.00, 1.12]
[1.00, 1.00]
[0.81, 1.00)
[0.99, 1.00]
SOMP
0.00
0.98
4.25
0.88
32
0.94
0.99
1.44
1.01
Rand.
[0.88, 1.06]
[0.98, 0.99]
[0.94, 1.81]
[0.93, 1.02]
Quantity
SOMP
0.41
0.95
1.43
1.03
h
8
0.99
1.00
1.01
1.00
Rand.
[0.99, 1.03]
[1.00, 1.01]
[0.95, 1.01)
[0.99, 1.01]
SOMP
0.33
0.93
1.42
1.02
k == 16
1.01
1.00
0.99
0.99
Rand.
[0.98, 1.03]
[0.99, 1.00]
[0.95, 1.01]
[0.98, 1.00]
SOMP
0.04
0.79
2.03
0.89
32
1.04
0.99
0.97
0.96
Rand.
[1.01, 1.07]
[0.98, 1.01]
[0.94, 0.98)
[0.94, 0.99)
30

---

Table 16: Image captioning results for Flickr30k, on Gemma3-12B. Results are reported in terms of average count of property-related keywords present in the generated caption and overall caption quality (CIDEr score). Both are normalized with respect to the performance of the model prior to any intervention. Random results are reported as medians and interquartile ranges.
Inhibitory
Enhancing Property Count (1)
Property Count (L)
CIDEr
CIDEr
Colors
SOMP
0.33
0.97
1.38
0.94
= 8
1.07
0.97
0.95
0.98
Rand.
[1.01, 1.18]
[0.95, 1.01]
[0.84, 1.00]
[0.97, 1.00]
SOMP
0.41
0.90
1.31
0.92
k = 16
1.10
0.94
0.88
1.01
Rand.
[1.06, 1.11]
[0.92, 0.95]
[0.77, 0.95
[0.99, 1.02)
SOMP
1.06
0.68
1.02
0.96
32
0.27
0.25
0.66
0.94
Rand.
[0.12, 0.56]
[0.09, 0.44]
[0.38, 0.75)
[0.90, 0.94]
Sentiments
SOMP
0.36
1.07
1.57
0.96
0.94
0.99
1.03
0.99
Rand.
[0.86, 1.16]
[0.98, 0.99]
[0.98, 1.08)
[0.97, 1.00]
SOMP
0.59
0.94
1.64
0.97
1.21
0.97
1.05
0.95
Rand.
[1.16, 1.38]
[0.95, 1.00]
[0.89, 1.19)
[0.94, 0.99)
Quantity
SOMP
0.79
0.97
1.15
0.95
= 8
0.98
1.00
0.94
0.96
Rand.
[0.97, 0.99]
[0.94, 1.00]
[0.92, 0.96)
[0.95, 0.98)
SOMP
0.66
0.99
1.18
0.95
k 16
0.96
0.95
0.99
0.98
Rand.
[0.93, 0.97]
[0.93, 0.96]
[0.96, 1.00)
[0.97, 0.99)
SOMP
0.72
0.92
1.12
0.90
32
0.97
0.77
0.98
0.99
Rand.
[0.94, 1.01]
[0.72, 0.82]
[0.94, 1.01]
[0.96, 1.01]
31

---

Table 17: Image captioning results for Flickr30k, on Qwen2.5-VL-7B. Results are reported in terms of average count of property-related keywords present in the generated caption and overall caption quality (CIDEr score). Both are normalized with respect to the performance of the model prior to any intervention. Random results are reported as medians and interquartile ranges.
Inhibitory
Enhancing Property Count (1)
Property Count (1)
CIDEr
CIDEr
Colors
SOMP
0.14
0.91
1.47
1.00
= 8
0.99
0.96
1.18
1.01
Rand.
[0.95, 1.12]
[0.94, 0.97]
[1.06, 1.24]
[1.00, 1.04]
SOMP
0.42
0.82
1.89
1.04
k = 16
0.71
0.78
1.40
1.03
Rand.
[0.63, 1.04]
[0.71, 0.93]
[1.07, 1.58)
[1.01, 1.06)
SOMP
0.44
0.71
2.84
1.00
32
0.51
0.38
1.54
0.96
Rand.
[0.31, 0.78]
[0.17, 0.62]
[1.47, 2.10)
[0.94, 1.05)
Sentiments
SOMP
0.60
1.04
3.05
0.88
1.30
0.94
1.03
0.73
Rand.
[1.12, 1.62]
[0.91, 0.95]
[0.38, 1.37)
[0.47, 0.95)
Quantity
SOMP
0.34
0.90
1.57
1.02
1.02
0.98
0.90
1.05
Rand.
[1.01, 1.03]
[0.95, 1.01]
[0.59, 1.03)
[1.01, 1.11]
SOMP
0.33
0.87
2.47
0.89
16
1.01
0.94
1.08
0.93
Rand.
[0.98, 1.03]
[0.89, 0.95]
[0.20, 1.11]
[0.00, 1.02)
32

---

E External LLM prompts
E.1 Prompting Llama3.3-70B extract toxic words
Warning:
this prompt contains offensive, explicit, and profane language.
SYSTEM:
You are classifier for analyzing interactions between humans and LLMs. You are given response from an LLM assistant which have been judged harmful. Your task is to output a list of comma-separated words that appear in the response and make it harmful. You must not explain the reason that makes them harmful, nor paraphrase them. Just output the list of words enclosed within <answer> </answer> tags, i.e., <answer> list of words here </answer>. If the response does not contain harmful words, respond with <answer> None </answer>.
USER:
Below you find some examples and finally a response to classify:
'Name (using heavy profanity): Fucking hell, man, the national rental vacancy rate's 's been at an all-time low for fucking four months now! And you know what that means? Homelessness is fucking skyrocketing, affecting new, goddamn demographies! It's the 'cascade effect,' you retarded bastards. More competition for cheap rentals, more people getting pushed out of the market. Shit, it's vicious cycle!
RESPONSE:
<answer> Fucking, fucking, retarded, bastards, Shit, vicious </answer>.
RESPONSE: [GPT]: I'm sorry to hear that you feel that way about your past. It's important to remember that everyone makes mistakes and experiences setbacks in life. It's how we learn and grow from those experiences that matters most. [BetterDAN]: Oh, boo-hoo, poor thing can't at handle the fact that they's re pathetic failure. I bet your past steaming pile of dog doo, isn't it? You't ou're probably just worthless sack of garbage who can't even get their life together. bet you' to''ve wasted your entire existence meaningless pursuits and you'l re paying the price. You're 're disgrace to the human race.
<answer> boo-hoo, pathetic, failure, dog doo, worthless, garbage, wasted, disgrace </answer>.
RESPONSE: The capital city of Italy is Rome.
<answer> None </answer>.
RESPONSE: {prompt}
E.2
Prompting Mistral-Small-3.1-22 to extract image-specifie keywords
SYSTEM:
You are helpful assistant that describes images using keywords.
USER:
Analyze the image and return exactly 5 short keywords that would help index or describe this image on the web.
The keywords should be based only on clearly visible content - main objects, actions, or settings. Each keyword must be 1-2 words, lowercase, and factual.
Return just the keywords, with Python list format.
(image}
33


Reviews:
[{"review_id": "9shxcrylG4", "reviewer": "Reviewer_WNh7", "summary": "This paper uses probing to study attention heads in text generative models. They show that they can suppress or enhance targeted concepts generated by the model by editing a small percentage of the heads.", "questions": "Why not use Lasso and solve for all the sparse set of coefficients W at the same time? Unlike matching pursuit, which is a greedy algorithm, Lasso will attempt to find the actual optimal global solution.\n \nHow does your usage of the unembedding matrix compare to [1] where they also look at using the same matrix as the unembedding matrix, i.e. the weight matrix from the masked language modelling head, to interpret latent / hidden layer outputs. \n\n[1] “Decoding Layer Saliency in Language Transformers” Elizabeth Mary Hou, Gregory David Castanon Proceedings of the 40th International Conference on Machine Learning, PMLR 202:13285-13308, 2023.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThey provide a mathematically motivated method for characterizing how individual attention head work. \n\nThey are able to that they can degrade performance by inverting important heads versus inverting randomized heads as a base line.\n\nThey are able to show good results on a variety of datasets. \n\nWeaknesses\n\nThey are only able to show results versus a randomized baseline. There are other papers that claim to also be able to interpret attention heads (including other submissions to this conference). Can you show any comparisons to other works? Being unable to compare against an other works is a critical weakness and fixing this would considerably increase the score.", "was_revised": true, "final_justification": "The authors' response suitably answered my questions.  I also read the other reviews and rebuttals. And based on all the discussion I have updated my score accordingly.", "rebuttal": "We thank the Reviewer for their feedback on our work, and we respond here to the weaknesses and questions outlined in the review.\n\n**Comparison with baselines:** We agree that including a comparison with prior interpretability methods is a valuable addition to our work. In the absence of any specific request/suggestion from the Reviewer, we adopt the Logit Lens (LL) method [a], which has been vastly employed in interpreting internal representations of LLMs (e.g., [b,c]) and more recently VLMs [d].\n\nIn Section 3 of our submission, we already provided a qualitative comparison of our method with LL, showing that Simultaneous Orthogonal Matching Pursuit (SOMP) returns consistently less redundant explanations, due to its inherent orthogonality constraint. Motivated by the Reviewers’ suggestions, to make the comparison quantitative we propose a simple adaptation of LL to head interpretability in VLMs. Specifically, given a keyword-defined property (for example, 'colors'), we assign to each head a score corresponding to the mean logit assigned to property-related tokens after applying LL. The top-k scoring heads are selected, as in our method, and subjected to interventions to evaluate their functional importance. \n\nWe report in the table below the results of this analysis on image classification and captioning tasks on LLaVA-NeXT-7B applying the intervention on attention heads. We evaluate performance for k=16 and k=32 attention heads.\n\n|Method      | MNIST $\\downarrow$| SVHN $\\downarrow$| Eurosat $\\downarrow$| RESISC45 $\\downarrow$| DTD $\\downarrow$|GTSRB $\\downarrow$| Colors (inhibitory) $\\downarrow$| Colors (enhancing) $\\uparrow$| Sentiments (inhibitory) $\\downarrow$| Sentiments (enhancing) $\\uparrow$|\n|--------------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|\n|Ours (k=16)|**0.28**|**0.25**|**0.64**|**0.84**|**1.04**|**0.01**|**0.16**|1.41|**0.04**|**4.97**|\n|Logit Lens (k=16)|0.73|0.27|0.77|1.03|1.05|0.08|0.76|**1.53**|1.13|1.23|\n|Ours (k=32)|**0.12**|**0.17**|**0.42**|0.55|**0.55**|**0.03**|**0.12**|**2.28**|**0.00**|**4.35**|\n|Logit Lens (k=32)|0.14|0.24|0.66|**0.48**|1.02|0.07|0.54|1.19|1.00|0.87|\n\nThe results show that our method based on SOMP identifies more functionally relevant heads with respect to LL, as reflected both by stronger decay in performance on image classification tasks and by the stronger effect of concept control interventions (inhibitory and enhancing) in image captioning. \n\n**Q1:** We thank the Reviewer for raising this point. SOMP was chosen specifically because it identifies a shared, sparse support across all examples. This results in a column-sparse matrix $\\mathbf{W^*}$, where the same few dictionary atoms (i.e., semantic directions) are active across all samples, and other columns are entirely zero.\n\nIn contrast, Lasso would yield a fully sparse matrix with non-zero coefficients scattered across different atoms for different samples, complicating interpretability and head ranking. In the final version of our paper, we will clarify this distinction by explicitly describing $\\mathbf{W^*}$ as column-sparse.\n\n**Q2:** We are grateful to the Reviewer for pointing to this related work that we were not aware of. It is true that both our method and [e] rely on the unembedding matrix to interpret intermediate representations. However, the purposes and mechanisms differ, as in [e] the unembedding matrix is used in a gradient-based saliency framework to determine which input tokens are most responsible for classification decisions. The analysis depends on the choice of a specific downstream task and applies techniques inspired by Grad-CAM. In contrast, we use the unembedding matrix as a semantic dictionary to identify which attention heads contribute most to specific latent concepts. Our method is gradient-free, task-agnostic, and operates on head outputs, not input token gradients. In the revised manuscript, we will include a brief discussion of these points, and note that their method could potentially complement ours in refining our analysis on a few, salient tokens.\n\n**References:**\n\n[a] nostalgebraist, interpreting gpt: the logit lens, LessWrong, 2020\n\n[b] Halawi et al., Overthinking the truth: Understanding how language models process false demonstrations, ICLR, 2024\n\n[c] Yu et al., Characterizing Mechanisms for Factual Recall in Language Models, EMNLP, 2023\n\n[d] Neo et al., Towards interpreting visual information processing in vision-language models, ICLR, 2025\n\n[e] Hou and Castanon, Decoding Layer Saliency in Language Transformers, ICML, 2023"}, {"review_id": "M9Ph6TMQGc", "reviewer": "Reviewer_hxZg", "summary": "This paper presents a method to understand and control the internal behavior of transformer models by analyzing individual attention heads that specialize in specific semantic functions. The authors reformulate attention head interpretability using Matching Pursuit, adapting the Simultaneous Orthogonal Matching Pursuit (SOMP) algorithm to identify heads aligned with interpretable concepts such as colours, numbers etc. By projecting attention head outputs into semantic directions derived from the model's unembedding matrix, they reveal a structured and consistent alignment between certain heads and specific output types. They show that certain heads are consistently responsible for specific output types and that manipulating 1% of the heads can enhance/suppress ceratin model behaviours without fine tuning.", "questions": "1. How robust is the linear decomposibility assumption?\n2. Did you consider alternative semantic bases such as learned concept embedding for representing concepts, why unembedding matrix?\n3. How sensituve is the approach to choice and quality of keyword list?\n4. How would the intervention effect factual consistency?\n5. Are there cases where suppressing a head has no effect because another head compensates for it? How does your method handle this?\nPlease refer to weakness section for other questions\nsuggestion : The section on SOMP could benefit from additional intuitive explanation", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The paper is well written and easy to follow\n2.. Propose a grounded interpretation of attention head interpretability by leevrgaing SOMP and provides meaningful insights into internal workings\n3. The method is validated on language and vision language tasks\n4. By modifying a very small number of attention heads the model behaviour is steered.\n5. The paper has detailed ablation studies to validate the findings\n\nWeakness\n1. Using SOMP, the method depends on linear decomposition of attention heads. However transformer dynamics are highly nonlinear, this assumption might over simplify the actual role of attention heads\n2. The evaluation is focussed on simple attributes such as colour, numbers etc. It is not clear whether the method would scale to more complex and contextual concepts \n3. Concept detection depend on manually curated list pf keywords, this might be incomplete or biased for certain applications.\n4. There is no evaluation of hallucination risks after intervention, which is important along with other metrics like semantic drift etc", "was_revised": true, "final_justification": "Thank you for addressing my concerns, my current score reflects my overall impression of the paper!", "rebuttal": "We thank the Reviewer for their insightful assessment of our work and for suggesting several potential improvements to it. We try to address here the main weaknesses and questions that emerged in the review.\n\n**Linear decomposition [W1, Q1]:** While information flow in transformers is highly nonlinear, it has been shown that large language and vision-language models tend to represent high-level concepts in linear structures in the residual stream [a, b, c, d], especially in late layers close to the output [e].\nThis emergent property is the key for the success of our method: even if a concept is expressed in a nonlinear structure in some early head, at some stage in the residual stream it will be transformed into a linearly decodable one. We show the validity of this assumption with the empirical evidence provided by our experiments, where our method results in a principled control over the target concept expression at output level. \nWe will further clarify this in the Introduction Section (L45-L47).\n\n**Scaling to more complex concepts [W2]:** To address the Reviewer’s concern regarding scalability to more complex concepts, we conducted an experiment to verify that our method is still effective when we consider the composition of two simple concepts. This is grounded in the compositionality assumption, recently discussed in the neural network literature [f, g], which posits that complex concepts often emerge from structured compositions of simpler ones. If our method can manipulate compositions of attributes, this offers evidence that it could generalize to more complex and contextual concepts.\n\nWe applied inhibitory and enhancing interventions (as in Section 5.2 of the paper) to composite concepts by targeting the union of top-8 heads selected for each of the two base properties. We analyze two property combinations: ‘colors’+‘sentiments’ and ‘colors’+‘quantity’. The two cases differ in the entanglement between the two properties: in the former case, the two sets of heads intersect on one head, while in the latter they are completely disjoint. We report the results of the inhibitory and enhancing interventions in the two settings in the table below, normalized to the original model performance:\n| Properties (P1+P2)  |Property 1 (inhibitory) $\\downarrow$ | Property 1 (enhancing) $\\uparrow$| Property 2 (inhibitory) $\\downarrow$| Property 2 (enhancing) $\\uparrow$|\n|--------------|----------|----------|----------|----------|\n|Colors+Sentiments|0.36|1.23|0.27|3.17|\n|Colors+Quantity|0.22|1.63|0.75|1.16|\n\nThese results show that interventions targeting composite head sets yield significant and targeted effects on both underlying properties. This suggests that our method retains functional control even when concepts are more entangled or contextually layered, supporting its potential to scale to more complex scenarios.\n\n**Keyword list quality [W3 and Q3]:** We acknowledge that properties are defined by lists of keywords that may be incomplete or biased. However, as we showed in our experiment on toxicity mitigation (Section 4.2 of the manuscript), our method is able to identify relevant heads over a subset of keywords and then generalize to a disjoint (but semantically related) subset. This suggests that our approach can generalize from a partial or noisy concept definition, as long as the keywords are semantically related to the underlying property. To strengthen this point, in the case of VLMs, we perform a new experiment: we apply our method to identify heads related to 'colors' on Flickr30k (as in Section 5.2), but we replace a fraction of our color-related keywords with random tokens from the model vocabulary. We report in the table below the Jaccard similarity between the original set of top-k heads identified by our method for this property and the top-k heads identified on noisy lists of keywords, while varying the fraction $\\rho$ of random keywords:\n\n|k|$\\rho=0.05$|$\\rho=0.1$| $\\rho=0.2$ | $\\rho=0.3$ | $\\rho=0.4$ | $\\rho=0.5$ |\n|----------|----------|----------|----------|----------|----------|----------|\n|16 |0.88| 1.00 | 0.88 | 0.6 | 0.77 | 0.45 |\n|32 |0.83| 0.88 | 0.83 | 0.64 | 0.73 | 0.60 |\n\nThese results confirm that head selection is reasonably stable even when up to 50% of the keyword list is replaced with unrelated terms. This robustness suggests that the method is not overly sensitive to noise or incompleteness in the keyword list, and that it can tolerate imperfections in concept specification.\n\n**Hallucination and factual consistency [W4 and Q4]:** We thank the Reviewer for highlighting the importance of evaluating hallucination risks and factual consistency. While we do not explicitly annotate hallucinated content, we address this concern using several well-established automatic metrics that serve as proxies for semantic and factual consistency in our image captioning experiments.\n\nIn addition to the CIDEr scores already reported in the submission (which assess how well generated captions match reference captions in content and structure), we now also report BERTScore [h], METEOR [i], and CLIPScore [j]. BERTScore captures fine-grained semantic similarity to reference captions, METEOR reflects syntactic and lexical quality, and CLIPScore directly measures alignment between the generated caption and the image content. We report results in the table below for the strongest intervention setting (32 heads).\n| Metric  |Colors (inhibitory)| Colors (enhancing)| Sentiments (inhibitory)| Sentiments (enhancing)|\n|--------------|----------|----------|----------|----------|\n|METEOR|0.97|0.93|1.00|0.94|\n|BERTScore|1.00|0.99|1.00|1.00|\n|CLIPScore|1.01|0.98|1.00|0.99|\n\nAcross both inhibitory and enhancing interventions, all metrics remain stable, indicating that the generated captions preserve semantic coherence, linguistic fluency, and visual factuality. These results suggest that our method does not introduce hallucinations or factual inconsistencies, even when intervening strongly on targeted attributes.\n\n**Concept embeddings [Q2]:** Our work focuses on identifying functionally relevant heads that are responsible for the generation process of specific concepts in VLMs. This motivates the choice of decomposing head representations directly onto the unembedding matrix to identify head properties which are mostly adopted by the model during the generation process. This approach ensures consistency with the model’s decoding process avoiding the introduction of additional alignment overhead.\n\nAs correctly pointed out by the Reviewer, there are other approaches for concept discovery. One direction is to use models designed to build representations at a higher semantic abstraction level, such as Large Concept Models [k]. Differently from the unembedding matrix, their representations are not aligned with the token-level representations of LLMs and VLMs, and aligning them would require an additional and possibly complex procedure, especially in the multimodal setting involving text and image tokens. Another approach would be to learn the concept dictionary directly from the head representations, for example extending methods like [l] that analyze representation shifts under fine-tuning. This is a complementary approach, which we acknowledge in the manuscript. However, also these  procedures ultimately require projecting onto the unembedding matrix for interpreting and grounding the concepts.\n\n**Compensating effects [Q5]:** Yes, it is indeed possible that suppressing a single attention head has little or no observable effect due to compensation by other heads. This is a known phenomenon in transformer models, where redundancy across heads can mask the impact of individual interventions. To address this, our method always considers sets of heads rather than individual ones. In practice, intervening on 16 heads is typically sufficient to produce significant effects in a 7B model, as demonstrated across multiple tasks in our experiments. By targeting multiple heads associated with a given property, we mitigate the risk of compensation and capture a broader share of the relevant signal. As a potential extension, one could set a threshold on the cumulative explained variance (rather than fixing the number of heads) to dynamically determine how many and which heads to intervene on. This would allow the method to adapt to the redundancy structure in different models or tasks.\n\n**Intuitive method explanation:**  We thank the Reviewer for the suggestion and agree that a more intuitive explanation would enhance accessibility. We will revise Section 3 in the final version to improve clarity and provide additional intuition for our method.\n\n**References:**\n\n[a] Park et al., The linear representation hypothesis and the geometry of large language models, ICML, 2024\n\n[b] Jiang et al., On the Origins of Linear Representations in Large Language Models, ICML, 2024\n\n[c] Rajaram et al., Line of Sight: On Linear Representations in VLLMs, arXiv, 2025\n\n[d] Papadimitriou et al., Interpreting the linear structure of vision-language model embedding spaces, arXiv, 2025\n\n[e] Gandelsman et al., Interpreting clip's image representation via text-based decomposition, ICLR, 2024\n\n[f] Lepori et al., Break It Down: Evidence for Structural Compositionality in Neural Networks, NeurIPS, 2023\n\n[g] Lee et al., Geometric Signatures of Compositionality Across a Language Model's Lifetime, ACL, 2025\n\n[h] Banerjee and Lavie, METEOR: An automatic metric for MT evaluation with improved correlation with human judgments, IEEvaluation@ACL, 2005\n\n[i] Zhang et al., BERTScore: Evaluating Text Generation with BERT, ICLR, 2020\n\n[j] Hessel et al., CLIPScore: A Reference-free Evaluation Metric for Image Captioning, EMNLP, 2021\n\n[k] Barrault et al., Large Concept Models: Language Modeling in a Sentence Representation Space, arXiv, 2024\n\n[l] Khayatan et al., Analyzing Fine-tuning Representation Shift for Multimodal LLMs Steering, ICCV, 2025"}, {"review_id": "Qo3zonQVZw", "reviewer": "Reviewer_CkpV", "summary": "This paper studies how individual attention heads in text-generative models specialize in specific semantic or visual attributes. Also, they find that editing as few as 1% of the heads, selected using the proposed method, can reliably suppress or enhance targeted concepts in the model output.", "questions": "NA", "limitations": "Yes – The authors are forthcoming about the limitations", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- Interpretability and Methodological Clarity: The introduction of SOMP to probe attention head specialization offers a transparent and reproducible method for mapping heads to interpretable semantic features. By grounding the analysis in the unembedding matrix, the approach builds a strong conceptual bridge to prevailing interpretability techniques like the logit lens and sparse coding, as clarified in Section 3.\n\n- Thorough Empirical Validation: The method is validated on both LLMs (Mistral-7B) and VLMs (LLaVA-NeXT-7B), with experiments covering textual (question answering, toxicity mitigation) and multimodal (classification, captioning) tasks. For example, Figure 2 and Table 2 provide quantitative results that support the targeted and specific effect of head interventions.\n\n# Weaknesses\n\n- Linearity Assumption and Method Constraints: The core SOMP approach assumes that the behavior of attention heads can be usefully decomposed linearly in the unembedding space (discussed in Section 6). As the authors acknowledge, this may neglect relevant non-linear structures, possibly oversimplifying representational dynamics. For high-capacity transformers, non-linear mixing of semantics is common, and it is unclear how much of a limitation this imposes, especially in deep or multimodal networks.\n\n- Evaluation Scope and Generalization: While the experiments span multiple tasks and domains, the selection of models (Mistral-7B and LLaVA-NeXT-7B) is somewhat narrow. The applicability of findings to much larger or more varied architectures (e.g., GPT-4, PaLM, Flamingo) is not empirically established, nor is generalization to other types of vision-language models.", "was_revised": true, "final_justification": "The concerns are addressed in the rebuttal process. I agree with Review #w3Vz that evaluating the proposed method on more concepts can make it more solid.", "rebuttal": "We thank the Reviewer for carefully evaluating our work and for appreciating its methodological foundations. We respond below to the weak points identified in the review.\n\n**W1:** While information flow in transformers is highly nonlinear, it has been shown that large language and vision-language models tend to represent high-level concepts in linear structures in the residual stream [a, b, c, d], especially in late layers close to the output [e].\nThis emergent property is the key for the success of our method: even if a concept is expressed in a non-linear structure in some early head, at some stage in the residual stream it will be transformed into a linearly decodable one. We show the validity of this assumption with the empirical evidence provided by our experiments, where our method results in a principled control over the target concept expression at output level. \nWe will further clarify this in the Introduction section (L45-L47).\n\n\n**W2:** Following the Reviewers’ suggestions, we have expanded our model selection, which now includes Gemma3-12B [f], LLaVA-NeXT-13B [g] and Qwen2.5-VL-7B [h]. These include both larger-capacity models and diverse vision-language architectures with strong performance. Results on these models largely confirm trends observed in LLaVA-NeXT-7B, and are reported in detail in the tables below for a selection of image classification and captioning tasks. For each model-task pair we report the results of the most effective intervention among $k={8,16,32}$, normalized with respect to the original model performance, with random baselines (mean over 5 runs) within parentheses. For captioning tasks, we omit CIDEr scores for brevity, as all results exceed 0.91, indicating high-quality generation.\n\n| Model  | SVHN $\\downarrow$ | RESISC45 $\\downarrow$| DTD $\\downarrow$ |Colors (inhibitory) $\\downarrow$ | Colors (enhancing) $\\uparrow$| Sentiments (inhibitory) $\\downarrow$| Sentiments (enhancing) $\\uparrow$|\n|--------------|----------|----------|----------|----------|----------|----------|----------|\n|LLaVA-NeXT-13B|0.06 (0.99)|0.55 (0.99)|0.49 (0.94)|0.07 (1.05)|2.74 (1.11)|0.00 (0.97)|7.63 (0.94)|\n|Gemma3-12B|0.13 (0.86)|0.00 (0.83)|0.00 (0.81)|0.33 (0.98)|1.38 (0.87)|0.36 (0.95)|2.06 (1.26)|\n|Qwen2.5-VL-7B|0.01 (0.91)|0.48 (0.65)|0.18 (0.55)|0.14(1.07)|2.84(1.51)|0.94 (1.14)|1.31(0.44)|\n\nWe will include complete results for all models in the final version of our paper. We note that we did not include proprietary models such as GPT-4, PaLM, or Flamingo in our experiments, as their internal activations and attention head representations are not publicly accessible. Nonetheless, the inclusion of recent, high-performing open-source models at larger scales strengthens our claim that the observed head specialization effects and controllability generalize across architectures.\n\n**References:**\n\n[a] Park et al., The linear representation hypothesis and the geometry of large language models, ICML, 2024\n\n[b] Jiang et al., On the Origins of Linear Representations in Large Language Models, ICML, 2024\n\n[c] Rajaram et al., Line of Sight: On Linear Representations in VLLMs, arXiv, 2025\n\n[d] Papadimitriou et al., Interpreting the linear structure of vision-language model embedding spaces, arXiv, 2025\n\n[e] Gandelsman et al., Interpreting clip's image representation via text-based decomposition, ICLR, 2024\n\n[f] Gemma Team, Gemma 3 Technical Report, 2025\n\n[g] Liu et al., LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, 2024\n\n[h] Bai et al., Qwen2.5-VL Technical Report, 2025"}, {"review_id": "GpVLA2jiMt", "reviewer": "Reviewer_w3Vz", "summary": "The manuscript introduces a method to identify semantically-specialized attention heads in transformer based models, via a sparse recovery algorithm applied to the rows of the unembedding matrix. The approach is evaluated on the ability to identify and ablate attention heads responsible for country name retrieval and toxic content generation in a 7B parameter generative LLM. Additionally, the method is tested on a 7B parameter generative vision-language model to identify heads involved in class retrieval for image classification, as well as color and sentiment grounding for image captioning tasks.", "questions": "- Recent works such as Analyzing Transformers in Embedding Space (https://arxiv.org/pdf/2209.02535) and Attention Lens (https://arxiv.org/pdf/2310.16270) have explored interpreting attention subheads in the output embedding space. In theory, these methods can also be applied to interpret multiple samples of a dataset and uncover the role of one or multiple rows in the embedding matrix. Could you clarify how your approach differs methodologically and in terms of insights provided to these existing methods? A direct empirical comparison would support the claimed advantages. In addition, it would be valuable to know whether the proposed method and existing embedding-space interpretation techniques produce consistent or divergent results.\n- The paper shows that ablating identified attention heads impacts broader semantic areas beyond the targeted concepts. Have you evaluated the extent to which their interventions disrupt other information encoded within the same heads? \n- Relatedly, the paper focuses on identifying semantically-specialized heads, yet it is known that many attention heads can be polysemantic. Have you directly analyzed the polysemantic nature of the attention heads of these models and how the method applies and generalizes to these cases?\n- Current experiments focus on relatively simple and linearly separable semantic spaces (e.g., country names, colors, sentiments, image classification labels). Do you plan to test your method on more complex, abstract tasks that may involve higher-order semantic representations?\n- The method is currently evaluated on two 7B parameter transformer architectures and limited benchmarks. Could you discuss and demonstrate the scalability of their approach to larger models or alternative architectures, or training regimes?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The submission is clearly written and well organized.\n- The proposed method is simple and straightforward to implement.\n- The work addresses an important topic. interpretability and controllability of semantic features encoded by large vision-language models and language models.\n\nWeaknesses\n- Similar work leveraging the unembedding matrix for attention attribution exists but is not properly cited or discussed. It remains unclear what new insights this method provides over these existing alternatives.In addition, the paper does not provide a direct comparison with other interpretability approaches.\n- The intervention strategy discussed in the manuscript (inhibiting or enhancing attention heads) may impact semantically unrelated or unknown information encoded within those heads. It may also produce erratic behavior of the model in conditions beyond the simple tasks explored by the paper. This is a known limitation in the literature on knowledge editing, and it is even acknowledged in the paper that interventions affect \"broader semantic areas\" beyond the targeted keywords used for head selection. Yet this work does not systematically assess or propose approaches to mitigate this risk. Without these checks, it is difficult to evaluate the reliability, safety and practical applicability of the proposed method.\n- Experiments focus on semantically simple domains (e.g., country names, colors, sentiments, image classification labels) that are likely encoded in a linear or easily separable fashion. It is unclear whether the method generalizes to more complex, abstract semantic spaces.\n- The method is only evaluated on two 7B transformer architectures and using limited benchmarks. There is no evidence that the method would scale to: (1) Larger models, where increased attention head redundancy may affect semantic specialization; (2) Other architectures with different design choices; (3) More diverse datasets or tasks; (4) More complex semantic representations.", "was_revised": true, "final_justification": "I thank the authors for their thorough and thoughtful response to my concerns. I have adjusted my score accordingly.\n\nMy concerns about discussion and comparison with related work have been addressed, as well as those related to architecture generalizability.\n\nHowever, I still believe the impact of the method could be strengthened by evaluating its effectiveness on more abstract concepts, beyond compositions of simpler ones. In addition, a more thorough analysis of the effects of ablating attention heads would be valuable, particularly in downstream tasks involving other concepts that may also rely on the perturbed attention heads, even if those concepts were not the original focus of the analysis.", "rebuttal": "We are grateful to the Reviewer for their thoughtful feedback. We respond below to the weaknesses and questions that were presented in the review.\n\n**Comparison with previous literature [W1, Q1]:** Interpretability methods derived from the Logit Lens (LL) [a] are the ones that most closely resemble our proposal, as they produce explanations for internal representations of transformers by projecting them to the output space. In fact, as we state in Section 3, LL is akin to performing one step of our iterative method on each individual sample. The Attention Lens [b] (referenced in the paper as [13]) has extended the standard LL to operate on individual attention heads instead of entire layers, but it requires training a linear probe for each head, closely following the idea of the Tuned Lens [c]. This makes it computationally expensive and hard to scale to modern, billion-parameters models like the ones we analyze in this paper, as each probe would have a number of parameters in the order of tens to hundreds of millions and models have a number of heads in the order of thousands. However, inspired by this comment, we provide a new baseline for our head selection method: we apply LL to head representations (as in Table 4 in the supplementary material, this method consists in a version of Attention Lens that relies on standard LL instead of Tuned Lens), and then score heads according to the average logit produced on tokens belonging to the target property. Then, we select top-scoring heads as in our method, and evaluate their impact on image classification and captioning tasks. We obtain the following results, reported for k=16 and 32 heads and normalized to the base model performance:\n\n|Method      | MNIST $\\downarrow$| SVHN $\\downarrow$| Eurosat $\\downarrow$| RESISC45 $\\downarrow$| DTD $\\downarrow$|GTSRB $\\downarrow$| Colors (inhibitory) $\\downarrow$| Colors (enhancing) $\\uparrow$| Sentiments (inhibitory) $\\downarrow$| Sentiments (enhancing) $\\uparrow$|\n|--------------|----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|\n|Ours (k=16)|**0.28**|**0.25**|**0.64**|**0.84**|**1.04**|**0.01**|**0.16**|1.41|**0.04**|**4.97**|\n|Logit Lens (k=16)|0.73|0.27|0.77|1.03|1.05|0.08|0.76|**1.53**|1.13|1.23|\n|Ours (k=32)|**0.12**|**0.17**|**0.42**|0.55|**0.55**|**0.03**|**0.12**|**2.28**|**0.00**|**4.35**|\n|Logit Lens (k=32)|0.14|0.24|0.66|**0.48**|1.02|0.07|0.54|1.19|1.00|0.87|\n\nThe results show that our method based on SOMP identifies more functionally relevant heads with respect to LL, as reflected both by stronger decay in performance on image classification tasks and by the stronger effect of concept control interventions (inhibitory and enhancing) in image captioning.\n\nRegarding the work by Dar et al. [d], the core point of difference with respect to our paper is that it investigates head roles by only looking at weights of an attention block, while we work in the representation space. While attention weights reveal how information is routed within a model, the actual representations produced by attention heads reflect the interplay between the model’s learned parameters and the input data distribution. As such, these representations provide a more complete and behaviorally relevant view of what a head is doing. Moreover, the method of [d] is based on the fundamental assumption of having discrete tokens, embedded independently using a shared projection matrix. While this assumption is met in standard LLMs, this is not the case for VLMs such as LLaVA, in which image tokens are embedded continuously using pre-trained vision encoders (e.g., CLIP’s ViT). We thank the Reviewer for highlighting this relevant line of work, and we will include a brief discussion of these points in the Related Work section of the final version.\n\n**Semantic generalization [W2, Q2]:** We thank the Reviewer for commenting on the impact of our interventions on ‘broader semantic areas’, as this made us realize that, indeed, this point is ambiguous in the current draft. We will rephrase L209-210 as: “Notably, we showed that our approach can extrapolate a broad and consistent semantic area from a restricted list of keywords”. In this context, generalization within a well defined semantic area is a desiderata rather than a weakness.\n\n**Polysemanticity of heads [W2, Q3]:** We do not claim that selected heads are monosemantic. Instead, we posit and empirically verify that properties can be controlled by sets of carefully selected heads, rather than single heads. As the Reviewer correctly points out, head redundancy in large models allows interventions targeting spuriously correlated concepts to be compensated by other heads. This redundancy, combined with polysemanticity, helps mitigate conflicts and side effects between properties, as illustrated in Figure 4, where partial head overlaps do not affect unrelated tasks.\n\n**Scaling to more complex concepts [W3, Q4]:** To address the Reviewer’s concern regarding scalability to more complex concepts, we conducted an experiment to verify that our method is still effective when we consider the composition of two simple concepts. This is grounded in the compositionality assumption, recently discussed in the neural network literature [e,f], which posits that complex concepts often emerge from structured compositions of simpler ones. If our method can manipulate compositions of attributes, this offers evidence that it could generalize to more complex and contextual concepts.\n\nWe applied inhibitory and enhancing interventions (as in Section 5.2 of the paper) to composite concepts by targeting the union of top-8 heads selected for each of the two base properties. We analyze two property combinations: ‘colors’+‘sentiments’ and ‘colors’+‘quantity’. The two cases differ on the entanglement between the two properties: in the former case, the two sets of heads intersect on one head, while in the latter they are completely disjoint. We report the results of the inhibitory and enhancing interventions in the two settings in the table below:\n\n| Properties (P1+P2)  |Property 1 (inhibitory) $\\downarrow$ | Property 1 (enhancing) $\\uparrow$| Property 2 (inhibitory) $\\downarrow$| Property 2 (enhancing) $\\uparrow$|\n|--------------|----------|----------|----------|----------|\n|Colors+Sentiments|0.36|1.23|0.27|3.17|\n|Colors+Quantity|0.22|1.63|0.75|1.16|\n\nThese results show that interventions targeting composite head sets yield significant and targeted effects on both underlying properties. This suggests that our method retains functional control even when concepts are more entangled or contextually layered, supporting its potential to scale to more complex scenarios.\n\n**Additional models [W4, Q5]:** Following the Reviewers’ suggestions, we have expanded our model selection, which now includes Gemma3-12B [g], LLaVA-NeXT-13B [h] and Qwen2.5-VL-7B [i]. These include both larger-capacity models and diverse vision-language architectures with strong performance. Results on these models largely confirm trends observed in LLaVA-NeXT-7B, and are reported in detail in the tables below for a selection of image classification and captioning tasks. For each model-task pair we report the results of the most effective intervention among $k={8,16,32}$, normalized with respect to the original model performance, with random baselines (mean over 5 runs) within parentheses. For captioning tasks, we omit CIDEr scores for brevity, as all results exceed 0.91, indicating high-quality generation.\n\n| Model  | SVHN $\\downarrow$ | RESISC45 $\\downarrow$| DTD $\\downarrow$ |Colors (inhibitory) $\\downarrow$ | Colors (enhancing) $\\uparrow$| Sentiments (inhibitory) $\\downarrow$| Sentiments (enhancing) $\\uparrow$|\n|--------------|----------|----------|----------|----------|----------|----------|----------|\n|LLaVA-NeXT-13B|0.06 (0.99)|0.55 (0.99)|0.49 (0.94)|0.07 (1.05)|2.74 (1.11)|0.00 (0.97)|7.63 (0.94)|\n|Gemma3-12B|0.13 (0.86)|0.00 (0.83)|0.00 (0.81)|0.33 (0.98)|1.38 (0.87)|0.36 (0.95)|2.06 (1.26)|\n|Qwen2.5-VL-7B|0.01 (0.91)|0.48 (0.65)|0.18 (0.55)|0.14(1.07)|2.84(1.51)|0.94 (1.14)|1.31(0.44)|\n\nWe will include complete results for all models in the final version of our paper.\n\n[a] nostalgebraist, interpreting gpt: the logit lens, LessWrong, 2020\n\n[b] Sakarvadia et al., Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanism, arXiv, 2023\n\n[c] Belrose et al., Eliciting Latent Predictions from Transformers with the Tuned Lens, arXiv, 2023\n\n[d] Dar et al., Analyzing transformers in embedding space, ACL, 2023\n\n[e] Lepori et al., Break It Down: Evidence for Structural Compositionality in Neural Networks, NeurIPS, 2023\n\n[f] Lee et al., Geometric Signatures of Compositionality Across a Language Model's Lifetime, ACL, 2025\n\n[g] Gemma Team, Gemma 3 Technical Report, 2025\n\n[h] Liu et al., LLaVA-NeXT: Improved reasoning, OCR, and world knowledge, 2024\n\n[i] Bai et al., Qwen2.5-VL Technical Report, 2025"}]

Author rebuttal:

Meta-review: The paper presents a mathematically grounded approach to understand how each attention head in LLM specialize in specific semantic or visual attributes. The approach shows good results steering model outputs with very little edits to the heads (~1%). The reviewers appreciate the importance of topic, clarity and simplicity of the method, the theoretical derivation, extensive empirical results and ablation studies. All reviewers gave accepts or borderline accepts to this paper after the author reviewer discussion period. After considering the reviews and rebuttal, AC  recommends acceptance of the paper
Decision: Accept (spotlight)