Spectral Filters, Dark Signals, and Attention Sinks (2024)

Abstract: Projecting intermediate representations onto the vocabulary is an increasingly popular interpretation tool for transformer-based LLMs, also known as the logit lens (Nostalgebraist). We propose a quantitative extension to this approach and define spectral filters on intermediate representations based on partitioning the singular vectors of the vocabulary embedding and unembedding matrices into bands. We find that the signals exchanged in the tail end of the spectrum, i.e. corresponding to the singular vectors with smallest singular values, are responsible for attention sinking (Xiao et al., 2023), of which we provide an explanation. We find that the negative log-likelihood of pretrained models can be kept low despite suppressing sizeable parts of the embedding spectrum in a layer-dependent way, as long as attention sinking is preserved. Finally, we discover that the representation of tokens that draw attention from many tokens have large projections on the tail end of the spectrum, and likely act as additional attention sinks.

heading: 1 Introduction
text: Large foundation models dominate the state of the art in numerous AI tasks. While we understand how these models work in terms of elementary operations, and black-box evaluations help characterize observable behaviours, we lack a clear understanding of the connection between the two.
There is a growing body of work providing insights into properties of model components, e.g. (Voita et al., 2019; Pimentel et al., 2020; Voita and Titov, 2020; Geva et al., 2022; Meng et al., 2022; Voita et al., 2023), as well as identifying and explaining fundamental phenomena, often with the support of simple models (Elhage et al., 2021, 2022; Olsson et al., 2022; Shai; Todd et al., 2023).
Most recent works assign a central role to the model’s residual stream (RS) as the shared communication channel between model components.
In this perspective, the probability distribution of a token is initialised from the projection of the embedding of the previous token through the unembedding matrix, and receives additive updates from attention heads and MLP components, each reading from the residual stream of the same or previous tokens. The role played by components is interpreted projecting their contribution on the probability distribution over vocabulary items, in what is referred to as the logit lens (Nostalgebraist; Geva et al., 2020). We extend this approach and introduce logit spectroscopy, the spectral analysis of the content of the residual stream and of the parameter matrices interacting with it. Equipped with this tool, we look at the part of the residual stream spectrum that is most likely to be neglected by the logit lens: the linear subspace spanned by the right singular vectors of the unembedding matrix with the smallest singular values. Drawing an analogy with “dark matter” in astrophysics, that interacts with light only indirectly, we dub projections onto this subspace dark parameters, features, activations etc.
We were motivated by the thought that LLMs could learn to use signals in the dark linear subspace to maintain global features responsible for long-range dependencies while minimizing their
4792
interference with the next token prediction. We discovered instead that dark signals are instrumental to implementing the recently described phenomenon of attention sinks (Xiao et al., 2023), of which we provide a detailed account. We also show that the negative log likelihood of pretrained models can be kept low despite suppressing large swaths of the unembedding spectrum, as long as the dark signals required for attention sinking are untouched. Finally, we find a significant positive correlation between the average attention received by a token and the relative prevalence of dark signals in its residual stream, and find evidence that tokens uniformly receiving substantial attention are likely acting as additional attention sinks.

heading: 2 The LLaMa2 models
text: We chose LLaMa2 models (Touvron et al., 2023) as the object of our study as they were the most competitive models with open-access weights at the time we started this work. In particular we studied pretrained models (without instruction fine-tuning) with 7B, 13B, and 70B parameters.
Table 1 presents the key dimensional hyperparameters of these models. All LLaMa2 models share the same tokenizer, with 32,000 vocabulary items. Table 3, in Appendix A, summarizes the standard notation we use to refer to model components, and a high-level diagram of the architecture is in Figure 2.
We point out a few differences between LLaMa2 models and the original transformer architecture (Vaswani et al., 2017). RMSnorm (Zhang and Sennrich, 2019) is applied before every attention component, MLP component, and the final unembedding projection. RMSnorm includes a learned rescaling vector that is applied after the normalization proper. In all cases, we absorb this rescaling vector into the matrices downstream from the rescaling: this is mathematically equivalent and simplifies the analysis.
LLaMa2 models use SwiGLU activation functions (Shazeer, 2020). This means that MLP components have three parameter matrices instead of the more familiar two.
Finally, LLaMa2 models use rotary positional embeddings (Su et al., 2023). While important for the functioning of the model, their effect is independent of the phenomena we are focusing on.

heading: 3 Model parameter properties
text: Let Wu = UuΣuV ⊤u be the singular value decomposition of the vocabulary unembedding matrix (and similarly for We). Figure 3 shows the distribution of the singular values (SVs) of Wu for LLaMa2 13B. There is a single large SV1 followed by a tail that declines only at the very end (the distribution for LLaMa2 7B and LLaMa2 70B is similar). The SV distribution of the We embedding matrix is also similar, but the top SV is only twice as large as the second one, with a longer “head” of relatively large SVs.
We use the adjective U-dark (E-dark) to characterize anything that happens in the linear subspace spanned by the 5% right singular vectors (RSVs) of Wu (We) with the smallest SVs, the dark basis.
We gain an initial insight into whether dark signals are exchanged by projecting rows and columns of parameter matrices on the dark bases. Wk, Wq, and Wv project the residual stream onto either the latent space used to compute attention scores or the latent space used to compute the attention output, so they “read” from the residual stream; similarly W1 and W3 map from the residual stream onto the activation layer of the MLP components. We project the columns of these matrices on the RSVs of Wu to estimate and visualize their aptitude to read from the dark subspace2. Conversely, Wo and W2 map from latent spaces back into the residual stream, therefore we project their rows to check their aptitude to write into the dark subspace. We computed and plotted the norms of the d vectors of dimension dh or dm obtained with these projections, e.g.:
ji = { ||(V ⊤u )iWy||2, y ∈ {k, q, v, 1, 3} ||(V ⊤u )iW⊤y ||2, y ∈ {o, 2} (1)
We discovered a great variety in where, in the bases formed by the RSVs of Wu and We, model components are equipped to read from and write into. Figure 4 gives a sense of such variety.
The projections of MLP matrices also clearly indicate that some MLP components can write mostly into the dark subspace (see Fig.5 for the projection of 13B/L0/W2 on Wu).
1Projecting unembeddings on the first singular vector shows that it is highly representative of token frequency.
2We adopt the convention that input vectors are row vectors and are multiplied by parameter matrices on the right.
There are therefore components that are equipped for communicating through the dark subspace. In the following section we explore if such communication actually takes place and how it manifests itself.

heading: 4 Spectral filtering
text: One possible explanation for the existence of components that communicate through the dark subspace is that they are useless, and the model learned to divert their output to subspaces with little bearing on the vocabulary logits. To see if this is the case, we perform a series of experiments, where we patch some of the intermediate representations by projecting them onto the RSVs of Wu and We with largest singular vectors, therefore removing dark signals. We measure the average negative loglikelihood (NLL) of tokens in a sample of prompts:
if dark signals are irrelevant noise then we expect NLL to be largely unaffected when they are filtered out.
We split the singular vectors of Wu into 20 bands: {Vu,1, . . . , Vu,20}, of cardinality d/20, and similarly for We. Vu,1 contains the 5% of singular vectors with largest singular values, Vu,2 the next 5%, and so on. Let Vu,j:k be the matrix obtained concatenating [Vu,j . . . Vu,k], and (Fig. 1) let Φu,j:k = Vu,j:kV ⊤u,j:k (similarly for We). We can then project any d-dimensional vector onto the U-dark (E-dark) space by multiplying them by Φu,20:20 (resp. Φe,20:20).
We form a hierarchy of nested filters Φu,1:k: multiplying a vector by Φu,1:k projects it onto the subspace of the kd/20 “least dark” dimensions. We also define filters that combine singular vectors of Wu and We (Fig. 6):
Ψk = (I − Φe,(k+1):20Φu,(k+1):20), k = 1, . . . , 19 (2) and we set Ψ20 = I . Multiplying a vector by Ψk filters away projections onto subspaces of both We
and Wu that get darker as k grows. Ψ19 filters out only vector components in the 5% dark subspace of both Wu and We. These definitions make Φu,k, Φe,k, and Ψk comparable when looking at the fraction of kept/discarded singular vectors, although, since it discards a double projection, Ψk retains more information. We create a dataset (ccnet-405) of 405 prompts (13,268 tokens) in English, from CCNET (Wenzek et al., 2019). We ran inference on ccnet-405 applying spectral filters to the output of MLP components, one by one (Fig. 2 (yellow box)) and measured the average NLL. We similarly measure NLL when filtering the RS after the contributions of a given target layer have been added to it (Fig. 2 (red box)). See App. B for implementation details.

heading: 4.1 Discussion
text: When filtering MLP layers of LLaMa2 13B, the effects of masking L0 and L3 dwarf all others.
When filtering the output of the MLP at L3 (Fig. 7) the loss remains poor until the last 5% of the spectrum is included for both the Φ spectral filters. The fact that the Ψ curve is always above the random filter indicates that this MLP exerts strong influence operating in the dark subspace. See Appendix C for a discussion of 13B/L0/MLP.
Figure 17 in App. C shows that MLP components with a similar propensity for writing dark signals exist also in LLaMa2 7B (L1) and 70B (L2 and L8).
We also look at what happens to the negative loglikelihood when filtering the residual stream after
a given layer (Figures 2 (red box) and 8). Even a mild bottleneck after L3 results in a significant increase in NLL, in line with the observation that the MLP in L3 writes in the dark space. If we filter only signals that are dark to both We and Wu (Fig. 8(b)), we see bottlenecks continuing to be more harmful than random direction removal (Fig.8(a)) up until L20-25.
Dark signals exist and play an important role in achieving low perplexity, but what is this role?

heading: 5 Attention sinks
text: (Xiao et al., 2023) describe attention sinks: the special Beginning of Sentence (BoS) ‘Token 0’ receives a disproportionate amount of attention.3 This happens because often an attention head should not activate in a given context, but the normalisation in the attention scores forces a constant amount of attention to be distributed to previous tokens. The model therefore learns to sink excess attention by allocating it to the BoS token and making allocating attention to the BoS token inconsequential, i.e. a ’no-op’.
Figure 9 shows how the norm of the Token 0 residual stream progresses over layers, and the contributions from MLPs and MHAs components. We plot the overall norm, and the norms of the projection on the U-Dark space and of the projection on its orthogonal complement (U-Light).4
After an initial phase of input enrichment that apparently does not need an attention sink, the MLP at L3 blasts off a vector of large norm and almost completely U-dark. This vector acts as an attention collector for heads in need of a sink according to the mechanism described in detail in Appendix D, and is kept around until the last few layers, where the combined action of MLPs and attention heads first erases it and then replaces it with the vector that encodes the probability distribution of the model over generation-initial tokens.
We confirmed that dark signals are primarily used to sink attention with an additional experiment (Fig. 10). We apply spectral filters at the exit of a layer but, rather than suppressing the filtered
3Xiao et al. (2023) include the first four tokens in their operational definition of attention sinks, for achieving effective streaming decoding. In our observations the BoS token is by far the one attracting the most attention.
4Note that the composition of the residual stream of Token 0 is always the same, irrespective of the prompt, because Llama 2 models initialize it with the embedding of the special <BoS> token, and since they are autoregressive there is no context that could influence it.
component, we add it into the corresponding residual stream of a different sample, at the same token position and layer. Since the content of the RS of the BoS is the same irrespective of the input, this procedure leaves it intact, while perturbing all other RSs.
The resulting NLL heatmap is in Fig.11 (left). Unlike in those in Fig. 8, there is no step-decrease in NLL when the last 5% of singular vectors is added. Conversely, if we apply spectral filters
only to Token 0, the step-decrease is clearly visible (Fig.11 (middle)). We conclude therefore that the primary function of the dark subspace is to enable the crucial attention sink mechanism.

heading: 6 Sink-preserving spectral filters
text: The finding that dark signals are essential to enable attention sinks leads to the question: what would be the impact of spectral filters that preserve dark signals but filter away one or more bands before them? Let Ωu,k be a new family of spectral filters (Fig. 12):
Ωu,k = ([Vu,1:k;Vu,20:20])([Vu,1:k;Vu,20:20]) ⊤ (3) Fig. 11 (right) shows that low values of NLL are achieved even when masking a significant number of components. As an example (Fig. 13) NLL grows only from 2.47 to 2.74 when suppressing 25% of the SVs by applying Ωu,14 after L12. We
validated these results by repeating the experiment with a sample of code from the DeepMind Code Contest dataset (Li et al., 2022) consisting of 117 prompts (14,641 tokens), Fig. 20, and with LLaMa2 7B and 70B (Fig. 19).

heading: 6.1 The effect of inhibiting attention sinks on generation quality
text: So far we have looked at the effect of spectral filters on the negative log-likelihood of prefixes that were fed as prompts to our models. It is interesting to also observe the impact on generated continuations when applying these same filters.
Layers 2-4 are the most fragile to random and Ψ filtering (Tab. 2), while generations remain coherent with Ω filtering irrespective of where the filter is applied, when 10-20% of the singular vectors are suppressed. We notice that the application of Ψ filters often results in the model entering repetitive patterns. This is consistent with the possibility that attention heads largely copy representations from the RSs of previous tokens, and inhibiting attention sinking results in over-copying. More examples can be found in App. E.

heading: 7 Dark Signals and Attention Bars
text: Inspecting attention matrices in our sample we see that there are frequently a few tokens that also receive a large amount of attention, giving rise to characteristic attention bars (e.g. Fig. 14). We hypothesized that such bars could correspond to tokens whose residual streams are similar to that of
Token 0, and therefore become additional attention sinks.
We define High-Mean Low-Variance (HMLV) token-layer pairs those with mean received attention from subsequent tokens above τµ = .018 and variance below τσ = .01. We selected thresholds manually so that most HMLV tokens appear as “attention bars” in attention matrices5. This heuristic yields 12,236 HMLV token-layer pairs for LLaMa2 13B on ccnet-405 out of a total of 404,748 tokenlayer pairs for the same tokens and layers considered. We ignore the BoS token, the last 4 tokens of each sequence, and layers 0-3, since the attention
5Empirically, this heuristic appears more accurate for layers from L15 on. It might be better, but more laborious, to define layer-specific thresholds.
sink is not in place in the 13B model yet. We define the U-Dark ratio as a measure of the prevalence of U-Dark signals in a representation:
udr(hlt) = ||Φu,20:20hlt||
||(I − Φu,20:20)hlt|| (4)
i.e. the norm ratio between the projection of the residual stream onto the U-dark space, and the projection on its orthogonal complement.
We also define the key dot-product as the average dot product between the keys projected through matrices Wk:
kdp(l) = 1 |H||T | ∑
h,t
⟨hltW hk,l, hl0W hk,l⟩ (5)
Intuitively, this value quantifies the extent to which a residual stream appears similar to the one of Token zero, as far as attention allocation is concerned.
We measured the average U-Dark ratio and key dot product for HMLVs, and contrasted them with the same statistics taken over all token-layer pairs. Results are in Fig. 15.
Starting around L13 the U-Dark ratio of HMLVs grows considerably, whereas it remains constantly below .75 for the overall population. At the same time, the gap in key dot product also grows, as the kdp for the overall population steadily decreases, while that for HMLVs stabilizes. Analogous plots for the LLaMa2 7B and 70B models show different profiles, but an even more marked difference between HMLVs and the general population (Appendix F). These measures support the hypotheses that HMLVs are token-layer pairs that appear very similar to Token 0, as far as attention allocation is concerned, and therefore become additional attention sinks.

heading: 8 Related work
text: In this work we adopt the framework introduced in (Elhage et al., 2021), which highlights the central role played by the residual stream as working
memory and communication channel across components, and suggests the presence of subspaces with differentiated functions. The logit lens was first introduced by (Nostalgebraist), and has been since used and extended in a number of interpretation studies, including (Dar et al., 2022; Geva et al., 2022; Belrose et al., 2023; Din et al., 2023; Katz and Belinkov, 2023; Voita et al., 2023). Of these, Dar et al. (2022) analyse models projecting parameter matrices onto the vocabulary space, an approach we are also following in Section 3.
Our hypothesis that the subspace most orthogonal to the vocabulary representation could encode non-sparse latent features that should not interfere with the distribution over the immediate next token appeared supported by (Elhage et al., 2022), where they show that, in basic models, important nonsparse features tend to be assigned principal components rather than being in superposition with sparse features. (Sharkey et al., 2022) showed that sparse autoencoders can recover a ground-truth assignment of features to linear subspaces by means of sparse autoencoders in an artificial and simplified setting and a 31M parameter LLM, with d = 256; (Cunningham et al., 2023) apply a similar method to larger Pythia 70M and Pythia 410M models. Our results with Ω filters with the much larger LLaMa2 models indicate that the residual stream could be used at less than full capacity, at least for some of the layers, raising the question of what model of superposition would best apply to them.
The concept of Attention Sinks was introduced by Xiao et al. (2023), who noticed that performance of streaming models dropped abruptly once the first tokens moved out of the attention sliding window context. We take here the next step in explaining how this process works, showing how it happens thanks to a very limited and specific portion of the spectrum.
Sharma et al. (2023) show that it is possible to preserve and even improve the performance of the GPT-J (Wang and Komatsuzaki, 2021) model while dropping large portions of component matrices based on the SVD decomposition of the matrices themselves. We also focus on spectral decomposition, but using the spectrum of the unembedding matrix, in accordance with the logit lens approach and the intuition that it represents a common ground for all the components in a model.

heading: 9 Conclusions and Future work
text: A better understanding of the inner workings of transformer models is important to find strategies to make them safer. In this work we explored a novel way to look at transformers, extending the logit lens approach into logit spectroscopy by introducing spectral filters. We explored the hypothesis that dark signals could be used to maintain global features while minimizing their interference with the next token, but we discovered that the main role of dark signals is in enabling attention sinking. We reconstructed how attention sinking works in LLaMa2 models, and we showed that, as long as attention sinking is preserved, they still achieve low negative log-likelihood even when significant portions of the unembedding spectrum are suppressed. Finally, we found a positive correlation between the attention received by a token and the relative prevalence of dark signals in its residual stream, especially in the upper layers.
The results on spectral filtering with dark signal preservation, combined with the observation that transformers are invariant to basis changes in the residual stream, suggest that it could be possible to first "canonicalize" a model so that its residual stream dimensions match the Vu columns, and then compress away from components dimensions that are low in the spectrum but are not used for attention sinking. We consider such a form of spectral compression an interesting direction for future work.
Finally, while our exploration of the connection between “attention bars”, prevalence of dark signals, and similarity with the Token 0 residual stream suggests that attention bars are additional attention sinks, increasing the granularity of the spectral bands in the dark subspace could lead to additional discoveries.

heading: 10 Limitations
text: We limited our attention to the LLaMa2 family of models, and only to the pretrained models prior to their instruction and safety fine-tuning. It is possible that other models do not realise attention sinks in the same way, or that they might not need them.6
The number of conditions tested in our detailed spectral filtering experiments meant that we could use only a text sample of limited size. While we did
6For example by using off by one softmax (Miller).
repeat some of the experiments using a code sample, we cannot exclude that there could be some differences when using samples from drastically different distributions, e.g. in languages using different scripts.

heading: 11 Ethical Considerations
text: While our aim in studying the behaviour of LLMs is to make them more transparent, controllable, and trustworthy, it is conceivable that increased understanding could also benefit malicious agents intending to undertake harmful actions. Besides these generic considerations, we cannot see problematic ethical issues arising from this work.

heading: Acknowledgements
text: We thank the Transparent LLM team of Meta, and in particular Lena Voita, Yihong Chen, Javier Ferrando, for important suggestions and insightful discussions. We are also grateful to the anonymous reviewers for their constructive criticism and suggestions.

heading: A Notation
text: Table 3 summarizes the notation used throughout the paper.
B Implementation details
Spectral filtering experiments were run on up to 8 NVIDIA A100 GPU servers with 80GB of memory each, the slowest (for the 70B models) taking up to 4 hours to complete. They were implemented modifying the public Llama inference code7. Inference used nucleus sampling with the default parameters (top-p = .9, T = .6).

heading: C Additional spectral filtering experiments
text: When filtering the output of the MLP of LLaMa2 13B at L0 with filters Φe,1:k log-likelihood remains poor (around 9) (Fig.16) until about 90% of We’s RSVs are retained. This indicates that the MLP in L0 exerts its effect by writing mostly into an E-dark subspace. The fact that the NLL recovers steadily starting from when about 40% of the RSVs of Wu are retained, and that the curve for the Ψk filters is close to the random masking, show that this MLP writes mostly in the target unembedding space.
Figure 17 plots the NLL loss when applying spectral filters to the MLP at L1 in LLaMa2 7B, and at L2 and L8 in LLaMa2 70B. The loss profiles of component 7B/L1/MLP (left) are similar to those of 13B/L3/MLP: it exerts its influence in a subspace that is dark to both embeddings and unembeddings, and indeed it is the MLP that creates the attention sink vector in the Token 0 residual stream. 70B/L2/MLP and 70B/L8/MLP are the components responsible for the attention sink in 70b (Fig. 18). Unlike with 7B and 13B, the attention sink vector in Token 0 is U-Dark but not E-Dark, and is formed in two steps at L2 and L8.
Figure 19 shows NLL loss heat-maps for LLaMa2 7B (top) and 70B (bottom) when spectral filters are applied to the RS at the layer on the vertical axis.
Figure 20 displays similar heat-maps for LLaMa2 13B, using a sample of code fragments from the solutions of the DeepMind Code Contest dataset rather than ccnet-405.
7https://github.com/facebookresearch/llama

heading: D Attention sink mechanics
text: In this appendix we offer a more detailed account of the attention sinking mechanism described in Section 5.
Consider the case when a given head H at layer l ≤ L is irrelevant when predicting a token t, perhaps because H specializes in a different kind of contexts, or in text from a different distribution (e.g. code, a different language, etc.).
Recall (Fig. 2) that the final representation hLt at a token t is the sum of the embedding of the previous token (BoS for Token 0) and of all contributions from MLPs and attention heads at all layers, including H . Since, in MHA components, nonlinearities are only involved in the determination of
attention scores, the overall contribution of head H to the RS of t can be decomposed into the sum of micro-contributions, each corresponding to a specific token s ≤ t. Each such micro-contribution is the projection of the hidden representation at a previous token s right before layer l through WHv and WHo scaled by the attention score as,t:
δ(H, t) = ∑
s≤t as,thsW
H v W H o (6)
where we omit the layer index l and transposition operators to avoid clutter. Since attention scores are the outputs of a softmax, they are non-negative and such that ∑ s≤t as,t = 1, therefore the contribution of H to the RS of t is a convex combination of the projection of the RSs hs of all tokens s ≤ t through the matrix product WHv W H o . Interestingly, LLaMa 2 parameters are such that the projection of the RS of Token 0 through WHv W H o has much smaller norm than the projection of all other input tokens, and this for all H .
Since Token 0 has no token to its left, all its attention is focused on itself. This means that Fig. 9 (bottom right) can also be read as a plot of the norms for the combined micro-contributions from Token 0 of all attention heads. Fig. 21 (left) shows the same plot limited to the layers where the attention sink is in place, whereas Fig. 21 (right) shows a similar plot for a random token (most tokens look like this). Note the different scale on the vertical axis: projections from Token 0 are much smaller compared to other tokens. They are also
darker. When they are added to the RS of a token t, they have negligible influence on the corresponding logits8. This in turn means that the (undesirable) perturbation from the irrelevant head H is minimized if the head assigns as much attention weight as possible to s = 0, i.e. a0,t = 1 − ϵ for a small ϵ > 0.

heading: E Generation examples
text: Tables 4 and 5 show continuations from a same prompt generated by LLaMa2 13B models with three different filters. The left column is from a model filtered at a single layer with the filter Ψ that ablates the double projection on the 20% (resp. 10%) dark subspaces; the middle column is from a model where the filtering removes 20% (resp. 10%) random projections; the right column is generated with an equivalent Ω filtering. The row header indicates the layer immediately after which the filter was applied. Since the Ψ filter ablates the result of a double projection, we would expect its impact to be less noticeable. This is not the case: both the
8We verified that the reduction in entropy due to projections from Token 0, when sampling with T = 1, is always one or two orders of magnitude lower than the reduction gained by the same projections from other tokens.
Ψ-filtered and the randomly-filtered models at L39 are disrupted already at 10%, but the randomlyfiltered one starts recovering already when the filter is moved to L4, while the Ψ-filtered one keeps generating nonsense as the filter is moved up to L10 and beyond. Generations with Ω filtered-models remain largely coherent irrespective of where the filter is placed, for levels of suppression up to 20%.

heading: F High-Mean Low-Variance received attention tokens
text: High-Mean Low-Variance token-layer pairs (HMLVs) are defined in Sec. 7 as token-layer pairs whose received attention over subsequent tokens has mean above τµ = .018 and variance below τσ = .01. HMLVs tend to be associated to characteristic attention bars in attention matrices. Figure 22 shows the U-Dark ratio and Key dot-product metrics, by layer, for HMLVs and for the general population. HMLVs have much larger U-Dark Ratio than the general population, and also much larger key-dot product with the BoS token, supporting the hypotheses that they appear similar to the BoS token to subsequent tokens, and therefore behave as additional attention sinks.
9The impact on the random-filter at L3 suggests that L3 is where the residual stream is used at its full capacity.

heading: Future Work
text: We consider such a form of spectral compression an interesting direction for future work. Finally, while our exploration of the connection between “attention bars”, prevalence of dark signals, and similarity with the Token 0 residual stream suggests that attention bars are additional attention sinks, increasing the granularity of the spectral bands in the dark subspace could lead to additional discoveries