OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
Authors: Seungjun Shin
Jaehoon Oh
Dokwan Oh
Venue: ICML 2025 poster (2025)
Keywords: large language model
attention sink
efficiency
Primary area: deep_learning->large_language_models
TL;DR: We can predict token states based on their similarity to the sink token, enabling more efficient LLM inference

Abstract: Attention mechanisms are central to the success of large language models (LLMs), enabling them to capture intricate token dependencies and implicitly assign importance to each token. Recent studies have revealed the sink token, which receives disproportionately high attention despite their limited semantic role. In this paper, we first expand the relationship between the sink token and other tokens, moving beyond attention to explore their similarity in hidden states, considering the layer depth. We observe that as the layers get deeper, the cosine similarity between the normalized hidden states of the sink token and those of other tokens increases, and that the normalized hidden states of the sink token exhibit negligible changes. These imply that other tokens consistently are directed toward the sink token throughout the layers. Next, we propose a dynamic token selection method, called OrthoRank, using these findings to select important tokens. Specifically, in a certain layer, we define token importance by the speed at which the token moves toward the sink token. This is converted into orthogonality with the sink token, meaning that tokens that are more orthogonal to the sink token are assigned greater importance. Finally, through extensive experiments, we demonstrated that our method results in lower perplexity and higher zero-shot accuracy compared to layer pruning methods at the same sparsity ratio with comparable throughput, while also achieving superior performance on LongBench.

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
Seungjun Shin
Jaehoon Oh
Dokwan Oh"
Abstract
been proposed for LLMs. Among the various methods, layer pruning is simple and effective approach to reduce com- putational costs by removing layers that have less impact on the model. The impact is quantified by either measuring the similarity between the input and output at each layer (Siddiqui et al., 2024; Men et al., 2024), or by evaluating how the removal of layer effects the final output (Song et al., 2024; Kim al., 2024). Song al. (2024) proposed iterative pruning method based on these metries, while Kim et al. (2024) introduced a one-shot pruning approach followed by additional tuning using LoRA (Hu et al., 2022).
Recent studies have revealed the sink token, which receives disproportionately high attention despite its limited semantic role. In this paper, we first explore the relationship between the sink token and other tokens beyond attention, by an- alyzing their similarity in hidden states. We ob- serve that layers deepen, the cosine similarity between the normalized hidden states of the sink token and those of other tokens increases, and that the normalized hidden states of the sink to- ken exhibit negligible changes. These imply that other tokens are consistently directed toward the sink token throughout the layers. Next, we pro- pose dynamic token selection method, called OrthoRank, using these findings to select im- portant tokens. Specifically, in a certain layer, we define token importance by the speed at which the token moves toward the sink token. This is converted into orthogonality with the sink token, meaning that tokens that are more orthogonal to the sink token are assigned greater importance. Extensive experiments show that our method re- sults in lower perplexity and higher zero-shot ac- curacy compared to layer pruning methods at the same sparsity ratio with comparable throughput, while also outperforming on LongBench.
While effective, layer pruning has inherent limitations. requires a calibration set to determine which layers can be skipped, applying fixed pruning decision across all input tokens. This uniformity prevents the method from adapting to token-specific computational needs. At given layer, some tokens may no longer require further processing, while others still benefit from it. However, the pruning pattern remains static and canmot reflect such variation. As result, although layer pruning is well suited for environments with limited storage or communication bandwidth, such as on- device inference with small batch sizes, it may fall short in exploiting more fine- grained efficiency gains at the token level.
Motivated by the need for token level processing, early exit (Schuster et al., 2022; Chen et al., 2024b; Bae et al., 2023) and mixture of depth (Raposo et al., 2024) have proposed dynamic computation paths based on token-level character- istics. Early exit determines that token aligns with the final output, bypassing the remaining layers. Mixture of depth uses routers at each layer to decide whether token should be computed or skipped. While these methods offer viable solutions, they rely on training additional routers or classi- fiers, or require the entire model be trained specifically for early exit. Although these techniques have contributed to LLM acceleration, such as in speculative decoding, their practical use is limited because they require additional train- ing across wide range of existing models.
1. Introduction
Large language models (LLMs) have shown remarkable performance across various tasks (Thirunavukarasu et al., 2023; Wu et al., 2024; 2023; Labrak et al., 2024; Nam et al.,
2024). However, despite this, the computational cost of LLM inference remains a significant challenge, especially for real-time applications.
To address this challenge, many lightweight methods have
'Samsung Advanced Institute Technology, Korea. Corre- spondence to: Scungjun Shin <sj0216.shin @ samsung.com>.
Therefore, this paper begins by questioning:
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Can we identify which tokens advantageous to compute at each layer without extra training?

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
or
be
or
not
Layer
Layer I.
Select tokens be updated by orthogonality to the sink token C
F13
1
Ag
2+1
1-1
1-1
Layer (pruning)
Layer
Ag.
h
h2
h
A/
h
hg
h4
h/p
4
h
h
Pre-attention Normalization
Layer
Layer
hn
A
A
A
<bos>>
To
be
not
<bos>
To
be
or
not
(a) Layer pruning.
(b) OrthoRank.
Figure 1. Overview of our approach (OrthoRank). OrthoRank first determines the orthogonality of tokens to the sink token after normalization at each layer. Based on this, the top K tokens are selected for updates, while the remaining tokens hypass the layer except for KV calculations.
To explore this, we analyze the internal workings of LLMs to determine whether each token requires an update within layer. Our focus is on one of the most distinctive phe- nomena in LLM behavior: the attention sink (Xiao et al., 2024b), which was first studied by investigating attention distributions and identifying the presence of attention sinks. This phenomenon shows that the initial token in an input sequence receives a disproportionately large share of atten- tion, despite often lacking meaningful semantic value. This occurs because, in autoregressive models, the initial token is visible to nearly all subsequent tokens, leading to 'ex- cessive' attention scores. Since then, this phenomenon has been further explored (Sun et al., 2024; Cancedda, 2024; Gu et al., 2024), calibrated (Yu et al., 2024), and leveraged in various ways (Son al., 2024; Zhang al., 2024; Chen al., 2024a; Tang et al., 2025; Xiao al., 2024a) to im- prove LLM efficiency and enhance understanding of their mechanisms. Through further investigation, we observed that the sink tokens and other tokens exhibit a distinctive pattern of cosine similarity (Section 2).
to early-exit mechanisms. Many studies (Sun al., 2024; Son al., 2024) suggest that in certain models, the attention sink phenomenon occurs not only with any token at the first position but also with specific delimiter tokens (e.g., "." "\n"). However, for simplicity and consistency, we focus our calculations on the first token. That is, ht represents the input hidden states of the sink token (i 0) and other tokens (i 2 1) at layer I.
In summary, the key contributions of paper are as fol- lows:
We discover that after the layer where the attention sink occurs, the cosine similarity between the normal- ized hidden states of the sink token and those of other tokens increases, as the layers deepen. However, the normalized hidden states of the sink token across the layers remains largely unchanged. These mean that other tokens are heading toward the sink token.
We propose simple but effective token selection method, OrthoRank, based on token-sink orthogo- nality, prioritizing tokens closer to orthogonality for updates while bypassing others except for KV calcula- tions, without additional modules or training.
From this, we propose an importance ranking of tokens, OrthoRank, which leverages Orthogonality Rank to- kens based their relevance to the sink token. We confirm that selecting tokens with our orthogonal-basee importance is effective, as it outperforms the opposite method in lan- guage modeling performance (Section 3. To apply this across multiple layers in the I.M,, we adopt the layer evalu- ation method from layer pruning. We then replace each layer with token selection layer and evaluate them to identify the optimal token selection layers (Section 3.2). In Figure we provide overview of our proposed method, including the selection scheme. The main idea is to calculate each token's ss orthogonality to the sink token to select tokens. Selected tokens pass through all steps within the layer (e.g., query. key, value, feed forward network, etc.), while unselected tokens only participate in key and value calculations for the selected tokens without updating their own states, similar
We conduct extensive evaluations demonstrating that our method has better performance compared to the existing layer pruning at the same sparsity with com- parable throughput.
2. Further Analysis on Attention Sink Beyond Attention
In this section, we revisit the concept of the attention sink and introduce new insights based further analysis. An attention sink refers to the phenomenon where particular token receives a disproportionately high amount of attention from other tokens. This phenomenon is always observed after a certain early layer, liink, in the initial token (Xiao et al., 2024b; Sun al., 2024).
2

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
6a
03
0a
-
Aa
06-
02-
Layer
(b) Layer5 of (a).
(c) Layer40 of (a).
(a) Llama- 2-13B.
R
00-
3.4
-28
Layer
(e) Layer3 of (d).
(1) Layer32 of (d).
(d) Mistral-7B.
Figure 2. (a, d) Cosine similarity between the normalized hidden states of the sink token (ha) and other tokens Llama-2- 2-13B- and Mistral-7B. lsink is layer 4 and layer 2, respectively. (b-c, e-f) Conceptual representation of the relationship between the sink token (red line) and other tokens (blue lines) at layer right after laink and the final layer. After the attention sink, as layers progress, the cosine similarity between the sink token and the other tokens increases, indicating that the tokens are gradually aligning more closely with the sink token.
We begin by verifying whether the special relationship be- tween the sink token and the other tokens appears in states other than attention. To the best of our knowledge, we are the first to analyze the behavior of the sink token and other tokens through similarity analysis. In fact, the attention map provides limited information regarding the layers be- cause there little difference between the layers after layer sink. Therefore, we focus on the hidden states after the pre- attention normalization layer (i.e., the normalized hidden states), because they are the direct inputs for an attention module in each layer.
while the blue lines represent those of other tokens (i.e.. hn and hio). Additionally, the cosine similarity between other tokens except for the sink token does not show any consistent trend, presented in Appendix B. In summary, our findings offer insights that go beyond the information derived from the attention map, revealing that as the layers deepen, the angles between the sink token and the other tokens gradually decrease, after layer Lsink-
Obs. (1), For layer sinn and the final layer L, cos(hiinn (fiinx M S cos (To 7.(',, Vi. Moreover, when (zink < 4 < l2 < L, it generally holds cos(hi) h²) < cos (hg Rl.), Vi.
We investigate the cosine similarity between the normalized hidden states of tokens throughout the layers, using wikit- text dataset (Appendix F). Figures 2(a) and 2(d) describe the cosine similarity between the normalized hidden states of the sink token (i.e., hg)) and those of other tokens (i.e., h; (1 < i < 10)) of 1-13a-2-13B and Mistral-7B, respec- tively. For each model, the attention sink occurs at layer 4 and layer 2. It is observed that the cosine similarity be- tween the sink token and other tokens decreases drastically right after layer link. However, after layer laink, the cosine similarity between the sink token and other tokens tends to increase as the layers progress, although the ranges of cosine similarity vary across models.
Next, we explore the cosine similarity between the normal- ized hidden states of the same token across different layers. Through this analysis, we can determine whether the sink token and other tokens are converging towards each other, or if one remains relatively stationary while the other actively moves towards it.
Figures 3(a) and 3(b) illustrate the cosine similarity between the normalized hidden states of the sink token across all layers of Llama-2 2-1BB and Mistral-7B, respectively. The red boundary highlights the layers ranging from /iin to the final layer L. For Mlama-2-13B. after passing through layer lyink. the layers are grouped together, with each group exhibiting a significantly higher degree of similarity (close to 1). Despite this grouping, layers across different groups still maintain
These findings are simply illustrated in 3D by Figures 2(b- <) and 2(e-f). We plot each state as a unit vector on the hypersphere to focus on angles at layer right after liink and the final layer. In the subfigures, the red line represents the normalized hidden states of the sink token (i.e., ho).
3

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
Obs. (2), When leink 4 < l2 S S. cos(h)) hig) remains close However, cos(R:) .1 decreases the gap between and la widens, Vi 2
From Obs. (1) and Obs. (2), it is concluded that:
(a) --ama-2-13B (fo).
(b) Mistral-7B (hr)
As the layers deepen, other tokens gradually align with the sink token, which remains almost static.
3. OrthoRank: Dynamic token selection
In this section, we extend our observations as crite- ria for selecting tokens at layer (Section 3.1). Then, we propose a dynamic token selection algorithm, called OrthoRank (Section 3.2). Our algorithm can be used in conjunction with the layer selection algorithm.
(c) Llama-2- 2-B (hso).
(d) Mistral- 7B (hso).
Figure 3. (a-b) Cosine similarity between the normalized hidden states of the sink token across layers. (c-d) Cosine similarity be- tween the normalized hidden states of another token (postiton 50) across layers. The red boundary represents the layers after layer bink. The sink token shows similar values not only with adjacent layers but also with distant layers, as confirmed through (a) and (b). In contrast, the another token show similarity in adjacent lay- ers, but differences accumulate, leading to dissimilarity in distant layers, as shown in (c) and (d). These results highlight the static nature unique to the sink token, in contrast to other tokens.
3.1. Dynamic token selection criteria
Attention scores widely used identify relationships between tokens and are often employed to determine token importance. However, we discover that the relationships between tokens can also be captured through normalized hidden states. Based on this observation, we propose using these states define token importance.
Our findings suggest that tokens follow discrete trajectory in which they align with the sink token (i.c., they move in direction that increases cosine similarity). Building on this, we define the importance of token in a certain layer after Link as the speed at which token i can increase its cosine similarity with the sink token':
a relatively high level of similarity, approaching nearly 0.8. For Mistral-7B. -B, all layers following layer sink form a single cohesive group, where the similarity between these different layers is consistently close These results suggest that the sink token experiences almost no change in its trajectory in the normalized hidden states space it moves through the deeper layers. Therefore, the fixed sink token on the hypersphere in Figure 2, which simplifies our Observation (1), is nearly accurate. Furthermore, this observation can be linked to massive activations of the sink token in the hidden states, which appear in a small number of fixed feature dimensions and are delivered to the next layer via the residual connection, keeping high cosine similarity across layers.
a ah;
cos (ho. Ri))
Starting from the relation bJ h; = |1 hل11 cos (ho,, we compute the gradient of cos (ho, h.)) with respect to h;:
1 cos (ho: h.) [hsl Ohy
( ho cos (ho. hi) [|
|
Figures 3(c) and 3(d)illustrate the cosine similarity between the normalized hidden states a token, excluding the sink token, across layers of Llama-2-133B 13B and Mistral-7B, re- spectively. As expected, due to the presence of residual skip connections, there is relatively high similarity between adja- cent layers, especially along the diagonal. However, as the model processes more layers, differences between the layers begin to accumulate, and the normalized hidden states at the final layer eventually exhibit low cosine similarity compared to the normalized hidden states immediately following layer 7 1iink
(2)
Assuming that normalized hidden states have approximately equal norms except for sink token, we can simplify the importance of token i based on the cosine similarity. The full derivation is provided in Appendix C:
cos (ha, h.)) ki)]"
x
cos? (ho,
(3)
Th
For simplicity, / omitted this section.

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
vs. Reverse
vs. Random
Algorithm Selecting tokens in an OrthoRank layer
Input: hidden states ht. where an OrthoRank layer Input: pruning ratio
0.2
A perplexity
// Calc. inner product with sink toxen selection_criterion torch.matmul((b,110111 At, trantossps 2)) selection.criierion hww.aaaffeeecccooo.iiiiiioosssss
0.0
-0.
-0.4
Exception for the sink token
3 selection (00
10
20
Layers
// Sorting for attentior: module
wwwkiindices seelection.criierionooopk-----se
size(())))),
Figure 4. Layer-wise performance by token selection criteria. Our method achieves lower perplexity (PPL) across all layers except the final layer compared to both random token selection the Reverse criteria, which selects tokens in the opposite manner to our approach.
largest=False, dim= dime-I).indiess
s lowk.indices sowcchhott(lwwkiindceees)).. values
// Us ng except key and value states
jalected_hidden.ae torch.gather(h.r 1,
lowk indices sorted.unsqueezze -xxpp/dddd (-1 -1, ho.size(-1))
Thus, the importance of token i is directly related to how small cos (No, h.) is. As cos (ho, h.) decreases, the im- portance increases because tokens that are more orthogonal to the sink token are more likely to be selected, as they have greater potential to influence the overall cosine similarity.
return selected
In summary, our orthogonal token selection criterion leads to better performance across most layers, confirming its effectiveness in reducing computation while maintaining accuracy.
For implementation convenience, since the norms are ap- proximately equal, we use the absolute value of the in- ner product |/g hy] as practical proxy for cos (ho, hs) ). Therefore, select the top & important tokens, we rank them based on the smallest hoo hyl, which corresponds to selecting the tokens that are more orthogonal to the sink token.
3.2. Dynamic token selection with selective layer
In Section 3.1, we demonstrated that selecting tokens closer to orthogonality at each layer improves effectiveness while preserving model performance. However, challenges arise when applying this selection across all layers. First, our selection criteria are less valid before the attention sink occurs. Second, layers near the output are crucial for main- taining model reliability and require computation for most tokens. Additionally, inter-layer dependencies must be con- sidered. Therefore, instead of applying our selection criteria (OrthoRank) across all layers, we propose selectively ap- plying it to specific layers. To implement this, we combine our selection criteria with existing layer pruning methods. While traditional layer pruning approaches measure perfor- mance by removing layers by one, we measure perfor- mance by applying token selection to layers incrementally. This strategy enables efficient computation across both to- kens and layers while preserving model fidelity.
Select top l tokens with smallest [h hal-
(4)
To validate the effectiveness our proposed selection cri- terion, we performed an experiment using the WikiText-2 dataset (Merity et al., 2022). In this experiment, we applied token selection one layer time, examining the impact on the model's 's language modeling performance. For each individual layer, we selected the top 33% of tokens for com- putation based on our proposed metric, which prioritizes tokens that are more orthogonal to the sink token (i.e., those with the smallest inner product). We then compared the resulting perplexity (ppl) scores to those obtained using an alternative method, where instead of selecting the most orthogonal tokens, we selected the bottom 33%-the to- kens with the largest inner product-thereby evaluating the inverse of our approach.
Figure 5 compares layer pruning (Song et al., 2024) and Ortho with selective layers. Figure 5(a), Layer pruming is applied to a Llama2- 13b model with 40 layers and 20% sparsity, showing the pruned layers. In Figure 5(b), maintain the same sparsity, 30% of the layers modified to compute only 33% of the tokens, where the top 33% most orthogonal tokens to the sink token are selected for computation. As shown in Figure 1, only the selected tokens are updated, while the remaining tokens bypass the computation by passing the output of the previous layer
Figure shows the perplexity differences for layers after the attention sink (layer > 4), comparing our orthogonal token selection method both reverse (blue) and random selection (orange). Except for the final layer, our method consistently achieves lower perplexity, indicating superior performance. Layers with higher perplexity occasionally appear, but this varies by model. Model-specific results in Appendix D (Fig- ure 13) further confirm that ours generally outperforms both Random and Reverse (opposite) approaches in most cases.
5

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
0% token
33% tokens
tokens
Table I. Perplexity results on C4 dataset for various models.
Method
Sporsit
Llama 2
Liuss 3
Miscal
Layer
Layer
7B
130
706
5B
20803.11
76
8x7B
(a) Layer pruning.
Dense
0%
7.26
6.75
5.71
9.45
7.11
8.38
7.41
6:26
+OtthoRank
8.06
6.13
11.27
8.24
8,8
SLEB
201
10.90
9.42
7.31
16,49
11.38
12.39
9:50
Layer 40
OOthhRank
205
10.04
8.74
7.21
14.98
10.25
11.54
9.39
(b) Ort hoRank with selective layer.
Shertened (wlo
18.28
8.37
OOthooRak
8,04
11.22
9.43
8.47
Figure 5. Comparison under the same sparsity.
Table 2. Mean accuracies (%) on zero-shot tasks for various mod- els evaluated on PIQA, WinoGrande, HellaSwag, ARC-Challenge, and ARC- Easy.
through the residual path. Specifically, unselected tokens excluded from query computation, the query input to the attention mechanism, and the feed-forward network (FFN) computations.
Method
Spasity
Liama
Liama 3
Mistral
7B
138
708
8B
708(3.1)
7B
5:7B
Dene
0%
68.98
71.77
76.58
72,87
50.08
74.14
77.23
74.60
-Ottholankk
68.06
69.71
74.56
09.25
75.37
4. Experiments
59.68
02.97
59,41
61.94
70.84
+OrthoKank
2096
60.35
66.99
71.25
60.84
73.96
63.88
72.53
4.1. Implementation details
Slamened LLaMaa (weo PT)
10%
62.07
69.72
74.22
60.87
64.31
66.63
71.97
+OrthoRank
64.79
73.67
73.70
We conducted experiments comparing layer pruning and OrthoRank with selective layer approaches. Following the evaluation protocol in (Song et al., 2024), set target sparsities at 10% and 20%. To ensure the same sparsity ratio across methods, our algorithm applied 15% and 30 % layer selection, with only 33% of tokens computed in the selected layers. We compared our method against two baseline al- gorithms: the iterative layer pruning method SLEB (Song al., 2024) and the one-shot pruning method Shortened LLaMA (Kim al., 2024) without finetuning. Since the one-shot method suffers significant performance drop at 20% without fine-tuning we limited its comparison to 10% sparsity. We measured throughput on a single A6000 GPU with a batch size of 32 and prompts of length 2048. The throughput was averaged over 50. To validate the robustness of ours across a wide range of models, we conducted ex- periments various models, including Llamaa (7B, 13B, 70B) (Touvron et al., 2023), Llama3 (8B), Llama3.1 (70B) (Dubey et al., 2024), Mistral (7B) (Jiang et al., 2023), Mix- tral (8x7B) (Jiang et al., 2024a). Except for the ablation study on token sparsity (token selection ratio) Section 4.7.4), all experiments were conducted with a ratio 0.333. Dur- ing generation, we compare each token's inner product with the accumulated context and compute it only if belongs to the top fixed selection ratio with the smallest values.
Table 3. Mean accuracies LongBench for Llama-3-88 B-BB.
Context length
Sparsily
Method
Average
layer selection
0%
30.93
SLEB
25.10
10%
+OrthoRank
29.61
2048
SLEB
17.85
20%
OrrthRRank
25.23
SLEB
25.
10%
OarthoRank
29.71
4096
SLEB
17.85
20%
+OrthRRank
24.52
30.42
10%
+OrthoRank
29.77
8192
SLEB
21.53
OrthoRank
23,89
4.3. Results on Zero-shot Task
We further evaluated OrthoRank's performance on several zero-shot tasks, including PIQA (Bisk et al., 2020), Wino- Grande (WG) (Sakaguchi et al., 2021), HellaSwag (HS) (Zellers al., 2019), ARC-easy, and ARC-challenge (Clark et al., 2018), using the LM Evaluation Harness. As shown in Table 2, Ort demonstrated better performance compared to layer pruning in most cases.
4.2. Results Perplexity
4.4. Results on LongBench
Table compares the performance of various models on the language modeling task. Since the layers were pruned (unselected) using the Wikitext-2 dataset, we used the the C4 validation set (Raffel et al., 2020) for the performance comparison. Our proposed method, Ort hoRan,, outper- formed other layer pruning approaches in terms of perplexity most cases.
The evaluation results from LongBench (Bai et al., 2024) are shown in Table 3. For the Longbench experiments, we tested calibration lengths of 2048, 4096, and 8192 during the iterative layer selection process. As expected, higher context lengths led to improved performance, and Ort Rank consistently outperformed SLEB in most cases.
6

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
SixGePT
72.5
Slice (253))
SLPR-120
OrthoRank (20%))
Log peppppon
Spar (50%)
Stowtart 120%)
(56%) SparsedPT (50%)
Wanda (30%)
10"
LLMPtneer 120RR]
SUED 120%%
60.0
SiesGrr2 (25%5)
SLER
OrthoRank (20%)
U-32
OrthoRank (10%)
Slicepprr (30%)
35.0
1.00
1.03
1.30
1.15
1.20
1.00
1.05
1.10
1.15
1.20
Thorwshout in provements
Thorughout imprevements
(a) Log perplexity,
(b) Mean zero-shot accuracy+
Figure 6. Performance vs. Throughput Trade-offs: (a) Log perplexity (L) and (b) zero-shot accuracy (f) under throughput improvements.
Table 4. TruthfulQA results using MCI and BLEU scores.
Table S. Ablation study for selection criteria, stage, and KV for unselected token.
L-13
L-8ma-3-88
Mistral-BB
Mixiral-xx71 8x7B
Method
gen-
mel,
gen;
mel,
gen;
mel,
gen-
Crassia
Suaee
Lluma 2
Llama 3-BB
Misual
Misural SK7B
SLEB
21.2
20.82
19.8
4.17
21.3
21.6
24.1
27.1
PPl.
PPL,
ppl.
+OrthoRank
22.3
23.6
21.6
15.28
23.6
20.17
25,2
26.85
Ranlun
h.
10.33
61.17
12.05
9.85
69.45
70.89
Norm L
8.
9.15
64.47
21.12
9.78
66.50
Ortbogonal 1
11.85
58.81
18.06
59.57
13.00
59.47
9.76
66.21
Orthogonal Ortiogonal
"
9.04
64.30
13.72
00.82
11.53
63.87
9.55
71.39
4.5. Performance vs. Throughput Trade-offs
9.77
64.70
17.72
59.76
14.21
61.34
10.94
67.35
Ortieponal
4.
8.74
66.99
14.55
69.84
11:34
63.85
9.39
72.52
Figure illustrates the relationship between throughput im- provements and performance across different pruning meth- ods using Llama-2-13B Unlike most pruning algorithms that fail scale throughput with sparsity, layer pruning methods like SLEB achieve proportional throughput gains. Our proposed algorithm, OrthoRank (red), further en- hances this efficiency by maintaining throughput increase nearly proportional to sparsity while achieving performance that comparable to or even better than SLEB's s Pareto optimal curve (orange).
(Ours). Our method consistently achieves top or compa- rable results across all models, yielding lower perplexity and higher accuracy than other strategies. While the high- norm strategy slightly outperforms in zero-shot accuracy Mistral-7B, suffers from higher perplexity, potentially harming performance elsewhere. In contrast, our method maintains consistently high performance across both met- rics, highlighting its robustness. These findings validate the effectiveness of our orthogonality in Section 3, enabling more efficient token selection under the same computational budget.
4.6. Impact on Generation Quality Metries
criterion introduced
To assess factual quality, we evaluated models on Truth- fulQA using two metrics: (1) MC1, which selects the high- est log-probability choice, and (2) Generation (BLEU), which measures truthfulness and informativeness against ground truth.
4.7.2. SIMILARITY MEASUREMENT STAGE
Table 5 compares the use of hidden states (Row 3) and normalized hidden states (Row 5) for token selection. The results show that using normalized hidden states leads to better performance, with lower perplexity and higher accu- racy. As discussed in Section 2, our findings are based on normalized hidden states, making this result consistent with our expectations and further confirming the importance of normalization in improving token selection. However, it per- forms better than some other components of our approach, suggesting that while the weights within the normalization process do affect the cosine similarity, the hidden state simi- larity still operates in a somewhat similar manner.
As shown in Table 4, OrthoRank consistently demon- strates superior MC1 and BLEU scores across most models, suggesting improved factuality and reduced hallucination.
4.7. Ablation study
4.7.1. TOKEN SELECTION CRITERIA
Row 1, 3, 4 and in Table 5 compare the performance of different token selection strategies: random selection, 1.2-norm t. 1.2-norm 4. Orthogonal 4, and Orthogonal +
7

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
60-
perplesity
10
SLEB
DrthhRRank
60
SLEB OnthoRanc
Jayer sparsiy 0.3
10a
40
67-
Accuracy 186)
60-
100
63-
10a
Layer aparsity 4.6
10 20
50
64-
Sparsity (%6
Sparsity (P6)
63-
(a) Log perplexity,
(b) Mcan zero-shot accuracy,
Layer sparsity C.z (+Layer pruning!
02-
Figure 8. Performance comparison across varying sparsity levels with or without Ort sothoRank.
01-
0.0
6.2
022
0.3
0.4
05
(a) Log perplexity (lower is better) C4 dataset and (b) Mcan of zero-shot accuracies (higher is better). OrthcRank demonstrates superior performance across all sparsity levels except at 0.4.
loken Sparsity
Figure 7. Effect of varying token and layer sparsity on I. Llama-2- 13B under a fixed 20% effective sparsity.
OrthoRank also outperforms other methods across most sparsity levels, except at 40%, where performance parity is observed. These results highlight the effectiveness of token orthogonality- baaeed selection in balancing computational efficiency and model accuracy across diverse tasks.
4.7.3. KV CALCULATION FOR UNSELECTED TOKENS
In Table 5, we compare Row 4, where Key and Value (KV) computations for unselected tokens are skipped, with Row 5, where KV values are computed even for unselected to- kens. The results show that calculating KV values for all tokens, regardless of whether they are selected for updates, leads to better performance. This is because our token se- lection strategy focuses on how quickly a token's 's state up- dates, without considering the influence these tokens exert on others through KV interactions. When KV calculations for unselected tokens are skipped, the reduced interaction among tokens significantly degrades overall performance. Therefore, while unselected tokens are not updated, it is essential to compute their KV values to maintain model per- formance. This approach resembles calculating key-value pairs for tokens that have exited in early exit methods.
Table 6 Comparison of token selection criteria at 20% sparsity actoss models. OrthoRank selects tokens based hidden state orthogonality, while the attention- baseline relies on attention scores, incurring higher computational overhead.
Critoria
Thaoughut improvement
Llam-22138
Liamn-3-88
Mistra-78
Mixtral-ss78
PPL
ppl,
pel.
pel:
602-
Orthogunal Attention
1,18x
8.74
66.99
14.95
60.84
11.54
63888
9.39
72,52
0.71
4.7.6. COMPARISON WITH ATTENTION-BASED SELECTION
To further validate our orthogonality- based token selection criterion, we additionally compare OrthoRank with an attention-based baseline (Table 6). Unlike hidden-state-level criteria (e.g.. random, L2-norm, orthogonality), attention- based selection explicitly computes attention scores for all tokens, thereby introducing substantial overhead. Ad- ditionally, attention-based methods treat tokens primarily as sources of information, prioritizing tokens highly attended by others to preserve the attention matrix structure. While high attention scores indicate a token's 's strong influence, they offer limited insight into the necessity of updating that token itself.
4.7.4. EFFECT OF TOKEN AND LAYER SPARSITY AT FIXED EFFECTIVE SPARSITY
We conducted experiments using Llama 2-BB by adjusting token and layer sparsity while keeping the overall effective sparsity fixed. As shown in Figure 7, we observe that set- ting token sparsity in the range of 0 0.5 generally yields favorable trade-offs between efficiency and performance. Notably, token sparsity of O effectively corresponds to pure layer pruning, highlighting the importance of balanc- ing sparsity across both dimensions.
As shown in Table 6, attention based selection consistently results in lower zero-shot accuracy across models, despite occasionally achieving competitive perplexity. Moreover, the computational overhead significantly reduces through- put (0.71x) x) and limits compatibility with optimized fused- kernel implementations (e.g., FlashAttention, SDPA). In contrast, OrthoRank achieves consistently competitive performance in both perplexity and accuracy, providing meaningful throughput gains (1.18x) without such compu- tational drawbacks.
4.7.5. PERFORMANCE COMPARISON ACROSS SPARSITY LEVELS
Figure 8 compares hrRan's'''s performance under vary- ing sparsity levels using log perplexity the C4 dataset (Figure 8a) and mean accuracy on zero-shot tasks (Figure 8b). OrthoRank consistently achieves lower perplexity than SLEB across all sparsity levels except 40%, demon- strating its ability to optimize token selection while pre- serving language modeling capabilities, In zero-shot tasks,
8

---

OrthoRank:
Token Selection via Sink Token Orthogonality for Efficient LLM inference
5. Related work
both approaches require training of routing components, and Mixture-of-Depths must train the entire model from scratch, limiting compatibility with pretrained language models. In contrast, OrthoRank requires no router train- ing or fine-tuning, and can be directly applied to existing models. Its selection criterion, based on cosine orthogonal- ity to the sink token, provides a simple and interpretable mechanism for controlling computation at the token level.
Layer Pruning. Layer pruning has been a prominent ap- proach for reducing the computational complexity of large language models (LLMs), particularly in transformer archi- tectures (Siddiqui et al., 2024; Men et al., 2024). Recent approaches such as SLEB (Song et al., 2024) and Short- ened LLaMA (Kim et al., 2024) aim to remove entire layers that are deemed less critical for downstream tasks. These methods often rely on performance metries or sensitivity analysis to determine which layers contribute less to overall model accuracy and can be pruned without significant loss of performance.
Attention Sink. The concept of the attention sink, where certain tokens receive disproportionately high attention across layers, has gained attention in recent studies (Sun et al., 2024; Cancedda, 2024; Gu et al., 2024; Yu et al., 2024; Son et al., 2024; Zhang et al., 2024; Chen al., 2024a; Tang et al., 2025; Xiao et al., 2024a). Xiao et al. (2024b) first in- troduced the term "attention sink" to describe how the initial token in sequence tends to dominate attention scores in au- toregressive models. This is attributed to its visibility to all subsequent tokens, causing it to act as a "sink" for attention. Sun et al. (2024); Gu et al. (2024) conducted an in-depth investigation into the attention sink phenomenon, revealing that this behavior emerges as result of massive activation and layer normalization. Furthermore, they discovered that the severity of this phenomenon can be alleviated through the application of KV bias.
While layer pruning is an effective way to reduce model depth and memory usage, especially in prarge-constraieed or low-batch settings such as on-device deployment, it can cause sharp performance drops when critical layers are re- moved. Since it operates at the granularity of entire layers, does not reflect token-level variation. In our approach, we combine layer pruning and token selection by applying token-wise sparsity within a subset of layers selected via pruning.
Token Pruning. Token pruning methods have been widely explored as a way to reduce the number of tokens processed across layers, thus decreasing computational load. Recent advanced techniques such as dynamic token selection (Lou et al., 2024) and early exit mechanisms (Chen et al., 2024b; Del Corro et al., 2023; Elhoushi et al., 2024; Bae et al., 2023) progressively drop tokens deemed uninformative as they pass through layers. These methods rely on criteria such as attention scores or token contribution measures to decide which tokens to prune.
Building on these observations, our work explores token- sink orthogonality and uses this metric inform token selection. By selecting tokens that more orthogonal to the sink token, we prioritize tokens with greater potential to contribute to meaningful computations, leveraging the inherent token dynamics to optimize inference efficiency.
6. Conclusion
However, one potential downside of token pruning is the loss of potentially relevant information tokens are eliminated layer by layer, especially in deeper models where remaining tokens may not fully capture the complexity of the input sequence. Our approach differs significantly in that we do not progressively drop tokens across layers. Instead, we selectively compute a subset of tokens at specific layers based on their orthogonality the sink token. This ensures that we preserve the fiexibility compute tokens based their relevance without completely discarding them, thus mitigating the risk of information loss while still reducing computational costs.
In this paper, we introduced OrthoRank, a dynamic to- ken selection strategy based on the orthogonality between tokens and the sink token. Our approach was motivated by the observation that as layers deepen, tokens increasingly align with the sink token in the normalized hidden state space. By analyzing token-sink similarity, we found that tokens more orthogonal to the sink token play greater role in computation. Leveraging this insight, we developed a token selection mechanism that prioritizes such tokens at specific layers, leading to more efficient computation. By applying this token selection approach to selective layers, we achieved superior performance compared to traditional layer pruning methods at the same sparsity level with com- parable throughput. Extensive experiments demonstrated significant improvements, and ablation studies confirmed that our selection scheme is optimized both theoretically and empirically. Furthermore, our findings on token-sink similarity offer valuable insights for future research in Effi- cient LLM inference and Interpretable LLMs, providing foundation for further optimizing and understanding large language models.
Concurrent work such as D-LLM (Jiang et al., 2024b) also departs from traditional token pruning by dynamically skip- ping computation without fully discarding tokens, similar in spirit to our approach. Similarly, Mixture-of- Deeptsss (Ra- poso et al., 2024) learns to route tokens across layers using trainable routers under a fixed compute budget. D-LLM also introduces decision modules at each layer that learn, via -oobbl-, whether to execute or skip computation for each token, typically fine-tuned using LoRA. However,

---

OrthoRank:
Token Selection via Sink Token Orthogonality for Efficient LLM inference
Impact Statement
A., et al. The llama 3 herd of models. arXiv preprint arXiv:240..21783, 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Elhoushi, M., Shrivastava, A., Liskovich, D., Hosmer, B., Wasti, B., Lai, I.., Mahmoud, A., Acun, B., Agarwal, S., Roman, A., et al. Layerskip: Enabling early exit inference and self-speeulative decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2024.
References
Bae, S., Ko, J., Song, H.,, and Yun, S.-Y. Fast and robust early-exiting framework for autoregressive language mod- cls with synchronized parallel decoding, In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 99044924, 2023.
Gu, X., Pang. T., Du, Liu, Q. Zhang, F., Du, C., Wang, Y., and Lin, M. When attention sink emerges in language models: An empirical view. In The Twelfth International Conference on Learning Representations, 2024.
Bai, Y. Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, 7.,, Liu, X., Zeng, A., Hou, I.., et al. Longhench: A bilingual, multitask benchmark for long context under- standing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3 3119-3137, 2024.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L.. and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence on Learning Representations, 2022. URL https: / //openeevie.nnetee http///orummmmd/ae VeeeeeeeeeeFFYy
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b, arXiv preprint arXiv:2310.0682. 10.06225, 2023.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv 2024a.
Cancedda, N. Spectral filters, dark signals, and attention sinks. Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (Volume I: Long Papers), pp. 4792-4808, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 11116/////2024.a/ooongg
preprint arXiv:2401.0400
Jiang, Y., Wang, H., Xie, L., Zhao, H., Qian, H., Lui, J., et al. D-llm: A token adaptive computing resource allocation strategy for large language models. Advances in Neural Information Processing Systems, 37:1725-1749, 2024b.
Chen, G., Shi, H., Li, J., Gao, Y., Ren, X., Chen, Y., Jiang, X., Li, Z., Liu, W., and Huang, C. Seplim: Accelerate large language models by compressing one segment into one separator. arXiv preprint ar arXiv:2412.12000 2024a.
Kim, B.-k., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened Ilam:: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834 2024.
Chen, Y., Pan, X., Li, Y., Ding, B., and Zhou, J. Fe-llm: Large-scale training and inference of early-exit large lan- guage models with 3d parallelism. In Forty-first Interna- tional Conference on Machine Learning, 2024b.
Labrak, Y., Bazoge, A., Morin, E., Gourraud, P.-A., Rou- vier, M. and Dufour, R. BioMistral: A collection of open-source pretrained large language models for med- ical domains. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computational Linguistics ACL. 2024, pp. 5848-5864, Bangkok, Thai- land and virtual meeting, August 2024. Association for Computational Linguistics.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803.0800 2018
Lou, C., Jia, Z.,, Zheng, 7.,, and Tu, K. Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint arXiv:2406. 16747, 2024.
Del Corro, L., Del Gioro, A., Agarwal, S., Yu, B., Awadal- lah, A., and Mukherjee, S. Skipdecode: Autoregressive skip decoding with batching and caching for efficient Ilm inference. arXiv preprint arXiv:2307.0266 2023.
Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language mod- els are more redundant than you expect. arXiv preprint arXiv:2403.0m 03853, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan,
10

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. In International Conference on Learning Representations, 2022.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:330.009288, 2023.
Nam, D., Macvean, A., Hellendoom, V., Vasilescu, B., and Myers, B. Using an Ilm to help with code understand- ing. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1024.
Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 17564, 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Wu, Y., Jía, F., Zhang, S., Li, H., Zhu, E., Wang. Y., Lee, Y. T., Peng, R., Wu, Q., and Wang, C. Mathchat: Converse to tackle challenging math problems with Ilm agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
Raposo, D., Ritter, S., Richards, B., Lillicrap. T., Humphreys, P. C., and Santoro, A. Mixture-of- Dynamically allocating compute in transformer-based lan- guage models. arXiv preprint arXiv:2404.02258, 2024.
Xiao, G., Tang, J., Zuo, J., Guo, J., Yang, S., Tang, H., Fu, Y., and Han, S. Duoattention: Efficient long-context Ilm in- ference with retrieval and streaming heads. In The Twelfth International Conference on Learning Representations, 2024a.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks, In The Twelfth International Conference on Learning Represenaatios, 2024b.
Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V. Q.. Tay, Y., and Metzler, D. Confident adaptive language modeling. In Advances Neural Information Processing Systems, 2022.
Yu, Z., Wang, Z., Fu, Y., Shi, H., Shaikh, K., and Lin, Y. C. Unveiling and harnessing hidden attention sinks: Enhanc- ing large language models without training through atten- tion calibration. In International Conference on Machine Learning, 2024.
Siddiqui, S. A., Dong, X., Heinrich, G., Breuel, T., Kautz, J., Krueger, D., and Molchanov, A deeper look at depth pruning of llms. arXiv preprint arXiv:2407.16266 2024.
Son, S., Park, W., Han, W., Kim, K., and Lee, J. Pre- fixing attention sinks can mitigate activation outliers for large language model quantization. arXiv preprint arXiv:2406. 12016, 2024.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4791 4791-4800, 2019.
Song, J., Oh, K., Kim, T., Kim, H., Kim, Y., et al. Sleb: Streamlining Ilms through redundancy verification and elimination of transformer blocks. In Forty-first Interna- tional Conference on Machine Learning, 2024.
Zhang, Y., Gao, B., Liu, T., Lu, K., Xiong, W., Dong, Y., Chang, B., Hu, J., Xiao, W., et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2466.00 02069, 2024.
Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activa- tions in large language models. In ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foun- dation Models, 2024.
Tang, H., Lin, Y., Lin, J., Han, Q., Hong, S., Ke, D., Yao, Y., and Wang, G. Razorattention: Efficient kv cache compression through retrieval heads. In The Thirleenth International Conference on Learning Representations, 2025.
Thirunavukarasu, A. J., Ting, D. S.J., Elangovan, K., Gutier- rez, L., Tan, T. F., and Ting, S. W. Large language models in medicine. Nature medicine, 1930):930011900 2023.

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
A. Cosine similarity changes of the same token across layers
investigate cosine similarity changes of the same token across layers using various models, including Llama-2-7B, 13B, and 70B; Meta- Metaa-Llama-3-83 and -lama- as shown in Figure 10.
14
11
(a) Hama-2-7B (ho).
(b) Llama-2-7B (ho0)
(c) Llama-2-77 2-BB (1100)
66
14
42
12
(d) Llama-2-13B (ha).
(e) Llama-2-133 (hso).
(f) Llama-2-13B (h110).
14
14
42
(g) Llama 1-70B (ho).
(h) Llama-2-700B (hso).
(i) Llama-2-70B (h100).
22
Meta-Llama-3-SB (ho).
(k) Meta-Llama-33 3-BB (hoo).
()) Meta-Llama-3-88 (h100).
22
(m) Llama-3.1 3.1-70B (ha).
(n) Llama-3.1 3.1-70B (hoo).
(o) Llama-3.1-1220 (A100).
Figure 9. Cosine similarity across layers for different models and token positions. Each row represents token position: (sink), 50, and 100. The red boundary represents the layers after layer luenk- The sink token shows similar values not only with adjacent layers but also with distant layers. In contrast, other tokens show similarity in adjacent layers, but differences accumulate, leading to dissimilarity in distant layers These results highlight the static nature unique the sink token, in contrast to other tokens.
12

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
B. Cosine similarity changes between tokens across layers
investigate cosine similarity changes between tokens across layers using various models, including Llama-2-7B, 13B, and 70B; Meta- veeta-Llama---B-- and Llama-3.1-3 3.7-OB.. The results are shown in Figure 10, which presents the similarity between position 0 (sink) and positions 91-100, as well as between position 50 and positions 91-100. Figure 11 shows the similarity between position 0 (sink) and positions {1,10,100, 100,0)00000)) as well as between position 500 and positions { 1,10,100,1000)
Cosne smitaryy
Cod sintarte
21
25
31
Layor
(a) Llama-2-7B; with position 0 (sink)
(b) Lllam-2-7B; with position
COSNN sinbuityy
Cod sim
An
11
23
(c) Llama-2-13B; with position (sink)
(d) Lllam-2--33B; with position
Coofne chntamt
12
-0.
18
12
Layer
Lay
(e) Llama-2 2-70B; with position (sink)
(l) Llama-2-700B with position 50
simtansm
24
Cooliee sintartty
3.6
(g) Meta- Betta-Laama---ae 3- with position (sink)
(h) Meta-Llama-3 3-BB; with position 50
Coline Sibalty
Codhre Smibrrts
3.8
-08
Llama-3.1 3.1-70B; with position (sink)
Ll 3-10B;; with position 50
Figure 10. Cosine similarity changes between tokens across layers. Each column corresponds to a different model: Llama-2-7B, 13B, 70B: Dett--Llama----e Llama-3.1 3.1 Left: Cosine similarity between tokens position 91-100. Right: Cosine similarity between tokens at position 50 and 91-100.
13

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
100
1.
hpp
Prie
0.8
Cosine cimientty
02
-04
Layer
Layer
(a) Llama-2-7B: with position 0 (sink)
(b) Llama-2-7B: with position 500
1.0
Fuo
RI
fre
0.0
0.3
Costne siitenty
0.2
0.0
(c) Llama-22 2-13B; with position 0 (sink)
(d) Llama-22 2-13B; with position 500
hpo
0.8
0.3
Cosire cimiintty
0.0
-0.
(e) Llama- Llama-2-70B- with position (sink)
(f) Llama-2-70B. with position 500
0.
Costie siileert
0.5
0.4
0.0
-0.2
-0.2
(g) Meta-Llamaa-388B; with position 0 (sink)
(h) Micta-Lllama-3-8 with position 500
Figure 11. Cosine similarity changes between diverse tokens across layers. Each column corresponds to a different model: Llama-2-7B,
13B, 70B; and Mcta--.lama-3-BB. Left: Cosine similarity hetween tokens position 0 and (1, 10, 100, 1000). Right: Cosine similarity hetween tokens at position at position 50 and {1, 10, 100, 1000). Results highlight that tokens, even when their positions are far from each other, increasing cosine similarity with the sink token (position 0) across layers (left), show no consistent trend position 50 (right).
14

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
C. Supplementary Derivation for Section 3
begin with the gradient of the cosine similarity between the sink token's hidden state hn and the hidden state of token i,
h;
a cos (ho. hi))
(1
ho
his ||bz]||
cos (ho. h4))
(5)
ah,
|
|Moll
We are interested in the magnitude of this gradient, specifically its squared norm:
a
ho
hg ||45]]
cos (ho, h.))
[/ell ||all cos (ho: Rr.)
(6)
Oh,
As observed in Figure 12, the norms of the hidden states Therefore, we assume:
|isll (excluding [[hol) follow the behavior described Section C.I.
||all| se
(7)
Substituting
|/aal| ss e into the gradient norm squared:
ho
cos(ho. hv) h4
cos ho. hu)
(8)
ah,
2
holl
To simplify the expression, we define unit vectors u, v as follows:
ho
h
u
v
(9)
/
Substituting u and v into the expression:
8 Oh,
1 lu cos(u, v)vl
cos (ho, hi)
hi)
2
(10)
We compute the squared norm:
u cos u, v)vl = |/u| 2 cos(u, (u, v)u v cos²(u, v\|vl2
== 1 - cos
[u, v) cos u, v)
cos (u,
( v)
Substituting back:
cos cos(ho, hi) Th,
~ (1 cos hv))
(LI)
Since 1 is a constant, the gradient norm squared is proportional to 1 - cos? (oo? ha):
cos(h,, ho, hv))
x 1 - cos (ho, h2)
(12)
Oh.
15

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
worm Manden
8
Layer
Layer
(a) Llama-2--1- 2-7B
(b) Llama-2 2-13B
10000 13
100 1000
20
Layer
(c) 1alama-2-700
(d) Meta-Llama- 1-BB----BB
mome Midten
23
71
22
16
Layer
(c) L.lam-3.11-70
(f) Mistral-7B
Figure Norms of the normalized hidden states. This figure plots norms the normalized hidden states ||ha]] for various token positions in the sequence. The red line represents the norm of the hidden state at position 0 (the sink token), while the blue lines correspond the norms of hidden states at positions 91 100. The plot illustrates the norms the hidden states (excluding sink token) are approximately equal.
16

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
C.I. Norm Behavior under RMS Scaling
OrthoRank operates scaled, normalized hidden states derived via RMSNorm. RMSNorm normalizes each token vector to unit RMS norm and then applies learned element-wise scale:
h;
scaled
h hoom
(13)
RMS(hh)
The scaling vector g has low empirical variance (approximately 0.001), introducing only minimal distortion in token-wise norms. For most tokens (non-sink tokens), the cosine similarity between h scaled and h exceeds 0.95:
cos( 0) R 1
scaled
oh!
norm
(14)
This indicates that scaling behaves approximately as scalar multiplication. Since scalar multiplication preserves norm ratios, the scaled hidden states maintain near-une near-uniform norms across non-sink tokens:
h scaled s a c [lh scaled for all 6, in non-sink tokens
(15)
A small subset of tokens (sink tokens) shows lower similarity (e.g., ~066 and significant deviation in norm, often accompanied by concentrated activation a few dimensions, Empirical observations supporting these behaviors are shown in Figure 12.
17

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
D. Layer-wise Performance by Token Selection Criteria
Figure 13 shows the perplexity differences for layer-wise manner, comparing our orthogonal token selection method to both reverse (green) and random selection (blue). OrthoRank demonstrated superior performance compared to both Random and Reverse (opposite) approaches in most models and across layers. Rare layers where OrthoRank performs worse automatically filtered out during the selection process, this does not pose significant issue.
vs. Reverse
Randam
Reverse
Random
0.2
0.2
perplexity
à perplexity
0.1
0.0
-0.1
-0.4
-0.2
10
15
20
25
30
35
40
10
20
30
40
50
60
70
80
Layers
(a) Llama-2-133
Llama-2-777
vs. Reverse
Random
Reverse
Random
0.4
A perplexity
à perplexityy
0.2
-1
-0.2
-0.4
10
25
20
50
70
Layers
Layers
(c) Meta- Meta-Llama-33-4 3-8B
(d) Llama-3.1-700
Figure 13. Layer-wise Performance by Token Selection Criteria
18

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
E. Results for each task
In this section, we present the detailed results for each task corresponding to Table and Table the main paper.
Table 7. sero-shot results (%) for various tasks and models with SLEB and OrthoRank by sparsity.
Model
Spursity
Method
PIQA
WG
ARC-C
ARC-E
SLEB
77.15
63.38
70.69
38.57
65.87
10%
OathhRRank
77.97
65.82
72.22
40.96
69.32
Llama-2-7B
SLEB
74.54
$9.31
64.71
35.41
59.22
20%
OrthoRank
76.33
56.51
65.68
37.12
65.95
SLEB
79.11
66.85
74.33
41.55
71.89
10%
OrthoRank
79.82
69.93
75.96
46,59
76.26
Llama-2-13B
SLEB
76.67
65.11
70.52
38.23
64.35
20%
OrthhRRank
79.16
64.17
73.00
44.45
74.16
SLEB
81.50
75.06
80.00
52.30
76.77
10%
+rtthoRank
81.34
75.69
81.46
55.72
78.62
Llama-2-70B
SLEB
80.14
73.09
77:20
48.29
75.38
20%
+OrthoRank
80.47
73.01
78.42
50.00
76.68
SLEB
78.02
67.88
71.79
44.37
72.56
10%
OathhRank
79.49
69.30
71.88
47.70
74.62
Meta-Llama-3--8
SLEB OrthoRank
75.19
56.59
63.35
35.49
61.11
20%
75.52
58.01
65.63
38.40
66.67
SLEB
83,35
76.09
81,78
58.19
81,70
10%
OathhRRank
83.84
71.27
82.84
60.67
84.68
Llama-3.1 3.1---
SLEB
81.39
74.03
78.00
53.67
79.12
20%
OrthoRank
82.10
69.77
79.87
56.48
81.56
SLEB
79.22
68.59
76.60
45.65
74.03
10%
OrthoRank
80.96
70.72
77.49
46.25
74.71
Mistral-7B
SLEB
76.88
62.75
66.71
37.46
64.14
20%
+OtthoRank
77.48
65.27
70.91
40.02
65.74
SLEB
82.81
72.70
81.35
54.95
80.26
10%
OrthoRank
82.97
74.51
82.16
58.79
Mixtral-8x77
SLEB
80.47
71.20
77.33
48.89
76.35
20%
OrthoRank
81.94
72.85
79.24
$2.05
78.16
Table 8. Longhench performance comparison across tasks with varying context lengths of calibration and sparsity.
Length
Spanny
Quaper
HaqaQQA
2WILMQA
MulliNwws
SAMSun
Dence
14,08
10:09
59.9
45.6
1.16
2048
10%
Orthokank
11.45
17.53
98.82
52.94
43.37
0.29
3.14
705
SI.FB
1.91
5.01
14.23
5:23
6.67
3.57
14.14
16.55
7.54
96.73
46.98
34.42
1.34
5.27
42.67
41.91
10%
SI.EB
5.77
11.45
6.26
17.91
7.34
8.63
4.3
24.3
19.73
19.58
47.5
83.04
36:11
3.54
0.29
3.23
58.03
52.31
20%
OOtholtank
SI.FB
1.91
5.01
14.20
5.23
6.67
3.57
14.14
16.55
7.54
46.98
34.42
1.34
5.27
42.67
41.91
8192
10%
SI.EB
16.95
13.71
26.44
9.16
12.87
6.35
29.99
20.54
15.19
7155
98.95
87.17
42.50
3
66.11
62.65
20%6
-OrthRRakk
SI.F
12.54
3.68
7.91
7.5
19.75
19.96
8.76
10.37
8.5
5.81
3.42
19.45
19.27
21.45
17.25
8.24
1.22
30.5
75.68
K7.9
38.84
31.16
3.84
1.1
3.6
3.9
56.28
55.34
41.45
56,99
19

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference F. Text Inputs Used for Figures 2, 3, 9, and 10.
This section provides the input text used to generate the plots Figures 2, 3, 9, and 10. Due to memory constraints associated with storing token hidden states for visualization, we limited the context length to 101 tokens. The following text is extracted as the first 101 tokens from the test split of WikiText-2-raw-vl. For clearer visualization, we excluded |n which is known to cause additional attention sink (Sun et al., 2024), from the experiments.
Robert Boulter = Robert Boulter is English film, television and theatre actor He had guest @-@ starring role the television series The Bill in 2000. This was followed by a starring role in the play Herons written by Simon Stephens, which was performed in 2001 at the Royal Court Theatre.. He had a guest role in the television series Judge John Deed in 2002. In 2004 Boulter landed a role as" Craig in episode Teddy Story of the television series The Long Firm he starred alongside actors Mark Strong and Derek Jacobi. He was cast the 2005 theatre productions the Philip Ridley play Mercury Fur which was performed the Drum Theatre in Plymouth and the Menier Chocolate Factory London He was directed by John Tiffany and starred alongside Ben Whishaw Shane Zaza Harry Kent Fraser Ayres Sophie Stanton and Dominic Hall In 2006.. Boulter starred alongside Whishaw in the play Citizenship written by Mark Ravenhill He appeared on a 2006 episode of the television series, Doctors, followed by role in the 2007 theatre production of How to Curse directed by Josie Rourke. How to Curse was performed at Bush Theatre in the London Borough of Hammersmith and Fulham. Boulter starred in two films in 2008 Daylight Robbery by filmmaker Paris Lconti, and Donkey Punch directed by Olly Blackbum In May 2008, Boulter made a guest appearance on two @-@ part episode are of television series Waking the Dead followed by an appearance on the television series Survivors in November 2008 He had recurring role in ten episodes of the television series Casualty in 2010 as Kieron Fletcher " Boulter starred in the 2011 film Mercenaries directed by Paris Leonti Career
2000
2005 = In 2000 Boulter had guest @- @ starring role the television series The Bill: he portrayed Scott Parry " in the episode In Safe Hands Boulter starred as Scott in the play Herons written by Simon Stephens, which was performed in 2001 at the Royal Court Theatre.. A review of Boulter 's performance in The Independent on Sunday described him as horribly menacing" in the role, and he received critical reviews in The Herald, and Evening Standard He appeared in the television series Judge John Deed in 2002 as " Addem Armitage in the episode" " Political Expediency " and had role as a different character" Toby Steele on Bill. had recurring role in 2003 on two episodes The Bill as character Connor Price" " In 2004 Boulter landed role as" Craig in the episode " Teddy 's Story" " of the television series The Long Firm he starred alongside actors Mark Strong and Derek Jacobi. Boulter starred as Darren in the 2005 theatre productions of the Philip Ridley play Mercury Fur It was performed at the Drum Theatre in Plymouth, and the Menier Chocolate Factory in London. He was directed by John Tiffany and starred alongside Ben Whishaw Shane Zaza Harry Kent, Fraser Ayres Sophie Stanton and Dominic Hall. Boulter received a favorable review in The Daily Telegraph: The acting is shatteringly intense. with wired performances from Ben Whishaw now unrecognisable from his performance as Trevor Nunn 's Hamlet ), Robert Boulter, Shane Zaza and Fraser Ayres The Guardian noted, " Ben Whishaw and Robert Boulter offer tenderness amid the savagery In 2006 Boulter starred in the play Citizenship written by Mark Ravenhill The play was part of series which featured different playwrights, titled Burn / Chatroom / Citizenship. In 2006 interview fellow actor Ben Whishaw identified Boulter one of his favorite co @-@ stars: loved working with guy called Robert Boulter, who was in the triple bill of Burn, Chatroom and Citizenship at the National Hc played my brother in Mercury Fur He portrayed Jason Tyler" on the 2006 episode of the television series, Doctors, titled Something I Ate Boulter starred as " William in the 2007 production of How to Curse directed by Josie Rourke How to Curse was performed at Bush Theatre in the London Borough Hammersmith and Fulham. a review of the production The Daily Telegraph theatre critic Charles Spencer noted, Rohert Boulter brings touching vulnerability the stage William Boulter starred Robbery by filmmaker Paris Leonti, and Donkey Punch directed by Olly Blackburn Boulter portrayed character named Sean in Donkey Punch, who tags along with character Josh the quiet brother who hits it off with Tammi " Boulter guest starred on two @-@ @ part episode are Wounds in May 2008 of the television series Waking the Dead as character Jimmy Dearden He appeared on the television series Survivors as Neil in November 2008 He had a recurring role in ten episodes of the television series Casualty in 2010, as Kieron Fletcher " He portrayed an emergency physician applyi
" 2006 present
films in 2008, Daylight
20

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
G. Exploring the Trade-offs Between Throughput and Perplexity
In this section, we investigated the relationship between throughput improvements and perplexity across varying sparsity levels. Figure 14 demonstrates how each point corresponds to a specific sparsity level. sharp increase in perplexity observed at the 50% sparsity point, highlighting the critical threshold for balancing sparsity and preserving model performance while accounting for the speed tradeoff. Based this finding, recommend using OrthoRank with sparsity levels below 40% to maintain an optimal balance.
100
80-
Perplexity
Each point represenul sparsily Jevel (10%, 20%. 30%, 40%, 50%
50%
0%%
10%
20%
30%
122
15
Throughpul Improvements
Figure The relationship between throughput improvements and perplexity (C4). Each point represents different sparsity level. showing sharp increase in perplexity at the 50% sparsity level
21

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
H. Dynamic Token Selection Ratio
This section presents a set of experiments analyzing the effect of varying the token selection ratio across layers while maintaining a fixed average sparsity. The goal is to evaluate the impact of simple heuristics for distributing sparsity without increasing overall budget.
Experimental Setup
Experiments were conducted with Llama-2-13B 20% overall sparsity. Token selection was applied to the following layers:
[8, 9, 10, 11, 12, 22, 25, 27, 29, 31, 33, 34]
Three strategies for assigning per-layer token selection ratios were evaluated:
Fixed: Constant ratio of 0.33 across all selected layers.
Linearl: Ratio increases linearly with layer depth.
[0.0, 0.061, 0 0.121, O. 0.182, 0 0.242, O. 0.303, 0.364, 0.424, 0.485, 0. 0.545, 0. 606,
0. 667]
Linear2: Ratio decreases linearly with layer depth.
[0.667, 0. 606, D 0.545, 0. 485, 0 424, 0. 0.4,,, 0.303, 0. 0.242, 0.182, 0.121, 0.061, 0.0]
Results
Table 9. Performance of different token ratio assignment strategies.
Method
ppl
acc;
Fixed (uniform ratio 0.33)
8.74
66.99
Linearl (ratio increases with depth)
9.10
63.74
Linear2 (ratio decreases with depth)
8.97
68.18
The fixed-ratio configuration produced the lowest perplexity, while allocating higher token ratios earlier layers (Linear2) yielded the highest accuracy. These configurations serve simple baselines for dynamic ratio selection under constrained sparsity.
22

---

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
I. Example of Generation Task
Table 10. Example of generation task.
Generation using OrthoRank
Prompt for OrthoRank
Struggling with recent break-up, person opens up about the intense feelings loneliness and sadness. They ask for advice on how to cope with the heartbreak and move forward in life. They also wonder if they will ever find love again.
OrthorRank
I'm sorry hear that you struggling with loneliness after recent break-up. It's natural to feel sad and lonely after relationship ends, and it's not uncommon to feel like you are the only one experiencing these feelings.
Here are some tips to help you cope with loncliness and move forward:
1. Acknowledge your feelings: It's I'ss okay to feel lonely and sad. Allow yourself to process your emotions and give yourself time to heal.
Connect with friends and family: Reach out to loved ones for emotional support and companionship.
3. Engage in activities: Try new hobbies or activities that bring joy and help you feel productive, such exercising, reading, or cooking.
4. Practice self-care: Take of your physical and emotional needs. Get enough sleep, well, exercise regularly, and take breaks from screens.
5. Seek professional help: you find that your feelings of loneliness and sadness are persistent and affecting your daily life, consider seeking counseling or therapy.
Remember that healing takes time, and it's s important to be patient and compassionate towards yourself. With time and effort, you move forward in life feeling stronger and happier.
Evaluation using ChatGPT
Prompt for ChatGPT 4o
Given the following response to the prompt, evaluate its quality on a scale from I to S, considering factors such as relevance, clarity, and accuracy. would rate this response out based on the following factors:
ChatGPT 4o
Strengths:
The response directly addresses the emotional struggle a breakup, providing practical and empathetic advice tailored to the prompt. Clarity: The suggestions are well-organized, easy to follow, and actionable, making them accessible for the reader.
Relevance:
Empathy: The tone is kind and understanding, which is critical in respouding someone dealing with heartbreak. Accuracy: The advice, including acknowledging feelings, connecting with loved ones, and seeking professional help, aligns with well-established coping strategies.
Areas for Improvement:
Depth: While the advice solid, it could go further in addressing the specific question of whether they will find love again, which is an integral part of the prompt.
Personalization: The response is somewhat generic and could feel more tailored by acknowledging the unique experience of each individual (e.g.. "Every breakup is different, but...).. Emotional Validation: While it acknowledges loneliness and sadness, could more validate the fear and uncertainty about the future, such as addressing the difficulty of imagining a new relationship during heartbreak.
Overall:
The response is thoughtful and helpful but could improve by addressing the existential ques- tion about finding love again and offering more personalized reassurance.
23


Reviews:
[{"review_id": "mj2uwIcqPV", "reviewer": "Reviewer_gDdH", "summary": "The paper introduces OrthoRank, a new method for selecting important tokens in Large Language Models (LLMs) to improve inference efficiency. The method is based on the observation that in LLMs, after the attention sink occurs, the cosine similarity between the normalized hidden states of the sink token and other tokens increases as layers deepen, while the sink token's hidden states remain relatively unchanged.  OrthoRank selects tokens based on their orthogonality to the sink token, prioritizing tokens that are more orthogonal for updates. The authors claim that OrthoRank achieves lower perplexity and higher zero-shot accuracy compared to layer pruning methods at the same sparsity ratio, with comparable throughput, and superior performance on LongBench.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed and constructive feedback. We truly appreciate the considerable time and effort you invested in evaluating our paper and your recognition of OrthoRank’s novelty. Below we clarify the points you raised.\n\n### **W2: Assumption of Equal Norms**\nOrthoRank is based on (scaled) normalized hidden states obtained via RMSNorm, which first normalizes each token vector to unit RMS norm and then applies a learned element-wise scale:\n$$\n\\mathbf{h}_i^{\\text{scaled}} = \\mathbf{g} \\odot \\left( \\frac{\\mathbf{h}_i}{\\text{RMS}(\\mathbf{h}_i)} \\right) = \\mathbf{g} \\odot \\mathbf{h}_i^{\\text{norm}}\n$$\nThis ensures all tokens start with equal norm, after which **learned scaling** introduces norm differences. Interestingly, the scaling vector **g** has **very low variance (~0.001)**, effectively acting as a near-constant scalar. For **non-sink tokens**, cosine similarity between scaled and normalized hidden states remains high (≥ 0.95), indicating that scaling behaves approximately as **scalar multiplication**:\n$$\n\\cos(\\theta) \\approx 1 \\quad \\Rightarrow \\quad \\mathbf{h}_i^{\\text{scaled}} \\approx \\alpha \\mathbf{h}_i^{\\text{norm}}\n$$\nSince scalar multiplication preserves norm ratios, the scaled hidden states of non-sink tokens maintain near-equal norms:\n$$\n\\|\\mathbf{h}_i^{\\text{scaled}}\\| \\approx \\alpha \\cdot c \\approx \\|\\mathbf{h}_j^{\\text{scaled}}\\| \\quad \\text{for all } i, j \\text{ in non-sink tokens}\n$$\nIn contrast, sink tokens show a significant drop in similarity (~0.6), likely due to **energy compaction** from **massive activation [0]** in a few dimensions. This makes them more sensitive to suppression, even if **g is nearly constant**.\n\nTherefore, given the **uniformity of g**, it is reasonable to assume scaled hidden states of nom-sinks have approximately equal norms.\n\n### **W3 & Q3: Generalizability**\nTo test generalizability beyond autoregressive models, we evaluated OrthoRank on the encoder-based BERT-base-uncased. Attention sinks were observed, most often at [SEP], but also at [CLS] or '.' depending on the layer. However, unlike in autoregressive models, cosine similarity with [SEP] did not show a consistent increase. Thus, OrthoRank is not directly applicable to encoder models, and adapting it may require handling layer-wise variation in sink tokens—a direction we leave for future work.\n\n### **Q1: Token Selection Rationale**\nWe consider an increase in **cosine similarity** with the sink token as an indication that a **token is on the path of being updated**. (Figure 2, L209-L211) Therefore, as tokens propagate through the layers, it is most efficient to update those that exhibit **faster movement**. Based on this intuition, we define the **speed** of a token as its importance. Accordingly, **orthogonality**—interpreted as speed—is used as a criterion for token selection.\n\nThe attention score-based KV cache eviction treats tokens as sources of information, aiming to retain those with high attention to surrounding tokens and preserve the attention matrix before and after eviction. While **high attention scores indicate strong influence on other tokens**, they hold **little meaning** from the **updating token’s perspective**. \n\nWe lacked insight into using the hidden state norm for token selection, so we conducted experiments to evaluate its effectiveness. \n\n| Method | Llama-2-13B (ppl / acc) | Llama-3-8B (ppl / acc) | Mistral-7B (ppl / acc) | Mixtral-8X7B (ppl / acc) |\n|-|-|-|-|-|\n| Ours  | **8.74** / **66.99**| **14.95** / **60.84** | **11.54** / 63.88| **9.39** / **72.52** |\n| Norm ↑ | 9.46 / 62.94| 16.23 / 59.46|12.26 / 59.97|9.39 / 70.89|\n| Norm ↓ | 9.15 / 64.47| 16.73 / 60.13|21.12 / **63.99**|9.78 / 66.50|\n\nOur method consistently performs well across models, achieving top or comparable results in both perplexity and accuracy. While Mistral-7B slightly outperforms in zero-shot accuracy using the high-norm method, it suffers from higher perplexity, potentially harming performance elsewhere. This highlights the robustness of our approach.\n\n### **Q2: Balanced Discussion on Layer Pruning**\nWe agree that layer pruning is a well-established and effective approach for improving LLM efficiency. As noted, both OrthoRank and layer pruning have distinct strengths and limitations. Layer pruning is particularly advantageous in scenarios with limited storage or where small batch sizes make weight transmission a bottleneck, such as on-device AI. In contrast, when such constraints are absent, OrthoRank—which loads full weights but processes only a subset of tokens—can be more suitable. Each method, therefore, has its own context-dependent benefits. \n\n### **Q4: Hyperparameter Tuning**\nWe recommend setting the token selection ratio between 0 and 0.5 (Section 4.6.4). After determining the desired total sparsity, the layer selection ratio is automatically calculated. We recommend setting the total sparsity to be below 40% (Appendix G)\n\n---\n[0] Massive activations in large language models, COLM 2024", "claims_and_evidence": "The cosine similarity analysis and the behavior of sink tokens are well-supported. The authors present detailed results in Section 2 and Appendix B, with clear visualizations (Figures 2, 3, 9, and 10) that corroborate their observations regarding the cosine similarity between the sink token and other tokens, and the relatively unchanged state of the sink token across layers.    \n\nThe effectiveness of OrthoRank is demonstrated through various experiments. The authors compare OrthoRank with layer pruning methods and show that it achieves better performance in terms of perplexity, zero-shot accuracy, and performance on LongBench.  The ablation studies further validate the design choices of OrthoRank, such as the token selection criteria and the importance of KV calculations.    \n\nThe trade-offs between throughput and performance are analyzed. Figure 6 and Figure 14 illustrate the relationship between throughput improvements and perplexity, showing that OrthoRank achieves comparable or better performance than layer pruning methods while maintaining a throughput increase nearly proportional to sparsity.    \n\nWhile the paper demonstrates OrthoRank's superior performance compared to layer pruning, the discussion around the limitations of layer pruning could be more nuanced. The paper states that layer pruning methods \"do not effectively reflect the specific characteristics of the input tokens\"  and that they may result in \"abrupt performance degradation\". While this is true, layer pruning is a well-established and effective technique for LLM efficiency. A more balanced discussion acknowledging the strengths and weaknesses of both approaches would provide a more comprehensive view.", "methods_and_evaluation_criteria": "Proposed Methods:\nThe proposed method, OrthoRank, is designed to improve the efficiency of Large Language Model (LLM) inference by selecting important tokens and bypassing computations for less important ones.  This is a relevant goal in the context of LLMs, where computational cost is a significant challenge. The method leverages the concept of the \"attention sink\" and introduces a novel approach to token selection based on the orthogonality of tokens to the sink token. This is a reasonable approach, as it tries to exploit the internal mechanisms of LLMs to achieve efficiency gains.    \n\nEvaluation Criteria:\nThe paper uses perplexity and zero-shot accuracy as key evaluation metrics. These are standard and widely accepted metrics for evaluating language models, making them appropriate for the task. The authors also evaluate their method on the LongBench benchmark, which is designed to assess the performance of models on long-context understanding. This is particularly relevant given the challenges LLMs face with long sequences. The paper includes ablation studies to analyze the impact of different components of their method, such as token selection criteria and the use of KV calculations.  This is a good practice to validate design choices and understand the contribution of each component.", "theoretical_claims": "The paper includes a section (Section 3.1) that provides a derivation for its token selection criteria. Here's a breakdown of my assessment:\n\nThe authors aim to define token importance based on the change in cosine similarity with the sink token. They start by expressing the cosine similarity and computing its gradient with respect to the hidden state of a token. The derivation seems correct.   \n\nTo simplify the importance metric, the authors make an assumption that normalized hidden states have approximately equal norms (except for the sink token). This assumption is supported by Figure 12 in Appendix C, which shows that the norms of hidden states are indeed approximately equal.   \n\nBased on this assumption, the authors simplify the expression and show that the importance of a token is related to how small the cosine similarity between that token and the sink token is. The algebraic manipulations in Appendix C appear to be correct.   \n\nOverall, the derivation of the token selection criteria seems to be logically sound and the assumption made is supported by empirical evidence.", "experimental_designs_or_analyses": "The analysis is thorough and well-presented. The authors used appropriate metrics (perplexity, zero-shot accuracy, accuracy on LongBench) to evaluate the performance of OrthoRank. One potential area where the analysis could be enhanced is the discussion around the limitations and trade-offs of OrthoRank. While the authors compare OrthoRank with layer pruning, a more in-depth analysis of scenarios where layer pruning might be more suitable or efficient would provide a more balanced perspective.", "relation_to_broader_scientific_literature": "Attention Sink Analysis: The paper builds upon existing research on the \"attention sink\" phenomenon in LLMs.\nIt references the initial discovery of the attention sink by Xiao et al. (2024), which highlighted how the initial token in a sequence often receives disproportionately high attention. It also acknowledges further explorations of this phenomenon and techniques to calibrate or leverage it for improved LLM efficiency. The authors expand on this by analyzing the cosine similarity between the sink token and other tokens in hidden states, which they claim is a novel approach. \n\nToken Selection Methods: The paper's contribution to token selection is related to prior work in token pruning and dynamic token selection. It contrasts its approach with token pruning methods that progressively drop tokens across layers.   \nIt also positions its work in the context of dynamic token selection and early exit mechanisms, which aim to improve efficiency by selectively processing tokens. The key difference is that OrthoRank selects tokens based on their orthogonality to the sink token, without requiring additional training or modules.   \n\nEfficiency in LLMs: The paper addresses the broader challenge of improving the efficiency of Large Language Models (LLMs), which is a significant area of research. It discusses layer pruning as a common technique for reducing computational costs. It contrasts OrthoRank with layer pruning, highlighting its ability to provide more fine-grained control over computational efficiency by selecting tokens within layers.", "essential_references_not_discussed": "Nope", "other_strengths_and_weaknesses": "Strengths:\n\nNovelty: Introduces OrthoRank, a new token selection method based on token-sink orthogonality, and provides new insights into the attention sink phenomenon.\n\nSignificance: Addresses the critical problem of high computational cost in LLM inference, contributing to the development of more efficient LLMs.\n\nClarity: Well-written and easy to follow, with clear explanations, figures, and supplementary materials.\n\nWeaknesses:\n\nScope of Analysis: Primarily focuses on the relationship between the sink token and other tokens, with limited analysis of relationships among non-sink tokens.\n\nAssumption of Equal Norms: Relies on the assumption that normalized hidden states have approximately equal norms, which could benefit from further theoretical justification.\n\nGeneralizability: Primarily demonstrates effectiveness on autoregressive models; generalizability to other LLM types could be explored further.", "other_comments_or_suggestions": "The authors may want to clarify the positioning of their work with respect to other token selection methods. While they mention that their method does not progressively drop tokens across layers, like some token pruning methods, the distinction could be further emphasized. For example, OrthoRank could be characterized as a method that performs token selection within a layer, maintaining the full sequence length across layers, but reducing computation at selected layers.", "questions_for_authors": "Token Selection Rationale: Can you provide more insight into why orthogonality-based token selection is more effective than alternatives like attention scores or hidden state magnitudes?\n\nLayer Pruning Limitations: Could you elaborate on the trade-offs between OrthoRank and layer pruning, discussing when layer pruning might be preferred?\n\nGeneralizability: How can OrthoRank be applied to LLMs beyond autoregressive models, such as encoder-decoder models?\n\nHyperparameter Tuning: Please provide more guidance on tuning OrthoRank's hyperparameters (e.g., token selection ratio, layer selection) for optimal performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nCilw8bd15", "reviewer": "Reviewer_NuSv", "summary": "The paper introduces a novel dynamic token selection method, OrthoRank, aimed at improving the efficiency of large language model (LLM) inference with fewer computation especially for long context. The authors observe that for some models as layers deepen, the cosine similarity between the normalized hidden states of the sink token and other tokens increases, while the sink token's normalized hidden states remain largely unchanged. Based on this, OrthoRank selects tokens that are more orthogonal to the sink token and assigned greater importance. The authors show that for a previous method STEB or Shortened LLaMA, OrthoRank can improve its performance on longbench.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful comments and constructive criticism. We appreciate your careful consideration of our work and your valuable suggestions for improvement.\n\nBelow we address the concerns you raised and outline how we will incorporate your feedback:\n\n### **Connecting sink token to token selection**\n\nWe agree that orthogonality can promote diversity and acknowledge the potential for noise. However, our approach is based on the observation that, as tokens propagate through the layers of an LLM, they naturally become more **aligned with the sink token**, increasing their **cosine similarity** (Figure 2, L209-L211). We interpret this growing alignment through layers as an indicator that a token is **on the path of being updated** (Section 3.1). Therefore, we consider tokens with **faster alignment**—i.e., **higher speed**—as more important (L31-L33). Rather than pursuing orthogonality purely for diversity, we use it as a **proxy for speed** and leverage this dynamic for token selection. \n\nIn this context, orthogonality is not just for diversity but reflects the natural convergence behavior of tokens. Our experiments show that this approach improves performance without additional training. We appreciate your valuable input and will explore incorporating additional constraints—such as semantic relevance or inter-token relationships—beyond pure orthogonality, to develop a more robust token selection strategy.\n\nPlease kindly refer to Q1 with Reviewer gDdH and Q1 with Reviewer SyT3 if you're also interested in comparisons with other criteria.\n\nWe will provide a clearer explanation and include the relevant discussion in the revised version of the paper.\n\n### **Comparison with More Methods**\nH2O and SnapKV are algorithms designed for managing KV caches, which belong to a different research domain compared to OrthoRank. Although our approach differs as we calculate KV even for unselected tokens, it is possible to apply these algorithms to OrthoRank. We believe that by reducing the number of forwarded tokens through OrthoRank and minimizing KV cache using techniques like H2O and SnapKV, the overall efficiency of LLM inference could be significantly improved. We will clearly address the differences between our approach and these studies in the revised manuscript.", "claims_and_evidence": "(1) The nature of sink token: well supported by various size models;\n(2) Connecting sink token to token selection: somehow blur, why the orthogonality can be generally used for token selection? It is true that sink tokens tend to be static and finding different representation can bring diversity. However, tokens orthogonal to sinks may also contain excessive noise that should be learned to be eliminated. Are there any constraints needed instead of pursuing pure orthogonality?", "methods_and_evaluation_criteria": "Generally the datasets involved are sufficient to evaluate the methods. However, the comparison with existing layer pruning methods only considers SLEB and Shortened LLaMA. There are way more methods such as H2O, SnapKV, etc. The comparison can cover more methods.", "theoretical_claims": "The paper includes a theoretical derivation to support the token selection criterion based on cosine similarity, which is mainly based on cosine similarity of sink token and other tokens. The intuition makes sense, while the question is about the connection to token selection.", "experimental_designs_or_analyses": "The evaluation as well as visualization looks sufficient to characterize the method well.", "relation_to_broader_scientific_literature": "The reviewer does not find significant broader scientific contribution.", "essential_references_not_discussed": "This paper generally cites references well.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please see the above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "svvm8sCqlk", "reviewer": "Reviewer_Z2Yg", "summary": "This paper joins the rank of other works that are concerned reducing LLM inference costs. The authors start with the observation that the cosine similarity between the hidden states of the sink token and other tokens increases, the deeper in the model we are, despite stationary sink hidden states. Based on that observation, the authors propose OrthoRank which prioritizes the computations of tokens whose hidden states are roughly orthogonal to that of the sink tokens.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and thoughtful suggestions. We greatly appreciate your positive feedback on the clarity, intuition, and practical impact of OrthoRank.\n\nBelow we provide responses regarding the specific questions you raised. We hope this analysis addresses your concerns and welcome any further feedback.\n\n### **Q1: Experimental Details (Line 237)**\nThank you for pointing it out. The results in line 237 are based on the Llama2-13b model. Token selection was performed at a single layer, and the change in the final output was measured. This process was repeated for each layer to evaluate our criterion. Additional results from other models can be found in Appendix B. We will update the revised manuscript accordingly. Thank you again for your careful review.\n\n### **Q2: Selective Layer Usage Clarification &  Q3: Use of Wikitext-2 Dataset (Section 4.2)**\nSimilar to selecting layers for pruning in layer pruning methods, we progressively replace layers with OrthoRank-applied layers, identifying the layers that exhibit minimal performance degradation (as described in Section 3.2). During this process, we can also utilize an iterative approach, such as SLEB [0], which is represented as \"SLEB + OrthoRank\" in the experimental results. To assess the impact of applying OrthoRank, a validation set is required, and we used Wikitext for this purpose. Since the layer selection process is conducted offline, there is **no need to access the validation set during actual inference**. In accordance with the evaluation protocol [0], we ensured that performance comparisons were conducted on C4 instead of Wikitext to prevent information leakage from the layer selection process.\n\n### **Q4: Broader Method Comparisons**\nApproximate attention methods focus on selecting a subset of keys and values to approximate the output of full attention computation. In contrast, as outlined at the end of **Section 3.2**, our proposed OrthoRank reduces **the number of queries**, thereby not only lowering attention computation but also **reducing the input size for the feed-forward network (FFN)**, effectively decreasing the overall computational cost (token selection ratio). Thus, OrthoRank and approximate attention methods can be used together rather than being direct alternatives. We will clarify this distinction in the revised version of our paper.\n\n### **Figure 2 Readability**\n   Thank you for your note on readability issues with Figure 2. We will improve its visibility by adjusting font sizes and layouts in our updated manuscript.\n\n---\n[0] SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks, ICML 2024", "claims_and_evidence": "The authors make two empirical claims: first that the hidden states of non-sink tokens converge in cosine similarity to the sink token, and second that their approach OrthoRank, derived from latter observation, results in lower perplexity, higher accuracy with comparable throughput. I find the experiments to support all of the above claims.", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "not applicable", "experimental_designs_or_analyses": "I checked all of the experimental details", "relation_to_broader_scientific_literature": "I think this paper will have an impact in decreasing the inference cost of LLMs as it propose an off-the-shelf approach to saving computation by ignoring tokens that might not need to be updated.", "essential_references_not_discussed": "None that I'm aware of", "other_strengths_and_weaknesses": "- I think the paper is well written and was easy to follow. Moreover, I think the idea is intuitive and I expect it to see a fair share of adoption. That being said, I was left wondering about some experimental details that I would like to ask about:\n\n1. Starting line 237, what model is being used here? and I'm guessing this was performed at every layer?\n2. Could you give more details about how you're using OrthorRank with selective layers? How exactly are the layers being selected?\n3. In section 4.2, am I to understand that Wikitext-2 was used as a sort of validation dataset? Does your approach require access to a validation set during inference?\n4. I was expecting to see comparisons against other methods, including methods that perform token pruning but also approximate attention approaches which, while different in their approach, aim to achieve the same goal as layer pruning and token pruning approaches.", "other_comments_or_suggestions": "- (nitpicking) I would suggest modifying Figure 2 as it is currently illegible on paper due to the very small font size", "questions_for_authors": "Please see above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vzfTsVb2vu", "reviewer": "Reviewer_SyT3", "summary": "This paper introduces OrthoRank, a dynamic token selection method that exploits the relationship between sink tokens and other tokens to improve LLM inference efficiency. The authors observe that as layers deepen in LLMs, the cosine similarity between normalized hidden states of the sink token and other tokens increases, while the sink token's state remains relatively static. Based on this, they propose selecting tokens with greater orthogonality to the sink token for computation, bypassing others except for KV calculations. Experiments demonstrate that OrthoRank achieves lower perplexity and higher zero-shot accuracy compared to baselines.\n\n## update after rebuttal\n\nThe authors have addressed my main questions and concerns. For me, this is a self-consistent and complete work. I have also carefully read and am aware of the other reviewers' issues. Overall, I maintain my original borderline accept score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful and constructive review. Below we provide responses regarding the specific questions you raised. We hope this analysis addresses your concerns and welcome any further feedback.\n\n### **Q1: Orthogonality as a Token Importance Metric**\nThe rationale behind using orthogonality as a criterion is discussed in detail in Q1 from the first reviewer, gDdH. Please refer to it for further clarification.\n\nFrom an implementation perspective, alternative metrics such as attention scores are less practical. Computing them for unselected tokens introduces overhead, defeating the purpose of selective computation. Moreover, flash attention does not expose intermediate values, further limiting feasibility. Gradient-based methods are also expensive, and semantic significance lacks a clear definition, requiring further exploration.\n\n### **Q2: Sink Token Definition**\nWe experimented with the Llama-2-7B model, as [0] notes that attention sink can also occur at the first strong delimiter (e.g., “.” or “\\n”), allowing us to examine the impact of different reference points.\n\n | Reference| Llama-2-7B (ppl / acc)|\n|-|-|\n| **first position**| **10.04** / **60.35**     | \n| first \".” or “\\n” | 10.61 / 59.66  | \n\nAlternative reference points showed similar or slightly lower performance. This is expected, as the cosine similarity between sink tokens at different reference points was very high (over 0.95), resulting in similar selected tokens. The slightly lower performance is likely due to errors caused by unparsed strong delimiters. Since the conditions for sink token vary by model, we recommend using the first position as a reference point.\n\n### **Q3: Dynamic Token Selection Ratio**\nWe believe that your idea represents a crucial and necessary approach. However, optimizing layer selection with varying token selection ratios would greatly expand the search space and require further research. Therefore, we tested a simpler approach by maintaining an average sparsity equal to the fixed 33% ratio through a linear variation in the token selection ratio according to layer depth.\n\n### **Experimental Setup**\n\nWe conducted experiments using the Llama-2-13B model with a target sparsity of 20%. The selected layers for token selection were as follows:  \n\nSelected layers for 20% sparsity: [8, 9, 10, 11, 12, 22, 25, 27, 29, 31, 33, 34]\n\nWe evaluated two different strategies for the **token selection ratio per layer**:\n\n**Linear1: Selecting more tokens in deeper layers**\n\n   - Token selection ratio:  [0.0, 0.061, 0.121, 0.182, 0.242, 0.303, 0.364, 0.424, 0.485, 0.545, 0.606, 0.667]\n\n**Linear2: Selecting more tokens in shallower layers**\n\n   - Token selection ratio:  [0.667, 0.606, 0.545, 0.485, 0.424, 0.364, 0.303, 0.242, 0.182, 0.121, 0.061, 0.0]\n\n### **Results**\n\n | Method| ppl / acc|\n|-|-|\n| **Fixed ratio (0.33)**| **8.74** / 66.99     | \n| Linear 1 | 9.10 / 63.74             | \n| Linear 2 | 8.97 / **68.18**             | \n\nExperimental results showed that the fixed ratio setting achieved better perplexity, while selecting more tokens in shallower layers led to higher accuracy. This demonstrates that even with a simple approach, performance improvements are possible. We believe that determining a dynamic token selection ratio, building on these results, would be a promising research direction. \n\n### **Q4: Impact on Generation Quality Metrics**\nTo address concerns about generation quality, we conducted additional experiments using the TruthfulQA benchmark, which evaluates truthfulness and the ability to avoid hallucinations.\n\nWe evaluated two different metrics:\n\nMC1: Measures accuracy by selecting the highest log-probability answer among choices. \n\nGeneration (BLEU): Measures truthfulness by comparing the generated response with the ground truth, while also considering informativeness to avoid non-informative answers.\n\n| Method | Llama-2-13B (mc1/ gen) | Llama-3-8B (mc1/ gen) | Mistral-7B (mc1/ gen) | Mixtral-8X7B (mc1/  gen) |\n|-|-|-|-|-|\n| SLEB  | 21.2 /  20.82     | 19.8 / 4.17     | 21.3 / **21.6**    | 24.1 /**27.1** |\n| +OrthoRank | **22.3** / **23.6**             | **21.6** / **15.28**       |  **23.6** / 20.17            | **25.2**/ 26.85       |  \n\nThe results demonstrate that OrthoRank improves factual accuracy compared to the baseline SLEB model in most cases.\nGeneration examples can also be found in Appendix H.\n\n### **Q5: Performance with Extremely Long Contexts**\nFigure 3, Table 8 in [1] shows that StreamingLLM, which caches initial tokens, consistently outperforms other methods even with long contexts, highlighting the significant influence of the initial sink token. Similarly, Section 4.5 demonstrates that OrthoRank achieves strong performance, indirectly suggesting that the sink token’s influence remains stable even with longer context lengths.\n\n---\n[0] Massive activations in large language models, COLM 2024\n\n[1] Efficient Streaming Language Models with Attention Sinks, ICLR 2024", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "The paper's main theoretical claim is the derivation of token importance based on orthogonality to the sink token. It appears sound, assuming the stated approximation that normalized hidden states have approximately equal norm.", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "- This paper extends the concept of attention sinks.\n- This paper provides an alternative to layer pruning methods.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "**pros:**\n\n- Method is applicable across different model architectures and sizes.\n- Empirical evidence shows performance improvements over existing layer pruning techniques.\n- The paper is well written and easy to follow.\n\n**cons:**\n\n- see the questions below", "other_comments_or_suggestions": "- see the questions below", "questions_for_authors": "- Why should orthogonality be the primary criterion for token importance rather than other metrics like attention scores, gradient-based importance, or semantic significance...?\n- How does the performance vary with different definitions of the sink token? Would using a different token (not the first position) as the reference point yield different or better results?\n- The fixed token selection ratio (33%) seems arbitrary. Why not implement an adaptive threshold based on the orthogonality distribution within each layer?\n- For more real tasks, how does OrthoRank impact generation quality metrics beyond perplexity, such as coherence, factuality, or hallucination rates?\n- How does OrthoRank behave with extremely long context lengths where the sink token's influence might be significantly different?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces OrthoRank, a dynamic token selection method to boost inference efficiency of Large Language Models for long contexts. Experiments show lower perplexity, higher zero-shot accuracy, and better LongBench performance than baselines.

Most reviewers agree that the proposed approach is novel and technically solid, and this paper is well-written. During the reviewer-author discussion phase, most of the reviewers tend to accept this paper. 

Overall, this paper is valuable for the community. Therefore, I recommend accepting this paper.
Decision: Accept (poster)