Star Attention: Efficient LLM Inference over Long Sequences
Authors: Shantanu Acharya
Fei Jia
Boris Ginsburg
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Transformers
Attention Mechanisms
Sparse Attention
Long Context
Inference Optimization
Distributed Systems
Primary area: deep_learning->attention_mechanisms
TL;DR: Training-free inference technique combining block-sparse and global attention for efficient long-sequence processing, achieving up to 11× speedup while retaining 97–100% accuracy.

Abstract: Inference with Transformer-based Large Language Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism. We introduce Star Attention, a two-phase block-sparse approximation that improves computational efficiency by sharding attention across multiple hosts while minimizing communication overhead. In the first phase, the context is processed using blockwise-local attention across hosts, in parallel. In the second phase, query and response tokens attend to all prior cached tokens through sequence-global attention. Star Attention integrates seamlessly with most Transformer-based LLMs trained with global attention, reducing memory requirements and inference time by up to 11x while preserving 97-100% of accuracy.

Star Attention: Efficient LLM Inference over Long Sequences
Shantanu Acharya'
Fei Jia"
Boris Ginsburg'
Abstract
these methods improve training efficiency, autoregressive decoding during inference still requires the model to attend to every previous token, resulting in high computational costs for long-context sequences. Other approaches attempt to optimize inference by segmenting long inputs into chunks, encoding them separately, and attending to these encoded chunks using the user query (Beltagy et al., 2020; Russak et al., 2024; Liao et al.. 2024). However, these methods of- ten require fine-tuning the model or introducing additional components that necessitate further training, limiting their out-of- applicability.
Inference with Transformer-based Large Lan- guage Models (LLMs) on long sequences is both costly and slow due to the quadratic complexity of the self-attention mechanism. We introduce Star Attention, two-phase block-sparse approx- imation that improves computational efficiency by sharding attention across multiple hosts while minimizing communication overhead. In the first phase, the context is processed using blockwise- local attention across hosts, in parallel. In the sec- ond phase, query and response tokens attend to all prior cached tokens through sequence- global attention. Star Attention integrates seamlessly with most Transformer-basee LLMs trained with global attention, reducing memory requirements and inference time by up to 11x while preserving 97-100% of accuracy.
We introduce Star Attention', a novel algorithm for efficient LLM long-context inference. This method is based on the observation that LLM inference usually has two stages: (I) prompt encoding, where the model processes input and stores KV vectors in the cache and (2) token generation, where model attends to the KV cache and autoregressively generates new tokens while updating the cache with the new KV vectors. In many long-context tasks, the input consists of long context followed by a short query and a short answer. The information needed for answering the query often localized within small parts of the context, meaning context tokens need only attend to nearby tokens, while query tokens need to attend to all prior tokens. Based on this observation, Star Attention utilizes two-phase approach shown in Figure 1:
1. Introduction
Recent Large Language Models (LLMs) can support con- texts up to millions of tokens in length (Gemini-Teann 2024; Anthropic, 2024; Meta-AI, 2024; Qwen, 2025), unlocking applications such as repository- document summarization, and large corpus retrieval. How- ever, processing such long sequences with LLMs requires substantial computational and memory resources due to the quadratic complexity of the self-attention mechanism.
code analysis, multi-
1. Context Encoding:
The context is divided into con- tiguous blocks and distributed across "context" hosts, with each host also receiving copy of the first block (an "anchor block"). Hosts compute self-attention only for their assigned blocks, without communicating with each other, reducing attention complexity from quadratic to linear with respect to context length. This distributed processing is similar to Ring Attention (Liu et al., 2024a) but without the "ring" communication during context encoding (Figure Ia).
The importance of long-contexs capabilities has driven sub- stantial research into addressing the computational chal- lenges of self-attention. Some approaches focus on reducing the need to fully materialize the attention matrix (Milakov & Gimelshein, 2018), leading to blockwise processing tech- niques (Dao al., 2022; Dao, 2024; Liu Abbeel, 2023) and further optimization through distributed computation across multiple compute units (Liu al., 2024a). While
2.
Query Encoding and Token Generation: The query is replicated across all hosts where initially attends to the KV cache on each host. Global attention is then computed by aggregating the results at a designated "query" host by efficiently communicating a single vec-
'NIIDIA. Correspondence to: Shantanu Acharya <shan- tanua @ nv rvidia.com>>, Fei Jia <fjia@nviiiacom>>
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
'Code: http://ghhhaaccmm////DAA/Aaaaa

---

Star Attention
C1
C2
c3
Cache
Cache
Layer N
Layer N
Layer N
Next Token Regrossive Decoding
Layer
Layer
Layer 2
Layer
Layer
Layer N
C2-RY
Layer N
Layer
Host
Host
Host
Update cache
+nnn-Oniinee Saftmace mode
Online Softmmaee
C1-V 2g
Layer 2
C2-VV ig
Layer
(3-MV Lg
Layer
C1
Online Softmar bi
Online Softmar
Anchor
Anchor
Layer 1
Layer
Layer
Host 3
Host 1
Host 2
(Query Host)
Query
Context
Query
(a) Phase I: Local Context Encoding with Anchor Blocks.
(b) Phase 2: Query Encoding and Output Generation with Global Attention.
Figure 1. Star Attention inference flow across two phases. (a) Context Encoding: The input context is partitioned into blocks and distributed across hosts, where each block (except the first) is prefixed with the anchor block (c1). Each host processes assigned block and stores the non-anchor portion of the KV cache. (b) Query Encoding and Token Generation: The query is broadcast to all hosts, which compute local altention using cached KVs. designated "query" host then aggregates softmax normalization statistics to compute global attention and generates the next token.
tor and scalar per token from each context host. Only the query host updates its KV cache during this stage (Figure lb).
Star Attention enables the context length scale linearly with the number of hosts by distributing the context process- ing across multiple hosts. Star Attention is compatible with most Transeormer-based LLMs trained with global atten- tion, operating seamlessly out-of-the-boy without additional model fine-tuning. Furthermore, we combine Star Attention with Flash Attention, allowing for additional speedup en- hancements. We evaluate Star Attention for Alama3.1188 and Llama3.1-70B (Meta-Al, 2024) on several long-context benchmarks. Star Attention achieves up to I1 times faster in- ference while maintaining 97-100% of the baseline accuracy.
Figure 2. Block sparsity pattern in Star Attention for a sequence partitioned into S context blocks a and query block q. Each context block attends only to itself and the "anchor block" whereas the query attends the entire input.
2.1. Phase 1: Context Encoding
Given an input sequence comprising context followed by a query q, the context e divided into n contiguous blocks: c = c2........ Ca]. where each block e; contains tokens. We introduce an anchor block mechanism, which, each block-excepp the first-is prefixed with the first block ci of the sequence, referred to the anchor block. This concatenation forms an augmented context c':
2. Star Attention Algorithm
Star Attention operates in two phases: (1) Context Encoding, where the long context is divided into contiguous blocks and is processed with local blockwise attention, and (2) Query Encoding and Token Generation, where the query is processed, and answer tokens are generated using global attention. Below, we detail each phase of the algorithm.
= [c1. (c1 02), (c1 cz). (c1 cx)
where each augmented block d; contains 2b tokens: b tokens from the anchor block 01 followed by tokens from the current block e (Figure 2). The positional indices of ci
2

---

Star Attention
Avg. Attn. Score
Avg. Attn. Score
Avg. Attn. Score
0.45
0.45
0.45
Actertion Sink
Attontion Sink
Attenkin Sinks
0.15
0.15
0.15
0
1000
2000
3000
1000
2000
3000
0
1000
2000
3000
Position IDs
Position IDs
Position IDs
(a) Global Attention
Blockwise Encoding
(c) Blockwise Encoding w/ Anchor Block
Figure 3. Attention distribution across the sequence during context encoding under different strategics in Phase (a) Global attention exhibits single attention sink at the sequence start. (b) Without anchor blocks, blockwise context encoding creates multiple attention sinks the start of each block. (c) With anchor blocks, attention sinks shift anchor tokens, yielding distribution that closely approximates global attention. The sequence is 4K tokens long and partitioned into 512-token chunks.
preserved, ensuring that its tokens retain their original position indices [0, 1,..., b- 1]. The augmented blocks are distributed across compute hosts, where each host computes attention over the 2b tokens from its assigned block c4 and generates the corresponding key-value (KV) vectors. While KVs for the anchor block с₁ discarded, the KVs for the current block
attention is computed as:
QKI vd
exp
AA
V/
(1)
/e k=
[QKA
exp
vdd
retained in the cache.
In addition to An, each host also stores the sum of the exponents 84 from the the local softmax operation (the denominator from Equation 1):
We observe that, without anchor blocks-i.. applying blockwise attention only to the original context e-the model fails to generate correct outputs. We conjecture this failure due to the incorrect approximation the attention patterns observed during phase 2 (Figure 3b), where multi- ple attention spikes, known attention sinks (Xiao al., 2024b), distributed across the sequence. These spikes occur because each block is processed independently, creat- ing an attention sink at the start of each block. As a result, the model struggles to effectively focus on relevant parts of the context. To address this issue, we prefix the blocks with the anchor block C1, shifting the attention sinks the an- chor tokens. By discarding the KVs of the anchor tokens the intermediate attention sinks are removed ensuring the atten- tion distribution of block-local attention (Figure 3c) closely approximates global attention (Figure 3a) while maintaining the computational efficiency of blockwise processing.
Lx
QKT m.k.
8h
exp
Vd
The query-host fy gathers the local attention Ad and the sums of exponents 8h from all hosts:
A [Ai, Ag Ar]
$ $1, sHl
The global softmax denominator, Sylohals is then computed as the sum of all local exponents:
Σ
Sglobal
8h
The query-host uses Sglobal to aggregate the local attentions to compute the global attention:
2.2. Phase 2: Query Encoding and Token Generation
In phase 2, global attention is employed to encode the query and generate output tokens by using a distributed softmax algorithm that eliminates the need to transfer KV cache between hosts (Figure 1b). A designated query-host hg coordinates this computation. The query broadcast to all hosts and transformed into the sequence 1, is the query length, and d is the attention head dimension. Each host h computes the local attention output A, for the query Q using its local key-value pairs KR, Vn E TR xd where 4g is the sequence length of the KV cache. The local
H
8h
Aglobal
A
(4)
Sglobal
This method ensures that the global attention scores are normalized correctly across all hosts. It requires the com- munication of only single scalar 8,, (the local sum of exponents) and a vector A₁, (the local attention) per token.
1R4xd where
The above formulations provide a simplified conceptual overview. In practice, for efficient inference, we use Flash Attention (Dao, 2024) to attend to the KV cache on each
3

---

Star Attention
Table Accuracy and relative inference speedup of Star Attention compared Ring Attention RULER across sequence lengths from 16K to 128K. Accuracy is reported as the absolute difference from Ring Attention and speedup reflects relative improvements in inference efficiency. Star Attention significantly accelerates inference with minimal accuracy loss.
Seq. Len. (K)
Block Size
Ring-Attm Acc.(%)
Star-Attn
Model
(K)
A Acc.
A Speedup
16
4
92.22
-0.94%
1.1x
Llama-3.1-8% 1omma-3.-8B--nsstruce
32
8
87.53
+1.17%
1.2x
(Meta-AI, 2024)
64
16
84.79
-1.42%
1.8x
128
32
76.31
-1.90%
2.7x
16
4
95.09
-2.71%
1.7x
Llama-3.1-70B 1--00B-Instruct (Meta-AI, 2024)
32
8
94.61
-2.55%
2.0x
64
16
88.54
-1.44%
4.7x
host and apply the log-sum--exp trick from online softmax (Milakov & Gimelshein, 2018) ensure numerical stability during global attention aggregation.
the impact of model scale using JJama-3.1-70B-Inntti Across all configurations, Star Attention demonstrates in- creasing speedup benefits with larger models and longer sequences.
Output generation and cache update. After computing the global attention output, the query-host hg generates the next token and its KV cache updated with the key and value vectors of the new token. This process is repeated for each generated token.
Baseline. We compare Star Attention against three strong baselines: (i) Ring Attention (Liu al., 2024a), a distributed attention mechanism that computes global block wise atten- tion by circulating each host's KV cache in ring pattern across all the hosts; (ii) StreamingLLM (Xiao et al., 2024b), sparse attention method that combines global sink tokens with sliding window attention. We use a configuration hav- ing 1000 global sink tokens along with a sliding window of 8000 tokens; and (iii) MInference (Jiang al., 2024), which utilizes three distinct sparse attention patterns, dynamically selecting the optimal pattern per head in an oflline search setting. Among these, only Ring Attention is distributed algorithm designed to scale inference across multiple GPUs. Since Star Attention also targets distributed efficiency, we report speedup metrics relative to Ring Attention, while accuracy comparisons are provided for all three baselines.
This two-phase mechanism--oce context encoding with anchor blocks in Phase 1 followed by global query encoding with token generation in Phase gives significant improve- ments in inference speed, while keeping the accuracy close to the global attention.
3. Experiments
We empirically evaluate Star Attention using several Ilama- based models across multiple long-context benchmarks with sequence lengths ranging from 16K to 1M tokens, assessing both its accuracy and inference speedup relative to estab- lished baselines. We also investigate the accuracy-sped trade-off's function of block size and provide a granular breakdown of Star Attention's 's effectiveness across different domains. Our results demonstrate that Star Attention con- sistently achieves peartparity with full global attention in accuracy while delivering substantial speedups, especially on large models and long-context tasks.
Configuration. We implement Star Attention in both HuggingFace Transformers library (Wolf al., 2020) and NVIDIA's TRT-LLM framework (NVIDIA, 2023). All experiments are conducted on NVIDIA A100 GPUs with bfloat 16 precision. Optimization techniques such as Flash Attention are applied uniformly across Star and Ring Atten- tion implementations to ensure fair comparison. Reported results are based on the HuggingFace implementation, with similar relative trends observed across TRT-LLM. Addi- tional details regarding our experimental setup can be found in Appendix B.
3.1. Setup
Models. We conduct experiments using both the base and instruct variants of 1 8B which support context lengths up 128K tokens (Meta-AI, 2024). To evaluate scalability beyond this range, we use gradientai-Llama InS--unstte and -anddeentai-Llmma------------tt 1048K that extend Llama-3-8B's context to 256K and IM tokens respectively (Gradient.ai, 2024). We further assess
Evaluation Benchmarks. We evaluate our method on three benchmarks, each testing unique aspects of long context understanding: (i) RULER (Hsieh et al., 2024); a syn- thetic benchmark with 13 tasks categorized into domains: Needle-in-a-Hasstiee (Retrieval), Multi-Hop Tracing, Ag-

---

Star Attention
gradientai/
gradientai/ Llama-3 -n---uutuuct11048
meta-llama/ LIama-3.1-80 1-8e-Base
Llama-3-88 3-mma-3-BB-o S--IInstuuct2224
accuracy (%)
90
RULER
80
70
Global Attention
Star Attention (Ours)
BABILong accuracy (%)
40
30
16K
32K
64K
128K
16K
32K
64K
128K
16K
32K
64K
128K
Sequence Length
Figure 4. Accuracy comparison of Star Attention and Global Attention RULER and BABILong from 16K 128K sequence lengths using various models. All runs use block and anchor block size set to one-quarter of the total sequence length. Star Attention maintains 97-100% of the accuracy global attention, and in some cases, even outperform it.
gregation, and Question Answering. (ii) BABILong (Kura- tov al., 2024): a benchmark of 5 tasks requiring reasoning over multiple supporting facts encoded in the context to generate accurate answers. (iii) InfiniteBench (Zhang al., 2024): a diverse collection of 10 real-world and synthetic tasks spanning summarization, multilingual QA, code de- bugging, and retrieval. Further details on the benchmarks and specific tasks can be found in Appendix C.
across all tasks up to 128K sequence length, with an accu- racy drop typically below 3%. However, we observe anoma- lies for the Llama-3.1-SB base model on BABILong, likely due to format-specing generation requirements that chal- lenge non-instruction-tune models, particularly at longer sequence lengths.
3.3. Comparison with Other Sparse Attention Methods
While our primary comparison focuses on Ring Attention (Liu et al., 2024a) due to its distributed design, we also evalu- ate Star Attention against two strong non-distributed sparse attention baselines: StreamingLLM (Xiao et al., 2024b) and MInference (Jiang al., 2024). These methods repre- sent alternative strategies for long-context efficiency under constrained compute budgets and provide complementary perspectives on accuracy trade-offs.
3.2. Results
Table I presents the accuracy and relative speedup of Star Attention compared to Ring Attention (representing full global attention in distributed setting) on RULER, across sequence lengths from 16K 128K tokens. In each set- ting, the context and the anchor block size are quarter of the total sequence length. Star Attention main- tains high accuracy, typically within 0-3% of global atten- tion, while delivering significant speedups, ranging from 1.1x to 4.7 x, depending on the model size and sequence length. The speedup becomes more pronounced for larger models. For instance, the Llama-3. 1.-maa-..-70B-BB1 Innsttrtttt model exhibits 4.7x acceleration at 64K tokens with minimal accuracy drop. This highlights Star Attention's suitability for high-throughput inference and its ability to preserve model's 's accuracy even with significantly reduced context window.
one-
Table 2 reports accuracy on RULER using Llama-3. 3-B-8B- Instruct across sequence lengths from 16K to 128K tokens. Star Attention outperforms both the methods, with the per- formance gap widening longer context lengths. Notably, Star Attention maintains accuracy closest the baseline (full attention) across all settings, demonstrating its robust- ness in extended-context reasoning.
To assess generalization beyond synthetic tasks, we fur- ther evaluate all methods on InfiniteBench. As shown in Table 3, Star Attention achieves the highest average ac- curacy across 10 diverse tasks spanning summarization, multilingual QA, code debugging, and retrieval. It excels especially in retrieval-heavy tasks such as PassKey, Num- Retr, and KVRetr while also delivering competitive results across other categories. These findings highlight Star Atten-
To evaluate generalization beyond RULER, we bench- mark Star Attention on BABILong using Llama-3.1 3.1-BBBBBBBB....... Base, -aatraa----aaaa---------nttut-ee Llama-3-8B-Ing miussructt10488 As shown in Figure 4, Star At- tention consistently achieves near- near-parity with full attention
and gradientai-
5

---

Star Attention
Table Accuracy comparison of different methods RULER from 16K 128K sequenee length using Llama-3.1 10-ma-3.-8B-Instrueee Star
Attention performs closest to Full Attention and outperforms others at longer sequences.
Methods
16K
32K
64K
128K
Average
Full Attn.
92.22
87.53
84.79
76.31
85.21
StreamingLLM
74.76
48.56
26.2
30.77
45.07
MInference
93.27
86.54
84.86
58.17
80.71
Star Attention
91.27
88.70
83.37
74.41
84.44
Table Accuracy comparison of different methods on InfiniteBench using 1 1-8t-Instttct.. Star Attention performs closest to Full Attention and outperforms others across all the diverse tasks.
En.
En.
En.
En.
Zh.
Code. Debug
Math. Find
Retr. PassKey
Retr. Num
Refr. KV
Methods
Avg.
Sum
QA
MC
Dia
QA
Full Attn.
31.91
25.92
69.43
21.5
31.95
16.75
24.29
99.15
99.66
60
48.06
StreamingLLMM
30.15
10.15
41.05
8.5
22.38
8.63
17.71
2.71
5.93
14.72
Minference
31.04
22
63.76
14.5
28.7
5.33
27.43
56.78
77.12
14
34.07
Star Attention
31.85
25.92
69
22
30.37
24.37
26.29
93.22
96.27
45.8
46.51
tion's ability to generalize beyond synthetic benchmarks and handle real-world, instruction-heavy tasks with long-range dependencies.
in Appendix B.
3.5. In-Depth Analysis on RULER Task Categories
To better understand the strengths and limitations of Star Attention, we analyze its performance across different task categories within the RULER benchmark. RULER com- prises five categories: Single-NIAH. Multi-NIAH, Multi- Hop Tracing, Aggregation, and Question Answering (QA). Figure 7 reports category-wise accuracy using the Llama- 3.1-8B- IIstruutt model sequence length of 32K and block size of 8K. We observe consistent trends across all sequence lengths, as detailed in Appendix D.
3.4. Trade-off between accuracy and speed
Figure 5a illustrates the effect of varying block size dur- ing context encoding, with the sequence length fixed at 128K tokens. Larger block sizes lead improved accuracy, highlighting the benefits of increased receptive fields for long-context comprchension.
Empirically, setting the block size to approximately one- quarter of the total sequence length strikes an effective trade-off between accuracy and speed. For sequence lengths exceeding 128K, we fix the block size at 32K tokens to prioritize inference speed. As shown in Figure this config- uration allows Star Attention achieve substantial speedups over Ring Attention while incurring only modest accuracy degradation. For instance, on the RULER benchmark with 1an--------nnnttt---eee-
Star Attention performs comparably to global attention in the Single-NIAH, Multi-NIAH, and QA categories. These tasks typically involve localized retrieval or reasoning. where attention primarily operates within near single context block. In contrast, Multi-Hop Tracing presents greater challenge. It requires propagating information across multiple hops within the sequence, demanding effective inter-block communication. Since Star Attention restricts K V-cachee access to the local block during context encoding, the model lacks mechanism for long-ran rangrrage token-to-tole to-token aggregation in this phase. Consequently, performance de- grades relative to global attention.
Star Attention achieves up to 11x x speedup while retaining accuracy comparable to Ring Attention. At IM tokens, the speedup increases to 16.9x with an accuracy drop of just 5.32%.
These findings demonstrate that Star Attention offers flex- ible control over the accuracy-efficiency trade-off. Larger block sizes allow performance to approach that of global attention, while smaller blocks enable higher throughput for aatency-sensitive applications. The appropriate configura- tion can thus be tuned based on available resources and task requirements. Additional experimental details are provided
Interestingly, Star Attention shows substantial gains in Ag- gregation tasks, especially those involving frequency analy- sis or summarization over distributed spans. Its chunk-wise encoding facilitates local aggregation within blocks, which is later synthesized during the global query phase. This
6

---

Star Attention
Seq Length: 128K
Seq Length: 128K
76
74
RUL EILER-MA Accuracy
95
72
RULER Accuracy
90
68
64
80
Global Atsention Attention
Star Attention [Ouns]
60
32K
Context Block Size
Anchor Block Size
(a) Accuracy vs. Context Block Size
(b) Accuracy Anchor Block Size
Figure 5. Impact of context and anchor block sizes on the accuracy of Star Attention at 128K sequence length with Llama-3.1 Instruct. (a) Accuracy function of context block size, with anchor block size matched (h) Accuracy as function of anchor block size, with context block size fixed at 32K. Larger block sizes yield consistent accuracy improvements, highlighting the benefit of broader receptive fields long-contex, understanding.
90
Spendiup
85
Bloce Size 33K 32k (Foedi
120
+2.00%
RULER Arcurary (86
30
11.11%
75
-1
Accuracy
20
85
60
35
(1/26)
11152)
0
MUE:NIH
Hop
Seq Length (Relative Block Sizee
RULER Tasks
Figure 6. Accuracy speed trade-off for Star Attention RULER with Llama3-88 -8-88B--lse -B--Instrctt-l 1048K as sequence length in- creases from 128K to IM with block size fixed at 32K. Star Atten- tion achieves up to 16.9x speedup with modest accuracy degrada- tion.
Figure 7. Accuracy of Star Attention compared Global Attention across five RULER task categories using Llama-3.J 1--8---8sBtnee Instruct at 32K sequence length with 8K block size. Star Attention matches or improves upon the baseline in most tasks, with significant gains in aggregation.
4.1. Position and Content of Anchor Block
two-phase process proves advantageous capturing com- mon patterns without needing full global context once. This analysis suggests that Star Attention is especially well- suited for retrieval and aggregation tasks, while highlighting opportunities for future work on cross bloss-block communica- tion.
In this section, we explore the role of anchor blocks during Phase that enables Star Attention to approximate global attention behavior. As outlined in Section 2.1, anchor blocks are crucial in managing the attention spikes generated at the start of each context block, helping Star Attention ap- proximate global attention (see Table 4 Drawing from the hypotheses on sink tokens in Xiao et al. (2024b), we consider two potential explanations for the effectiveness of anchor blocks: (I) the model may develop bias toward the absolute position of the anchor block, or (2) the semantic content of the anchor block is essential for maintaining per- formance. To better understand how anchor blocks enable Star Attention to approximate global attention distribution, we test both the hypotheses. We conduct experiments on the Llama-3. 1-81-BB--. 1-8B--Bnnee Instructt model, varying both the position and
4. Ablation Study
The ablation experiments focus on the Needle-in-a-Haysttee (NIAH) task, which tests model's 's ability answer queries based on small, relevant piece of information ("needle"") embedded within a large context ("haystack"). To increase the task's 's complexity, we explore three variations from the RULER benchmark (Hsich et al., 2024): Single-NIAH, Multi-key NIAH, and Multi-query NIAH.

---

Star Attention
Table Impact of anchor block position and content Star Attention accuracy using 1lama-3.1 1-8B- 1-8-tnsrructt RULER-NIAH 64K
and 128K sequence lengths. Each configuration has the anchor block size equal to the context block. The A values indicate absolute accuracy degradation relative global attention. Results show that anchor content critical, while position IDs haveminor effect. Missing poorly constructed anchors lead to significant degradation.
RULER-NIAH (%)
Experiments
64K
A64k
128k
A128k
Global attention
99.50
98.49
No anchor block
60.11
-39.59%
73.75
-25.12%
Content set first-block, position IDs are:
randomly sampled from [0, current block)
96.79
-2.72%
97.16
-1.35%
same as previous block
97.35
-2.16%
96.80
-1.71% %
same as first block
97.61
-1.90%
97.54
-0.96%
Position IDs set first-block, content is:
constant token (ex: " "or" the" or ! random tokens shuffled first block tokens
0.00
-100.00%
0
100.00%%
90.55
-8.99%
82.63
-10.15%
92.96
-6.57%
90.76
-3.26%
first block tokens
97.61
-1.90%
94.94
-0.96%
Previous-block used anchor
94.20
-5.33%
96.13
-2.40%
content of the anchor block. We evaluate two configurations: a block size of 16K for sequences of length 64K, and a block size of 32K for sequences of length 128K, in both the cases, with anchor block size matching the context block size.
To examine the roles of both position and content, we experiment with using the previous block as the anchor block. For example, for a block of size 16K starting at position 32K, the anchor block would be the block with position IDs from 16K to 32K. This configuration has lower accuracy comparing to using the first block as the anchor(Table 4).
Previous block as anchor block:
Position of anchor block: Here, we fix the content of the anchor block to the first context block and vary its position IDs. We test three scenarios (I) the position IDs are randomly sampled from the range [O, starting position of the current block] (e.g., for a block starting at position 32K, position IDs are sampled from [0. 32K] ); (2) the position IDs are derived from the previous block (e.g., for a block of size 16K starting at position 32K, position IDs are sampled from [16K, 32K] (3) the position IDs fixed to the first block (our proposed approach). As shown in Table 4, varying the position of the anchor block has minimal impact on accuracy.
In summary, we found that while the positional placement of the anchor block is not important, its content critical for optimal performance.
4.2. Size of Anchor block
As discussed in Section 3.4, larger block sizes improve the accuracy of Star Attention. this section, we analyze the impact of varying anchor block size while maintaining a fixed block size 32K for sequence length of 128K. As illustrated in Figure 5b, increasing the anchor block size enhances model accuracy, with the best performance ob- served when the anchor block size equals the context block size. Although Figure 3b demonstrates that attention spikes predominantly occur the first few tokens, reducing the number of tokens the anchor block leads to a substantial drop in performance. This suggests that a larger anchor block is critical for maintaining model accuracy, despite attention spikes being concentrated the beginning of the sequence. This observation implies that the anchor block's ss effectiveness is not solely due to its role in managing atten- tion sinks but may involve other underlying factors. These findings remain consistent across both base and instruct models, as well as for all sequence lengths. Further investi-
Content of anchor block: anchor block to that of the first block but vary its content. We explore several configurations (as shown in Table 4): (i) single repeated token (e.g.,
We fix the position IDs of the
the', or '.));
(ii) random tokens; (iii) shuffling the tokens of the first block; and (iv) using the original first block content (the proposed approach). Our results show that the content of the anchor block significantly impacts performance, with the original first block content yielding the best results. This outcome suggests that since global attention performed during Phase 2, is important for the local context blocks to attend to anchor blocks whose content reflects what the model would see during global attention.
8

---

Star Attention
gation into why the anchor block size must be equivalent to the context block size is left for future work.
support. Star Attention is orthogonal these methods and can be integrated with them to further enhance inference efficiency.
5. Related Work
6. Conclusion
To address the computational challenges of long-context inference in LLMs, various techniques have emerged to mitigate memory usage and enhance inference speed.
In this paper, we introduced Star Attention, a novel block- sparse attention mechanism designed enable efficient inference on long sequences in wansformer-basee LLMs. The method operates in two phases: (1) context tokens processed using blockwise-local attention, with the context segmented into blocks where each block is prefixed with an anchor block; and (2) then the query and response tokens attend to all prior cached tokens through sequence-globa attention. Star Attention delivers up to 1 Ix speedup over Ring Attention while maintaining 97-100% accuracy, sig- nificantly enhancing both memory efficiency and inference speed. Despite these advances, several open questions re- main. The role and optimal size of anchor blocks relative to context blocks require further exploration. Additionally, while Star Attention performs effectively with block sizes set to one-quarter of the sequence length, accuracy degrades when using smaller blocks on longer sequences. Future work will focus refining the anchor block mechanism and improving performance on more complex long-context tasks to enhance the scalability and robustness of Star At- tention.
Blockwise and Distributed Attention Computation: Flash Attention (Dao et al., 2022; Dao, 2024) introduces a blockwise GPU-efficient implementation of exact atten- tion, reducing both memory footprint and runtime. Building on this, distributed approaches such as Liu et al. (2024a) and Shyam al. (2024) partition the computation of self- attention and feed-forward networks across multiple devices, employing sophisticated communication-coopptaii lap to improve scalability. General distributed strategies (Shoeybi et al., 2019; Huang et al., 2019; Li et al., 2023; Meta-Al, 2021) provide frameworks for dividing the compu- tational load effectively across multiple accelerators. These methods, however, still compute dense global attention, which becomes prohibitively expensive at longer sequence lengths. Star Attention leverages the distributed nature of these approaches but reduces attention complexity through a two-phase block- sparse approximation that avoids com- puting the full attention matrix.
over-
Sparse attention methods reduce the quadratic complexity of self-attention through structured or learned sparsity patterns (Zhang et al., 2023; Tang et al., 2024; Child et al., 2019), achieving linear log-linear scaling in sequence length (Dai et al., 2019; Qin et al., 2024). Beltagy et al. (2020) introduced sliding window attention combined with global tokens, which was adapted by Xiao et al. (2024b) for streaming generation via atten- tion sinks. Jiang et al. (2024) focuses on identifying and leveraging dynamic sparse patterns, particularly accel- erate the pre-filling stage. More recently, Titans (Behrouz et al., 2024) augment LLMs with neural memory modules for long-horizon reasoning. Star Attention's 's first phase is conceptually similar to streaming methods, but differs by utilizing global attention during decoding- ppreeeriing com- patibility with pretrained models without retraining.
Sparse Attention:
Acknowledgements
We thank Kefeng Duan, Santiago Akle, Vahid Noroozi, Somshubra Majumdar, Jocelyn Huang, Zhiyuan Jerry Lin and NVIDIA Long Context team for helpful discussion and feedback.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Memory Optimization: Maintaining the KV cache during autoregressive decoding is major memory bottleneck. KV cache compression (Ge et al., 2024; Munkhdalai et al., 2024; Sun et al., 2024; Liu et al., 2024b; Wu et al., 2024) and low- rank approximation methods (Hu et al., 2022) have been proposed to trade precision for reduced memory. Recent sys- tems explore eviction-basee memory management strategies that allow LLMs to operate over virtually infinite contexts (Zhao al., 2024; Han al., 2024; Xiao et al., 2024a), often requiring architecture changes or specialized runtime
Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku, 2024. URL htLLhttps://www edn. anthropi c. com/ 0018857627/Model1 /MtdeMModlll/// Caweoddee__ar___auuudeee 3.f.f
7b7bb
Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning to memorize at test time, arXiv preprint arXiv:2501.00663, 2024.
Beltagy, I., Peters, M. E., and Cohan, A. Long-
9

---

Star Attention
former: The long-document Transformer. arXiv preprint 00Xiv:0004005/50, 2020.
Jiang, H., Li, Y., Zhang, C., Wu, Q., Luo, X., Ahn, S., Han, Z., Abdi, A. H., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L, MInference 1.0: Accelerating pre-filling for long-context LLMs via dynamic sparse attention. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Child, R., Gray, S., Radford, A., and Sutskever, I. Gen- erating long sequences with sparse transformers, arXiv preprint arXiv: 10509, 2019.
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. and Salakhutdinov, R. Transformer-XL. Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics (ACL). 2019.
Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., and Burtsev, M. BABILong: Testing the Limits of LLMs with Long Context Reasoning- -n---- Haystack. arXiv preprint arXiv:2406. 10149, 2024.
Li, S., Xue, F., Baranwal, C., Li, Y., and You, Y. Sequence parallelism: Long sequence training from system per- spective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pupers), 2023.
Dao, T. FlastAttention-22 Faster attention with better paral- lelism and work partitioning. In International Conference on Learning Representations (ICLR). 2024.
Dao. T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. FlashAt- tention: Fast and memory-effictee exact attention with 10 no-awrreness In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2022.
Liao, Z., Wang, J., Yu, H., Wei, L., Li, J., Wang, J., and Zhang, W. E21lm: Encoder elongated large language models for long-context understanding and reasoning. arXiv preprint (.Xxv:2409906679 2024.
Ge, S., Zhang. Y., Liu, L., Zhang, M., Han, J.. and Gao, Model tells you what to discard: Adaptive kv cache compression for llms. In International Conference on Learning Representations (ICLR), 2024.
Liu, and Abbeel, Blockwise parallel transformers for large context models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Team. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2403.055300 2024.
Liu, H., Zaharia, M., and Abbeel, P. Ringattention with blockwise transformers for iefi-infi context. In Inter- national Conference on Learning Representations (ICIR), 2024a.
Gradient.ai. RULER Gradient's IM context length Llama-3-70B, 2024. URL / ai/ 1m- cn-coetext-
http://tleee----rrdieeeeeeee 1ength-11ama-33
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., and Hu, X. KIVI: tuning-free ffren asymmetric 2bit quantization for KV cache. In The Twelfth Interna- tional Conference on Learning Representations (ICML), 2024b.
Han, C.. Wang, Q., Peng. H., Xiong, W., Chen, Y., Ji. H., and Wang, S. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), 2024.
Meta-AL AL. Fully sharded data parallel: faster Al training with fewer GPUs, 2021. URL https: /t////enn //ngi neering. fb. ct..co/22000 2ttomm/22100777 /pp//pn////pp///////////eeeeee// source/ podp///fsdp/.
Hsieh, C.-P. Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. RULER: What's the real context size of your long-com congeconeett language models? In First Conference on Language Modeling (COLM), 2024.
Meta-AI. Introducing Llama 3.1: Our most capable models to date, 2024. URL wtttthtt///// http://aaaaaaaaaaa////e .eeta. com/blog /meta- "meta-llame 1aa---1mma-------
Milakov, M. and Gimelshein, N. Online normalizer cal- culation for softmax. arXiv preprint 2018.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICIR), 2022.
Munkhdalai, T., Faruqui, M., and Gopal, S. Leave con- text behind: Efficient infinite context Transformers with infini-attention. arXiv preprint urXiv:2404.099 2024.
Huang, Y., Cheng,
Y., Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q V., Wu, Y., et al. GPipe: Efficient training of giant neural networks using pipeline parallelism. In Advances in Neural Information Process- ing Systems, 2019.
NVIDIA. Tensorrt-llm: An optimized library for large language models, 2023. URL https: httpp//b///tuu./////// om/NVIDDAA/TensoRTT-
10

---

Star Attention
Qin, Z., Sun, W., Li, D., Shen, X., Sun, W., and Zhong, Y. Lightning attention-2: A free lunch for handling unlim- ited sequence lengths in large language models. arXiv preprint arXiv:2401.04658 2024.
Zhang, Z.., Sheng, Y., Zhou, T., Chen, T., Zheng, L.., Cai, R., Song, Z., Tian, Y., Re, C., Barrett, C., Wang. Z.,, and Chen, B. H2o: Heavy-hitter oracle for efficient genera- tive inference of large language models. In Advances in Neural Information Processing Systems (NewrIPS), 2023.
Qwen. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2025.
Zhao, Y., Wu, D., and Wang, J. Alisa: Accelerating large language model inference via sparsity-aware kv caching. In 51 Annual International Symposium on Computer Architecture (ISCA), 2024.
Russak, M., Jamil, U., Bryant, C., Kamble, K., Magnuson, A., Russak, M., and AlShikh, W. Writing in the margins: Better inference pattern for long context retrieval. arXiv preprint arXiv:2408. 149900008.14906, 2024.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-LMM: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv:/909.0800 2019.
Shyam, V., Pilault, J., Shepperd, E., Anthony, Q., and Mil- lidge, B. Tree attention: Topology-aware decoding for long-context attention gpu clusters. arXiv preprint 27Xi::2408.04093, 2024.
Sun, Y., Dong, I.., Zhu, Y., Huang, S., Wang. W., Ma, S., Zhang, Q., Wang, J., and Wei, F. You only cache once: Decoder-decoder architectures for language models. arXiv preprint arXiv:24055.05244 2024.
Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikei, B., and Han, S. Quest: query-aware sparsity for efficient long-conter Ilm inference. The Twelfth International Conference on Learning Representations (ICML), 2024.
Wolf, T., Debut, L., Sanh, V., et al. Transformers: State-of- the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing:
System Demonstrations, 2020.
Wu, J., Wang, Z., Zhang, L., Lai, Y., He, Y., and Zhou, D. Scope: Optimizing key-value cache compression in long- context generation. arXiv preprint 136649,4 2024.
Xiao, C., Zhang, P., Han, X., Xiao, G., Lin, Y., Zhang, Z., Liu, Z., and Sun, M. Infllm: Training-free free long-context extrapolation for llms with an efficient context memory. arXiv preprint orXiv:2402.046))
2024a.
Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Ef- ficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations (ICML), 2024b.
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, 7.., and Sun, M. BBBehch: Extending long context evaluation beyond 100K tokens. In Proceedings of the 62nd Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), 2024.

---

Star Attention
A. Star Attention Pseudo-code
Algorithm Star Attention Phase Context Encoding
Require: Context e, Block size h
L length(c)
2: Split into = [L/b] blocks, such that c [e1, 02,... Gn]
3: for = 2 to do
4: c/ < (c)
5: end for
6: for each host concurrently do
7: Initialize an empty list
8: end for
9: Distribute augmented blocks 12.. dn across all hosts
10: for each host concurrently do
11: for each assigned block cg do
12:
Compute attention over 26 tokens in c Generate KV cache for c'
13:
14: Discard KV cache for anchor block Cj
Append remaining KV cache (for 6;) to kv
end for
17: end for
Algorithm 2 Star Attention Phase 2: Query Encoding and Token Generation
Require: Query tokens number of output tokens No, KV cache kリ,, of each host from Phase
1: Designate one host as the query-host hy
2: Broadcast query tokens to all hosts
3: Initialize input.tokens <- q
4: Initialize output.tokeee tokens
5: for = 1 to 11, do
6: for each transformer layer do
7: for each host h concurrently do
8: Compute query, key, and value vectors (Q, K, V) using input.tokens
9:
hy then Append the new K and V vectors to kUR,
10:
end
12:
Compute local attention scores A for query Q using the local KV cache kuy Compute local log-sum-exp sn. (logarithm of the softmax denominator) end for
13:
14:
15:
Gather all A and sn, from hosts: = [$1, s2....,......................... sh].
A = [A1, As,
Aal
16:
Initialize Sglobal +- 81, Aglohal < A
17:
for h = 2 to Update global log-sum-exp Sylohal using online softmax: Sglobal + Sglobal + log (1 + exp(s). - $global)
19:
Update global attention scores:
Aglobal + exp(sh Sglobal) Aglobal exp(An -globbl) A
end for
21: end for
22: Generate the next output token and append it to output takens
23: Set inkutstokens +- [new output token]
24: end for
25: return oulpul Jokens
12

---

Star Attention
Table Accuracy versus speed trade-off for Star Attention compared to Ring Attention RULER. The A for attention shows the absolute accuracy degradation and the relative speedup compared to the baseline. When the block size remains fixed and the sequence length increases, Star Attention achieves exponential speedup over Ring Attention the cost of slightly more accuracy degradation.
Seq. Len. (K)
Block Size
Ring-Attn Acc. (%)
Star-Attn
Model
(K)
A Acc.
A Speedup
128
32
77.39
+0.96%
2.7x
1048K
256
32
74.44
-0.77%
10.8x
(Gradient.ai, 2024)
512
69.30
-6.73%
16.2x
1024
63.70
-5.32%
16.9x
Llama-3.1-70B Inntruutt 128K (Meta-Al, 2024)
64
16
88.54
-1.44%
4.7x
128
16
65.29
-7.47%
8.7x
Table Time per sample (seconds) for Llama3. 1-8B-Instruct model with dense, ring, and star attention, using A100 GPUs. Vanilla autoregressive generation encounters out-of- mo-omr (OOM) at 128K sequence length. performs best in short context scenarios (i.c. sequences upto 32K tokens) hut in long context scenarios, star attention demonstrates significant speedup.
Seq. Length (K)
Time Per Sample (s)
Vanilla
Ring
Star
16
7
10
9
32
10
12
10
64
18
22
12
128
OOM
53
20
B. Experiment Details
B.1. Baseline Comparison
Our implementation utilizes the HuggingFace Transformers library (Wolf et al., 2020), which currently lacks support for mudti-node inference. As result, when performing inference with the Llama-3.1 8B model using standard causal autoregressive generation on sequences exceeding 64K tokens with 6 precision across A.00 GPUs, we encounter out-of-memory (OOM) errors. Given these limitations, adopt Ring Attention practical and relevant baseline for evaluating Star Attention's 's performance on sequences up to million tokens in length.
Table 5 shows speedup obtained by Star Attention over the baseline on sequences over 128K tokens. For such long sequences, we freeze the block size to 32K sequences to optimize for speed. This setting shows upto 16.9x inference speedup with just 5.32% accuracy degradation compared to the baseline. Table 6 presents the time per sample for vanilla autoregressive generation, Ring Attention, and Star Attention across sequence lengths ranging from 16K 128K. The results indicate that both Ring and Star Attention can process sequences up 128K tokens on 8 A000 GPUs, whereas vanilla autoregressive inference encounters OOM issues beyond 64K tokens. For sequence lengths below 32K, vanilla inference is faster than the distributed attention mechanisms, primarily due to the GPU communication overhead incurred in the distributed setups. However, in long context scenarios i.c. on sequence lengths exceeding 32K tokens, Star Attention begins to demonstrate clear performance advantages. As demonstrated in Table 5, the speedup achieved by Star Attention increases significantly with longer sequence lengths.
B.2. Hardware for Inference Speed
We use A100 GPUs to run all our inference speedup experiments. Table 7 describes the number of GPUs and the number of parallel workers used to obtain the inference speed numbers for Ring Attention and Star Attention for each sequence length. In all these experiments, the anchor block size in Star Attention was kept same as the context block size.
13

---

Star Attention
Table Resources used for the speedup experiments
Model Size
Seq. Length
# GPUs
# Workers
16K
128K
8
8B
256K 512K
16
1M
32
16
16K 32K
8
4
70B
64K
16
4
128K
32
8
B.3. Prompt Templates
Prompt template for base models:
[context] (ranttexx((quueeeeeeeeeee (query) (answer.prefix.
Prompt template used for Llama-3 and Llama-3.1 Instruct models:
< </begin.of.toxt >< start.
You are helpful assistant. < (ttooiiddd(sttttttthtteeee id >user<| ben
{context (onntext[qqury]]) {query] leot id|><| start header >assissttnnt end header id/>
(answer.prefix)
The portion blue is processed during Phase for blockwise context encoding, while the remaining text gray processed in Phase for query encoding and token generation. The {context) and {query)(answee prefix} denote the context and the query portion of the input prompt, respectively. The (answer.prefix] is only relevant for the RULER benchmark.
C. Evaluation Benchmarks
This benchmark comprises 13 tasks covering domains such as Needle-in-a- -aale-in-a-Haystack (Retrieval), Multi-Hop Tracing, Aggregation, and Question Answering. Each task comprises 500 samples. For the ablations, we choose four Needle-In-A-Haasttac (NIAH) tasks where Paul Graham essays serve as the distractor text (haystack); Single Single MultiKey and MultiQuery. In these tasks, key-value pair is concealed within long context, and the model must identify the value corresponding to the key based on the provided input query. Table 8 presents the configurations of all the tasks in RULER.
RULER:
BABILong: In BABILong, we choose tasks (shown in Table 9), each containing a 1000 samples. These tasks are generated by simulating a set characters and objects engaged in various movements and interactions across multiple locations. Each interaction is represented by factual statement, and the objective answer questions based on the facts derived from the current simulation.
InfiniteBench: This benchmark comprises 10 real-world and synthetic tasks, each crafted to assess different aspects of language processing and comprehension in extended contexts. Details of each task is shown in 10
14

---

Star Attention
Table 8. Configuration of RULER tasks
Task
Haystack
Keys
Values
Category
Outputs
Name
Type
Type
Type
Single 1
noise
words
numbers
Single 2
book
words
numbers
Single
book
words
uuids
NIAH (Retrieval)
MultiKey 1
book
words numbers
MultiKey 2
line words numbers
MultiKey 3
kv uuids
uuids
VultiValue
book
words numbers
4
MultiQuery
book words numbers 4
Multi-Hop Tracing
Variable Tracking
Common Words Extraction Frequent Words Extraction
Aggregation
Question
QA (squad)
Answering
QA 2 (hotpotqa)
Table 9. Configuration of tasks BABILong
Task
Name
# Facts per task
qal
single supporting fact
10
qa2
two supporting facts
2 68
qa3
three supporting facts
4-
qa4
two arg relations
2
qa5
three arg relations
2 126
Table Configuration tasks InfiniteBench
Avg. input tokens
Avg. output tokens
Task name
Context
samples
En.Sum
Fake Book
103
171.5k
1.1k
En.QA
Fake Book
351
192.6k
4.8
E..MC
Fake Book
229
184.4k
5.3
En.
Script
200
103.6k
3.4
Zh. Zh.QA
New Book
175
2068.66
6.3
Code. Debug
Code Document
394
114.7k
4.8
Code.Run
Synthetic
400
75.2k
1.3
Math.Cale
Synthetic
50
43.9k
43.9k
Math.Fina
Synthetic
350
87.9k
1.3
Retrieve. Passkey
Synthetic
590
122.4k
2.0
Netrieve.Number
Synthetic
590
122.4k
4.0
Retrieve.K
Synthetic
500
89.9k
22.7
15

---

Star Attention
D. RULER Analysis
Global Attention
Star Attention (Ours)
Seq Length: 16K
Seq Length: 32K
Seq Length: 64K
Seq Length: 128K
100
Accuracy (%)
75
50
25
S-MAH
M M-NIAH
Multi-Hop
Agg.
QA
S-NIAH
M- M-NIAH
Multi-Hpp
Agg.
QA
S-NIAH
M-NIAH
Multi-Hop
Agg.
QA
S-NIAH
M-NIAH
Multi-Hop
Agg.
QA
Figure 8. Accuracy of Star Attention using 1amma-3.1-8B-Immaa
on the 5 categorics of tasks in RULER on sequence lengths of 16K, 32K, 64K, and 128K. In all experiments, the block size and anchor block size NIAH and QA tasks, Star Attention retains upto 97-100% accuracy of the baseline. The Multi-Hop Tracing task is notably challenging because it requires inter-block communication, which leads to expected performance degradation. Interestingly, Star Attention performs better with sequence lengths of 128k on this task, but this may be due to noise given the suboptimal baseline. In aggregation tasks, Star Attention show significant improvement as distributed local attention helps model such summarization tasks.
one-quarter of the total sequence length. For the
16


Reviews:
[{"review_id": "MozeEUBYdc", "reviewer": "Reviewer_x5MA", "summary": "This paper presents Star Attention, a novel two - phase block - sparse approximation algorithm for efficient LLM inference over long sequences. The self - attention mechanism in Transformer - based LLMs has quadratic complexity, making long - sequence inference costly and slow. Star Attention addresses this issue by dividing the inference process into two phases. In Phase 1, the context is partitioned into blocks and distributed across multiple hosts. Each host computes local attention within its assigned block, which reduces the attention complexity from quadratic to linear. In Phase 2, the query is broadcast to all hosts, and global attention is computed at a designated query - host by aggregating local attention results. This approach enables the context length to scale linearly with the number of hosts. Experiments on Llama - based models show that Star Attention can achieve up to 11x faster inference speed compared to Ring Attention while maintaining 95 - 100% of the accuracy.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer x5MA's constructive feedback. Below, we address the identified weaknesses regarding anchor blocks, performance on complex tasks, and block size dependency:\n\n### **1. Role and Configuration of Anchor Blocks:**\n- **Anchor Block Size:** The performance peak when the anchor block size equals the context block size can be attributed to Star Attention’s design constraints. Since Phase 1 uses causal attention (as in decoder-only LLMs), anchor blocks cannot extend beyond the preceding context block without violating causality. Setting the anchor size equal to the context block size represents the maximum usable context while respecting this constraint — effectively bringing Star Attention closer to full attention in terms of receptive field.\n- **Anchor Block Content and Position:** As shown in Section 4.1, using the first context block (`c1`) as the anchor yields optimal performance. The rationale is that Phase 2 performs global attention; therefore, Phase 1 context blocks should attend to anchor tokens representative of the initial sequence context that the full model would observe in a global attention scenario. Our analysis in Section 4.1 further indicates that if the anchor block content is fixed to `c1`, variations in the position IDs assigned to these anchor tokens during local attention computation have a minimal impact on overall performance.\n\n### **2. Performance on Complex Tasks:**\n- Star Attention, like other sparse attention mechanisms, trades off some accuracy for substantial inference speedups. Across a range of tasks — including question answering (Figure 7) — it retains 95–100% of dense attention performance while delivering substantial inference speedups.\n- For tasks requiring complex reasoning and inter-block context (e.g., Multi-Hop Tracing), we acknowledge that the absence of direct cross-block attention in Phase 1 introduces challenges. Despite this, Star Attention maintains up to 93% of the dense attention accuracy on these tasks (Figure 7).\n- To assess generalization on more complex, real-world tasks, we evaluated Star Attention on InfiniteBench, a diverse benchmark covering multilingual QA, retrieval, math, code debugging, summarization and more. As shown in Table B, Star Attention matches or closely tracks full attention across all categories and outperforms prior sparse inference baselines by a significant margin.\n\n**Table B: Accuracy on Infinite Bench (Llama-3.1-8B-Instruct)**\n| Methods            | En. Sum | En. QA | En. MC | En. Dia | Zh. QA | Code. Debug | Math. Find | Retr. PassKey | Retr. Num | Retr. KV |  Avg. |\n| :----------------- | :-----: | :----: | :----: | :-----: | :----: | :---------: | :--------: | :-----------: | :-------: | :------: | :---: |\n| *Full Attn.  (Baseline)* | *31.91* | *25.92* | *69.43* | *21.5* | *31.95* | *16.75* | *24.29* | *99.15* | *99.66* | *60* | *48.06* |\n| StreamingLLM       |  30.15  | 10.15  | 41.05  |   8.5   | 22.38  |    8.63     |   17.71    |     2.71      |   5.93    |    0     | 14.72 |\n| MInference         |  31.04  |   22   | 63.76  |  14.5   |  28.7  |    5.33     | **27.43** |     56.78     |   77.12   |    14    | 34.07 |\n| **Star Attention** | **31.85** | **25.92** | **69** | **22** | **30.37** | **24.37** |   26.29    | **93.22** | **96.27** | **45.8** | **46.51** |\n\n- Improving the performance on tasks requiring deeper inter-block reasoning in the early encoding stages remains an important direction for future work.\n\n### **3. Dependency on Block Size and Flexibility:**\n- The dependency on block size reflects a common trade-off in blockwise attention: smaller blocks enable faster inference but may limit context aggregation, especially in longer sequences. Our experiments (Figure 5) show that setting the block size to ~1/4 of the total sequence length achieves a strong balance of accuracy and speed. However, the block size depends on the user and they can choose smaller block sizes to prioritize higher inference speeds, accepting slightly more accuracy degradation.\n- For variable-length or real-time inputs, users can set the block size based on the maximum expected sequence length or system constraints. As shown in Figure 6 and Table 5, even with smaller blocks (1/8, 1/16, 1/32 of the sequence), Star Attention retains up to 90% of dense attention performance while delivering up to 17× speedups — allowing users to flexibly tune accuracy vs. latency based on application needs.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "no", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths\n1. Star Attention significantly speeds up LLM inference on long sequences. It manages to achieve a remarkable speedup, up to 11 times faster than the baseline in some cases. This is a huge improvement, especially considering the increasing demand for processing long - context data in applications like large - scale document analysis and multi - document summarization. For example, in the experiments with Llama - based models, it clearly outperforms the Ring Attention baseline in terms of inference time.\n2. Despite the significant speed improvement, Star Attention can preserve 95 - 100% of the accuracy of global attention. This means that it doesn't sacrifice much in terms of the model's ability to understand and process the input accurately. Whether it's in simple retrieval tasks or more complex question - answering tasks, it can still provide reliable results.\n3. The two - phase design is really smart. By separating context encoding and query encoding, it takes advantage of the characteristics of different parts of the input sequence. The use of anchor blocks in Phase 1 helps to manage the attention spikes and approximate global attention, which is a great way to optimize the attention mechanism. Also, the distributed softmax algorithm in Phase 2 enables efficient global attention computation without excessive communication overhead.\n4. It is compatible with most Transformer - based LLMs trained with global attention. This means it can be easily integrated into existing models without the need for complex fine - tuning. This makes it very practical and convenient for researchers and engineers who want to improve the performance of their LLM - based systems.\n\nWeaknesses\n1. Although anchor blocks play a crucial role in Star Attention, there are still some aspects that need further exploration. For instance, the exact reason why the anchor block size needs to be equal to the context block size for optimal performance is not fully understood. Also, the relationship between the position and content of the anchor block and the model's performance could be studied more deeply. This lack of understanding may limit the further optimization of the algorithm.\n2. In more complex tasks like Multi - Hop Tracing and some types of Question Answering tasks, Star Attention shows a slight decline in performance. These tasks require the model to have a deeper understanding of the context and often need inter - block communication. Since Star Attention lacks effective inter - block communication during context encoding, it struggles to perform as well as in simpler tasks. This means that there are still limitations when applying Star Attention to tasks that demand high - level context comprehension.\n3. The performance of Star Attention is highly dependent on the block size. While setting the block size to one - quarter of the total sequence length seems to work well in most cases, using smaller blocks on longer sequences leads to accuracy degradation. This restricts the flexibility of the algorithm in different scenarios. For example, in some real - world applications where the sequence length may vary unpredictably, it may be difficult to choose the optimal block size.", "other_comments_or_suggestions": "no", "questions_for_authors": "see weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4j0XprIkmM", "reviewer": "Reviewer_dnhh", "summary": "This paper propose star-attention which combines a streamingllm attention for the prefill stage and a dense attention for the decoding stage. Specifically, the author implement the streamingllm pre-fill with blocks, where the computing are partioned across the query dimension. The sink and local blocks are packed and distributed, this implementaion, compared to ring-attention requires less communication and thus avoid the ring style data transfer. For the decoding stage, the query are distributed and lse and other inner states are transfered back. The experimental results show that it achieve about 95% accuracy while being upto 11x faster than ring-attention.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer dnhh for their feedback and acknowledge their points regarding novelty, motivation, baselines, and benchmarks. We address these points below:\n\n### **1. Novelty and Relation to Prior Work:**\n- While Star Attention draws inspiration from prior work like StreamingLLM and attention sinks, it introduces a distinct two-phase distributed inference architecture. Unlike StreamingLLM, which struggles with long-context retention and is not inherently distributed, Star Attention processes the context in parallel without inter-host communication in Phase 1 and leverages global attention only during decoding. This results in both accuracy preservation and significant latency reduction.\n- Regarding “tri-shape attention,” we were unable to identify a specific paper with that terminology. If you are referring to a particular method, we would appreciate a citation to better contextualize and compare it in the final version.\n\n### **2. Motivation for Sparse Prefill vs. Dense Decoding:**\n- In Phase 1 (sparse prefill), attention is localized, as context tokens generally require only local neighborhood interactions. This allows efficient processing via distributed blockwise attention. In contrast, during query encoding and decoding (Phase 2), the tokens must integrate information from the entire context, necessitating dense attention. This design reflects practical needs in long-context tasks and allows us to optimize for both throughput and accuracy.\n- Conversely, during the query processing and response generation phase (decoding phase), the query tokens and subsequent generated tokens often require access to information scattered throughout the entire preceding context to formulate an accurate response. Therefore, employing dense global attention in Phase 2, accessing the full cached KV state from Phase 1, is crucial for preserving the model's understanding and generation capabilities, particularly for tasks requiring synthesis of information from distant parts of the context.\n\n### **3. Comparison with Sparse Attention Baselines:**\n- In response to the reviewer’s feedback, we extended our baseline comparisons to include StreamingLLM and MInference. Results are shown in Table A.\n\n**Table A: Accuracy on RULER (Llama-3.1-8B-Instruct)**\n| Methods            |  16K  |  32K  |  64K  |  128K | Average |\n| :----------------- | :---: | :---: | :---: | :---: | :-----: |\n| *Full Attn. (Baseline)* | *92.22* | *87.53* | *84.79* | *76.31* | *85.21* |\n| StreamingLLM       | 74.76 | 48.56 | 26.2  | 30.77 |  45.07  |\n| MInference         | **93.27** | 86.54 | **84.86** | 58.17 |  80.71  |\n| **Star Attention** | 91.27 | **88.70** | 83.37 | **74.41** | **84.44** |\n\n- The results show that Star Attention outperforms all the other sparse KV methods. Furthermore, due to the distributed nature of Star Attention, since there is no inter-block communication during phase 1, the latency on 128K sequence length is equivalent to the latency that the model will have when processing a sequence length of 64K (32K block + 32K anchor), since each host will process each block parallely.\n- Unlike methods such as MInference, which may require offline analysis to determine optimal sparsity patterns, Star Attention can be applied directly to most pretrained Transformer models without model-specific tuning or preprocessing.\n\n### **4. Evaluation on Comprehensive Benchmarks:**\n- In response to the reviewer’s suggestion, we expanded our evaluation to include InfiniteBench—a more comprehensive benchmark suite for long-context performance. As shown in Table B, Star Attention outperforms other sparse baselines across diverse tasks, demonstrating its robustness beyond synthetic datasets like RULER and BABILong.\n\n**Table B: Accuracy on Infinite Bench (Llama-3.1-8B-Instruct)**\n| Methods            | En. Sum | En. QA | En. MC | En. Dia | Zh. QA | Code. Debug | Math. Find | Retr. PassKey | Retr. Num | Retr. KV |  Avg. |\n| :----------------- | :-----: | :----: | :----: | :-----: | :----: | :---------: | :--------: | :-----------: | :-------: | :------: | :---: |\n| *Full Attn.  (Baseline)* | *31.91* | *25.92* | *69.43* | *21.5* | *31.95* | *16.75* | *24.29* | *99.15* | *99.66* | *60* | *48.06* |\n| StreamingLLM       |  30.15  | 10.15  | 41.05  |   8.5   | 22.38  |    8.63     |   17.71    |     2.71      |   5.93    |    0     | 14.72 |\n| MInference         |  31.04  |   22   | 63.76  |  14.5   |  28.7  |    5.33     | **27.43** |     56.78     |   77.12   |    14    | 34.07 |\n| **Star Attention** | **31.85** | **25.92** | **69** | **22** | **30.37** | **24.37** |   26.29    | **93.22** | **96.27** | **45.8** | **46.51** |\n\nWe thank the reviewer again for raising important points. Their comments led us to incorporate broader baselines and more diverse evaluations, which we believe have significantly strengthened the work. We look forward to incorporating these additions more prominently in the camera-ready version.", "claims_and_evidence": "-", "methods_and_evaluation_criteria": "The strength of this paper:\n1. This paper propose a solid sparse attention approach that does pre-fill sparsely while does decoding densely. And experimental results demonstrate the effectiveness of this approach.\n\n2. The proposed method is especially suitable for distributed system where the block-wise strategy can be applied directly for this method.\n\nThe limitation of this paper:\n1. The proposed star-attention is simply a streamingllm + a dense bottom window. It's very similar to the vanilla streamingllm and exactly the same compared to tri-shape attention. In general, the proposed algorithm does not provide novel knowledge to the community.\n\n2. The author fail to qualify the motivation of dense decoding verse sparse pre-fill. Unlike tri-shape attention, which provides a good reason for why decoding phase requires a complete access of all history tokens.\n\n3. The author fail to include reasonable baselines to compare with. Star-attention as a lossy attention variant should be compared against for example streamingllm, Minference, or more recent flex-prefill, instead of comparing against ring-attention.\n\n4. The author only report the performance on the the synthetic benchmark RULER, which may fail to test all aspects of star-attention. Should also report metrics on more comprehensive testset such as Infinite-bench, scbench etc.", "theoretical_claims": "-", "experimental_designs_or_analyses": "-", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "1. What is the motivation of dense query and sparse pre-fill?\n2. How well star-attention perform against other sparse attention baselines? (see above.)\n3. How well start-attention perform on comprehensive long-context benchmarks? (see above.)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xx1XTguoQG", "reviewer": "Reviewer_EMsM", "summary": "The paper introduces StarAttention, a sparse attention method for encoding long-context by distributing chunks of context over GPUs.  Unlike Ring Attention, Star Attention uses only local (in-chunk) attention for the majority of the context, allowing for a substantial speedup. Each block attends only to itself and an anchor block; the query at the end of the long input then attends over all input chunks, using a lazy softmax accumulation. Star Attention substantially reduces latency at a small performance cost; ablations show that maintaining an anchor block of meaningful context and limiting the number of total chunks is important to maintaining performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer EMsM for their insightful comments and suggestions. We address each point below:\n\n### **1. Clarity of Abstract Wording:**\nThank you for pointing out the ambiguity in the abstract's final sentence. You are correct that the “up to 11x” improvement specifically refers to inference speed and throughput, not memory. While Star Attention does reduce memory usage (due to sharded attention and localized KV caching), the 11x figure quantifies speedup. We will revise the abstract to explicitly state that the 11x gain pertains to speed/throughput, and mention memory reduction separately, to prevent misinterpretation.\n\n### **2. Evaluation Benchmarks:**\nWe agree that results on real-world or less synthetic benchmarks are valuable. In response, we have extended our evaluation to include InfiniteBench, a long-context benchmark with diverse and challenging tasks. Additionally, we also added comparisons to some of the other sparse attention methods as well such as StreamingLLM and MInference. The results are summarized in the table below.\n\n**Table B: Accuracy on Infinite Bench (Llama-3.1-8B-Instruct)**\n| Methods            | En. Sum | En. QA | En. MC | En. Dia | Zh. QA | Code. Debug | Math. Find | Retr. PassKey | Retr. Num | Retr. KV |  Avg. |\n| :----------------- | :-----: | :----: | :----: | :-----: | :----: | :---------: | :--------: | :-----------: | :-------: | :------: | :---: |\n| *Full Attn.  (Baseline)* | *31.91* | *25.92* | *69.43* | *21.5* | *31.95* | *16.75* | *24.29* | *99.15* | *99.66* | *60* | *48.06* |\n| StreamingLLM       |  30.15  | 10.15  | 41.05  |   8.5   | 22.38  |    8.63     |   17.71    |     2.71      |   5.93    |    0     | 14.72 |\n| MInference          |  31.04  |   22   | 63.76  |  14.5   |  28.7  |    5.33     | **27.43** |     56.78     |   77.12   |    14    | 34.07 |\n| **Star Attention** | **31.85** | **25.92** | **69** | **22** | **30.37** | **24.37** |   26.29    | **93.22** | **96.27** | **45.8** | **46.51** |\n\n### **3. Discussion of Related Work:**\nThank you for suggesting the additional relevant references. We will amend the related work section in the revised manuscript to include a more detailed discussion and comparison with related methods.\n\n### **4. Impact of Block Size and Number of Blocks (Q1):**\n- Figure 5 illustrates accuracy as a function of block size, holding the total sequence length fixed. We observe that larger blocks result in better approximation to global attention and higher accuracy.\n- Figure 6 and Table 5 (Appendix C.1) explore scaling behavior with fixed block size (e.g., 32K) while increasing sequence length up to 1M tokens. We find that Star Attention retains upto 90% of full attention accuracy even at 1M tokens, with up to 17x speedup.\n\nThis suggests a graceful degradation, not a sharp drop-off. The method remains robust up to at least 1M tokens, especially with larger blocks.\n\n### **5. Size of Anchor Block relative to the Block Size:**\nSince Phase 1 uses causal attention (as in decoder-only LLMs), anchor blocks cannot extend beyond the preceding context block without violating causality. Setting the anchor size equal to the context block size ensures maximal usable context per block while preserving autoregressive constraints, effectively bringing Star Attention’s receptive field closer to that of full attention.", "claims_and_evidence": "Yes; however, I think the last line of the abstract could be worded more carefully-- it currently almost seems to imply that memory requirements are *also* reduced by 11x. I think this is a miscommunication and not an overclaim.", "methods_and_evaluation_criteria": "Yes; I think evaluating on RULER and BABILong is reasonable, although it would be a bonus to also see results on a less synthetic task.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes; I think RingAttention is an appropriate comparison point, and using Flash Attention for each is appropriate.", "relation_to_broader_scientific_literature": "I think this is a nice contribution to efficient long context; while there have been a number of prior works that encode context in chunks and then do some kind of aggregation by retrieval, fusion, or overlapping, this is (to my knowledge) the first work to apply the attention sink + local context method to efficiently prefill a long input's KV cache.", "essential_references_not_discussed": "I think there could be more discussion of streaming methods for long context that evict the middle cache, keeping only the sink + local context-- these are conceptually adjacent, although they do differ from Star Attention in that they fully evict the middle cache. Some notable examples would be [StreamingLLM](https://arxiv.org/abs/2309.17453) (which is already cited), [LM-infinite](https://arxiv.org/abs/2308.16137), and [InfLLM](https://arxiv.org/abs/2402.04617). \n\n[TurboRAG](https://arxiv.org/abs/2410.07590) may also be relevant, although it also differs significantly from this setting-- in particular, I believe they train their model to adapt to sparse attention patterns from stacking reused KV caches from pre-encoded documents.", "other_strengths_and_weaknesses": "I think the ablation of what should go in the context block (and whether its positional IDs matter) is interesting and useful! I appreciated the analysis.", "other_comments_or_suggestions": "(Related to the questions below) I think more understanding of how many blocks can be used before performance breaks down would be helpful. The settings proposed (generally 4 context blocks, with an anchor block of the same size as the context blocks) seems like a reliable setting, but it would be nice to understand how much this method is robust to varying the block size.", "questions_for_authors": "Q1. Given a fixed context length, how does the performance vary with block size? Given a fixed block size (e.g. 16k tokens), how many blocks can you add before performance severely declines? There is some discussion of this (and it seems that generally less blocks is better), but how sharp is the dropoff? It would be helpful to understand how dramatic this performance dropoff is.\n\nQ2. You state that a larger anchor block size is critical to maintaining accuracy, and so the anchor block should be the same size as the context block. Is there any benefit to having the anchor block *larger* than the context block? (e.g. if I can only process 32k tokens on each single parallel worker, is it best to have 16k anchor + 16k context, or would 24k anchor + 8k context be better?).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SmiaGRa3ek", "reviewer": "Reviewer_dKGV", "summary": "This paper proposes Star Attention, which improves the LLM inference efficiency by sharding attention across multiple hosts.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer dKGV for their detailed and insightful feedback. Below, we respond to the concerns regarding system performance analysis, literature coverage, compatibility, and presentation.\n\n### **1. System Performance Analysis:**\n- While we agree that kernel-level profiling and analysis across hardware configurations (e.g., PCIe vs. NVLink interconnects) can provide valuable insights, our primary focus in this work is to introduce and validate the Star Attention algorithm from an algorithmic and end-to-end performance standpoint. We view detailed kernel-level profiling and hardware-specific optimization as an exciting avenue for future work that could further enhance deployment efficiency.\n- All experiments were conducted under consistent hardware and software environments for Star Attention and the baselines, ensuring that the reported relative speedups (e.g., up to 11x on Llama-3.1-8B-Instruct for long contexts) fairly capture the algorithmic advantage.\n- Notably, Star Attention’s phase 1 avoids inter-host communication entirely, unlike Ring Attention. Even assuming an infinitely fast interconnect, global attention still incurs computation proportional to the total number of tokens per host, while Star Attention reduces this via localized blockwise attention in phase 1.\n\n### **2. Literature Survey (KV Cache Optimization):**\n- Thank you for the helpful references [1–3]. We will revise the related work section to include and discuss these references.\n- Based on the reviewer's feedback, we conducted new comparisons between Star Attention and other sparse KV cache methods such as StreamingLLM and MInference, on long-context inference tasks. *Star Attention outperforms these alternatives in both accuracy and scalability*, as shown below:\n\n**Table B: Accuracy on RULER (Llama-3.1-8B-Instruct)**\n| Methods            |  16K  |  32K  |  64K  |  128K | Average |\n| :----------------- | :---: | :---: | :---: | :---: | :-----: |\n| *Full Attn. (Baseline)* | *92.22* | *87.53* | *84.79* | *76.31* | *85.21* |\n| StreamingLLM       | 74.76 | 48.56 | 26.2  | 30.77 |  45.07  |\n| MInference         | **93.27** | 86.54 | **84.86** | 58.17 |  80.71  |\n| **Star Attention** | 91.27 | **88.70** | 83.37 | **74.41** | **84.44** |\n\nWe also evaluated Star Attention on the InfiniteBench benchmark. Given its broad coverage across multilingual and programmatic tasks, we include this for further evidence of Star Attention’s generalization:\n\n**Table C: Accuracy on Infinite Bench (Llama-3.1-8B-Instruct)**\n| Methods            | En. Sum | En. QA | En. MC | En. Dia | Zh. QA | Code. Debug | Math. Find | Retr. PassKey | Retr. Num | Retr. KV |  Avg. |\n| :----------------- | :-----: | :----: | :----: | :-----: | :----: | :---------: | :--------: | :-----------: | :-------: | :------: | :---: |\n| *Full Attn.  (Baseline)* | *31.91* | *25.92* | *69.43* | *21.5* | *31.95* | *16.75* | *24.29* | *99.15* | *99.66* | *60* | *48.06* |\n| StreamingLLM       |  30.15  | 10.15  | 41.05  |   8.5   | 22.38  |    8.63     |   17.71    |     2.71      |   5.93    |    0     | 14.72 |\n| MInference          |  31.04  |   22   | 63.76  |  14.5   |  28.7  |    5.33     | **27.43** |     56.78     |   77.12   |    14    | 34.07 |\n| **Star Attention** | **31.85** | **25.92** | **69** | **22** | **30.37** | **24.37** |   26.29    | **93.22** | **96.27** | **45.8** | **46.51** |\n\n### **3. Compatibility with Parallelism Methods and Serving Frameworks:**\n- **Model Parallelism [4, 5]:** Star Attention is complementary to tensor and pipeline parallelism. These can be applied within each host, while Star Attention governs how input context is distributed across hosts via sequence parallelism. No assumptions are made about the intra-host setup.\n- **Serving Frameworks [6-7]:** Star Attention is compatible with modern LLM serving frameworks. For instance, memory management techniques like PagedAttention [6] can be applied within each host to efficiently handle the local KV cache. Because Star Attention distributes context blocks across hosts and isolates them during Phase 1, such memory-optimized serving layers can operate independently within each node. Similarly, the two-phase structure of Star Attention aligns well with structured execution frameworks like SGLang [7], and can potentially be integrated into such systems without requiring fundamental changes to their scheduling or runtime semantics.", "claims_and_evidence": "Please see **Other Strengths And Weaknesses**.", "methods_and_evaluation_criteria": "Please see **Other Strengths And Weaknesses**.", "theoretical_claims": "Not applied here.", "experimental_designs_or_analyses": "Please see **Other Strengths And Weaknesses**.", "relation_to_broader_scientific_literature": "Please see **Other Strengths And Weaknesses**.", "essential_references_not_discussed": "Please see **Other Strengths And Weaknesses**.", "other_strengths_and_weaknesses": "**Strengths**:\n1. The paper is easy to follow, with clear writing and presentation.\n2. Evaluation results are good.\n\n**Weaknesses**:\n\n1. The main concern I have with this paper is the lack of system performance analysis of proposed method. While the authors present comprehensive algorithm results over various benchamarks, a more detailed performance breakdown at kernel-level would make this method more convincing. Moreover, since sequence parallelism depends on hardware setup, such as GPU accelerator and interconnect (PCIe/NVLink), a more fine-grained analysis for different settings would also provide insights.\n\n2. The literature survey is also not comprehensive. \nFor instance, there are many other works that focuses on KV cache optimization from architecture/system angeles [1-3].\n\n3. The authors should also further discuss its compatibilty with existing parallelism methods [4-5] and popular LLM serving frameworks [6-7].\n\n4. The title of the paper should emphasize '**distributed LLM inference**' as the proposed method demands multi-GPU/node setup to work.\n\n\n[1] ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching, ISCA 2024.\n\n[2] InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management, OSDI 2024.\n\n[3] FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving, Arxiv 2025.\n\n[4] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, Arxiv, 2019.\n\n[5] GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism, NeurIPS 2019.\n\n[6] Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023.\n\n[7] SGLang: Efficient Execution of Structured Language Model Programs, NeurIPS 2024.", "other_comments_or_suggestions": "Please see **Other Strengths And Weaknesses**.", "questions_for_authors": "Please see **Other Strengths And Weaknesses**.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a practical and efficient method for long-sequence LLM inference, offering substantial speedups (up to 11×) while maintaining high accuracy (95–100%). It is compatible with existing Transformer models and demonstrates strong performance compared to Ring Attention and Sparse Attention. The two-phase design is well-motivated, and the empirical results—strengthened by a thorough rebuttal—show consistent performance across diverse tasks and benchmarks. However, concerns remain regarding the method's novelty, with some reviewers noting similarities to existing sparse attention mechanisms. 

The submission has polarized reviewers. One reviewer recommends acceptance, highlighting originality and empirical strength; another leans toward rejection due to perceived lack of novelty and insufficient motivation. Two reviewers rate the paper as weak rejects but appreciate its engineering contributions.

Despite some valid critiques, the paper demonstrates clear technical contributions, practical relevance, and thorough empirical validation. The authors actively addressed all reviewer concerns during rebuttal, and the method’s scalability and compatibility with existing LLM architectures are compelling.

The paper should be accepted, conditional on improving discussion around prior work (especially streaming variants and attention sparsification techniques), and clarifying performance metrics (e.g., decoupling memory vs. speedup claims). A more comprehensive system-level analysis could further strengthen the paper.
Decision: Accept (poster)