reviews: 
Comments: The paper introduces FlexPrefill, a novel sparse attention mechanism aimed at improving the efficiency of long-sequence inference for large language models (LLMs). The proposed approach addresses the computational challenges associated with long-sequence processing by dynamically adjusting sparse attention patterns and computational budgets based on input characteristics. FlexPrefill consists of two key components: Query-Aware Sparse Pattern Determination, which selects between diverse and structured attention patterns using Jensen-Shannon divergence, and Cumulative-Attention Based Index Selection, which ensures efficient attention computation by dynamically selecting query-key indices based on attention scores. Experimental results demonstrate  improvements in both speed and accuracy compared to existing methods, such as Streaming LLM and Minference.
Strengths: 1. Originality: The paper presents an adaptive and flexible approach to sparse attention that addresses the limitations of fixed sparse attention mechanisms. The real-time adjustment of sparse attention patterns is a notable advancement in handling varying input needs during long-sequence inference, contributing to the growing body of research focused on optimizing LLM efficiency.

2. Quality: The quality of the experimental setup is high, with comprehensive comparisons to baseline methods like Streaming LLM and Minference. The authors provide detailed analyses that demonstrate improvements in both speed and accuracy, supporting the efficacy of the proposed method. The ablation studies further strengthen the validity of the claims by examining the contributions of each component of the approach.

3. Clarity: The paper is well-structured and clearly written, making it accessible to a broad audience. The authors provide a thorough explanation of the motivation, methodology, and experimental results.
Weaknesses: 1. Complexity of the Proposed Sparse Attention Mechanism: The multi-step decision-making process, involving both Query-Aware Sparse Pattern Determination and Cumulative-Attention Based Index Selection, introduces additional complexity to the attention mechanism. A detailed computational complexity analysis is missing, which raises questions about how the overhead introduced by this process is justified by the computational savings achieved through sparse attention.

2. Lack of Detailed Comparison of Computational Savings: The paper lacks a comprehensive comparison of how much computation is saved by using FlexPrefill compared to the baselines presented in Tables 1 and 2. This information would be valuable in quantifying the benefits of the proposed approach.

3. Limited Comparison with Recent Related Works: The paper does not include comparisons with other recent related works, such as Han et al. (2024) and Xiao et al. (2024a). Including these comparisons would help to better position the contributions of FlexPrefill within the context of current research.

4. Mixed Performance Benefits: The reported performance benefits compared to the Full Attention model on long-context benchmarks are mixed, as shown in Tables 1 and 2. Additionally, the computational savings compared to the Full Attention model and other baselines are not adequately reported, which limits the understanding of the trade-offs involved.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 4