Kati8/ICLR_2023_2025 huggingface dataset:
authors: Tri Dao
reviews: 
Comments: This paper describes FlashAttention-2 which improves upon FlashAttention by introducing "tweaks" to improve performance on GPUs.   The paper claims the tweaks improve performance by increasing occupancy and use of matrix-multiply hardware (tensor cores).  The paper reports a bit under $1.3\times$ wall clock speedup versus FlashAttention.
Strengths: Improving training speed of LLMs is of great interest to many.
Weaknesses: Could do a better good job explaining how a given "tweak" helps achieve a given improvement (occupancy, use of tensor cores).
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: This paper presents improvements to FlashAttention (Dao et al., 2022), a established method for efficiently computing attention through fused ops. These improvements are designed specifically for better parallelism and work partitioning in GPUs, resulting in the development of FlashAttention v2. Performance benchmarks have been conducted for both training and inference phases. Additionally, the authors provide comprehensive results from training end-to-end GPT-style models with 1.3 billion and 2.7 billion parameters and 2k and 8k context sizes.
Strengths: There are many things to like in this paper, such as:

1. The paper is well-written
2. The improvements are well-explained and justified
3. The paper covers both training and inference time optimization
4. The results are encouraging

In summary, I expect widespread adoption of FlashAttention v2 within the community. Furthermore, the methods proposed and utilized in this paper could inspire the creation of more efficient components in machine learning.
Weaknesses: FlashAttention v2 has a notable limitation: it relies on recent, specialized GPU architectures like the A100 (and H100). Additionally, the requirement for custom CUDA kernels adds a layer of complexity. 

A small critique is that Figure 3 could benefit from a more descriptive caption.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 3 good
reviews: 
Comments: Scaling Transformers for longer sequences holds the promise of enhancing language modeling and understanding complex inputs, but is hindered by the attention layer's quadratic scaling in memory and runtime. FlashAttention has mitigated this by bringing linear memory usage and considerable runtime speedup, yet it still lags behind the efficiency of optimized matrix multiplication operations. To address this, FlashAttention-2 is introduced with improved work partitioning, yielding a significant speedup and reaching closer to the efficiency of matrix multiply (GEMM) operations. Empirical validation shows that FlashAttention-2 significantly increases the training speed of GPT-style models on both A100 and H100 GPUs.
Strengths: * The proposed platform-specific optimizations are clever and sound.
* The resulting software artifacts are useful and have has the potential to benefit both researchers and practitioners.
Weaknesses: * The work is mostly engineering-focused, with several "tweaks" made to FlashAttention.
* The performance gains are relatively marginal, especially when compared to those of the original FlashAttention over the baseline.
* The absence of an ablation study makes it difficult to pinpoint the exact sources of efficiency.
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 3 good
reviews: 
Comments: This work introduces enhancements to the FlashAttention framework (Dao et al., 2022), a well-established technique for accelerating attention computations via kernel fusion. The proposed refinements target improved GPU parallelization strategies and task distribution, leading to FlashAttention v2. The authors conduct thorough performance evaluations across both the training and inference stages. Furthermore, they demonstrate end-to-end results by training GPT-like language models ranging from 1.3B to 2.7B parameters with context windows of 2k and 8k tokens.
Strengths: The submission has numerous commendable qualities:

1. The manuscript is clearly and effectively written
2. The proposed enhancements are presented with solid technical justification
3. Both training and inference performance improvements are addressed
4. The empirical results are compelling

Overall, the work appears poised for significant adoption across the research and engineering communities. The techniques and strategies employed may also catalyze the development of additional optimized modules within deep learning systems.
Weaknesses: A significant constraint of FlashAttention v2 is its dependence on cutting-edge GPU hardware such as the A100 and H100 processors. Furthermore, the implementation necessitates hand-optimized CUDA kernels, which introduces additional engineering overhead.

As a minor observation, Figure 3 would benefit from an enhanced and more detailed figure caption for improved clarity.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 3 good