Kati8/ICLR_2023_2025 huggingface dataset:
authors: Xiaoran Liu
authors: Hang Yan
authors: Chenxin An
authors: Xipeng Qiu
reviews: 
Comments: This paper studies how shall we scale the transformer context length up via modifying RoPE. Different from popular approaches that increase the rotary base, authors found both increasing and decreasing the rotary base could significantly enhance its extrapolation performance.
Strengths: 1) This paper studies a very interesting problem: How shall we modify the 10^4 rotary base? 
2) Well-written. 
3) Some reasoning and insights are very cool. Such as the Q1 and Q2 in the introduction.
4) The theoretical analysis of the benefits of a smaller rotary base would be helpful to the long seq community.
Weaknesses: 1) I have one strong concern about the evaluation of this paper. This paper only evaluates the scaled model on the long-context perplexity and short-context well-established benchmarks. Low ppl on long context language modeling cannot really reflect the good results on long-range reasoning. The short-context results can only show the proposed approach is not that harmful on short tasks. If so, there is no evidence showing that the proposed framework can really improve the long-range real-world tasks, which is exactly what we espect in the long-range modeling research.

2) There is no related work section. It is "okay" for expert readers, but it would be very helpful to make a broader impact if authors can write one. I think authors should at least include:
- scaling seq len with position embedding. The authors discussed some of these approaches, but it would be better to summarize them in a thread.
- scaling seq len with efficient implementation, such as Flash attention 1&2 [1]
- scaling seq len with a distributed system, such as Sequence Parallelism [2][3]
 

[1] https://github.com/Dao-AILab/flash-attention

[2] Li, Shenggui, et al. "Sequence parallelism: Long sequence training from system perspective." arXiv preprint arXiv:2105.13120 (2021).

[3] Jacobs, Sam Ade, et al. "DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models." arXiv preprint arXiv:2309.14509 (2023).

3) It would also be very interesting to study the filling-in-the-middle pattern after applying the proposed trick. Is the filling-in-the-middle related to the position embedding? This is just a bonus question. It is an okay paper without this study, but I suggest authors to study this to deliver more insights.

I give a relatively negative score at this stage because of the lack of real-world task evaluation, but the insights of this paper are interesting to me. Authors need to further justify them. I can improve my score if the authors can add the real-world long-context evaluation into the draft.
Recommendation: 8: accept, good paper
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: Rotary Position Embeddings have played a significant role in improving the scalability of LLMs. However, RoPE faces problems when trying to extrapolate beyond the length the model was trained on.

Various approaches have been taken to solve the issue with extrapolation, and this paper goes in depth describing why these issues exist and derives pretty good solutions as well. Apart from this, it presents a good understanding of why some of the existing methods work and why a base of 10,000 is not a good value for RoPE's base.

All the claims are very well substantiated with theoretical as well as empirical analysis.
Strengths: - Pretty useful theorems that shows why and how RoPE's beta impacts extrapolation.
- Results on critical base, d_{extra}, and T_{extra} and their relationship is a very useful contribution in this area.
- Both experimental and theoretical analysis are strong.
Weaknesses: A bit more clarification around the calculation of critical beta, and why extrapolation capabilities exist even when beta values are large could be added since we will see OOD embeddings as soon as T is larger than T_{train} during inference.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: Rotary Position Embeddings have become instrumental in enhancing the efficiency of large language models. Nonetheless, RoPE encounters significant challenges when attempting to generalize beyond sequence lengths seen during training. This work provides a thorough analysis of the underlying causes of these limitations and proposes well-motivated solutions. Additionally, it offers valuable insights into the effectiveness of existing mitigation strategies and explains why the conventional base value of 10,000 proves suboptimal for RoPE. All arguments are rigorously validated through both theoretical derivations and comprehensive experimental evidence.
Strengths: - Derives insightful theorems demonstrating the mechanisms through which RoPE's beta parameter influences length extrapolation performance. - Establishes meaningful relationships between critical base, d_{extra}, and T_{extra}, providing important takeaways for practitioners in this domain. - Combines rigorous mathematical analysis with solid empirical validation.
Weaknesses: The presentation could benefit from additional detail regarding the derivation of critical beta values and the theoretical rationale for why extrapolation capabilities persist in cases where beta parameters are relatively large, particularly given that out-of-distribution embeddings emerge whenever the sequence length exceeds T_{train} during inference.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent