reviews: 
Comments: This paper focuses on the extrapolation capability of Rotary Position Embedding (RoPE) in large language models (LLMs). The authors first observe that pre-training RoPE with smaller or larger bases can significantly enhance its extrapolation performance. Then, they propose the Scaling Laws of RoPE-based Extrapolation, a unified framework that describes the relationship between extrapolation performance and base value as well as tuning context length from a periodic perspective. Furthermore, they explain the origin of the RoPE-based extrapolation issue by the critical dimension for extrapolation. Finally, they achieve extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B.
Strengths: - Observing that pre-training RoPE with smaller or larger bases can significantly enhance its extrapolation performance, providing a new perspective for extrapolation research.
- Proposing the Scaling Laws of RoPE-based Extrapolation, offering a unified theoretical framework for RoPE extrapolation.
- Explaining the origin of the RoPE-based extrapolation issue through the critical dimension for extrapolation.
- Achieving extrapolation up to 1 million context length within only 16K training length on LLaMA2 7B and 13B, demonstrating the potential of RoPE extrapolation.
- Providing guidance for other extrapolation methods, such as dynamic NTK and Code LLaMA.
Weaknesses: - In fact, the analysis results of this article are basically available in the YaRN[4] paper, which had pointed out that certain dimensions have high frequency and enough turns are made during training, therefore, the training of these dimensions is sufficient and should remain unchanged during extrapolation; for those dimensions that are less than one circle, only interpolation (PI) can be used to ensure numerical stability. This is obviously the equivalent expression of the proposed concept "critical dimension" in this work.
- Most experimental comparisons only report ppl, which does not represent real long context ability. The authors should conduct experiments on some practical benchmarks and tasks. Eg. PASSKEY RETRIEVAL[2], the Hugging Face Open LLM Leaderboard[3] (ARC, HellaSwag, TruthfulQA, etc).
- The RoPE extrapolation methods discussed in the paper may have certain limitations in practical applications, it is just verified over perplexity, but not enough practical application scene.
- Compared to baselines, it lacks some comparison results over computational resources and memory usage.
- Missing reference. The work should compare results with ReRoPE[1].

[1] J. Su. Rectified rotary position embeddings. https://github.com/bojone/rerope, 2023.

[2] A. Mohtashami and M. Jaggi. Landmark attention: Random-access infinite context length for transformers, 2023. arXiv: 2305.16300.

[3] Hugging Face. Open LLM Leaderboard, 2023. URL https://huggingface.co/spaces/ HuggingFaceH4/open_llm_leaderboard.

[4] YaRN: Efficient Context Window Extension of Large Language Models
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 3 good
Clarity: 3 good