Long Input Benchmark for Russian Analysis
Authors: Igor Churin, Murat Apishev, Maria Tikhonova, Denis Shevelev, Aydar Bulatov, Yuri Kuratov, Sergei Averkiev, Alena Fenogenova


Reviewer paper summaries: This paper presents a novel benchmark for evaluating Long Input Large Language Models (LLMs) in Russian. The benchmark consists of 21 datasets, which include translations from existing resources, adaptations of existing texts, and newly generated datasets (some created automatically by combining existing texts and others annotated by humans). The datasets are organized into four groups based on increasing levels of difficulty. Additionally, the paper includes a test section to assess the performance of various state-of-the-art LLMs on this benchmark
Summary of strengths: - The paper is well-written and easy to follow, with a clear and well-organized structure. 
- The problem is effectively contextualized and motivated. 
- The benchmark appears novel, particularly given the lack of similar evaluations for the Russian language. 
- The numerous tables help the reader's understanding of the baseline models' performance. 
- The Appendix provides extensive supplementary information.
Summary of weaknesses: - Although the authors discuss the issue of data leakage in the Appendix, I believe this aspect should be explored more in-depth. The resource is invaluable, but it remains unclear whether the performance of the tested models is due to effective pre-training and fine-tuning or because these models have previously seen the data.
- As a benchmark, I expected to see an online ranking and evaluation procedure for newly submitted models, potentially contributed by the community.
Comments, suggestions and typos: - Third contribution point (lines 64-65): is not clear what you mean, do you release some kind of software? Or the online leaderboard? 
- Lines 321-322: "The dataset is balanced across answers" do you mean across Yes/No? What about the Topics distribution?
- Line 494: "pretrain once" -> "pre-trained ones"
- Footnote number 38 in the Appendix: you repeated "and are given and are".. I guess you can remove "and are given".
Overall assessment scores: 4