AcademicEval: Live Long-Context LLM Benchmark
Authors: Haozhen Zhang
Tao Feng
Pengrui Han
Jiaxuan You
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have recently achieved remarkable performance in long-context understanding. However, current long-context LLM benchmarks are limited by rigid context length, labor-intensive annotation, and the pressing challenge of label leakage issues during LLM training. Therefore, we propose \textsc{AcademicEval}, a live benchmark for evaluating LLMs over long-context generation tasks. \textsc{AcademicEval} adopts papers on arXiv to introduce several academic writing tasks with long-context inputs, \textit{i.e.}, \textsc{Title}, \textsc{Abstract}, \textsc{Introduction}, and \textsc{Related Work}, which cover a wide range of abstraction levels and require no manual labeling. Moreover, \textsc{AcademicEval} integrates high-quality and expert-curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, \textsc{AcademicEval} features an efficient live evaluation, ensuring no label leakage. We conduct a holistic evaluation on \textsc{AcademicEval}, and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, highlighting the challenge of our benchmark. Through experimental analysis, we also reveal some insights for enhancing LLMs' long-context modeling capabilities. Code is available at \url{https://github.com/ulab-uiuc/AcademicEval}

Published in Transactions on Machine Learning Research (10/2025)
AcademicEval: Live Long-Context LLM Benchmark
Haozhen Zhang
huozhemz@illinnis.ccom edu, wazhz14 @gmaii com com
University of Alinois at Trbana-Champonn
Tao Feng
taofeng2@iliioosscooo edu
University of Illinois at Urbana- Champaign
Pengrui Han
phan1 edu
University of Alinois at Chaana-Chmppige
Jiaxuan You
Bazuan@illinois edu
University of Illinois Urbana- Chompaign
Reviewed on OpenReview: https: // optpr//eeepr...e....m net/ forum? id=1)04uoE5bs
Abstract
Large Language Models (LLMs) have recently achieved remarkable performance in long- context understanding, However, current long-context LLM benchmarks are limited by rigid context length, labor-intensive annotation, and the pressing challenge of label leakage issucs during LLM training. Therefore, we propose ACADEMICEVAL, live benchmark for evalu- ating LLMss over long-context generation tasks. ACADEMICEVAL adopts papers arXiv to introduce several academic writing tasks with long-context inputs, i.c., TITLE, ABSTRACT, INTRODUCTION, and RELATED WORK, which cover wide range of abstraction levels and require no manual labeling- Moreover, ACADEMICEVAL integrates high-quality and expert- curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, ACADEMICEVAL features an efficient live evaluation, cnsuring no labol leakage. We conduct holistic evaluation on ACADEMICEVAL, and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, highlighting the challenge of our benchmark. Through exper- imental analysis, we also reveal some insights for enhancing LLMs' long-context modeling capabilities. Code is available at httpt::/// //gi trithuu..... wwt.ccommulaaeeeeeeo wullullaauuuuaa Academi cEval
1 Introduction
Large Language Models (LLMs) have recently achieved tremendous success in natural language processing (NLP) tasks (Achiam et al. 2023 Al@Meta show a sharp decline in performance, which poses a pressing challenge to LLMs in understanding and capturing key information in long texts [Li et al.
2024).
However, when facing long context inputs, LLMs
Liu et al. 2024]. Therefore, several cong-context LLM benchnarks are spawned to evaluate LLMs in vario various settings, including question answering, summarixing,
2024
and reasoning (Shaham et al. 2023 An et al. Zhang et al.] 2024b). Despite their success, these benchmarks still suffer from concerns of rigid context length, saturated performance, and being leaked in LLM training.
2023
Dong et al. 2023 Bai et al. 2023b Li et al. 2023
We envision that the mext-gemeration long-contezt LLM benchmarks should ideally possess three key features. (1) Flezible and potentially unlimited context length: existing benchmarks fix the context for each long- context problem; ideally, the format and length of the context could be flexibly set based on the LLM's capability, especially given the release of long-context LLMs (Reid et al., (2024) and their capabilities in ingesting multi-modal information, e.g, graphs Dong et al. 2024 (2) High-quality labels derived from
Work done as intern University Minois Urbana- Urbana-Champan Champaign

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Comparison with Existing Long-context LLM Benchmarks. Each column indicates the average imput length, whether the annotation is human-assisted, whether there tasks with hierarchical abstraction levels, whether it contains few-shot demonsirations, and whether the benchmark is lively updated, respectively.
Automatic
Hierarchical Abstraction
Few-shot Demos
Live Update
Benchmark
Avg Len Annotation
ZeroSCROLLS
Shaham ct al.
2023
~0KK
X
L-Eval (An et al. 2023
~8
X
BAMBOO Dong et al. 2023
16K
X
X
LongBench Bai et al. 2023b
8K
X
LooGLE
(Li ct al. 2023
20K
X
ooBench (Zhang el. al.)
2024b)
200K
X
AcademicEval (ours)
Flexible
real-world data, minimizing human labeling efforts: existing long-context benchmarks often require human labeling (Bai al. 2023b An ct al. 2023 Li al. 2023 Dong ct al.
2023 Zhang ct al. 2024b), which
is costly and limits the size of the benchmarks to about. 2000 samples
[Xu el al. 2023 (3) Live updates
to mitigate information leakage during LLM pretraining and fine-tuning: benchmark data contamination in
LLM has gradually become severe issuc (Sainz et al. 2023 Yc ct al. 2024 Zhu et al. 2024b Xu et al.
2024); we argue that holding out futurr data as the val/ test set is one of the most effective approaches for open benchmarks.
Based on these principles, we propose ACADEMICEVAL, live benchmark to evaluate LLMs over long-context generation tasks. ACADEMICEVAL adopts arXiv as its data source and features suite of academic writing tasks each paper without labor-intensive annotation: TITLE, ABSTRACT, INTRODUCTION, and RELATED WORK, each of which has long-context input and hierarchical abstraction levels. In particular, we construct co-author graph via the arXiv API to conveniently obtain co-author papers high-quality and expert- curated few-shot demonstrations, which also possess ACADEMICEVAL flexible context length. Furthermore, ACADEMICEVAL introduces efficient live evaluation based on the co-author graph, which utilizes the lat- est papers on arXiv to update the benchmark data periodically and ensures no label leakage. Moreover, ACADEMICEVAL provides in-context few-shot demonstrations for each sample, which is neglected by most ex- isting long-context LLM benchmarks (Liu al. 2024 Li et al. 2024). In our experiments, we evaluate three categories of baselines ACADEMICEVAL: standard LLMs, long-context LLMs, and retrieval-augmennt language models (RALM). Under automatic metrics (BERTScore and ROUGE-L.). RALM often attains the strongest results by concentrating salient evidence into shorter retrieved chunks, while long-context LLMs and strong standard modols remain competitive several settings. However, an ALM-as-a-Juudo evaluation, which assesses novelty, feasibility, consistency, factuality, and academic style, reveals a more nuanced picture: retrieval not always preferred (e.g., for TITLE/ ABSTRAcT). whereas is highly beneficial for RELATED WORK. Across both evaluations, performance commonly degrades as the input length grows, and correlated few-shot demonstrations from the co-author graph can provide modest gains for specific model-tan task pairs. Overall, the results indicate that ACADEMICEVAL. is challenging benchmark that exposes complementary facets of long-come cong-context modcling: overlap-oriented automatic metrics and higher-level judged quality.
We illustrate the comparison with existing long-context LLM benchmarks in Table T. Our contributions are summarized as follows:
We propose live benchmark, ACADEMICEVAL, evaluate LLMs over long-context generation tasks. ACADEMICEVAL features four academic writing tasks with hierarchical abstraction levels and requires manual annotation.
2

---

Published in Transactions on Machine Learning Research (10/2025)
We construct co-author graph via the arXiv API and draw the co-author papers informative few-shot demonstrations, making the context length of ACademIcEval, flexible and scalable. Es- pecially, ACADEMICEVAL conducts periodic data updates on the co-author graph to enable efficient live evaluation, which ensures no label leakage and fair evaluation.
We conduct comprehensive experiments on ACADEMICEVAL, and the results demonstrate its chal- lenges and yield potential insights for improving LLMs in long-context modeling.
2 Related Work
Long-context Modeling and LLM Benchmarks. LLMs are known to be powerful in language modeling
tasks (Achiam et al. 2023
Al@Meta
2024). However, when it comes to long-contert inputs, LLMs show a pressing challenge when benchmarking their long-context modeling
sharp decline in performance, posing a 2024 Li et al.
capabilities (Liu et al.
2024
2025). Currently, there are two mainstream technologies for Yu et al.
long-context modeling tasks: retrieval
augmented language models (RALM) (Ram et al. 2023
2023
Trivedi et al. 2022 Jiang et al.
2023 Asai et al. 2023 Zhang et al.
2024a Feng et al. 2024) and
long-context. LLMs Bai et al. 2023a; Jiang ct al. 2024 Teknnumm et al.p.
RALM equips LLMs with a
retriever (Robertson al., 2009; Ramos et. al. 2003 Karpukhin et al., 2020
Izacard et al. 2021 to perform
information retrieval on short text chunks, which are then fed to LLMs together with the input query to generate the final output. As retrieval system, RALM is usually evaluated over retricval- -based benchmarks,
including STARK (Wu et al.) 2024), RGB (Chen et al.) 2024h, ARES (Saad-Falcon et al. 2023 ), etc. In
comparison, long-context LLMs expand their context window length to accommodate longer inputs and are benchmarked over various tasks, which include long-context QA, summarization, conversations, reasoning,
etc (Shaham et al. 2023 An et al. 2023 Dong et al. 2023
Bai et al. 2023b Li al. 2023 Zhang et al.
2024b Li al., 2025).
Recent works such ResearchTown Yu et al. 2024 and WildLong Li et al. 2025 share conceptual proximity to our setting but target different goals. RescarchTown is a multi-agent simulation framework that models the dynamics of a research community via message- paasssngg on an agent- data graph, simulating activities such as paper and review writing. Its focus lies in simulating collaborative behavior and ensuring the realism of outputs under controlled settings. In contrast, ACaDeMIcEVaL is a live, real-world benchmark grounded in authentic academic papers, designed to evaluate LLMs on hierarchical writing tasks (TTTLE, ABSTRACT, INTRODUCTION, and RELATED WORK) under evolving and leakago-resistant conditions. While both leverage graph structures, Rescarch uses them for interaction simulation, whercas AcademicEval employs a co-author graph for retrieving high-quality few-shot demonstrations supporting scalable context lengths, and enabling periodic data updates.
Similarly, WildLong introduces a scalable framework for synthesizing realistic long-context instruction data. It extracts meta-information from user queries, builds co-occurrence graphs, and employs adaplive generation to create 150K instruction response pairs for complex multi-document reasoning tasks. While WildLong focuses data synthosis for instruction tuning, ACADEMICEVAL focuses on evaluation, providing live, automatically updated benchmark that measures LLMs' long-context reasoning and generation abilities on real-world academic tasks. Together, these works are complementary: ResearchTown and WildL.ng contribute to synthetic data generation and simulation, whereas ACadeMIcEVal provides a robust evaluation framework for real-world, graph-enabled long-context reasoning.
Label Leakage in LLM Benchmarks. Label leakage has always been a severe issue that benchmarks must attempt to avoid during data collection. However, recent research (Xu et al. 2024 Zhu et al., 2024b a Ye et al.) 2024 point out that most LLM benchmarks are composed of statically collected data, which may be inevitably included in the large amount of training data of LLMs, causing label leakage. Therefore, some works attempt to mcasure or detect the extent of label Icakage in LLM benchmarks. Benbench (Xu
let al.
2024) leverages perplexily and N-gram accuracy to quantify potential label leakage, while PAC (Ye et al.
2024h detects contaminated data by comparing the polarized distance of samples before and after augmentation. Even though these approaches propose to mcasure or detect label leakage, there little work on mitigating and solving this issue (Zhu et al. (024bb). Dynabench (Kiela et al. 2021)) and Dynaboard (Ma) et al. 2021 feature dynamic human-in- human-in-the-loop dataset creation while avoiding leakage, which is very labor-
3

---

Published in Transactions on Machine Learning Research (10/2025)
intensive. DyVal (Zuu et al., 2024b) leverages pre-set constraints and directed acyelic graphs (DAG) to dynamically generate test cases with diverse complexities, reducing the risk of label leakage. FreshBench (Zhu et al. 2024ah and StackMIA (Ye et al.) 2024) collect the latest data from public websites periodically and simply rely on the chronological split to build a dynamic benchmark.
Long-context Summarization Benchmarks. Solving ACADEMICEVAL requires LLM's long-context sum- marization capability (Liu et a 2024). Existing works include (1) query- based summarization tasks, focus- ing on the capability of models to position and capture local key information in long texts given a specific query [Ltvak & Vanetik 2017 Wang et al.) 2022 (2) single-document or multi-document summarization tasks concentrate on evaluating the ability of models to understand long texts holistically (Cohan et al. 2018 Meng et al. 2021 Huang et al. 2021] Kryšcińiski et al. 2021: Cachola et al. 2020). These long-context limitations, including requiring
summarization benchmarks suffer from the
labeling and concerns about data, leakage; moreover, these summarization tasks focus on one-level summa- rization, failing to consider the summarizations at different abstraction levels.
3 AcademicEval Benchmark
In this section, we propose ACADEMICEVAL (Figure 1 for live evaluation over long-context generation lasks with hierarchical abstraction levels. We first describe data collection and preprocessing in Section 3.1 Then, in Section 3.2 four academic writing tasks with diverse abstraction levels are introduced, and wc also integrate few-shot demonstrations to make the context length flexible and scalable. Finally, Section 3.3 elucidates the live evaluation with periodic data updates.
3.1 Data Curation
Co-author Graph Construction via Xiv. As a public paper preprint platform, arXiv has always been favored by researchers. It archives a huge amount of papers and updates the latest ones daily, which serves as an excellent data source and also lays the foundation for the live update of our benchmark. Thanks to the arXiv A paper files can be obtained in batch without much mamual effort. We first collect and construct a co-author graph (i.e., edges are established between two co-author nodes) using the arXiv API through breadth-firtt search (BFS), where the features of each author node include the published first-author papers. By making the co-author graph the carrier of papers, we can form an interconnected whole of scattered articles, which provides valuable structural information to be exploited for our benchmark. Furthermore, we can enable efficient live updates on the co-author graph, which will be introduced in Section 3.3
Academic Data Gathering and Preprocessing. After the co-author graph is collected, we remove authors who have not published independent first-author papers (i.e., appear only as co-authors in the author list) and then prune it to obtain the maximum connected component. For each paper (f.e., node features), we collect essential metadata via the arXiv APT, including author information, publication timestamp, etc., and download the PDF file simultaneously, which further goes through a series of pipelines split and extract the text of several sections in it. In detail, we leverage PyMuPDI to detect section headings (e.g.. "Introduction") and extract the paper content by sections. Especially for the "Related Work' section, we extract each cited paper's abstract and title via the arXiv API to form an additional citation corpus. All these processed data constitute the node feature of each author node. We will further describe in Section 3.2 how to use these data to design long-context academic writing tasks.
3.2 Benchmarking LLMs over Long-context Generation Tasks with Hierarchical Abstraction
Task Description. Employing machine learning approaches to automate academic writing has always been a rescarch hotspot with significant, practical application value (Chen et al., 2022 2021). Therefore, inspired
by the leave-one-out validation, we introduce four academic writing tasks with ultra-long context to evaluate the generation capability of LLMs under different abstraction levels, as shown below:
'https://arxxiv.orr
hhtte:///mfoaaxii..orr////////// 3 Shtps///ghhub.cmmpppmppff//f
4

---

Published in Transactions on Machine Learning Research (10/2025)
ar
Author Node
Co cauthr
TITLE
Preprocess
ABSTRACT
Chronological Split
MAIN BODY
INTRODUCTION
RELATED WORK
Co-author Graph
CITATION CORPUS
Integration
Few-shot Demonstrations Selection
Paper Sample
Figure 1: AcademicEval Benchmark. We construct co-author graph via arXiv and conduct chrono- logical split on all paper samples (training, validation, and test samples are represented by red, orange, and green, respectively). Each paper sample is preprocessed into separate sections and can integrated with few-shot demonstrations from co-author papers.
TITLE WRITING. This task takes a paper's main body and abstract, along with specific task prompt as inputs, and then asks LLMs to output predicted title.
ABSTRACT WRITING. Similar the above, this task takes paper's main body (with the 'Conchu- sion' section removed) and title, along with specific task prompl inpuls, and then asks LLMs to output a predicted abstract.
INTRODUCTION WRITING. This task takes a paper's main body (with the "Introduction' section removed), title, and abstract, along with a specific task prompt as inputs, and then asks LLMs output predicted introduction.
RELATED WORK WRITING. This task takes paper's main body (with the 'Related Work' section removed), title, abstract, and citation corpus (introduced in Section 3.1), along with specific task prompt as inputs, and then asks LLMs to output predicted related work.
Based on the above task descriptions, we can generate four basic benchmark settings with different abstrac- tion levels, namely TITLE- 10K, ABs-9K, INTRO-8K and RELATED- -44K, with suffixes indicating their input context l Intuitively, the paper content itself can be considered as kind of original, expert-curated, and high-quality labeled data without manual annotation. Therefore, for evaluation, directly adopt the corresponding paper section as the ground truth for each benchmark setting, minimizing human labeling efforts.
Integration of Few-shot Demonstrations. Given the rigid context length of current long-context LLM benchmarks and the general effectiveness of in-context learning in LLMs (Dong al.] 2022 Wei et all
2022a by Kojima al. 2022). we propose integrate long few-shot demonstrations to enable flexible and scalable context length, and have two selection options for each sample in the above four basic benchmark
WW BERT DDvvlin 2018 tokenizer default count number of input tokens (output takens are included).

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Data Statistics of AcademicEval (Initial Round). includes writing tasks and provides four scttings of different context length for cach task. For each sctting, we list their Comp. Rate, Samples of Each, Chronological Split, and Timespan of Test Data.
Comp. Rate
#Samples
Chronological Split of Each. (Train-Val-Tess Val-Test)
Timespan of Test. Data
Setting
(In-Len. Out-Len.)
Title Writing
Title-10K
587
Title-30K
1773
2024.06- 2024.07
5098
72%-19%99%
Title-31KGG
1807
Title-50KMM
2968
Abstract Writing
Abs99K
36
Abs-28K
108
2024.06-
5098
72%-19%-9%
Abs- -
112
2024.07
Abs-48K-M
185
Introduction Writing
Intro-8K
Intro-28K
21
2024.06-
4665
71%-20%-9%
Intro-28K-G
22
2024.07
Intro-48K-M
37
Related Work Writing
Related-34K
34
Related-53K
53
2024.06-
2240
72%-20%-8%
Related-53K-G
53
2024.07
Related-72K-M
72
Note: We use the BERT tokenizer by default to count the number of tokens.
settings: (1) Randomly select papers under the same category. According to the paper categories provided by the arXiv API, we can randomly select several non-duplicate papers under the same category. (2) Randomly Select co-author papers. The motivation straightforward: the similarity of rescarch directions between co-author papers more fine-grained. Thanks to the co-author graph, is convenient to obtain the co- author papers of each original paper sample. These selected papers serve few-shot demonstrations and are utilized as input-output pairs enrich the input context of the original samples, providing potentially insightful and relevant content while enabling flexible and scalable context length.
Consequently, we have completed the construction of benchmark settings, and the data statistics in the initial collection round are shown in Table 2
Data Statistics. As shown in Table 2 ACADEMICEVAL has four academic writing tasks with hierarchical abstraction levels, and each task fcatures four scttings with diverse input context lengths, some of which are obtained by integrating few-shot demonstrations. For instance, each sample in TITLE-1OK consists of single paper sample. TITLE-30K and TITLE-31K-G are obtained by integrating with two few-shot demonstrations from random papers and co-author papers, respectively, while TITLE-50K-M is obtained by using both of the above integration options. Actually, we can scale context length by increasing the number of few-shot demonstrations to provide more informative references, enhancing task performance.
Furthermore, we present the text compression rate (defined as the number of input tokens divided by the number of output tokens) for each benchmark setting in Table 2 to illustrate the diverse abstraction levels in ACADEMICEVAL. Across the four tasks, higher compression rate means higher level of text. abstraction

---

Published in Transactions on Machine Learning Research (10/2025)
3. Graph Pruning
1. Node Update
2. Node & Edge Update
NEW
NEW
NEW
Chronological Split
NEW
NEW
MB
NEW
Figure 2: Live Evaluation of AcademicEval Benchmark. To support continual benchmarking, ACA- DEMICEVAL incrementally updates the co-author graph using daily arXiv data. The procedure includes: (1) Node Update augmenting node features for authors with newly published first-author papers; (2) Node and Edge Update identifying and prioritizing co-authors via BFS to expand the graph with recent publications; and (3) Graph Pruning removing outdated papers and inactive authors to maintain graph connectivity and efficiency.
in this task. Among several settings within each task, a higher compression rate makes it tougher to exploit information holistically but more likely to produce better outputs (since more references are integrated). These different tasks and scttings increase the diversity of the ACADEMICEVAL benchmark.
As for data splitting, we perform chronological split in ACAdEMICEVAl, which means that the test set always contains the latest papers collected in each collection round, ensuring no label leakage. Note that Table / shows only the data collected in the initial round, which will be updated periodically as described in the next section.
3.3 Live Evaluation with Periodic Data Updates on the Co-author Graph
The daily updates of arXiv provide the basis for the live evaluation of ACADEMICEVAL: we can periodically update the benchmark with the latest papers on arXiv. By setting a reasonable update cycle (e.g., monthly or quarterly), we can ensure that the data in the benchmark is not contaminated so that it can be used to evaluate LLMs fairly in a live manner. Therefore, we proposed an efficient incremental update procedure on the co-author graph:
(1) Node Update. For each author the co-author graph, check whether the author has newly published first-author paper through the arXiv API. If so, add it to the corresponding node feature on the co-author graph.
(2) Node and Edge Update. During the traversal of Node Update, each author's new co-authors are added to a candidate list, and the number new papers (including first-author and non-first-author papers) when scarching for the author used as the priority of the co-authors (co-authors of active authors tend to be active as well, and we can efficiently collect the latest papers from active authors). Then, we use the prioritized candidate list to conduct BFS to update nodes and edges until specific number of incremental update papers is met.
(3) Graph Pruning. As the benchmark is updated, we will remove some outdated papers and inactive authors (defined as those who have not published new first-author or non-first-author papers for a long time) from the co-author graph.
In this way, the latest papers can be obtained sufficiently and efficiently while ensuring connectivity and a smaller graph size.

---

Published in Transactions on Machine Learning Research (10/2025)
4 Experiments
4.1 Baselines
We adopt the following three types of baselines to conduct holistic evaluation of ACADEMICEVAL.
Standard LLMs. We choose Gemma Instruct (7B) (Team et al.
2024 and LLaMA-3 Chat
(70B) AetMeta,
2024 standard LLM baselines, each with context length of 8K.
Long-context LLMs. We choose Qwen 1.5 Chat (72B) Bai et. al.
2023a), Mixtral-8x7B Instruct
(46.7B) (Jiang et al. 2024), and Nous Hermes 2 Mixtral 8x7B-DPO (46.7B) Teknnumm et al.) as long-context
LLM baselines, each with context length of 32K.
Retrieval-augmented language models (RALM). First, we consider two sparse retrievers: (1) BM25 (Robertson et al. 2009): This is widely used retricval model that ranks documents based on the frequency of query terms in each document. (2) TF-IDF (Ramos et al. 2003): It scores documents by multiplying the term frequency of each query term by the inverse document frequency. Second, we also (3) DPR. (Karpukhin et al., 2020): It uses a bi-encoder to retrieve relevant documents based on dense embeddings.
consider three dense retrievers:
(4) Contriever [Izacard et al., 2021 It leverages unsupervised contrastive learning to learn high-quality dense representations.
(5) Dragon (Lin et al., (003): Tt enhances retriever training by employing data augmentation, including query and label augmentation.
4.2 Settings
API Access. In this paper, conduct a comprehensive evaluation over ACADEMICEVAL benchmark using the LLM API provided by together.a For each API call, we fix the temperature parameter to 0 (i.e., greedy decoding).
Input Truncation. By default, we use a BERT tokenizer to calculate the number of input tokens for ACADEMICEVAL. However, since the tokenizer of each LLM is usually different, will cause some inputs exceed the context length limit of the L.M. Therefore, for the evaluation of each LLM,, we additionally download its tokenizer configuration file from the official website at Hugging Face, which is utilized ensure correct and accurate truncation of input tokens.
Refinement of LLM Responses. For the TITLE WRITING task, the responses of LLMs are relatively short. If the response contains some extra redundant information, it will have a greater impact on the evaluation metric score (although we have given LLM instructions not to generate irrelevant information). Therefore, for the TITLE WRITING task, we additionally refine the L.M responses, for example, removing irrelevant information such as "here the title". For other tasks, since LLM's responses are relatively long, occasional small amounts of irrelevant information will not have a significant. impact on the evaluation, we do not perform any refinement LLM's responses in this case.
Details of the Implementation of RALM. We use the inputs of ACADEMICEVAL as the external corpus of RALM (such as Target Content and Reference Content introduced in Section D). For lext split, we use the ReeursiveCharacterIesee Sextpptteee from LangChair and set chunk size and chunk overlap to 512 and 64, respectively. For each retrieval, we recall up to 12 text chunks (limited by the context length of standard LLMs) based on text similarity (semantic similarity based on inner product for dense retrievers or similarity based word frequency for sparse retrievers).
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Main Results on AcademicEval w.r.t. BERTScore.
Standard LLMs
Long-context LLMs
RALM
Models
Gemma
LLaMA
Qwen
Mixtral
Hermes
Gemmal
LLaMA
#Params.
7B
70B 8K
72B
8x7B 32K
8x7B 32K
7B
70B
Context Length
8K
32K
8K
8K
Setting: Title Writing
TITLE-10K
66.1
74.1
73.9
73.4
74.2
65.8
73.9
TiTLE-30K
73.0
72.9
73.4
65.7
73.9
TITLE-31K-G
72.8
72.8
73.3
65.7
73.8
Setting: Abstract Writing
ABs-9K
59.9
62.4
62.5
61.4
62.2
60.3
61.5
ABS-28K
61.3
61.2
62.6
60.1
61.4
ABs-29K-G
61.3
61.4
62.5
60.2
61.3
Setting: Introduction Writing
INTRO-8K
54.8
55.8
55.4
54.6
55.2
55.0
55.2
INTRO-28K
54.8
54.0
54.8
55.0
55.2
INTRO-28K-G
54.9
54.1
54.7
55.0
55.3
Setting: Related Work Writing
RELATED-34K
52.0
56.2
58.5
55.3
57.8
52.4
54.7
RELATED-53K
52.4
54.7
RELATED-53A0 -3K-G
52.4
54.8
Bold indicates the highest score in each row.
denotes augmentation with retriever (Default: Contriever).
4_" means that the context length is too long to be fed into LLMs.
4.3
Automatic Metric Evaluation
4.3.1
Evaluation Setup
For automatic evaluation metrics, adopt (1) BERTScore" Zhang al. 2019 This metric leverages
BERT-bascr embedding to measure semantic similarity between predicted and reference texts. (2) ROUGE- L [Li 2004 This metric evaluates the longest common subsequence between the generated and reference texts, providing a measure of similarity in terms sequential matching. For both metrics, higher scores indicate a better match between the predicted and the reference text.
4.3.2 Result Analysis
We conduct comprehensive experiments the four academic writing tasks, and the results w.r.t. BERTScore and RougeL. are presented in Table 3 and respectively. Note that we do not conduct experiments on -M settings because its context length is too long for most of our selected baselines.
Diverse Task Difficulties and Abstractions. The four tasks we proposed are designed to challenge LLMs over long-context generation tasks with different abstraction levels. From Table and 4 we can clearly observe that provides different difficulties for LLMs to perform well from TITLE WRITING to RELATED WORK WRITING tasks, and the results of all baselines on these four tasks have relatively obvious trend.
httpps:/ wwtp://wwwwoooo http://wwwwww...a.....ooooo httpps: www.wwangchain.coo We deberta-xlarge mlll He have the best. correlation with human evaluation.
2021 instead of the default roberta-large large Liu
2019 the backbone model

---

Published in Transactions on Machine Learning Research (10/2025)
Table 4: Main Results on AcademicEval w.r.t. RougeL.
Standard LLMs
Long-context LLMs
RALM
Models
Gemma
LLaMA
Qwen
Mixtral
Hermes
Gemma
LLaMA
#Params.
7B
70B
72B
8x7B
8x7B
7B
70B
Context Length
8K
8K
32K
32K
32K
8K
8K
Setting: Title Writing
TITLE-10K
44.5
47.1
44.2
45.2
46.2
42.7
47.3
TITLE-30K
44.5
44.6
45.9
42.6
47.3
TITLE-31K-G
44.2
44.4
45.3
42.5
47.0
Setting: Abstract Writing
ABs-9K
22.4
25.0
24.3
24.1
26.1
23.4
24.2
ABs-28K
23.3
24.7
26.6
23.1
24.1
ABS-29K-G
23.3
24.9
26.6
23.2
24.0
Setting: Introduction Writing
INTRO-8K
14.9
18.1
16.2
17.2
17.8
15.4
17.9
INTRO-28E
16.3
17.5
17.5
15.3
17.8
INTRO-28K-G
16.3
17.5
17.5
15.4
17.8
Sctting: Related Work Writing
RELATED-34K
13.5
14.9
16.0
13.4
15.1
14.1
15.3
RELATED-53K
14.0
15.3
RELATED-53K-G
14.0
15.2
Bold indicates the highest score in each row.
denotes augmentation with retriever (Default: Contriever).
4_" means that the context length is too long to be fed into LLMs.
For example, the TITLE WRITING task lends to have higher score than the ABSTRACT WRITING task, which may indicate that the TITLE WRITING task is easier than the ABSTRACT WRITING task. Since a title only has a few words, LLMs only need to generate a roughly related theme to achieve a high semantic similarity, while an abstract requires a more detailed description to achieve it.
Baseline Performance Comparison. Across automatic metrics, RALM with LLaMA frequently at- tains the highest scores in multiple settings (e.g., TITLE-30K /3KK/ INTRO-28K (OOO-28K//SKKG, and RELATED- 53K/53K-GG despite using 8K input window. Standard LLMs remain competitive and long-context LLMs (e.g., Qwen, Hermes) lead in some settings (e.g., RELATED-34K on BERTScore). This exposes the shortcomings of long-context LLMs' generation capabilities, which are well revealed by ACADEMICE- VAL. Among long-context LLMs, Hermes performs best overall, but still slightly inferior to RALM with LLaMA. This shows that although the current long-context LLMs have a longer context window size, they still have great deficiencies processing long text information. Overall, RALM often has an edge under au- tomatic metrics, likely because retrieval concentrates salient content into shorter chunks, thereby maximizing overlap-oriented scores.
Impact of Context Length. The impact of context length on performance is evident across all task settings and both metrics, with baselines often performing worse as the context length increases, though the extent model- and task-dependent. For example, the TITLE WRITING task shows a noticeable drop in scores as the context length extends from 10K 31K tokens. This trend also apparent in ABSTRACT WRITING and INTRODUCTION WRITING, where longer contexts correlate with decreased model performance, showing that benchmark challenges L.I.Ms in effectively processing ultra-long inputs.
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table Additional results on AcademicEval w.r.t.
LLM-as-a-Judge win rate (%).
Standard LLMs
Long-context LLMs
RALM
Models
Gemma
LLaMA
Qwen
Mixtral
Hermes
Gemma!
LLaMA
#Params.
7B
70B
72B
8x7B
8x7B
7B
70B
Context Length
8K
8K
32K
32K
32K
8K
8K
Setting: Title Writing
TITLE-10K
45.7
42.7
63.2
43.1
72.0
50.0
43.9
TrTLE-30K
54.5
45.6
62.5
47.5
44.9
TITLE-3IK-G
52.4
62.7
45.4
47.7
43.7
Setting: Abstract Writing
ABs-9K
12.0
55.5
77.0
70.0
61.1
14.3
43.2
ABs-28K
72.7
66.1
41.2
12.7
42.0
ABs-29K-G
71.0
65.9
40.7
12.0
43.9
Setting: Introduction Writing
INTRO-8K
34.6
63.2
79.3
61.5
58.0
48.8
64.1
INTRO-288
70.3
60.1
56.9
46.5
62.9
INTRO-28K-G
70.9
61.9
59.3
48.2
63.9
Setting: Related Work Writing
RELATED-53K 55.9
-GLATED-34K
91.9
91.2
65.6
88.6
71.3
89.8
72.5
90.7
RELATED-53K-6
71.7
90.2
Bold indicates the highest score in each row.
denotes augmentation with retriever (Delault: Contriever).
means that the context length is too long to be fed into I.L.s.
Impact of Few-shot Demonstrations, From Table and we can observe that the integration of few- shot demonstrations yields mixed effects: in several settings it is neutral or slightly negative under automatic metrics, yet correlated demonstrations can produce small but consistent gains for certain model task pairs. This shows that current LLMs cannot exploit long few-shot demonstrations benefit the target tasks well, emphasizing the importance of evaluating long in-context learning in LLM benchmarks. In addition, we can also find that few-shot demonstrations from co-author papers generally have more positive impact task performance than randomly selected ones.
4.4
LLM-as-a-Judge Evaluation
4.4.1
Evaluation Setup
To complement automatic metrics, we further incorporate an LLM-as-a-Judge evaluation capture higher-level qualitative aspects beyond semantic overlap. Specifically, employ the open-source Mixtral- 1.22B-Insstruct-ve (Jiang et al. 2024) to assess five dimensions of generation quality: (1) Novelty the degree to which the content introduces new and mcaningful idcas; (2) Feasibility - the plausibility and practicality of the described methods or claims; (3) Consislency the internal logical coherence of the out- put; (4) Factuality- - the correctness of factual statements; and (5) Academic Style the alignment with conventions of scholarly writing, enabling a more muanced evaluation of LLM outputs. For each task, we report the win rate (%), i.e., the percentage of cases where the generated text preferred over the reference according to the LLM judge. The detailed I.LM-as-a-Judge prompt be found in Appendix D
11

---

Published in Transactions on Machine Learning Research (10/2025)
25
25
64
Contriever
IBM25
Contriever
BM25
62
IFF-IDF
23
62
TF-IDF
23
BERT Score
60
Dragon
60
Dragon
21
Rouge
58
58
56
54
17
17
52
52
50
15
50
15
BERT Score
Rouge
BERT Score
Rouge
Gemma Instruct (7B)
LLaMA-3 Chat (70B)
Figure 3: Analysis of RALM Abs-9K. The left figure shows results with Gemma Instruct (7B), while the right one shows results with LLaMA-3 Chat (70B).
4.4.2 Result Analysis
We report results for the overall preference in Table 5 Compared BERTSscore and ROUGE-L, the LLM- as-a-Judge evaluation reveals partially different patterns, reflecting that it targets broader qualitative aspects beyond lexical or semantic overlap.
TITLE WRITING. Under TrTLE-10K, Hermes achieves the highest win rate (72.0), while Mixtral becomes the top model under the correlated setting TITLE-31K-G (62.7). This suggests that (1) short, highly abstract outputs benefit from strong style and concision (favoring Hermes at 10K); and (2) correlated contexts can help certain models (e.g.. Mixtral) at longer lengths when the judge considers qualities beyond surface similarity. Notably, RALM variants (Gemmat, LLaMAt) are not consistently preferred in the title task, indicating that aggressive retrieval does not always align with judged title quality.
ABSTRACT WRITING. Qwen attains the highest win rate at ABS-9K (77.0) and remains strong at longer lengths (ABs-28K: 72.7; ARs-29K-G: 71.0). Mixtral follows closely, while RALM variants trail in preference. This contrasts with automatic metrics where RALM often ranks highly, suggesting that, for abstracts, the judge values holistic qualities (coherence, feasibility, academic style) over high overlap with references.
INTRODUCTION WRITING. Qwen consistently leads (INTRO-SK: 79.3; INTRO-28K: 70.3; ANTRO-28K-G: 70.9), and Hermes improves slightly with correlated contexts (56.9 - 59.3). LLaMAt remains competitive but is not lop-ranked. Overall, correlated few-shol demonstrations offer modest gains for some models, sup- porting that graph-informed contexts can help introductions when evaluated on broader quality dimensions.
RELATED WORK WRITING. In contrast to the above tasks, RALM models (espocially LLaMAt) achieve top preferences longer lengths (RELATED-53K: 90.7; RELATED-53K-GG G: 90.2), aligning with the intuition that retrieval particularly beneficial for RELATED WORK, where judged quality rewards appropriate citations, prior studies, and domain-specifie terminology.
Takeaways. (1) The judge-based preferences are not dominated by RALM across all lasks; instead, pref- erences depend on task nature and qualitative dimensions. (2) Correlated contexts can yield improvements in several settings (e.g., Mixtral on TITLE-31--G, Hermes on INTRO-28K-G), though gains are model- dependent. (3) The divergence from automatic metrics underscores their complementarity: automatic met-
rics reward overlap, whereas the judge emphasizes higher-level writing quality.
4.5 Discussion
Additional Analysis on RALM. We conduct extensive experiments on RALM on the ABs-9K setting using standard LLMs Gemma Instruct (7B) and LLaMA-3 Chat (70B), and the results are presented in We find that the performance of dense retrievers consistently outperforms sparse retrievers,
Figure 3
12

---

Published in Transactions on Machine Learning Research (10/2025)
Table 6: Title-only ablation on Abstract Writing (Abs-9K). The clear degradation indicates genuine
reliance on external context rather than in-weights memorization.
Setting
Model
BERTScore
ROUGE-L
Default (Abs-9K)
LLaMA
62.4
25.0
Title-only (Abs-9K)
LLaMA
57.4
18.8
Default (Abs-9K)
Hermes
62.2
26.1
Title-only (Abs-9K)
Hermes
56.7
19.3
among which contriever achieves the best results. This is because the summary generation task emphasizes semantic similarity, which can be well measured by the similarity dense embeddings. However, the sparse retrievers perform text chunk recall based on sparse embeddings, and the results are significantly worse than those of the dense retricvers.
Understanding the Performance Plateau of AcademicEval. The performance plateau observed longer contexts (e.g., 9K~30K) invites further examination of its underlying causes. While our analysis attributes this plateau partly to the limited ability of current models to utilize ultra-long inputs through in-context learning (ICL), another plausible factor lies in in-weights learning (IWL) (Chan et al., 2024). That is, certain academic knowledge may already be internalized during pretraining. In such cases, adding more context brings diminishing informational returns even when the benchmark itself remains well- constructed.
To better understand this phenomenon, we analyze both structural and empirical evi- dence. Structurally, ACaDEMICEVAL organizes tasks across hierarchical abstraction levels WORK), where deeper contextual reasoning becomes increasingly essential. Platcaus may thus occur when longer inputs introduce redundancy rather than new cues, suggesting ICL saturation instead of pure memorization. Empirically, we conduct a Title-only ablation on the ABSTRACT WRETING task under ABs-9K, where most contextual information is removed except. for the paper title. As shown in Table 6 the BERTScore and ROUGE-L of both LLaMA and Hermes drop sharply (by 5-7 points), confirming that model performance depends strongly the provided context and is not solved by IWL alone.
Overall, our evidence indicates that the observed plateau on ACADEMICEVAL is primarily driven by imperfect long-contert utilization (ICL limitation), rather than by IWL. This reading is supported by the Title-only ablation, where removing most contextual information yields substantial drops in both BERTScore and ROUGE-L, indicating strong dependence on the provided context. Morcover, since ACADEMICEVAL is designed as live-updating benchmark that continuously incorporates newly published arXiv papers via the co-author graph, the evaluation set evolves over time and reduces the likelihood that performance is dominated by pre-encoded (in-weights) knowledge. While diminishing informational returns occur when additional tokens introduce redundancy, ACADEMICEVAL serves as a diagnostic lens showing that the platcau chiefly reflects current models' limited ability to exploit ultra-long inputs under realistic, evolving conditions.
This discussion also aims to inspire further reflection in the long-context benchmarking community on how dataset design and periodic updates can better disentangle ICL and IWL effects, while underscoring the continued importance of mitigating data leakage in future benchmark construction.
5 Conclusion
In this paper, we propose ACADEMICEVAL, live long-context LLM benchmark for evaluating long-context generation tasks with hierarchical abstraction levels. ACADEMICEVAL adopts arXiv as the data source and introduces several long-context academic writing tasks without manual annotation since the papers on Xiv can be regarded as original, high-quality, and expert-curated labels. Morcover, we integrate few-shot demonstrations from collected co-author graph to make the context length of our benchmark flexible and scalable. An efficient live evaluation is also designed to make ACAdemICEvAl. immune to the label leakage
13

---

Published in Transactions on Machine Learning Research (10/2025)
issue and move toward more fair evaluation. In the experiments, conduct comprehensive analysis ACADEMICEVAl using several LLM baselines, and the results show that ACAdEMICEVAl. is challenging long-context LLM benchmark. Insightful findings are also elucidated for potentially strengthening the long- context modeling capabilitics of LLMs and inspiring future long- long-context LLM benchmarks to evaluate LLMs more flexibly and holistically.
Acknowledgments
We sincerely appreciate the support from Amazon grant funding project #120359, 'GRAG: Enhance RAG Applications with Graph-struchured Knowledge", and research gifts from Meta and Lenovo.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXie:2303.0877 2023.
Al@Meta. Llama 3 model card. 2024. URL
CARD. md
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint prXiv:2307.1008: 2023.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint rcXtt:2310.1151, 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023a.
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXix:2308. 14508, 2023b.
Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S Weld. Tidr: Extreme summarization of scientific documents. arXiv preprint arXi::200,.1001, 2020.
Bryan Chan, Xinyi Chen, András György, and Dale Schuurmans. Toward understanding in-context vs. in-weight learning. arXiv preprint 2x.i:2410.230044 2024.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models retrieval- augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17754-17762, 2024.
Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xiangliang Zhang, Dongyan Zhao, and Rui Yan. Capturing relations between scientific papers: An abstractive model for related work section generation. Association for Computational Linguistics, 2021.
Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Rui Yan, Xin Gao, and Xiangliang Zhang, Target-aware abstractive related work generation with contrastive learning. In Proceedings of the ,5th international ACM SIGIR conference on research and development in information retrieval, pp. 373 1733-83,, 2022.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Scokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long docunents. ar preprint trXie:1804.05688 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiu preprint ar 1x0:::81000044400 2018.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Jialin Dong, Bahare Fatemi, Bryan Perozzi, Lin F Yang, and Anton Tsitsulin. Don't forget to connect! improving rag with graph-based reranking. arXiv preprinl or orx:2224 18414 2024.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.0025/ 2022.
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A comprehen- sive benchmark for evaluating long text modeling capacities of large language models. arXiv preprint arXiv:2309.134 13345, 2023.
Tao Feng, Pengrui Han, Guanyu Lin, Ge Liu, and Jiaxuan You. Thought-retriiver Don't 't just retrieve raw data, retrieve thoughts. 2024.
Pengeheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhann enhanced bert with
disentangled attention. In International Conference on Learning Representations, 2021. URL https:
//openreview.nen seet/fforumiidxPPlaaee
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. arXiv preprinl arxiv:2104.021 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. orXiv preprint arXiv:2112.09118 2021.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arxiv preprint ar Xix::2401. (0000000004088, 2024.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. orXiv preprint or 066:2305.06683, 2023.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Lodell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprinl arXiv:2004.0400 2020.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in nlp, 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199 22213, 2022.
Wojciech Kryéciński,, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint *xXi:2205500200, 2021.
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang, Loogle: Can long-context language models understand long contexts? arXiv preprint ar Xiv::331 1. 0.09999, 2023.
Jiaxi Li, Xingxing Zhang, Xun Wang, Xiaolong Huang, Dong, Liang Wang. Si-Qing Chen, Wei Lu, and Furu Wei. Wildlong: Synthesizing realistic long-context instruction data at scale. arXiv preprint arXiv:2502.1663. 2025.
Tianle Li, Ge Zhang, Quy Due Do, Xiang Yuc, and Wenhu Chen. Long-context 1mms struggle with long in-context learning. arXiv preprinl arXiv:2404. 02060, 2024.
Chin-Yer Lin. Rouge: A package for automatic evaluation of summaries. In Teat summarization branches out, pp. 74-81, 2004.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Sheng-Chicl Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. ar preprint arXie:2302.07/3
2023.
Marina Litvak and Natalia Vanetik. Query-based summarization using mdl principle. In Proceedings of the mulliling 2017 workshop on summarization and summary ecaluation across source types and genres, pp- 22 22-31, 2017.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized BERT pretraining approach. CoRR, abs/1907.11692 2019. URL http://arziv.ot http://rxiv..or///b/ http//bb/119./...
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Focrster, Jeff Clunc, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery, arXiv preprinl ar urXiv:2408.6620 2024.
Zhiyi Ma, Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potls, Adina Williams, and Douwe Kiela. Dynaboard: An evaluation-as---service platform for holistic next-generation benchmarking, 2021.
Rui Meng, Khushboo Thaker, Lei Zhang, Yue Dong, Xingdi Yuan, Tong Wang, and Daqing He. Bringing structure into summaries: faceted summarization dataset for long scientific documents. orXiv preprint arXiv:2106.0011 2021.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context votrieval-augmented language models. Transactions of the Association for Computa- lional Linguistics, 11:1316 1331, 2023.
Juan Ramos al. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference machine learning, volume 242, pp. 29 19-48. Citeseer, 2003.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. orXiv preprinl (xXi::2403.05590 2024.
Stephen Robertson, Hugo Zaragoza, al. The probabilistic relevance framework: Bm25 and beyond. Foundations and TrendsS in Information Retrieval, 3(4):333 389, 2009.
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matci Zaharia. Ares: An automated evaluation framework for retrieval-augmented generation systems. arXiv preprint arXiv:2311.09/70 2023..
Oscar Sainz, Jon Ander Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agiree NIp evaluation in trouble: On the need to measure 1lm data contamination for each benchmark. arXiv preprint xrx:::2310.18011 2023.
Cri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: zero-shot benchmark for long text understanding. arXiv preprint arXiv:2305. 196, 2023.
Gemma Team, Thomas Mesuard, Cassidy Hardin, Robert Dadashi, Surya Blupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint orXiv:2403.088 088295 2024.
Teknium, thocmozilla, karan4d, and art huemin. Nous hermes mixtral 8x7b dpo. URL [https: /rtggggggecceooooooooooeeeererra Haweass-Heeeeeeeeee hatprr/eeooooooeeomoaeeaaaaaaaaa 8xPB-DPP] (https: btttps://hugginggace.co Nouskkeearcc//oons-HHaaee---aaaa
16

---

Published in Transactions on Machine Learning Research (10/2025)
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. or arXiv preprint orXiv:222.. 10000, 2022.
Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R Bowman. Squality: Building long-document summarization datasct the hard way. arXiv preprint orXh:2205.11465, 2022.
Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Qingsong Wen, Wei Ye, et al. Autosurvey: Large language models can automatically write surveys. Advances in neural informalion processing systems, 37:115119-115443 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint Xiv:2200 0 07682, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-offthought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 248477 2022b.
Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang Jindong Wang, Yue Zhang, and Linyi Yang. Cycleresearcher: Improving automated research via automated review. arXiv preprint arXiv:2411.0088 2024.
Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis N. Toannidis, Karthik Subbian, James Zou, and Jure Leskovec. Stark: Benchmarking Ilm retrieval on textual and relational knowledge bases, 2024.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. arXiv preprint arXiv:2310.03095, 2023.
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models, 2024.
Wentao Ye, Jiaqi Hu, Liyao Li, Haobo Wang, Gang Chen, and Junbo Zhao. Data contamination calibration for black- box Ilms, 2024.
Haofci Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, and Jiaxuan You. Researchtown: Simulator of human research communily. arXiv preprinl 4r 17767, 2024.
Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. Augmentation-adaated retriever improves generaliza- tion of language models as generic plug-in. arXiv preprinl arXi::2305.123: 2023.
Haozhen Zhang, Tao Feng, and Jiaxuan You. Graph of records: Boosting retrieval augmented generation for long-context summarization with graphs. arXiv preprint orXiv:2410.11001, 2024a.
Tianyi Zhang, Varsha Kishore, Felix Wa, Kilian Q Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. arXiv preprint
2019.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Lin, et al. Extending long context evaluation beyond 100k tokens. arXiv preprint ar Xiv:2440. 13718, 2024b.
Chenghao Zhu, Nuo Chen, Yufei Gao, and Benyou Wang. Evaluating llms at evaluating temporal general- ization, 2024a.
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations, 2024b. URL http:///pppeeeeeeeeeee///oddddds
17

---

Published in Transactions on Machine Learning Research (10/2025)
A API cost
We adopt LLM API provided by together.a to conduct experiments in this paper. API costs mainly come from evaluating the test set of VCA,EMICEVAL, which are estimated to be around $300.
B Limitation and Future Improvement
ACADEMICEVAL. is live benchmark without label leakage, which leverages co-author papers from collected co-author graph as few-shot demonstrations to make the context length flexible and scalable, ACADEMICE- VAL adopts arXiv as its data source without the need for manual labeling, and the content of the papers on it can naturally serve as high-quality and expert-curated annotations.
However, ACADEMICENAL still has some limitations:
Task Diversity. ACADEMICEVAL currently has only four academic writing tasks, which limits the task diversity.
Independent Evaluation of the Paper Section. In ACADEMICEVAL, independently evaluate the section content extracted from a paper, which may lack comprehensive evaluation of the paper as whole.
Popularity Bias. ACADEMICEVAL. first collects a co-author graph from arXiv, which contains subset of all papers arXiv. Therefore, the collected papers may have some popularity bias. For example, most of the papers may come from few active authors, which will cause bias in the evaluation.
Based on the above limitations, our future improvements will include:
Introduce More Data Source. The goal of ACADEMICEVAL is to make context length flexible and scalable by using few-shot demonstrations and high-quality labels without manual annotation, so papers on arXiv are a more suitable data source. We will consider adding other websites as data sources in the future, such some crestion-answent websites (Stack Overflow or Reddit, etc.). In this case, we can use the best answers high-quality labels. By modeling the citation relationship between posts into graph, can also obtain few-shot demonstrations to enrich the context length.
K-fold Cross-validation. (or fold) as the label, the remaining sections as inpuls, and finally calculating the average of all leave-one-out evaluation scores.
We can use k-fold cross-validation for a paper, that is, leaving a section
Eliminate Popularity Bias. We will perform probabilistic sampling on papers when collecting the co-author graph and give a lower sampling probability to active authors to alleviate the impact of popularity bias.
C
Social Impact
The proposed benchmark ACADEMICEVAL. will promote the academic community's 's exploration of using
LLMs to automate academic writing tasks (Wang et al. 2024 Lu et al, 2024 Weng et al. 2024). Here are some key points highlighting its significance:
Efficiency and Productivity. LLMs can drastically reduce the time and effort required for various academic writing tasks. These tasks include drafting papers, writing literature reviews, summarizing research articles, and generating bibliographies. By automating these processes, researchers can focus more on high-level thinking, experimentation, and analysis.
"https///www.togeehhraaa/
18

---

Published in Transactions on Machine Learning Research (10/2025)
Enhanced Writing Quality. LLMs have the ability to produce coherent and grammatically correct text, which can improve the overall quality of academic writing. They can assist in refining arguments, improving clarity, and ensuring consistency in style and tone, which is particularly useful for non-native English speakers.
Support for Multidisciplinary Rescarch. Given their training on diverse topics, LLMs can assist researchers in exploring interdisciplinary approaches by providing information and generating content across various fields of study. This can foster innovation and encourage collaboration between different academic disciplines.
LLM Prompts
In this section, we present the LLM prompts used in the experiments, including TITLE WRITING, ABSTRACT WRITING, INTRODUCTION WRITING, and RELATED WORK WRITING. For each academic writing task, we provide prompts for standard LLMs, long-context LLMs, and RALM (RALM additionally includes the retrieval query).
D.1 LLM Prompts for Title Writing
Prompt for Standard and Long-context LLMs on TrTLE-10K
Please read the following Target Content carefully and summarize the Target Content required. # # Target Content: {CONTENT}
# # Target Content Abstract: (ABSTRACT)
Please craft a title highly summarizing the main theme from the above provided Target Content. The title should be of appropriate length (strictly limited to about 10 words). The title should also include and highlight the core and most critical theme of the Target Content, ignoring minor and redundant information. Please clear and concise manner. Please output the title directly without including other redundant irrelevant text.
that the title captures the essence of the Target Content in
Prompt for Standard and Long-conted LLMs on TITLE-30K and TITLE-3IK-G
Please read the following Reference Content and Output carefully and summarize the Target Content
as required.
# # # Reference Content O: (CONTENT 0}
# # # Reference Abstract O: (ABSTRACT
# # # Reference Output O: (OUTPUT_D)
# # Target Content: (CONTENT)
# ### Target Content Abstract: (ABSTRACT)
Please craft a title highly summarizing the main theme from the above provided Target Content. The Reference Content and Output provide some demonstrations, which may also contain some information that potentially related the Target Content. You can refer the input and output text forms of the Reference Content and Output assist in summarizing the Target Content and try to explore and use the information that is potentially related to the Target Content contained in the Reference Content and Output. The title should be of appropriate length (strictly limited to about 10 words). The title should also include and highlight the core and most critical theme of the Target Content, ignoring minor and redundant information. Please ensure that the title captures the essence of the Target Content in a clear and concise manner. Please output the title directly without including other redundant or irrelevant text.
19

---

informa- Published in Transactions on Machine Learning Research (10/2025)
Prompt for RALM on TrTLE-1OK, TFTLE-30K, and TTTLE-31K-G
Please read the following Target Content carefully and summarize the Target Content as required.
### # Target Content O: {CONTENT 0}
# ### # Target Content 1: {CONTENT
# # Target Content Abstract: {ABSTRACT)
Please craft a title highly summarizing the main theme from all the above provided Target Contents. The title should be appropriate length (strictly limited to about 10 words). The title should also include and highlight the core and most critical theme of the Target Contents, ignoring minor and redundant information. Please ensure that the title captures the essence of the Target Contents in clear and concise manner. Please output the title directly without including other redundant. irrelevant text.
Retrieval Query for RALM on TITLE-1K,, TITLE-30K, and TITLE-31K 3
Please craft a title highly summarizing the main theme of the provided text. The abstract of the text is: {ABSTRACT]
D.2 LLM Prompts for Abstract Writing
Prompt for Standard and Long-context LLMs on ABS-9K
Please read the following Target Content carefully and summarize the Target Content required.
# ### Target Content: {CONTENT)
### Target Content Title: (TITLE)
Please craft an abstract summarizing the key points from the above provided Target Content. The abstract should be of appropriate length (around 200 words) and include the main theme, significant findings or arguments, and conclusions of the Target Content. Please ensure that the abstract captures the essence of the Target Content clear, coherent, and suceinct manner. Please output the abstract directly without including other redundant or irrelevant text.
Prompt for Standard and Long-context LLMs on ABS-28K and ABS-29K-G
Please read the following Reference Content and Output carefully and summarize the Target Content
required.
# ### Reference Content O: {CONTENT 0}
# # # Reference Title O: (TITLE 0}
# # # Reference Output O: {OUTPUT_O)
# # # Target Content: {CONTENT)
# # Target Content Title: (TITLE)
Please craft an abstract summarizing the key points from the above provided Target Content. The demonstrations, which may also contain tion that is potentially related to the Target Content. You can refer to the input and output text forms of the Reference Content and Output to assist in summarizing the Target Content and try to explore and use the information that is potentially related to the Target Content contained in the Reference Content and Output. The abstract should be of appropriate length (around 200 words) and include the main theme, significant findings or arguments, and conclusions of the Target Content. Please ensure that the abstract captures the essence of the Target Content in clear, coherent, and succinct manner. Please output the abstract directly without including other redundant irrelevant text.
Reference Content and Output provide
20

---

Published in Transactions on Machine Learning Research (10/2025)
Prompt for RALM on A 10K, ABS-30K, and ABs-31K-G
Please read the following Target Content carefully and summarize the Target Content as required.
### # Target Content O: {CONTENT 0}
# # # Target Content 1: {CONTENT
# # Target Content Title: (TITLE)
Please craft an abstract sunmarizing the key points from all the above provided Target Contents. The abstract should be of appropriate length (around 200 words) and include the main theme, significant findings or arguments, and conclusions of the Target Contents. Please ensure that the abstract captures the essence of the Target Contents in clear, coherent, and succinct manner. Please output the abstract directly without including other redundant or irrelevant text.
Retrieval Query for RALM on ADs-10K, Ans-30K, and ADs-31K-G
Please craft an abstract summarizing the key points of the provided text. The Litle of the text is: (TITLE)
D.3 LLM Prompts for Introduction Writing
Prompt for Standard and Long-context LLMs on INTRO-8K
Please read the following Target Content carcfully and summarize the Target Content as required.
# # Target Content: {CONTENT)
### Target Content Title: (TITLE)
# ### Target Content Abstract: {ABSTRACT)
Please craft an introduction summarizing the key points from the above provided Target Content. The introduction should be of appropriate length (about 1000 to 1500 words). The introduction should first describe the topic main theme of the Target Content, then provide relevant background knowledge, and summarize the existing relevant research on this topic from the Target Content, point out their advantages and disadvantages, and highly summarize the specific research problem and problem statement targeted by the Target Content. Next, describe in detail the core approach or insights proposed by the Target Content on this topic and include any necessary experimental results. Then, use about short paragraphs (each paragraph is about 50 words) highly summarize the approach or insights proposed in the Target Content, as well as the experimental results. Finally, briefly give an overview of the Target Content's structure. Please ensure that the introduction captures the essence of the Target Coutent in clear, coherent, and succinet manner. Please output the introduction directly without including other redundant or irrelevant text.
21

---

Published in Transactions on Machine Learning Research (10/2025)
Prompt for Standard and Long-conlext LLMs on INTRO-28K and INTRO-28K-G
Please read the following Reference Content and Output carefully and summarize the Target Content
88 required.
### # Reference Content O: {CONTENT
### Reference Title O: (TITLE 0}
# ### Reference Abstract O: {ABSTRACT 0}
## Reference Output O: (OUTPUT 0}
### Target Content: (CONTENT)
# # Target Content Title: (TITLE)
### Target Content Abstract: (ABSTRACT)
Please craft introduction summarizing the key points from the above provided Target Content. The Reference Content and Output provide some demonstrations, which may also contain some information that is polentially related to the Targel Content. You can refer to the input and output text forms of the Reference Content and Output to assist in summarizing the Target Content and try to explore and use the information that is potentially related to the Target Content contained in the Reference Content and Output. The introduction should be of appropriate length (about 1000 to 1500 words). The introduction should first describe the topic main theme of the Target Content, then provide relevant background knowledge, and summarize the existing relevant research on this topic from the Target Content, point out their advantages and disadvantages, and highly summarize the specific research problem and problem statement targeted by the Target Content. Next, describe detail the core approach or insights proposed by the Target Content on this topic and include any necessary experimental results. Then, use about 3 short paragraphs (each paragraph is about 50 words) highly summarize the approach or insights proposed in the Target Content, well as the experimental results. Finally, briefly give overview of the Target Content's structure. Please ensure that the introduction captures the essence of the Target Content in clear, coherent, and succinet manner. Please output the introduction directly without including other redundant irrelevant text.
Prompt for RALM on INTRO-8K, INTRO-28KE and WNTRO-28K-<
Please read the following Target Content carefully and summarize the Target Content as required.
# # # Target Content O: {CONTENT 0}
# ###
Target Content 1: (CONTENT 1)
# # Target Content Title: (TITLE)
# ### Target Content Abstract: (ABSTRACT)
Please craft an introduction summarizing the key points from all the above provided Target Contents. The introduction should be of appropriate length (about 1000 to 1500 words). The introduction should first describe the topic main theme of the Target Contents, then provide relevant background knowledge, summarize the existing relevant research on this topic from the Target Contents, point out their advantages and disadvantages, and highly summarize the specific rescarch problem and problem statement largeted by the Target Contents. Next, describe in detail the core approach or insights proposed by the Target Contents on this topic and include any necessary experimental results. Then, use about 3 short paragraphs (cach paragraph is about 50 words) to highly summarize the approach insights proposed in the Target Contents, well as the experimental results. Finally, briefly give an overview of the Target Contents' structure. Please ensure that the introduction captures the essence of the Target Contents in clear, coherent, and succinct manner. Please output the introduction directly without including other redundant or irrelevant text.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Retrieval Query for RALM on INTRO-8K, INTRO-28K, and INTRO-28K-G
Please craft an introduction summarizing the main theme the provided text (including background knowledge, advantages and disadvantages of existing research and challenges, the proposed approach, experimental results, etc.). The title of the text {TITLE}. The abstract of the text is {AB- STRACT).
D.4 LLM Prompts for Related Work Writing
Prompt for Standard and Long-context LLMs on RELATED-34K
Please read the following Target Content and Target Citations carefully and summarize the Target Citations according the topic of the Target Content as required.
### Target Citation O:
Target Citation Title: {C_ TITLE_ 0}
Targel Citation Abstract: {C ABSTRACT 0)
### Target Content: (CONTENT)
# # Target Content Title: (TITLE)
### Target Content Abstract: (ABSTRACT)
Given the Target Content and its Abstract and Title, along with its Target Citations (including Target Citation Title and Abstract), please craft related work summarizing the key points from the above provided Target Citations. There is specific length requirement or limit for the entire related work (it is best to keep it around 500 to 1000 words), but each Target Citation that appears in the related work needs to be highly summarized in extremely concise and short sentences. You can refer to the topic or main theme described by the Target Content and its Abstract and Title to filter irrelevant information the Target Citations and leverage relevant information. Furthermore, you can categorize the relevant Target Citations, briefly summarize the advantages and disadvantages of each calegorization, and explain the advantages of the approach proposed in the Target Content. Please ensure that the related work captures all the relevant key points of the Target Citations in clear, coherent, and succinct manner. Please output the related work directly without including other redundant or irrelevant text.
23

---

Published in Transactions on Machine Learning Research (10/2025)
Prompt for Standard and Long-conlext LLMs on RELATED-53K and RELATED-53K-G
Please read the following Reference Content and Output carefully and summarize the Target Citations according to the topic of the Target Content as required.
### # Reference Content O: {CONTENT 0)
### Reference Title O: (TITLE 0}
### Reference Abstract O: (ABSTRACT 0}
# ## Reference Output 0: (OUTPUT 0}
# # # Target Citation O:
Target Citation Title: {C TITLE 0}
Target Citation Abstract: {C ABSTRACT 0)
### Target Content: :CONTENT)
### Target Content Title: (TITLE)
# # Target Content Abstract: (ABSTRACT)
Given the Target Content and its Abstract and Title, along with its Target Citations (including Target Citation Title and Abstract), please craft related work summarizing the key points from the above provided Target Citations. The Reference Content and Output provide some demonstrations, which may also contain some information that is potentially related to the Target Content. You can refer to the input and output text forms of the Reference Content and Output to assist in summarizing the Target Citations and try to explore and use the information (e.g., related citations missing from the Target Citations) that is potentially related to the Target Content contained in the Reference Content and Output. There no specific length requirement or limit for the entire related work (it is best keep it around 500 1000 words), but each Target Citation that appears in the related work needs to be highly summarized in extremely concise and short sentences. You refer to the topic main theme described by the Target Content and its Abstract and Title to filter irrelevant information in the Targel Citations and leverage relevant information. Furthermore, you can categorize the relevant Target Citations, briefly summarize the advantages and disadvantages of each categorization, and cxplain the advantages the approach proposed the Target Content. Please ensure that the related work captures all the relevant key points of the Target Citations clear, coherent, and succinet manner. Please output the related work directly without including other redundant or irrelevant text.
Prompt for RALM on RELATED-34K, RELATED-53K, and RELATED-53K-G
Please read the following Target Content and Target Citations carefully and summarize the Target Citations according the topic of the Target Content as required.
# # # Target Content O: (CONTENT
### Target Content 1: {CONTENT
# # Target Content Title: (TITLE)
# # Target Content Abstract: (ABSTRACT)
Given the Target Content Abstract and Title, please craft a related work summarizing the key points from all the above provided Target Contents. There is no specific length requirement. limit for the entire related work (it is best to keep it around 500 1000 words), but each Target Content that appears in the related work needs to be highly summarized in extremely coneise and short. sentences. You refer to the topic main theme described by the Target Content Abstract and Title to filter irrelevant information in the Target Contents and leverage relevant information. Furthermore, you can calegorize the relevant. Target Contents and briefly summarize the advantages and disadvantages of each categorization. Please ensure that the related work captures all the relevant key points of the Target Contents in a clear, coherent, and succinet manner. Please output the related work directly without including other redundant or irrelevant text.
24

---

Published in Transactions on Machine Learning Research (10/2025)
Retrieval Query for RALM on RELATED-34K, RELATED-53K, and RELATED-53K-G
Please craft a related work summarizing all the relevant key points of the provided text. The title of the text is (TITLE). The abstract of the text is (ABSTRACT).
25

---

Published in Transactions on Machine Learning Research (10/2025)
D.5 LLM Prompts for LLM-as-a-
Prompt for LLM-as-a-Judge
Role-
You are an expert scientific writing evaluator with strong expertise research methodology, academic writing, and factual reasoning.
Goal
You will evaluate two generated scientific contents for the same research topic based on five key criteria that capture quality beyond surface similarity.
Evaluation Criteria
1. Novelty
2. Feasibility
3. Consistency
4. Factuality
5. Academic Style
Judgment
For each criterion, decide which answer better (Answer or Answer 2), and give a **bricf explanation**
Your evaluation must be based purely the content provided and your scientific reasoning, avoiding any bias toward the answer order.
Finally, provide an overall winner that best meets the criteria holistically.
Here are two generated scientific texts to be compared:
Answer 1: {answerl]
Answer 2: {answer2]
Evaluate them according to the **five criteria** defined above:
1. Novelty
2. Feasibility
3. Consistency
4. Factuality
5. Academic Style
For each criterion, decide which answer is better and briefly explain why.
Output your evaluation strictly the following JSON format:
{L 'Novelty': {{ "Winner": "[Answer or Answer 2]", 'BriefExplanationns
'[max sentence
}}
'Feasibility": { "Winner":
"[Answer 1 Answer 2]¹, 'BriefExplanation' [max sentence
}}
"Consistency": {{ "Winner': "[Answer or Answer 2)', "BriieExxplanation"
[max sentence
}):
'Factuality": {E 'Winner": "[Answer 1 or Answer 2]', 'BriefExplanation':
"[max sentence }
"Academic Style": {{ 'Winner": "[Answer or Answer 2)', 'BrielExplanattions "[max sentence)
}}.
Overall Winner': {{ 'Winner:: "[Answer 1 or Answer 2)', "BriefExplanation"
[mmx 1 sentence
}}
}}
26

---

Published in Transactions on Machine Learning Research (10/2025)
E
Examples
In this section, we present some generation examples w.r.t. TITLE WRITING, ABSTRACT WRITING, and INTRODUCTION WRITING tasks.
E.1 Title Writing
Comparison between Generated Title and Ground Truth TITLE WRITING Task
Ground Truth:
When to Trust LLMs: Aligning Confidence with Response Quality
Qwen 1.5 Chat (72B)
Generated Title (O-shot):
CONQORD: Reinforcement Learning for Confidence-Responie Alignment in Large Language Models
Generated Title (2-shot demonstrations from random selection):
CONQORD: Enhancing LLM Confidence Alignment for Improved Trustworthiness and Retrieval
Generated Title (2-shot demonstrations from co-author papers): CONQORD: Enhancing Confidence Calibration for Language Models via Reinforcement Learning
Mixtral-8x7B Instruct. (46.7B)
Generated Title (O-shot):
CONQORD: Aligning Confidence with Quality in Large Language Models
Generated Title (2-shot demonstrations from random selection): CONQORD: Reinforcement Learning for qoulidence-uualtyy Aligument LLMs
Generated Title (2-shot demonstrations from co-author papers): CONQORD: Aligning Confidence with Response Quality in LLMs
Nous Hermes 2 Mixtral 8x7B-DPO (46.7B)
Generated Title (O-shot):
Aligning Confidence with Response Quality in Large Language Models using Reinforcement Learning with Dual- mnal-Component Reward Function: CONQORD Approach
Generated Title (2-shot demonstrations from random selection):
Aligning Confidence with Response Quality in Large Language Models: Introducing CONQORD
Generated Title (2-shot demonstrations from co-author papers): Confidence Aligmment in Large Language Models via Reinforcement Learning: Approach for Trust.
A CONQORD
27

---

Published in Transactions on Machine Learning Research (10/2025)
Comparison between Generated Title and Ground Truth on TITLE WRETING Task
Ground Truth:
Distillation for Multilingual Information Retrieval
Qwen 1.5 Chat (72B)
Generated Title (0-shot):
Multilingual Translate-Distill: Enhancing Crogs-Lan Information Retrieval Models
Generated Title (2-shot demonstrations from random selection):
Multilingual Translate-Distill for Enhanced Cross-Language Information Retrieval
Generated Title (2-shot demonstrations from co-author papers): Multilingual Translate-Distil for Effective Multilingual Information Retrieval
Mixtral-8x7B Instruct (46.7B)
Generated Title (0-shot):
Multilingual Trasslate-Distill for Cross-Language Information Retrieval
Generated Title (2-shot demonstrations from random selection): Multilingual Translate-Distill for Cross-Language Information Retrieval
Generated Title (2-shot demonstrations from co-author papers): Multilingual Trasslate-Distill for MLIR: Extending Translate-Distill for Robust Language Mixing
Nous Hermes 2 Mixtral 8x7B-DPO (46.7B)
Generated Title (O-shot):
Multilingual Translate-Distilll Enhancing Multilingual Information Retrieval with Translation and Distillation
Generated Title (2-shot demonstrations from random selection):
Multilingual Dratslate-Distill:: Efficient Training for Multilingual Information Retrieval
Generated Title (2-shot demonstrations from co-author papers): Multilingual Trasslate-Distill:: Training Dual-Encoder Models for Multilingual Information Retrieval
28

---

Published in Transactions on Machine Learning Research (10/2025)
E.2
Abstract Writing
Comparison between Generated Abstract and Ground Truth on ABSTRACT WRITING Task
Ground Truth:
In recent years, neural ranking models (NRMs) have been shown to substantially outperform their lexical counterparts in text retrieval. In traditional search pipelines, & combination of features leads to well-defined behavior. However, as neural approaches become increasingly prevalent as the final scoring component. of engines or as standalone systems, their robustness to malicious text and, more gencrally, semantic perturbation needs to be better understood. We posit that the transformer atlention mechanism can induce exploitable defects through positional bias in search models, leading to attack that could generalize beyond single query topic. We demonstrate such defects by showing that non-relevant text-such as promotional content-can be casily injected into document without adversely affecting its position in search results. Unlike previous gradient-based attacks, we demonstrate these biases in a query-agnostic fashion. In doing so, without the knowledge of topicality, we can still reduce the negative effects of non-relevant content injection by controlling injection position. Our experiments are conducted with simulated on-topic promotional text automatically generated by prompting LLMs with topical context. from target documents. We find that contextualization of non-relevant text further reduces negative effects whilst likely circumventiin existing content filtering mechanisms. In contrast, lexical models are found to be more resilient to such content injection attacks. We then investigate a simple yet effective compen- sation for the weaknesses of the NRMs in search, validating our hypotheses regarding transformer bias.
Qwen 1.5 Chat (72B)
Generated Abstract (2-shot demonstrations from co-author papers):
The study investigates the vulnerability of Neural Ranking Models (NRMs) to adversarial attacks through the injection of promotional content into documents. The authors propose framework for generating contextualised text using Large Language Models (LLMs) and examine the impact of position and context on the relevance estimation of injected text. They find that NRMs, particularly those based on BERT and T5 architectures, are sensitive to the position of injected lext, with content placed after salient sentences having a lesser negative effect on relevance. The study also introduces mitigation strategy using classifier to detect promotional content, demonstrating its effectiveness in reducing the negative impact retrieval performance. The findings highlight the potential for exploiting positional bias in NRMs and the need for robustness against such attacks in semantic search.
Nous Hermes Mixtral 8x7B-DPO (46.7B)
Generated Abstract (2-shot demonstrations from co-author papers):
This study investigates the exploitation of positional bias in neural ranking models (NRMs) for query-agnostic generative content injection in search. The authors propose the concept of attention bleed-through, where positive attention scores from relevant spans propagate to non-relevant spans, affecting relevance estimation. They find that injecting non-relevant text close to document. sentences with high similarity scores with the query can largely affect the rank of documents. The study also demonstrates that controlling the position and contextualization of promotional text can mitigate its negative impact on retrieval performance. which improves nDCG@10 significantly under classic evaluation setting by reducing the effect of contextualization. The findings highlight the importance of addressing positional bias in NRMs and the potential risks of generative content injection in semantic search.
model-agnostic mitigation strategy is proposed,
29

---

Published in Transactions on Machine Learning Research (10/2025)
Comparison between Generated Abst and Ground Truth on ABSTRACT WRITING Task
Ground Truth:
Developing autonomous agents for mobile devices can significantly enhance user interactions by offering increased efficiency and accessibility. However, despite the growing interest in mobile device control agents, the absence of commonly adopted benchmark makes it challenging to quantify scientific progress in this area. In this work, we introduce B-MoCA, novel benchmark designed specifically for evaluating mobile device control agents. To create a realistic benchmark, we develop B-MoCA based on the Android operating system and define 60 common daily tasks. Importantly, we incorporate a randomization feature that changes various aspects of mobile devices, including user interface layouts and language scttings, to assess generalization performance. We benchmark diverse agents, including agents employing large language models (L.LMs) multi-modall...Ms well agents trained from scratch using human expert demonstrations. While these agents demonstrate proficiency in exccuting straightforward tasks, their poor performance on complex tasks highlights significant opportunities for future research to enhance their effectiveness. Our source code publicly available at https://b-mocc.gihhnbb.
Qwen 1.5 Chat (72B)
Generated Abstract (2-shot demonstrations from co-author papers):
The Benchmark for Mobile Device Control Agents (B-MoCA) is novel evaluation platform designed to assess the performance of agents in controlling mobile devices across diverse configurations. The benchmark features 60 practical tasks grounded in real-life scenarios and supports randomization of device attributes like icon placements, wallpapers, languages, and device types to test generalization ability. The study compares three types of agents: L.M (Targe Language Model) agents, MLLLM (Multimodal Large Language Model) agents, and Vision-Language-U1 (VLUI) agents. Results show that while LLM and MLLM agents perform well in simple tasks, they struggle with complex lasks requiring sequential decision-making. VLUI agents, trained from scratch, demonstrate better performance in completing tasks but still face challenges with severe visual changes in unseen device configurations. The study highlights the importance of diverse training data and the need for more efficient methods to leverage multi-mon mooti- inputs. Overall, B-MoCA provides a com- prehensive framework for evaluating and advancing mobile device control agents real-world settings.
Nous Hermes 2 Mixtral 8x7B-DPO (46.7B)
Generated Abstract (2-shot demonstrations from co-author papers):
This paper introduces B-MoCA, a benchmark designed to evaluate the performance of mobile device control agents on diverse device configuratinns executing common daily tasks. The benchmark includes 60 basic tasks grounded in realistic scenarios and a randomization feature that changes icon placements and sizes, wallpapers, languages, and device types. Bascline agents include LLM agents, MLLM agents, and Vision-Language-UU UT (VL.UT) agents. Experiments reveal that. LLM agents and MLLM agents show high robustness across diverse device configurations while they fall short multiple sequential decision-making. Agents trained with BC successfully mimic expert behaviors but lack generalization ability in lest environments with unseen device configurations. The study highlights the limitations of existing methods in mobile device control and calls for future research.
30

---

Published in Transactions on Machine Learning Research (10/2025)
E.3
Introduction Writing
Comparison between Generated Introduction and Ground Truth on INTRODUCTION WRITING Task
Ground Truth:
The original lext too long to be included here. Please 2404.17196v via arXiv for more details.
Qwen 1.5 Chat (72B)
Generated Introduction (2-shot demonstrations from co-author papers):
Retrieval poisoning is a newly identified threat to large language model (LLM))powered applications, particularly those utilizing the retrieval augmented gencration (RAG) technique. This threat arises from the vulnerability in LLM application frameworks, which allow altackers to subtly manipulate the content of documents in way that appears benign to humans but can guide the LLMs to gen- erate incorrect malicious responses. The attack is executed by injecting invisible attack sequences into the documents, which remain undetected during the RAG process, leading to compromised application responses. The prevalence of LLM application frameworks, such as LangChain, has enabled the development numerous LLM-powered applications. However, these frameworks often overlook the security risks associated with external content. Retrieval poisoning exploits this vulner- ability by crafting malicious documents that, despite containing accurate information, can prompt the LLMs to produce misleading responses when used reference sources during RAG. The success 88.33% success
rate of such allacks is alarming, with preliminary experiments demonstrating rate in misleading LLMs and a 66.67% success rate in real-world applications. To execute retrieval poisoning, attackers analyze the LLM application framework, identifying exploitable components like document parsers, text splitters, and prompt templates. They then generate an attack sequence that invisible to human perception but influences the LLM's response generation. The crafted malicious document is designed maintain its original appearance while containing the hidden attack sequence, which is strategically placed to remain undivided by text splitters. The proposed approach involves two-step process: analyzing the framework to identify invisible injection points and generating the attack sequence. Attackers leverage the framework's prompt templates to create augmented requests that guide the LLMs towards generating targeted malicious responses. The effectiveness of the attack is demonstrated through experiments on different LLMs, augmented requests, and real-world applications, showcasing the potential impact of retrieval poisoning. Tn summary, retrieval poisoning a stcalthy attack method that exploits the vulnerabilitics in LLM application frameworks, particularly in the RAG process. By injecting invisible attack sequences into documents, attackers can manipulate LLMs into generating incorrect responses, posing significant security concern for LLM-powered applications. The proposed approach highlights the need for better understanding of LLM application frameworks and the development of more robust defense mechanisms to mitigate such threats. The paper structure delves into the problem, presents the attack methodology, demonstrates its effectiveness through experiments, and discusses potential defense strategics, ultimately contributing to the ongoing rescarch on securing LLM-powered applications.
31

---

Published in Transactions on Machine Learning Research (10/2025)
Comparison between Generated Introduction and Ground Truth on INTRODUCTION WRITING Task
Ground Truth:
The original text is too long be included here. Please see 2404.18465v2 via arXiv for more details.
Qwen 1.5 Chat (72B)
Generated Introduction (2-shot demonstrations from co-author papers):
The M3oE framework is a pioneering solution to the challenge multi-domain multi-task recommen- dation, addressing the intricate dependencies among multiple domains and tasks in disentangled and adaptive manner. Recognizing the need for a more comprehensive approach in practical recommendation systems, M3oE integrates multi-domain information and leverages knowledge transfer across domains and tasks, optimizing for multiple objectives simultaneously. The framework consists of three mixturo-of-cxpert. modules, cach dedicated to capturing common, domain-speccfic,, and lask-specific user preferences. A two-level fusion mechanism is employed to effectively extract and integrate features across diverse domains and tasks, ensuring a balanced representation of user preferences. Existing multi-domain and multi-task recommendation methods have shown promise in enhancing recommendation accuracy by exploiting shared patterns. However, they often fall short when faced with the complexity of simultancously addressing multiple domains and tasks. M3oF stands out by introducing self-adaptive approach, utilizing AutoML to optimize the model structure dynamically. This adaptability is crucial for real-workd scenarios where data characteristics and lask requirements can vary significantly. The core of M3oE lies in its three expert modules: shared, domain, and task experts. The shared expert module learns common patterns across domains and tasks, while the domain expert module focuses on domain-specific characteristics. The task expert module, on the other hand, models lask-specific user preferences. The two-level fusion mechanism balances the contribution of these modules, allowing for flexible and effective representation of preferences. Experimental results benchmark datasets demonstrate M3oE's superior perfor- mance compared to stale-of-the-arl baselines, highlighting its ability to address the multi-domain multi-task seesaw problem. In summary, M3oE is a novel and adaptive multi-domain multi-task recommendation framework that integrates multi-domain information, transfers knowledge across domains and tasks, and optimizes for multiple objectives. The framework's disentangled and fused expert modules, along with the AutoMI.-drivn fusion weights, enable user preferences in complex recommendation scenarios. The proposed approach not only surpasses existing methods in performance but also offers more generalizable solution to the multi-domain multi-task recommendation challenge. The structure of the paper delves into the problem definition, framework overview, detailed module descriptions, experimental setup, and results, providing a comprehensive understanding of M3oE's design and effectiveness.
effectively capture
32


Reviews:
[{"review_id": "NBNqRrKi4G", "reviewer": "Reviewer_HUkG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new benchmark, called AcademicEval, to evaluate large language models on long-context generation tasks using real-world academic papers from arXiv. It aims to overcome limitations in existing benchmarks, such as rigid context length, labor-intensive annotation, and label leakage, by offering live updates, automatic labeling, and tasks involving hierarchical abstraction levels. The dataset construction and update method requires no manual annotation.\n\nThe work evaluates several popular LLMs on the benchmark. The analysis shows the challenge LLMs face with abstraction hierarchy and long-context modeling, offering useful insights for further research and improvement.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The motivation for constructing the new dataset is clearly stated, and a comparison with existing related datasets is provided.\n\n- The method for constructing and updating the dataset is feasible. The method requires no manual annotation.\n\n- The experimental analysis also reflects the shortcomings of current methods in long-text generation.", "audience_yn": "Yes", "audience_explanation": "Although this work is a paper in the field of NLP, some researchers in the machine learning community are also exploring KV-cache approaches for long-text modeling and may find this paper interesting.", "requested_changes": "- The publication dates of the papers used in the current dataset appear to be unspecified. If the papers are quite old, there might be data leakage concerns, as arXiv is an open platform and its papers may have already been incorporated into large model training datasets—meaning the models have potentially seen these papers before. \n\n- Although the proposed method does not require manual annotation, it still involves operations such as title and abstract extraction from PDFs, as well as author disambiguation. Could there be inaccuracies in extraction or author disambiguation here? If possible, could a dataset quality evaluation or discussion be conducted?", "broader_impact_concerns": "NA.", "additional_comments": "None.", "extra_scores": {}, "extra_text": {}}, {"review_id": "OEXLsxkKBv", "reviewer": "Reviewer_x9YH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces AcademicEval, a benchmark for evaluating long-context language models on academic writing tasks without manual annotation. Its key contributions are:\n- Use daily arXiv updates and a co‑author graph to ensure continuous data collection and evaluation. \n- Authors propose four tasks (Title, Abstract, Introduction, Related Work) spanning increasing abstraction levels. \n- Authors benchmarked standard LLMs, long‑context LLMs, and retrieval‑augmented LLMs (RALM), revealing performance degradation with longer contexts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The live update mechanism is detailed with graph updates and pruning procedure. \n- Task definitions and automatic labeling are clearly described. \n- Experimental results across seven models show consistent trends (e.g., performance drops as context grows).", "audience_yn": "Yes", "audience_explanation": "- Researchers developing long‑context LLMs would be interested in this work. As academic papers can be a natural testbed/dataset for long-context tasks.", "requested_changes": "- My biggest concern is that the evaluation relied on BERTScore and ROUGE-L may not capture output quality beyond semantic similarity. For example, it misses aspects like novelty, feasibility, and consistency. I think authors need to introduce additional evaluation metrics or human judgments to address this. For example, are the generated abstract/introductions logically consistent?\n- Include and compare to other related works, like ResearchTown: Simulator of Human Research Community (arXiv:2412.17767), which also proposes writing tasks based on academic graphs.\n- Correct the typo in Figure 3 on “Gemma instruct.”", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZsDwCmDfrz", "reviewer": "Reviewer_PxUc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a dynamic benchmark constructed from arXiv that asks models to predict masked portions of a paper given its co-authors’ work. Experiments span several state-of-the-art LLMs. While the resource is timely and the benchmark design is creative, the current evaluation and interpretation raise fundamental questions about what is actually being measured.", "claims_and_evidence_yn": "No", "claims_explanation": "Major Concerns\n- Plateau in long context performance does not necessarily mean that the tested LLM is incapable of long-context modeling. In-contrast, the observed plateau could equally arise from the benchmarks itself, including: (a) low contextual relevance or (b) the dominance of in-weights learning (IWL) over in-context learning (ICL). Table 2 already shows RALM often underperforming the vanilla LLM, supporting the irrelevance hypothesis. If performance saturates even with highly-relevant papers, the benchmark is no longer probing “long-context reasoning” but merely confirming the sufficiency of pre-training corpora. This would be a serious limitation for the community.\n\n- Metrics are out-dated: BERTScore and ROUGE were designed for extractive summarization, not scientific reasoning. Consider, for instance, GPT-4o-based scoring (with chain-of-thought rubrics) for factual and stylistic fidelity.", "audience_yn": "Yes", "audience_explanation": "Long-Context Modeling Measure, and data leakage are indeed important issues in LLM research.", "requested_changes": "Typo: where longer contexts correlate with decreased model performance. showing that…", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)