Benchmarking Long-Context Language Models on Long Code Understanding
Authors: Jia Li, Xuyuan Guo, Lei Li, Kechi Zhang, Ge Li, Jia Li, Zhengwei Tao, Fang Liu, Chongyang Tao, Yuqi Zhu, Zhi Jin
Accepted at: acl25

LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding

Jia Li1,2, Xuyuan Guo3, Lei Li4, Kechi Zhang1,2, Ge Li1,2*, Jia Li σ5, Zhengwei Tao1,2, Fang Liu6, Chongyang Tao6, Yuqi Zhu 1,2, Zhi Jin1,2*
1Key Lab of High Confidence Software Technology (PKU), Ministry of Education
2School of Computer Science, Peking University, China
3School of Software & Microelectronics, Peking University, China
4The University of Hong Kong 5College of AI, Tsinghua University
6School of Computer Science and Engineering, Beihang University
{lijiaa,lige,zhijin}@pku.edu.cn, {guoxy26,nlp.lilei}@gmail.com

Abstract

Current advanced long-context language models offer great potential for real-world software engineering applications. However, progress in this critical domain remains hampered by a fundamental limitation: the absence of a rigorous evaluation framework for long code understanding. To gap this obstacle, we propose a long code understanding benchmark LONGCODEU from four aspects (8 tasks) to evaluate LCLMs’ long code understanding ability required for practical applications, including code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long code documentation understanding. We evaluate 9 popular LCLMs on LONGCODEU (i.e., 6 general models and 3 code models). Our experimental results reveal key limitations in current LCLMs’ capabilities for long code understanding. Particularly, the performance of LCLMs drops dramatically when the long code length is greater than 32K, falling far short of their claimed 128K~1M context windows. In the four aspects, inter-code unit relation understanding is the most challenging for LCLMs. Our study provides valuable insights for optimizing LCLMs and driving advancements in software engineering.

1 Introduction

Recent advances in long-context language models (LCLMs) like Gemini-1.5 (Team et al., 2024) and GPT-4o (GPT, 2024), which support context windows exceeding hundreds of thousands of tokens, offer unprecedented potential for real-world software engineering applications. These models promise transformative improvements in related downstream tasks, such as repository-level code generation (Zhang et al., 2024b; Bi et al., 2024), real-world GitHub issues resolution(Jimenez et al., 2023), and long code summarization (Dhulshette et al., 2025). However, progress in this critical area remains hampered by a fundamental limitation: the lack of rigorous evaluation frameworks for long code understanding—a capability essential for real-world software development tools that require accurate code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long code documentation understanding in code repositories.

Current benchmarks generally fall into two categories, which face five fundamental limitations that hinder comprehensive evaluation of LCLMs’ long code understanding capabilities. The first category includes studies such as RepoQA (Liu et al., 2024), whose task design has insufficient diversity like only focusing on needle function search. While these evaluations are useful, they do not capture the real-world full range of code understanding capabilities needed for practical coding scenarios. Additionally, benchmarks like L-Eval An et al. (2023) further compound these limitations by using synthetic “long code” through a

def calculate_positive_average(lst):
    pos_n = [i for i in lst if i > 0]
    return sum(pos_n) / len(pos_n) if pos_n else 0

def upper_and_sort(str_list):
    upper_str = []
    for string in str_list:
        upper_str.append(string.upper())
    return sorted(upper_str, key=len)

(a) A long code with independent functions

import codecs
def decode_utf7(enc):
    return codecs.decode(enc, 'utf-7')

def namespace(self):
    data = self._command_and_check("namespace")
    parts = []
    for item in parse_response(data):
        prefix, separator in item:
            prefix = decode_utf7(prefix)
            parts.append(prefix, to_unicode)
    return Namespace(parts)

(b) A real-world long code with non-standalone functions

Figure 1: Examples of a synthetic long code with independent functions and a real-world long code with non-standalone functions. Dependencies are highlighted.
simple joining of independent code snippets. ② This approach overlooks the natural dependencies between code segments, as shown in Figure 1. ③ These studies also face issues with data contamination. They neither enforce temporal constraints on code release dates nor address potential model pre-exposure through evaluating on previously published datasets. ④ Their maximum supported context length of 36.5K tokens also falls far short of stress-testing the claimed 128K-1M context windows of modern LCLMs. The second category includes benchmarks like Long Code Arena (Bogomolov et al., 2024), LongBench (Bai et al., 2023), SWE-bench (Jimenez et al., 2023), and DevEval (Li et al., 2024b), which evaluate long-context understanding based on performance in downstream tasks. ⑤ However, these benchmarks entangle code understanding with other task-specific challenges like code generation and bug fixing. This makes it hard to determine whether performance limitations are due to a lack of code understanding or other factors.

To address these limitations, we propose LONGCODEU, a benchmark designed to isolate and comprehensively evaluate LCLMs’ capacity to understand and reason about real-world, dependency-rich, long code contexts. Our benchmark offers the following key features:

• Comprehensive Tasks stem from Practical Applications. We evaluate LCLMs from four aspects (8 tasks) to evaluate the LCLMs’ long code understanding capability required for practical applications, including code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long documentation understanding.
• Extra-long Code Context. Each task contains around 500 gathered long codes. The lengths of examples change in 0~8K, 8~16K, 16~32K, 32~64K, and 64~128K following the normal distribution, which far exceeds the maximum length of 36.5K supported by existing benchmarks (An et al., 2023).
• Real-world Repository. The benchmark is collected from real-world code repositories. Long code consists of one or more real code file contents, instead of being composed of multiple independent code units like current benchmarks (Liu et al., 2024).
• Reducing Data Contamination. We collect up-to-date code repositories that are created after 2024-06 from GitHub1, which are later than most prevailing LCLMs’ cut-off dates thus reducing the risk of data contamination.

We evaluate 9 popular LCLMs, which contain 6 general models (i.e., GPT-4o (GPT, 2024), Claude-3.5-Sonnet (cla, 2024), Gemini-1.5-Flash (Team et al., 2024), DeepSeek-V2.5 (Bai et al., 2023), Mistral-v0.3 (Jiang et al., 2023), and Phi-3.5 (Abdin et al., 2024)) and 3 code models (i.e., DeepSeek-Coder-V2 (Bai et al., 2023), Qwen2.5-Coder (Hui et al., 2024), CodeLLama (Roziere et al., 2023)) on LONGCODEU. The experimental results reveal key limitations in current LCLMs’ capabilities for long code understanding. Especially, LCLMs’ performance drops dramatically when the long code length is greater than 32K, falling far short of their claimed context windows such as 128K-1M tokens. In the four aspects, inter-code unit relation understanding is the most challenging for LCLMs. Our findings provide valuable insights for optimizing LCLMs and driving advancements in software engineering.

2 Related Works

Benchmarks on Long Code Understanding. Existing studies can be mainly categorized into two types. The first category predominantly explores the long code understanding ability required in downstream applications like the needle-in-a-haystack task, but they face significant limitations. RepoQA (Liu et al., 2024) is a pioneer introduced needle function retrieval task. However, the single task is insufficient to evaluate the complex long code understanding ability. L-Eval (An et al., 2023), a widely-used benchmark, has limitations as well. It constructs artificial "long code" by concatenating independent code snippets, ignoring context dependency in real-world source code. These limitations highlight the need for a more comprehensive and reliable benchmarking approach for evaluating LCLMs’ long code understanding ability. The second category includes research like LCArea (Bogomolov et al., 2024). They verify LCLMs’ long code understanding ability by measuring their performance on downstream tasks such as LongBench (Bai et al., 2023), SWE-bench (Jimenez et al., 2023), and EvoCodeBench (Li et al., 2024a). Although they provide an intuitive way to assess LCLMs, it suffers from a significant drawback. The performance of downstream tasks is confounded by

1https://github.com/
Table 1: The comparison between existing benchmarks and LONGCODEU. #Num is the abbreviation of number. #Div Tasks refers to diverse tasks. #High Disp represents high dispersion. #Max-L and #Avg-L mean the maximum length and the average length of long code. #Trunk-L means whether each example has the length label. #Doc refers to documentation related to repositories. #Task represents the number of tasks (i.e., examples).

<table>
  <tr>
    <th>Benchmark</th>
    <th>#Num</th>
    <th>#Div Tasks</th>
    <th>#High Disp</th>
    <th colspan="3">Extra-long Data</th>
    <th>Real-world Repository Code</th>
    <th>#Doc</th>
    <th>#Num</th>
    <th>Reduce Data Leaking</th>
    <th>Data Time</th>
    <th>Data Scale</th>
    <th>#Task</th>
  </tr>
  <tr>
    <th colspan="14">The second category benchmarks (Only some benchmarks are listed)</th>
  </tr>
  <tr>
    <td>LongBench [6]</td>
    <td>2</td>
    <td>X</td>
    <td>X</td>
    <td>–</td>
    <td>0.4K</td>
    <td>X</td>
    <td>Function</td>
    <td>X</td>
    <td>–</td>
    <td>2023.02–2023.08</td>
    <td>–</td>
    <td>1,000</td>
    <td>–</td>
  </tr>
  <tr>
    <td>LC-Arena [8]</td>
    <td>6</td>
    <td>X</td>
    <td>X</td>
    <td>–</td>
    <td>–</td>
    <td>X</td>
    <td>File</td>
    <td>X</td>
    <td>62</td>
    <td>2023.01–2024.05</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>LONGPROC [30]</td>
    <td>1</td>
    <td>X</td>
    <td>✓</td>
    <td>–</td>
    <td>2K</td>
    <td>✓</td>
    <td>Function</td>
    <td>X</td>
    <td>0</td>
    <td>No Limit</td>
    <td>200</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>DevEval [21]</td>
    <td>1</td>
    <td>X</td>
    <td>✓</td>
    <td>–</td>
    <td>0.5K</td>
    <td>✓</td>
    <td>File</td>
    <td>X</td>
    <td>164</td>
    <td>2023.11–2024.02</td>
    <td>–</td>
    <td>1,825</td>
    <td>–</td>
  </tr>
  <tr>
    <th colspan="14">The first category benchmarks</th>
  </tr>
  <tr>
    <td>RepoQA [22]</td>
    <td>1</td>
    <td>X</td>
    <td>X</td>
    <td>16K</td>
    <td>–</td>
    <td>X</td>
    <td>Function</td>
    <td>X</td>
    <td>50</td>
    <td>No Limit</td>
    <td>500</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td>L-Eval [5]</td>
    <td>1</td>
    <td>X</td>
    <td>✓</td>
    <td>36.5K</td>
    <td>31.5K</td>
    <td>✓</td>
    <td>Function</td>
    <td>X</td>
    <td>0</td>
    <td>No Limit</td>
    <td>90</td>
    <td>–</td>
    <td>–</td>
  </tr>
  <tr>
    <td><b>LONGCODEU</b></td>
    <td>8</td>
    <td>✓</td>
    <td>✓</td>
    <td>128K</td>
    <td>54.8K</td>
    <td>✓</td>
    <td>File</td>
    <td>✓</td>
    <td>116</td>
    <td>2024.06–2024.11</td>
    <td>3,983</td>
    <td>–</td>
    <td>–</td>
  </tr>
</table>

multiple factors, and these works do not decouple the long code understanding ability independently, which is orthogonal to our objective-evaluating LCLMs’ capacity to understand and reason about real-world, dependency-rich, long code.

Long Context Language Models. Recent studies have explored diverse ways to extend large language models’ context window size. The direct way is to fine-tune models on long sequences (Wu et al., 2021), but they are often effort-costing. Some approaches involve additional fine-tuning for a longer context (Xiong et al., 2023; Chen et al., 2023a,b,c; Peng et al., 2023). They down-scale the input position indices to match the original context window size of models with several training steps. There are also some training-free studies, which use window attention to clip the long sequences (Han et al., 2023; Ding et al., 2023; Xiao et al., 2023). Concurrently, a series of approaches modify the relative distance to extend the extrapolation length (Zhang et al., 2024a; Jin et al., 2024). In this paper, we construct a comprehensive benchmark to evaluate LCLMs’ long code understanding ability.

3 LONGCODEU Benchmark

In this section, we first introduce long code understanding tasks (\S3.1). Then, we describe the construction process of LONGCODEU (\S3.2). Finally, we present the evaluation metrics (\S3.3).

![Four understanding aspects in LONGCODEU.](page_101_1017_627_246.png)

Figure 2: Four understanding aspects in LONGCODEU.

3.1 Tasks

In real-world software development, long code understanding is usually oriented towards code repositories. These repositories take functions as basic code units, establish relations among units to achieve complex applications, and introduce documentation to describe code-related information. LCLMs with good long code understanding abilities should be able to perceive and understand basic code units, relations among units, and associated code documentation, which is essentially required for dealing with downstream tasks such as repository-level code generation, issue resolving, and long code summarization. In this paper, we propose LONGCODEU to comprehensively evaluate LCLMs’ long code understanding ability from four aspects: code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long documentation understanding. Four aspects of LONGCODEU are shown in Figure 2

These long code understanding tasks share the following procedure: given an instruction, long code, and anchor input, LCLMs output the desired answer. The instruction briefly describes the request of each task. The anchor input is the detailed demand such as a code unit. In LONGCODEU, instruction and anchor input are generally short, while long code is long containing 0~128K tokens.

3.1.1 Code Unit Perception

Understanding long code, particularly in code repositories, requires identifying its numerous functions, as they form the foundational code unit for comprehending long code’s overall functionality. In this paper, we treat a function as the code unit and introduce the code unit perception task to evaluate LCLMs’ code unit identifying ability in long code. Concretely, this task requires LCLMs to identify all defined functions in long code and return
their corresponding function names, where long code is composed of one or more code file contents collected from real-world repositories. This ability is the cornerstone for downstream tasks.

3.1.2 Intra-Code Unit Understanding
Based on code unit perception, we further evaluate LCLMs’ ability to understand the internal logic and semantics of code units.

Code Unit Data Flow Analysis. In this section, we propose the code unit data flow analysis task that verifies whether LCLMs can understand the internal logic of code units by tracking data flow to a certain extent. Given a code unit in long code and a variable name in the code unit, LCLMs are required to figure out lines where the value of the given variable changes by tracing data flow. For example, after executing the line “upper_string += 2”, the value of variable “upper_string” changes. The given code unit is randomly selected from long code and its position in long code is also random. Evaluating code unit understanding ability is significant for LCLMs to discover potential vulnerabilities, repair vulnerabilities, and optimize code.

Code Unit Semantic Analysis. In addition to analyzing the data flow of code units, we propose the code unit semantic analysis task to further verify LCLMs’ intra-code unit understanding ability. The task asks LCLMs to return a code unit from the long code that satisfies the given description. The long code contains real-world code files, where all descriptions of code units are removed to prevent LCLMs from acquiring clues in them. This task requires LCLMs understand the semantics of code units and then return the desired code unit.

3.1.3 Inter-Code Unit Relation Understanding
In real-world long code, especially in code repositories, code units are non-standalone. Grasping code unit relations is essential for LCLMs to understand the complex functionality of long code, where code unit relations mainly containing dependency relations and semantic relations. The dependency relation indicates the calls among code units. The semantic relation focuses on the functional similarities of code units.

Dependency Relation Analysis. (T1) Given a code unit, this task requires LCLMs to find code units that are invoked by the given unit from long code, where the long code covers one or multiple code files and is collected from the same repository with the given code unit. The dependency relation analysis ability can assist LCLMs correctly identifying other code units related to vulnerable units and determining vulnerability scopes in real-world applications. (T2) Considering that in real-world applications, apart from long code as LCLMs’ input, developers usually use natural language requirements to interact with LCLMs. Thus, LCLMs also need to understand dependency relation between long code and requirements. Given a natural language description, this task requires LCLMs to find code units from the long code that are invoked for generating the desired code that satisfies the given description. The ability can ensure that LCLMs successfully invoke existing code units in repository-level code generation and correctly integrate generated code into the current repository.

Semantic Relation Extraction. Even if two code units have no dependency relationship, they might be semantically similar such as having similar implementation or logic. This section analyzes semantic relations of code units in long code. (T1) Given a code unit, this task asks LCLMs to extract semantically similar code units with the given unit from long code. Extracting semantic relations of units can effectively help LCLMs to improve software development efficiency by reusing similar code units when programming, and enhance software maintainability such as finding potentially similar vulnerabilities among semantically similar units. (T2) As described in dependency relation analysis (T2), understanding the semantical relations of code units and natural language requirements is more in line with practical development scenarios. For instance, in repository-level code generation, developers input a requirement to LCLMs. LCLMs can find semantically similar units with the given requirement from the current repository and then refer to these units to generate desired codes. In this task, LCLMs need to extract semantically similar code units to a given requirement, which challenges LCLMs to understand the semantics of units in long code and reason their semantic relations.

3.1.4 Long Documentation Understanding
In real-world software engineering, code documentation also plays a crucial role. It encompasses a diverse range of code-related information including descriptions of code units, usage patterns, architecture designs, and more. Consequently, it is essential not merely to verify the long code under-
Table 2: Statistics of LONGCODEU. #Num means the number of examples in each task. #C-File represents whether the output can be obtained by aggregating cross-file content. #Avg-L is the average length of the output. #Gran means the granularity of the output.

<table>
  <tr>
    <th rowspan="2">Task</th>
    <th colspan="2">Input</th>
    <th colspan="3">Output</th>
  </tr>
  <tr>
    <th>#Num</th>
    <th>Format</th>
    <th>#C-File</th>
    <th>#Avg-L</th>
    <th>#Gran</th>
  </tr>
  <tr>
    <td>CU_P</td>
    <td>487</td>
    <td>Code</td>
    <td>x</td>
    <td>0.4K</td>
    <td>Name</td>
  </tr>
  <tr>
    <td>CU_SA</td>
    <td>500</td>
    <td>Code</td>
    <td>x</td>
    <td>0.2K</td>
    <td>Function</td>
  </tr>
  <tr>
    <td>CU_DFA</td>
    <td>500</td>
    <td>Code</td>
    <td>x</td>
    <td>0.03K</td>
    <td>Line</td>
  </tr>
  <tr>
    <td>DRA_T1</td>
    <td>500</td>
    <td>Code</td>
    <td>✓</td>
    <td>0.3K</td>
    <td>Function</td>
  </tr>
  <tr>
    <td>DRA_T2</td>
    <td>500</td>
    <td>Code</td>
    <td>✓</td>
    <td>0.3K</td>
    <td>Function</td>
  </tr>
  <tr>
    <td>SRE_T1</td>
    <td>500</td>
    <td>Code</td>
    <td>✓</td>
    <td>1.0K</td>
    <td>Function</td>
  </tr>
  <tr>
    <td>SRE_T2</td>
    <td>500</td>
    <td>Code</td>
    <td>✓</td>
    <td>1.1K</td>
    <td>Function</td>
  </tr>
  <tr>
    <td>LDU</td>
    <td>500</td>
    <td>Document</td>
    <td>x</td>
    <td>0.7K</td>
    <td>NL</td>
  </tr>
</table>

standing ability, but also to analyze the long code documentation understanding ability of LCLMs. We introduce the long documentation understanding task. Given long documentation and a code unit name such as a function name contained in the documentation, this task requires LCLMs to extract the related information to the unit name. We ensure that the long documentation contains the related knowledge of the given unit name, aiming to effectively evaluate LCLMs. Analyzing long documentation is critical to verify the performance of LCLMs in real-world software development.

3.2 Benchmark Construction

The pipeline for constructing LONGCODEU encompasses 6 stages as follows.

Stage ①: Repository Selection. Referring to the TIOBE index (Tio, 2024) for programming language popularity, the most popular language is Python. Thus, we conduct experiments on Python and will expand to other languages in the future. In Python source platforms, PyPI is a rich Python package index tool. We identify the top 10 popular programming topics in PyPI and obtain the top 50 packages with the most stars in each topic. We then select repositories following four criteria: open-source repositories, created after 2024-06, non-fork and non-malicious, and more than 50 stars. Finally, we crawl these repositories from GitHub and obtain 116 real-world repositories.

Stage ②: Code Parse. We use tree-sitter2 to design a code parser. This parser identifies code units defined in repositories, traces the definition of code units, and analyzes unit relations: First, it performs static analysis of each file in a repository and extracts unit names defined in it. Then, it executes unit symbol navigation, finding the definitions of all code units in the repository. Finally, it extracts unit names invoked in a code unit to grasp their dependency relations. Combining the three steps, the parser can traverse predefined code units within a repository and obtain intricate dependencies. To obtain semantic relations among units, we apply an advanced embedding model to encode code units, and use the cosine similarities of their representations to measure semantic relations.

Stage ③: Requirement Collection. We extract requirements contained in the signature of code units with tree-sitter and invite two developers to check requirements. Not that it is enough to construct examples for T2-type tasks even if some code units do not contain requirements in repositories.

Stage ④: Documentation Annotation. We collect documentation from collected code repositories and select the documentation with standard: being easy to distinguish the related information of code units. Then we invite two developers to manually label 500 examples.

Stage ⑤: Deduplication. For tasks whose input is code unit, we exclude trivial units (e.g., empty or initialization functions). To ensure the quality of LONGCODEU, we randomly select files from repositories as long codes, and ensure long codes of all examples in each task are non-repetitive.

Stage ⑥: Benchmark Construction. Based on the above stages, we construct around 500 examples for each task supporting the maximum length of 128K tokens. We make LONGCODEU satisfy following goals: comprehensive tasks from practical applications, extra-long code context, real-world repositories, and reducing data contamination.

3.3 Automatic Evaluation

We focus on evaluating the long code understanding ability required for LCLMs to complete downstream tasks. Tasks in LONGCODEU commonly require LCLMs to retrieve dispersive snippets from long code and execute reasoning. Thus, we mainly adopt metrics used in retrieval tasks to measure LCLMs, including recall and precision.

For proper evaluation, we refine existing metrics according to the output granularity of tasks:
① Output with code lines refers to the code unit data flow analysis task, which extracts several lines from long code. We first use exact match (EM) to measure each line in outputs. Then, we calculate recall and precision based on EM, acquiring EM-R and EM-P metrics. ② Output with code unit names
is related to the code unit perception task. We first find the correct names in output and then calculate the longest common subsequence (LCS) of unit names between output and ground truth, since it not only reflects whether the unit name is correct but also indicates LCLMs’ ability to perceive the position of units in long code. Then, we calculate recall and precision based on LCS, named LCS-R and LCS-P. ③ Output with code units refers to code unit semantic analysis and inter-code unit relation understanding tasks. Code unit semantic analysis only returns one unit. We use CodeBLEU as its metric, where it is a popular metric to indicate the consistency of two code sequences. For inter-code unit relation understanding, it returns multiple code units. We first determine whether the name of each generated unit is in ground truth. Then, we calculate CodeBLEU of each unit if it exists in the ground truth. For the code unit whose name is not in the ground truth, its CodeBLEU value is set to 0. Based on CodeBLEU value of each unit, we finally calculate CodeBLEU-based recall and precision, dubbed CB-R and CB-P. ④ Output with descriptions is related to the long documentation understanding task. Considering that documentation contains many text descriptions, we employ BLEU for evaluation, which is commonly used to measure the consistency between two sequences in the natural language process field.

These automatic evaluations can effectively measure LCLMs’ long code understanding ability since the outputs of each task are deterministic and task’ features are incorporated into these metrics. Figure 5 shows the automatic evaluation correlates well with human annotation, which further demonstrates the reliability of our automatic metrics.

4 Experiment Setups

In this section, we aim to answer the following research questions through a series of experiments.
RQ1. How is the long code understanding ability of LCLMs? We evaluate LCLMs’ long code understanding ability on LONGCODEU in §5.1.
RQ2. What is performance of LCLMs across long code lengths? We explore the performance of LCLMs on long code with different lengths. §5.2 demonstrates LCLMs’ performance comparison across long code lengths.
RQ3. How do developers select models in real-world application scenarios? Based on the experimental results, we summarize the empirical lessons we learned, aiming to help developers select suitable LCLMs (§5.3).

4.1 Base LCLMs

We evaluate 9 advanced LCLMs on LONGCODEU, which contain 6 general models (i.e., GPT-4o (GPT, 2024), Claude-3.5-Sonnet (cla, 2024), Gemini-1.5-Flash (Team et al., 2024), DeepSeek-V2.5 (Bai et al., 2023), Mistral-v0.3 (Jiang et al., 2023), and Phi-3.5 (Abdin et al., 2024)) and 3 code models (i.e., DeepSeek-Coder-V2 (Bai et al., 2023), Qwen2.5-Coder (Hui et al., 2024), CodeLlama (Roziere et al., 2023)). DeepSeek-R1 and GPT-03-mini are newly released LCLMs, but their availability through invoking API is unstable or limited to usage frequency. We can not present their performance now and will evaluate them in the future.

4.2 Experimental Setup

We use greedy search for all experiments. We evaluate LCLMs within their maximum context window length. In the semantic relation extraction task, we use an advanced embedding model stella_en_400M_v5 to encode code units and natural language requirements with the 1,024 dimension version, and select the top five similar code units by calculating the cosine similarities of their embeddings. Constrained by computing resources, we evaluate DeepSeek-V2.5 by invoking API3 provided by DeepSeek. Although the context length of DeepSeek-V2.5 achieves 128K tokens, the API provided by DeepSeek only supports up to 60K tokens. Thus, we analyze DeepSeek-V2.5 on long codes with 0~64K tokens.

5 Results and Analysis

5.1 Long Code Understanding Capability

Table 3 presents the performance of LCLMs on LONGCODEU. For LCLMs with a context length of less than 128K tokens, we only evaluate them within their supporting maximum lengths.

Comparison across LCLMs. We observe significant performance gaps among LCLMs. For similar-scale models, the performance of code LCLMs is better than the counterpart of general models on most tasks. For example, Qwen2.5-Coder outperforms Phi-3.5 24.31% on intra-code unit understanding (the second-aspect task) in terms of CodeBLEU, and DeepSeek-Coder-V2 achieves 11.75% average improvements on

3https://api-docs.deepseek.com/api/deepseek-api
Table 3: The performance of LCLMs on LONGCODEU. We only report recall-based results (EM-R, LCS-R, and CB-R) due to page limitation. The precision-based results (EM-P, LCS-P, and CB-P) can be found in Appendix A.

<table>
  <tr>
    <th>#Param</th>
    <th>Context Size</th>
    <th colspan="2">Task</th>
    <th colspan="6"></th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>CU_P</th>
    <th>CU_SA</th>
    <th>CU_DFA</th>
    <th>DRA_T1</th>
    <th>DRA_T2</th>
    <th>SRE_T1</th>
    <th>SRE_T2</th>
    <th>LDU</th>
    <th>#Avg</th>
  </tr>
  <tr>
    <th colspan="11">Open-Source LCLMs</th>
  </tr>
  <tr>
    <td>Qwen2.5-Coder</td>
    <td>7.6B</td>
    <td>128K</td>
    <td>43.47</td>
    <td>71.06</td>
    <td>74.01</td>
    <td>30.38</td>
    <td>9.59</td>
    <td>24.34</td>
    <td>21.81</td>
    <td>21.83</td>
    <td>37.06</td>
  </tr>
  <tr>
    <td>DeepSeek-Coder-V2</td>
    <td>15.7B</td>
    <td>128K</td>
    <td>38.67</td>
    <td>65.21</td>
    <td>48.42</td>
    <td>47.26</td>
    <td>22.92</td>
    <td>24.61</td>
    <td>26.21</td>
    <td>50.69</td>
    <td>40.49</td>
  </tr>
  <tr>
    <td>CodeLlama</td>
    <td>33.7B</td>
    <td>16K</td>
    <td>68.57</td>
    <td>62.41</td>
    <td>79.87</td>
    <td>68.82</td>
    <td>34.94</td>
    <td>44.48</td>
    <td>36.34</td>
    <td>46.92</td>
    <td>55.29</td>
  </tr>
  <tr>
    <th colspan="11">Open-Source LCLMs</th>
  </tr>
  <tr>
    <td>Phi-3.5</td>
    <td>3.8B</td>
    <td>128K</td>
    <td>39.92</td>
    <td>46.75</td>
    <td>49.52</td>
    <td>30.76</td>
    <td>9.66</td>
    <td>18.99</td>
    <td>14.48</td>
    <td>34.14</td>
    <td>30.53</td>
  </tr>
  <tr>
    <td>Mistral-v0.3</td>
    <td>7.3B</td>
    <td>32K</td>
    <td>57.42</td>
    <td>63.90</td>
    <td>58.00</td>
    <td>46.66</td>
    <td>18.92</td>
    <td>33.91</td>
    <td>32.50</td>
    <td>58.64</td>
    <td>46.24</td>
  </tr>
  <tr>
    <td>DeepSeek-V2.5</td>
    <td>236B</td>
    <td>128K</td>
    <td>70.58</td>
    <td>82.11</td>
    <td>77.47</td>
    <td>72.25</td>
    <td>56.80</td>
    <td>49.08</td>
    <td>47.42</td>
    <td>85.85</td>
    <td>67.70</td>
  </tr>
  <tr>
    <th colspan="11">Proprietary Source LCLMs</th>
  </tr>
  <tr>
    <td>Claude-3.5-Sonnet</td>
    <td>–</td>
    <td>200K</td>
    <td>43.82</td>
    <td>40.60</td>
    <td>45.65</td>
    <td>29.37</td>
    <td>28.70</td>
    <td>26.55</td>
    <td>27.77</td>
    <td>41.81</td>
    <td>35.53</td>
  </tr>
  <tr>
    <td>Gemini-1.5-Flash</td>
    <td>–</td>
    <td>1000K</td>
    <td>58.45</td>
    <td>83.46</td>
    <td>80.37</td>
    <td>72.51</td>
    <td>46.42</td>
    <td>39.84</td>
    <td>38.69</td>
    <td>81.43</td>
    <td>61.39</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>–</td>
    <td>128K</td>
    <td>56.42</td>
    <td>86.76</td>
    <td>87.87</td>
    <td>71.58</td>
    <td>48.88</td>
    <td>44.45</td>
    <td>43.14</td>
    <td>87.54</td>
    <td>65.83</td>
  </tr>
</table>

Figure 3: Performance comparison across tasks and long code lengths on LONGCODEU (grey blocks indicate unavailable configurations). The rate of performance degradation exhibits task-specific and model-specific patterns.

inter-code unit relation understanding (the third-aspect task) in CB-R. Among open-source models, DeepSeek-V2.5 performs the best, which is related to its large number of parameters. In proprietary source models, GPT-4o achieves the best performance, while Claude-3.5-Sonnet is not satisfactory.

Comparison across Tasks. In the four aspects, LCLMs perform the best in code unit perception and long documentation understanding, and achieve moderate results in intra-unit code understanding. Inter-code unit relation understanding is the most challenging. Their performances are reasonable. Because it’s a fundamental ability for LCLMs to understand code documentation and perceive code units. Based on this, LCLMs understand intra-unit code and then analyze inter-code unit relations, thereby comprehending long code. When observing the third-aspect task (i.e., inter-code unit relation understanding) closely, it can be seen that the performance of dependency relation analysis is lower than that of semantic relation extraction. We also find that no LCLM outperforms others on all tasks. For instance, in terms of scale-similar code models and general models, the former outperforms the latter in code unit perception, while code models perform worse than general models in long documentation understanding.

5.2 Performance across Long Code Lengths

To address RQ2, we classify examples into five buckets according to the long code length, including 0~8K, 8~16K, 16~32K, 32~64K, and 64~128K on each task. We conduct experiments on these classes to investigate the true context ability supported by LCLMs in long code understanding. Figure 3, Appendix A, and B show the results of LCLMs across long code lengths.

Our experimental results reveal key limitations in current LCLMs’ capabilities for long code understanding. We can also observe a negative correlation between the long code length and the performance of LCLMs. LCLMs’ performance drops dramatically when the long code length is greater than 32K tokens, falling well short of their claimed 128K~1M context windows. Besides, when the long code contains 64~128K tokens, the performance of LCLMs is near to 10 or even close to 0 on some tasks such as dependency relation analy-
sis and semantic relation extraction. In addition, the degradation slopes of performances vary by task. For example, there is a large slope in long documentation understanding, which means that LCLMs are suitable for processing relatively short documentation. The slope on code unit understanding including code unit data flow analysis and code unit semantic analysis tasks is relatively small. LCLMs fail to model code context effectively in their claimed context windows.

5.3 Empirical Lessons.

Based on the above experiments, we summarize the empirical lessons we learned as: ① The small-size LCLMs such as Qwen2.5-Coder can satisfy developers’ need for code unit perception and understanding if long code length is less than 16K. Otherwise, we suggest choosing larger-size models. ② For understanding long code documentation with more than 32K tokens, it is recommended to use GPT-4o and Gemini-1.5-Flash. Otherwise, Mistral-v0.3 and DeepSeek-Coder-V2 can achieve satisfying performances. ③ For tasks with high requirements for understanding inter-code unit relations, we suggest selecting the most powerful LCLMs that developers can access. Because when long code length exceeds 64K which is common in repository-level tasks, strong LCLMs also achieve weak performances. These poor performances explain why powerful LCLMs perform unsatisfactorily in repository-level downstream tasks. For example, GPT-4o only achieves 4.00% Success@1 on repository-level code translation benchmark-RepoTransBench (Wang et al., 2024).

6 Discussion

6.1 Case Study

We analyze the outputs of LCLMs, particularly GPT-4o, on the inter-code unit relation understanding task. Appendix D presents two notable examples. GPT-4o often extracts code units that are structurally similar or share overlapping tokens but have distinct or even opposite functionalities. This highlights the need to enhance models’ ability to distinguish confusing code units.

6.2 Code Understanding or Memorization?

We collect a small number of early-released code repositories from GitHub that LCLMs have potentially encountered during LCLMs training, aiming to analyze the dependency degree of LCLMs on memorization and long code understanding ability. Concretely, we enter only an instruction and anchor input to models, withholding the long code context, and assess their performance. We select tasks where models can work normally even without long code context, such as code unit semantic analysis and dependency relation analysis (T2). As shown in Figure 4, the memorization performance (w/o context) is much lower than the results with long code context, even though models might have met these contexts when training. The \( \delta \) score (the results with context minus the performance without context in yellow) relieves the memory phenomenon (Yu et al., 2023) and also reveals the significant importance for measuring LCLMs’ long code understanding ability.

![Bar chart comparing code understanding vs. memorization on CU_SA and DRA_T2 tasks](page_820_120_474_312.png)

Figure 4: Assessing long code understanding vs. memorization on CU_SA (left) and DRA_T2 (right) tasks.

6.3 Reliable Evaluation Metrics?

Reliable evaluation metrics are essential for assessing long code understanding. We measure the consistency between our metrics and human evaluation by selecting 20 GPT-4o outputs on each task and inviting two advanced developers to manually evaluate them. Using Kendall-Tau \( \tau \) (Kendall, 1938), we found that the average \( \tau \) value is at least 0.75 across all tasks, with the minimum value exceeding 0.7 (Figure 5). This demonstrates a strong correlation between our metrics and human evaluation, confirming the reliability of our metrics.

6.4 Further Evaluation of LCLMs on DCC and CCI Tasks

We conduct a code change impact (CCI) analysis task to further evaluate LCLMs’ long code understanding ability. Meanwhile, we also evaluate LCLMs on understanding dynamic code characteristics (DCC) like runtime behavior tracing. The details and results are shown in Appendix C.
Figure 5: The value of Kendall-Tau \( \tau \) between our automatic metrics and human evaluation.

7 Conclusion

In this paper, we propose a comprehensive long code understanding benchmark LONGCODEU. It introduces four aspects (8 tasks) to evaluate LCLMs’ long code understanding ability required for practical applications, including code unit perception, intra-code unit understanding, inter-code unit relation understanding, and long code documentation understanding. Our experimental results reveal key limitations in current LCLMs’ capabilities for long code understanding. When the long code length contains more than 32K tokens, the performance of LCLMs drops dramatically, falling far short of their claimed 128K~1M context windows. We hope our findings can provide valuable insights for optimizing LCLMs and driving advancements in software engineering.

8 Acknowledgement

This research is supported by the National Key R&D Program under Grant No. 2023YFB4503801, the National Natural Science Foundation of China under Grant Nos. 62192731, 62192730, 62192733, 62072007, and the Major Program (JD) of Hubei Province (No.2023BAA024).

Limitations

This paper proposes a benchmark - LONGCODEU to evaluate the long code understanding ability of long context language models from four aspects which are essential capabilities required for LCLMs to complete real-world downstream tasks. Based on LONGCODEU, we evaluate 9 popular LCLMs and analyze their strengths and shortcomings. We think that DevEval has three limitations.

① LONGCODEU is a monolingual benchmark (i.e., requirements in English and code in Python) and ignores other languages. In practice, LLMs require understanding requirements in different natural languages (e.g., Chinese, Spanish) and generating programs in various programming languages (e.g., Java, C). Thus, we plan to build a multilingual LONGCODEU in future work.

② Most recently, there are a few newly released LCLMs such as DeepSeek-R1 and OpenAI o3-mini-high. Constrained by the availability or stabilization of API, we do not provide the performance of these newly released models. In the future, we will evaluate their long code understanding abilities on LONGCODEU.

③ In our experiments, we only consider the long code with 0~128K tokens, although some LCLMs have supported longer context windows, e.g., Claude-3-5-Sonnet with 200K context size and even Gemini-1.5-Flash with 1000K context size. We will continue to update and evolve our benchmark, in order to support LONGCODEU to measure LCLMs on longer codes.

References

2024. Claude 3.5 haiku. https://www.anthropic.com/clause/haiku.

2024. Gpt-4o. https://openai.com/index/hello-gpt-4of/.

2024. Tiobe-index. https://www.tiobe.com/tiobe-index/.

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harikrat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.

Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2023. L-eval: Instituting standardized evaluation for long context language models. arXiv preprint arXiv:2307.11088.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508.

Zhangqian Bi, Yao Wan, Zheng Wang, Hongyu Zhang, Batu Guan, Fangxin Lu, Zili Zhang, Yulei Sui, Hai Jin, and Xuanhua Shi. 2024. Iterative refinement of project-level code context for precise code generation with compiler feedback. arXiv preprint arXiv:2403.16792.

Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria
Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie van Deursen, Maliheh Izadi, et al. 2024. Long code arena: a set of benchmarks for long-context code models. arXiv preprint arXiv:2406.11612.

Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, and Lidong Bing. 2023a. Clex: Continuous length extrapolation for large language models. arXiv preprint arXiv:2310.16450.

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023b. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595.

Yuhan Chen, Ang Lv, Ting-En Lin, Changyu Chen, Yuchuan Wu, Fei Huang, Yongbin Li, and Rui Yan. 2023c. Fortify the shortest stave in attention: Enhancing context awareness of large language models for effective tool use. arXiv preprint arXiv:2312.04455.

Nilesh Dhulshette, Sapan Shah, and Vinay Kulkarni. 2025. Hierarchical repository-level code summarization for business applications using local llms. arXiv preprint arXiv:2501.07857.

Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Nanning Zheng, and Furu Wei. 2023. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486.

Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. 2023. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137.

Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, et al. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressaud, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770.

Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zuiru Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325.

Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81–93.

Jia Li, Ge Li, Xuanming Zhang, Yunfei Zhao, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, and Yongbin Li. 2024a. Evocodebench: An evolving code generation benchmark with domain-specific evaluations. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, and Mengfei Yang. 2024b. Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 3603–3614. Association for Computational Linguistics.

Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. 2024. Repoqa: Evaluating long context code understanding. arXiv preprint arXiv:2406.06025.

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shipello. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071.

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530.

Yanlin Wang, Yanlin Wang, Suiquan Wang, Daya Guo, Jiachi Chen, John C. Grundy, Xilin Liu, Yuchi Ma, Mingzhi Mao, Hongyu Zhang, and Zhibin Zheng. 2024. Repotransbench: A real-world benchmark for repository-level code translation. CoRR, abs/2412.17744.

Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862.

Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2023. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453.

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, et al. 2023. Effective long-context scaling of foundation models. arXiv preprint arXiv:2309.16039.
Xi Ye, Fangcong Yin, Yinghui He, Joie Zhang, Howard Yen, Tianyu Gao, Greg Durrett, and Danqi Chen. 2025. Longproc: Benchmarking long-context language models on long procedural generation. arXiv preprint arXiv:2501.05414.

Jifan Yu, Xiaozhi Wang, Shangqing Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hamming Li, et al. 2023. Kola: Carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296.

Kechi Zhang, Ge Li, Huangzhao Zhang, and Zhi Jin. 2024a. Hirope: Length extrapolation for code models using hierarchical position. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13615–13627.

Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024b. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. arXiv preprint arXiv:2401.07339.
A  Rrecision-based results in RQ2

We present the performance of LCLMs on several tasks which can be measured by precision-based metrics in Figure 7.

B  Recall-based performance of LCLMs in moderate-length (16~64K)

We also conduct experiments to evaluate LCLMs on moderate-length (16~64K) codes. As shown in Figure 6, the performances of all LCLMs on the moderate-length scenario need to be improved.

<table>
  <tr>
    <th></th>
    <th>CLP</th>
    <th>CL_SA</th>
    <th>CL_SPA</th>
    <th>SRC_T1</th>
    <th>SRC_T2</th>
    <th>SRC_T3</th>
    <th>LAD</th>
  </tr>
  <tr>
    <td>Mistral-v8.2</td>
    <td>37.7</td>
    <td>52.2</td>
    <td>43.7</td>
    <td>52.3</td>
    <td>26.1</td>
    <td>29.4</td>
    <td>25.3</td>
    <td>48.3</td>
  </tr>
  <tr>
    <td>PPL-3.5</td>
    <td>39.4</td>
    <td>39.1</td>
    <td>48.9</td>
    <td>38.9</td>
    <td>7.1</td>
    <td>17.0</td>
    <td>12.2</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>DeepBase-V2</td>
    <td>39.2</td>
    <td>61.5</td>
    <td>53.1</td>
    <td>48.4</td>
    <td>27.9</td>
    <td>22.6</td>
    <td>29.9</td>
    <td>59.3</td>
  </tr>
  <tr>
    <td>Quetz-3-Coder</td>
    <td>38.6</td>
    <td>69.4</td>
    <td>68.8</td>
    <td>21.7</td>
    <td>7.1</td>
    <td>17.5</td>
    <td>15.2</td>
    <td>13.4</td>
  </tr>
  <tr>
    <td>DeepBase-V5</td>
    <td>37.7</td>
    <td>81.2</td>
    <td>71.3</td>
    <td>75.6</td>
    <td>56.5</td>
    <td>38.5</td>
    <td>37.6</td>
    <td>61.5</td>
  </tr>
  <tr>
    <td>Gemini-1.1-Torch</td>
    <td>38.2</td>
    <td>81.1</td>
    <td>78.4</td>
    <td>68.1</td>
    <td>48.1</td>
    <td>33.4</td>
    <td>38.8</td>
    <td>78.1</td>
  </tr>
  <tr>
    <td>Claude-3.5-Secret</td>
    <td>38.6</td>
    <td>87.7</td>
    <td>83.8</td>
    <td>80.4</td>
    <td>72.7</td>
    <td>17.4</td>
    <td>18.9</td>
    <td>71.6</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>38.6</td>
    <td>87.7</td>
    <td>83.8</td>
    <td>80.4</td>
    <td>72.7</td>
    <td>17.4</td>
    <td>18.9</td>
    <td>71.6</td>
  </tr>
</table>

Figure 6: Performance of LCLMs on all tasks within 16~64K code length.

C  The performance of LCLMs on DCC and CCI tasks

Given a modified code unit and its original unit, CCI requires LCLMs to identify other units that are affected by the given code unit changing. We use tree-sitter to analyze dependency relations and identify affected units. We invite two developers to check the affected units. For the DCC task, given a code unit name and its specific inputs, it requires LCLMs to output lines that are executed based on the specific inputs. We use BLEU as the evaluation metric. From Table 4, we can find that the dynamic code characteristics task and the code change impact analysis task are difficult for LCLMs.

D  Case Study

Figure 8 shows a representative error case in the dependency relation analysis task. We can find that the output of GPT-4o extracts an error code unit "stream_async" that is confusing to correct invoked function "stream" since the two code snippets have similar structures.

Figure 9 demonstrates a generated result of GPT-4o in the semantic relation extraction task. We can observe that the output contains an error “delete” function which has opposite functionalities to the anchor input, i.e., the given natural language description.

E  Instructions in Our Benchmark

We present the instructions employed in diverse tasks. Each instruction has undergone iterative refinement to ensure that different models can not only achieve relatively high metrics but also produce outputs that appear satisfactory and are applicable to real-world development scenarios when using these instructions. These instructions standardize the output format of the models (even though some models may not output strictly in accordance with these specifications), facilitating the parsing of the streaming output of the models using regular expressions during the evaluation process.
Table 4: The performance of LCLMs on DCC and CCI tasks.

<table>
  <tr>
    <th></th>
    <th>Qwen2.5-Coder</th>
    <th>DeepSeek-Coder-V2</th>
    <th>CodeLlama</th>
    <th>Phi-3.5</th>
    <th>Mistral-v0.3</th>
    <th>DeepSeek-V2.5</th>
    <th>Claude-3.5-Sonnet</th>
    <th>Gemini-1.5-Flash</th>
    <th>GPT-4o</th>
  </tr>
  <tr>
    <th>CCI</th>
    <td>24.62</td>
    <td>40.81</td>
    <td>47.91</td>
    <td>27.25</td>
    <td>55.92</td>
    <td>68.07</td>
    <td>42.39</td>
    <td>70.34</td>
    <td>67.31</td>
  </tr>
  <tr>
    <th>DCC</th>
    <td>53.13</td>
    <td>57.58</td>
    <td>67.88</td>
    <td>57.08</td>
    <td>38.46</td>
    <td>72.51</td>
    <td>52.30</td>
    <td>83.08</td>
    <td>57.40</td>
  </tr>
</table>

Figure 7: Performance comparison across long code lengths on tasks which can be measured by precision-based metrics. (grey blocks indicate unavailable configurations). The rate of performance degradation exhibits task-specific and model-specific patterns.
You're an expert Python programmer...Extract all functions and methods that the list function calls from the long Python source codes.
...
Do not include any explanations or additional text.

The long Python source codes are as follows:
...
def stream(
    ...
) -> Iterator[ExecutionInstance]:
    ...
    page = self.page(
        date_created_from=date_created_from,
        date_created_to=date_created_to,
        page_size=limits["page_size"],
    )
    return self._version.stream(page, limits["limit"])

...
async def stream_async(
    ...
) -> AsyncIterator[ExecutionInstance]:
    ...
    page = await self.page_async(
        date_created_from=date_created_from,
        date_created_to=date_created_to,
        page_size=limits["page_size"],
    )
    return self._version.stream_async(page, limits["limit"])

...

The programming requirement of the list function is:
...
def list(
    ...
) -> List[ExecutionInstance]:
    """
    Lists ExecutionInstance records from the API as a list.
    ...
    :param datetime date_created_from:
    ...
    :returns: list that will contain up to limit results
    ...

(a) Input

#### function_1 #####
def stream_async(
    ...
) -> AsyncIterator[ExecutionInstance]:
    ...
    page = await self.page_async(
        date_created_from=date_created_from,
        date_created_to=date_created_to,
        page_size=limits["page_size"],
    )
    return self._version.stream_async(page, limits["limit"])

(b) Output

Figure 8: For the dependency relation analysis task, the output of GPT-4o extracts a error code unit “stream_async” that is confusing to correct invoked function “stream”.
You're an expert Python programmer...Identify and extract the five functions or classes that are most semantically similar to the __call__ function from the long Python source codes.
...
Do not include any explanations or additional text.

The long Python source codes are as follows:
...
def create(self) -> AssistantsKnowledgeInstance:
    """
    Create the AssistantsKnowledgeInstance
    ...
    ...
    return AssistantsKnowledgeInstance(
        self._version, payload, assistant_id=self._solution["assistant_id"]
    )
...
def delete(self) -> bool:
    """
    Deletes the AssistantsKnowledgeInstance
    ...
    ...
    return self._version.delete(
        method="DELETE",
        url=self._uri,
    )
...
def forward(
    ...
) -> Tuple[torch.Tensor, torch.Tensor]:
    """
    Applies the poison noise and compute the loss with respect to the target gradient.
    ...

The programming requirement of the list function is:
...
def __call__(self, id: str) -> AssistantsKnowledgeContext:
    """
    Constructs a AssistantsKnowledgeContext
    :param id: The knowledge ID.
    ...

(a) Input

...
######
Top-3 ######
def delete(self) -> bool:
    """
    Deletes the AssistantsKnowledgeInstance
    ...
...
######
Top-5 ######
def create(self) -> AssistantsKnowledgeInstance:
    """
    Create the AssistantsKnowledgeInstance
    ...
...

(b) Output

Figure 9: For the semantic relation extraction task, the output contains an error “delete” function which has opposite functionalities to the anchor input, i.e., the given natural language description.
(a) Code Unit Perception

You’re an expert Python programmer. Identify all defined Python functions and extract their function names from the long Python source codes.

Only extract the Python function names that are defined in the long Python source codes.
Do not extract the Python function names that are mentioned but not defined in the long Python source codes.
Do not extract the Python function names that are defined in the classes of the long Python source codes.
All extracted Python function names are stored in a list Function_names according to the order these function names appear in the long Python source codes.

Make sure the output is strictly in this format:
“
Function_names = [Function_name_1, Function_name_2, ..., Function_name_k]
”
Do not include any explanations or additional text.

(b) Code Unit Data Flow Analysis

You are an expert Python programmer. Identify the {Target_function_name} function from the long Python source code and extract the lines where the value of the {Target_variable_name} variable changes within the {Target_function_name} function.

Only extract lines where the value of the {Target_variable_name} variable changes.
Do not extract lines where the {Target_variable_name} variable is referenced but its value does not change.

Return the extracted lines according to the order they appear in the {Target_function_name} function.

Make sure the output is in the following format:

“
##### The extracted code line ######
{extracted_code_line}
##### Line number ######
{line_number}

##### The extracted code line ######
{extracted_code_line}
##### Line number ######
{line_number}
...
##### The extracted code line ######
{extracted_code_line}
##### Line number ######
{line_number}
”

Do not include any explanations or additional text in the output.
(c) Code Unit Semantic Analysis

You’re an expert Python programmer. Understand the long Python source codes. Identify and extract the function that satisfies the given programming requirement.

Extract the entire source code of the function that satisfies the given programming requirement.
Do not only extract the name of the function that satisfies the given programming requirement.

Extracted the entire source code of the function:
“
def function_name(parameters):
# function body
”
Do not include any explanations or additional text.

(d) Dependency Relation Analysis (T1)

You’re an expert Python programmer. Extract all functions and methods that the {Target_function_name} function calls from the long Python source codes.

Only extract functions and methods from the given long Python context.
Do not extract functions and methods from the Python standard library or third-party libraries.
Only extract functions and methods that the {Target_function_name} function directly calls.
Do not extract functions and methods that the {Target_function_name} function indirectly calls.
Extract the entire source code of functions and methods that the {Target_function_name} calls.
Do not only extract the name of functions and methods that the {Target_function_name} function calls.
Ensure indentation is preserved.

Please follow the format to complete the skeleton below:
“
##### function_1 ######
def function_name_1(parameters):
# function body

##### function_2 ######
def function_name_2(parameters):
# function body

##### method_1 ######
def method_name_1(self, parameters):
# method body

##### method_2 ######
def method_name_2(self, parameters):
# method body
”

Do not include any explanations or additional text.

(e) Dependency Relation Analysis (T2)

You’re an expert Python programmer. Understand the Python programming requirement of the {Target_function_name} function. Extract all functions and methods that the {Target_function_name} function calls from the long Python source codes.

Only extract functions and methods from the given long Python context.
Do not extract functions and methods from the Python standard library or third-party libraries.
Only extract functions and methods that the {Target_function_name} function directly calls.
Do not extract functions and methods that the {Target_function_name} function indirectly calls.
Ensure indentation is preserved.

Do not include any explanations or additional text.
(f) Semantic Relation Extraction (T1)

You’re an expert Python programmer. Identify and extract the functions or classes that are most semantically similar to the {Target_function_name} function from the long Python source codes.

Extract the entire source code of functions and classes that are the top-5 semantically similar to the {Target_function_name} function.
Do not only extract the name of functions and classes that are the top-5 semantically similar to the {Target_function_name} function.
The order of extracted five functions or classes is in order of decreasing similarity to the {Target_function_name} function. Ensure indentation is preserved.

**Do not extract the target function {Target_function_name} itself.**

Please follow the format to complete the skeleton below:

"######## Top-1 #######
def name_1(parameters):
# function body

######## Top-2 #######
def name_2(parameters):
# body

######## Top-3 #######
def name_3(parameters):
# body

######## Top-4 #######
def name_4(parameters):
# body

######## Top-5 #######
def name_5(parameters):
# body
"

Do not include any explanations or additional text.
(g) Semantic Relation Extraction (T2)

You’re an expert Python programmer. Understand the Python programming requirement of the {Target_function_name} function. Identify and extract the functions or classes that are most semantically similar to the {Target_function_name} function from the long Python source codes.

Extract the entire source code of functions and classes that are the top-5 semantically similar to the {Target_function_name} function.
Do not only extract the name of functions and classes that are the top-5 semantically similar to the {Target_function_name} function.
The order of extracted five functions or classes is in order of decreasing similarity to the {Target_function_name} function. Ensure indentation is preserved.

**Do not extract the target function {Target_function_name} itself.**

Please follow the format to complete the skeleton below:

“
#### Top-1 #####
def name_1(parameters):
# function body

#### Top-2 #####
def name_2(parameters):
# body

#### Top-3 #####
def name_3(parameters):
# body

#### Top-4 #####
def name_4(parameters):
# body

#### Top-5 #####
def name_5(parameters):
# body
”

Do not include any explanations or additional text.

(h) Long Documentation Understanding

You are an expert Python programmer. Understand the multiple natural language documentations and identify the one that describes the {Target_API_name} API.

Each documentation is labeled with a sequence number and enclosed between special markers: <BEGIN> indicates the start of a documentation, and <END> indicates its conclusion.

Extract and return only the complete documentation corresponding to the {Target_API_name} API, exactly as it appears.

Please follow the format to complete the skeleton below:

“
# The sequence number
{sequence number}

# The complete documentation
{complete documentation}
”

Do not include any explanations or additional text.

Reviewer paper summaries: This paper introduces LONGCODEU, a novel benchmark designed to systematically evaluate long-context language models (LCLMs) on realistic code understanding tasks. The work addresses critical limitations in existing evaluations, which often rely on synthetic data, lack task diversity, and fail to capture the complexity of real-world codebases (e.g., cross-file dependencies, unit relationships). The benchmark includes 3,983 test samples across eight tasks spanning four dimensions: code unit awareness, intra-unit understanding, inter-unit relationships, and long documentation comprehension. Built from 116 recent Python repositories (post-June 2024 to mitigate data contamination), LONGCODEU features code lengths up to 128K tokens with a natural token distribution. Experiments on nine LCLMs reveal significant gaps between claimed context window capacities and actual performance, particularly in inter-unit reasoning (>32K tokens) and documentation comprehension. The paper provides actionable insights for model selection and highlights key challenges for future research, such as long-range dependency modeling and code-aware attention mechanisms.
Summary of strengths: This study pioneers a comprehensive evaluation framework for Long-Context Language Models (LCLMs) in code understanding, transcending conventional needle-in-haystack tests through three key innovations: (1) Contextual realism using authentic code repositories (average 16K tokens, maximum 128K tokens) with cross-file dependencies and state-preserving functions, (2) A four-tier cognitive hierarchy (unit awareness → intra-unit logic → inter-unit collaboration → documentation grounding) mirroring human code comprehension patterns, and (3) Quantifiable performance analysis revealing critical limitations. Our findings demonstrate severe performance degradation beyond 32K tokens (e.g., 40%+ CodeBLEU drop at 128K context) and systemic deficiencies in inter-unit reasoning (30% accuracy) and documentation alignment. Rigorous temporal filtering of post-training cutoff code samples effectively mitigates data contamination risks. By open-sourcing the first benchmark for long-code understanding with real-world token distributions, this work establishes standardized evaluation protocols and provides practitioners with actionable insights for optimizing LCLMs in enterprise development environments. The community-oriented benchmark enables direct comparison of model capabilities across hierarchical reasoning levels, addressing a critical gap in long-context AI research.
Summary of weaknesses: The study exhibits limitations in model behavior analysis. While it clearly demonstrates performance trends, there remains insufficient exploration of the underlying mechanisms behind models’ difficulties in capturing inter-unit code relationships—such as attention head interaction patterns or positional encoding biases. The evaluation framework currently overlooks dynamic code characteristics like runtime behavior tracing and debugging capabilities, while documentation-related tasks predominantly focus on information extraction rather than generative tasks (e.g., synthesizing functional summaries of code).
Technical implementation details lack comprehensive elaboration. The improvement roadmap for "code-sensitive attention" is described at a high level without granular technical specifications, and critical aspects of tree-based parsing utilities—including handling strategies for anomalous syntactic structures and error recovery mechanisms—are not adequately addressed. Comparative analyses with domain-specific baselines remain underdeveloped, as existing discussions fail to rigorously contrast the performance of dedicated code models (e.g., GraphCodeBERT) against general-purpose LCLMs on LONGCODEU tasks, thereby limiting the generalizability of conclusions.
Regarding data construction strategies, the inclusion of 128K-token samples raises distributional validity concerns. Such extreme-length samples occur infrequently in real-world codebases, and overemphasizing these edge cases may compromise the evaluation’s representativeness for typical development scenarios. Supplementing experiments with results from moderate-length code segments could strengthen the ecological validity of findings.
Comments, suggestions and typos: 1. Add a case study contrasting model failures in inter-unit tasks (e.g., misidentified call chains).
2. Include ablation studies on input chunking strategies (e.g., sliding window vs. hierarchical).
2. Consider adding a task for code change impact analysis (e.g., "If function X is modified, which other units break?").
4. Discuss applicability to non-Python languages.
Overall assessment scores: 4
Metareview score: 4
Metareview: This paper introduces LongCodeU, a comprehensive benchmark designed to assess the long-context code understanding abilities of large language models (LCLMs). The benchmark is structured across four cognitively motivated aspects—code unit perception, intra-unit understanding, inter-unit relation understanding, and long documentation comprehension—each comprising realistic tasks grounded in post-June 2024 code repositories.

Reviewers are uniformly positive, assigning overall scores (4.0) and high confidence (4) across the board. They praise the paper’s real-world relevance, wide task coverage and thorough experiments.  In the rebuttal, the authors respond in detail to each reviewer’s concerns. They conduct additional experiments (e.g., evaluating DeepSeek-R1, introducing new tasks such as Code Change Impact Analysis), provide extended error analyses and visualizations, clarify the evaluation framework, and commit to an open and dynamic benchmarking website. 

Overall, I agree with the reviewers' positive assessment. The benchmark meets an important need by highlighting gaps in current LCLMs’ long-code handling abilities. While some improvements remain—particularly in error analysis, cover non-python language, and ensuring data stays relevant over time—the paper delivers clear and useful results. Therefore, I recommend acceptance to the main conference.
Summary of reasons to publish: LongCodeU is grounded in real-world software engineering practices, addressing a major gap in evaluating LCLMs on code beyond synthetic "needle-in-a-haystack" tasks. The benchmark targets multiple core skills needed for long code comprehension, from basic code unit perception to inter-unit relationships.

Tests with nine popular LCLMs show marked performance drops past 32K tokens, offering actionable guidance for future work on large-context models.