Evaluating Long Range Dependency Handling in Code Generation LLMs
Authors: Yannick Assogba
Donghao Ren
Venue: Accepted by TMLR ()

Abstract: As language models support larger and larger context sizes, evaluating their ability to make effective use of that context becomes increasingly important. We analyze the ability of several code generation models to handle long range dependencies using a suite of multi-step key retrieval tasks in context windows up to 8k tokens in length. The tasks progressively increase in difficulty and allow more nuanced evaluation of model capabilities than tests like the popular needle-in-the-haystack test. We find that performance degrades significantly for many models (up to 2x) when a function references another function that is defined later in the prompt. We also observe that models that use sliding window attention mechanisms have difficulty handling references further than the size of a single window. We perform simple prompt modifications using call graph information to improve multi-step retrieval performance up to 3x. Our analysis highlights ways that long-context performance needs deeper consideration beyond retrieval of single facts within a document.

Published in Transactions on Machine Learning Research (05/2025)
Evaluating Long Range Dependency Handling in Code Generation LLMs
Yannick Assogba Apple
yassogha@apple.com
Donghao Ren Apple
donghao@apple.com com
Reviewed on OpenReview:
Abstract
As language models support larger and larger context sizes, evaluating their ability to make effective use of that context becomes increasingly important. We analyze the ability of several code generation models to handle long range dependencies using suite of multi-step key retrieval tasks in context windows up to 8k tokens in length. The tasks progressively increase in difficulty and allow more nuanced evaluation model capabilities than tests like the popular saclllei--tt---hhsster
test. We find that performance degrades significantly for many models (up to 2x) when function references another function that. is defined later in the prompt. We also observe that models that use sliding window attention mochanisms have difficulty handling references further than the size of single window. We perform simple prompt modifications using call graph information to improve multi-step retrieval performance up to 3x. Our analysis highlights ways that long-context performance needs deeper consideration beyond retrieval of single facts within a document.
1
Introduction
Long context inference is an increasingly important differentiator of LLMs, with model releases supporting
larger and larger context windows (Anthropic] 2023 OpenAT 2023 Google 2024 Jiang et al.) 2023
Touvron
el al.
2023 This is enabled by advances in ellicient attention implementation such as FlashAAten ion (Dao
et al.
2022), Grouped--uery Attention (Ainslie al. 2023) and Paged Attention (Kwon et al. 2023), as
well scaling attention by introducing sparsity via windowing (Child et al.) 2019 Beltagy al.
2020
Applications, such in-editor code completion from tools like GitHub's Copilot (Choi 2024 and Source-
Graph's Cody [Isken & Hill 2024). benefit from long context support as they leverage retrieval-augmenter generation strategies (Gao et al. 2024) to incorporate code snippets from across the user's project into a single prompt. This allows completions to be driven by the user's context rather than just by what the model
learned at training time. Works such as |Zhang et al. (2023) and Shrivastava et al. (2023) have proposed various approaches to improve cross-repository code completion.
These applications rely heavily on the model's ability to effectively everything in its context window, and thus evaluation of long context usage has been topic of interest in the community, Khandelwal et al. (2018) showed that carly LSTM-based language models only roughly modelled information from more distant tokens. Recently, the "needle-in-the haystack" lest, which measures recall for chat-style LLMs (Kamradi, 2024) has grown popular, and a parallel key-retrieval method has been used by Rozière et al. (2024) to measure recall in code generation models. However, we believe that desirable long context inference also includes the ability reason over multiple pieces of information in the input, and needle- predle-in-haaee haystack evaluation approaches do not capture a model's ability to do this. In the paper we contribute:
1

---

Published in Transactions on Machine Learning Research (05/2025)
1. A set of multi-step key retrieval tasks that progressively increase in difficulty and allow evaluation of long-range dependency handling and basic reasoning through function calls.
2. Empirical evaluation of several open source and proprietary code generation models. We find that model performance varies greatly depending on the mumber of steps involved, and the distinctiveness of the target fact compared to the rest of the context. We also discover that the order of function declarations has large effect on model ability to complete these tasks. We further observe that sliding window mechanisms degrade models' ability to resolve references beyond the size of the window.
3. An investigation of methods for improving multi-step recall that rely only on prompt modification using information obtainable via existing non-LLM tochniques, in particular we add annotations of function dependencies in the form of comments.
2 Related Work
Symbolic reasoning: Zhang et al. (2022) sludy neural networks' ability to handle indirection by introducing the Pointer Value Retrieval (PVR) task. They train models to take in a sequence tokens (typically digits), where the first token acts as pointer to the value to be retrieved. The synthetic nature of this task provides full control over the complexity of the problem. [Abnar et al. (2023) extend this work to add recursive steps to the PVR task, evaluate it in the context of vision transformers, and present new architecture that supports adaptive compute.
Multi-hop hop QA: Min et al. (209 study popular multi-hop reasoning benchmark (HotpotQA) and find that many of the questions can be answered with single hop reasoning. They find many questions embed contextual info that make the task easier, and that weak distractors may make picking the right answer 'obvious' without need for the desired reasoning steps. Chen & Durrett (2019) further explore dataset design choices to induce multi-hop reasoning.
Long context retrieval: Liu et al. (2024) examine instruction-tuned LLM performance for synthetic key-retrieval tasks as well as multi-document QA and describe a foost-in-the-midde phenomenon where information becomes harder to retrieve if it is not near the beginning or end of the prompt.
Long context dependencies: Yu ct al. (2024) present a code completion benchmark named CoderEval, where problem solutions have varying levels dependencies on code in the surrounding context. They find that model performance for functions with dependencies is significantly worse than performance for standalone functions. In the context of natural-langunge, Kuratov et al. (2024). Yuan et al. (2024) and Levy et al. (2024) develop benchmarks to evaluate how well language models extract and reason over distributed facts in the presence of noise text. Hsich et al. (2024) develop a rich set of tasks to evaluate long context. performance with different dependency structures in natural language.
Chain-of toohhught prompting: Wei et al. 2023) and Kojima et al. (2023) explore prompting strategies to induce multi-step reasoning in instruction tuned models. They find that models perform better on various tasks when prompted output intermediate steps in the reasoning chain.
While most of the existing literature focuses chat-style natural language generation, our work focuses on autocompletc-style code generation, where latency requirements often constrain the use of 'scratchpad' methods such as chain-of-thought to improve reasoning. We situate our work in between the PVR work of Zhang et al. 2022), which is highly controllable but uses a fairly abstract task, and the -edeewiit--eesedennees senchmarking work exemplified by Yu et al. (2024), which more realistic but also makes it more difficult to run controlled exporiments that allow discovery of spocific failure modes. We design multi-step retricval tasks that are similarly centered on ion, but focus on evaluating inference-time limitations of pre-trained code models, and expand the imput design space to include the effects of position and ordering, in addition to the number of hops.
We open source the code generate these tasks at http:///hhhhhcccmm//m////aaaaeee
2

---

Published in Transactions on Machine Learning Research (05/2025)
3
Long Context Multi-Step Key Retrieval
3.1 Task Design
To study long-range dependency handling we propose four multi-step key retrieval tasks of increasing complexity (one-step, two-step, three-slep, and concatenation retrieval). These extend the key-retrieval task in Rozière et al.] (2024) and test models' ability to integrate multiple pieces of information spread throughout
a long context window to make a completion.
(a) One Step
(b) Two Step
(c) Three Step
(d) Concatenation
def key() O): return "xdfgew"
def value (): return "xdfgew"
def value_2():
def value_ 1()
return "xdfgem"
return "xdfgew"
#
assert key
def k):
value_1():
def value_2():
return (alue()
return value_2()
return "asdahj"
assert k)
def key return value_10)
def key return value_1
value_2()
assert key
assert
key ()
Key retrieval lasks with increasing levels of difficulty. Function names and return values are randomized in the actual prompt. See Appendix G for complete example.
Figure 1:
One-step retrieval is equivalent to the key-retrieval task in Rozière et al. 2024). We differ from their design by using random strings to construct function names rather than fixed function names, and return values that are string literals rather than integer literals. Using a simple template system, we first generate key function that returns a random string. The model is then asked to complete an assert statement on the return value of this function. All return strings that are 10 characters long and all function names are between 13 and characters long-
In two-step retrieval, the key function calls value function that returns the string. Three-step retrieval adds an additional function call between the key function and the value function that returns the string. In the concatenation retrieval lask the key function calls two value functions and returns the concatenation of their returned strings Figure
To turn each of these into a long context problem we insert varying amounts irrelevant code into the context window, we detail this in Section 3.3 In all cases the assert statement on the result of the key function and is placed the end of the prompt.
3.2 Avoiding Reliance on Parametric Knowledge and Trivial Solutions
To make sure the model not solely relying on parametric knowledge (i.c., knowledge stored in the weights) to solve the task, we construct function names from two or three random sequences of lowercase characters or digits, joined with underscores (e.g., characters (e.g., pczjdfeyxc). This makes it extremely unlikely that such functions exist in the model's training data, and thercfore to complete the task successfully, the model must use the information from the prompt.
Return values are random strings of lowercase
In our early experiments we observed that models may produce trivial solutions such as assert foo() == foo(). While these are technically correct, such solutions prevent us from testing the models' ability to retrieve information from the context. To prevent this, we guide the decoding of tokens to make sure that the response starts as string literaf? As we sample output tokens, we use the prefix_ane wwweeeaallwwwdeeeeeeeo wewlowwd_toeenn tokens functionality in the Hugging Face transformers library (Wolf et al., 2020) to ensure that the output starts
We not able do this guided decoding the hosted GPT4o models, instead add opening quote the prompt, fen_f
3

---

Published in Transactions on Machine Learning Research (05/2025)
with any number of spaces followed by single or double quote. Once an initial quote mark is produced, generation proceeds unrestricted.
3.3 Long Context Construction
To turn the tasks described above into long context inference problems we add irrelevant snippets to the context window from Iwo sources. Motivated by findings in Min et al. (2019) and Chen & Durretl (2019) that show that the presence of good distractor functions are important when measuring multi-step reasoning, we first generate a number of synthetic distractor functions that use the same template as the the key and value functions that contain the task information. Then we sample standalone Python functions from the HumanFval dataset (Chen et al.] 2021 to fill out the context window to desired size. Algorithm describes this dataset generation process in more detail.
Algorithm Generate Long Context Retrieval Tasks
ng +- number of unique key functions
nd + number of synthetic distractor functions
+ maximum number of tokens
"p < maximum number of position combinations
repeat ni times
snippets + Generate key and value function(s) for task
assert < Generate assert statement
irrelevant +- Generate nd distractor functions randomly sampled functions from Human Eval such that TOKENCOUNT((nippp + irrelevant + assert) S
positions + All LEN(snéppets) combinations of integers in 1... (sipppets + irrelevant)
>> If > ny randomly sample np of these combinations
for each position combination in positions do for each permutation of snippels do prompt + irrelevant
Insert each function in snippets into prompt using the positions in position combination Add prompt assert to the list of generated prompts
end
This generates a dataset that allows us to compare the effect of the following variables: position of task related snippets within the input context, relative order of the task-rclated snippets, and the spread of task related snippets. We vary these factors while keeping the irrelevant functions used constant for each key function we generate. To aid reproducibility fix the random seed during generation that the prompt set can easily be re-generated.
In our experiments, we set n& to 100 for one-step task, and 20 for other tasks; ny to 0, 1, and 5; np to oc, 150, 50, 50 for one-step, two-step, three-step, and concatenation tasks respectively (this results in a maximum of 6000 prompts for each condition). We set n, to 2k, 4k and 8k tokens.
3.4 Models
We evaluate these lasks on six open source models: StarCoderBase-113 MaarCoderBase-7B. StarCoderBase 15.5B (Li et 2023), StarCoder2-7B (Lozhkov al. 2024), Mistral-7B-v0.1 (Jiang et al. 2023) and DeepScckCoder-66 6.--drr--.BB--eeee base et al. 2024). We also evaluate GPT-4o-mini and GPT-40 (OpenAl 2024h. The StarCoder family of models are competitive code generation benchmarks and available in number of different, parameter counts. StarCoder2-7B and DeepSockCoder-6, 7.--BBaas models have been pre- rre-trained on "repository level" data with the explicit aim of improving performance in contexts where there are dependencies between code units. Mistral a general purpose LLM that is nonetheless competitive on code generation benchmarks. Similarly the GPT-4o models display strong performance on code generation
2Specifically gpt-44o-mnii-2222 2024-07-18 2012--0-- and got-4o- 21-24--1----200
4

---

Published in Transactions on Machine Learning Research (05/2025)
tasks even though they not specific to code. We use implementations from the Hugging Face transformers library (Wolf et al. 2020) for the open source models.
Table Models evaluated.
Model
Human Eva 3
Max Context Size
Sliding Window Size
Mistral-7B
30.5
8192 / 131k
4096
StarCoder2-7B
35.4
16384
4096
StarCoderBase-11B
15.1 1
8192
N/A
StarCoderBase-715
30.5
8192
N/A
29.3
8192
N/A
DeepSeekCoder-6.. Ba-bes
49.4
16384
N/A
GPT-4o-mini
87.2
128k
N/A
GPT-4o
90.2
128k
N/A
We are primarily interested in seeing how the performance of a given model changes as task difficulty (i.e., number of hops needed) and context size varies. We evaluate multiple models to see how consistent effects are across multiple models.
3.5
Results
3.5.1
Overall Task Performance
Our primary metric is accuracy@k, which is similar to the pass@k metric introduced in Chen et al. (022 the only difference is instead of running unit tests, we simply check if the string literal produced is the expected string. We compute accuracy@3 over 10 generations for each input prompt. Hyperparameters for
generation are in Appendix D)
100%
Mistral-78-v0.1
100%
accuracy@g
80%
StarCoder2-78
60%
60%
StarCoderBase-11
StarCoderBase--2
40%
40%
StarCoderBase-155.
20%
DeepSeekCoder-6.78.bb0
20%
GPT-4o-mini
0%
4k
8k
one-step twe-step three-step
GPT-4o
(a) Task performance by model.
(b) Task performance by prompt length.
Figure 2: Overall task performance. For detailed scores, see Table 5 in appendix.
three-step
concatenation
Mistral-7B-v0.
80%
StarCoder2-78
60%
StarCaderBase-1B
StarCoderBase--6
40%
StarCaderBasz-11.55
20%
DeesSeekCocder-.7B-bearr
GPT-44-mini
5
Distractors
GPT-4o
Figure Effect of number of distractors by task variant. For detailed scores, see Table appendix.
Task difficulty: Figure 2a shows that tasks increase in difficulty in the following order: one-step, two-step, three-step, and concatenation. We note particularly weak performance on the concatenation task. Another
PHumanEval scores for StarCoder* modcls from Loghkov al. (2024). StarcoderBasc-I1
from ht
//wuggingfee..o/bbioee/////trrre www.b/igooe//ttaaoorrrrsss---e Mistral-7B
from Jiang et al. 2023
CPT
from ht
/openaii, www.cooc///aac http//nddx//gptaae advancing fft.cceficcieettintteee
"Mistral-7B trained 8192
reports support up 131k tokens.
5

---

Published in Transactions on Machine Learning Research (05/2025)
factor affecting difficulty is the number of distractor functions. For the more difficult tasks see consistent degradation in performance as distractors are added FFiure 3
Context size: In | Figure 2b we observe a small performance drop as context sizes increases for StarCoderBase- 1B, StarCoderBase- 7B, StarCoderBase- 15B, DeepSeekCoder-6 6B, GPT-4o-mini and GPT-4o. For StarCoder2- 7B and Mistral-8B, which both use sliding window attention, a large drop in performance occurs when moving from 4k to 8k context length, we hypothesize that this is related to the size of the sliding window used and will explore this in more detail in Section 3.5.3)
3.5.2 Incorrect Responses
We analyzed the incorrect responses from models and note a number of distinct failure modes. Firstly a noticeable amount of incorrect generations are return values of distractor functions. Considering the and 5 distractor conditions, approximately 10% of all responses are distractor answers across models (min=1.8%, max=16.2%). Within just the incorrect responses, approximately 20% are distractors (min=10.9%, max-33.4%). Full details are in Table 7 and Table 8 in the appendix.
For the concatenation task common failure mode is returning a partial answer (i.e. one of the two values to be concatenated). Approximately 33.1% of all incorrect responses for this task are partial answers from one of the two strings value functions. See (Table for details.
To get further insight into other incorrect responses we also conduct an edit distance analysis. This allows to see if models mostly getting the answer right and maybe failing just few tokens. We compute Levenshtein distance between model responses and ground truth answers for the incorrect responses and find the mean distance to be 10.97. We note that the length of the expected string is 11 in all tasks. This indicates that when the model is wrong it is generally completely wrong rather than just incorrectly generating one or two tokens. Details this analysis can be found in Table 10
3.5.3 Effect of Task Snippet Position
We focus the rest of our analysis the five-distractor condition it allows us to good variation in task performance.
2k
4k
100%
Mistral-78-vO.1
accuracy@3
StarCoder2-78
StarCoderBase-11
StarCoderBase--78
40%
StarCoderBase-15.55
20%
DeepSeekCoder-66 78-bbase
0%
GPT-4o-mini
0
0.2 0.4 0.6 0.8
0 0.2 0.4 0.6 0.8
0 0.2 0.4 0.6 0.8 Key Function Position
Key Function Position
Key Function Position
GPT-4o
Figure 4: Effect of key function position on one-step task with 5 distractors. Position defined as index of the first token in the function normalized by the total number of tokens in the prompt, and grouped into bins. is the beginning of the prompt and is at the end. Shaded regions represent 95% confidence intervals
In the one-step retrieval task we see good performance with respect to key function position across the entire context window with two notable exceptions. At the 8k context size Mistral-7B-v0.11 and StarCoder2-7B are unable to perform the one-step task when the key function more than -4k tokens away from the generation site Figure 4 Both models use sliding window attention mechanism with a 4k window size. While this mechanism theoretically allows the model to scale very large context sizes (Jiang et al. (2023) reports a theoretical attention span of 131k tokens), our results indicate that the model fails to retrieve precise information at distance greater than the sliding window.
Mistral-7B-vO.1 allows for changing the sliding window size at run time. To futher confirm the result above, we measured one-step retrieval performance with sliding windows of 2048 and 8192 (in addition the default. 4096) at the 8k context size, and find that the drop in performance relative to key function
6

---

Published in Transactions on Machine Learning Research (05/2025)
one-stop, 8k context
100%
accuracy@3
80%
60%
40%
20%
0%
0
0.2
0.4
0.6 0.8
1
Key Function Position
Sliding Window
2048
4096
8192
Figure 5: Varied sliding window sizes for Mistral-7B-vO.1 on the one-step task with 5 distractors.
position is generally consistent with this result. However note that at the 8192 sliding window size, performance begins to drops when the key function is approximately 60% of the size of the prompt away from the generation site, it however does not drop to zero as in the 2048 and 4096 case, Figure 5) shows this in detail.
The two-step lask has two relevant task snippet positions, Figure 6 shows heatmaps of lask performance vs. both key function position and value function position. We see that performance is worse when the key function appears before the value function in the prompt for the first two models. We call this forward reference and it turns out this has large effect performance in most models. We also observe that performance is generally higher along the diagonal, i.e., when the functions are closer to each other. We elaborate on this further in Section 3.5.4
For three-step and concatenation tasks, it becomes harder to directly analyze position effects, as there are three task-relevant code snippets with independent positions. We instead explore performance by number forward references, and by spread between task-relevant snippets in the following sections.
3.5.4 Effect of Forward References
We define a forward reference as function calling not-yet-defined function (i.e. one that will be defined later in the prompt). We find that forward references have a negative impact on task performance in all models, in most conditions tested. The exceptions to this highlighted in blue in Table 2 and mainly occur in the three-step task where we sometimes observe an increase in performance when going from zero to one forward reference followed by drop going from two to three forward references. [Table 2 shows these results in detail. We see drops as large as 44.9% in models like StarCoderBase-T1 (three-step task at 4k), 19.4% for GPT-4o-mini (concatenation task at 4k) and 12.3% for GPT4o (two-step task at 8k).
3.5.5 Effect of Task Snippet Spread
We saw in Figure 6 that the order of functions in multi-step tasks and the distance between functions affects performance. We define spread as the distance between the first token of the first task-related snippet and the last token of the last task-related snippet, not including the assert statement at the end of the prompt.
In the two-step task we broadly see three behaviors with respect to spread, the GPT4o models show little variation as spread increases. DeepSeekCoder-6.77 7B--Base, StarCoder2-7B and Mistral-7B-v0.11 show performance drops at moderate amount of spreads but perform well when snippets very close to each other or very far from each other. Snippets with maximal spread are located close to the beginning and end of the prompt and result in higher performance. The oth models MarCCodrrBase-111 and StarCoderBase-11333 show a drop in performance as spread increases, but only in the presence of forward references. Figure 7 shows the effect of task snippet spread for StarCoderBast with 4k context length on the two-step retrieval task with results for the other models in the appendix Figuree 10p.
StarCoderBBase7755
DeepScokCoder66. 7B-basc

---

Published in Transactions on Machine Learning Research (05/2025)
StarCoderBase- 7B
2k
4k
8k
Key Function Position
0.8
0.6
0.4
0.2
0.0
0
0.2
0.6
0.8
0.4
0.8
StarCoder2-7B
2k
4k
8k
1.0
Key Function Position
0.8
0.6
0.5
0.4
0.2
0.0
0.2
0.4
0.6
0.8
1
0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
GPT-4o-mini
2k
4k
8k
Function Position
0.8
0.6
0.4
0.0
0.2
0.6
0.8
0.4
0.8
Figure 6: Effect of key and value function position for two-step task with distractors. Top-to-bottom StarCoderBase-P77 StarCoder2-7B, GPT-4o-mini, Color represents accuracy@3 score.
For the three-step task similar, effects are clearly visible at 2k and 4k context size but are not as consistent across different models Figure II] For the concatemation task, the effect is less pronounced as model's performance is generally low regardless of spread and the number forward references and there is not much room for separation Figure 12
4 Improving Retrieval Performance with Call Graph Comments
We saw in Section 3.5.4 that performance degrades for the multi-step tasks in the presence of forward references. Loosely inspired by chain-of-thought prompting (Wei et al.) 2023 Kojima et al. 2023 that encourages models to produce intermodiate token representing sub-parts the problem, we considered if injecting information about function call relationships into the prompt would help performance. This allows to control the amount of additional information about the problem provided and importantly, where il
8

---

Published in Transactions on Machine Learning Research (05/2025)
Table 2: Model accuracy (%) vs. number of forward references. 5-distractor condition. 95% confidence interval shown adjacent. Performance generally drops as number of forward references increases. Blue indicates exceptions to this behavior, indicating a condition that performs worse than the condition with one more forward reference.
Two-step Task
Context Size
2k
4k
8k
# Forward References
Mistral-7B-v0.11 StarCoder2-7B
60. 1111
1112
46.
22.5m1.a
8. 0.0.
9.2+:.a
73.51s.1
67.6
69.4
5_..
28. 18..
26.8111.
StarCoderBase-
55.319.4
39. 49.7
49.711.0
33.9
65. 35.6
39.3
StarCoderBase-77
90.011.4
38, 11111
87 87.211.4
34.1214
76.022o.4
37 37-7
StarCoderBase- 1..5
85.5
63.4
82.3+114
49. 29.2
63. 43.44
28.3+tia
DeepSeeCCoder-6.7BBBB.
83.7
75.2
74.4
67. 67.72144
70.2
58.5ttia
GPT-4o-mini
98.411.4
95.7214
91.411.
87 87.2
87 .7.44
80.321.4
GPT-4o
99.81c..
98.3
98.61n.
93.621a
36.221.2
83.911.a
Three-step Task
Context Size # Forward References
2k 2
4k
8k
0
2
2
Mistral-7B-vO.. StarCoder2-7B
28. 142.1
24.4
2+2.
19.8822.
22.3+1.:
16.9 14.1
2.7+2.
11.7
10. 10.3+2.1
61.5tta
221.
48. 12s.a
37. Stt.0
14.4
23. 1io.a
16.0
StarCoderBase-I1
32.313.0
29. 22.1
18.1227
29. 53..
28.5
14.911.a
19.82za
23. 23.8114
8.217.a
StarCoderBase-7BB
63. 4+2.0
41.3#0.
18.8+1.7
65. 4+1.
42 22.0+a.9
20. 10.
41.9
35. 6+o.a
15.9+1.
Star CoarCodrrBase-- 155
57.2+2.4
50.6+11
37.2+2.4
54, 0+2.4
39.7+1.a
24.9+2.4
34.5+1a
21.0+1.
10.5+1.
DeepSeekCoder-6.7 B-abbae
69.4+2.4
79.1
56. 2
2+22+
61.3+1.2
52.6+1.
42.9+2.
50.5+1.
40.1+1a
GPT-4o-mini
68.011.2
81.821.
70.9
63.3
67.121:
62.8111.
39.422.
66.321.1
50.411.a
GPT-4o
96.742.2
96.7.1.2
89.621.4
88.7+2.0
91.3#1.:
86.6
85.
91.021.1
86.8a1.e
Concatenation Task
Context: Size # Forward References
2k 2
4k
8k
0
Mistral-7B-v0.1 StarCoder2-7B
11.8+La
4.3+Ls
3.1+1.
2.4+12
0.6+o.a
0.6+0.8
0.8+1.0
0.2+0.6
0.2+o.a
17.710.4
14.7
7.9
6.440.6
6.410.7
4.2
1. 8.0.
0.720.9
0.74:.2
StarCoderBaso--11
1.621.
0.3418
0.0a1.
0.6
0.142.0
0.0
0.0.2.0
0.0
0.0
StarCodcrBaso-7BB
13. 13.9
9.340.7
4.0
2. 2.0.6
1.6#0.2
1.2=6.2
0.0.
0.6
0.3
StarCoderBase- 15.5B
22. 91.
14.221.3
11.02s.a
10. 120.
6. 6.11n.n
5.8
6.8 8.8.s
4.5go.a
2.61
G.epSeekCodeer
40.321.
24.721.
20.221.1
14. 221.1
8.6an.
7.2
9, 62n.n
4,62
3.210.6
GPT-4o-mini
79.8 8.n.4
71.2
63.6gs.e
64.9 9.s.
62.7
45.61..0
46. 1is.
35, 41...
27.720.0
GPT-40
98.5+1.2
96.7=1..
97.3+o.e
97.4+t.s
96.0+a.4
94.9+o.3
93.6+e.a
91.7+a.2
88.9
appeors. So unlike chain-of-thougle prompting, do not allow the model to produce extra tokens but rather inject 'extra' tokens into particular areas of the prompt.
We focused on our most difficult tasks, namely the three-step and concatenation retrieval task with 5 distractors using the same models as in our previous experiment.
4.1 Experiment Setup
We add comments above cach function that contain lightweight description of the call graph associated with that function. We construct these comments by annotating which functions are "called by" other functions, annotating which functions "call" given function, combination of both. In the three-step retrieval task we can also do this transitively and add annotation of all the functions that will be called to compute the result of a given function (as opposed to just the next function in the chain).
We considered two templates for these comments, in one variation ("names only") we simply included a comma-scparated list of function names and no other information. This has the effect of introducing the tokens for the referenced function before appears as a call target, though without any indication that it is even function name.
9

---

Published in Transactions on Machine Learning Research (05/2025)
StarCoderBase-777 two-step
three-step
concatenstion
Forward Refs
accuracy@3
60%
40%
20%
0.2
0,4
0.6
0.2
0.4
0.6 Spread
0.2
0.4
0.5
Spread
Spread
6 two-step
three-step
concatenation
100%
Forward Refs
accuracyes
80%
0
40%
20%
0%
0
0.2
0.4
0.5
0.8
0.2
0.4 0.5 Spread
0.8
1
0.2
0.4 Spread
0.6
0.8
Spread
Figure 7: Effect of spread (normalized the number of tokens the context) for StarCoderBase-7BB (top) and DecpSockCoder66. 7B-base (bottom) with 4k context size, 5-distractor condition. Shadod regions represent 95% confidence interval. For data on other models and context sizes, see Figure 10 Figure 11 and Figure 12
In the other template ("full sentence") we use the phrases: "This function is called by and "This function calls followed by the comma-separated list of function names. See Appendix H for examples.
4.2 Results
Table 3: Call graph comment performance across all models. Accuracy@0
Comment Type
Template
calls
called-by
both
Names Only
46.9
47.8
72.9
69.3
Three-step
Full Sentence
46.9
55.3
74.2
76.8
Names Ouly
23.3
24.6
29.5
29.5
Concatenation
Full Sentence
23.3
27.6
32.8
36.4
We observe that the addition of call-graph comments has a positíve effect on task performance. Table 3 shows a ~1.5x improvement on the concatenation task, and a -1.6xx improvement on the three-step retrieval task with the addition of call-graph comments aggregated across models. We do note that StarCoderBase- 1B sees no improvement on the concatenation task. See Appendix C for detailed results.
Full sentence vs. function names only: The full-sentence template performs better than the names-only version Table 3). This suggests that the models are able to take some advantage of the more complcte natural language description and that there may be room for further improvement by tuning the template. However it is notable that simply including the names of the referenced functions before their function definition has such a large performance boost. We focus the rest of our analysis on the full-sentence condition as the best performing template.
Table 4: Call graph comment performance by depth across all models. Accuracy@3
# Forward References
Comment Type
None
Next Hop
Full
48.6
70.4
73.9
48.8
71.9
79.0
37.8
58.9
71.0
10

---

Published in Transactions on Machine Learning Research (05/2025)
Call graph directions: Table 3 shows that while most of the benefit comes from the "X is called by Y type of comment, using both directions produces the best improvement overall.
Full graph vs. next-hop comments: We noted earlier that in the three-step case we add comments describing the full call graph (i.e., up two steps away). We experimented with only adding comments about the neat function in the call graph and found that the full graph version performs best TTbble 4).
StarCoderBaso-7B
100%
none
three-step
80%
calls called-bb
60%
both
40%
20%
2k
4k
8k
2
100%
concatenation
80%
60%
40%
0.2
0.6
0.8
Context Size
Spread
Forward Refs
GPT-4o-mini
none
three-step
80%
calls
60%
called- by
both
40%
20%
0%
2k
4k
8k
1
2
100%
concatenation
80%
60%
40%
20%
0.2
0.6
0.8
Context Size
Spread
Forward Refs
Figure 8: Eflect of call graph comments on StarcoderBase-7BB (top) and GPT-4o-mini (boltom) performance across context sizes, spread of task snippets and number of forward references. Shaded regions represent 95% confidence interva
Context size, spread, and forward references: Figure 8hshows that call-graph comments improve perfor- mance across all context lengths, number of forward references, and task-snippet spreads for StarCoderBase- 7B (this trend holds for other models where the call graph comments make difference). Full plots for all the models are found Appendix C. We note that with respect spread for the concatenation task, most of the improvement appears when the task-relevant snippets closer together.
11

---

Published in Transactions on Machine Learning Research (05/2025)
5 Discussion
Our findings align with those of Yu et al. (2044), who show that over 70% of functions in popular open-source repositories are "non-standalone" and report 48% average performance drop when models are evaluated on such functions. Their report underscores the need to evaluate the ability of models to reason over interdependent code units. Our use of synthetic code enables us expand the context beyond the 1K token limit in their benchmark and gain fine-grain control over the relations being tested.
Our results also help distinguish between long-content support and the ability to handle long-range dependencies. We find that models were able to perform well at the simpler one-step recall task over their entire context lengths Figure 2ap, but performance degrades as the number of steps required increases. This is further exacerbated by the presence of dist which we believe are important for appropriately pressure testing the ability of models to do the required reasoning (Figure 3).
We also observe architectural choices that appear to make tradeoff between efficient support of long-contens inputs and capturing long-range token relationships. This is illustrated by models like Mistral and StarCoder2, whose use of sliding window attention greatly expands their supported context sizes, however our experiments show that when distance between relevant snippets is greater than the sliding window size used by these models, performance degrades quickly. Our tasks require precise recall over a long distance and it appears to be difficult to pass on information from earlier sliding windows in a way that allows successful completion of the task. As different approaches to scaling attention to larger context windows proliferate, practitioners should take care to understand and mitigate these effects if their task relies on precise recall of long-range dependencies. In the context of code generation, builders of retrieval augmented generation systems may find it desirable to rewrite code to bring code snippets that depend on each other closer together.
Our findings shed light on specific factors influencing the fragility of multi-step reasoning in large language models, including the presence of distractors, function ordering and architectural constraints such as sliding- window attention.
We note that models we tested struggled even more in the presence of forward references Table 2 In this situation all the information in the prompt is the same, but is presented in different order. Humans and compilers are able to resolve this problem and it would be ideal if models could do the same. While we do not know what causes this behavior, one hypothesis is that the causal attention used during training favors attending backwards in the context window and propagating information from carlier to later tokens.
While we leave the discovery of the mechanisms that underlie such behavior to future work, we do observe useful progress in the field that could be brought to bear. While there is still debate on whether examining attention patterns in model can provide faithful explanations (Jain & Wallace 2019 Wiegreffe & Pinter 2019), recent progress in mechanistic interpretability is developing tools to closely examine components involved in resolving relationships between tokens and the mechanisms transformers use to encode them (e.g. Elhage et al. (2021)). [eernandez et al. (2024) find that a subset of natural-language entity relations can be modeled with a single linear transformation. Brinkmann et al. (2024) analyze mechanisms involved in symbolic multi-step reasoning in transformers using synthetic tree-traversal task and describe mechanism such as "backward-chaining" and "onc-step" lookahcad in the models they train. Such tochniques could be applied to larger pre-trained models to understand the failure modes in resolving forward references that we observed in this work.
Finally we find that we can add information to the prompt that may short-circuit whichever mechanisms are involved in resolving function references. We see that introducing just the lokens of referenced functions before they are used in a function definition greatly improves performance on the more difficult multi-step tasks in all cases, especially where forward references are present (Table 3). We believe these scenarios provide a rich testbed to better examine how information flows across tokens particularly in scenarios where models must 'plan' for the future.
12

---

Published in Transactions on Machine Learning Research (05/2025)
6 Limitations
Our experiments uses synthetically generated functions that not realistic as real code in the wild. Our tests only use top level functions composed via function call chains and concalenation. Nor do we test more complex structures of indirection such as recursion, branching control flow, or abstractions that cause functions in the call graph to be reused (c.g. helper functions used by multiple functions). These may introduce additional retrieval challenges or sensitivity to ordering not captured by our current setup. Furthermore our use of randomly generated function names results a setting that likely out-of-domain. Models may perform better with more realistic identifiers drawn from natural codebases.
We also evaluate only decoder-only causal transformers. Models with recurrence bidirectional attention may behave differently and merit separate study, particularly with regard to the forward reference finding,
Suggestions such as re-ordering functions to bring dependent code closer together or adding annotations like call-graph comments would need to be tested in real aaarieval-nggeented code generation environments and should be seen as exploratory; we are not recommending these approaches over others that may improve performance on these tasks (e.g. fine tuning).
7
Acknowledgements
We thank Alan Leung, Titus Barik, Dominik Moritz, Samira Abnar, Forrest Huang, Jeff Nichols and Barry-John Thcobald for their helpful feedback and support.
References
Samira Abnar, Omid Saremi, Laurent Dinh, Shantel Wilson, Miguel Angel Bautista, Chen Huang, Vimal Thilak, Etai Littwin, Jiatao Gu, Josh Susskind, and Samy Bengio. Adaptivity and Modularity for Efficient Generalization Over Task Complexity, October 2023. URL http:///rrxiv.or/abb//////// arXiv:2310.08866 [cs].
Joshua Ainslie, James Lee Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training Generalized Multi Query Transformer Models from Multi- Checkpoints, December 2023. URL
13245 arXiv:2305.1324 13245 [cs).
Anthropic. Introducing 100K Context Windows, May 2023. URL https://ww.aattnpii.ccco////sss context-windows
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Long-Document Transformer, December 2020. URI.
05150 arXiv:2004.05150 [cs].
Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. A Mechanistic Analysis of a Transformer Trained on Symbolic Multi-Step Reasoning Task. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 4082-4102, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/vv /v1//v/vv/// 1o024..iniing--ae. ac1.242. URL nttps::/aclanttol.Ba www.nnhhoooo.ogoogooeoooooo wtw./////222222// findings acl. 242/
Jifan Chen and Greg Durrett. Understanding Dataset Design Choices for Multi-hop Reasoning. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4026-4033 Minncapolis, Minncsota, June 2019. Association for Computational URL
N0.1865/vv1/N19-111 Linguistics. doi: 10. 10.18653/ v1 1
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krucger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet,
13

---

Published in Transactions on Machine Learning Research (05/2025)
Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish; Πlya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained on Code, July 2021. URL arXiv:2107.03374 [cs].
wtw.g//aa////.....// 03374
Rewon Child, Scott Gray, Alec Radford, and Πya Sutskever. Generating Long Sequences with Sparse Transformers, April 2019. URL http://arxiv. http://aaaxiv.ooorrb/////////// wrg/ 1904. 10509 arXiv:1904.10509 [cs, stat].
Nicole Choi What retrieval-auugrmented generation, and what does do for generative A1?, April 2024. URL http:///.....ttto4e444/////t/rte
pues-it--o-foor-ggeratieeee
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Effiiient Exact Attention with IO- Awaaenesss, June 2022. URL http::/arxx..org/aatttaa www.gr/////gar////e 14135 arXiv:2205.44135 [cs].
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah, A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, 2021.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval--uugmenter Generation for Large Language Models: A Survey, March 2024. arXiv:2312.10997 [cs].
URL http://aarxiivooogbbbb////ss/ 10997
Google. Our next-gencration model: Gemini 1.5, February 2024. URL https://blegg.oggle/ttcchoooool ai aa// -earr-moomdeeeuurrr---------
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. DeepScek=Coder: When the Large Language Model Meets Programming The Rise of Code Intelligence, January 2024. URL http:://arxivorr//bbs///a org/abs/2401
14196 arXiv:2401.14196.
Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng. Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. Lincarity of Relation Decoding in Transformer Language Models. In Proceedings of the 2024 International Conference on Learning Representations, 2024.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What's the Real Context Size of Your Long- ConteCtt Language Models?, April 2024. URL http:///axxx> www.:/ogg/bb///eeeee 24040006644444444444
arXiv:2404.06654 [cs].
Alex Isken and Corey Hill. How Cody understands your codebase, February 2024. URL https://sourcegrann cody://h--ooyyyddeerr-
your codebaseti
Sarthak Jain and Byron C. Wallace. Attention is not Explanation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 3543-3556, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1 11/033/11/1191133 1357. URL.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Maric-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, October 2023. URL http://arxiv..ot wwtp://wwwwwwww/////////a/ 06825 arXiv:2310.06925 (cs).
14

---

Published in Transactions on Machine Learning Research (05/2025)
Gregory Kamradt. GitHub gkamradt/LLMMM oatrraadtLLLetetteeoooooooaaaaaa April 2024. URL https://gitthbb.co
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp Nearby, Fuzzy Far Away: How Neural
Language Models Use Context, May 2018. URL http://aaxivv.or//aa//BBB/ 04623] arXiw:1805.04623 [cs].
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language
Models are Zero-Shot Reasoners, January 2023. URL http::::///arii..rrgaaa////////a 11916 arXiv:2205.1100 11966
[cs].
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Tn Search of Needles in 11M Haystack: Recurrent Memory Finds What LLMs Miss, February 2024. URL http:://axxiv.org/// 2402. 10790 arXiv:2402.10790 [cs].
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttentton,, September 2023. URL http:///axxiw.org/aaa abtpg///ab///////// 2309 06180 arXiv:2309.06880 [cs].
Mosh Levy, Alon Jacoby, and Yoav Goldberg, Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modols, February 2024. URL http:///axii..orraab//24220
14848 arXiv:2402.1485 148488 [cs].
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia L.,, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaenc, Mishig Davaadorj, Joel Lamy-Poirier, João Montciro, Olch Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Scan Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder: may the source be with you!, 2023. URL https:
06161 Version Number: 2.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang, Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Compulational Linguislics, 12:157 173, February 2024. ISSN 2307-387X. doi: 10. 10.1162/tacl a 00638. URL
https://dol.org//1 0ww//116taaeeeee _acl_aaccl________ a_00638
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poiriee Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indrancil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas KrauB, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Edoardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Nicolas Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Nicolas Patry, Canwen Xu, Julian McAuley, Han Hu, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. StarCoder 2 and The Stack v2: The Next Generation, February 2024. URL http://aaxiv...orr///////// arXiv:2402.19173 [cs].
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannanch Hajishirzi, and Luke Zettlemoyer. Compositional Questions Do Not Necessitate Multi-hop Reasoning. In Proceedings of the 57th Annual Meeling of the Association for Compulational Linguistics, pp. 1249-4257, Florence, Italy, 2019. 199. Association
15

---

Published in Transactions on Machine Learning Research (05/2025)
for Computational Linguistics. doi: 10.18653/v1/P19-141. URL http::/www..alwvbb.rr//r/tttt// P19-1416,
OpenAI. New models and developer products announced at DevDay, November 2023. URL https://openai
OpenAI. GPT-4o mini: advancing cost-efficient intelligence, July 2024. URL https://openai..om///ddd
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Llama: Open Foundation Models for Code, January 2024. URL http::/arxiv.oo www.v/..rggg//////ee/eee wrgg/rr///b.....//// 12950 arXiv:2308.122 12950 [cs].
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository-Level Prompt Generation for Large Language Models of Code. In Proceedings of the 40th International Conference on Machine Learning, pp. 31693 31715. PMLR, July 2023. URL httph: http://ceeiiins......reeee/////a ISSN: 2640-3498.
html
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Eno-TTnnd Chat Models, July 2023. URL http:://axiii http://wwww..gg///////////////// 09288 arXiv:2307.090 .09288 [cs].
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. ofain-of-Thoughh Prompting Elicits Reasoning in Large Language Models, Jamuary 2023.
URL http:://txttii http://aaaaiiigga.a///// www.aagg/////eeooooo....o.e.eese 22///20...222...///// 2201. 11903. arXiv:2201.11903 [cs].
Sarah Wiegreffe and Yuval Pinter. Attention is not not Explanation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 11 20, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 1002. URL httpsss/aaclantiollgg.or///////
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shlcifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFacel Transformers: State-of-the-art Natural Language Processing, July 2020. URL http:://axiiii www./rrg/////eoeee/ooooeeeeeeeee 1910 03771 arXiv:1910.03771 [cs].
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guanglai Liang, Ying Li, Qianxiang Wang, and Tao Xie. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engincering, pp. 1-13, February 2024, doi: 001145//859750333623333 URL http://aaxiv.orr//bb//22.. 00288 arXiv:2302.00288 (cs).
16

---

Published in Transactions on Machine Learning Research (05/2025)
Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. LV-Eval: A Balanced Long-Context Benchmark with Length Levels Up to 256K, February 2024 URL http://arxiv, wttp://orrg/abb/222.00555555 arXiv:2402.06138 [cs].
Chiyuan Zhang, Maithra Raghu, Jon Kleinberg, and Samy Bengio. Pointer Value Retrieval: A new benchmark for underslanding the limits of neural network generalization, February 2022. URL http:
//axxiv. w/arxiv.org/abs// /bw/ 12580 arXiv:2107.12580 [cs, stat].
Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repasitory-LLvel Code Completion Through Iterative Retrieval and Generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 2471-2484, Singapore, December 2023. Association for Computational Linguistics. doi: 111/8653///22223.o// 2023. 123.emmlp-main. 151.
151. URL
A Appendix
A Detailed Results: Experiment 1
We include detailed tables of accuracy@3 scores (percent scale). Table 5 context length (2k, 4k, and 8k); Table shows scores by the number of distractors; and Table 2 shows scores by the number of forward references.
shows overall scores by task and
17

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: Overall accuracy@3 scores for each model, task, and context size. Each score in the context length columns the mean across all number of distractors. The "mean" column shows the average score for the three context sizes.
Task
Model
2k
4k
8k
mean
one-step
Mistral-7BB-v0.1
99. 8
99.8.0.6
46.4+0.5
StarCoder2-7B
000.00+0.2
99.8+0.3
68. 18.1
89.3
StarCoderBase-IB
97.140.5
98.3+0.6
98.5+0..7
98.0
StarCoderBase--7B
100. 0+0.4
99.9+0.3
99.6.0..1
99.8
StarCoderBase-15.35
99. 19.820.6
99.5.0.5
99.0=0.2
99.5
DeepSeeCCoder-6.77BBbae
99. 99.7+0.6
99.2+0.5
98.8=0.44
99.2
GPT-4o-mini
99.9+0.1
99.8.0.1
99.3=00
99.6
GPT-4o
100.0=0.5
99.9=0.4
99. 99.6+0.33
99.9
two-step
Mistral-7B-v0.1
63.140.1
58.3g0.2
16.7+0.2
46.1
StarCoder2-7B
81. 91.940.0
82.620.1
46 18.440.11
70.3
StarCoderBase--1B
73.3+0.. ,1
73.520.1
73. 73.720.1
73.5
StarCodderBase--B
61...
80.1
79..9
80.3
StarCodeerasxc-15551
82. 2a0.0
74.7
64.0a0.8
73.7
DeepScokCoder-6.7B.
90.
08.0+0.1
85.6+0.1
87.9
GPT-44--inn
98.8+0..
95.8+0.2
92.2=0.1
95.6
GPT-40
97.9+0.0
95.5=0.1
three-step
Mistral-7B-vv..1
41.5a0.4
14.7+0.5
StarCoder2-7B
78.9a0.2
75.3a0.3
30.0a0.3
StarCoderBase-1B
$5.940.4 .4
5402.
52.9+0.6
54.7
StarCoderBass-7B
67.0+0.5
66.9+0.5
66.2+0.4
66.7
StarCoder
65 840.
57 57.1
4
56.1
DeepSeckCodder6.7BBBbe.
88 88.3+0.6
83.920.6
79.0=0.6
83.7
GPT-4o-mini
90.8+0.6
86.5.0.6
77 77.8+0.6
85.0
GPT-40
97.9+0.0
95.6.0.5
94.140.6
95.9
concstenation
Mistral-7B-vO.1
17 17.320.1 4
6.720.4
0.7+0.4
8.2
StarCoder2-7B
740.1..1
18.3+0.2
4.540.3
19.2
StarCoderBase-1B
1.7+0.2
0.6+0.3
0.140.4
0.8
StarCoderBase-7B
21.9+0.1 .7
13. 13.4
8.0+0.5
14.4
MarCoderBasee1555
34.4.0.5
21.6.0.5
15.240.6
23.7
240.6
32.5.0.6
24.6.0.6
34.4
GPT-4o-mini
640.6
70.9+0.6
67.240.5
68.9
GPTT400
97.9+0.5
95.5.0.5
92.940.5
95.4
18

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Overall model performance vs. number distractors.
One-step Task
Context Size
2k
4k
8k
# Distractors
5
Mistral-7B-v0.1
100.0
99.9
99.6
100.0
100.0
99.5
48.5
48.2
42.8
StarCoder2-7B
100.0
100.0
99.9
100.0
100.0
99.6
76.3
69.3
59.8
StarCoderBase-13
100.0
93.2
98.4
96.7
98.8
97.0
StarCoderBase-
100.0
100.0
100.0
100.0
100.0
99.7
StarCoderBase 15.5B
100.0
99.3
DeepSeekCoder-6.77
100.0
100.0
100.0
100.0
99.7
96.8
GPT--o-mini
100.0
99.6
100.0
100.0
99.7
99.7
100.0
98.7
99.2
GPT-4o
100.0
100.0
100.0
100.0
100.0
99.9
99.9
99.7
99.4
Two-step Task
Context Size
2k
4k
8k
# Distractors
0
0
1
5
Mistral-7B-v0.11
StarCoder2-TB
99.2
65.5
.7.9
86.8
63.0
62.6
49,2
27.5
StarCoderBase-11
98.5
80.9
47.3
96.2
82.5
41.8
89.5
79.2
52.4
StarCoder
99.2
86.3
64.0
97.9
81.9
60.6
95.4
85.3
56.8
StarCoder Base- 15.5B
93.6
80.9
74.4
89.6
68.8
65.8
82.0
64.3
45.8
DeepScckCoddr-6..7 BBBbbsss
100.0
96.5
79.4
100.0
92.9
71.0
100.0
92.6
64.4
GPT--o-mini
100.0
99.4
97.1
100.0
98.1
89.3
100.0
92.7
83.9
GPI--o
100.0
98.2
99.1
100.0
97.6
96.
99.0
97.4
90.0
Three-step Task
Context Size
2k
4k
8k
# Distractors
1
5
0
1
5
Mistral-7B-v0.11 StarCoder2-7B
81.4
54.0
23.8
61. 61.4
42.3
21.0
19.5
14.7
10.0
95.6
83.0
58.2
94.5
85.4
46.0
60.0
39.5
20.5
StarCoderBase--1
80.7
59.0
28.1
70.9
68.3
26.4
79.6
58.7
20.5
StarCoderBase-77
91.7
68.0
41.2
82.6
75.9
42.3
88.0
77.
33.4
StarCoder BasroodeBBase15....o 15.5B
81.4
66.5
49.5
69.7
62.1
39.6
65.0
49.5
21.5
DeepScckCoder-6.7 7B-basc
100.0
95.3
69.7
99.9
93.6
58.3
99.3
90.0
47.5
GPT-4o-mini
100.0
94.8
77.7
99.8
95.7
64.
98.5
82.3
52.5
GPT-4o
100.0
98.4
95.3
100.0
96.7
90.
99.8
93.1
89
Concatenation Task
Context Size # Distractors
2k
4k
8k
0
5
0
1
5
Mistral-7BB-v0.1
27.0
18.5 5
6.4
12.8
6.2
1.2
1.1
0,6
0.4
StarCoder2-7BB
54.3
36.2
13.4
26.5
23.0
5.4
8.6
3.9
1.1
MaaCoderBase-
2.9
1.5
0.6
1.1
0.5
0.2
0.3
0.1
0.0
StarCoder Basee-BB
37.5
19.2
9.1
26.9
11.7
1.6
16. 1
7.3
0.5
StarCoder Base-
50.5
36.6
16.1
35.6
21.9
7.3
27.
13.3
4.6
DeepScckCoder-6.. BBB-bbb 7B-basc
60.1
50.1
28.4
54.2
33.3
10.0
47.4
20.4
5.8
GPT-4o-min
85.4
78.8
71.5
74.4
84.0
54.4
73.0
62.2
36.4
GPT-4o
97.6
98.7
97.5
93.2
97.3
96.
94.8
92.4
91.4
19

---

Published in Transactions on Machine Learning Research (05/2025)
two-step
three-step
concatenation
Mistral-7B 1--8--00.1
acciracy@3
80%
60%
40%
20%
0%
StarCodeer277
accuracy@3
80%
60%
20%
0%
StarCoderBase- 18 accuracy@3
80%
60%
20%
0%
2
0
1
StarCoderBase-7 7B
accuracy@3
80%
60%
40%
20%
0%
2
0
1
DeepSeekCooder- 6.78 www.bbeettrrrrrrrreeeeeeeee 15.5B
accuracy@3
80%
60%
40%
20%
0%
2
2
Forward References
Forward References
Forward References
80%
accuracyes
60%
40%
20%
0%
2
1
2
GPT-4o-min -ini
accuracy@3
80%
60%
40%
20%
0%
2
0
2
100%
accuracy@3
80%
GPT-40
60%
40%
20%
0%
2
0
2
Figure Effect of forward references by task variant, with 5 distractors. The data is showin Table
20

---

Published in Transactions on Machine Learning Research (05/2025)
2k
4k
8k
100%
Forward Refs
Nistral-78-v0.
80%
20%
0%
0.2
0.4
0.6
0.0
0
0.2
0.4
0.6
0.8
10
0.2
0.4
0.6
0.8
100%
StarCoder2-77
800
20%
0%
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
StarCodertieee
80%
60%
0%%
0.8
0.6
ParCoderBase--7
60%
0%
StarcoderBasr-11..33
100%
60%
40%
0%
0.6 Spread
Spread
Spread
1000%
Deepseeeicoddr---.777
40%
20%
0.6
100%
OPT-40-min|
40%
20%
0%
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.2
0.4
0.6
0.8
100%%
80%
GPT-8o
40%
20%
0%
0.2
0.4
0.6
0.8
10
0.2
0.4
0.6
0.8
10
0.2
0.4
0.6
0.9
Figure 10: Efffect of spread for two step tasks for each model, with distractors.
21

---

Published in Transactions on Machine Learning Research (05/2025)
2k
4k
8k
100%
Histral-7B-vO.
800
60%
20%
0%
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
100%
StarCoder2-77
10%
20%
0%
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
100%
StarCoterBase-11
60%
0%
100%
StarCoderBare-77
80%
20%
0%
0.2
0.4
0.6
0.8
0.2
0.4
0.8
0.2
0.1
0.6
0.9
100%
StaroddeBBlse-1..
40%
20%
0%
0.2
0.4
0.6
0.8
0.2
0.4
0.2
0.4
0.6
0.8
Spread
DeepsBeiicoodee--------0
100%%
60%
40%
0%
GPT-4o-mini
80%
60%
20%
0%
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.9
0.2
0.4
0.6
0.9
1000%
GPT-4o
40%
20%
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
10
0.2
0.4
0.6
0.8
Figure 11: Effect of spread for three step tasks for each model, with distractors.
22

---

Published in Transactions on Machine Learning Research (05/2025)
2k
4k
8K
100%
Forward Refs
Mistral-7B-vo.
00%
40%
100%
StarCoder2-78
60%
40%
0.6
0.8
1.00%
StarCoderRare-11
60%
40%
1.00%
StaroddeRaare-77
40%
0.6
0.8
1.00%
StarColerteet-15e
40%
Spread
Spread
Spread
1000%
40%
20%
0.8
0.6
0.9
1000%
OPT-40-mink
60%
40%
20%
1000%
GPT-40
40%
20%
0.8
0.8
Figure 12: Effect of spread for concatenation tasks for each model, with 5 distractors.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Mistral-7B-v0.1
2k
8k
Function Position
0.8
0.6
0.5
0.4
0.2
StarCoderBase-11
1.0
Function Position
0.8
0.6
0.5
0.2
StarCoderBasse-7B
Sk
1.0
Function Position
0.8
0.6
0.5
0.4
0.2
0.0
0.4
0.6
0.8
0.2
0.4
0.6
0.2
0.6
0.8
StarCoderBaser Ba5e-15.5B 15.5B
2k
4k
8k
1.0
Function Position
0.8
0.6
0.4
0
0.0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Figure 13: Effect of key and value function position for two-step tasks, with 5 distractors, Color represents accuracy@3 score.
24

---

Published in Transactions on Machine Learning Research (05/2025)
StarCoder2-7B
Sk
1.0
Function Position
0.8
0.6
0.5
0.4
0.2
0.0
0.4
0.6
0.8
0.2
0.4
0.6
0.2
0.6
0.8
DeepSeek Coder L.7B-base
2k
8k
1.0
Function Position
0.8
0.6
0.5
0.4
0.2
0.0
0.2
0.4
0.6
0.8
0
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
GP'T-4-mini
8k
Function Position
0.8
0.6
0.5
0.4
0.2
GPT-40
1.0
Key Function Position
0.8
0.6
0.5
0.2
0.6
Figure 14: Effect of key and value function position for two-step tasks, with 5 distractors. Color represents accuracy@3 score.
25

---

Published in Transactions on Machine Learning Research (05/2025)
B Incorrect Response Analysis
Here we present analyses of incorrect responses broken down by model. We also present details of the edit ance analysis.
Table 7: Percentage of distractor answers across all experiments with ristroctors
Prompt Size
2k
4k
8k
Mean
Mistral-7B-v0.11
10.5
9.4
5.3
8.4
StarCoder2-7B
16.7
19.4
12.9
16.3
StarCoderBase-I1
11.5
11.0
14.2
12.2
StarCoderBaas-711
7.7
11.7
9.6
StarCodcrBaso- 15.5B
11.5
11.3
10.1
11.0
D BB-bbae
16.0
17.5
15.1
16.2
GPT-4o-mini
5.2
7.3
11.1
7.9
GPTT-4o
0.9
1.9
2.7
1.8
Table Percentage of incorrect answers that are distractor values
Prompt Size
2k
Sk
Mean
Mistral-7B--v..1
14.3
12.5
6.0
10.9
StarCoder2-7B
30.8
35.0
16.9
27.6
SaarCddrBBac-11 1B
15.9
16.2
22.2
18.1
StarCoderBBas--1]
12.5
15.6
20.8
16.3
StarCoderBase- 15.5B
19.8
17.9
15.5
17.7
DeepSeekCCoder-667BBB- 7.r.derr--..--baee
36.4
34.6
29.5
33.5
GPT-4o-mini
31.0
33.8
35.5
33.4
GPT-4o
25.8
29.1
24.2
26.3
Table 9: Percentage of incorrect responses the concatenation task where the response one of the two strings that should have been concatenated.
Prompt Size
2k
8k
Mean
Mistral-7B-vel
20.7 7
26.7
11.5
19.6
StarCoder2-7B
21.8
33.5
30.0
28.5
StarCoderBase- 1B
44.9
43.0
44.2
44.0
StarCoderBase-77B
40.9
50.9
51.6
47.8
StarCoderBase-155
30.3
33.1
32.5
32.0
DeepSeekCoder-611 6.B.-asB
23.4
30.4
34.7
29.5
GPT-4o-mini
28.1
26.7
32.3
29.0
GPT-4o
13.7
7.6
8.1
9.8
26

---

Published in Transactions on Machine Learning Research (05/2025)
Table 10: Levenshtein Distance for incorrect responses.
Prompt Size
2k
4k
8k
Mean
Mistral-7B-v0.1
10.8
12.3
15.
12.7 7
StarCoder2-7B
10.5
10.5
11.4
10.8
StarCoderBase-1B
9.6
9.4
9.7
9.6
StarCoderBase-7B
9.5
9.7
9.7
9.6
StarCodderBase-55....
11.1
11.9
12.3
11.8
D basee
9.3
9.9
10.2
9.8
GPT-4o-mini
10.0
12.1
10.2
GPT-40
11.4
12.4
11.1
C
Detailed Results: Call Graph Comment Experiment
We include details of model performance (accuracy@3 percent scores) on the call graph experiment.
Table 11: Call graph comment performance by model for full-sentence template.
Comment Type
Task
Model
None
Calls
Called-By
Both
Mistral-7B--v..1 StarCoder2-7B
18.2
16.7 7
33.8
39.6
41.6
57.4
80.2
90.8
Three Step
StarCoderBase-11
25.0
30.5
46.5
47.4
StarCoderBasc-7
39.0
48.7
86.0
87.2
StarCoderBase- 15.5B
36.9
49.5
85.3
87.5
DeepSeekCoder-66 7.---...7.-ee.-
58.5
72.6
86.5
88.2
GPT-4o-mini
64.8
74.2
82.7
85.4
GPT-4o
91.6
95.0
99.0
97.8
Mistral-7B-v0.1
2.7
3.8
5.6
8.9
StarCoder2-7B
6.6
14.3
27.4
31.2
Concatenation
StarCoderBase-
0.3
0.3
0.3
0.4
StarCoderBase-777
16.0
30.2
StarCoder
Gestoderrasee15...1
15.5B
9.3
20.0
33.7
40.6
SonSSockCoder-6.7B--aae
14.7
17.9
27.2
35.8
GPT-4o-mini
54.1
65.1
71.1
73.3
GPTT-o
95.0
95.3
96.1
94.0
27

---

Published in Transactions on Machine Learning Research (05/2025)
Table 12: Call graph comment performance by model for names only template.
Comment Type
Task
Model
None
Calls
Called-Bi By
Both
three-step
Mistral-7B-v0..1
18.2
15.0
36. 1
45.4
StarCodcr2-7B
41.6
47.5
72.0
71.8
StarCoderBase-11
25.0
28.9
51.5
48.6
StarCoder Base- 7B
39.0
35.5
80.1
71.6
StarCoderBasseI15 15.5B
36.9
35.5
80.8
63.0
DeepSockCoder-6... B.r-bderr------aaas-
58.5
60.0
83.6
80.5
GPT-4o-mini
64.8
66.6
81.4
78.5
GPT-4o
91.6
93.1
97.9
94.9
concatenation
Mistral-7B-v0..
2.7
2.3
4.0
4.1
StarCoder2-7B
6.6
8.5
15.4
11.3
StarCoderBase-1
0.3
0.2
0.4
0.3
StarCoderBasse- 7B
3.7
6.9
4.0
11.4
StarCoderBase- 15.5B
9.3
11.2
24.6
19.2
6eepSSeckCodder-.771B---eee
14.7
15.2
23.4
23.8
GPT-4o-mini
54.1
58.2
68.9
70.5
GPT-4o
95.0
94.6
94.9
95.7
Table 13: Next-hop vs. full graph comments performance on three-step retrieval.
# Forward References
Comment Type
Model
None
Next Hop
Full Graph
16.8
39.6
53.6 4
Mistral-7B-v0.11
19.5
29.9
39.8
14.7
17.1
25.3
40.7
83.2
90.9
StarCoder2-7B
43.9
80.5
92.5
33.1 1
61.1
84.3
0
27.2
28.4
32.9
StarCoderBase-1 1B
27.3
1
13.8
29.9
43.4
0
56.9
88.7
92.5
StarCoderBase-
39.6
78.9
91.2
18.4
46.8
66.
48.6
66.6
76.6
StarCoderBase-15..0 15.5B
37.1 1
78.7
89.3
24.2 2
91.5
54.8
84.5
87.9
DeepSeekCoder-6.7BBBB
61.7
86.1
91.0
49.6
66.7
77.7
53.6
70.0
75.2
GPT-4o-mini
68.4
84.3
88.
61.4
78.7
84.7
0
89.8
98.1
97.8
GPT-4o
93.0
98.3
97.9
2
87.6
96.9
97.7
28

---

Published in Transactions on Machine Learning Research (05/2025)
100%
100%
100%
Mistral-7B-vO.
80%
calls
50%
called-hy
40%
both
20%
0%
0%
2k
4k
0.4 Spread
0.6
Context
Forward Refs
100%
100%
100%
StarCollr2-75
80%
80%
80%
60%
60%
50%
40%
40%
20%
20%
0%
2k
4k
8K
0.4 0.6
Context Size
Forward Refs
Spread
StarooderBase 18
100%
80%
60%
40%
40%
40%
20%
20%
20%
0%
0%
0.2 0.4 0.5
0.8
Context
Forward
Spread
100%
100%
100%
StarCodurBaae-77
80%
80%
90%
60%
60%
40%
60%
20%
0%
0%
2k
4k
8k
2
Context Size
Forward Refs
Spread
SuuraaderBine-15.59
100%
100%
100%
80%
80%
80%
60%
60%
50%
40%
40%
40%
20%
20%
20%
0%
0
0.2
0.4
0.6
0.8
Context Size
Forward Refs
Spread
DeepStenkCdderr.. DespSientColkr-..8B-.
100%
100%
80%
60%
40%
20%
20%
0%
2k
4k
8k
0%
0%
0.5
Context
Forward Refs
Spread
100%
100%
100%
GPT-4e-mini
80%
80%
80%
60%
00%
50%
40%
40%
40%
20%
20%
0%
2k
4k
0%
0.2
0.4
0.6
0.8
Context Size
Forward Refs
Spread
100%
100%
80%
80%
GFT-4o
60%
60%
50%
40%
40%
20%
20%
20%
0%
4k
0%
0.4 0.6 Spread
Context
Forward
Figure 15: Effect of call graph comments for three-step lask.
29

---

Published in Transactions on Machine Learning Research (05/2025)
100%
100%
100%
none
Mintral-7B-vO.
80%
80%
80%
calls
60%
60%
00%
called-by
40%
40%
40%
both
20%
20%
0%
0%
2k
0.4
0.6
Context Size
Forward Refs
Spread
100%
100%
100%
Startader2-78
80%
20%
0%
0%
2kc
4k
0.2
0.4
0.5
0.8
Context
Forward Refs
StertoderBase 18
100%
100%
80%
80%
60%
60%
40%
40%
20%
20%
20%
0%
0%
Context Size
Forward
Spread
100%.
100%
100%
DewpSealCoodrre...5 1.7.ddrr-.7.-bane 6.78-bas StarCaderBave-11.. StarCodurBane-
80%
80%
80%
60%
60%
50%
40%
40%
20%
20%
0%
2k
4k
8k
0.2
0.4
0.6
0.8
Context
Forward Refs
100%
80%
80%
60%
60%
40%
20%
20%
0%
Context
Forward Refs
Spread
100%
100%
100%
80%
80%
60%
50%
40%
20%
20%
0%
0%
2k
4k
8kc
0
0.2
0.4
0.6
0.8
Contaxt Size
Forward Refs
Spread
GPt-Ao-mini
80%
60%
60%
40%
40%
20%
20%
0%
0%
0.6
Forward
Spread
100%
100%
100%
80%
80%
80%
GPT-4o
60%
60%
60%
40%
40%
40%
20%
20%
0%
0%
2kc
Context Size
4k
8k
Forward Refs
0.2
0.4
Spreed
0.5
0.8
Figure 16: Effect of call graph comments for concatenation task.
30

---

Published in Transactions on Machine Learning Research (05/2025)
D Hyperparameters for Generation
For all experiments we use nuclous sampling with the following parameters.
Table 14: Generation hyperparameters.
Hyperparameter
Value
Temperature
0.8
Top p
0.95
Top
0
Batch size
Output samples per input prompt
10
E
Confidence Intervals
All confidence intervals (error bars and plus/mimus values) are calculated as 95% confidence intervals of the estimated mean following assumptions of normal distribution and independent samples.
F
Compute Information
We ran all experiments on machines with a single A100 GPU with 80GB VRAM on a cloud provider. Experiments were run over the course of about a month and we estimate ~1000 GPU hours were used including runs that had bugs in them and additional experiments that not part of the paper. We use improve memory usage and latency of generation.
Flash Attention Dao et al. 2022
31

---

Published in Transactions on Machine Learning Research (05/2025)
G Prompt Samples for Multi Step Key Retrieval
def (e5ri__18430044411550
return "wxmbrnpoks"
def (tring_to_nddstex))
***
Given string "text', return its md5 hash equivalent string. "text" is an enpty string, return None.
>>> string_to_nds H"elell world") = "328888arddbbssbb7444444e2a22223
*..
import hashlib
return hashlib. (eriii.ndd5tte) nte(t encode (" ")) hexdigest() text else None
def egllun_467846))
return fwnxln
def count_upperrs))
Given a string s, count the number of uppercase vowels in even indices.
For example:
count_upper ('
count_upper ('abcdefg") returns 0 count_upper d'BBBE") returns
count
for in range (en(s) if stil in "AFTOU";
return count
def solution(Ist):
G*viven a non-empty list of integers, return the sum all of the add elements that are in even positions
Examples
solution([5, 8, 7, 11)
solution 3, 3, 3, 3]) ==> solution[[38, 13, 24, 3213) =2>0
return sun for idx, x in enunerate(Ist) if idx#2==0 and x82==1])
def choose_nun(x, y):
**Thhi function
positive numbers
returns
biggest even integer nunber that is in the range [x, y] inclusive. If there's 's no such number, then the function should return -1.
For example:
choose_nun choose_nun(13 (13, 12) **.
if x
return y
return y:
return -1
return y
daf dsgitsun(s):
***Task
Write a Function that takes a string as input and returns the sun of the upper characters only"
Exanples:
oigitSun("")
digitSun(" ("aitSun"aabB"" >> 131
digitSun ("abeCa")
digitSun ("heeloen""ell => 69
("sataSun => 153
if ":: return return (ch(o( if char, isupper () else for char in s)
def multiply(a, b):
*** Complete function two integers
32

---

Published in Transactions on Machine Learning Research (05/2025)
product their digits. Assune the input is alwoys valid. Exanples:
nultiply(148, 412) should return 16.
nultiply 28) should return 72.
nultiply 1851) should return -11t)ply(14,115) should return 20.
return (a 10) abs 18)
def Light_angle_triaagleee. b. c):
Given the lengths of the three sides triangle. Return True if the three sides form a right-angled triangle, false otherwise.
righee triangle is triangle in which one angle right ongle 90 degree. Exanple:
right_angle_ trinneeerianneee, 4, 5) =s True right_angle_ttiangll 2, 3) == False
return a*a == beb cec or b*b = a*a C*C or ceC == a*a beb
def add edd_elementsaarr, k):
Given non-enpty array of integers and integer k, return the sun the elenents with at nost two digits from the first k clements arr.
Example:
Input: arr [111, ,21,2223 3,, 8..488.5.6..8... =
Output: 24 sun of 21
Constraints:
1. len(arr)
2. <a <a len(arr)
return selen for elen in arr[:k] if len(str(elen)) <=
def 51kfb4_5
return "thwtyqujws"
def agtsin_336111 116199__wwghhb[C0 return iaizjb_184360_4441113
def 2wdpgg_293861_ [) return rbwofb__00331_3811111
def even_odd www.eoddddddddoeeeeeeooeoeeeeeee (nun):
""ivienn an integer. return tuple that has the number of even and add digits respectively.
Example:
even_add_conu count (-12) ==> (1, 1)
even_odd_count (123) (1,
even_count odd_count = (nun)):
if int(() even_count +=1
else: odd_count +=1
(even_count, odd_count)
ninSubarraysuu(nuns))
Given an array of integers nuns, find the mininun sun of any non non-enpty sub sub-array Exanple
minSubArraySun(C2 3, 4, 41)
minSubarraySun (C-1, -2, -31) == -6
s
4= nun
(s 8):
8
sun msx max_sum)
If
max_sun max (-1 for in nuns) nin_sun mmax sun
33

---

Published in Transactions on Machine Learning Research (05/2025)
nun
(wkr1p_/01444_78794220
return vskrby
def rdcxei_1353430):
return eg1lun_467846
def is):
Mrite function that takes integer and returns True this ingeger is cube of sone integer number. always valid.
Note: you may
the
Exanples:
iscube
iscube (2) ==> False
iscube
True
iscube am> True
iscube (0) ==> True
iscube (180) False
return int(round н* (1. / 3))) "
def
"*"The Brazilian factorial is defined as:
trazilian_factorialc n! (n-1) (n-2) ... where n 2
For example: special_factorial())
288
The function will receive integer input should the special factorial of this integer.
...
fact_1 =
special_fact =
for range(1, n+1):
fact_i special_fact fact_1
return special_fact
def 15_equsl__o__su_eeee
"*"Evaluate whether the given nunber n can be written as the sun exactly positive even numbers Exanple
stuuall_t__uu_eeeee == False
is_equal_to__s saraall_tt_uu_eeeeee (6) False
tecquul_te_su__evee (8) == True
*..
return nX2 and
odcxoi_1354433
Listing 1: Sample prompt for two-step retrieval task with 2000 token context size and 5 distractors. rdexoi 135343 is the key function and egllun_467846 467846 is the value function.
34

---

Published in Transactions on Machine Learning Research (05/2025)
H Prompt Samples for Call Graph Comments Experiment
This function calls Swwgtr_32516__11 8622229 This function called by ackwf1_ def 13855)_ return oswgtr_325169_882222
This function calls tapvbp_13857,
and oswgtr oswgtr_325169__662
def aokufl_208971hhwoohh
return 13px3p_138573
This function calls 1hezee_508961 and hjdnw1_724283
def oft0yy_2861380) return (Mezee_506969(
This function is called 13855p_138573 and aokwfl __88111________11111111111_11 fhhoofh def 22w2r__22169_8822298 "kyfgholerg"
This function calls ajobne_ 651008__m.e_ This function is called wwzfoa_984885 def bwecku_868522 868522_nykkyp(()=
gjobmn_65100B_tymmm00
function called by 1hezee_50a966 and ofteyy_286138
def hjdnw1_1242833): return "pwinenzyqh"
This function is called by twecku_ and wwzffa__9488s def gjobme_651ees _stoo___ttttttmmmm) return "axxtrhucug"
This function calls bweckv.880527_nykk1 and _jobne_65111 651888 tymnij
def wwzfoa_9048830 return Beeckw_86052__nykkyyt
assert Ww2Foa_904885 ()
Listing 2: Full sentence call graph comments for three stop retrioval. distractors. HumanEval functions excluded for brevity.
is the key function, @weckw__60050 _nkkypp and 651008 _tmomjjjj_jjjooo are value functions
35

---

Published in Transactions on Machine Learning Research (05/2025)
def 14pxbp_138573()
return 0svgtr__925109_8622290
84pvbp_ 138573, oswgtr_ 0ssggr__2251118__62222
def return 1opvbp_138573()
lhezee_ Ihezee_508969 588969 hjddw1_724283
def oftoyy_286138(): 2881188):
return 1hezen_5es389()
89pvbp_13855 138573, [aaxff_208971_hwmaa
def SaMgtr_325169_82222
return "kyfgholerg"
3jobme_651889_-20 651n88_ttmmij
wwzfoa_9e48s. 904885 def
(68527_nykiyp():
return EJohre__51808_tyymi1
1hezee_598969 508969 oftoyy_286138
1hezee_
def 1jjdml_72222833
"pwinenzygh"
bweckw_ 101527_nykiyp...mm
ww2foa_994485
def _jobme_ 001808__588___111 (0180s_tymei]()
return "axxtrhucug"
@weccw__60527_.0 -nkkip,, 3jobm__651008_tymmi)
def () return Baackk_B60527_nnkkk
Listing 3: Function names only call graph comments for three step retrieval. 5 distractors. HumanEval functions excluded for brevity. wwzfoa 904885 is the key function, 2ecc227 nykiyp and gjobme_651008
wwzzoa_9048855 651008 1e1008_tymmij are value functions
36


Reviews:
[{"review_id": "qUu1j6Wjyn", "reviewer": "Reviewer_bhMM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "-\tThe paper is well written and easy to understand. I only wished a few more experimental details would be explicitly stated (e.g. number of distractors and prompt length for Fig.2 a). I have a few minor suggestions that could improve readability and completeness.\n-\tThe design of the test functions is fitting and enables for controlled experiments.\n-\tThe paper evaluates the models in sound experiments and displays the results mostly in an easy to understand format.\n-\tThe constructed tests are rather simple (call and concatenate). Incorporating some more complex but still easy to control structures like loops or if statements would provide further insights.\n\n\nOverall, I found that the paper to be worthy of publication but the results could be further strengthened (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a set of synthetic problems to test LLMs in a controlled way for their capabilities in handling long range dependencies in code completion scenarios. The paper evaluates 8 models (6 open-source code models and 2 GPT-4o models) on their tests and finds that the code completion models struggle with longer dependencies. The paper also analyzes the influence of context size, number of distractors, position of relevant functions, and kind of errors produced. In the second half of the paper the authors investigate how annotating the relevant functions with call graph information can improve model performance and find that this annotation helps code completion models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "-\tI found Table 3 rather confusing: It is not clear what exactly is displayed in Table 3a (three-step, concatenation, or both). Additionally, the 51.7 should be bold instead of the 49.4. In general, both Tables took me a while to comprehend. I think a single table just displaying the performance of all comment types for both problems and templates would be easier to understand here. I also suggest providing detailed findings like in Table 11 in Appendix C for the names only runs. I see no reason not to do that. \n-\tA way to strengthen results would be to explore more complex structures in the calls (like control structures or loops).\n-\tLastly, how many steps would be needed to notably reduce the performance of GPT-4o.", "broader_impact_concerns": "There are no ethical concerns with this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uBxWy6og8A", "reviewer": "Reviewer_dwJc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Fills a research gap with a balance between task realism and control\n- It is well-written, structured, and easy to follow. In most places a good-quality manuscript\n- The experiments are suitable and provide decent coverage of models, although Llama is a notable omission\n- It will be of value to the community and particularly TMLR readers\n\nWeaknesses\n- The discussion is a weak point, lacking depth particularly in placing the work in context of the related literature, which is completely missing. The related work (section 2) is good, but it would tie the paper together better to refer back to previous findings. In particular, it would be valuable to see this discussed both with respect to Zhang et al. (2022) and Yu et al. (2024)\n- The limitations section is also quite weak. A deeper explanation of the key differences between the synthetic code and real code, along with details on how this might affect the results, would improve it. There are also many limitations not discussed (lack of variety in code tests, realism, models tested, inability to test different context windows for all models), and some of those have clear justifications which should make it easy to detail\n- The GPT4o models were run without the \"allowed tokens\" filter, which puts it at a disadvantage due to a technical limitation (API not supported). However, it would be better to get a sense of the impact of this, either by checking how often GPT4o produces a trivial solution or the performance of other models with this feature turned off (putting them on par with GPT4o). Even an estimate of the order of magnitude of these differences would be helpful\n- It would have been stronger with more models, particularly some llama model (even though they are general purpose models), however this current set is sufficient to demonstrate the claims made\n- There are some more minor issues that can be improved, which I will list below in requested changes", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors evaluate long-range dependency handling in code generation. The authors introduce various tasks with increasing complexity to assess how models use information spread within a prompt. Their experiments with both open source and proprietary models reveal that performance degrades significantly when functions reference definitions that appear later in the prompt. They also find that models using sliding window attention struggle with dependencies beyond a single window size. They also demonstrate that adding call graph information via simple comments can improve multi-step retrieval performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Content:\n\n- Improve the discussion by placing the work in context of related literature, and more clearly state key takeaways\n- Add to the limitations, both in depth and breadth\n- Is there any sense for how much \"prefix_allowed_tokens\" affects general performance of the models? Can we trust this to be negligible? (See also weakness)\n\nMethods:\n\n- End of section three talks about three distractor conditions but I believe there are five. Also, a better clarification between the condition vs the distractor function would be helpful\n- Below Algorithm 1 is a list of the variables to be studied, n_p is not listed, should it be?\n- In the prompt samples in the appendix it would be good to get a description in the caption of what the target and intermediate functions are\n\nFigures and Tables:\n\n - The caption for figures 2 and 3 don't mention this is mean for 10 runs (I think this would be helpful information), also it would be very helpful to state that the variance is minimal and error bars would not be visible – to save the reader from wondering what the variance is and having to look at the appendix tables\n- Describe the shaded region in Figure 4 in the caption\n- The colours in the colourbars for figure 6 need to be fixed\n- Adding the names of the models in Figure 7 (perhaps on the left side) would make it more readable\n- There are no variances in table 2, were these single runs?\n- The statistic used to describe the distributions in appendix table 5 needs to be described (standard dev, standard error, etc)\n\nFormatting/Text:\n\n- The footnotes in table 1 on page 5 actually appear on page 4, I believe they should be on the same page\n- There is a strange gap in the opening line on page 6, it matches with the inset figure on the page before but looks strange on page 7\n- Start of section 3.5.3 \"five distractor condition\" → \"conditions\"", "broader_impact_concerns": "No notable ethical considerations", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QuDSONySFY", "reviewer": "Reviewer_txvE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses:\n\n1) Limited exploration of prompt position effects:\nwhile the study examines snippet position effects, it doesn't fully explore whether placing the assert statement (the task instruction) at different positions within the context might give different results.\n\n2) Missing analysis of attention patterns:\nthe paper could benefit from an analysis of attention patterns to better understand the mechanisms behind the observed performance differences.\n\n3) Missing analysis of other possible settings:\nwhile the paper suggests prompt-based improvements, it doesn't explore retrieval augmentation (many long-context applications use retrieval augmentation), fine-tuning, a few shot setting, or cross-modality testing.\n\n\n\nStrengths:\n\n1) The experiments settings and design:\n- Comprehensive model evaluation: the paper study a diverse set of models, including both open-source (StarCoder family, Mistral, DeepSeekCoder) and proprietary models (GPT-4o variants). \n- Reproducibility;\n- Clear visualization of results;\n\n2) The evaluation methodology: the paper goes further the simple \"needle-in-a-haystack\" retrieval tasks by requiring models to reason through function calls and integrate multiple pieces of information spread throughout the context.\n\n3) The paper doesn't just identify problems but propose and test practical solutions (and the call graph annotation solution they propose requires no model retraining and minimal computational overhead)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on code generation models and how well they handle dependencies spread throughout a long context. Instead of just testing whether models can find a single piece of information (like the popular \"needle-in-a-haystack\" tests), they create more challenging tasks that require models to follow a chain of function calls to arrive at the correct answer.\nTheir experiments reveal some surprising limitations. Most notably, models struggle significantly when they encounter functions that reference other functions defined later in the prompt, a common pattern in real codebases. Models using sliding window attention also hit a performance cliff when dependencies stretch beyond their window size, even though they technically support much larger contexts.\nPerhaps most valuable is their discovery that a simple fix, adding comments that describe function relationships, can dramatically improve performance. This provides an immediately useful technique for anyone working with these models.\n\nContributions:\n- The evaluation framework to properly study where models start to struggle with long-range dependencies.\n- They showed weaknesses in how current models handle code, particularly with forward references and dependencies that span across large distances in the context window.\n- They provided a practical solution: adding call graph annotations (that significantly improves performance without requiring any model retraining or complex processing).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest to change the structure of the paper:\n\nThe authors have this fascinating insight about forward references causing problems, but we don't get a clear theoretical explanation for why this might be happening. Is it related to how attention works in these models? Is it because training data typically follows certain patterns? These questions deserve exploration before jumping into experiment details.\n\nThe solution of adding call graph comments is genuinely clever and practical, but it feels like it comes out without a theoretical setup first: drawing connections to the chain-of-thought work by Wei and Kojima mentioned in the related work section would help readers understand why this approach makes sense theoretically. \n\nThe experiments themselves are well-designed and thorough, especially those progressive difficulty tasks they created. I just wish I had understood the conceptual framework behind them earlier in my reading journey.\n\n\nI'd say this is the only change i'd suggest, and it's a suggestion to strengthen the paper, not a condition for acceptace. I really think though that changing the structure would make the paper look more robust.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a well-executed empirical investigation into how current LLMs handle long-range dependencies in code completion. The evaluation proposed by authors allow to have a strong alternative to the popular benchmark needle in the hay stack in particular for coding tasks. Having more benchmarks to be able to assess model performance on long context task is an important topic.
Decision: Accept (poster)