CALLME: Call Graph Augmentation with Large Language Models for Javascript
Authors: Michael Wang
Kexin Pei
Armando Solar-Lezama
Venue: COLM 2025 (2025)
Keywords: Javascript
program analysis
TL;DR: Handling edge cases in call graph construction for Javascript that cannot be handled with static analysis using large language models.

Abstract: Building precise call graphs for Javascript programs is a fundamental build-
ing block for many important software engineering and security applications
such as bug detection, program repair, and refactoring. However, resolving
dynamic calls using static analysis is challenging because it requires
enumerating all possible values of both the object and the field. As a result,
static call graph construction algorithms for Javascript ignore such dynamic
calls, resulting in missed edges and a high false negative rate. We present
a new approach, CALLME, that combines Language Models (LMs) with a
custom static analyzer to address this challenge. Our key insight is in using
LMs to incorporate additional modalities such as variable names, natural
language documentation, and calling contexts, which are often sufficient to
resolve dynamic property calls, but are difficult to incorporate in traditional
static analysis. We implement our approach in CALLME and evaluate it
on a dataset of call edges that are dependent on dynamic property accesses.
CALLME achieves 80% accuracy and .79 F1, outperforming the state-of-the-
art static analyzer by 30% and .60, respectively. To study the effectiveness
of CALLME on downstream analysis tasks, we evaluate it on our manually
curated dataset with 25 known Javascript vulnerabilities. CALLME can
detect 24 vulnerabilities with only 3 false positives, whereas static analysis
tools based on current call graph construction algorithms miss all of them.

Published as conference paper at COLM 2025
CALLME: Call Graph Augmentation with Large Language Models for Javascript
Michael Wang
Kexin Pei
MIT CSAIL Cambridge, MA 02139, USA mi27950@mit. edu
Department of Computer Science University of Chicago Chicago, IL 60637, USA kpei@uchicago.co edu
Armando Solar-Lezama MIT CSAIL Cambridge, MA 02139, USA asolar@csail wwm.iii... edu
Abstract
Building precise call graphs for Javascript programs is fundamental build- ing block for many important software engineering and security applications such as bug detection, program repair, and refactoring. However, resolving dynamic calls using static analysis is challenging because it requires enumerating all possible values of both the object and the field. As a result, static call graph construction algorithms for Javascript ignore such dynamic calls, resulting in missed edges and a high false negative rate. We present a new approach, CALLME, that combines Language Models (LMs) with a custom static analyzer to address this challenge. Our key insight is in using LMs to incorporate additional modalities such as variable names, natural language documentation and calling contexts, which are often sufficient to resolve dynamic property calls, but are difficult to incorporate in traditional static analysis. We implement our approach in CALLME and evaluate it on a dataset of call edges that are dependent on dynamic property accesses. CALLME achieves 80% accuracy and .79 F1, outperforming the state-of-ther art static analyzer by 30% and .60, respectively. To study the effectiveness of CALLME on downstream analysis tasks, evaluate it curated dataset with 25 known Javascript vulnerabilities. CALLME can detect 24 vulnerabilities with only false positives, whereas static analysis tools based on current call graph construction algorithms miss all of them.
manually
1 Introduction
Call graph construction has been a prerequisite for many critical software analysis tasks, such as code optimization (Fink et al., 2008; Malavolta et al., 2023), bug detection (Brown et al., 2017; Cai et al., 2023a; 2021), taint analysis (Kang et al., 2023), and software maintenance and inspec- tion (Feldthaus et al., 2013). However, dynamic languages such Javascript present unique challenges for call graph construction. The dynamic nature of Javascript alongside the size of sophisticated frameworks such as Reactand ÁngularJS makes call graph construction very dif- Javascript call graph construction (Antal et 2023) has
ficult. For example, recent recent shown that even the best static analyzer only obtains 0.43 detection score, suffering from high false positives and false negatives. Moreover, the majority of the existing static analyzers tested simply ignore the method calls when they are across multiple files (Antal et al., 2023).
A key challenge for constructing call graphs statically is Javascript's flexible object model, which allows properties to be created and deleted at runtime. Specifically, dynamic property accesses, where the property being accessed depends runtime-computed string, are esti-
1

---

Published as a conference paper at COLM 2025
mated to cause 70% of missed edges in static call graph construction algorithms (Chakraborty et al., 2022). Recent static call graph construction algorithms specifically ignore reasoning about most dynamic property accesses due to their runtime-dependent behavior, and thus miss any calls that are computed dynamically from objects (Feldthaus et al., 2013; Nielsen et al., 2021a).
Figure shows an example of dynamic property access, where the property access call is made in line 13 to a function field on the object Analyzing the call requires field-sensitive pointer analysis to determine the points-to set of o "greet" #firstUser]] However, the property names of o are com- puted dynamically in loop in lines 3-8. 9
var {
2
function createdreetfuncttionn { (user this getisssg var greetUser "greet" [[greetUser] new function() console, log(arguments A
3
5
7
user);
}
Computing field-sensitive pointer analy- sis with the presence of dynamic property accesses is thus a prohibitively expensive process. The presence of dynamic proper- ties increase pointer analysis runtime from O(N9) to O(N)), where N is the size of the program (Sridharan et al., 2012). As result, traditional field-sensitive analyses are un- able to handle large Javascript frameworks such as jQuery and react.
creattereestFunciions//
12 var firstUser getussgeussrrssee (0); "l'greet" firstUser] ("hello");
Figure 1: A basic example of a dynamic prop- erty access function call on line 13. The value of firstUser is determined at runtime, making it difficult for static analyzers.
In this paper, we introduce CALLME, an approach to specifically target dynamic property access calls for Javascript call graph construction. CALLME has two stages: statement selection and inference.
Statement selection. We develop JSelect, a custom static analyzer that efficiently selects the relevant statements needed to determine if a call site calls specified function.
Inference. The output of JSelect is further processed by a Language Model (LM) which determines whether the call site can call the function.
Importantly, CALLME does not replace the need for traditional call graph construction algorithms. Rather, CALLME is intended to augment traditional call graph construction specifically for dynamic property accesses. To the best of our knowledge, CALLME is the first static analysis system capable of resolving dynamic property accesses without runtime information.
Reasoning about dynamic property accesses for Javascript makes a good target for LMs for several reasons. First, existing solutions already unsound and incomplete, so solution that works well in practice can be competitive even without theoretical guarantees introducing an unsound LM does not worsen any existing guarantees. Second, LMSs can take advantage of useful sources of information that are difficult to encode as traditional static analysis rules. In Figure 1, for example, is fairly straightforward to infer that the function call made in line 13 likely refers to the function defined in line 5, given the variable names such as "greet" and "User". Similarly, inferring the relationship between words such as "greet" and "hello" often relies on knowledge of natural language. Additionally, LMs can infer high-level design patterns, such visitors and builders, which can be challenging to incorporate into traditional static analysis.
We evaluate CALLME on a dataset of caller-callee edges collected by dynamic analy- sis (Chakraborty et al., 2022), where we specifically target dynamic property accesses. Im- portantly, the dataset consists only of calls that were missed by Approximate Call Graph (ACG) (Feldthaus et al., 2013), recent call graph construction algorithm. CALLME is able to resolve 75% of calls with an Fl score of 0.79. Jelyy, the recent open source implementation based off of the siate-of-the-rreee static analysis framework JAM (Nielsen et al., 2021a), is only able to resolve 11% of these calls with an F1 score of .19 by handling dynamic property accesses that start end with hard-coded strings, e.g., obj ("foo" ().
2

---

Published as conference paper at COLM 2025
show how CALLME can be applied downstream program analysis tasks such as vulnera- bility detection by resolving edges that are currently undetectable by call graph construction algorithms due to their use of dynamic property accesses. We manually searched multiple datasets of known Javascript bugs real-world projects and identified 25 function calls across 23 projects resulting in bugs vulnerabilities that are undetectable with current call graph construction due to dynamic property accesses. These security bugs include prototype pollution, cross-site scripting, command injection, and arbitrary file overwrites. We build scanner using CALLME to search for these vulnerable function calls, which resolves 24 out of 25 vulnerable calls with only 3 false positives.
2 Motivation
As an example to motivate our approach, consider a simplified code snippet (Figure 2) from the jQuery framework (openjsf.on containing call that is ignored by static analyzers. In the following, we discuss several features of Javascript that make analyzing the call difficult.
Functions objects. Functions ob- jects which can have assigned fields them- selves. This is shown on line where the initial jQuery object is a function but also has a field called extend. Functions can also be passed around as objects, as shown in the each function on line 28.
jQuery
context); {
2
return new jQuery. initt))
1. Functions are objects.
5
2.Dynamic additions/uses.
Dynamic additions/uses. Properties and fields can be added objects dynamically, such as adding extend to the juuery object on line 9, and then using extend add the each and show properties. The properties can then be overwritten, as shown on line 30.
3.Arity mi nismacching.
9
(Query.extent function() {
10
for (i=0; arguments. Trguments.Length; i++
11
for ( name arguments(i]
12
this[name] = options [name];
14
Arity mismatching. Functions can be called with any number of arguments, as shown the extend function on lines 8 and 10.
15
};
17
extend
18
each function( obj, cb )
19
for ( ( in obj ) cb.call(obj [i], obj [i]);
Computed names. Properties can be read and written by computed names, as seen on line 29. Additionally, precise modeling
21
23
showALL function() {
24
return shovilide( this, true );
of functions is necessary for analyzer. In our example, an analyzer would need to model the functionality of each in der to understand what happens on line 28, and would need to model the functional- ity of extend to understand what happens on line 17. Without the explicit modeling of extend, there is no way for static ana- lyzer to know that the showAlL and each functions are added to jQuery itself.
25
}
Computed
26 });
27
28
jQuery.ead each (["show"], functionn1 (1, name)
29
var cssFn jQuery. fn (name + "All"]
30
JQuery [name] = func tion(speed)
31
return speed null ?
32
css httr.aappyytthisaaggm...
33
this. allssanimate/eiim (name,, speed); };
34
However, for human looking at the code snippet in Figure 2, there are several things that indicate that the function call cssFn. apply (this, (thiss,arument arguments) on line 32 can refer to the function showAll on line 23. There are natural language hints and background knowledge of programming patterns that are helpful. For example, the anony- mous function defined on line 28 takes two parameters, i and name, common pattern where
});
Dynamic praperty access call.
Figure 2: A modified code snippet taken from jQuery showing several features of Javascript that make static analysis difficult. The dynamic property access However, information such as the variable names allow an LM successfully resolve the call.
line 29, and the function call on line
3

---

Published as conference paper at COLM 2025
CaLLME
Extract relevant slice inpul program
function logValue key) obj [key] //call
Function LogValue (ob3, key)
return obj [key]
Legra Uue (6B1
function getàge (name) { return db. lookup (name) age;
Function getàge (nane)
zeturn
bonat HBBE
db. lookup (nane) agg;;
const user =
resuls
Logra (opec Key)
getName:
"Alice",
getáge: () =>
getName: () >> "Alice",
LM checko indrect cell
getàgo ("Alice"),
getàge:
great: () sb "BL!"
getAge ("Alice"),
In the following program slice, does the call on line refer function line 57
):
const unused add, 20);
logAll (user);
const unused = add 20) logA11 (usez):
Emnoston Soguacos (obd)
const key "greet";
result LogValue (user, key): consola. 1og (zesult);
const "great")
5007
Bonat resule
LogValue (ueer,
console.. log (xesult);
Figure 3: An overview of our system CALLME. JSelect selects the relevant statements from the original program, which then gets formulated into prompt for an LM.
i refers to index and name refers to the value of list. each is word commonly used in programming when iterating over lists, so it is fairly easy to infer that j juuery. each is iterating through each element in the list of its first argument, even without looking at the definition of each on line 18. While these types of heuristics are very difficult to encode in traditional static analyzer, they can be incorporated into machine learning models and large language models. optimized for code generation and understanding released by Meta Al in 2023 (Rozière et al., 2024a), successfully resolves the call on line 32.
each
a 34-billion parameter large language model
3 Approach
CodeLlama-34b is able to identify the call in Figure 2 because of its small size at 35 lines. In real- world program, the call site and target function could be several thousand lines apart, with most of the code being irrelevant to the function call we are trying to re- solve. Thus, we need a way to au- tomatically find the relevant lines to feed into the LM. We implement custom statement selection algo- rithm, JSelect, which outputs the relevant lines to be formulated into prompt for an LM. An overview of CALLME can be seen in Figure
can.each(options function(attr) { options. attr attr:
if Scanner.attriburte. aattributes [attr])
Scanner.attriburee [attr] (options, el);
};
});
Global variables are defined elsewhere.
Figure 4: We want to provide local information around the call site as well information about the global vari- able S Scanner whose definition is several hundred lines away from the call on line 4 in the snippet.
3.1 JSelect
Our statement selection algorithm, JSelect, is an prota--rocedural def-use analysis combined with a simple window around the call site. show an example of our statement selection in Appendix B in Figure 6 and Table with other statement selection methods we tested.
4

---

Published as a conference paper at COLM 2025
Typically, static program slicing (Weiser, 1984) would be used to identify the relevant state- ments to the function call. The goal of static slicing is to identify all statements that can affect specified variable, and has been used with machine learning algorithms to perform type inference for Python (Yan et al., 2023). Unfortunately, static program slicing for Javascript runs into the same problem call graph reconstruction, it requires pointer analysis to collect the data flow graph. Sridharan et al. showed that a traditional field-sensitive pointer
analysis has an O(N*) runtime for an N-statement Javascript program (Sridharan et al., 2012). In Section 4.1, we show that a field-sensitive pointer analysis is unable to finish on large Javascript libraries within 12 hours. To the best of our knowledge, there are no static slicing algorithms using field-sened sensitive pointer analysis for Javascript that scale to large frameworks like React. Instead, we opt for cheap and scalable analysis in JSelect to identify the relevant statements for the call sites.
Call site statement selection. The intuition behind our approach is to collect the immediate context in which the call is invoked, as well as to capture references to global objects. For example, in Figure 4, we want to capture that the call on line 4 is happening inside of an if statement which is inside of a loop. However, we also want to provide information about the global variable Scanner.
The implementation of JSelect relies on parsing SarhiinJavvscctti Javascript AST's to extract variable informa- tion. We build our implementation using Esprima (esp) and Esrefactor (Ariya), two static analysis tools for parsing Javascript AST's. First, JSelect performs a static scope analysis in order to each variable to its set of references and declaration. JSelect takes in a call site location and identifies what variables are being referenced in the call itself (in Figure 4, the variables are Scanner. att wetriiie ributes, attr, options, el). JSelect collects each of the statements that modifies or uses one of the variables. The final output of JSelect which is sent to the LM con- tains all of these statements along with the statements immediately surrounding the call site. This approach combines the local context with information about any global variables that are being referenced. JSelect is not path or flow-sensitive, and is not inter-procedural An inter- procedural analysis would require pre-existing call graph, leading to a chicken-and-egg problem.
Target function statement selection. There is often useful information around the function definition for the target function (If were trying to determine whether call site calls function B, the target function is B). For example, the target function may be assigned as field to an object. In this case, it would be very helpful to include the entire object in the input to the LM. To obtain the relevant information, we use Esprima (esp) to obtain the AST node corresponding to the target function. We then provide the statements of the parent of the target function node to the model. If the parent node is too big (sometimes the parent node is the entire program), we use a window of 50 lines before and after the target function. We set the limit of the parent node 200 lines.
3.2 Prompt Formulation
Once we have filtered the relevant statements from the programs, we form the input to the language model. We base our prompting strategy off of how a human might resolve an indirect call. Our prompting has three steps: program understanding, call site intention inference, and final prediction. Our final prompting template can be seen in Figure 5 in Appendix A.
Program understanding. First, the model should gain an understanding of what each statement in the program is doing. We ask the model to interpret the slice from JSelect line-by-line to get a general understanding of the code. Section 4.2 shows the performance of CALLME with different methods of interpreting the code line-by-line, such as asking the model to explain each line or asking the model to simulate the execution line-by-line.
Call site intention inference. Next, we ask the model to reason about the call site itself. We provide the call, the line number, as well as information on the variables referenced in the call site. This provides additional information about what the call is being used for and helps determine whether it is likely to be match with the target function provided later.
5

---

Published as conference paper at COLM 2025
Final prediction. Finally, ask the model to predict whether the call site in question refers to the specified function, By this point, the model has reasoned about the program snippet as whole well as how the call site of interest is used.
4
Evaluation
Our primary baseline is Jelly, an open-source call graph construction algorithm written by the authors of JAM (Nielsen et al., 2021a), which handles dynamic property accesses with well indirect calls. The static analysis for Jelly is based
string prefixes and postfixes off three static analysis tools for Javascript-
(Nielsen et al., 2021a), ACG (Feldthaus et al., 2013), and Tapir (Meller et al., 2020a). First, analyze how well CALLME performs in resolving dynamic property calls. Next, we explore how the design decisions in CALLME affect the performance. Finally, we show how CALLME can be used with downstream program analysis tasks such as bug detection. We conducted the experiments on a Linux server with two AMD EPYC 7763 64-Cor Processors, 128 cores, 1024GB RAM, and 4 NVIDIA RTX 6000 Ada Generation GPUs.
Dataset. We use the dataset from Chakraborty et al. (2022) root cause analysis of Javascript call graphs (Chakraborty et al., 2022), con- sisting of caller-callee pairs generated by performing dynamic analysis on the
Program
#Lines
Inter-File
Total
AngularJS
12,091
41
207
Backbone
9,003
22
46
KnockoutJs
1,044
24
Knockbackjs
15,836
24
85
popular TodoMVC Suite (Chakraborty et al., 2022). We identified 660 caller- callee pairs that were missed by ACG due to a dynamic property access. For each caller-callee pair, we generate negative example by selecting random function for the same caller for total of 1,320 total caller-callee pairs evenly split between positive and negative samples. Statistics for each framework in the TodoMVC Suite can be found in l.
CanJs
11,371
34
91
React
24,855
11
57
Mithril
1,433
25
27
Vue
7,667
15
61
Vanillajs
14
jQuery
9,526
7
48
Total
93,557
184
660
Table 1: Dataset statistics. Inter-file refers to the number of caller-callee edges that are in different files, and Total refers to the total number of calls.
4.1 Performance TodoMVC
Table 2 presents the results of CALLME while using differ-
Model Detect Miss
FP
Acc.
F1 Prec. Recall
ent LM backends. We test the CodeLlama models (Roz- ière et al., 2024b), Llama-3.3- 70B-Instruct (Grattafiori et al., 2024), and GPT-4 (Achiam et al., 2023). CALLME with CodeLlama-34b achieves an Fl score of .79 and is able to detect almost 7x more calls Jelly while maintaining a tol- erable false positive rate. Ad- ditionally, as shown in Table 9 in Appendix D, 39/ /1 of the calls detected by Jelly are in Knockback,js, due to Knockback using more dynamic property accesses with string concatenations. Excluding Knockbackjs, Jelly can only detect 5.5% of calls, while CALLME is able to detect 75% of all calls in dataset. Additionally, shown in Table 8 in Appendix D, results remain stable even when calls are resolved across different files. GPT-4 and 3333m-3.3' false positive rates much lower than everything else, with almost
Jelly
71
589
19
.501
.19
.79
.11
CodeLlama-7B
490
170
339
.61
.71
.59
.74
CodeLlama-13B
435
225
230
.66
.66
.65
.66
CodeLlama-34B
497
163
102
.80
.79
.83
.79
Llama-3.3-70B
280
380
18
.70
.59
.94
.42
GPT-4
292
368
13
.71
.61
.96
.44
Table 2: Final results the TodoMVC benchmark using different models for the final inference. FP stands for False Positive.

---

Published as a conference paper at COLM 2025
9x fewer false positives than the next lowest model, but suffers from lower recall. CALLME configurable with different LM backends, so users can use Code or GPT-4 based on their precision requirements.
Runtime. As noted in prior work (Sridharan et al., 2012), a field-sensitive pointer anal- ysis is intractably slow for large programs. To test, we existing static tools on the largest files in the TodoMVC benchmark. WALA performs standard Andersen's alias analysis with call site abstractions. TAJS is static analysis tool based on abstract interpretation. TAJS errored out on all of the programs except for React due to unsupported Javascript features. We set the time-out threshold 12 hours.
pointer analysis using two
As CALLME is meant to augment existing static analyzers, we use an analogous setup for our exper- iment. We run Jelly on each pro- gram in Table 3 and find all calls that do not have a target function. We then run CALLME once for each call. This is likely an over- approximation of CALLME's run- time in practice, as we do not antic- ipate CALLME being used to con- struct an entire call graph. How- ever, seen in Table 3, CALLME is still able to scale to much larger programs than traditional field- sensitive pointer analysis.
Total Runtime (Hours)
Program #Lines #Calls
CALLME
WALA
TAJS
AngularJS 28,363 1,349
2.6
TO (12+) N/A
React 21,641
2,307
4.5
T.O T.O (12+)
Jquery 9,205 728
1.4
T.0 (12+)
N/A
Ractive 9,133 900
1.7
T.O
N/A
Blocks 14,724 1,031
2
TO (12+) N/A
Average 16,613 1,263
2.5
N/A
N/A
Table 3: Runtime performance of CALLME compared to field-sensitive pointer analysis. TAJS errored out on everything except for React. #Calls refers to the num- ber of call sites which Jelly does not return any target functions.
4.2 Ablations
We tested multiple statement selec- tion methodologies and prompting
formats achieve the best tradeoff between scalability and accuracy. We perform the same experiment as in Section 4.1 on the TodoMVC dataset.
Statement selection. We evaluate the effect of dif- ferent statement selec- tion methodologies in CALLME's performance. Our goal is to find the fastest analysis that still gets robust performance. We show an example pro- gram in Figure as well as the output of various slicers in 7. We test using simple window around the caller and callee, a program slicer tracking the flow of func- tion values (Feldthaus et al., 2013), thin slic-
Improve (+/-)
Slicing Method Detect Miss FP
Detect Miss
FP
Def-Use + Window
497 163
102 0%
0%
0%
Window Only
531 129
299
+7.1%
-25.6%
+193.1%
Def-Use Only
491 169
195
-1.0%
+4.3%
+91.1%
Thin-Slicing
452 208
170
99.7%
+27
66.7%
Full-Slicing
438 222
163
13.2%
+36.4% +59.8%
Table Ablation on various slicing methods. We treat the first the baseline and then compute the improvement of other slicing methods. Green Performance improvement. Red = Performance decrease. Using a window slightly improves the detection and miss rates, but greatly increases the number of false positives.
ing (Sridharan et al., 2007), and following def-use information of variables used in the call sites. Additional information on each of the statement selection methodologies can be found in Appendix B.
The results the slicing ablation study can found in Table 4, where we find that combining Def-Use chains and a simple window performs best. The improvement seems to come primarily in the false negative and false positive rates, where the next best slicing method still has 60% more false positives. We find evidence that incorporating static analysis leads to
7

---

Published as a conference paper at COLM 2025
dramatic improvements over only using LMs. Simply using the 50 statements before and after the call sites almost triple the false positive rate. This is likely due to the fact that simply using window includes the context in which call is made, but likely does not include information about the relevant variables used in the call itself.
Prompting. We tested multiple prompting strategies, such asking the model to perform ab- stract interpretation, asking the the model. Details all our prompting strategies can be found in Appendix C.
to analyze the code in English, and directly prompting
The results of the study can be found in Table 5, where we find that asking the model to per- form abstract interpretation has the best results. We can see that giving the model instructions as well as generating a high level summary through abstract interpretation or English interpre- tation of the code significantly improves performance over the direct and explain approaches. Interestingly, abstract inter- pretation performs better than English. This is likely due to it being slightly more granular and providing bet- ter information about the in- dividual variables.
Improve (+/-)
Prompt Method Detect Miss FP
Detect Miss
FP
Abstract Interpret.
495 162
102 0%
0%
0%
English Interpret.
450 209
94
-10%
+29%
-8.5%
Instruct
490 169
165
-1.%
+4.3%
+59%
Two-Step
304 171
119
-6..8%
+5.6%
+16.7%
4.3 Bug detection.
Direct
525 135
277
+6.1%
-20%
+171.6%
Explain
471
189
225
-5.1%
+16.7%
+120.1%
In order to determine CALLME's effectiveness in a downstream program analysis task, we use CALLME to help identify bugs which rely on re- solving dynamic property accesses. We simulate a taint analysis scenario, where the analysis needs to discover all paths from any tainted sources to vulnerable sinks. For example, user might want to locate all call sites to library function which writes information to the filesystem to ensure that is free from unsafe user input. An example of a real-world security vulnerability that requires resolving dynamic property call can be found in Appendix E in Figure 7.
Table 5: Ablation on various prompts. We treat the first row of each design as the baseline and compute the improvement of other alternatives. Green = Performance improvement. Red Performance decrease.
Our dataset includes multiple examples of security bugs that require resolving dynamic property access calls such as Cross Site Scripting, Arbitrary File Overwrites, Prototype Pollution, Improper Access Control, and Remote Code Execution.
Case study methodology. We manually searched through several hundred examples from three datasets of known Javascript bugs, SecBench.js, BugAid, and Vulnerable Functions in the Wild for bugs which required resolving dynamic property accesses and found 24 separate bugs. For each, we manually identified the target buggy function in question. Next, we built an automated detection tool which performs the following steps:
1. Scans the code for dynamic property access calls with Esprima (esp) and EsRefac- tor (Ariya), which returns all the locations in the program where dynamic property access calls occur.
2. For each call discovered by our scanner, ran CALLME to obtain the relevant state- ments to the call site.
3. Formulates the statements into a prompt as described in Section 3 with the vulnerable function to determine whether the call could refer to the vulnerable function or not and query CodeLlama-34b.
Our final dataset has 25 dynamic property access calls which resolve to vulnerable functions and 66 which do not.
8

---

Published as a conference paper at COLM 2025
Code obfuscation. Language models trained on code have been shown to be brittle to semantics- semantics-preserving program transformations (Miceli-Barone et al., 2023; Zeng et al., 2022). To test our CALLME's robustness, we obfuscate our samples using Uglify]S (Mishoo) and repeat our experiment. We replace all variable names with single letters, such as a and b, and compress the AST with Uglify]S's built-in compressor. It is important to note that analyzing obfuscated code is significantly harder task. Prior work shows that language models are very brittle to name changes (Miceli- Barine et al.,, 2023), as replacing random variable names causes an average of 81% % performance decrease in BLEU-4 on code summarization (Zeng et al., 2022).
Case study results. As shown in Table 6, CALLME successfully resolves 24 true positives with only three false positives and a single false negative. Closer introspection into the fail- ures show that the false negative uses function parameters as part of the indirect call, which are not handled by JSelect as discussed in Section 3. Additionally, performance remains relatively robust on obfuscated dataset, where CALLME can still identify almost 67% of true positives.
Normal
Obfuscated
Prediction
True Pos.
True Neg.
True Pos.
True Neg.
Pred. True
24
17
9
Pred. False
1
63
8
57
Table 6: Results of case study on obfuscated and un-obfuscated samples.
5 Related Work
Program analysis Javascript.. There have been several approaches static analysis for Javascript et Jensen et 2009; Sridharan et al., 2012; Moller et al., 2020b; Nielsen et al., 2021b; Li et al., 2022; Kang al., 2023). Several tools also explicitly target call graph con- struction (Feldthaus et al., 2013; Nielsen et al., 2021a; Madsen et al., 2015). Additionally, these systems have difficulty scaling up to large programs, as shown Section 4.1. Madsen ef al. generates call graph using static analysis, but only handles event listeners. ACG (Feldthaus et al., 2013) and JAM (Nielsen et al., 2021a) generate call graphs for Javascript programs that can handle multiple files and are scalable, but ignore dynamic property accesses.
There have been several analysis techniques to apply additional analysis specifically for certain features such as dynamic reads and writes (Park et al., 2013; Ko et al., 2019; Stein et al., 2019; Kim et al., 2014; Madsen & Andreasen, 2014; Park et al., 2016). One popular technique to use dynamic information to focus static analyzer on dynamic structures collected at runtime, and has led to many improvements in reasoning about dynamic data (Wei et al., 2016b; Wei & Ryder, 2014b; Wei et al., 2016a; Wei Ryder, 2013; 2014a; 2012; Wei, 2012; Chakraborty et al., 2024; Laursen et al., 2024). However, these approaches are not purely static like CALLME, as they require the program to be executed to collect the runtime information.
LMs for program analysis. LMs have been used for many program analysis task such as type inference (Peng et al., 2023; Wei et al., 2023; Wang al., 2023b), fuzzing (Xia et al., 2024; Yang Deng et al., 2023), vulnerability detection (Mathews et al., 2024; Liu et al., 2023), resource leak detection (Wang et al., 2023a; Mohajer et al., 2023), code summarization (Cai et Geng et al., 2024; Ahmed et al., 2024; Wang et al., 2022), and fault localisation (Wu et al., 2023). DLInfer (Yan et al., 2023) and TypeGen (Peng et al., 2023) both use static program slicing as well as machine learning model for type inference on Python. CALLME differs from the prior work as is the first to perform call graph construction. Unlike many prior works, CALLME is meant to augment current static analysis tools on certain edge cases, rather than replacing them. To the best of our knowledge, CALLME is the first work to explicitly tackle an inter-procedural analysis task like call graph construction using LMs.
et

---

Published as conference paper at COLM 2025
6 Limitations
Single link prediction. CALLME is designed to predict a single edge between specified call site and function. Thus, is not scalable to build an entire call graph using CALLME, and is not designed for task such IDE support like ACG (Feldthaus et.,,, 2013) its own. However, as many program analysis tasks only require discovering single link, CALLME can still be useful in many cases demonstrated in Section 4.3. Additionally, CALLME is meant to be used alongside existing algorithms to handle some difficult edge cases rather than replacing them.
7 Conclusion
We present new approach, CALLME, which combined static program analysis with LLMs to perform call graph construction for Javascript. Given full program, CALLME first runs the static analysis, JSelect, to select the relevant statements. Next, CALLME formulates prompt with the output ]Select and queries an LLM to determine whether specified call site calls target function. We evaluate CALLME on the TodoMVC benchmark (Chakraborty et al., 2022) and find that CALLME is able to resolve 75% of all calls with an Fl score of .79. The next best tool, Jelly, is only able to resolve 11% of these calls with an F1 score of .19. Additionally, show how CALLME can find vulnerabilities that are currently undetectable by static analyzers, where we are able to find 24 out of 25 vulnerabilities with only 3 false positives.
References
URL WwL.httpps//epprrin..rr//oo///ed
URL URLhttps://ww.nnnjss htttpp/s//ww..pppssss..c//caaaaa proccssss........s.....
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar arXiw:2333.08774, 2023.
Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl T. Barr. Automatic semantic augmentation of language model prompts (for code summarization) 2024.
Gábor Antal, Péter Hegedús, Zoltán Herczeg, Gábor Lóki, and Rudolf Ferenc. javascript call graph extraction solved yet? comparative study of static and dynamic tools. IEÉE Access, 27222666-22244 2023. doi: 10.109//ACCESS.2223.32222
Ariya. Ariya/esrefacton URL www.hhtpss/oo httpps://ggitub...cmrrrraa//eees
Fraser Brown, Shravan Narayan, Riad S. Wahby, Dawson Engler, Ranjit Jhala, and Deian
Stefan. Finding and preventing bugs in javascript bindings. In 2017 IEEE Symposium Security and Privacy (SP), pp. 559-578, 2017. doi: 10.1109/SP:200 SP..//....0.......
Yuandao Cai, Peisen Yao, and Charles Zhang. Canary: practical static detection of inter- thread value-flow bugs. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, PLDI 2021, pp. 1126-1140, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383911 doi: 10.1145/3453483.444400 URL htt.httts:/doi..rggggoo http:///o0ooo1rr11111111////3//0 1ee.o..111133451 34555/3333343334449999 3454099.
Yuandao Cai, Peisen Yao, Chengfeng Ye, and Charles Zhang. Place your locks well: Under- standing and detecting lock misuse bugs. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 3727-3744, Anaheim, CA, August 2023a. USENIX Association. ISBN 978-1- 939133-37-3, URL https:ttp/// //www. usenix.e.....oeeeeeeo. httpa//www.g..oeeeeeeee////ieuua presentation/ cai- yuandao.
10

---

Published as a conference paper at COLM 2025
Yufan Cai, Yun Lin, Chenyan Liu, Jinglian Wu, Yifan Zhang, Yiming Liu, Yeyun Gong, and Jin Song Dong. On-the-fly adapting code summarization on train- able cost-effective language models. In A. Oh, T. Neumann, A. Glober- son, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Informa- fion Processing Systems, volume 36, pp. 2023b. URL https://proceediing wwwp:eiiiddiinsneeii........ cttpcc/pppppapccccccepcccpccepee wilesspaper/eeeeeeeeaeraaaaaeeee FTL/:166666fbb002227777666660011 Paper- Ceee--ooooeeeeoe pdf.
Curran Associates, Inc.,
Madhurima Chakraborty, Renzo Olivares, Manu Sridharan, and Behnaz Hassanshahi. Auto- matic Root Cause Quantification for Missing Edges in JavaScript Call Graphs. In Karim Ali and Jan Vitek (eds.), 36th European Conference on Peject-Oriented Programming (ECOOP 2022), volume 222 of Leibniz International Proceedings in Informatics (LIPIcs), Pp. 9p.3:1-35 3:1-3:28, Dagstuhl, Germany, 2022. Schloss Dagstuhl - Leibniz-Zentrum für Informatik. ISBN 978-3- 95977-225-9, doi: 10.4230 http:///PPccs..OO22000 entities/document// http://ttt...4.0//111. 4230/LIPICs ECOOP 2022.3.
URL hts:: //rrrpd....p////////////////// dags tuhlt de/
Madhurima Chakraborty, Aakash Gnanakumar, Manu Sridharan, and Anders Moller. Indirection-boundeed call graph analysis. In 38th European Conference on Object-Oriented Programming (ECOOP 2024), pp. 10-1.Schloss Degstuhl-LeibmiizZZenrr für Informatik, 2024.
Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models, 2023.
Asger Feldthaus, Max Schäfer, Manu Sridharan, Julian Dolby, and Frank Tip. Efficient construction of approximate call graphs for javascript ide services. 2013 35th International Conference on Software Engineering (ICSE), PP. 752-761, 2013. 6606621.
doi: 10.1109 1.00//ISS..013.
Stephen J. Fink, Eran Yahav, Nurit Dor, G. Ramalingam, and Emmanuel Geay. Effective typestate verification in the presence of aliasing. ACM Trans. Softw. Eng. Methodol,, 17(2), may 2008. ISSN 1049-331X. doi: 10.1145 00.114/1344820.0 11111/33822211111 1145/1348250. 1348255.
URL
Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM Interna- fional Conference Software Engineering, ICSE '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400702174. doi: 10.145/3397500.36681444 URL 1145/3597503. 01155/33557503336614444, 3608134.
https: //tos. org/10. 11t.:/0000/111111555555500000000
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. ar Xrv preprint ar Xixv:2407.21788. 2024.
Simon Holm Jensen, Anders Møller, and Peter Thiemann. Type analysis for javascript. In Jens Palsberg and Zhendong Su (eds.), Static Analysis, pp. 238-255, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. ISBN 978-3-642-03237--..
2009.
Mingqing Kang, Yichao Xu, Song Li, Rigel Gjomemo, Jianwei Hou, V. N. Venkatakrishnan, and Yinzhi Cao. Scaling javascript abstract interpretation to detect and exploit node.js taint-style vulnerability. In 2023 JEEE Symposium on Security and Privacy (SP), pp. 1059-1076, 2023. doi: P0t.9//555211222...1212
Se-Won Kim, Wooyoung Chin, Jimin Park, Jeongmin Kim, and Sukyoung Ryu. Inferring grammatical summaries of string values. In Jacques Garrigue (ed.), Programming Languages and Systems, pp. 371-391, Cham, 2014. Springer International Publishing. ISBN 978-3-319. 12736-1.
Yoonseok Ko, Xavier Rival, and Sukyoung Ryu. Weakly sensitive analysis for javascript object-manipulating programs. Software: Practice and Experience, 49(5):840 8884 2019.
11

---

Published as conference paper at COLM 2025
URL URLhttp::/oonnineitbrrrr wwlpe...w....................... com/ doi: 1002/ stt.
doi/abs/
Mathias Rud Laursen, Wenyuan Xu, and Anders Møller. Reducing static analysis unsound- ness with approximate interpretation. Proceedings of the ACM Programming Languages, (PLDI):1165-1188/ 2024. 2024.
Hongki Lee, Sooncheol Won, Joonho Jin, Junhee Cho, and Sukyoung Ryu. Safe: Formal specification and implementation of a scalable analysis framework for ecmascript. 2012.
semanticscholaa o
http://ppw...aaaato.oooooooooro/ org/CorpusIII 14633517.
Song Li, Mingging Kang, Jianwei Hou, and Yinzhi Cao. Mining node vulnerabilities via object dependence graph and query. In 31st USENIX Security Symposium (USENIX Security 22), pp. 143-160, Boston, MA, August 2022. USENIX Association. ISBN 978-1- 939133-31-1, URL https: www.wusen...or...mmw org/ wttp:///w..oooeeeeee////eeeeeeaa presentation/
song.
Puzhuo Liu, Chengnian Sun, Yaowen Zheng, Xuan Feng, Chuan Qin, Yuncheng Wang, Zhi Li, and Limin Sun. Harnessing the power of llm to support binary taint analysis, 2023.
Magnus Madsen and Esben Andreasen. String analysis for dynamic field access, In Albert Cohen (ed.), Compiler Construction, pp. 197-217, Berlin, Heidelberg, 2014. Springrr Berlin Heidelberg. ISBN 778-3-642-54807--
Magnus Madsen, Frank Tip, and Ondrej Lhoták. Static analysis of event-driven node.js javascript applications. SIGPLAN Not., 50(10):505-519, oct 2015. ISSN 0362-1340, doi: 10.1 01144/5/2288955288822777 URL
http:/o../11.1151111 22555/288885555 2814272
Ivano Malavolta, Kishan Nirghin, Gian Luca Scoccia, Simone Romano, Salvatore Lombardi, Giuseppe Scanniello, and Patricia Lago. Javascript dead code identification, elimination, and empirical assessment. JEEE Transactions on Software Engineering, 49(7)33692-3714, 2023. doi: 00.1199/TSE.22233227777
Noble Saji Mathews, Yelizaveta Brus, Yousra Aafer, Meiyappan Nagappan, and Shane McIntosh. Libezpeky: Leveraging large language models for vulnerability detection, 2024.
Antonio Valerio Miceli-Barone Barone, Fazl Barez, Joannis Konstas, and Shay B. Cohen. The larger they are, the harder they fail: Language models do not recognize identifier swaps in python, 2023.
Mishoo. Mishoo /uglffy/ Javascript parser mangler compressor / beautifier toolkit. URL h http//ggtttt.tt// cww/mishoomeoooocccccccoooo..oot commmm/shoo//lggffyy...
Mohammad Mahdi Mohajer, Reem Aleithan, Nima Shiri Harzevili, Moshi Wei, Alvine Boaye Belle, Hung Viet Pham, and Song Wang. Skipanalyzer: A tool for static code analysis with large language models, 2023.
Anders Moller, Benjamin Barslev Nielsen, and Martin Toldam Torp. Detecting locations in javascript programs affected by breaking library changes. Proc. ACM Program, Lang,, 4(OOPSLA), nov 2020a. doi: 10.1145/3428255 URL https://doi.org: 11155 3428255.
1145/
Anders Møller, Benjamin Barslev Nielsen, and Martin Toldam Torp. Detecting locations in javascript programs affected by breaking library changes. Proc. ACM Program. Lang,, 4(OOPSLA), nov 2020b. doi: 10.1145/3428255 URL 3428255.
1145/
Benjamin Barslev Nielsen, Martin Toldam Torp, and Anders Møller. Modular call graph construction for security js
In Proceedings of the 30th ACM SIG- SOFT International Symposium Software Testing and Analysis, ISSTA 2021, pp. 29-41, New York, NY, USA, 2021a. Association for Computing Machinery. ISBN 9781450345999 doi: 10.1145/3460314 3464836. URL htt.ttttp///ooo... .rg. 1145/3466319 1144/44433391933448888 3464836.
12

---

Published as a conference paper at COLM 2025
Benjamin Barslev Nielsen, Martin Toldam Torp, and Anders Moller. Semantic patches for adaptation of wwwaaassrppprgggaaan to evolving libraries. libraren In /EE//AC 43rd International Conference on Software Engineering (ICSE), pp. 74-85, 2021b. doi: 10.1109 (001109/1CSE43390222200 00020.
Founaiioonn openjsf.org. URL.htps://jquery www.uuuurrrr..o....
Foundation
Changhee Park, Hongki Lee, and Sukyoung Ryu. All about the with statement in javascript: removing with statements javascript application, In Dynamic Languages Symposium, 2013. URL https: /api. www.nticsmconttoo htt.tsss//ppp..ssenniccccllllla lar. org/ Org/CorpusIDD 106189122
Changhee Park, Hyeonseung Im, and Sukyoung Ryu. Precise and scalable static analysis of jquery using a regular expression domain. UIGPLAN domain. 52()):25-3)) nov 2016. ISSN00622
1340. doi: 10.1145/30933342982222 2989228.
URL ht /ttp:/ooiiiii http://oo..or///.// 1145/3093334.
Yun Peng, Chaozheng Wang, Wenxuan Wang, Cuiyun Gao, and Michael R. Lyu. Generative type inference for python, 2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024b.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024a. URL https: /ttp/xaxivv/////////////////at www.aaiwww...aaas....oe/ee///eee orgg/bbs2300. 12950.
Manu Sridharan, Stephen Fink, and Rastislav Bodik. Thin slicing. In Proceedings the 28th ACM SIGPLAN Conference on Programmning Language Design and Implementation, PLDI '07, pp. 112-122, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781 7781595936332. doi: 10.1455/120074412507465 URL / 1145/ 1250734. 1250748.
Manu Sridharan, Julian Dolby, Satish Chandra, Max Schäfer, and Frank Tip. Correlation tracking for points-to analysis of javascippt.. In James Noble (ed.), ECOOP 2012 Object- Oriented Programming, pp. 435-458, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 97--3--4222-1110
Benno Stein, Benjamin Barslev Nielsen, Bor- Yuh Evan Chang, and Anders Møller. Static analysis with demand-driven value refinement. Proc. ACM Program. Lang,, ((OPSSL) oct 2019. doi: 10.1145 1344556366666.. URLhttps://doi. orgp 10. 1145/3360566.
Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R. Lyu. No more fine-tuning? an experimental evaluation of prompt tuning in code intelli- gence. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium the Foundations of Software Engineering, ESEC FSE 2022, pp. 382-394, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 978 9781450394130. doi: 01115//544250.554911 URL https:: http:::/ddooooo////// 10. 1145/3540250 3544440255.354113 3549113. htte://.11/111111111111
Chong Wang,Jianan Liu, Xin Peng, Yang Liu, and Yiling Lou. Llm-based resource-oriented intention inference for static resource leak detection, 2023a.
Jianan
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi. Codet5+: Open code large language models for code understanding and generation, 2023b.
13

---

Published as a conference paper at COLM 2025
Jiayi Wei, Greg Durrett, and Isil Dillig. Typet5: Seq2seq type inference using static analysis, 2023.
Shiyi Wei. Blended analysis for javascript: practical framework to analyze dynamic features. In Proceedings of the 3rd Annual Conference pystemms,rograammiiigg and Applications: Software for Humanity, SPLASH '12, pp. 101-102, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 9781450315630. doi: 10.1445/2847166234477 URL
http:://dof..rgg /ttp///...../11111111/18/8/11111 1101//381//2111111 2384758.
Shiyi Wei and Barbara Ryder. A practical blended analysis for dynamic features in javascript. 08 2012.
Shiyi Wei and Barbara G. Ryder. Practical blended taint analysis for javascrippp In Proceedings the 2013 International Symposium on Software Testing and Analysis, ISSTA 2013, pp. 336-346, New York, NY, USA, 2013. Association for Computing Machinery, ISBN 978145221133311 doi: 10.1145 011114//243370024455755 URL https: //oii http://o...rrr/.... 1ttgg//.... 11......114/222/66611 /ttp://6000..0111111111////3/331
2483788.
Shiyi Wei and Barbara G. Ryder. State-sensitive points-to analysis for the dynamic behavior javasciiipt objects. In Proceedings of the 28th European Conference on ECOOP 2014 Object-
Oriented Programming Volume 8586, pp. 1-26, Berlin, Heidelberg, 2014a. Springer-Verlag, ISBN 9783662442012. doi: 10.1007 9978-3-662-44202-922 URL https: http::::///ooooorg...
662 032-62622422-22222 44202-222222 9.
Shiyi Wei and Barbara G. Ryder. Taming the dynamic behavior of javascript. In Proceedings of the Companion Publication of the 2014ACM SIGPLAN Conference on Systems, Programming, and Applications: Softwure for Humanity, SPLASH '14, pp. 61-62, New York, NY, USA, 2014b. Association for Computing Machinery. ISBN 9781 450332088. doi: 10.1155/266035222600333 URL WRR.https://do/ wrg/le.greoo.o 005...111/////.22222222 2660393.
Shiyi Wei, Omer Tripp, Barbara G. Ryder, and Julian Dolby, Revamping javascript static analysis via localization and remediation of root causes of imprecision. Proceedings of the 2016 24th ACM SIGSOFT International Symposium Foundations of Software Engineering, FSE 2016, pp. 487-498, New York, NY, USA, 2016a. Association for Computing Machinery. ISBN 9781450342186. doi: 2950290 2950338.
URL https://doi.one
1145/
Shiyi Wei, Franceska Xhakaj, and Barbara G. Ryder. Empirical study of the dynamic behavior of javascript objects. Softw. Pract. Exper,, 46(7):8678889, jul 2016b, ISSN 0038-0644. doi: 00.1002/spe.2334 URL httpt::://ool 11. 1000//pee.... 2334.
Mark Weiser. Program slicing. IEEE Transactions Software (ttpgieerrnnn.S--000333332---33 1984. doi:
Yonghao Wu, Zheng Li, Jie M. Zhang, Mike Papadakis, Mark Harman, and Yong Liu. Large language models in fault localisation, 2023.
Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang. Fuzz4all: Universal fuzzing with large language models, 2024.
Yanyan Yan, Yang Feng, Hongcheng Fan, and Baowen Xu. Dlinfer: Deep learning with static slicing for python type inference. In 2023 IEFF/ACM 45th International Conference on Software Engineering (ICSE), pp. 2009-2021, 2023. doi: 10.1109 (CCE44619.2023.000 000700
Chenyuan Yang, Yinlin Deng, Runyu Lu, Jiayi Yao, Jiawei Liu, Reyhaneh Jabbarvand, and Lingming Zhang. White-box compiler fuzzing empowered by large language models, 2023a.
Chenyuan Yang, Zijie Zhao, and Lingming Zhang. Kernelgpt: Enhanced kernel fuzzing via large language models, 2023b.
14

---

Published as a conference paper at COLM 2025
Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Lingming Zhang. An extensive study on pre-trained models for program understanding and generation. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2022, pp. 39-51, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393799, doi: org/10. 1145 3539677.333666
35343390 URL https / /doi.
15

---

Published as conference paper at COLM 2025
[CALLER SNIPPET from ]Select]
Please perform abstract interpretation the previous piece code line by line. After each line you interpret, provide the output from abstract interpretation. Focus specifically on call "[CALL]" is on [LINE] Please note that this is not complete piece of code, and is snippet from larger body.
[MODEL RESPONSE]
What does [CALL]" call on line [LINE] do? Below some information about the variables that are used.
Varl, which defined on Var2, which is defined line of the previous snippet.
previous snippet.
[NODEL RESPONSE]
[LINE] the following program snippet? Please use the information provided previausly and answer yes or no. [TARGET FUNCTION SNIPPET from JSelect]
the [CALL]" call refer the function [TARGET FUNCTION]
[MODEL RESPONSE]
Figure 5: The final prompting template for CALLME.
var obj = ();
Cal true;
[_["call_func]]" function() {
console.log/ lou('eucl))"""
}
obj["call. function() {
console.logy
}
10
11 function (ulLFunc(useSStringg
(userString "func1" userString == "func2")
13
14
if (callUserString == true) {
"call_" userString;
16
obj [a]();
17
Seed statement
18
else
19
obj (bserStttig = "foo";
21 }:
23
function foo()
24
callFunc(" "fuunc1")
26
foo();
Figure An example of program demonstrate slicing.
A
Appendix
B Statement Selection Details
Simple Windowing: Our baseline approach is to use a simple window around the call site and callee function without any further analysis. This determines whether LLMs perform call graph reconstruction without being augmented by traditional static analysis tools. In our experiments, found window size of 50 to work best.
Full Slicing with ACG: As CALLME is meant to be used alongside existing algorithms such as ACG, use the output of WALA's ACG implementation to construct slices. We
16

---

Published as conference paper at COLM 2025
Table 7: The output of different slicing methods on Figure 6.
Slicing Option
Slice
var obj 0; [_ = function()
console. log ("rouuecc11( ffucc"";
obj [ = function() console. (""uncc22""";;;;
function callFunc { if (userString = "func1" II userString = "func2") { if (callUserString
Full Slicing
true) { var a = "call_" + userString; ob](a]));
}
("funcc1;;;
var obj (); function callFunc(userString {
Thin Slicing
a "call_" userString;
[b]
}
CallFunc("func1"
var obj = 0
func1"] function() {
console. log( ("funcuc""";;
}
Def-Use
obj ["call_ func2"] function() {
console. ( "func2");
var a "call" + userString; obj [a]();
obj [userString] = "foo";
choose ACG rather than Jelly as its implementation inside of WALA allows for querying of the points-to graph generated during call graph construction. ACG generates a simple flow analysis (Feldthaus et al., 2013), which only tracks the flow of function values rather than all objects. This allows it to scale to large frameworks such as React and jQuery. We use the output from the flow analysis to build system dependence graph for slicer. WALA constructs an Intermediate Representation (IR) from the source code. We use the built in source mapping to find all IR statements that correspond with the caller line. Each statement is a seed statement. Full slicing returns the set of all statements that can influence the value of one more specified seed statements by tracing all data and control dependencies from any variables in the seed statement, and is repeated recursively to find all statements. Table 7 shows the output of our full slicer on the code from Figure 6. Note that ACG only tracks function values rather than all objects, leading to the definition of ca lUserString on line 2 not being included in the slice.
Thin Slicing with ACG: Full program slices often return more information than is necessary. To remedy this, Sridharan et al. developed an approach called thin slicing (Sridharan al., 2007), which helps limit the size of the slices. Thin slicing ignores value flow of base pointers as well as control dependencies. In contrast, if one of the seed statements contains a value read from global object, full slice would include all writes to that object, whereas a thin slice would ignore all writes to the object. Table 7, we can see that the thin slice ignores all control dependencies and all other writes to obj.
17

---

Published as conference paper at COLM 2025
Model Detect Miss
FP
Acc.
Prec. Recall
Jelly
177
0
52%
1.0
.04
CodeLlama-7B
108
76
49
66%
.69
.59
CodeLlama-133
133
51
35
76%
.79
.72
CodeLlama-347
121
63
21
77%
.85
.66
GPT-4
98
86
3
76%
.97
.53
Table 8: Results to determine CALLME's ability to resolve calls from different files.
Use- We follow the protocol described in Section 3. The Def-Use slice only contains statements that reference the one of the variables the seed statement. In the code snippet in Figure 6, these are any statements that refer obj or a. We test both using only the Def-Use information well as combining it with naive window, as seen in Table
C Prompting details.
Program understanding with Abstract Interpretation. The approach described in Section 3. We ask the model to perform abstract interpretation line-by-line in the first prompt. We then ask the model to analyze the purpose of the call. Finally, we ask the model whether the call refers to the target function.
Program understanding with English Interpretation. This serves as a baseline to determine how much asking the model to perform abstract interpretation helps. We use the same prompts as the abstract interpretation approach, except we ask the model to create textual descriptions of each line without mentioning abstract interpretation.
Instruct. We guide the model step by step through resolving a dynamic property access call using the same methodology that person would. The prompts in order are follows:
1. We ask the model to analyze the variables used in the call, and provide the variable information that is provided by JSelect as well as the statements.
2. We ask the model what the purpose of the call is for and how it is used.
3. We ask the model whether the call refers to the target function and provide the statements for the target function output by JSelect.
Two-step. We first provide the model with the relevant statements for the call site, and ask the model to analyze what is happening at the call site. After the model responds, we then provide the model with the callee function and ask whether the call site refers to the callee function. The purpose of the two-step prompt is to see if the model is able to determine what the important pieces of information are in resolving a dynamic property access.
Direct. In our direct prediction approach, we provide the model with the relevant statements for the call site well the callee function, and ask the model predict whether the call site can refer to the callee function.
Explain. We perform the same test in the direct prompts, but also ask the model to explain its reasoning before providing an answer.
D Additional Results
E Vulnerability Example
Motivating example. To show how resolving dynamic property accesses can be crucial to vulnerability detection, consider the following code snippet from the NPM package find process in Figure 7, which has over 1.2 million weekly downloads (fin).
18

---

Published as conference paper at COLM 2025
Detect
Miss
FP
Benchmarks
CALLME Jelly
CALLME
Jelly CALLME Jelly
AngularJS
136
12
71
195
25
11
Backbone
30
16
44
1
Knockoutjs
24
2
0
22
0
KnockbackJs
70
39
15
46
15
4
CanJs
71
6
20
85
18
2
React
45
2
12
55
15
0
Mithril
27
26
0
Vue
48
13
57
1
Vanillajs
13
14
0
jQuery
32
45
0
Table 9: Results separated by program and compared to Jelly, the state of the art call graph construction algorithm for Javascript.
const finders
darwin function cond
exec cmd);
Command inj unjection!
}. android: function(cond)
Let cmd ps
10
exils:exec((md))
11
}
Linuxx darwin
12
13
}
14
15
function findProcess (cond) {
16
let platform platfor;;
17
let find finders (platform)
18
if (typeof find =u= "string")
find finders(find];
20
21
find (cond) (resove,,, reject);
22
}
23
module. exports findProcess
Figure 7: A security vulnerability from findProoees.. Because cond is an argument to the darwin function, whether there is command injection on line depends on whether cond. pid was sanitized before the call to darwin. Resolving the call on line 21 requires resolving the dynamic property access on line 17.
There is potential command injection vulnerability in the function darwin on line 6. If an attacker can control cond. pid, they can execute arbitrary commands as cond. pid is passed into exec. However, cond is function argument, whether not there is vulnerability depends on whether it was sanitized before the function call. The function call is made on line 21, where if finders Iplatform darwin, then the darwin function is called with unsanitized input. In order to discover the vulnerability, à static analyzer would need to determine whether the dynamic property access on line 17 can refer to the darwin function on line 2. As result, current static call graph algorithms such as JAM (Nielsen et al., 2021a) and ACG (Feldthaus et al., 2013) miss this dynamic property access and the vulnerability is undiscovered.
19


Reviews:
[{"review_id": "FuGdicUvp6", "reviewer": "Reviewer_JHym", "summary": "CALLME addresses a longstanding weakness of JavaScript static analysis: resolving calls made through dynamic-property access (eg `obj[prop]()` ). It does so in two phases:\n\n1. JSelect builds a slice of context that captures only what an analyst would read.\n\n2. The slice and a candidate callee are fed to an LLM (best results with Code-Llama-34 B) via an “abstract-interpretation” prompt that asks, “Does this call invoke function F?”\n\nOn 660 challenging edges from the TodoMVC benchmark, CALLME improves recall from 11 % → 75 % and F-score from 0.19 → 0.79 over the best previous static tool (Jelly). A case study of 25 real vulnerabilities shows 24 detcted with only 3 false positives.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Solid work: strong results. The hybrid slice-plus-LLM design shows an order-of-magnitude accuracy boost that immediately improves downstream security and refactoring tools.", "reasons_to_reject": "None. I see no substantive flaws.", "questions_to_authors": "1. How will candidate callees be generated at scale without exploding the number of LLM queries?\n\n2. What, if anything, is special about the remaining 25 % of missed edges?\n\nPresentation: please increase the resolution of Figure 3! Its current blurriness undermines an otherwise excellent paper.", "extra_scores": {}}, {"review_id": "V9RE84uIcD", "reviewer": "Reviewer_NECr", "summary": "This paper proposes CALLME, a hybrid technique that combines LLMs and static analysis for call graph construction of JavaScript programs. CALLME implements a lightweight intra-procedural def-use analysis to select relevant statements and uses LLMs to predict. The evaluation on curated datasets of call edge and bug detection show promising results.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ First hybrid “LLM + static analysis” approach aimed specifically at resolving dynamic property accesses in JavaScript.\n\n+ Reported significant gains +30% in accuracy and 0.60 in F1 over the best‑published static analyzer on the chosen benchmark and near‑perfect recall in the bug detection study.\n\n+ Potential for broad impact for bug‑finding, refactoring, and IDE tooling with better call graph construction.", "reasons_to_reject": "- Evaluation datasets are cherry-picked to favor CALLME. More specifically, the call‑edge dataset contains only the 660 edges missed by ACG; so baselines are guaranteed with low recall. On bug detection study, 24 vulnerabilities are manually selected so that require dynamic‑property resolution while other bug classes are ignored. I suggest authors to evaluate on larger benchmarks (such as SecBench.js vulnerabilities).\n\n- Negative edge might be weak that might inflate the precision of the proposed approach.\n\n- No evaluation on complete TodoMVC dynamic traces.\n\n- Some inconsistencies in reporting (different accuracy numbers reported: 80% in abstract, 75% in introduction and conclusion; Detect and Miss in some rows of Table 2&4 do not add to 660...).", "questions_to_authors": "", "extra_scores": {"ethics_concerns_details": "No"}}, {"review_id": "cLpoF9fJsu", "reviewer": "Reviewer_vcVB", "summary": "This paper introduces CALLME that combines the Large Language Model with the static analysis tools to resolve dynamic property calls in Javascript. Dynamic property calls are tough to resolve using static analysis. CALLME first uses the JSelect algorithm to select the potential relevant statements and then construct the prompt for the LLMs to resolve the dynamic property access.\n\nIn the paper, CALLME is applied to the Javascript root cause analysis benchmark with manually crafted caller-callee pairs, and shows superior performance compared with the state-of-art tools. Furthermore, CALLME is shown to be helpful in detecting vulnerabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The research problem is well-motivated and suitable to use LLMs to solve.\n2. CALLME shows improved performance in resolving a dynamic property access calls.\n3. Ablations clearly demonstrate the effectiveness of different design choices.", "reasons_to_reject": "1. Overall, I do not think COLM is the best venue for this paper for the following reasons. First, the audience generally has limited background in software analysis, especially for Javascript. Some terms, such as 'inter-procedural analysis task', 'taint analysis', and 'abstract interpretation', need to be explained more clearly in the paper. Second, CALLME is mainly a combination of software tools and large language models, with little innovation on the language model side.\n\n2. The problem addressed in the paper is relatively narrow, focusing only on resolving dynamic calls in Javascript.\n\n3. The evaluation could be improved. The current evaluation does not provide an apple-to-apple comparison. In TodoMVC, CALLME is tested as a binary classification method, while the baseline is used to construct a call graph. This setting does not make sense to directly compare CALLME and Jelly. I recommend that the authors propose a more realistic use case where CALLME is evaluated in combination with Jelly and showing improved constructed call graph. Binary classification seems unreasonable in this particular dataset TodoMVC. Furthermore, authors could consider using CALLME with existing static analysis tools on a real use case or a more complex task, rather than only on a curated subset. For example, showing that CALLME can reduce the error-out rate or runtime of existing tools.\n\n4. The presentation of the paper needs improvement. Many results and technical details, such as statement selection strategies, are placed in the appendix, which makes the main paper difficult to follow. Figure 3 lacks the caption and necessary text to demostrate the workflow of CALLME. I also suggest including at least one example from the TodoMVC dataset to help readers better understand the task.", "questions_to_authors": "1. For TodoMVC, a random function is used as negative example. If I understand correctly, the task is a binary classification task. Does it align with the actual use case of CALLME where there could be many candidates?\n\n2. What techniques do you use to find the potential target functions? How to precisely find the target functions based on AST node?", "extra_scores": {}}, {"review_id": "gTlnqatHVc", "reviewer": "Reviewer_7TUv", "summary": "This paper explores the use of LLMs for call graph analysis for Javascript programs, specifically for resolving dynamic calls that are beyond the scope of static analysis. Experimental evaluation shows that a combination of statement selection and LLM prompting significantly improves the recall on dynamic calls compared to a baseline system using traditional static analysis. The implemented system CaLLME is apparently the first static analysis system capable of resolving dynamic property access without runtime information.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Novel approach to call graph analysis, as well as a novel task for LLMs.\n- Promising experimental results with some analysis.", "reasons_to_reject": "- There is no obvious reason to reject, but there are aspects of the evaluation that at least requires some clarification.\n- The presentation can be made clearer.", "questions_to_authors": "It is not clear to me how challenging the evaluation data sets are. If I understand correctly, the TodoMVC data set originally consists of positive examples of caller-callee pairs, and has been augmented with negative pairs constructed by pairing a caller in a positive pair with a random function. This raises the question of whether the negative examples are at all plausible, which may undermine the reliability of the evaluation. Similarly, the case study reported in Section 4.3 seems to be based on a small number of hand-picked examples, which makes it hard to assess how challenging the evaluation is, especially in the absence of a baseline.\n\nAnother aspect of the evaluation is the choice of baselines. for the TodoMVC evaluation, the baseline is Jelly, which is described in the opening paragraph of Section 4, but there is no information about how this system was selected, nor what other systems could have been chosen. And, as already noted, in the case study in Section 4.3, there is no baseline at all.\n\nIn my view, the narrative style in Section 4 can be improved by clearly separating experimental setup, results, and analysis, instead of presenting everything in one narrative flow, although I am aware that this partly a matter of individual taste.\n\nI also find the first argument in favor of using LLMs (on page 2) a bit strange. Essentially, what the authors say is that, since all existing methods, are unsound and incomplete, it is okay to use another unsound and incomplete method. This may be true, but it is hardly a strength of the proposed method.\n\nTypos and such:\n- Page 2: applied downstream -> applied to downstream\n- Page 3: Under \"Arity mismatching\", there is a reference to line 8 in Figure 2, which is a blank line.\n- Page 7, Table 4: There is a missing plus (or minus) sign in the last column of the \"Thin-Slicing\" row.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces CALLME, a hybrid approach that combines large language models (LLMs) with static analysis to resolve dynamic property accesses in JavaScript, a longstanding challenge for static call graph construction. The authors demonstrate that CALLME significantly outperforms state-of-the-art static analyzers by using a lightweight statement selection technique (JSelect) to build relevant context before querying an LLM. The evaluation shows impressive gains in both a call resolution benchmark and a downstream security vulnerability detection case study. The work represents a meaningful contribution to the emerging field of neuro-symbolic program analysis, demonstrating how LLMs can extend the capabilities of traditional static tools in areas where purely symbolic reasoning breaks down due to dynamic language features.

Pros:
- Addresses a well-defined, challenging problem in JavaScript static analysis that traditional symbolic methods struggle with
- Shows significant performance improvements over state-of-the-art static analyzers (75% recall vs 11% for Jelly)
- Demonstrates practical utility in downstream tasks like vulnerability detection

Cons:
- Limited evaluation scope focused specifically on dynamic property accesses
- Evaluation methodology presents CALLME as a binary classifier rather than demonstrating integration into a full call graph construction pipeline
- Small-scale case study for vulnerability detection (24 examples)
Decision: Accept