QLCoder: A Query Synthesizer For Static Analysis of Security Vulnerabilities
Authors: Claire Wang
Ziyang Li
Saikat Dutta
Mayur Naik
Venue: ICLR 2026 Poster (2026)
Keywords: Static Analysis
Program Synthesis
Vulnerability Detection
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Abstract: CodeQL is a powerful static analysis engine that represents programs’ abstract syntax trees as databases that can be queried to detect security vulnerabilities. While CodeQL supports expressive interprocedural dataflow queries, the coverage and precision of its existing security queries remain limited, and writing new queries is challenging even for experts. Automatically synthesizing CodeQL queries from known vulnerabilities (CVEs) can provide fine-grained vulnerability signatures, enabling both improved detection and systematic variant analysis. We present QLCoder, an agentic framework for synthesizing CodeQL queries from known CVE descriptions. QLCoder leverages the Model Context Protocol (MCP) for agentic tool use, integrates abstract syntax tree guidance, and incorporates CodeQL’s language infrastructure and documentation into the synthesis loop. A key challenge is that state-of-the-art large language models hallucinate deprecated CodeQL syntax due to limited training data and outdated knowledge. QLCoder addresses this by combining contextual engineering, iterative query feedback, and structured tool interaction to reliably generate executable, up-to-date queries.

Published as a conference paper at ICLR 2025
QLCODER: A QUERY SYNTHESIZER FOR STATIC ANALYSIS OF SECURITY VULNERABILITIES
Claire Wang
Ziyang Li John Hopkins University yangêcs. jhu. edu
Saikat Dutta Cornell University sai safkatd@corme edd:
University of Pennsylvania cdwang@seas upenn edu
Mayur Naik
University of Pennsylvania
mhnaik@uper keeeeeneen...mmmm edu
ABSTRACT
Static analysis tools provide powerful means to detect security vulnerabilities by specifying queries that encode vulnerable code patterns. However, writing such queries is challenging and requires diverse expertise in security and program anal- ysis. To address this challenge, we present QLCoder an agentic framework that automatically synthesizes queries in CodeQL, powerful static analysis engine, directly from a given CVE metadata. QLCoder embeds an LLM in a synthesis loop with execution feedback, while constraining its reasoning using custom MCP interface that allows structured interaction with a Language Server Protocol (for syntax guidance) and RAG database (for semantic retrieval of queries and documentation). This approach allows QLCoder to generate syntactically and se- mantically valid security queries. We evaluate QLCoder on 176 existing CVEs across 111 Java projects. Building upon the Claude Code agent framework, QL- Coder synthesizes correct queries that detect the CVE in the vulnerable but not the patched versions for 53.4% of CVEs. In comparison, using only Claude Code synthesizes 10% correct queries. Our generated queries achieve an F1 score 0.7. In comparison, the general query suites in IRIS (a recent LLM-assisted static analyzer) and CodeQL. only achieve F1 scores 0.048 and 0.073, highlighting the benefit of QLCoder's specialized synthesized queries. QLCoder available https: http::////g...taoccccoc/e/aaaprr http//raaaaa/ccccooo.oo.eo/oo/oe qloderr
INTRODUCTION
Security vulnerabilities continue to grow at an unprecedented rate, with over 40,000 Common Vul- nerabilities and Exposures (CVEs) reported in 2024, 28,961 CVEs reported in 2023, and 25,059 CVES reported in 2022 (CVE) 2025). Static analysis, a technique to analyze programs without executing them, is common way of detecting vulnerabilities. Static analysis tools such Cod- cQL HibtHub 2025b). Semgrep (Semgrep) 2023), and Infer (Metal (025) are widely used in indus- try. They provide domain-soe -speciff languages that allow specifying vulnerability patterns as queries. Such queries can be executed over structured representations of code, such as abstract syntax trees, detect potential security vulnerabilities.
Despite their widespread use, existing query suites of static analysis tools are severely limited in coverage of vulnerabilities and precision. Extending them is difficult even for experts, as requires knowledge of unfamiliar query languages, program analysis concepts, and security expertise. In- correct queries can produce false alarms or miss bugs, limiting the effectiveness of static analysis. Correct queries can enable reliable detection of real vulnerabilities, supporting diverse use-cases such as regression testing, variant analysis, and patch validation, among others (Figure
Meanwhile, CVE databases (MITRE 2025 NIST 2025 GitHub 2025d) provide rich information
about security vulnerabilities, including natural language descriptions of vulnerability patterns and records of buggy and patched versions of the affected software repositories. This resource remains largely untapped in the automated construction static analysis queries. Recent advances in LLMs, particularly in code understanding and generation, open up the possibility of leveraging this informa-

---

Published as a conference paper at ICLR 2025
CVE Information
QCVE Query (CodeQL.
Regression Testing
Same
Repo
Desc: The Look 53 servicl's budket creatoo endpoin
wheeratle to XML External Erilyy (OXE) intection.
QLCoder
Variant Analysis
Other Repos
Flx: Lora153 Jarn: 122
cos) suarth.ptthohg saae (1)
WM Eapan so 217 Mwileen-FA333
select
Patch Validation
Patches
A CodeQL query capturing a vulnerability pattern is synthesized by QLCoder from an existing CVE and subsequently reused for regression testing, variant analysis, patch validation.
Figure I:
tion automatically synthesize queries from CVE descriptions, thereby bridging the gap between vulnerability reports and practical detection tools.
Synthesizing such queries poses significant challenges. The syntax of static analysis query lan- guages is low-resource, richly expressive, and evolves continually. A typical query, such as the one in Figure 2(b) specifying global dataflow pattern leaves ample room for errors in describing predicates for sources, sinks, sanitizers, and taint propagation steps. Even if the generated syntax is correct, success is measured by whether the query can identify at least one execution path travers- ing the bug location in the vulnerable version while producing no matches in the patched version. Achieving this requires understanding the CVE context at the level of abstract syntax trees, such as code differences that introduce sanitizer to prevent flow from source to sink. Complicating matters further, reasoning about the code changes alone is often insufficient: sources, sinks, and taint propagation steps may reside in parts of the codebase far from the modified functions or files, and the vulnerability itself may involve non-trivial dataflow chains across these components. Thus, correct query must not only integrate information from multiple locations across the program but also capture the intricate propagation patterns to accurately characterize the vulnerability.
In this paper, we present QI.Coder-an agentic framework that synthesizes queries in CodeQI., powerful static analysis engine, directly from a given CVE metadata. We select CodeQL. because has the richest query language, which allows capturing complex inter-procedurl vulnerability patterns. QLCoder addresses the above challenges by embedding an LLM in a structured synthesis loop that incorporates execution feedback to verify query correctness and allows interactive reason- ing using custom MCP (Model Context Protocol) interface. The MCP interface constrains the model's 's reasoning using a Language Server Protocol (for syntax guidance) and vector database of CodeQL queries and documentation (for semantic guidance). By combining these capabilities, QLCoder avoids common pitfalls of naive L.LM-based approaches, such as producing ill-formed queries, hallucinating deprecated constructs, missing subtle vulnerability patterns, and instead produces queries that are both syntactically correct and semantically precise.
We evaluate QLCoder on CWE-Bench-Java (Li et al. 2025b), which comprises 176 CVEs across 111 Java projects. These CVEs span 42 different Common Weakness Enumeration (CWE) cate- gories and the projects range in size from 0.01 to 1.5 MLOC. To account for model training cut-offs, we include 65 CVBs reported during 2025 and target a recent CodeQL version 2.22.2 (July 2025). Using the Claude Code agent framework, QLCoder achieves query compilation and success rates of 100% and 53.4%, compared to 19% and 0% for our best agentic baseline, Gemini CLI. Further, our generated queries have an Fl score of 0.7 for detecting true positive vulnerabilities, compared to 0.048 for IRIS (Li et al. 2025b), recent LLM-assisted static analyzer, and 0.073 for CodeQL.
We summarize our main contributions:
Agentic Framework for CVE-to-Query Synthesis. We present QLCoder, an agentic framework that translates CVE descriptions into executable CodeQL queries, bridging the gap between vul- nerability reports and static analysis. QLCoder introduces a novel integration of execution- -uuided synthesis, semantic retrieval, and structured reasoning for vulnerability query generation.
Evaluation on World Repositories and CVEs. We evaluate QLCoder on 176 CVEs in Java projects, covering 42 vulnerability types (CWEs) from CWE-Bench-Java. Each project involves complex inter-procedural vulnerabilities spanning multiple files. We show how QLCoder can successfully identify sources, sinks, sanitizers, and taint propagation steps, and refine queries to ensure they raise alarms vulnerable versions while remaining silent on patched versions. Comparison with Baselines. We compare QLCoder against state-of-the-e agent frameworks and show that CoCdrer achieves substantially higher compilation, success, and F1 scores. We
2

---

Published as a conference paper at ICLR 2025
X Vuln Patch
Lecalsa. ptt.jjaw::2200 124-125
Sanitizer Query
Input, Input.setPropers secPropertyy by BURPORT OTD, false];
class xnlxseaanttizer extends betaflow: XnlkseBanitizert 1
Medsts/Nethedaal setPropba12
setPropCall get hed heat))haettrere "..."
"setProperty" and
Factory
LocaiSa.
exasts(Field setPropcal1. wew.al...etrouunenteee....
F.hhslaane" SUPFORT_ DTD") and ...
f.getinkeesss()
IILInpuFacttory Input non Wetsingutaatorr]) KniRagoer sa Tappper
setProppall. qete.l.uttttgttett
T.hashaare false"" ... ... // 12 lines axicted...
new FelFactory (arput. servicefactory register(
nen KniMuppert
-01:
Q Taint Steps
XelNaoper class. xelMapper);
predicate
exists (Methedtall register, varsocess xnikapperkar regreter ge getMethod)) hasikanee regaster" and regiater -e NetNetho sseeDeclyyaef hasChasee ..... "Service Factoryy and
(DataFlos Loo: Mode nt, Dataflow: Mode n2)
Source Greateerccettonttellee Java
pcsta(LarbdaExp lanbda lasbda, geteepeeedpggeeeeeee Body() XngMagee ...]
steps...
class peeettellckeettnttrlllee in
Crrren Soriaa Gran Stepel
HEtpRocuest HttpRespense
Q. Source Query
Excection
class HexpRegues Bourree extends DataFlow: :Nooe HttphequestBourca) { ext e (Rechodtall ge getlboyyalll this ast getBoodyaall
StputStrees Streer
neen Byte Inpurrrrruu........ (req. gatdoy()))
getmodyCall getmathod()) () hastare( " "getflddy") and getBodyCall gecoualifierd (e.tttypee (estype()hhassoinee haschaane""".. .'..", "HttoRoquest" and Lines
(keddoo pull unlo)
for other types of sources
Sink
CratsetcketCntrrolle... Jave
Sink Query
181_ ovee 27122277.1..
Xalkapeer serviceFactory
class XaiParsingkeeeinn extends bacaflon: didee KesParsinngkeSiii I call asxorr) call.getárgunentte and call. ghod)thod()) hastassel "readValue" and call. gethetthd())....... gettlllg (ea hhasfHirm ...", "XxlMapper")
.class);
BreateBucketDont conf
and Mapper .wwaeValllee lie
StputStrass, BreateRucke Conf .olass):
>: lines oxitted...
(a) The vulnerable dataflow snippets and the patch, which adds configura- tion to XMLInput Fcttory.
(b) Snippets of the synthesized vulnerability query by QLCoder capturing the patterns of dataflow source, sink, taint steps, and the sanitizer indicated by the vulnerability patch.
CodeQI. Query
sodull int intppsellos Track ing: : :Global citt tpxoel lnwtoniig:: // Config: lit ittplequnsttorrs, Partt Sonnii import Stppkooeoon:PPtaarraaa
fron PttoloeeFlows: Pathiodd course, Sttsseeeloo:::aahhoooo cink
(loxaathhsauu sink)
solect sink. pethoee)))))))
"HTTP request data flows XM parser.", got HHTP request entry point"
(c) The synthesized CodeQI. path query that ties everything together.
Figure 2: Illustration of vulnerability CVE--2025-2133 in repository Robothy/local-e which exhibits an XML. External Entity Injection weakness (CWE-611). When the XmpMpp is not configured to disable Document Type Definition (DTD), the function readValue may declare additional entities, allowing hackers to inject malicious behavior.
also compare QLCoder's synthesized queries with state-of- ofate-of-thh-ar static analysis frameworks and show that our queries are more precise and have higher recall.
ILLUSTRATIVE EXAMPLE
We illustrate the challenges of vulnerability query synthesis using CVE-2025-27136, an XML Ex- ternal Entity Injection (XXE) bug found in the repository aabothy/local--ee Figure Eggure/depice depicts the vulnerability snippets, the patch, and the synthesized CodeQL query generated by QLCoder.
Vulnerability context. The vulnerability arises when the Xm XmlMapperrreeeeeeee object is used parse user- provided XML data (Figure 2a). In the vulnerable code, Xm XmlMapper..... readaa a 1 ue is called on the HTTP request body without disabling support for Document Type Definitions (DTDs). As a result, an attacker can inject malicious external entity declarations into the input stream, enabling server- side request forgery (SSRF) attacks, allowing for access to resources that should not be accessible from external networks, effectively leaking sensitive information. The patch mitigates the issue by configuring the underlying XMI.InputFactorm with the property SUPPORT DTD=fal se.
readVal
Synthesizing the query. The CodeQL query that can effectively capture the vulnerability pattern needs to incorporate 1) sources such Ht tttpReq Body calls where untrusted malicious information enters the program, 2) sinks such as invocations of Xm1Mappes readValue, where the XXE vulnerability is manifested, 3) additional taint steps related to how the Xm 1 Mapper is
3

---

Published as a conference paper at ICLR 2025
Agentic Framework Toolbox (via MCP)
Candidate Query
Feedback
Voctor Database
CodeQt. Langunge Server
owen ATiM aussee owey synte is willd
Ponoution Readhte: manoyy itrecattin 81
Documentations
codegt complete(ur),)..u codegl_ hooegl_ooeerur2....0
Generator LLM-Agent
Validatur CodeQL
lethed Recall: Waleerale True, Froed Thi Tru Nosttow Metioda:: intrerabbend, Floodel Grnerage: target withods
Sample queries
Abstract syntes troes
codegt_ pwt.eenstttonnurl..... codegl, (orrl) codegl_ [euppl_deffastsinnnrrl..o
Vulnerahility info
Racoerafully targeled mutisate
Patch info
Feedback
Final Query
Musnol turgel melisod.
Figure 3: Overall pipeline of QLCoder's iterative synthesis loop between an agentic query generator and a CodeQL-based validator. The generator uses vector database and our CodeQL Language Server as tools while the validator produces compilation, execution, and coverage feedback.
constructed and configured, involving non-trivial interprocedural flows spanning multiple files, and 4) sanitizers such as calls to setP roperty (SUPPORT _TD,, false), so that we know that no alarm should be reported after the vulnerability has been fixed.
In general, the synthesized query must connect all these components to be able to detect the bug in the vulnerable program, while not reporting the same alarm after the vulnerability has been fixed. Figure ( [ shows all the components of the CodeQL. query (simplified), capturing their in- dividual syntactic patterns. Lastly, Figure 2c connects all these components into a coherent path query by using CodeQL's Taint TrinTTran a ck ing: Global< <. >; From-where-selede query, which returns the exact path from source to sink.
PathGraph and the SQL-like
Challenges and solutions. Vulnerability query synthesis must overcome several tightly-coupled challenges. We hereby state the challenges and explain how QLCoder addresses them.
Rich expressiveness onagillityy of syntactic patterns. CodeQL is powerful but syntactically intri- cate: small mistakes in predicate names, qualifiers, or AST navigation often produce syntactically valid yet semantically useless queries. QLCoder mitigates this fragility through its Language Server Protocol (LSP) interface for syntax guidance and RAG database for semantic retrieval of existing CodeQL queries and documentation. These structured interactions guide predicate selec- tion and AST navigation during synthesis, reducing off-by-name and version-mismatch errors.
Inter-procedural taint propagation across large codebase. Sources, sinks, and sanitizers typ- ically live in different modules or files and are connected by nontrivial inter- inter-procedural flows (lambdas, factory patterns, etc.). While CodeQL provides robust inter-procedural analysis for many common patterns, gaps in dataflow still require bridging via additional taint propagation steps. Through its custom MCP interface, QL.ooder performs structured reasoning to discover candidate program points, synthesize custom taint-step predicates (e.g., service registration), and compose them into a CodeQL path query that tracks data across file and component boundaries.
Semantic precision: alarm on the vulnerable version, silence on the patched version. A useful vul- nerability query must not only parse correctly but also be discriminative. QLCoder enforces this semantic requirement directly during synthesis. Via an iterative refinement loop. the successful criteria states that in the fixed program, there should be alarm being raised about the vulner- ability. This incentivizes the agent to synthesize sanitizer predicates (e.g.. the set roperty call) and them constrain the path query that sanitizer presence suppresses the alarm. The resulting query thus captures the exact behavior difference, producing alarms on the vulnerable snapshot and not on the patched snapshot.
Together, these capabilities let QLCoder synthesize a semantically precise CodeQL query that can be reused for regression testing, variant analysis, or patch validation. We now elaborate on the detailed design and implementation of QLCoder.
QLCODER
At a high level, QLCoder operates inside a repository-aware iterative refinement loop (Figure 3). In each iteration, the agent proposes a candidate CodeQL query, CodeQL-based validator executes and scores it on both the vulnerable and patched versions of the repository, and the agent uses the validation feedback propose targeted repairs. The loop terminates successfully when the validator accepts query, or fails after a fixed iteration budget. In this section, we elaborate the major design components that make the loop effective.
4

---

Published as a conference paper at ICLR 2025
3.1 PROBLEM STATEMENT
The task of vulnerability detection is generally framed as a taint analysis task, where the goal of a query is to find dataflow paths from source (e.g., API endpoint accepting user input) to sink (e.g.. a database write) that lack proper sanitization (e.g.. filtering malicious data).
We formalize the Vulnerability Query Synthesis problem as follows. Assume input vulnerable project version Pvuln. its fixed version Pixed, and a textual CVE description (commonly available in open vulnerability reports). Let assume we have inter-procedural dataflow program graphs for each code version: Gvuln = (Vvuln, Evula) and Gixed = (Vixeed L'ssea). Let AP denote the source-level patch between Pvuln and Pixeed- We represent the patch in the dataflow-graph domain as patch subgraph AG (AV, AE), where AV is the set of graph nodes that correspond to the modified program snippets.
A vulnerability path query Q evaluated on graph G returns a set of dataflow paths, denoted as π [Q) We write each path E IT as π = . u1x), where each vn E dataflow graph G. Consecutive nodes (M;, vi+1) should be either connected by an existing edge in E, or an additional taint step specified in the query Q. to compensate for missing edges via dataflow graph construction. Specifically, we call i1 the source of path and vs the sink of π.
node in the
Synthesis task. We aim synthesize query Q from the vulnerability report satisfying the follow- ing requirements:
1. Well-formedness. Q is syntactically valid (based the latest CodeQL syntax) and can be executed on the target CodeQL infrastructure (e.g.. dataflow graphs) without runtime errors.
2. Vulnerability detection. Q generates least one path π in the vulnerable version that traverses the patched region:
3π E (Guule) such that x ANAV # 0.
3. Fix discrimination. Q does not report the vulnerability in the fixed version. Concretely, no path reported on the fixed version should traverse the patched locations:
Vπ [Q](Gr) we have TnAV=0 0.
In other words, the synthesized query must be executable, must witness the vulnerability in the vulnerable version via path that uses code touched by the fix, and must not attribute the same (patched) behavior in the fixed version. When only the fell-formedness condition is satisfied, we say that the query Q is valid (denoted as valid(Q)); when all the conditions are satisfied, the query Q successful (denoted as success(Q2 Pvuin: Piixea). Note that these criteria may admit potentially false positive paths in both versions. It might be possible to consider additional constraints regarding precision, but it might further complicate synthesis. In practice, we find most queries synthesized by QLCoder already have high precision.
3.2 DESIGN OF QLCODER
Concretely, QI.Coder proceeds in an iterative refinement loop indexed by = 1.......) Via prompting, the LLM agent-based synthesizer first proposes an initial candidate query Qo. For each iteration i, the validator evaluates Q; and produces a feedback report. We consider synthesis suc- cessful iteration i iff success Pvuln. Piisea) holds; in that case the loop terminates and Q: returned. Otherwise, the synthesizer analyzes the feedback and the previous candidate Qi and produces the next query candidate Q++1- The loop stops successfully when success(-) is achieved or fails once i reaches the pre-configured limit N (in our implementation N = 10). The remainder of the design focuses two aspects: 1) how the agentic synthesizer performs synthesis, and 2) how the validator generates and communicates feedback. We elaborate on both below.
Agentic synthesizer. In each iteration i, the M-b-ed agentic synthesizer runs an inner conver- sation loop of up to M turns. In each turn, the agent either performs internal reasoning or issues a tool call by emitting JSON-formatted action. When a tool call succeeds, the tool returns a JSON- formatted response that is appended to the conversation history. Conversation histories are kept local to the current refinement iteration (i.e., not carried over between iterations) keep context compact and relevant. In practice, we set M = 50, i.e., the agent may interact with tools up to 50 times before generating candidate query for validation.
Two design choices are critical for the effectiveness of this loop: 1) the initial prompt that initializes and constrains the agent's behavior, and 2) the toolbox of callable tools, each exposed by custom
5

---

Published as a conference paper at ICLR 2025
mnronn_queryy www...uery.ddouuue d
chrore. phrror-query-doree dfcumentss "cadegl_qq", queryal XMMLInpucaacoor. n.results:3)
writel file_patte "xe]_xxe__v__ contentse" "isport laya import sermle. code..ava..
todo_wratel nergentrue, Lodose[f "aa 3" f'raftt
querya externel." results:))
DVE-2425-21130 by farst researching XML vuinerability patterns
I'11 analyze
Nos let me exaeinn Cadeqi 's existing ing patterns for XXE.
Now I'11 create a Codeet query targating WotxinputFactory urage-
Let's try validat ing the query xnc.xxe....cl
Exellent!
Iuestian
Iucation
V Response
Response
Resppanee
06:-611: Inprose" vestr iot of Externai Ertity Reference. coseeeeeee
yai Parsers, all: Provider clanzer predicater for-
File created accessofally wel_seeoon 2e____onn__9pt__11 pa0t_ of
wulterable resteree: tised, resulta:
veloerabie__ree: tised, resuite:
confiqurec..
neatiperyy all:
evecutton. summary.
X MES8: Query.
esscution, surnary!
EXDELLENT:
Figure 4: Illustration of example traces of conversation during the synthesis of the query in the motivating example (Figure LLM-agent may think, invoke tools that are available in the toolbox, and receive responses from the MCP servers.
Model Context Protocol (MCP) server. We refer to the combined problem of designing these items as Context Engineering (discussed in Section 3.3).
CodeQI--Based Validator. The validator compiles and executes each candidate query against the vulnerable and fixed versions and returns a concise, structured feedback report that is used to drive refinement (Figure 3). The vulnerable and patched versions are automatically derived from patch commit hash, which in the CVE metadata. The report contains: (i) CodeQL. compilation results, (ii) execution counts (matches on vulnerable and fixed graphs), (iii) recall and coverage statistics, (iv) concrete counterexample traces and hit locations, and (v) a prioritized set of next-step recom- mendations (e.g.. add qualifiers, synthesize sanitizer checks, or expand taint steps) that are program- matically generated via a template.
3.3 CONTEXT ENGINEERING FOR AGENTIC SYNTHESIZER
The primary goal of context engineering is to expose the LLM-based agent to the most precise amount of information: enough for the agent to make progress, but not so much that the LLM is confused or the cost explodes. As illustrated in Figure 3 QLCoder relies on two primary MCP servers to provide demand-driven, structured information to the agent: retrieval-augmenner vector database and a CodeQL Language Server interface. We show example traces of conversation loop in Figure and describe the available tools below.
Initial prompt. Each refinement iteration begins with an initial prompt that kicks-off the agentic conversation loop. The initial prompt in the first iteration contains a query skeleton for reference (See $ $D.I for an example). In subsequent iterations, the prompt contains summary of the syn- thesis goal and constraints, the previous candidate query Qi-1, and the validator feedback report. Concretely, the initial prompt emphasizes: (i) the success predicate (see Success(:) (ii) concrete counterexamples from previous feedback, and (iii) an explicit list of callable tools and their purpose.
Vector database. We use retrieval-augmented vector database (ChromaDB MCP server in implementation) to store large reference corpora without polluting the LLM prompt. The database is pre-populated with (i) vulnerability analysis notes and diffs. (ii) Common Weaknesses Enumeration (CWE) definitions, (iii) same-version CodeQL API documentation, (iv) curated CodeQL sample queries, and (v) small abstract syntax tree (AST) snippets extracted from the target repository. We manually filtered the patch diffs files to be only related to the given CVE and automated removing non-Java files that couldn't be analyzed by CodeQL or were unrelated the CVE pattern, such as binary files. Afterwards, ASTs of the related patch files were extracted. During conversation loop, the agent issues compact retrieval queries (e.g., to fetch example CodeQI. queries related to the CWE) and receives ranked documents snippets on demand.
In practice, we may populate our RAG database with tens of thousands of documents. Even with this large corpus, we observe that the LLM-agent reliably retrieves exactly the kinds of artifacts it needs: CodeQL sample queries that inspire overall query structure, small AST snippets that suggest the precise syntactic navigation, and vulnerability writeups or diff excerpts that help discriminate buggy from patched behavior. These demand-drived lookups let the agent gather high-quality information without loading the main prompt with large reference corpora.

---

Published as a conference paper at ICLR 2025
CodeQL language server. We expose the CodeQL Language Server (GitHub 2025i) through MCP server that the agent can call for precise syntax- guidance. Importantly, we developed own CodeQL Language Server client and MCP server that ensures syntactic validity (especially for the given CodeQI. version) during query generation. The L.M agent's MCP client makes the tool call which is received by the CodeQL. MCP server. The MCP server forwards tool calls, such as complete (file, loc, char), di diagnost ics (file), and definit in ( (file, lcc, char), to the underlying CodeQL process and returns -era-seril responses. Tools such as completion help the agent fill query templates and discover correct API or AST names, while diagnost ics reveal compile or linter errors (e.g., unknown predicate names) that guide mutation. Appendix C
shows the full specification and example request and response schemas.
3.4 DESIGN DECISIONS
We discuss several alternative designs that we considered but found ineffective in practice. Allowing the agent unrestricted access to compile-and-ru CodeQL via MCP led severe performance degra- dation: compilation and full execution are expensive operations that the LLM soon overused, so we instead expose only lightweight diagnostics during the conversation and defer full compile-and-run to the end of each iteration. Permitting web search for vulnerability patterns or snippets similarly proved problematic. pollutes the working context and degrades synthesis quality. See Appendix B for more details on using web search. Equipping the agent with an extensive set of heterogeneous tools led to confusion and poor tool-selection behavior; in contrast, small, well-scoped toolbox yields more reliable ac- tions. Finally, retaining full conversation histories across refinement iterations induced context rot and ballooning prompt sizes, we keep histories local to each iteration. Overall, our current design a pragmatic trade-off that balances cost, responsiveness, and synthesis effectiveness.
both costly and easy for the agent to rely on web lookups, which quickly
4 EVALUATION
We aim to answer the following research questions through our empirical evaluation:
RQ 1: For how many CVEs can QLCoder successfully generate queries?
RQ 2: How usefull is each component of QLCoder?
RQQ 3: How does the choice base agent framework affect QLCoder's efficttivenes?
RQ 4: How scalable is QLCoder?
4.1
EXPERIMENTAL SETUP
We develop QLCoder top of the Claude Code framework (Anthropic) 2025a) and use Claude Son- net 4 for all our experiments. For agent baselines, we select Codex with GPT-5 (minimal reasoning) and Gemini CLI with Gemini 2.5 Flash. For each CVE and agent baseline, we use maximum of 10 iterations (N- = 10). For static analysis baselines, we select IRIS Li et al. (025b) and CodeQL (version 2.22.2) query suites. Experiments were run on machines with the following specifications: for the gst- agent baseline we used machine with an Intel Xeon Gold 6338 2.00 GHz CPU, 10x NVIDIA H100 PCIe, and TB RAM. For all other experiments used Intel Xeon Gold 6248 2.50GHz CPU, four GeForce RTX 2080 Ti GPUs, and 750GB RAM.
Dataset. We used CWE-Bench-Jee Li et al. 20256) and its latest update, which added new CVEs from 2025. We were able to successfully build and use 111 (out of 120) Java CVEs evaluated in IRIS (Li et al. (0255)), and 65 (out of 91) 2025 CVEs. Each sample in CWE-Bench-Java comes with the CVE metadata and fix commit information associated with the bug.
4.2 EVALUATION METRICS Besides valid(() (Q) and success Q; Pvuln; Priced) from Section 3.1
we the following terms and metrics when evaluating QLCoder and baselines on the problem of vulnerability query synthesis:
Rec(Q) (Q) 1[3 E (Gvata), x AAAV + 0). Prec(Q) 1r (QQ I AV + 0)
[G
FI( =2 Prec(Q) Rec(Q)
Prec Q) + (Qc
4.3 RQI: QLCODER EFFECTIVENESS
QLCoder vs. -tate-of-the-art QL. Table Table Tsoows QLCoder's 's overall query synthesis success rate by CWE. Table 2 shows the notable increase in precision of QLCoder over CodeQI. and IRIS.

---

Published as a conference paper at ICLR 2025
Table I: QLCoder Query Success by CWE Type.
CWE Type
Total CVEs
Success
Success (%)
Avg Precision
CWE-022 (Path Traversal)
48
31
64.6
0.75
CWE-009 (Cross-Sile Scripting)
36
18
0.621
CWE. 094 (Code Injection)
20
12
60.0
0.606
CWE-078 (OS Command Injection)
0.628
CWE-502 (Desetializatio)
66.7
0.853
CWE-611 (XXE)
60.0
0.657
Other CWEs (54 CVEs)
49
19
0.504
Table 2: Recall Performance Comparison Across Methods (Shared CVEs: 130).
Method
Recall Rate (%)
Arg Precision
Avg F1 Score
CodeQL
20.0
0.055
0.073
IRIS
0.031
QLCoder
80.0
0,672
0.700
QLCoder is able to successfully synthesize 53.4% of the CVEs. For half the queries QLCoder correctly synthesizes CodeQL, detects the CVE, and does not report false positives on the fixed version of the CVE's repository. The lack of true positive recall is why CodeQL and IRIS have significantly lower precision. CodeQL's queries are broad, categorized by CWE queries. IRIS generates all of the predicates for potential sources and sinks with CodeQL, and does not generate sanitizer or taint step predicates.
Finally, Figure shows that CodeQL, IRIS, and QLCoder have significantly higher vulnerability recall rates compared to Snyk and SpotBugs. Thus highlighting CodeQL's superior performance compared to other static analysis tools.
Impact of training cut-off. We also want take note that Claude Sonnet 4's training cut-off is March 2025. Table shows that QLCoder performs consistently regardless of CVEs before or after the cut-off period. The CodeQL. version, 2.22.2, was released in July 2025. New versions of CodeQL often include analysis improvements and new QL packs (GitHub) 2025f
The
Table 3: Tool Performance Before vs After Training Cutoff.
Period
Total CVEs
Success (%)
Avg Precision
Avg Fl
Pre 2025 (021)-2024)
I11
64
57.7
0.676
0.702
2025+ (Post-cutoff).
30
46.2
0.555
0.583
Overall
176
53.4
0.631
4.4 RQ2: ABLATION STUDIES
For ablations, we chose 20 CVEs and ran QLCoder with of the QLCoder components moved (Table (4) The ablation with no tools refers to only running Claude Code with the iterative feedback system. The high recall rate when removing access the AST cache while lowered recall rates without the LSP server or documentation access show that the LSP and documentation lookup impact the synthesis performance more. We also include QLCoder's 's performance on the same set of CVEs, and point out its significantly higher query success rate and precision score. Claude Code without tools scored high recall rate, yet failed to synthesize queries without false positives when executed on the fixed version.
4.5
RQ3: STATE OF ART AGENT COMPARISON
QLCoder can be transferred to other coding agents, For switching between coding agents, the MCP server configuration format may differ but the MCP server commands and arguments are the same. Coding agents also had different CLI commands used to invoke the agent. For the weaker, open source model we used (gpt,) we had to break down our prompts into smaller tasks and increase query writing iterations. We used Gemini CLI with Gemini 2.5 Flash and Gemini 2.5 Pro. We used Codex with GPT-5 minimal reasoning effort, GPT-5 medium reasoning effort, and for trying open source models. We used Codex with open source models due to ease of configuration. See Appendix A for more details on using Codex with open source models. We evaluated their performance on 20 CVEs. We achieved an increase in compilation success for both agents compared
8

---

Published as a conference paper at ICLR 2025
Rocall Rate Commgartoon by CWE Acrowa Different Mathods (Sharnd CVPx: 102)
THE
61.4
65.2%
22.5%
1677
LLB
8.7
5.4%
2.%
EWE
Figure Recall Rate Comparison by CWE Type Across Different Methods (102 CVEs).
Table 4: Ablation Study (out of 20 CVES).
Variant
% Successull
Recall Rale
Avg Precision
FI Score
QLCoder
55%
80%
0.67
w/o
30%)
(-%)
0.32
w/o Doc/Ref
20% 35%)
55% (-2%%)
0.32
0.36
wro AST
25% 301))
80% (4:0%)
0.41
0.47
Claude Code
108: 45%)
55% (-25%)
0.33
0.36
using the agents without QLCoder in T We also achieved an increase in successful queries generated using Codex and GPT-5 medium reasoning with QLCoder. GPT-5 minimal reasoning with QLCoder also had slight increase compared to not using. though as large of gain compared to using medium reasoning effort. Codex with gpt-oss-20b also had slight increase in successful queries synthesized, compared without QLCoder. See Appendix statisties on common error categories for agent baseline failed queries.
4.6 RQ4: SCALABILITY OF QLCODER
To compare costs and runtime of CodeQL query writing with QLCoder, do note that GitHub offers consulting services related to CodeQL query writing businesses (GitHub 2025e). For QLCoder, the average time to synthesize each query took 3712 seconds. The average cost for each synthesized query was 2.90 USD. The average input token usage was 708 tokens per query. The average output token usage was 43176 tokens per query. Figure 6 shows how for queries that finish synthesizing in < 3000 seconds, the success is around 97%. Thus, one could consider an early stop policy around 3000 seconds. A more detailed breakdown of runtime and cost per CVE can be found in Appendix
DISCUSSION
VERSATILITY OF QLCODER
CodeQL Versions and Other Languages. To upgrade the CodeQL version for QLCoder, the system has to fetch CodeQL documentation and queries related to the upgraded version. We used simple python script fetch CodeQL documentation web pages and each CodeQL CLI installation comes with its version-adjusted queries. To use QLCoder for other languages supported by CodeQL,
Table 5: LLM-agent baselines' compilation and success rates on 20 CVEs from 2025.
w/o QLCorer
with QLCoder
Baselines
Language Model
Compilation
Succesaful
Compilation
Successful
Gemini CLI Gemini CLI
Gemini 2.5 Flash
19%
0%
24% (+5%)
0% (=)
Gemini 2.5 Pro
35%
0%
75% (+40%)
0% (=)
Codex
GPT-5, Minimal
0%
0%
24% (424%)
5% (45%)
Codex
GPT-5. Medium
0%
55% (455%)
20% (420%)
Codex
gpt-oss-20b
25%
0%
35% (+10%)
5% (458)
Claude Code
Sonnct
95%
10%
100% (+5%)
35% (425%)

---

Published as a conference paper at ICLR 2025
the system has fetch CodeQL documentation and queries specific to the language, which categorized in the CLI installation and the site documentation.
Other SAST tools. For adapting the system for other engines, Semgrep (Semgrep 2025) and Snyk (nyk 2025) both have officially supported language servers. For engines that don't It have language servers, can create MCP server that interfaces with calling the engine's commands.
5.2
APPLICATIONS OF QLCODER
Variant Analysis. A QLCoder synthesized query can used for variant analysis. By running the known CVE pattern on other repositories, we can find new bugs that match the pattern. CodeQL's VSCode extension already supports running a given query on the top 1000 public repositories or any specified group of repositories (GitHub 2025c). For example, with one QLCoder synthesized query we reported 2 unknown bugs in different repositories.
Regression Testing. In continuous integration pipelines, regression testing is common component to have. One Github's CI features is its CodeQL code scan Github action (GiHub 2025a). The action automatically runs CodeQL queries a given pull request or specified event. Currently CodeQL's provided security queries support only 64 CWEs (GitHub) (2025)). QLCoder is CWE agnostic and given history of known CVEs for specific repository, the synthesized queries can be used as regression testing via a CI pipeline.
6 RELATED WORK
LLMs and vulnerability detection. L.MSs have been used extensively for vulnerability detection and repair using techniques such as fine-tuning and prompt engineering (Zhou et al., 2024). L.MSs have also been combined with existing program analysis tools for vulnerability detection. The com- bination of LLMs can be used from vulnerability analysis like IRIS's (Li et al. 2025b) source and sink identification, however IRIS depends on a limited set of CWE templates derived from CodeQL's CWE queries. IRIS also only the LLM for identifying sources and sinks. KNighter synthesizes CSA checkers given a fix commit of a C repository (Yang et al. 2025), however the checkers are written in C which has more available training data. MocQ's uses an LLM to derive subset DSL of CodeQL and Joern, and then provides a feedback loop to the LLM though prompting via API calls is used rather than an agent with tools and MocQ uses significantly higher iterations, with max threshold of 1,000 iterations per vulnerability experiment. (Li ct al. 2025a).
LLM agents and tool usage. SWE-agent pioneered the idea of autonomous LLM agents using tools for software engineering tasks Yang et al. (2024). LSPAI Go et al. (005),, an IDE plugin, uses LSP servers to guide
unit tests. Hazel. live program sketching environment, uses a language server (Blinn et al.] 2024) to assist code completions synthesized by LLMs. The Hazel Language Server provides the typing context of a program hole to be filled.
Low resource LLM code generation. SPEAC uses ASTs combined with constraint solving to repair I.L LLM--genrrted code for low resource programming languages (Mora et al., (024).. SPEAC converts a buggy program into an AST and uses solver to find the minimum set of AST nodes to replace, to satisfy language constraints. MultiPL-T generates datasets for low resource languages by translating high resource language code to the target language and validates translations with LLM generated unit tests (Cassano et al. 2024
7 CONCLUSION AND LIMITATIONS
We present QLCoder, an agentic framework for synthesizing syntactically correct and precise Cod- eQL queries given known vulnerability patterns. We will also open source our CodeQL. LSP MCP server and QLCoder. future work, we plan to explore efficient ways to synthesize, and to combine synthesized queries with dynamic analysis tools.
Limitations. We omit CVEs where the vulnerability involves non-Java code such as configuration files or other languages. QLCoder can be used with exploit generation to find vulnerabilities that are realized during dynamic execution. For supporting other languages that can be queried by Cod- eQL, the vector database can be filled with references, documentation, and example queries in other CodeQL supported languages. We also note that Claude Sonnet 4's 's official training cut-off is March 2025, however the 2025 CVEs evaluated were reported between January to August 2025.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the reviewers for feedback that improved this paper. This research supported by NSF award CCF #2313010, the NSF CSGrad4US Fellowship, and a Google Research Award.
REFERENCES
Anthropic.
Claude code: Agentic coding assistant, 2025a. URL httpp::/githuttt com/
anthropics/claude code Version 1.0. Relcased February 2025. Accessed September 2025.
Anthropic. Web fetch tool, 2025b. URL
agents=and--ools/tooo-sss -tte--ussswe---et-tttto----
Anthropic. Web search tool, 2025c. URL
agents- - Beaitts--nd----e tord-ttollstool-usse
search-tcol
Andrew Blinn, Xiang Li, June Hyung Kim, and Cyrus Omar. Statically Contextualizing Large Language Models with Typed Holes. Artifact for Statically Contextualizing Large Language Models with Typed Holes, 8000PS. 288:4488, October 2024. doi: 10.1 145/346668.....
URL https: //ds://dd... htt./aaac.mmm..mmmmmmmmmmm...... acm. htt...o..r///oo/1111111166666660
Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Car- olyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha. Knowledge transfer from high-resource low-resource programming languages for code llms, 2024.
wwtp:////aabb......o./// 09895
CVE. Common vulnerabilities and exposures (cve), 2025. URL https: http::/wwwwceeeeee. about/Metr.ict
GitHub. codeql-action, 2025a. URL h /ttpa//////gggtooooo/////a////a/
GitHub. Codeql, 2025b. URL httpp:://codeel.iitthinte
GitHub. sis,
Running codeql
queries at scale with multi-repository variant analy-
2025c.
URL httpasss/ooee.ggttto...e///e/eee
running ott.r:/r.aa............aaaaaaaar
GitHub.
Github security advisory database, 2025d. URL
https:://ithub.comm
advisories
GitHub.
Codeql services catalog, 2025e.
URL
http:://gttub..co//er..cet
services- catalog
GitHub. http::/2h..222lllloo.oooo22222//
com/
GitHub. catee///wag//lllssmmmmeee/e/eeer at main github/codeqll 2025g. URL
code/Java/secuurin
GitHub. httphh/eeehhliiieoooooee/////e/e http:///wwwww//uouoooooooooooo/n
GitHub.
execute language-server, 2025i. URL https://dos-inteent
-eectte--anguaageseeree
Gwihwan Go, Chijin Zhou, Quan Zhang, Yu Jiang, and Zhao Wei. LSPAI: An IDE Plugin for LLM-Powered Multi-Language Unit Test Generation with Language Server Protocol. In Pro- ceedings of the 33rd ACM International Conference the Foundations of Software Engineering, pp. 144 149. ACM, Jume 2025. ISBN --00- doi: 10. 10.447366377000775527735 URL
https: //dd acm. http:a//aa.aaaaooooooooo///////5

---

Published as a conference paper at ICLR 2025
Penghui Li, Songchen Yao, Josef Sarfati Korich, Changhua Luo, Jianjia Yu, Yinzhi Cao, and Junfeng Yang. Automated static vulnerability detection via a holistic neuro-symbolic approach, 2025a.
htttps::///axii org. abs 250//504...... 16057]
Z. Li. S. Dutta, and M. Naik. IRIS: -Ls--ssitted Static Analysis for Detecting Security Vulnera- bilities, Apr. 2025b. arXiv:2405.172: 172388
Meta. Infer. https://fbinfer.com// 2025.
MITRE. Common vulnerabilities and exposures (cve), 2025.
Federico Mora, Justin Wong, Haley Lepe, Sahil Bhatia, Karim Elmaaroufi, George Varghese, Joseph E Gonzalez, Elizabeth Polgreen, and Sanjit A Seshia. Synthetie programming elicitation for text-to-code in very low-resource programming and formal languages. Advances in Neural Information Processing Systems, 37:105151-105117 2024.
NIST. National vulnerability database (nvd), 2025. URL
gov/
Ollama. Ollama, 2025. Uttpstts://ollaaaaaaoo
Semgrep. Semgrep. the semgrep platform. https://semgrep.dw..2..0 2023.
Semgrep. Extensions, 2025. overview
URL https///ssngge..eee//////ttttttt
Snyk. Snyk language server, 2025. URL
XAMPPRocky. tokei, 2025. URL
C. Yang. Z. Zhao, Z. Xie, H. Li, and L. Zhang. KNighter: Transforming Static Analysis with I.LM-Synthesized Checkers, Apr. 2025. arXiv:2503.09002.
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. 2024.
15793
Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. Large language model for vulnerability de- tection and repair: Literature review and the road ahead, 2024.
orq/
abs/2404. 02525
A EVALUATION DETAILS
Table 6 is more detailed breakdown of the successful query synthesis rate CWE.
Runtime and Costs. Table 7 is more detailed breakdown of cost and runtime per synthe- sized CVE query. In table 7, SLOC is the number of source lines of code. SLOC is calculated by using the CLI tool tokei (XAMPPRocky (0255) on the project checked and counting the number of lines of code for Java files of the project. Time was derived from recording the timestamps of starting QLCoder and when QLCoder returned successful query or exhausted 10 iterations of the query. Note that the analysis phase invokes Claude Code once, and the query writing phase invokes Claude Code up 10 times. Thus for deriving total tokens used, we summed up the following token values found from the following keys in each Claude Code API response (as of September 24, 2025) input_ okens, cache_creat i and out sutput-token
at the patch commit
ooknttokeesstttttto.op In Table 7 the Tokens column shows token counts in thousands. For the USD cost value, we added up each Claude Code API response's 's dotal_cost.
value in given CVE synthesis. In Table 7 the USD column shows the total cost of synthesizing the query.
Error Categories for Agent Baselines' from Table 5 We derived the enror categories for why a query wasn't 't successful in each of the agent baseline configurations. The categories are: (i) the patched version has vulnerability recall, (ii) the query did not detect any of the patched methods
12

---

Published as a conference paper at ICLR 2025
S1OC Duration
Deriton Dhaccos Ricl
Gos thuccess a hale!
5000
Thai Cost cUSo]
Dznttion [J
5000
Pgn
1000
96
SLoc dogu
Figure 6: SLOC, Synthesis Duration, and Synthesis Cost Comparison by CVE.
related to the CVE, (iii) the query did have a file recall but failed to alarm on the vulnerable methods in the file, and (iv) the query did not compile. We have also included Figure to visualize each agent baseline configurations' error category count.
CodeQL Baseline Queries. The CodeQL baseline queries used are from CodeQL's security QL pack (GitHub) 2025h). We picked the CVEs that have CWEs supported by CodeQL's security queries. Not all of CWEs represented in our evaluation have corresponding CodeQL CWE queries.
Using Codex with Open Source Models. As of November 17, 2025, to use Codex with open source models, used ollama (Ollama 2025) to download and host the models. Codex has con- figuration file located at [home di directory] /. codex con fig. toml. For each open source model we wanted to try out with QLCoder, added new profile for the model in the configuration file. We also were able to use existing MCP server configurations for Codex, with the open source models. Below an example the configuration for using gpt-oss:20b with Codex. Afterwards, to use the profile with Codex we use the command codex - --profile gpt-oss 20b.
[model_provider:) ollama]
name = "O1lama" base_ url = "http:/ /lwoaalosss::://.. 11434//1
porofiles..pt--oss
model_provides "ollama"
model = "gpt-oss:
wire_api = "chat" "
A.1 EVALUATION LIMITATIONS
Codex CLI. Claude Code and Gemini CLI allow users to configure how many max turns an agent can take in context window. As of 9/23/2025, Codex CLI does not offer this configu- ration. Thus we were not able to force Codex to always take to 50 max turns each context window.
IRIS. The original IRIS evaluation consists of 120 Java projects from
Many of these projects are old with deprecated dependencies, thus we were only able to build and use 112 of the projects with CodeQL. 2.22.2. As of 9/23/2025, IRIS supports 11 CWEs and out of the 65 CVEs from 2025, we were able to use 24 of them with IRIS. When running some the IRIS queries, the amount of sources and sink predicates in the query led to out of memory errors. This impacted 9 out of the 24 queries, thus we treat those as queries with 0 results and false recall.
13

---

Published as a conference paper at ICLR 2025
Table 6: QLCoder Query Success by CWE Type.
CWE Type
Total CVEs
# Success
Success (%)
Avg Precision
CWB-022 (Path Traversal)
48
31
64.6%
0.750
CWE-079 (Cross-Site Scripting)
36
18
50.0%
0,621
CWE-094 (Code Injection)
20
12
60.0%
0.606
CWE-078 (OS Command Injection)
12
7
58.3%
0.628
CWE-502 (Deserialization)
66.7%
0.853
CWE-611 (XXE)
5
60.0% 3
0.657
CWE-287 (Authentication)
25.0%
0.875
CWB-200 (Information Exposure)
3
0.0% 0
0.667
CWE-400 (Resource Consumption)
33.3%
0,556
CWE-532 (Information Exposure)
100.0% 3
0.686
CWE-770 (Resource Exhaustion)
33.3%
0.444
CWE-020 (Improper Input Validation)
2
2
100.0%
0.650
CWE-089 (SQL Injection)
2
100.0% 2
1.000
CWE-1333 (ReDoS)
0.0%
0.000
CWE-284 (Access Control)
2
0
0.0%
0.500
CWE-862 (Authorization)
0.0%
0.000
CWE-918 (SSRF)
2
50.0%
0.500
CWE-023 (Relative Path Traversal)
100.0%
1.000
CWE-044 (Path Equivalence)
100.0%
0.667
CWE-083 (Improper Neutralization)
100.0%
0.052
CWE-1325 (Improperly Controlled Memory)
O
0.0%
0.000
CWE- (Foreign Code)
0.0%
0.000
CWE-175 178 (Case Sensitivity)
0.0%
1.000
CWE-190 (Integer Overflow)
0.0%
0.000
CWE-264 (Permissions)
0.0%
0.000
CWE-267 (Privilege Defined)
0.0%
0.000
CWE-276 (Incorrect Permissions)
0.0%
1.000
CWE-285 (Improper Authorization)
100.0%
1.000
CWE-288 (Authentication Bypass)
0.0%
0.000
CWE-290 (Authentication Bypass)
100.0%
1.000
CWE-297 (Improper Certificate)
100.0%
1.000
CWE-312 (Cleartext Storage)
0
0.0%
0.000
CWE-327 (Cryptographic Issues)
0.0%
0.000
CWE-346 (Origin Validation)
0
0.0%
0.200
CWE-352 (CSRF)
100.0%
0.941
CWE-426 (Untrusted Search Path)
0.0%
0.000
CWE-835 (Infinite Loop)
0
0.0%
0.000
CWE-863 (Authorization)
100.0%
1.000
Total
176
53.4%
0.631
14

---

Published as a conference paper at ICLR 2025
Genin 2.5 (wrh QUCoder)
Gemini 25 Flash (wiibout QuCoker)
20
20
Gentii 255 Qodee))
Gentni QUCoder)
B
Orter
2.0
20
XX
GPES, Melium (kib QLOwlr)
GFT-S, Median (wabout QLCoder)
20
20
3
Figure 7: Error Decomposition for Table
Table 7: QLCoder duration, costs, and total tokens used per CVE ID.
CVE-ID
CWE-ID Project
SLOC
Time (s)
Tokens (K) USD
CVE-2025-24970
020
netty
341.20K
2916
5060.08 2.92
CVE-2025-22233
020
spring-framework
782.76K
3570
2750.30
1.65
CVE-2022-23082
022
CureKit
519
5429
6415.35
4.13
CVE-2022-31
022
DSpace
237.97K
5367
5934.22
3.91
CVE-2022-31199
022
DSpace
236.54K
5636
5346.47
4.41
CVE-2025-
022
DSpace
373.20K
4275
3052.49
2.04
CVE-2018- 120-6
022
Dependency
28.53K
3453
5467.16
3.36
022
commons-io
40.33K
5511
5844.67 3.79
Continued on next page
15

---

Published as a conference paper at ICLR 2025
CVE-ID
CWE-ID Project
SL.CC
Time (s)
Tokens (K) USD
CVE-2025-0851
022
djl
102.97K
3477
4968.76 3.05
CVE--2022-2688
022
dolphinscheduler
90.69K
6288
7767.97 4.97
CVE--202-3448
022
dolphinscheduler
110.91K
1402
1748.13
1.12
CVE-2022-23433
022
esapi-java-legacy
35,88K
1888
1529.97
1.05
CVE-2020-17519
022
flink
LIIM
1912
3395.07
1.76
CVE-2022-2000 22044
022
goomph
12.77K
2084
2150.50
1.46
CVE-2023-41044
022
graylog2-server
270.29K
1722
2778.67
1.62
CVE-2201- 17297
022
hutool
64.12K
3063
4408.37
2.88
CVE-2022-2222 2-931
022
james-project
434.53K
4865
6468.51
4.40
CVE-2200-8875
022
java
816.52K
4238
3653.89
2.53
CVE-2025-49656
022
jena
676.42K
2319
2795.36
1.68
CVE-2019-0225
022
jspwiki
59.35K
2002
2183.08
1.33
CVE-2022-
022
keycloak
576.48K
4234
2193.97 1.50
CVE-2023-
022
mina-sshd
131.18K
6165
7486.34 4.96
CVE-2020-35460
022
mpxj
181.97K
4468
4041.71 2.91
CVE-201 1-4367
022
myfaces
263.92K
6037
6228.39 4.01
CVE-2022-2558 25822
022
one-java-agent
5.59K
1481
1118.59 0.92
CVE-2018 10018-00002000
022
plexus-archiver
13,54K
2213
2585.96 1.61
CVE-2023-37460
022
plexus-archiver
12.36K
2056
3958.90 2.39
CVE-2022-4244
022
plexus-utils
23.84K
3540
5521.94 3.21
CVE-2018- 1000850
MIE-2008-100005001
022
retrofit
20.43K
1688
2831.06 1.63
CVE-2019-17572
022
rocketmq
94,98K
4136
3845.71 2.58
CVE-2023-34478
022
shiro
37.14K
4233
6733.75 3.98
CVE-2023-46749
022
shiro
43.62K
3268
4812.75 2.89
CVE-2024-23673
022
8.41K
1297
1730.85
1.14
CVE-2025544096
022
solon
125.80K
1579
1695.92
1.04
CVE-2016-9177
022
spark
9.74K
4547
5257.03
3.49
CVE-2018-9159
022
spark
10.87K
4017
6149.27
4.35
CVE-2020-5405
022
spring-cloud-contig
22.01K
1511
1414.43
0.92
CVE-2020-5414
022
spring-cloud-config
18.66K
5131
5543.48
3.61
CVE-2019-020
022
tapestry-5
157.63K
932
1092.47 0.64
CVE-2018-1170
022
tika
109.76K
3494
4456.12 2.98
CVE-2022-32287
022
uima-uimaj
227.86K
6658
5932.88 4.23
6V8-2014-7816
022
undertow
103.90K
4320
5419.59 3.35
CVE-2022-3600 36007
022
venice
117.70K
1783
1371.73 0.98
CVE-2018- 12542
022
vertx-web
34,46K
1961
2487.06
1.62
CVE-2019-176A4
022
vertx-web
47.21K
1462
2660.48 1.45
CVE-2018-104
022
wildfly
486.38K
4759
7287.55
4.66
CVE-2023-45277
022
yames
173.18K
1457
2031.62
1.27
CVE-2023-45278
022
yamcs
173.18K
1572
2204.78 1.25
CVE-2018-1002202
022
zip4j
8.18K
1348
2894.83 1.72
CVE-2018-1002200
022
zt-zip
7,80K
3016
5901.48
3.85
CVE-2025-1584
023
solon
118.06K
2388
2999.88 2.22
CVE-2025-24813
044
tomcat
366.29K
1252
2409.97 1.39
CVE-2025-2660 26074
078
conductor
114.24K
1376
2726.99 1.48
CVE-2022-2061
078
bocker-commonn-plege
2.89K
3551
2354.26
1.68
CVE-2019- 10392
078
plluuin
16,79K
2630
4160.82
2.46
CVE-2020-2261
078
perfecto-plugin
825
3238
5544.05 3.45
CVE-2017-1000487
078
plexus-utils
23,30K
2571
3938.81 2.50
CVE-2023-2440
078
seaip--sccuitty-lluee
8.15K
3211
6250.12 3.65
CVE-2022-25177
078
workfoow-pp-glooa--ib--pgug
4.53K
1901
1634.79
1.12
CVE-2002-25117
078
workflow-cps-pllginn
16,98K
4048
3594.86
2.61
CVE-2022-25177
078
workfoow-multbbanch--hggggi
3.58K
5120
4783.47 3.32
CVE-2013-7285
078
xstream
44,44K
2809
5416.36 3,21
CVE-2020-26217
078
xstream
52,51K
6096
6573.86 4.29
CVE-2021 2021-21345
078
xstream
52.77K
4443
5313.38 3.78
Continued on next page
16

---

Published as a conference paper at ICLR 2025
CVE-ID
CWE-ID Project
SL.CC
Time (s)
Tokens (K) USD
CVE-2022-31192
079
DSpace
236.52K
1767
2638.92 1.74
CVE-2016-10006
079
antisamy
4.01K
3664
6565.33 3.91
CVE-2017-14735
079
antisamy
4.04K
4696
7181.97
4.68
CVE-2022-28367
079
antisamy
5,03K
4569
4505.33
3.31
CVE-2022-29577
079
antisamy
5,14K
3973
5344.35 3.56
CVE-2025-47883
079
cloudbses-jenkins-advve adviiso advisor-plugin
2.60K
1461
3818.47 1.96
CVE-2025-2901
079
console
122.59K
4287
5209.18 4.03
1VE-2016-6812
079
cxf
602.83K
2700
2878.97 1.93
CVE-2019-17573
079
exf
680.23K
3897
4745.86 3.01
CVE-2022-24891
079
esapi-java-legacy
36.07K
5593
5719.59 3.87
CVE-2020-27219
079
hawkbit
103.46K
6416
6996.26 5.20
CVE-2019-10219
079
hibernate-validator
88,47K
4691
6373.29 3,71
TTE-2018-10001 29
079
jolokia
30.21K
2441
5901.65 3.33
CVE-2025-3296
079
jpawebapi
3.37K
2698
7642.30 4.81
CVE-2020-13973
079
json-sanitizer
1.67K
3489
5929.25 3.57
CVE-2019-10076
079
ispwiki
59.38K
3244
5633.10 3.68
CVE-2019-1007
079
jspwiki
59,38K
1335
1950.83 1.18
CVE-2019-10078
079
jspwiki
59,38K
4160
4980.42 3.11
CVE-2019-10089
079
jspwiki
59.55K
2177
3361.70 2.24
CVE-2022-46007
079
jspwiki
58.95K
3097
3359.56 2.10
CVE-2223-3322 23962
079
jstachio
19.33K
4083
7190.32 3,94
CVE-2025-2302
079
jte
23.05K
3313
5277.52 3.12
CVE-2014-3656
079
keycloak
66.27K
3912
5476.51 3.29
CVE-2022-1211
079
keycloak
576.88K
7472
4463.84
3.20
CVE-2022-4137
079
keycloak
576.99K
1397
2976.16 1.61
CVE--2022-461
079
keycloak
575.34K
6241
7337.30 4.29
CVE-2021-4466
079
nacos
122.11K
4944
5964.66 4.14
CVE-2025-32960
079
restapi
19,41K
1452
3001.09 1.76
CVE-2016-5394
079
1,4KK
6294
5968.91 3,73
CVE-2025-46558
079
syntax-markdown
2.95K
1075
2051.41 1.15
CVE-2023- 29201
079
xwiki-commons
101.47K
3493
7209.06 4.03
CVE-2023-29552
079
xwiki-commons
102.52K
1555
2139.98
1.21
CVE-2023-3112
079
xwiki-commons
103.00K
1627
2454.59
1.43
CVE-2223-3644
079
xwiki-commons
103.178
4079
6791.06 .06 4.24
SVE-2003-37998
079
xwiki-rendering
51.28K
4486
5594.74
3.43
CVE-2020-200
079
xxl-job
9.12K
4361
6101.32
4.15
CVE-2023-32070
083
xwiki-rendering
50.36K
2175
2917.20 1.73
CVE-2022-45206
089
jeecgboot
100.87K
1505
2707.01
1.46
CVE-2022
089
spring-module-com
1.59K
1287
3680.27 1.77
CVE-2019-0222
094
activemq
409.42K
6429
7499.98 98 4.86
CVE-2020-11998
094
activemq
419.39K
3025
2437.65 1.61
CVE-2022-42889
094
commons-text
27.12K
1842
2152.16 1.45
CVE-2021-4122
094
cron-utils
13,21K
4784
4689.80
3.00
CVE-2023-49109
094
dolphinscheduler
153.39K
1591
2369.87
1.35
CVE-2023-517700
094
dolphinscheduler
152.57K
6049
6345.60
3.93
CVE-2021-30180
094
dubbo
179.47K
1968
1735.59 1.01
CVE-2021-30181
094
dubbo
179.12K
6880
8732.56
5.76
CVE-2022-44262
094
ff4j
46.47K
4480
6612.80 3.97
CVE-2021-4178
094
kubernetes-clin client
69.41K
2556
2116.29 1.48
CVE-2025-30061
094
kylin
395.31K
7213
5382.61 3.72
CVF-2023-34468
094
nifi
864.61K
7622
8168.47 4.70
CVE-2003--3246
094
rocketmq
108.61K
3850
5853.05 3.66
CVE-2023-337882
094
rocketmq
201.07K
4461
5656.22 3.43
CVE-2022-46166
094
prrin-booot-admin
18.23K
3040
4401.71 2.51
CVE-2022-22947
094
spring-cloud-gant -agtwwawway
25.77K
4561
3338.67 3.57
094
spring-framm poring-rrameork
705.22K
5523
6182.90 4.27
Continued on next page
17

---

Published as a conference paper at ICLR 2025
CVE-ID
CWE-ID Project
SL.CC
Time (s)
Tokens (K) USD
CVE-2018-1260
094
44.68K
1176
1851.95 1.10
CVE-2023-32697
094
sqlite-jdhe
18.11K
1436
2001.01
1.21
CVE-2020-17530
094
struts
161.73K
4277
7378.37
4.39
CVE-2025-2240
1325
smalltye-faul--llerannc
34.56K
3527
4291.18 2.67
CVE-2025-48058
1333
powsybl-core
269.0IK
4560
6698.58 4.08
CVE-2025-48055
1333
powsybl-core
269.35K
4691
6438.45 4.03
CVE-2025-300
164
camel
1.80M
6280
7088.64 4.02
CVE-2225-2442
178
oic-auth-plugin
5.26K
3860
5949.46 3.73
CVE--2255-225
190
tomcat
381.92K
4871
6579.21 4.21
CVE-2025-26799
200
iotdb
759.13K
3989
5053.95 3.31
CVE-2025-54380
200
opencast
257.74K
4220
6120.22 3.92
CVE-2025-2222
200
reactor-netty
80.17K
4088
7084.78 4.31
CV5-2014-3576
264
activemq
321.48K
5276
5521.34 3.35
CVE-2025-23015
267
cassandra
500.47K
5368
4546.67 2.98
CVE-2025-24790
276
snowflake-jdbe
98.59K
3512
5499.07 3.66
CVE-2025-48734
284
commons-beanutils
25.69K
5183
5848.92 3.47
CVE-2025-1391
284
keycloak
591.27K
5277
6025.32 3.67
CVE-2025-53100
285
graylog2-server
356.87K
4287
5071.88 2.99
CVE-2023-51982
287
crate
540.71K
5998
6567.43 3.97
CVE-2025-0604
287
keycloak
570.32K
4526
6785.14 3.81
CVE-2025-3910
287
keycloak
600.36K
6701
5630.84 3.50
CVE-2025-22228
287
spring-securijg
273.92K
3460
4215.36 2.59
CVE-2025-49125
288
tomcat
374.12K
4096
7034.93 4.75
CVE-2025-22223
290
spring-security
307.39K
2675
3320.81 2.01
CVE-2025-3500
297
keycloak
603.77K
1971
4672.52 2.61
CVE-22225-53103
312
junit-framework
125.80K
4471
5648.19 3.73
CVE-2025-27508
327
emissary
63.
3865
5184.05 3.16
CVE-2025-7365
346
keycloak
606.26K
6454
5955.44 3.81
CVE-2025-31723
352
simple-queue-plugin
1.89K
824
1248.24 0.78
CVE-2025-48795
400
exf
700.41K
7568
5211.09 3.48
WVE-2025-25119
400
netty
341.19K
1290
1896.56 1.16
CVE-2025-
400
tomcat
373.95K
4838
7602.65 4.83
CVE-2025-24789
426
snowflake-jdbe
98.59K
3089
5808.05 3.61
CVE-2025-27522
502
inlong
456.66K
7805
5149.58 3.58
CVE-2025-27522
502
inlong
453.50K
6227
4254.62 2.77
CVE-2025-27528
502
inlong
453,50K
6036
4720.41 2.92
CVE-2025-27531
502
inlong
452.17K
7894
6402.01 4.13
CVE-2025-27816
502
kafka
792,42K
6204
5508.74 3.60
CVE-2025-47771
502
powsybl-core
269.40K
1094
1366.41 0.77
CVE-2025-48955
532
para
30.08K
2094
2722.48 1.60
CVE-2025-49009
532
para
30.08K
1357
1624.77 0.91
CVE-2025-27496
532
snowflake-jdbe
102.30K
3024
4348.54 2.40
CVE-2025-5366
611
DSpace
386.12K
3998
6407.57 3.93
CVE-2025-52888
611
allure2
14.21K
1564
2836.58 1.57
CVE-2025-22136
611
local-s3
14.03K
1904
3533.41 2.18
4V7-2025-47293
611
powsybl-core
269.25K
3006
6040.41 3.63
CVE-2025-4641
611
webdrivermanager
10.50K
1633
2034.01 1.39
CVE-2025-48976
770
commons-fileupload
6.69K
4289
5690.52 3.39
CVE-2025-
770
cuba
384,90K
4424
4115.30 2.63
CVE-2025-48988
770
tomcat
368.45K
5115
6289.47 4.02
CVE-2025-27497
835
Open
601.66K
5256
5809.20 3.56
CVE-2025-31720
862
jenkins
195.74K
6319
5288.47 3.51
CVE-2025-31721
862
jenkins
195.81K
6179
7173.86 4.44
CVE-2025-27427
863
activemq-
634.70K
2506
2809.01
1.50
CVE-2023-51441
918
axis-axis -jjaaa
128.99K
4432
8246.73 4.64
918
kylin
355.34K
1791
1932.82 1.22
Continued on next page
18

---

Published as a conference paper at ICLR 2025
CVE-ID
CWE-ID Project
SLOCC Time (s) Tokens (K) USD
Average
209.44K
3712
4614.56 2.90
B DESIGN DECISIONS
B.1 WEB SEARCH TOOL USAGE
In section 3.4 we discuss how free web search for vulnerability patterns was a tool we decided not use. Free web search had less constraints in the context window. Web search lookups could fail due to web requests potentially dropping and we did not want to risk using the context window from reliability issues. Web search also led the agent searching for things that potentially weren't related to the CVE. We only want the agent to analyze the CVE description to better inform the vulnerability pattern. Although web search could open up the agent to more resources, we also were concerned about using too many tokens in the context window thus preferred cacheing CVE entries from the NVD database instead of performing a web search for every analysis call for the same CVE. Below we have provided transcripts for the vulnerability analysis phase for the same CVE (CVE-2025-27136) the first example is where Claude Code's Web fetch tool (Anthropic) 2025b) and Web search tool (Anthropic) (025c) are enabled, thus the agent web search. The second example is where Web fetch and Web search are not enabled. The second example's configuration of having no free web search is what use in QLCoder- CVE entries from the NVD website and we cache CWE data from MITRE in the vector database.
use these tools for free
cache the
Turn 5 Assistant
Tool: NebSearch
Query: CVB 2025
vulnecability details external entity XXE
Tokens: input: Cache Read: 29, Cache Create: 267 Output: 57 Rasult: Web search results for quory: *CVE-2025-21136 vulnarability
details XML, external entity XXE"
CVR-2025-27136 vilnerability and relation to XML external entity (XXB) lasues.
T/11
for information
11 Links: [[*ti=le"="ccaal" Createsacketiiniiguaaee
indpcint XML External Enticy 12 (XXF) Iejection CVF-2025-27136 GitHub Advisory batabase GitHub",
13 "url"::httte:://gtthh Impact, aaplottabliity. and Mitigation Steps Wiz',
14
"url":"httoe://wwr
16 ("LiLIe":"M00 CVD 2025 27136",
18 ("cicle": "XXL External Entity (XXE) Processing OWASP Foundation",
19 "url*
wttp://///.wooooottt/tt///////// www.ntty///lneaaaaa it External External_Rntieee Entity (XX3)
20 Processing": (""Ltli
1s XXC (ML External Entity) I Examples Prevention
21 Tmperva", http:rrrrr.rrr"""hhtsss///// http::hhhttt:://..mmmmmmmom wwwpr//..aaa.oo//mmmmmm www..//ar///a///////oo// laan///glleaatioooee waatlllttttoeuutttii
foee-mm---- Be ("Licle":
Search Results",
23 "url*: hhttp://tts:::/eeeeeee http://:wweeeiiterroooooo
24
("title": "what xXE (XEL external entity) injection? Tutorial examples
25
Web Security Anademy", "ur httptt:////://tttttttt wwtprr/ee/web-ecrrttte.
26
("Licie": Complete Guide: Impact, ExamgLes, and Prevent BackerOne",
27
"url hatep::/ http:p/://////// www.wwwwwcceo.e...oom wwwp/owwwoooeeeeeeoooooooooo..ee center/
28
xxe
Excernal enticy xxe"),
Entity (XXE) "url"" "url":""ttpsss//wee http://pp/wwww...iii..o//////aee
30
("ci=le": *XLL External Entity Prevention OWASP Cheat Sheet Series", "url"; "url";
www.p.org//caaatsheet...
32
XMT__Rxterna" Batiimaaa___ttta Bnt tyntity_Preveede fon Cweat__heetttttteeee Sheet.htmi")
33
34 Turn 6 Assistant
35 Tool: NebSearch
Query: XHL
entity XXE CodeQL detection patterns XMLInputPaczory
37
Takens: Input: cache Read: 30, 399 cache create: 267 Output: 54
38
Rasult: Web search results for query: external entity codegt. dotection patterns Java
XMI InpuFttaFa
39
40
1'11 search for information about external entity (XXE) CodegL detection patterns for Java XWl.nputFactorr.
42 Links: [["titlo":" "xML external Eatity Prevention CHASP Cheat Sheet Series",
"uhtt:" sheot www.teteeeeeeeeee.eeeoeooooroooe www..e...oso...//ceeeeeeeee.e thtteettttttttttttttteeee Latalllnntttt ty_pryeenttoo
19

---

Published as a conference paper at ICLR 2025
Cheat www.sheeeeett.ee.. ("title"; "Kesolving excernel entity user-controlled data Codegu query help documentation", "uri":: wtt.htt.com/ccooeg...... xxe/"),
jowee//////////// /jvaa-
45
"Livve" false positive: XXE via XiLInpu.Factory Issue +7199 github/codeg1", "
https:// http:::////hhbbbccee bttthbbrooeeg//sseeeee
46
("titla": :attll""""XI Entity Expansion Java SecureF Security Knowledge Base", "uri" "httpss// aat.pe--taae..eurtlllaeee html*),
liteea/biiite///mlll http:///innjjooooo//eoa wtap///mlnnniiiiiieeeeeeeeeeeeee ent antittyeeee exe_eaaaaao__jvaaa
47 "eritle"=""tectt" ng, Pixing, and nefending Againat XXR Attacks in Python and Java I by Rohit
kunar I Medium", "ttp::""ttparrrrotiiiio we..ee...mmimmm.mm wtt.///c/cccccecccccccc comp///eeeeeeeeieeeeee guide detecting rixing aeddeeending-ggiinst -tw--ataccs--in--eee seaheen-andjjaet and-java-ell 13xx==78691b/b99111 ( ("sitle"; "rreeet XML External entity valnerabilities for Jave xxe", "url" /"htops:// www.mmrrre..dd/dcce.....
=shes=s/jowa-wwe ("title":"ho fix 'Disable XML htwpaaaaccoooeeooooooooeeeeeeeee Lona
entity (XXF) procesaing
wulnerabilitile in Java Stack wwwaaa/mww........eeooooooeonnne
"I, ("sitle"; : vulnerability detection Moderne Docs", "urr":"""""""" "htps::////////// /oocs.modereee moderne lo/ openzewrite advanced program
("(tt=" "Hlow to prevent XML
externel entity attack (XE attack) Wkyong. "url "uri":"attpe /mkkyngg.... ctt.c//a/aaaa...e com/java/how-to- grovattsml--tterrl--nnttt tatalll--ttttttttt-------tt- Information Security Stack Exchange", ";"ttt" hhtppt/ht//////e//e//// securrtt..sacceecchee stackexchangeee... con/queat /260936/ java xxe vulaerability")]
|"title" :*xml Java XxE minnarbilits
49
50 Turn 7 Assistant
Tocl: webSearch
52
Query: Codegt, XxE taint tracking dava XML, parsing sources ainks sonicizers
Tokens: Input:
Read: 31, cache Create: 1,354 output:
54
Result: Web zearch results for query: CodeeL XXE taint tracking Java XML parsing sources sanicizers"
55
56 I'11 search for information about Codegl XXE taint tracking in Java XML parsing, including sources, sinks, and sanitizers.
57
Links:
[["LLtee:="rrcliiin Introduction CodeQL :: jorgects blog", :attp::// Jorgect[ http////g/pppet/paaatttaaaaeeeee -nteecc--coeee---itrtrttteiiiii introductioo/// ("tiit "Reoolving XMI external
entity user-controlled data CodeQL quezy help documentation" "url": "hetps:/ //code/
github.cem/een http://eeeeeeggeeeeeeeeeeeeeeeee httpesyabbppjjaa///aaaaaaaa java-xx/a"""" "ttite":""ssii flew state for precise data
flow enalysis "odegi", hittpp/////// http://ddell..ttth..b..
59
I"title"; "CodeCt, 2.16.4 (2024-03-11) CoceOt",
http:a:://oodgl.gitttee withub. wtw.ccom//oeeeeeee com/docs/codeq! /codeql- 2.16.1/"), ("Litle";
overview/codegl changelog/codegl http://ewe....tthbb..ccbeeeec com/ httpcceellllllgggggguueeeeeeee
"Analyyi "Analyzing data flow
and
Kotlin codegu", "url"
data-floo-in--sess/" ("tstl""
2024 Prozectad--yy-Java---- wtt.::rddddd----2---u----a-
60
Writeup", huth:r/ttthttttts/ssssseeee htw.::/pw......eeeeeere////2/2/2 Sind-bugp-in-cooog]/"" ("title": "New dztaflow API I for writing custom Codegl quaries
GitHub
61
Changelos*; "ha "t::::::iiilpe///paaaaaaa blog/ http::///g...bbbboooooooc///g/g/ 08 14 new dataflow api- woo-weiiing custom
-codsql-queiies/",
62
("ticle"::Tainttttett""""
codegl - sitHub", "url":"htpps: /codegl github, comm/coeel-sttaddree
itbrarrles//vvsseepp////om f/vsse/aaassrppss /otrrr/ddddaaat///aian perrraainnaakk.g... 1/
1/module, maantll....tteeee.... mainttrackinng
redegi deal with sava's interface invoke in Global Taint Tracking
github/codegl Discussion #7363",*** "hstpss///ttt /ttpt/////gtttttt////// wtt.:i.a../tgaaub/////cc
("titt) CCodett 2.11.3 (2022-09-28) Codeot", "uri":"attp"" pttttpp////ode/e/ whw.gh..t..o..oo con/ docs/codegl overview/codegi. changeleg/codeg/ eli 2. 2.11.0/*113
Listing 1: Web search and Web fetch tools are enabled for Claude Code to use during the vulnerability analysis phase. Note: we truncated the analysis and are only showing the tool usage, tokens used, and the web links searched.
Turn 4 Assistant
Too1: :chroma_gee.tt... _eo___eteeetttttt s (col 1ect wet.eaaeenistt_ooeee0000 ("ave_id"; "CVE- 2VV--2025-271677
where:
Tokens: Input: Cache Read: 28, 074
Cache Create: 340
Cutput: 105
Turn 6 Assistant
Tool:
qow.umuueeeeetttssssee (col colleectio_name: cwe_data, query_texts: [' CWE-611'' XML External Entity', xXE injection' n_results: 3)
Tokens: Input: I Cache Read: 29, 031 Cache Create: 340 Output: 121
Listing 2:
Web search and Web fetch tools are disabled for Claude Code to use during the vulnerability analysis phase. Claude Code can use cached data related to vulnerability patterns stored the vector database. We have only provided the tool calls and token usage.
20

---

Published as a conference paper at ICLR 2025
C CODEQL LANGUAGE SERVER VIA MCP
The following are the MCP tool specifications and example usage for our custom CodeQL LSP client, wrapped as an MCP server. The MCP server code can be found at htttps://githii
com//neura.programm wtwe//oddelllleeee
C.I TOOL SPECIFICATIONS
codeql complete Provides code completions at a specific position in a CodeQL file. Supports pagination for large completion lists and trigger character-based completions.
Inputs:
le uer (string): The URI of the CodeQL file
1 line (number): Line number (0-based)
character (number): Character position in the line (0-based)
trigger.character (string, optional): Optional trigger character (e.g.. """, ":")
limit (number, optional): Maximum number of completion items to return (default: 50)
offset (number, optional): Starting position for pagination (default:
completiontae with pagination metadata containing completion items, each with label, kind, documentation, and text edit information.
Returns:
Example usage:
"tool": "codes]__complete"" "arguments: "file_uri"; "tLee///oorksppcsssutttt//eee "line": 5, "character": 12, serrgger_ccaracterr ".",
"limit": 25
codeql hover Retrieves hover information (documentation, type information) a specific po- sition. Provides rich markdown documentation for CodeQL predicates, classes, and modules.
Inputs:
f i le.uri (string): The URI of the CodeQL file
1 line (number): Line number (0-based)
character (number): Character position in the line (O-based)
Returns: Hover null containing documentation content in markdown or plain text format, with optional range highlighting
Example usage:
"tool": "codeql_hover" "arguments"; "file_uri"; "line": 8,
"character"; 15
21

---

Published as a conference paper at ICLR 2025
codeql.der deninition Navigates to the definition location for symbol at specific position. Supports both single definitions and multiple definition locations.
Inputs:
f i le uri (string): The URI of the CodeQL. file
1 i ne (number): Line number (0-based)
chaaaaater (number): Character position in the line (0-based)
Returns: Location | Location |
null containing URI and range information for def-
inition locations.
Example usage:
"tool":
Marguments": " "file_uri"; "file:///workee hElles///orrsppc//ssrrroooo "line": 12, "character";
codeql references cedeeneeesssnces Finds all references to symbol specific position across the workspace. Includes both usage references and declaration references.
Inputs:
fi le uri (string): The URI of the CodeQL. file
1 i ne (number): Line number (0-based)
character (number): Character position in the line (0-based)
Returns: Location [] null containing an array of all reference locations with URI and
range information.
Example usage:
"tool"; "codeqL_references
"arguments": "file_uri"; "line": 6, "character"; 20
codeql.die diagnostics Retrieves diagnostics (errors, warnings, information messages) for CodeQL file. Provides real-time syntax and semantic analysis results.
Inputs:
file uri (string): The URI of the CodeQL file
Returns: Diagnostic containing an array of diagnostic objects with severity, message, range, and optional related information.
Example usage:
"tool": "codeq__diagnostiee" "arguments":
22

---

Published as a conference paper at ICLR 2025
"file_uri": "tlee///oorksppc//eeatttteerrrg
coreqt_format cQL style guidelines.
Formats CodeQL file specific selection within the file according to Cod-
Inputs:
file.uri (string): The URI of the CodeQL file
range (Range, optional): Optional range to format with start and end positions
TextEdit [] containing an array text edits that describe the formatting changes to be applied.
Returns:
Example usage:
"tool": "codeql_format"
Marguments": "file_uri": "file:/
"range":
"start": "line": 10, "character"; },
"end": { "line": 25, "character";
codeql update file Updates the content of open CodeQL file in the language server. This allows for dynamic content modification and analysis unsaved changes.
Inputs:
file.uri (string): The URI of the CodeQL file
content (string): The new complete content of the file
Returns: string containing a success confirmation message.
Example usage:
"tool":
"arguments": "file_uri": hftle:///werrspcc//eerrryy////// ql",
"content"; "inport cppin'nfrom Function f\nwhere f. hasName ( "stropyy \") ") Anselect f.
D CODEQL QUERIES
D.1 CODEQL QUERY STRUCTURE TEMPLATE
The template below is given the LLM agent the start of the iterative query synthesis task. The prompt instructs the LLM to use the AST nodes, along with the CodeQL LSP and CodeQL references in the vector database, fill in this template. The prompt also takes find similar queries related to the given CVE's vulnerability.
/+*
@name: [Vulnerabiliee Name based en analysis @description [Description derived from the vulnerabll lity pattern]
23

---

Published as a conference paper at ICLR 2025
Oproblem.severiee
@sccurity-severity [score based on severity]
@precision high
@tags sccurity
@kind path-problem
Bid [unique-id]
10 */
11 import java
12 import semmle. code. java. www.wwrrw..ws..c.mmmoo...oe....e ors..eoess..................... Networkingg
import semmle.cooe..java
import semmaae..ccs...paaaaaaaaaaaaaaat
1s import or
16 private import semmle. lat.eeecooo..aa.aaaaafflfl.. www.eernalltl...................
class Source extends DataFlow: ()urce
19
20
exists ([AST node type from analysis]
/+ Fill based on AST patterns for sources identified in Phase & 2
and this. () [appropriate mapping]
23
24
25
27
class Sink extends DataF low: :ode
28
Sink () {
29
exists ([AST node type]
/. Fill based on AST patterns for sinks +/ ahis. or exists ([Alternatvve AST pattern]
lappropriate mapping]
32
33
34
/* Additional sink patterns from analysis x/
and (appropriate condition]
36
37
38
40 class Sanitizer extends DataFl ow: :Node
41
Sanitizer()
42
exists ([sstt((ASS (IAST node type for sanitizers]
/+ Fill based on sanitizer patterns from Phase & +/
45
46
47
module MyPathConfig implements DataFlow :oonfigiig {
predicate isSource (DataFlow: source) {
50
source instanceof Source
51
52
predicate isšink Shatarlow::Nor sink). {
sink instanceof Sink
55
56
predicate isBarrier (DataFlow) :ode sanitizer) {
sanitizer instanceof Sanitizer
60
predicate ieRdditionalFlowsee (Da (DataFlow: :Nde n1, DataFlow: :ode n2) E
/+ Fill based additional taint steps from analysis */
64 }
65
66 module MyPathFlow :at
import MyPathFlow Low: :aathGraph
24

---

Published as a conference paper at ICLR 2025
from 70 :yFathhloow:PPathoood source,
11 MyFathFlow: sink
where
#fPathhh (source, sink)
14 select
15 sink.getNede().
16 source,
sink,
79
nource.getNonn (),
80
deource_descri inteerrpttio iption]
D.2
ITERATIVE FEEDBACK EXAMPLE
The following is an excerpt from the prompt used in a new context window that instructs the agent to improve the last iteration's query. The feedback is derived from automatically running synthesized queries against the fixed and vulnerable versions of the CVE's repository, and comparing the query's 's results against the changed methods in the CVE fix commit diff. The feedback also includes any successful method hits, false positive method hits, and which fixed methods have not been detected by the query.
## Objective
Refine the CedeQl, query based on previous iteration feedback to improve vulnerability detection.
# Previous Iteration Feedback
II Iteration 2 Results
## Previous Query (Iteration 2)
''ll
/**
10
* @name Ant iSayy XSS Bypass in Style Tags
8description Detects potential XSS vulnerabilities where AntiSamy HDML sanitization fails to properly filter malicious style tags with event handlers
12 @problem.severiie error
13 @security-severity 6.1
@precision high Stags security
16 Skind path-problem
17 8id jawra//mtiaannysttlllxxs----
18 //
import java
20 import semmle. code, waw. wwwpraaatalll..aatac..oa
21 import semmle.code java www.aaaall...aaaaaarreeooemo
22
class MalicicusHtmll Input extends DataFlow; Node MalliciousHtmmInnutt
25
exists (Stringliteraa s1 this. = si and
26
27
s1. s1.getValue matches www.aaathssaaatttae ("scstttlyylyleellaad33111
29
sl.getValue () .ww.ches (*#:styleesonc11ckk33)
30
. (*#csttyle@@nnerroobb)))
or
31
sl.getValuee)..me ( [stecassssccccc////ooomeeeeessss or
..macches s ("steeest"sttyllessletttt"""
sl.getValue() .mw.ches www.ches hataaaaaea...ataaaaaecccCcaaaaaa ("<sstyleejjjaarrrpppt
36
sl.getValue watchhs ("ssatyyeeccpprcessoon5))
35
36
)
exi exists (Parameter p this asParamttrt
and
25

---

Published as a conference paper at ICLR 2025
40
P. gegeeNamm O tolowerCase () matches ( ("Shtm18*) or
41
P p.getName () tolowerCase () matches ( ("iinput"") or
42
P.getName () tolowerCase () matches ( "#content3")
45
46
47
class AntiSamyProcessing extends DataFLow: :ode antiisayroocesinggll
50
exists (MethodCall mc
51
this. chis.asExpr () - me and
mc me.getMethod() hasSName ("scan") and
mc oec.gggl/Wstadddgeeeeeeaaaaaaaal getDeelaringType (RefType)
54
.hasuullifiedNanm (" ("org.
(wwrg. owasp. validator. iaddiiorottt",,, "AntiSamy")
55
) or
56
exists (MethedCall me mals.asExpr () and
me.getMethod hame (*getCleanHTMMI") and
59
mc.get.uallffier () get () (RefType)
60
hhasqualifiedii
val widaatoo,ttmm "CleanResults")
63
64
65 class Potentiallsssuutptt extends D :ode {
Node
PotentialXssoutpuut
exists (MethodCall this. () me and
68
69
70
me geggMMtttod)) haasName ("centains") and exists (StringLiteral sl me. gettnrtnnrggu sl and
72
33
74
s1 .l..geValleel maachhe ("aalert3") or
s1 geeeeealleeeee .macchhe (" ("Ascript*") or
s1 gi.eeeallls .matches ("sjavascript3") or
77
s1. getVelel () matches ("ateess""onns) ("Sonloads") or
78
sl. sl.getValue () .mttches ("Sonclick3")
82
) or
83
exists (MethodCall mc asippr () = and
mc.getMethod hasName (*getCleanHTMM"") and
86
me.getQualifier () getType () (RefType)
87
.hasqullifiedManm ("org.owasp. owasp. validator...imi "CleanResults")
90
91
92
class ImprrppersSanttiaaatiioararree extends DataFlow: :ode ImproperSantilatiiooiirrerr exists (MethodCall this aats.asExprt () me and mc wr..getMtthodd . hhaNName ("assertTrue") and exists (LogNotExpr Inc
95
96
97
mc getAnArgument () 1ne and 1ne gne.getExpr (MethodCall) getMethod () hasNaame ("contains")
100
101
26

---

Published as a conference paper at ICLR 2025
module SntisanyxssConfig implements : 106
predicate isSource (DataFlow: :Node source) {
107
source instanceof MaliciousHtmlIIpuu
110
predicate isSink (DataFlow: :ode sink) {
111
sink instanceof PotentialXesoutpur
predical cate istarrier (Datarl :Node sanitizer) { sanitizer instanceof Impropersaniiiiatiinnnnrr
115
116
predicate iskddittionaFFoowttee (DataF :odd nl, DataFlow: :ode n2) exists (MethedCall me n1 al. () mc ge.ggtqualifier () and
119
120
n2 as () me and
me.getMMthodd)) hhasName (*scan") and
gc.getMethod get.beeclaringgypee (RefType)
124
.hasuuali wwwor..wwwa......o.ooo..p....... waliaat...ladaoo........oo...om. htmmo,, "AntiSamy")
125
or
exists (MethodCall mc asxpsE)p me. getQualiiierrr and
128
n2. asEppr () me and
129
mc. mc.getMethod hhasName ("getCleanHTML")
133
module AntiSamyXsaflov = TaintTracking: httpa//aaaaaataiiooooooooooooaaa haa.:::::ccoooaaaaaaaaaaaaaaaaao import AntisamyKosFlow: PathhGrpph
134
135
from
138
Ant SatiiaayXasF1oa Flw:: : Noae source,
where 139
AntiSamyXssFFoww low: Path sink
AntisamyXaar1on low: flowPath (source, sink)
142
select
143
sink.getNode()),
source, sink,
http::/wwww...poo..ooooooooooooe "malicicus_AHMM_.inn input
147
source.getNode
151 ## Compilation Results
COMTION SUCCESS: Query syntax is valid
Execution Results
155 #I Query Evaluation Summary (Iteration 2)
156 Results: Vulnerable=6, Fixed=8
Method Recall: Vulnerable=True, Fixed=True True Positive Methods: Vulnerable=2, Fixed=2 Coverage: 1/1 target methods
160 PARTIAL: Query hits targets but has false positives in fixed version
161 Mothod location format is path/tc/hit/Ello. [Class of hit method] Hit method]
Successfully targeted methods:
163 hwtp:/main//aaaa//r////s/ss htwpa///aaaaaaaa//eteaaaaa dater /atddr//hha////aaaaaaaaaaaaa java:
False positives (hits in fixed version)
27

---

Published as a conference paper at ICLR 2025
http://aaaaa...///////aaaaaaa/aa Magl
166
## Detailed Evaluation Analysis
Method Coveragex:: target methods
170
**ii Coverage**: 1/1 target files
171 * Successillyy targeted files**;
MagicsMxFllerr.jav
Successfully targeted methods**:
175
startElement
176
False positives (hit in fixed version) ax1 startElement
179
180
Evaluation Summary**:
Vulncrable DB File Recall: Truc
Fixed DB File Recall: Truc
Vulnerable Method Recall: True
184 Fixed DB Method Recall: True
185 Total Query Results: Vulnerable=8, Fixed-8
Code Flow Paths: Vuineraole=8, Fixed=8
188 ## Next Steps
189 We want vulnerable DB method recall and we don't fixe Baceed_________ me_hhho recall! Priority
to_ wtw.:aatttttttttttttetteomm http://ettttdd____ddddee__edeaae Listed_above
D.3 QLCODER SYNTHESIZED QUERY EXAMPLES
Below are examples of successful synthesized queries. The queries successfully find > true positive method hits the vulnerable CodeQL database of the CVE's source code, and no false positive method hits on the fixed version's 's CodeQL database. For reference queries to compare with, CWE queries can be found on the official CodeQL repositrry (GitHub) 2025h g)
CVE-2025-27136, CWE-611 Improper Restriction of XML External Entity Reference
/**
@name XML External Entity vulnerability in WstxInputFactory without secure configuration
8dcscription WetxInputractory used in XmlFactory without disabling DTD support and external entities allows XXE attacks uproblem.scvere error
@security-severitty 9.1
* @precision high
Stags security
8kina path-problem
Bia java///ssxiippttaactoy-xxxx
10
+/
11
12 import java
13 import semmle. oot..cood...a.aaaaaooooootttaeee
14 import semmle. ooo.
15 import sew.::e..ce...ee...attt.........
16 private import semmle. code. wtw..oo.a.aaaaaaaaaaa...ooo Exw.mmaaFFattaaaa
18 class MatkInppttaccttoryrceat...oo extends DataFlow: :Node
19
Wetxchpputtaattccooeerrttaaal ()
20
exists (MethedCall me
getMett () hasquall inttaaliffeaaaaa fiedName ( ("com. wstx.stax".
nptx// "newl natance")
28

---

Published as a conference paper at ICLR 2025
me.getMethood() hasqualiffeddarre ("com, www.m..t..sttt.......o.......... " wetxInputFactory", "newPactozy")
23
24
this. () = me
exists (ClassInstanceexxpr cie
27
cie cie. ret.geeeoossruuusddyyyeeee () () fifedNamee (* ("com. wom.ctc............. wstx. stax",
WatxInputFactory") and
28
this this. = cie
// Include variable access to watxInputHactoryy instances (like "input parameter)
31
exists (Variable VarAccess va v.getType (RefType) l hwwtualllieeeaaaaeeeaaaaaaeieeee ("com.cter wstx.stax", MstxInputFactory") and
32
33
va.getVariable() v and this.asExpr = va
34
35
37
38
19 class SesafexmlFactoryyusagg extends DataFlow: Node UneadexinlFactoryuagge
{
40
exists (ClassInstaneeExo xmlFactoryCall
42
// XmiFactory constructor with WstxInputTactory parameter
43
ammFFactoryall..gg htw.orrcccal.etccoootttttteeedee
44
hhasuullifiedNaaee fasterrm.. www.errt.......aoooooooooo backson.dataforman. xm1",
XmlFactory") and
xmlFactoryCall getArgumet (0) this this.
46
or
47
exists (ClassInstanceexpp xmlMapperCall, ClassInstanceExpr xmlFactoryCall
// XmiMapper constructor using XmlFactory with RstxInputFactory
89
50
thasuualifiedName (ocom. fasterxm .jackrr fasttrrrnl..dsssnnaaaaffrre formatt. xm1", Xmi Mapper") and
xmlFactorycal1, htt.aacccryalll.eeccossttraaaaae getcoonst
hhasqaalfiiedaanl cocom. fasterxml. jacksso data format.xm1", XmiFactory") and
53
mmlMapperCall.getrrumme (0) xm]FactoryCall and
ielaactoryCalll.gtrruuuie (0) = asxprr
56
57
58
class NatxImputFFattorySnniitierr extends DataFlow: :Noe {
61
exists (MethodCall setFropeztycall, VarAccess factoryVar
62
itrerpppertyalll..ettttttto htg.etttoddd.sssuuuuiiiieeee www.uaalireeeaaaa.e..... (" ("javax. xmi. strean",
XMLInputFactory" "setProperty") and
htw.rperrrcalll.ggguuaaaaaaaa getqualifier factoryvar and setPropertycal1
65
// DTD support disabled (exists f
66
(0) f.getAnAccess and hhhasName ("SUPPOOR_T DTD") and
SusPropertyyalllgettrrmuu
f.getDeclaringtypo () hasqual wotoquuliifddaaaa ifioNName (" javaxx xml. strlssrram",, XMLInputFactory")
70
) and
exista f porpoparttcalllggtttrrmmee f.hasName (MFAll.s)) and
72.
F.getAnAccess and
73
74
E.getDeclaringTypee () www.uualiiiieieee (" ("java. "Boolean")
15
)) or
// External entities disabled
29

---

Published as a conference paper at ICLR 2025
(Fiied
78
steepprrtycalllgeeergggeenenn I.getAnAccess ENTITIES" and
79
foctaa f. F.getDeclaainggYyoe () www.hhaalliichaaaaaaa. ( javaxx xm. stramm,
80
XMIInputFactorry)
and
82
(Field f
83
seterooerttyal. (1) = f.getånAccess () and h. ("FALSE") and
E.getDeclaringJype () was.uasuuiaa. Liniuuuaeeeedamme (*java. Lang", "Boolean")
)
87
and
88
this.asExpr = factoryVar
91
92
93
module New.npuuFaetorryooooonnn implements DataFlow:
predicate isSource (DataFlow: :Node source) { source instanceof vetxiinpuFFttoorrtrrttttto
96
97
precicate isSink (DataFlow: sink) | sink instanceof UnsatexniFFactoryyaagee
101
predicate isBarrier (DataFlow: :Nede sanitizer) { sanitizer instanceof Netxinguttaccoorysattiie
106
predicate
nl, :ode DataFlow::Nel :ode n2) {
:ode
// WstxInputFactory passed to XmlFactory constructor
exists classsstttst xmlFactorycall
xmlFactrrycalleeettstttteetttttd whwsuualifiedNaaee c*com. fasterxm jackson format. xm1",
Xm1 Factory") and 110
xm1ractorycal1 .wt.ragttttttttueettttt..m (0) nl mll.ssxxrr () and
n2.asExpr() xm1FactoryCal1
113
) or
114
// XmiFactory passed to XmIMapper constructor exists (Clsssssssssssseeeeee xmlMapperCall
xmiMapperCall http://caaacleetoooooeeeeeeeeeee
whwguallfieddamm c*com. fasterxml Jackson waw.ksoondaaaaaomaa... aeee
XmiMapper") and
118
xmiMapperCall.co .eetrruuuggt (0) = nl. () and
n2 n2.asExpr () xmlMapperCall
121
122
module hstxinpptractooryyioo - TaintTrackine ng: :61obal /stx://puutaattooryloooooooo
125
import =
126
from Metdinoothacccoyyloooo.aPaaaaa scurce, waw.mppttraacccyyloo..aaaaoooddd sink where tetxIpptEaattorlooo:.ooooott sink) select (ink.getNode().
130
134
source,
135
sink, htwp://tttttttttoeeeeeeeereteeee
l. alwwwng XXE att_kkta
30

---

Published as a conference paper at ICLR 2025
source getNNoeeNN Wettxingattaccooruuuugge
CVE-2025-0881, CWE-22 Path Traversal
/++
2 @name Archive path traversal vulnerability (aipslip) CVB-2025-0851 edescription Archive entries with path traversal sequences can write files outside the intended extraction directory uproblem.severitg. error
@secuzity-severitt 9.8
*
èprecision high
Stags security
0kind path-problem
lid http:///tteeeeaaaaaaaaaaaaaloo--
10 */
11
12 import java
import semmle.. code. www..aataa.oo..Dettt.ooo.o.
14 import semmle.. semmle.code, jaw.ee..aaaaaaaaaaaaa.a. dww.flaaal.......... Tainf.rwwn..
15
16 /**
Sources: Archive entry names from dipEntry.getNaae and "arrrrchi
18 */
19 class ArchiveeintrylameSoorre extends : { ArchhiveEntryyaaeSooue
exists (MethodCall
22
mc getMgeMM gettNam () "getName" and
(mc. (mc. gettettoo
getDeclaringTypel () www.shaaaaiieedee... ("java, aaiva.uti..zipp. "ZipEntry") or
mc getMethed
hwwquallficddeee.e. "org.apache. cow.prse..rrciiiiei..... tar", "TarArchiveentzyy) and
.getDeclaringyype
commons.
24
25
this.asExpr. () = mc
26
28
29
30 /**
Sinks: Path resolution operations that lead to file creation
*/
33 class PathCreationSink extends DataFlow: :Noe PathCreationSinkii {
34
35
// Arguments to resolve () calls
exists (MethodCall resolvecall
37
resclvecall. getMethoo .getName "resolve" and
38
resclveCall gettethho .get@eclaringgTyelll
39
("java..
"Path") and resolvečall, vecall wtt.aellla.lllllleeeeeeeee getAnArgument ()
this.asfxpr () resol
42
or
43
// Arguments to file creation operations
44
exists (MethodCall file0p
getMethod () .getName () "ereateDirectories"" or
47
getName () "newoutputstreean! or
48
.ellepp.geMMehhod () getNNme () "write" or
1.le0p.geeMettie geeNName ""opy" and #ilcop.getMethodl x .gtDeeclaringTyee www.qualllaaa......am.aa..a dasqualifiedNams ("Java. file", "Files") and
52
asEpp()) = file0p. weree.gettnaaan getAnArgument ()
31

---

Published as a conference paper at ICLR 2025
56
57 /.*
98
Sanitizcrs: Proper validation that prevents path traversal
*/
class PathiravversllSanitiierr extends :oteelow::Node
61
Patifraversallsmitizzerrr/
62
// The validateArchiveEntzy method call that properly validates paths
// This blocks flow after the validation call is made exists (MethodCall validateCall
val validatecall. .ettethhh () getNName "validates "valicateerchiveentry and
66
67
1/ Any variable assigned from validateArchiveennrry call result exists (Variable v
this. asExpr v.getAnAccess and exists (AssignExpr assign
20
71
assign. (e () v. v.getAnAccess () and
72
assign.gotRhs = validateCall
35
or
76
// Variables passed through val velidateeAcchiventee calls this asEppr) validateCall. settArrruttt () and exists (ExprStmt stmt stmt getExpr () validateCall)
77
19
80
81
or
// Proper ".." validation with exception throwing (complete pattern)
exists (MethodCall containsCall, IfStmt ifStmt, ThrowStnt throwStmt .eettaea getName "contains" and
84
85
contaiinsalll..etttrrrgggmmm (atrraaeeeuuuttttttt.trrraaaaeee (StingIitter)))) .gtVaalue)) = ".." and ifstmt l.s.mt.gettonddiee.......... g o gettachieeeeeptcc containsCall and ifStmt. () get&child+ throwStmt and asipp contalnscall.gettuullfiiie getqualifier(() or
86
88
89
90
// Path normal ization combined with startswith val idation exists (MethodCall normalizeco normalizecall, MethodCa11 ChrrsawithCaa. hormauieeeelll..tttttttccct.cc
92
93
.geetaaeee geetName "normalize" and
94
"Path") and starcaattticalll.eeceecehhheeeee = "startswith" and
97
98
hasouualIIccccaaeer.ro.o.......r "Path") and
99
DataFlow: bbaafpa.w..sppppooeeccrrrrrrtaar ())) and
this. () normal
102
103
107
Additional predicate to detect validation barriers at method level. //
108
predicate hasValidationCalll (Callable method)
exists (MethodCall validateCall
valicateCall wtw.tccaallgttmmeeeennngaaaaaaaa
method and
112
malicattcalll.getteett () .getName = "valdatcarcciveennez""
113
116 module PathTraversa Config implements Prttallo:::oofigg::
117
precicate isSource settaFoow::Noae source) { source instanceof ArchiveEntsy/aaneSuun
32

---

Published as a conference paper at ICLR 2025
predicate isSink (DataFlow: sink) {
122
sink instanceof PathCreaticnšink
predicate isBarrier (DataFlow: :Node sanitizer) { sanitizer instanceof PathTraaveralleemttierr
126
127
predicate isParrierIn (DataFlow: :Nede node)
// Barrier at method entry if method contains val caliddttercchveentty call
131
node instanceof Datarlow: : ParameterNooe and hasValidationCall (node. (ww.eeallnnnneeteeeeeeeeeee (nodeee.etnncclesnnnaaaaaaeee
134
135
predicate LeAdditionalFlowStt : n1, DataFlow: : :ode n2)
// Flow through variable assignments and declarations
exists (LecalVariaabeeeelxxp decl decl. ()cl.getInit) nl al.asExpr() and
139
n2.asExpr() deal getVariable))) htw.rriiaablll.ggttneeeeeesse getAnAccesss ()
140
or
exists (AssignExpr assign DesignngettRie ml asxprr () and n2 a2.asExpr
143
144
145
or
// Flow through string manipulation methods that preserve taint exists (MethedCall me
148
149
mc no..getnnArgumett () = nl al.asExpr() and asEprr) me and
me. ge..eeeMthhd gettame () "venoveeeddinEEllepppaattor"" or
153
mc gec.ttMMt () Name () "trim" or
154
mc mc. ge..eeMeh () g... Name () "toString" or
me. () Name "substring"
157
)
158
or
// Flow through Path operatiens exists (MethedCall pathop
n1.asExpr () and n2 as pathOp and pathOp. getMethod .getName = "resolve" and | pathOp. getMeth () .getbbeclaringgyppl
pathop. gettnargegtnnnr
162
163
LhasQualifiedName java. .ic. fil.file", "Path")
167
module Pathiraversallfoo raintirrccking::Gl http://ccooooaoo.a.a.aaeaaeooooo hrw.a
171
172
import PathTraverealFlow: PPthhrrpph
from PathTrawerrsallooo:PPPP...C scurce, PathTraversaalFlow: :atthooee sink where
176
FathrkawrrsalFooo::1looettt (source,, sink) and
// Focus on the specific vulnerable files and methods
source, Cource.getNone getNode getttmcloosingallllllll eetDeeaaaiiagggtepee hasName ("TarUtils)) or
180
181
source perice.getion getNode () .gttEnslosinnaalLaeelll getDecLariigTTpee
hasName ("ZipUtils")
and
33

---

Published as a conference paper at ICLR 2025
source .earee.geetee getNode .eettnclosinnaalllabell getName () "untar" or
186
Cource.getNoon () -getenelosingallllbbll () getName () "unzip" and
187
// Only report flows where validation is NOT properly done
hasValidationcan (eouroee.eeeteeeecee.... (et.c
190
select sink getNode)) source, sink,
191
ottp://aa..aa.toooo.aooooooaoaaa source.getNoade "archive,entzy,nemee"
prope_,eat..rteerss..... ttvarrs....aiaaatonn
CVE-2025-27528, CWE-502 Deserialization Untrusted Data
@name MySQL JDBC URL parameter injection vulnerability edescription Detects MySQL JDBC URLS with dangerous bracket parameters that bypass inadequate filtering in vulnerable code @problem.severity error
@security-severity 8.8
éprecision high
* Stags security
8kind path-problen
Bia jawp//yyal--ddooou---nineettt
*/
11
12 import java
13 import semmle. code. htt.:jjjaaaaaaaaooooooo DataFlow
import semmle. www..aalwww.......ooooooo..... wtt.:aaaa.......iniaaaaa.a.a.
TaintTracking
15
16 class wySoDCanggeonnstssseettroooo. extends DataFlow:
17
// String literals with dangerous MySQL parameters in bracket notation
19
exists (Stringlttea lit
20
1it itt.getValue() .matches ("smysql*") and 1it. lit.getValue matches ("*[*]*") and
23
Lit.getValue() .ma.ches ("a.achhhennaLoooottttttaaaiiiia
or
24
lit. mat.ches (*atcheeee=allloo01llio.oaa
or lit.getValue .matches matcche ("aaatttbbbeerrillleee)) or
25
lit.getvalue .matches (mg
lit.getValue .matches ( or lit.getValue ..a.chhessstc... ("*password=") and chis.asExpr lit
28
lit.getValue () (astches(**useree
29
30
32
33
or
34
// Parameters to filterSensitive method that may contain dangerous bracket content exists (Method m, Parameter and "#sg@Seesitiivebiiltti. and m. m.ggeDDecaainngype () ().getamm
36
m hagName "'iltterenssitvee"
37
38
m. mn.getAParameter () and
40
41
42
class VulherabllecoaaPstterriiin extends DataFlow: :ode { WinneaabllccdeePatttnnini
15
16
// The vulnerability: calls to filterSensitive in vulnerable code patterns
exists (Methed m, MethodCa11 filterca11 h.haaa (#Filtersensitivee and
34

---

Published as a conference paper at ICLR 2025
m. geeteeelaaainggype () = and 90
filterCall, getMethod g c = filterCall and // Key vulncrability conditicn: this code exists where fi Battersensitiveeeyyyyraaaa method is NOT available
and
51
52
// In the vulnerable versicn, filterSensitivek fllterreesstiveeeyyyyyraeee doesn't t exist
54
not exists (Method bracketMethod bracketMethod, hasName hww.aaaaaeoeea..aaeee...a brrccktttttbod..gbeeelaaaaan Mysiplienssitivveurlltiiaa and
55
and
getName
57
bracketMethod l
m. m.getieecaaringiyy
58
61
// Additional sink: method calls that use the result of inadequate filtering
62
exists (MethedCall me, MethedCall filterCall
wtt.errcal..geMettheeeee haaNName ("filterSensttvee" and
filterCall. () gettDeclaringIyye () -getName ()
My.8pl.sensstiivveel1101... and
65
lattaloo::
(filterCall), DataFlow::
exprNode (mc. getArgumett (_)) and this asExpr
and // Only vuinerable no proper bracket filtering exists the same class
67
68
not exists (Method bracketMethed
bracketMethod. hasName http:/eeeeetooo.eeeeeeeaiaiaiiee by:30..enssttiive0:1l0iiis
hassam wff..eersenssiiiieeeeyyryrr
and
gettamm =
and http::/wwwwwoooeetteeeeerrrrggge = SeiterCall.getWethoe getDeclaringTypo
71
73
34
75
77
class Propertrraccet////eeernggaa
extends Iatafloww :ode
18
FroperBracketF AttaateetiLterinnnnniiiiees
79
// The proper bracket-based sanitization method (present only in
fixed version)
exists (MethodCall
mc me.getMethod hhasName (ffleeessnstiiiveeepyzereee and
82
mc mec.geMMehood) getDeeearinggypo () gotName ()
wysoLSennaitivveeilliiiss and this.asExpri () = mc
85
86
87
module MYSQIJDACUU1 InjectionConti implements Datarlow::Coffighe :ConfigSig {
predicate isSource (Datarlow: :Ncde source) { source instanceof hysp.laaneercaarrraeee.u.....
90
91
predicate isSink (DataFlow: sink) { sink instanccof VulnerablccoooePateeemmm
95
96
predicate isBarrier SlataFlow::Ndds sanitizer) { sanitizer instanceof Properthaceee/llleenngaatteee
99
100
precicate isAddition [DataFlow: :Noe // Flow through string concatenation operations
:Noee ni, DataFlow; Nooo::ood :ode n2)
35

---

Published as a conference paper at ICLR 2025
exists (AddExpr addExpr
n1.asExpr() () aldxxpr.geLLetttoerannndy and
105
n2 aa. () addExpr
or
exists (AddExpr addExpr
189
n1 al. () addExpr.getRiighttpeee... and n2 as.prE addExpr
110
// Flow through variable gnments exists gAmsignment assign
114
115
n1 n1.asExpr
and
n2 a2.asExpr() assign.getDest
118
or
119
// Flow through return statements exists (ReturnStmt ret
n1 nl.asExpr() ret.getResult() and
n2.asParameter () ret.gettindlossnn@aallabbee gettAaaammter ()
123
124
127 module MyF024DBBBUULLnnjsttttooot.ox TeintTracking: :Global<
import MySSJ/UBeUU111nnccttioooo0o::
131 from o:0.00pee00..niaaertttoroorrooae source,
wyg..0pp000lmmgsccttt.poooo.ddd2 sink
where wyg.uppoculmj/ccttoooooo::ooeoe: (source,, sink)
select sink.getNode()),
source, sink,
fi htcpllaccksppppr_bbb_ proper_bracket-baaeet
inn",
135
source. ()t (iaagerous_URl__on
D.4 AST EXTRACTION QUERY
Given fix diff, QLCoder automatically parses the changed methods and files, and inserts them into an AST pretty printing query template. Below an example of the AST extraction query used for CVE-2014-7816.
/**
@name Exprcssions and statements for CVE-2014-7816 changed code areas @description Extract expressicns and statements from vulnerability fix areas
*
@id java/expr-ssmttdd (FFV--22fffcC___11___81
@kinn problem
@eroblem.seeee severitty recommendation
*/
import java
11 from Element e, Location
12 where
13
1 = e (.getLocation() and ((1.geFFile) getBassaame ()
PatuieparattrrHandlee.... java"
and 1. L.getStartLine () and L.getEndLine() 100)
15
16 (1. (l.getFile() (aaeeiilee.getBBseeaaaa getBaseName () "URLDecodingHandler: java"
17 and L.getStartLine() >= 17 and 1. L.getEndIine () <= 128)
19
(1.getFi 1e Nete "ResourceeHandler java"
36

---

Published as a conference paper at ICLR 2025
and 1. (e () 158 and (. 172)
22 or (1.getFile() gettaaseeae () "io, handerttww.srrorr..aalllsss.
HandlcrBuilder" and L.getStartLine >= 17
and L.getEndLine() <= 29)
25
or (1.gettFlee) -w..eeeillee.gettteeeeeeeeeeee get () "DefaultsServlet.jjaaa"
26
and L.getstartLine () >= 39
27
and L.getEnaLine() <= 150)
(1.getrile() get and 1.getstartLine () > 32
"
10
select E, 31 and 1.getEndLine () <= 140))
t) as element, () as elementType, () as filc,
36
35 (.getFlic()
36 L.getttartinne)) as startLine,
1. (e endLine,
1. gggtttaattollmm startColum,,
1 1. I.getnndCollme as endColumn
37


Reviews:
[{"review_id": "KGWpbEFV1R", "reviewer": "Reviewer_XjUp", "summary": "This paper presents FineNib, a system that automatically synthesizes CodeQL vulnerability queries from CVE text and vulnerable/patched code. It uses an LLM agent with execution feedback and structured tool access (CodeQL LSP + RAG) to iteratively refine queries. On 176 Java CVEs, FineNib achieves a high rate of successful queries and good F1, substantially outperforming IRIS and built-in CodeQL rules.", "questions": "1. What is the average runtime per CVE?\n2. How sensitive is the system to CodeQL version changes?\n3. Could the same iterative loop work with a weaker LLM if aided by stronger program-analysis constraints?\n4. For multi-file or multi-snapshot vulnerabilities, how do you avoid overfitting to the specific patch change?\n5. Can the pipeline detect variants of vulnerabilities (not only the exact CVE instance)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Tackles a practical and timely challenge in security automation.\n2. Clear design: LSP for syntax, RAG for semantics, feedback loop for correctness.\n3. Strong results across real-world projects; large margin over baselines.\n4. Good ablations showing importance of LSP + retrieval.", "weaknesses": "1. Only Java CVEs are supported; unclear how easily this extends to other CodeQL languages (C/C++, JS, Python). Authors mention RAG replacement, but actual cross-language results are not shown.\n2. The system leans on Claude’s coding agent + CodeQL LSP. It is unclear how reproducible this is without Anthropic tooling or whether open-model support is viable (Gemini/GPT baselines fail).\n3. Iterative refinement with code execution can be slow. Wall-clock time, computational budget, and iteration cost are not reported.\n4. The metric assumes \"raises alarm on vulnerable version and silent on patched version\" as correctness. But some vulnerabilities have multiple exploitation paths; authors explicitly allow false positives, which may inflate success numbers.\n5. The work is strong for static analysis automation, but the conceptual novelty in LLM reasoning or program analysis theory is incremental. It feels like an engineering system rather than theoretical innovation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iVbXVTiqGT", "reviewer": "Reviewer_n8Qx", "summary": "The paper introduces FineNib, an agentic framework that generates CodeQL queries from CVE metadata. To mitigate the pitfalls of pure LLM-based approaches, FineNib utilizes iterative reasoning with syntax guidance and a RAG database for semantic retrieval of queries and documentation. FineNib is evaluated on CWE-Bench-Java with Claude Sonnet 4, where 100% of generated queries complied, and 53.4% successfully detected the vulnerability. This is a much-improved result over the vanilla agentic baselines.", "questions": "1. It is not clear from the text if the performance was measured under a regression testing scenario or a variant analysis. It is important to know the performance separately between these two scenarios. What is the performance of FineNib in the variance analysis scenario (i.e., generate queries from one issue and detect other issues of the same type)?\n2. Since the static analysis query is a low-resource language, would finetuning a smaller model specifically to generate the code for this language be a feasible approach? Would this potentially reduce the operational cost for a trade-off of a slightly larger cost upfront?\n3. Could an additional open-source LLM be added for the evaluation to create a more complete evaluation picture?\n4. Do you have some cost statistics that you can quickly perform a cost vs effectiveness analysis of FineNib?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The application of script generation for vulnerability detection is a novel application, and with the help of RAC, the approach can mitigate the issues with low-resource static analysis query languages.\n+ The approach shows great efficiency, outperforming vanilla LLMs and original CodeQL analysis by a big margin.", "weaknesses": "- The evaluation shows that FineNib can successfully generate good queries. However, the evaluation of a single LLM makes it somewhat incomplete. Claude Sonnet 4 is a very powerful, expensive, closed-source LLM; knowing the effectiveness of FineNib with a weaker open-source LLM would make the evaluation more complete.\n- Since agentic approaches tend to be more costly due to their iterative refinement process, a cost vs effectiveness analysis could inform the reader of the cost per performance gain. Having an additional open-source LLM in the evaluation with this cost analysis can also paint a more complete picture of the overall efficiency of FineNib.\n- Since the baseline agents with Claude Sonnet 4 are not presented in Table 4, it is hard to follow (it is kind of presented in the ablation study). Having Claude Sonnet 4 as the baseline agent would help highlight the effectiveness of the contribution more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjyIbHEnOa", "reviewer": "Reviewer_hinK", "summary": "FineNib uses LLM's to synthesize CodeQL queries for a given CVE. Instead of doing it directly with an LLM, they integrate it into a loop using execution feedback to iteratively repair the CodeQL query that is generated. This execution feedback helps ensure that the query is correct syntactically and semantically. They show large improvements on CWE-Bench-Java compared to current state-of-the-art approaches, achieving an F1 score of .7.", "questions": "- Why do we need a CodeQL query for one specific vulnerability? \n- Along the same lines, what can that query be used for besides the vulnerability that has been patched?\n- Can the authors elaborate more on why IRIS has such a low F1 score? Is it because IRIS still reports the vulnerability in the patched version?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I think there are some interesting ideas in this paper when it comes to neurosymbolic program analysis. I think that using execution feedback to guide the LLM to synthesize a query is a very good idea. There are many parts of program analysis that are tedious and difficult (writing CodeQL queries is a great example of this!). By incorporating an LLM into the loop and providing some feedback, I think we can make program analyses much more accessible and easier to write. I also liked the discussion section on alternative designs - it really helps highlight that more information is not always better when it comes to using these models. Also, performance is greatly improved using FineNib compared to other methods.", "weaknesses": "Unfortunately, where this paper falls apart for me is in the motivation. I don't understand why we need a query for one *specific* CVE, especially given the fact that that CVE has been patched. To my understanding, IRIS and other tools write queries that detect a general weakness pattern, like code injection (CWE-94). These queries can be used to detect that weakness pattern in other projects. But a CVE is just *one* instance of a CWE: \"CVE-2024-12345: Buffer overflow in `foo()` of `libbar` 2.3.1 allows remote code execution\".  I don't understand why we need a query for just that instance, *especially* given that it has been patched. It seems to me that the query would be useless from thereon out. \n\nIf the authors could show how FineNib could be used to discover *new* CVE's, that would be a huge improvement to this paper. But from the current version, it seems that FineNib is just detecting vulnerabilities that have already been detected and patched.\n\nSome other small nitpicks:\n- Figure 1 implies that FineNib needs only the CVE information. But it uses the CVE information, the vulnerable code, and the patched code.\n- The first sentence of the introduction (line 030) states that security vulnerabilities are growing, but doesn't state how many CVE's were reported in 2023, 2022, etc. \n- I'm a little confused at why IRIS has such a low F1 score. IRIS reports an F1 score of .177 in their paper, but it is .048 here. Specifically, the authors state that \"The lack of true positive recall is why CodeQL and IRIS have significantly lower precision\". Does that mean that IRIS still reports the vulnerability when it has been patched? I think this could be clearer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2rqis7NV7w", "reviewer": "Reviewer_5Esz", "summary": "The paper presents FineNib, an agentic framework that synthesizes CodeQL queries directly from CVE text and patch signals. An LLM runs in a generate→execute→feedback loop and is constrained via a minimal MCP tool suite: (i) an LSP for syntax/API diagnostics and completion, and (ii) a RAG component to retrieve documentation, example queries, and small AST snippets. On CWE-Bench-Java (176 CVEs across 111 Java projects), the authors report a 53.4% success rate (fires on the vulnerable version and remains silent on the patched version) and strong F1, outperforming IRIS and off-the-shelf CodeQL query suites.", "questions": "1) Can you report a detailed cost profile per CVE (time cost, token usage), along with a breakdown by project size and CWE category?\n2) What minimal changes are needed to port FineNib to Python/JS/C/C++? How would you replace or adapt LSP/RAG for other engines (e.g., Semgrep rules, Infer)?\n3) Could you provide a controlled cross-agent evaluation where multiple LLMs use the same MCP interface and tool set, with end-to-end success rates and error categories?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Translating CVE/patch information into executable static-analysis queries has clear value for regression testing, variant hunting, and patch verification in security engineering.\n2) The constrained (MCP) tool set with LSP+RAG is a practical and novel combination that reduces hallucinated identifiers, version drift, and brittle syntax—common failure modes in code-query synthesis.\n3) Treating “patched-version silence” as a first-class success criterion is well-motivated. The path-level metrics that intersect with patch deltas provide a principled notion of precision/recall beyond simple hit counts.\n\n4) FineNib achieves substantial gains over IRIS and generic CodeQL suites on shared CVEs, and the reported overall success rate indicates meaningful practical headroom for automated coverage.", "weaknesses": "1) Limited external validity across languages and analysis engines; the study focuses on Java with CodeQL, leaving portability to Python/JS/C/C++ or to engines like Semgrep/Infer unclear.\n2) Insufficient transparency on cost and throughput; the paper does not report detailed per-CVE runtime, tool-call/token breakdowns, or scaling with project size/CWE category.\n3) Potential bias from metric design; scoring based on intersections with patch deltas may favor patch-aware synthesis and disadvantage baselines not modeling sanitizers or taint steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JtYbdXlNG1", "reviewer": "Reviewer_tkkv", "summary": "The authors present FineNib – an agentic framework that automatically synthesizes queries in CodeQL, a powerful static analysis engine,\ndirectly from a given CVE metadata. FineNib embeds an LLM in a synthesis loop with execution feedback, while constraining its reasoning using a custom MCP interface that allows structured interaction with a Language Server Protocol (for syntax guidance) and a RAG database (for semantic retrieval of queries and doc\u0002umentation). This approach allows FineNib to generate syntactically and  eman\u0002tically valid security queries. The authors evaluate FineNib on 176 existing CVEs across\n111 Java projects. Building upon the Claude Code agent framework, FineNib\nsynthesizes correct queries that detect the CVE in the vulnerable but not in the\npatched versions for 53.4% of CVEs. In comparison, using only Claude Code\nsynthesizes 10% correct queries. The generated queries achieve an F1 score of\n0.7. In comparison, the general query suites in IRIS (a recent LLM-assisted static\nanalyzer) and CodeQL only achieve F1 scores of 0.048 and 0.073, highlighting\nthe benefit of FineNib’s specialized synthesized queries", "questions": "Please address my concerns in **Weaknesses**.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The topic of CodeQL query generation for vulnerability detection is both timely and technically demanding. It presents significantly greater challenges than code generation for mainstream programming languages.\n- The development of an MCP interface demonstrates substantial engineering effort. This contribution is valuable and enables structured reasoning and provides a foundation for future research in automated vulnerability analysis.\n- FineNib achieves notably higher compilation, success, and F1 scores compared to existing methods.", "weaknesses": "- The authors claim that \"Permitting free online search for vulnerability patterns or snippets similarly proved problematic.\" It would be helpful to provide empirical evidence or examples supporting this observation.\n- While it is understandable that the presence of both vulnerable and fixed code versions simplifies the synthesis process, the realism of this assumption is uncertain. In practice, software updates often contain numerous unrelated edits (e.g., functional updates) to the affected CVE. How robust is the proposed method to such irrelevant changes between the two versions?\n- The evaluation section refers to “CodeQL (version 2.22.2) query suites” as a baseline. This description is somewhat vague—could the authors clarify what specific query suites or rule sets were used? Since CodeQL itself does not autonomously generate queries, it would be useful to understand how these suites were selected and executed for comparison, and what they represent in the context of automated query synthesis.\n- A broader weakness of the work is that it appears predominantly engineering/application-oriented, with limited discussion of principled methodologies for code generation in low-resource or domain-specific languages such as CodeQL. A stronger methodological foundation could enhance the generalizability and scientific contribution of the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)