## 3 Methodology
### Data Collection  
To construct a representative and well-structured dataset for evaluating the vulnerability detection capabilities of LLMs, this study leverages annotated test cases retrieved from SARD. Specifically, the experiment focuses on the 2024 CWE Top 25 Most Dangerous Software Weakness, as published annually by the CWE project on its official platform. According to the latest SARD technical report [47], as of October 2024, the dataset contains over 150 documented vulnerability types and more than 450,000 labeled test samples. A significant portion of these vulnerabilities appears in C/C++ programs, primarily due to the languages' low-level memory manipulation features, such as pointer arithmetic and manual memory management. These features substantially increase the risk of introducing critical security flaws. Accordingly, this study adopts C/C++ as the primary target language. However, the proposed methodology remains language-agnostic, as it leverages the reasoning and representation power of LLMs to capture universal patterns in code semantics and control flow. This generalization capacity allows the approach to be extended to other programming languages, including Java, Python, and HTML. Using SARD's structured query interface, all available test cases associated with the selected Top 25 CWE categories in C/C++ are retrieved. The detailed vulnerability categories and their sample statistics are listed in Table 1.  
Each SARD test case comprises several components, including source code files containing vulnerabilities, supplementary test scripts, and a Static Analysis Results Interchange Format (SARIF) property file. SARIF, defined by the Organization for the Advancement of Structured Information Standards (OASIS), is a standardized format that promotes the interoperability of static analysis tools and streamlines the sharing and interpretation of vulnerability data [48]. Each SARIF file contains structured metadata such as vulnerability type (_ruled_), state (_bad_ or _good_), programming language, upload timestamp, and the precise location of vulnerable code segments. An excerpt from a SARIF file corresponding to a CWE-119 test sample is shown in Figure 2.  
In the example provided, the _ruled_ indicates a _\"buffer overflow\"_ issue categorized under CWE-119. The _state_ field is marked as _\"bad\"_, signaling that the file contains a known vulnerability. The vulnerability location is pinpointed by the _locations_ field, which shows that the vulnerable statement appears at line 39 in the file _\"src/figured-29-windows.cpp\"_. Notably, a single SARIF file references multiple vulnerable files and multiple vulnerabilities within each file, making it a rich and informative source for detection evaluation. However, as summarized in Table 1, the number of available test cases varies substantially across CWE categories, ranging from fewer than 10 to several thousand per category. To ensure balanced evaluation while maintaining computational feasibility, the following sampling strategy is applied: for vulnerability types with fewer than \(T\) = 120 samples, all available test cases are retained. For those exceeding \(T\) samples, \(T\)+ representative cases are randomly sampled to preserve diversity while limiting evaluation cost. Additionally, SARD includes relatively fewer non-vulnerable (_\"good\"_ state) samples. To mitigate class imbalance, all good-state test cases are included to better balance the  
\begin{table}