VulLibGen: Generating Names of Vulnerability-Affected Packages via a Large Language Model (2024)

Abstract: Security practitioners maintain vulnerability reports (e.g., GitHub Advisory) to help developers mitigate security risks. An important task for these databases is automatically extracting structured information mentioned in the report, e.g., the affected software packages, to accelerate the defense of the vulnerability ecosystem. However, it is challenging for existing work on affected package identification to achieve high precision. One reason is that all existing work focuses on relatively smaller models, thus they cannot harness the knowledge and semantic capabilities of large language models. To address this limitation, we propose VulLibGen 1, the first method to use LLM for affected package identification. In contrast to existing work, VulLibGen proposes the novel idea to directly generate the affected package. To improve the precision, VulLibGen employs supervised fine-tuning (SFT), retrieval augmented generation (RAG) and a local search algorithm. The local search algorithm is a novel postprocessing algorithm we introduce for reducing the hallucination of the generated packages. Our evaluation results show that VulLibGen has an average precision of 0.806 for identifying vulnerable packages in the four most popular ecosystems in GitHub Advisory (Java, JS, Python, Go) while the best average precision in previous work is 0.721. Additionally, VulLibGen has high value to security practice: we submitted 60 <vulnerability, affected package> pairs to GitHub Advisory (covers four ecosystems) and 34 of them have been accepted and merged.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9767–9780 August 11-16, 2024 ©2024 Association for Computational Linguistics
However, it is challenging for existing work on affected package identification to achieve high precision. One reason is that all existing work focuses on relatively smaller models, thus they cannot harness the knowledge and semantic capabilities of large language models.
To address this limitation, we propose VulLibGen 1, the first method to use LLM for affected package identification. In contrast to existing work, VulLibGen proposes the novel idea to directly generate the affected package. To improve the precision, VulLibGen employs supervised fine-tuning (SFT), retrieval augmented generation (RAG) and a local search algorithm. The local search algorithm is a novel postprocessing algorithm we introduce for reducing the hallucination of the generated packages. Our evaluation results show that VulLibGen has an average precision of 0.806 for identifying vulnerable packages in the four most popular ecosystems in GitHub Advisory (Java, JS, Python, Go) while the best average precision in previous work is 0.721. Additionally, VulLibGen has high value to security practice: we submitted 60 <vulnerability, affected package> pairs to GitHub Advisory (covers four ecosystems) and 34 of them have been accepted and merged.

heading: 1 Introduction
text: With the increasing usage of third-party software packages, their security vulnerabilities pose great
∗*Corresponding author 1Our data and code can be found at https://github.
com/q5438722/VulLibGen.
Github Advisory
Description Jenkins Mail Commander Plugin for Jenkins-ci Plugin 1.0.0 and earlier stores passwords unencrypted in job config.xml files on the Jenkins controller where they can be viewed by users with Extended Read permission, or access to the Jenkins controller file system.
GitHub Advisory Database / GitHub Reviewed / CVE-2020-2318
challenges to software and network systems. A recent study (Wang et al., 2020) shows that 84% third-party packages contain security vulnerabilities and 60% of them are high-risk ones. To mitigate the security risks, security practitioners maintain databases (e.g., NVD and GitHub Advisory) for each unique vulnerability (i.e., Common Weakness Enumeration or CVE). These databases provides metadata including the vulnerability description, affected packages, and affected versions. For example, Figure 1 shows the report of CVE-20202318. By providing the affected package names and versions, developers can be aware of the vulnerable packages and quickly apply patches/fixes; the affected packages also help with security knowledge management and task prioritization.
In recent years, automatically identifying package names and versions given the CVE report has become a popular task (Chen et al., 2020; Dong et al., 2019; Haryono et al., 2022; Lyu et al., 2023; Chen et al., 2023b) due to the high cost of maintaining such information. GitHub Advisory and other databases rely on the community to first submit the package names, the database maintainers then validate the submitted information before merging
9767
them to the database (Wu et al., 2024). Nevertheless, GitHub users sometimes submit incorrect information (Haryono et al., 2022; Dong et al., 2019), while the maintainers’ reviewing process takes a long time (GitHub, 2024b).
Several existing works have studied automatic affected package identification (Chen et al., 2020; Dong et al., 2019; Haryono et al., 2022; Lyu et al., 2023; Chen et al., 2023b), however, it is challenging for them to achieve high precision due to the limited model size they employ. Existing works typically rank/retrieve the package name from a list of pre-defined packages by computing the similarity between the vulnerability description and the package description. Since the time complexity of retrieval is linear to the size of the list (e.g., 435k Java packages), the cost of each model inference has to be kept quite low and only smaller models have been used, e.g., BERT and linear regression (Chen et al., 2020; Lyu et al., 2023; Haryono et al., 2022; Chen et al., 2023b).
To leverage the extensive knowledge and semantic capabilities of LLMs for this task, we propose a different strategy: we directly generate the affected package names using LLMs, rather than retrieving them. Our framework, VulLibGen, is the first framework to directly generate the mentioned package name given the vulnerability description. Since generation requires the LLM inference to be invoked only once, our approach can easily scale to larger language models such as Llama-13B and Vicuna-13B on a single GPU server.
Our initial investigation shows that the raw outputs of LLMs suffer from several types of errors including hallucination. Motivated by an empirical study of these errors, we propose the following techniques for improving VulLibGen. First, we leverage supervised fine-tuning (SFT), in-context learning, and retrieval-augmented generation (RAG) to enhance the domain knowledge of a LLM. Second, we propose a novel local search technique to post-process the raw output for reducing hallucination, i.e., ensuring that the generated package name actually exists. Our local search algorithm matches the raw output with the closest existing package name based on the edit distance. Since the sub-level package information (e.g., mailcommander in Figure 1) is often more directly mentioned than the top-level package information (e.g., org.jenkins-ci.plugins), our local search algorithm first matches the suffix before the
prefix. We evaluate VulLibGen on four vulnerability ecosystems: Java, JS, Python, and Go. Our evaluation attains three main findings. First, we observe that the precision of VulLibGen (0.806) significantly outperforms existing ranking approaches using smaller models (Chen et al., 2020; Haryono et al., 2022; Lyu et al., 2023; Chen et al., 2023b) (0.721) and the computational time costs are comparable. Second, our ablation studies show that SFT, RAG, and local search all help improve the precision of VulLibGen and SFT contributes to the most improvement. In particular, the fine-tuned open-source Vicuna-13B outperforms the unfinetuned commercial ChatGPT and GPT-4 models. Our local search algorithm can significantly reduce the hallucination in the original LLM output, and it is especially helpful for longer package names such as Java and Go. Third, VulLibGen provides high value to security practice: at the time of the writing, we have submitted 60 pairs of <vulnerability, affected package> to GitHub Advisory (25 Java, 14 JS, 11 Python, 10 Go) and 34 of them have been accepted and merged.

heading: 2 Existing Work on Vulnerable Package
text: Identification
This section summarizes existing work on affected package identification and analyze the scalability challenge. Formal Definition of Affected Package Identification. Given a security vulnerability (CVE) submitted to a software ecosystem (e.g., GitHub Advisory), the goal of affected package identification is to link the description q of the CVE to an existing software package name p (e.g., a Maven or PyPi package) that is affected by the CVE. An example of the linked package can be found in Figure 1, where the description mentions the affected package org.jenkins-ci.plugins:mailcommander. Smaller Models Have Lower Precision. Existing approaches on vulnerable package identification (Chen et al., 2020; Dong et al., 2019; Haryono et al., 2022; Lyu et al., 2023) all suffer from lower precision (Lyu et al., 2023; Chen et al., 2023b). Given the vulnerability q, existing works rank all packages p of the ecosystem by computing the similarity score between the descriptions of q and p. Due to the large number of packages (e.g., Maven has 435k packages), existing work cannot afford using large language models to compute the score.
All of them thus rely on smaller models, e.g., logistic regression (Gupta et al., 2021; Lyu et al., 2023) and BERT (Haryono et al., 2022; Chen et al., 2023b). Despite various methods introduced for improving the precision (Dong et al., 2019; Anwar et al., 2021), the precision remains low (Chen et al., 2023b). Existing Work’s Efforts on Scaling to Larger Models. To improve the precision, existing work leverages re-ranking with the BERT model (Chen et al., 2023b). More specifically, they first use TFIDF to rank all packages in the ecosystem (435k in Java and 506k in Python), then re-rank the top-512 packages using BERT. The re-ranking approach achieves a reasonable precision with a saved inference cost, but there remains a large room for improving the precision (Chen et al., 2023b).

heading: 3 Two Challenges with LLM Generation
text: In contrast to existing work, we propose the first work, VulLibGen, that leverages LLMs for affected package identification. Due to the scalability challenge of the retrieval approach, our approach directly generates rather than retrieves the affected package. VulLibGen thus only need to invoke the LLM inference once for each vulnerability q. Nevertheless, there exist two challenges with the generative approach. Challenge 1: Lack of Domain Knowledge. The first challenge is that there may exist a knowledge gap for the LLM to generate the correct package. This is because the description may not contain the full information about the affected package name. For example, CVE2020-2167 in Table 1 is about the Java package
com.openshift.jenkins.openshift-pipeline, but the the description does not mention the word "Jenkins". To predict the correct package name, the LLM has to rely on domain knowledge to complete this information. Existing work have used various methods to bridge the knowledge gap of LLMs, e.g., supervised fine-tuning (Prottasha et al., 2022; Church et al., 2021) and retrieval augmented generation (Lewis et al., 2020; Mao et al., 2020; Liu et al., 2021; Cai et al., 2022). Challenge 2: Generating Non-Existing Package Names. Following a previous study on Reddit2, the second challenge is that the LLM may generate library names that do not exist in the ecosystem. Existing work has adopted post-processing to reduce the non-existing package issue in code translation and program repair (Jin et al., 2023; Roziere et al., 2022). Following existing work, we can potentially leverage post-processing by matching the generated package with the closest existing package.
To understand whether post-processing is promising for solving Challenge 2 and to study how to design the post-processing algorithm, we conduct an empirical study on ChatGPT’s incorrect response, the study result can be seen in Table 1. The study uses 2,789 Java vulnerability descriptions collected in a recent work (Chen et al., 2023b). We divide all ChatGPT responses into four types: 1. the package is incorrect but it exists (13%, 23% of errors); 2. the package does not exist and is partially correct (34%, 58% of errors); 3. the package is completely incorrect (11%, 19% of errors). 4.
2https://www.reddit.com/r/ChatGPT/comments/ zneqyp/chatgpt_hallucinates_a_software_library_ that/
the package is correct (42% of all cases); From the study result, we draw the conclusion that post-processing by matching is a promising approach to solve Challenge 2. This is because the majority errors are Type 2 errors, while post-processing is the most effective in helping with Type 2 errors. For example, for CVE-2020-2167, ChatGPT generates org.jenkinsci.plugins:openshift-pipeline. While the suffix is correct, the prefix and suffix never co-occur in any existing package name. We can fix this case by matching the prefix to the closest co-occured one. By applying a naive edit-distance matching on the ChatGPT output, the precision is improved from 42% to 51%.

heading: 4 VulLibGen Framework
text: To address the two challenges in LLM generation, we employ the following techniques: first, we use supervised fine-tuning and in-context learning to enhance the domain knowledge in LLM; second, we further employ the retrieval-augmented framework (RAG) to enhance the knowledge when SFT is not easy; third, we design a local search technique which alleviates the non-existing package name problem. The VulLibGen framework can be found in Figure 2 3.

heading: 4.1 Supervised Fine-Tuning/In-Context Learning
text: To solve the first challenge (Section 3), we incorporate supervised fine-tuning (Prottasha et al., 2022; Church et al., 2021) and in-context learning (Dong et al., 2022; Olsson et al., 2022) in VulLibGen. For SFT, we use the full training data (Table 2); for ICL, we randomly sample 3 examples from the training data for each evaluation vulnerability. For both SFT and ICL, the input and output of the LLM follow the following format: Input: the same prompt as Figure 2 3, Output: "The affected package is [package name]". The hyper-parameters used for ICL and SFT are listed in Table 11 of Appendix.

heading: 4.2 Retrieval-Augmented Generation (RAG)
text: To further enhance the LLM’s domain knowledge especially when SFT is not easy (e.g., ChatGPT
3Our prompt in Figure 2 is: "Below is a [Programming Language] vulnerability description. Please identify the software name affected by it. Input: [DESCRIPTION]. The top k search results are: [L1][L2]· · · [Lk]. Please output the package name in the format "ecosystem:library name". ### Response: The affected packages:".
and GPT4), we employ retrieval-augmented generation (RAG) in VulLibGen. Retriever Setting. Given the description of a vulnerability, our retriever ranks existing package names in an ecosystem (Table 2) based on the similarity score between the vulnerability description and the package description. The descriptions of Java, JavaScript, Python, and Go packages are obtained from Maven 4, NPM 5, Pypi 6, and Go 7 documentations. For example, the description of the package org.jenkins-ci.plugins:mailcommander is “This plug-in provides function that read a mail subject as a CLI Command.”. Our retriever follows (Chen et al., 2023b)’s re-ranking strategy, i.e., first rank all packages (e.g., 435k in Java) using TF-IDF, then re-rank the top 512 packages using a BERTbase model fine-tuned on the same training data in Table 2.

heading: 4.3 Local Search
text: To solve the second challenge (Section 3), we incorporate post-processing in VulLibGen. Based on the empirical study results in Section 3, we design a local search technique to match the generation output with the closest package name from an existing package list (Algorithm 1 in Appendix 10.1).
Algorithm 1 employs the edit distance as the metric and respects the structure of the package name. Formally, a package name can be divided into two parts: its prefix and suffix (separated by a special character, e.g., ‘:’ in Java). The prefix (e.g., the artifact ID of Java packages) specifies the maintainer/group of this package, and the suffix (e.g., the group ID of Java packages) specifies the functionalities of this package. Specifically, Java, Go, and part of JS packages can be explicitly divided while Python and the rest of JS packages only specify their functionalities in their names. We denote the prefix of a package name as empty if it can not be divided.
Algorithm 1 first compares the generated suffix with all existing suffix names and matches the suffix to the closest one. After fixing the suffix, we can then obtain the list of prefixes that co-occur at least once with this suffix. We match the generated prefix with the closest prefix in this list. The reason that we opt to match the suffix first is twofold. First, our study shows that the vulnerability description
4https://mvnrepository.com 5https://www.npmjs.com 6https://pypi.org 7https://pkg.go.dev
more frequently mentions the suffix than the prefix: among all 2,789 vulnerabilities investigated in Section 3, their description mentions 12.4% of the tokens in the prefixes of the affected packages and 66.0% of the tokens in their suffixes of the affected packages; second, our study also shows that each suffix co-occurs with fewer unique prefixes than conversely. In all 435k Java packages, each prefix has 5.86 co-occurred suffixes while each suffix has only 1.13 co-occurred prefixes on average. As a result, it is easier to identify the prefix by first matching the suffix, and then matching the suffix with the co-occurred prefix list.

heading: 5 Evaluation

heading: 5.1 Evaluation Setup
text: Dataset. Among existing work on affected package identification (Chen et al., 2020; Haryono et al., 2022; Lyu et al., 2023; Chen et al., 2023b), two datasets are frequently used: VeraCode (Chen et al., 2020) and VulLib (Chen et al., 2023b). In this work, we choose to use VulLib instead of VeraCode because it is of better quality. The VulLib dataset contains 2,789 Java vulnerabilities collected from GitHub Advisory. Each package name in VulLib is manually verified by security experts from GitHub Advisory (GitHub, 2024a). In contrast, VeraCode is not verified thus it is prone to errors 8. Since VulLib only focuses on Java, we further extend it to JS, Python and Go by collecting the data from GitHub Advisory following a similar workflow as VulLib.
The statistics of our dataset are listed in Table 2. In total, our dataset includes 2,789 Java, 3,193 JS,
8VeraCode does not use explicit package names ecosystems. For example, the affected package of CVE-2014-2059 is “org.jenkins-ci.main:jenkins-core” while VeraCode labels it as three packages: “jenkins”, “openshift-origin-cartridgejenkins”, and “jenkins-plugin-openshift”
2,237 Python, and 1,351 Go vulnerabilities, respectively. To the best of our knowledge, this is the first dataset for identifying vulnerable packages with various programming languages. For each PL, we split the train/validation/test data with the 3:1:1 ratio. The split is in chronological order to simulate a more realistic scenario and to prevent lookahead bias (Kenton, 2024). Comparative Methods. To evaluate the effectiveness of VulLibGen, we contrast it with four existing ranking approaches, FastXML (Chen et al., 2020), LightXML (Haryono et al., 2022), Chronos (Lyu et al., 2023), and VulLibMiner (Chen et al., 2023b) for comparison. Recent studies (Lyu et al., 2023; Chen et al., 2023b) show that they outperform other approaches, such as Bonsai (Khandagale et al., 2020) and ExtremeText (Wydmuch et al., 2018). Models in VulLibGen. The models we evaluate for the VulLibGen framework include both commercial LLMs, e.g., ChatGPT (gpt-3.5-turbo) and GPT4 (gpt-4-1106-preview), and open-source LLMs, e.g., LLaMa (Touvron et al., 2023) and Vicuna (Chiang et al., 2023).
We assess open-source LLMs in two scenarios: few-shot in-context learning using 3 examples ran-
domly sampled from the training data and supervised fine-tuning using the full training data. For the open-source LLMs, we use ICL/SFT + RAG + local search, whereas for commercial LLMs, we use RAG + local search only. Evaluation Environments Our evaluations are conducted on the system of Ubuntu 18.04. We use one Intel(R) Xeon(R) Gold 6248R@3.00GHz CPU, which contains 64 cores and 512GB memory. We use 8 Tesla A100 PCIe GPUs with 40GB memory for model training and inference. In total, our experiments constitute 200 GPU days (32 groups in RQ1 + 68 groups in RQ2, and each group costs 0.25 GPU days across 8 GPUs). Metrics. Following previous work (Chen et al., 2023b, 2020), we use three metrics for evaluating VulLibGen and baselines: Precision@k, Recall@k, and F1@k (k = 1, 2, 3).

heading: 5.2 Evaluation of VulLibGen
text: In this subsection, we evaluate the effectiveness of VulLibGen. We seek to answer the following research question: How does VulLibGen compare to existing work on identifying vulnerable packages? Overall Performance: Existing Work vs. VulLibGen. In Table 3, Table 4, Table 5, and Table 6 (Appendix), we compare the performance of VulLibGen and baselines. From these tables we can observe that for all programming languages, VulLibGen achieves substantially higher precision compared to existing work. As a result, by leveraging LLMs, VulLibGen can effectively generate the
names of affected packages with high precision. Overall, VulLibGen using supervised fine-tuning on the Vicuna-13B model has the best performance. Fine-tuning Vicuna-13B even outperforms the larger ChatGPT and GPT4 models on all datasets besides Java. As a result, the knowledge gap of LLMs can be effectively bridged by leveraging supervised fine-tuning. In Table 12 of Appendix, we further report statistical significance tests (Kim, 2015) between the overall bestperforming generative approach (i.e., VulLibGen using Vicuna-13B SFT) and the best-performing existing work (i.e., VulLibMiner (Chen et al., 2023b)). The p-values in all tests are smaller than 1e-5. When Is VulLibGen More Advantageous? From Table 3 observe that the gap between VulLibGen Vicuna-13B SFT and the best-performing existing approach for each programming language are 0.041, 0.031, 0.11, and 0.157. By comparing with the data statistics in Table 2, we can see that this gap is highly correlated with #Unique packages in the dataset and #Total packages in the ecosystem. In general, VulLibGen is less advantageous when
the output package name is longer and has a larger token space.
Efficiency of Existing Work vs. VulLibGen. In Figure 3, we visualize the actual computational cost and precision of each method in Table 3. We further mark the upper bound of the ranking-based approach using LLMs for comparison. The Precision@1 is upper-bounded by the recall@512 of TF-IDF, i.e., the best possible Precision@1; while the time cost is upper-bounded by the time cost of invoking the 13B model 512 times (10 mins).
We can observe the VulLibGen achieves a sweet spot in the effectiveness and efficiency trade-off. When compared with existing work, VulLibGen achieves a better Precision@1 while the time cost is comparable to the best-performed existing work (Chen et al., 2023b). When compared with the upper bound, VulLibGen achieves a slightly lower Precision@1 while consuming less than 1/100 time and computation resources.

heading: 5.3 Ablation Studies on VulLibGen
text: In this subsection, we conduct ablation studies on the three components of VulLibGen: supervised fine-tuning, RAG, and local search. SFT’s Improvement. By comparing the results of in-context learning vs supervised fine-tuning in Table 3, we can see that SFT outperforms ICL by a larger margin. This result indicates that for the 7B and 13B models, supervised fine-tuning on the full training data is essential in bridging the models’ knowledge gap. RAG’s Overall Improvement: Table 7 shows the improvement of our RAG technique in Precision@1. Specifically, it improves the Precision@1 by 9.3%, 1.8%, 8.9%, and 15.7% on each programming language, respectively. These improvements indicate that our RAG technique is effective in helping generate the names of vulnerable packages. We further report paired t-test results for Table 7 in Table 13 of Appendix.
Table 7 also indicates that RAG’s improvement in commercial LLMs is higher than that of opensource LLMs. Especially in Go vulnerabilities, our RAG technique improves the Precision@1 by 57.6% and 31.0% on ChatGPT and GPT4. The main reason is that both ChatGPT and GPT4 do not have sufficient domain knowledge about Go packages as they are relatively newer than packages of other programming languages (Hall, 2023). RAG Improvement vs. k/Retrieval Algorithm Choice. We evaluate whether k and the choice of retrieval algorithm affect the end-to-end effectiveness of VulLibGen. Specifically, we focus on Java vulnerabilities (as Java package names are the most difficult to generate). The result can be found in Table 8.
For k, we conduct an Analysis of Variance (ANOVA) (St et al., 1989) among the Precision@1
of six representative numbers of RAG packages (ranging from 1 to 20). Although k = 20 has a slightly higher precision than k = 1 for both TFIDF and BERT, this difference is not significant. In fact, the paired t-test results show that there is no significant difference among the Precision@1 of different k values (p = 0.814 for TF-IDF and p = 0.985 for BERT).
As for the retrieval algorithm, we observe that Precision@1 with TF-IDF results is quite similar to that of non-RAG inputs, and the Precision@1 with BERT results is substantially higher than that of non-RAG/TF-IDF results. As a result, it is essential to use BERT-retrieved results in RAG. Local Search’s Improvement. Table 9 shows the end-to-end improvement in Precision@1 of VulLibGen before and after local search. Our local search technique improves the Precision@1 by 3.43%, 1.02%, 1.57%, and 6.20% on each programming language. We further report paired t-test results for Table 9 in Table 14 of Appendix.
We make the following observations. First, local search is more effective on commercial LLMs (an average improvement of 4.58%) than fine-tuned open-source LLMs (an average improvement of 2.29%). Since commercial LLMs are not finetuned, local search plays an important role in improving the effectiveness of generation. Second, local search is more effective on Java and Go than JS and Python. The reason is that since Java and Go packages are longer (8-14 tokens), LLMs are more prone to generating partially correct, non-existing outputs (i.e., Type 2 error in Table 1). Local search can effectively reduce this type of error.

heading: 5.4 Evaluating VulLibGen Performance in Real World Setting
text: To examine VulLibGen’s performance in the real-
world setting, for each programming language, we randomly sample and report a subset of <vulnerability, affected package> pairs that are not listed in GitHub Advisory (Java: 25, JS: 14, Python: 11, Go: 10). We use VulLibGen to generate the package names and submit the generated names (VulLibGen with Vicuna-13B) to GitHub Advisory.
At the time of the writing, the results are summarized below. Java: 21 of them have been accepted and merged into GitHub Advisory. Among the remaining 4 packages, 2 of them are considered non-vulnerabilities, and 2 of them are considered incorrect affected packages. JS, Python, and Go: 13 of them have been accepted and merged into GitHub Advisory (2 JS, 8 Python, 3 Go). The details of these packages are listed in Table 15 of Appendix.
This result highlights the real-world performance of VulLibGen in automatically identifying affected package names.

heading: 6 Related Work
text: Vulnerable Package/Version Identification. Numerous existing works have proposed methods to improve the precision of affected package identification. Multiple existing works model this prob-
lem as a named entity recognition (NER) problem, i.e., extracting the subset of description about the package (Dong et al., 2019; Anwar et al., 2021; Jo et al., 2022; Kuehn et al., 2021; Yang et al., 2021) or version (Dong et al., 2019; Zhan et al., 2021; Zhang et al., 2019; Backes et al., 2016; Zhang et al., 2018; Tang et al., 2022; Gorla et al., 2014; Wu et al., 2023). The NER approach works well for the software version identification since many version numbers are already in the description (Dong et al., 2019). On the other hand, the package names are often only partially mentioned (e.g., CVE-20202167 in Table 1), therefore the NER approaches are less effective (Lyu et al., 2023). Another branch of work models the package identification problem as extreme multi-label learning (XML) where each package is a class (Chen et al., 2020; Haryono et al., 2022; Lyu et al., 2023). However, these methods are limited to less than 3k classes (the labels in their dataset). Finally, (Chen et al., 2023b) leverages the re-ranking approach using BERT; however, there still exists a gap between their method’s precision and the best possible performance (Table 3).
Retrieval vs Generation. Existing work has investigated scenarios of replacing the retrieval with generation. For example, Yu et al. (Yu et al., 2023) leverages LLM to generate the context documents for question answering, rather than retrieving them from a text corpus. Their experiment shows that the generative approach has a comparable performance to the retrieval approach on the QA task. However, since our task requires us to generate the exact package name, their conclusion is not directly transferrable to our task.
Retrieval-Augmented Generation Retrievalaugmented generation (RAG) (Lewis et al., 2020; Mao et al., 2020; Liu et al., 2021; Cai et al., 2022) is a widely used technique and has shown its effectiveness in various generation tasks, e.g., code generation or question answering. Specifically, RAG enhances the performance of a generative model by incorporating knowledge from a database so that LLMs can extract and comprehend correct domain knowledge from the RAG inputs.
Reducing Hallucination. In Section 3, we show that ChatGPT’s raw output package name may not exist. This phenomenon is similar to hallucination (Ji et al., 2023; Tonmoy et al., 2024), which occurs in various LLM-related tasks. Among hallucination reduction approaches, postprocessing (Madaan et al., 2023; Kang et al., 2023)
is a widely used one. For example, in code-related tasks, existing work (Jin et al., 2023; Chen et al., 2023a; Zhang et al., 2023; Huynh Nguyen et al., 2022) adopts post-processing techniques to reduce/rerank programs generated by LLMs, e.g., using deep-learning models, test cases, or compilers to determine whether a generated program is correct and remove incorrect programs. However, such techniques cannot be directly adopted in our task because validating the generated names of affected packages is relatively difficult. It requires a Proof-of-Chain (PoC) (Mosakheil, 2018), which is often unavailable due to security concerns. Therefore, we design our local search algorithm focusing on Type 2 errors in Table 1.

heading: 7 Conclusion
text: In this paper, we have proposed VulLibGen, the first framework for identifying vulnerable packages using LLM generation. VulLibGen conducts retrieval-augmented generation, supervised finetuning, and a local search technique to improve the generation. VulLibGen is highly effective, achieving an average precision of 0.806 while the best SOTA approaches achieve only 0.721. VulLibGen has shown high value to security practice. We have submitted 60 pairs of <vulnerability, affected package> to GitHub advisory and 34 of them have been accepted and merged.

heading: 8 Limitation
text: Our work has several limitations, which we plan to address in our future work: Challenges in Generating Long and Complex Package Names. As discussed in Section 5.2, the effectiveness of VulLibGen depends on the token length and the number of unique packages. Table 3 shows Java is more challenging than others while having the highest token length and unique packages (Table 2). To improve the generation of complicated languages such as Java, we plan to further enhance the knowledge of LLM using techniques such as constrained decoding (Post and Vilar, 2018). We leave this as our future work. In particular, it may pose further challenges to generate packages that are exceptionally long. To understand the distribution of token lengths, we report the quantile statistics of the token lengths in Table 10 of Appendix. Table 10 shows that the majority of the package names are shorter than 20 tokens, therefore, exceptionally long package names are
very rare. Challenges in Generating Package Names with Limited Ecosystem Knowledge. Though VulLibGen has demonstrated its effectiveness in four widely-used programming languages, some other programming languages, e.g., C/C++, do not have a commonly used ecosystem that maintains all its packages. Thus, it is difficult to generate/retrieve the affected packages of C/C++ vulnerabilities as we do not have specific ranges during the RAG step of VulLibGen. Exploring how to generate RAG results without a commonly used ecosystem (e.g., Maven or Pypi) or collecting other useful information for RAG is the future work of this paper.

heading: 9 Ethical Consideration
text: License/Copyright. VulLibGen utilizes opensource data from GitHub Advisory, along with four third-party package ecosystems. We refer users to the original licenses accompanying the resources of these data. Intended Use. VulLibGen is designed as an automatic tool to assist maintainers of vulnerability databases, e.g., GitHub Advisory. Specifically, VulLibGen helps generate the names of affected packages to complement the missing data of these databases. The usage of VulLibGen is also illustrated in Section 4 and our intended usage of VulLibGen is consistent with that of GitHub Advisory (GitHub, 2024a). Potential Misuse. Similar to existing open-source LLMs, one potential misuse of VulLibGen is generating harmful content. Considering that we use open-source vulnerability data for LLM finetuning, the LLM might view harmful content during this step. To avoid harmful content, we use only reviewed vulnerability data in GitHub Advisory, so such misuse will unlikely happen. Overall, the scientific and social benefits of the research arguably outweigh the small risk of their misuse.

heading: 10 Appendix

heading: 10.1 Our Local Search Algorithm
text: Algorithm 1: Local Search Input :rawName, a generated package name Output :vulnNames, names of affected packages.
// Pre-process on name list 1 nameDict, suffixes =← {}, ∅; 2 for name ∈ nameList do 3 prefix, suffix← name.split(“/:”); 4 nameDict[suffix].add(prefix);
// Search the closest prefix/suffix 5 prefix, suffix← rawName.split(“/:”); 6 edit.weight← (Winsert,Wdelete,Wreplace); 7 suffix′ ← argmin
s∈suffixes edit(suffix, s);
8 prefixes← nameDict[suffix′]; 9 if prefixes.isEmpty() then
10 return {suffix′};

heading: 11 else
text: 12 prefix′ ← argmin
p∈prefixes edit(prefix, p);

heading: 13 return {prefix′} : {suffix′};
text: The pseudocode of our local search algorithm is shown in Algorithm 1. The input of this algorithm includes one package name generated by LLM together with the name list of existing libraries under the same ecosystem. The output of this algorithm is the name of one existing package that is the closest to the generated package name.
In Lines 1-4, we pre-process the name list of candidate packages. Now that we divide a package name into its prefix and suffix, we first construct the dictionary nameDict that maps a suffix into its corresponding prefix.
In Lines 5-13, we search for the closest package name of the input package name, rawName. In Line 7, we use its suffix, suffix to find its closest and existing suffix, suffix′. Then in Lines 8-13, we first determine whether it contains a corresponding prefix. If it has no prefix (e.g., a Python package), we directly return the closest suffix. Otherwise, we find its closest prefix, prefix′, from all prefixs that correspond to suffix′. Additionally, in Line 6, we manually set the weight used in calculating the edit distances because LLMs change the package names in terms of tokens instead of characters. Thus, the weight of inserting one character should be smaller than that of deleting and
replacing one, and we set the empirical weights as follows, Winsert = 1,Wdelete = 4,Wreplace = 4.

heading: 10.2 Statistical Significance Test between VulLibGen and Baselines
text: VulLibGen vs the Best Baseline. Table 12 shows the P-values between the best Precision@1 of the generative approach (i.e., VulLibGen using Vicuna13B SFT) and the best Precision@1 of existing work (i.e., VulLibMiner (Chen et al., 2023b)), i.e., the P-values for Table 3. From this table, we can observe that the P-values of all tests are smaller than 1e-5, indicating the significant improvement of VulLibGen’s effectiveness. Significance of Using RAG. Table 13 shows the Pvalues between VulLibGen’s Precision@1 before and after RAG in Table 7. We highlight the Pvalues that are larger than 0.05 in Table 13. We can observe that RAG significantly improves the effectiveness of VulLibGen in most combinations of LLMs and programming languages. Significance of Using Local Search. Table 14 shows the P-values between VulLibGen’s Precision@1 before and after local search in Table 9. We highlight the P-values that are larger than 0.05 in Table 14. We can observe that local search’s improvement is less significant in JS and Python and more significant in Java and Go.

heading: Future Work
text: Our work has several limitations, which we plan to address in our future work: Challenges in Generating Long and Complex Package Names. As discussed in Section 5. 2, the effectiveness of VulLibGen depends on the token length and the number of unique packages. Table 3 shows Java is more challenging than others while having the highest token length and unique packages (Table 2). To improve the generation of complicated languages such as Java, we plan to further enhance the knowledge of LLM using techniques such as constrained decoding (Post and Vilar, 2018). We leave this as our future work. In particular, it may pose further challenges to generate packages that are exceptionally long. To understand the distribution of token lengths, we report the quantile statistics of the token lengths in Table 10 of Appendix. Table 10 shows that the majority of the package names are shorter than 20 tokens, therefore, exceptionally long package names are
very rare. Challenges in Generating Package Names with Limited Ecosystem Knowledge. Though VulLibGen has demonstrated its effectiveness in four widely-used programming languages, some other programming languages, e. g. , C/C++, do not have a commonly used ecosystem that maintains all its packages. Thus, it is difficult to generate/retrieve the affected packages of C/C++ vulnerabilities as we do not have specific ranges during the RAG step of VulLibGen. Exploring how to generate RAG results without a commonly used ecosystem (e. g. , Maven or Pypi) or collecting other useful information for RAG is the future work of this paper