Abstract: Public vulnerability databases such as CVE and NVD account for only 60% of security vulnerabilities present in open-source projects, and are known to suffer from inconsistent quality. Over the last two years, there has been considerable growth in the number of known vulnerabilities across projects available in various repositories such as NPM and Maven Central. Such an increasing risk calls for a mechanism to infer the presence of security threats in a timely manner. We propose novel hierarchical deep learning models for the identification of security-relevant commits from either the commit diff or the source code for the Java classes. By comparing the performance of our model against code2vec, a state-of-the-art model that learns from path-based representations of code, and a logistic regression baseline, we show that deep learning models show promising results in identifying security-related commits. We also conduct a comparative analysis of how various deep learning models learn across different input representations and the effect of regularization on the generalization of our models.

Introduction: The use of open-source software has been steadily increasing for some time now, with the number of Java packages in Maven Central doubling in 2018. However, BIBREF0 states that there has been an 88% growth in the number of vulnerabilities reported over the last two years. In order to develop secure software, it is essential to analyze and understand security vulnerabilities that occur in software systems and address them in a timely manner. While there exist several approaches in the literature for identifying and managing security vulnerabilities, BIBREF1 show that an effective vulnerability management approach must be code-centric. Rather than relying on metadata, efforts must be based on analyzing vulnerabilities and their fixes at the code level. Common Vulnerabilities and Exposures (CVE) is a list of publicly known cybersecurity vulnerabilities, each with an identification number. These entries are used in the National Vulnerability Database (NVD), the U.S. government repository of standards based vulnerability management data. The NVD suffers from poor coverage, as it contains only 10% of the open-source vulnerabilities that have received a CVE identifier BIBREF2. This could be due to the fact that a number of security vulnerabilities are discovered and fixed through informal communication between maintainers and their users in an issue tracker. To make things worse, these public databases are too slow to add vulnerabilities as they lag behind a private database such as Snyk's DB by an average of 92 days BIBREF0 All of the above pitfalls of public vulnerability management databases (such as NVD) call for a mechanism to automatically infer the presence of security threats in open-source projects, and their corresponding fixes, in a timely manner. We propose a novel approach using deep learning in order to identify commits in open-source repositories that are security-relevant. We build regularized hierarchical deep learning models that encode features first at the file level, and then aggregate these file-level representations to perform the final classification. We also show that code2vec, a model that learns from path-based representations of code and claimed by BIBREF3 to be suitable for a wide range of source code classification tasks, performs worse than our logistic regression baseline. In this study, we seek to answer the following research questions: [leftmargin=*] RQ1: Can we effectively identify security-relevant commits using only the commit diff? For this research question, we do not use any of the commit metadata such as the commit message or information about the author. We treat source code changes like unstructured text without using path-based representations from the abstract syntax tree. RQ2: Does extracting class-level features before and after the change instead of using only the commit diff improve the identification of security-relevant commits? For this research question, we test the hypothesis that the source code of the entire Java class contains more information than just the commit diff and could potentially improve the performance of our model. RQ3: Does exploiting path-based representations of Java source code before and after the change improve the identification of security-relevant commits? For this research question, we test whether code2vec, a state-of-the-art model that learns from path-based representations of code, performs better than our model that treats source code as unstructured text. RQ4: Is mining commits using regular expression matching of commit messages an effective means of data augmentation for improving the identification of security-relevant commits? Since labelling commits manually is an expensive task, it is not easy to build a dataset large enough to train deep learning models. For this research question, we explore if collecting coarse data samples using a high-precision approach is an effective way to augment the ground-truth dataset. The main contributions of this paper are: [leftmargin=*] Novel hierarchical deep learning models for the identification of security-relevant commits based on either the diff or the modified source code of the Java classes. A comparative analysis of how various deep learning models perform across different input representations and how various regularization techniques help with the generalization of our models. We envision that this work would ultimately allow for monitoring open-source repositories in real-time, in order to automatically detect security-relevant changes such as vulnerability fixes.

Question: How long is the dataset?
Answer: 2022