Patent Description:
Modern software development relies upon open source libraries to allow the rapid prototyping and development of new code. As open source libraries are created and revised, bug fixes and vulnerabilities are not always publicly published. Relevant forums and developer notes often contain pertinent information regarding the stability of open source libraries, but it is difficult to use this information due to the large volume of data.

Embodiments of the disclosure may be better understood by referencing the accompanying drawings.

The description that follows includes example systems, methods, techniques, and program flows that embody embodiments of the disclosure. However, it is understood that this disclosure may be practiced without these specific details. For instance, this disclosure refers to K-nearest neighbors in illustrative examples. Aspects of this disclosure can be also applied to other machine learning algorithms. In other instances, well-known instruction instances, protocols, structures and techniques have not been shown in detail in order not to obfuscate the description.

Advantageous features are described in the dependent claims.

To aid in software development, software developers typically adopt issue-tracking and source control management systems. These tools are popular for open source projects and modern software development. Developers work on reported issues in these systems, then commit corresponding code changes to a source control management system. Bug fixes and new features are frequently merged into a central repository, which is then automatically built, tested, and prepared for a release to production, as part of the DevOps practices of continuous integration and continuous delivery ("CI/CD"). However, numerous security-related issues and vulnerabilities are identified and patched without public disclosure due to the focus on fast release cycles and the lack of manpower and expertise. An estimated <NUM>% of vulnerabilities in open source libraries are not disclosed publicly. In addition, fixes to these undisclosed vulnerabilities are sometimes not reported ("silent fixes").

A system to create a stacked classifier model combination or classifier ensemble has been designed for identification of undisclosed flaws (e.g., bugs and vulnerabilities) in software components (e.g., open source libraries) on a large-scale. This classifier ensemble is capable of at least a <NUM>% improvement in precision. The system uses a K-folding cross validation algorithm to partition or split a sample dataset and then train and test a set of N classifiers with the split dataset. At each test iteration, trained models of the set of classifiers generate probabilities/predictions that a sample has a flaw, resulting in a set of N probabilities or predictions for each sample in the test data. With a sample size of S, the system passes the S sets of N predictions to a logistic regressor along with "ground truth" (represented with labels) for the sample dataset to train a logistic regression model. The trained classifiers and the logistic regression model are stored as the classifier ensemble. The classifier ensemble can then be used to generate a probability that a software component or project has flaw. Output from the classifier ensemble can be inserted into a database of identified flaws, which can then be used to guide developers in open source library selection in rapid software development. For instance, an agent can be programmed to monitor or analyze a software project and access the database to identify vulnerabilities in the software project.

<FIG> depicts the training and deployment of an automatic vulnerability identification system for open source libraries. A list of open source data sources <NUM> is shared with a data scraper <NUM>. The data scraper <NUM> scrapes data sources from a number of open source data repositories <NUM> via a network <NUM> to accumulate commit messages and bug reports (hereafter "CBR data") <NUM><NUM>. A natural language processor (hereafter "NLP") <NUM> processes the CBR data <NUM>. The NLP <NUM> generates vulnerability vectors <NUM> based on the CBR data <NUM> and provides them to a vulnerability classifier ensemble generator ("generator") <NUM>. The generator <NUM> uses the vulnerability vectors <NUM> and "ground truth" labeled vulnerability data <NUM> to generate a vulnerability classifier ensemble <NUM>. The vulnerability classifier ensemble1 <NUM> is deployed to identify vulnerabilities in software components which may be in a software project, open source project, and/or open source library. The ensemble <NUM> generates a probability value that a software component has a flaw and this probability value in association with identification of the software component is inserted into a vulnerability database <NUM>.

Referring back to the data scraper <NUM> prior to generation of the classifier ensemble <NUM>, the data scraper <NUM> collects the CBR data <NUM> from the open source data repositories <NUM> over the network <NUM> based on the list of open source libraries <NUM>. The data scraper <NUM> collects CBR data <NUM> by collecting "software artifacts" based on relevant text. A software artifact may be a commit message, comment in a forum, bug report, issue, text extracted from any one of these, etc. The scraper <NUM> determines relevant text based on known organizational structures for data in the open source data repositories <NUM>. For example, a forum may be known to label topics by open source library name and organize comments with a particular spacing. The data scraper <NUM> can extract the text from each comment of each relevant topic. In general, the data scraper <NUM> is programmed to gather data from interfaces oriented toward human accessibility. The data scraper <NUM> can also be configured to gather data from data repositories with various formats.

Example data sources of the CBR data <NUM> include forums, open source library repositories, and blog posts, which are elements of the open source data repositories <NUM>. The CBR data <NUM> may contain a combination of natural language and code-specific language. The NLP <NUM> is designed to process this language combination of the CBR data <NUM>. The code-specific language may include universal resource locators ("URLs"), types of software vulnerabilities, abbreviations (e.g., XSS), acronyms, and other software development or programming specific terminology. Commit messages and bug reports, titles, descriptions, comments, numbers of comments, attachment numbers, labels, dates of creation, and last edited dates may be extracted by the data scraper <NUM> as the CBR data <NUM>, as this information is generally available in most bug reports and is relevant to vulnerability detection. Among the selected features, titles, descriptions, comments, and labels are text features that contain semantic information which can be processed by the NLP <NUM>. The numbers of comments and attachments are numeric features which may reflect the attention and resolution a bug report received. Similarly, the difference between the created and last edited date reflects the resolution time, allowing vulnerabilities to be correlated with corresponding versions of open source libraries.

The NLP <NUM> takes the CBR data <NUM> as input and performs language processing operations to generate the vulnerability vectors <NUM>. The vulnerability vectors <NUM> comprise frequency counts of tokens (words or phrases) from the CBR data <NUM> which indicate the presence of a vulnerability. The NLP <NUM> generates the vulnerability vectors <NUM> by recording words or phrases in the CBR data <NUM> indicative of vulnerabilities. The NLP <NUM> may have been manually trained by embedding words known to be indicative of vulnerabilities into a vector space to generate relationships between different words. Words or phrases in the CBR data <NUM> are then evaluated based on their proximity or association with the words embedded in the vector space. The functionality of the NLP <NUM> is explored in greater detail by the <FIG> description below.

The NLP <NUM> provides the vulnerability vectors <NUM> as input to the generator <NUM>. The generator <NUM> uses the vulnerability vectors <NUM> and the vulnerability data <NUM> to generate a classifier ensemble based on training a number N of classifiers with a K-splitting algorithm and using logistic regression. Using N classifiers improves performance for imbalanced datasets.

The generator <NUM> trains and tests the N classifiers to accurately identify vulnerabilities by classifying the vulnerability vectors <NUM> and validating identifications with the vulnerability data <NUM>. This training and testing generates for each vulnerability vector a prediction from each trained classifier. Assuming S of the vulnerability vectors <NUM> were in the testing data, the generator <NUM> generates S sets of N predictions. The generator <NUM> then uses the predictions made by the N trained classifiers with the vulnerability data <NUM> to train a logistic regression model. The generator <NUM> forms the ensemble <NUM> by combining the trained N classifiers with the trained logistic regression model. To combine, the generator <NUM> can create program code that linearly combines output probabilities for a software component from the trained N classifiers to the logistic regression model and program code to fit the linearly combined probabilities to the probability distribution of the logistic regression model.

The ensemble <NUM> can be used in coordination with the NLP <NUM>, as well as the data scraper <NUM> The NLP <NUM> can be used to generate vectors from information about a code unit or a software project. The data scraper <NUM> can operate in conjunction with the NLP <NUM> and the ensemble <NUM> to periodically or on demand scrape one or more target repositories. The vulnerability database <NUM> is updated with results from the ensemble <NUM>. The vulnerability database <NUM> can be accessed prior to employing the ensemble <NUM> to avoid repeated processing of the same open source components. Embodiments may further enhance the quality of the database <NUM> with expert evaluation of the components identified as having or being vulnerabilities by the ensemble <NUM>. The open source components (e.g., libraries or projects) can be stored in association with the vulnerability prediction or probability value. When presented for evaluation, the database <NUM> can present the components identified as vulnerabilities with their associated confidence values for sorting or filtering to allow the experts to evaluate more efficiently.

The client systems 126a-c submit requests for vulnerability identification by sharing the client data <NUM> with the client request manager <NUM>. The client manager <NUM> acts as an interface between the client systems 126a-c and both the ensemble <NUM> and the vulnerability database <NUM>. The client request manager <NUM> receives the client data <NUM> from the client systems 126a-c. The received client data <NUM> may comprise operational parameters and a list of open source libraries with specified versions. The operational parameters may include a Boolean denoting whether a client system has requested a new search of the open source libraries or a probability threshold for vulnerability identification. The client request manager <NUM> may provide received client data <NUM> to the AVIS <NUM> and the vulnerability database <NUM> for vulnerability identification as appropriate.

The client request manager <NUM> accesses the vulnerability database <NUM> to determine whether appropriate vulnerability information is available for the open source libraries in the client data <NUM>. In some cases, the vulnerability database <NUM> may return query results that indicate appropriate information is available for some open source libraries in the client data <NUM> but not others. In such cases, the ensemble <NUM> can process the client data <NUM> based on those open source libraries for which the vulnerability database <NUM> does not have information. The new data is stored or merged with existing data in the vulnerability database <NUM>. The vulnerability database <NUM> then returns appropriate information for each of the open source libraries in the client data <NUM> to the client request manager <NUM>.

For example, the client system 126a may request vulnerability data for an open source library titled "openL. The client system 126a sends the request to the client request manager <NUM>, which accesses the vulnerability database <NUM> accordingly. The vulnerability database <NUM> returns results that indicate whether or not any known vulnerabilities for "openLibraryV1. <NUM>" are stored therein. The vulnerability database <NUM> may report that the open source library "openLibraryV1. <NUM>" has vulnerabilities. If the vulnerability database <NUM> does not contain vulnerability data for "openLibraryV1. <NUM>", this result can trigger the client request manager <NUM> to invoke the ensemble <NUM> to automatically identify vulnerabilities for openLibraryV1. The identified vulnerabilities are then stored in the vulnerability database <NUM> and communicated to the client request manager <NUM>, which returns the relevant vulnerability data to the client system 126a. In some cases, the client system 126a may directly request a new search be performed for "openLibraryV1. <NUM>" to ensure that vulnerabilities are detected. This request may be made based on operational parameters specified by the client system 126a in the client data <NUM>.

<FIG> depicts example client systems interacting with a vulnerability database and an automatic vulnerability identification system through a client request manager. A client request manager <NUM> transmits client data 224a and 224b between client systems 222a and 222b, a vulnerability database <NUM>, and an automatic vulnerability identification system (AVIS) <NUM>. The vulnerability database <NUM> comprises open source library data 206a-d. The AVIS <NUM> comprises an AVIS controller <NUM>, a data scraper <NUM> with access to open source data repositories <NUM> over a network <NUM>, an NLP <NUM>, and a vulnerability classifier ensemble <NUM>. The AVIS <NUM> generates client vulnerability data 226a from the client data 224a.

For the example system, the client request manager <NUM> and the vulnerability database <NUM> are shown to exist on the same device. This allows the client request manager <NUM> direct access to the open source library data 206a-d. In some cases, the client request manager <NUM> and the vulnerability database <NUM> exist on different devices and may interact over a wired or wireless connection. In some cases, the client request manager <NUM>, the vulnerability database <NUM>, and the AVIS <NUM> may exist on the same device.

The vulnerability database <NUM> contains the open source library data 206a-d for four different open source libraries. The open source library data 206a comprises a table of version histories for the open source library with associated vulnerability data. The vulnerability data may comprise a number of vulnerabilities for each version of the open source library and probabilities associated with each identified vulnerability. In some cases, each of the number of vulnerabilities may be identified and alternative open source libraries may be added to the vulnerability data manually or automatically from predetermined vulnerability data repositories. Similar information is contained in the open source library data 206b-d.

The client request manager <NUM> receives the client data 224a and 224b from the client systems 222a and 222b. The client request manager <NUM> determines whether relevant information is present on the vulnerability database <NUM>. For example, the client data 224b from the client system 222b may pertain to the open source library <NUM>, which has relevant information in the vulnerability database <NUM> stored in the open source library data 206d. Thus, the client request manager <NUM> may send the open source library data 206d to the client system 222b as the client vulnerability data 226b in response to the request made by the client system 222b.

If the vulnerability database <NUM> does not have relevant information based on the client data 224a, the client request manager <NUM> invokes the AVIS <NUM>. The client request manager <NUM> shares the client data 224a with the AVIS controller <NUM>. The AVIS controller <NUM> initiates a series of operations to identify vulnerabilities using the data scraper <NUM>, the NLP <NUM>, and the vulnerability classifier ensemble <NUM>. Once identified, vulnerability data is returned to the client request manager <NUM> as the client vulnerability data 226a by the AVIS controller <NUM>. The client request manager <NUM> can store the client vulnerability data 226a as open source library data on the vulnerability database <NUM> and returns the client vulnerability data 226a to the client system 222a.

The data scraper <NUM> takes as input the client data 224a from the AVIS controller <NUM>. The data scraper <NUM> collects commit messages and bug reports pertaining to open source libraries listed in the client data 224a by scraping data from open source data repositories <NUM> over the network <NUM>. The data scraper <NUM> outputs the aggregate information as CBR data to the NLP <NUM>.

The NLP <NUM> analyzes the CBR data from the data scraper <NUM> to generate a set of vulnerability vectors. The vulnerability vectors comprise information indicative of vulnerabilities from the CBR data. The generated vulnerability vectors are sent to the vulnerability identifier <NUM> for processing. The functionality of the NLP <NUM> and the vulnerability vectors is explored in greater detail by the <FIG> description.

The vulnerability classifier ensemble <NUM> generates a vulnerability probability value for each of the vulnerability vectors. Any identified vulnerabilities are returned to the AVIS controller <NUM>, which returns the identified vulnerabilities as the client vulnerability data 226a to the client request manager <NUM>. The generation of the vulnerability identifier <NUM> is explored in greater detail by the <FIG> description.

The client request manager <NUM> stores the received client vulnerability data 226a as open source library data or merges it with existing open source library data 206a-d. Any open source library data relevant to the client data 224a is returned to the client system 222a as the client vulnerability data 226a.

In some cases, the vulnerability database <NUM> and the AVIS <NUM> may communicate to identify and store vulnerabilities in open source libraries without being prompted by a client system. The vulnerability database <NUM> may also request the identification of vulnerabilities for open source libraries based on the passing of a predetermined amount of time since the last update to the vulnerabilities of the open source library.

<FIG> is a flowchart of example operations for generating a classifier ensemble for flaw identification. The description of <FIG> refers to a generator as performing the example operations for consistency with <FIG>. The generator has access to program code that implements N classifiers. The number and type of classifiers can vary.

At block <NUM>, a generator partitions software development data with M samples into K datasets or folds. The M samples may be M software artifacts or M messages, commits, etc. The folding parameter K may be chosen based on a number of open source components indicated in the software development data. The generator may invoke a defined function/method for K-fold cross validation and pass as parameters the value for K and a reference(s) to the dataset including the M samples. By partitioning the software development data, classifiers can be trained and tested without overfitting.

At block <NUM>, the generator begins an operative loop from <NUM> to K (or <NUM> to K-<NUM>) according to the splitting/folding algorithm being used. The loop comprises the operations of blocks <NUM>, <NUM>, <NUM>, and <NUM>.

At block <NUM>, the generator begins a nested loop N=<NUM> to M, for the M classifiers. The loop comprises the operations of blocks <NUM> and <NUM>. The classifier of a current iteration is referred to as the Nth classifier.

At block <NUM>, the generator trains the Nth classifier on all but the Dth dataset and tests the Nth classifier with the Dth fold. This allows the Nth classifier to be trained with a large dataset while reserving the Dth fold for testing. The generator uses indices provided from invocation of the K-fold function on the dataset as parameters into a function defined for training each classifier. For instance, a classification label of "<NUM>" may indicate that a data sample has a flaw based on expert knowledge or "ground truth. " The generator can be programmed to interpret a probability above <NUM> as classification related to a flaw and a probability of <NUM> and below as not related to a flaw. The generator can convert the probability to a binary classification value unless the classifier being trained generates a binary classification value. In addition, a classifier being trained may have a designated output for a first classification and a designated output for a second classification. The output node of the classifier with the greater probability would be selected by the generator. After a training iteration that involves training a classifier with samples in the training folds produces a trained classifier, the generator uses the index or indices into the test fold for the Dth iteration to input samples of the Dth fold into the trained model. For example, training the K-nearest neighbors classifier comprises training the classifier on each data sample (e.g., vector corresponding to a software artifact) in all folds other than the Dth fold. This training of the Nth classifier generates or results in a trained K-nearest neighbors classifier. The trained K-nearest neighbors classifier is then tested with the Dth fold by inputting the samples from the Dth fold. For each data sample S in the testing fold, the generator stores the prediction. Thus, the generator stores a probability PNS made by the Nth classifier for the sample S in the Dth fold as the Nth component of a set or vector of predictions for sample S.

At block <NUM>, the generator stores the predictions of the Nth trained classifier for samples in the Dth dataset fold as the Nth component of the Sth probability vector. This iterative training and testing of each classifier for each dataset fold yields SxM predictions which are stored as probability vectors. The stored probabilities can be associated with the corresponding classification labels for the samples within the probability vectors or a separate data structure. Embodiments can associate the classification labels with the probabilities different. For example, these can be associated by sample identifiers or the structures can be ordered consistently to allow for a same index to be used across structures for a same data sample. The probability vectors contain the predictions of the N classifiers for the S samples and are used for training a logistic regression model.

At block <NUM>, the generator determines whether there is an additional classifier to train and test. For example, the generator determines whether the loop control variable N is equal to M. If there is an additional classifier to train and test, the flow of operations returns to block <NUM>. If there is not an additional classifier, the flow of operations continues to block <NUM>.

At block <NUM>, the generator determines whether there is an additional dataset fold or partition. For example, the generator determines whether the loop control variable is equal to K. If there is an additional fold, the flow of operations returns to block <NUM>. Otherwise, the flow of operations continues to block <NUM>.

At block <NUM>, the generator feeds classified vulnerability data (classification labels) and the probability vectors to a logistic regressor. The logistic regressor trains a logistic regression model with the classification labels and probability vectors. The logistic regressor trains a model based on a logistic probability distribution given by: <MAT> where p(x) is the probability of a prediction being accurate, x is a probability vector, and β<NUM> and β<NUM> are parameters that best fit a logistic probability curve to the probability vectors. The parameters β<NUM> and β<NUM> are determined by fitting the above equation to the classified vulnerability data as a function of the probability vectors. Once determined, the parameters β<NUM> and β<NUM> specify the logistic probability distribution or the trained logistic regression model. To train the logistic regression model, the regressor linearly combines the probabilities of each probability vector and accepts as input features the linearly combined probabilities and the corresponding classification labels. The regressor can use stochastic gradient descent to estimate the values of the model parameters or coefficients β<NUM> and β<NUM>. Training continues until a specified accuracy threshold is satisfied.

A predetermined probability threshold can be used to turn the logistic regression model into a binary classifier. For example, if a probability threshold of p = <NUM> is used, vulnerability vectors which give a probability p(x) ≥ <NUM> will return a <NUM> (indicating a vulnerability) and vulnerability vectors which give a probability p(x) < <NUM> will return a <NUM> (indicating no vulnerability). The selection of a probability threshold is discussed in greater detail by the <FIG> description.

At block <NUM>, the generator stores as a classifier ensemble a combination of the M trained classifiers and the trained logistic regression model that satisfies the accuracy threshold, which may be based on specified precision and recall rates. To create the classifier ensemble, the generator constructs program code that passes an input vector for a software artifact or software component to each of the M trained classifiers, stores the outputs of the M trained classifiers as a set or vector, and passes that set or vector of M trained classifier outputs to the trained logistic regression model. The output of the trained logistic regression model can be a binary classification that is then associated with the software component represented by the input vector, or can be a probability value that is associated with the software component.

The above training refers to an initial training and testing to form a classifier ensemble. This initial training would likely have a large-scale training data set that encompasses a large time period (e.g., years or months). After training and deployment, the classifier ensemble continues to be trained with smaller training datasets, depending upon use. In contrast to the initial training, the output of subsequent training is used (e.g., stored for expert evaluation and/or inserted into a vulnerability database). The subsequent "training" datasets may be obtained from scheduled, periodic running of the data scraper or specific invocation of the data scraper to collect CBR data from a particular data source or set of data sources. Furthermore, classifier ensembles can be trained for different data sources. For example, a classifier ensemble can be trained for each flaw (bug or vulnerability) tracking tool or repository. Classifier ensembles can be trained for different open source repositories.

<FIG> depicts an example automatic vulnerability identification system. Vulnerability data <NUM> and classified vulnerability data <NUM> are both partitioned into K datasets which are fed to a K-stacking algorithm to generate K sets of training data <NUM> and testing data <NUM>, respectively. The training data <NUM> is used to train a set of classifiers 416a-f. The training data <NUM> and the testing data <NUM> are fed to the classifiers 416a-f to generate classifier data 418a-f. The K sets of the classifier data 418a-f are combined as probability vectors folded together to form K-fold classifier data <NUM> and classified vulnerability data <NUM> are fed to a logistic regressor <NUM> which outputs a trained logistic regression model <NUM>. An AVIS <NUM> with an AVIS controller <NUM>, a data scraper <NUM>, and a natural language processor <NUM> combines the logistic regression model <NUM> and a set of trained classifiers <NUM> to form the classifier ensemble <NUM>.

Each of the classifiers 416a-f are iteratively trained K times with different folds. Specifically, the classifiers 416a-f are iteratively trained with all but the holdout fold or partition of the vulnerability data <NUM> and the classified vulnerability data <NUM>. The trained classifiers 416a-f generate a set of classifier data 418a-f for the holdout partition of the vulnerability data <NUM> in each iteration.

As the classifiers 416a-f are trained and tested, each of the K sets of classifier data 418a-f are combined as the probability vectors <NUM>. Once K iterations of the training and testing have been run, the probability vectors <NUM> and the classified vulnerability data <NUM> are fed to the logistic regressor <NUM>, where the probability vectors <NUM> and the classified vulnerability data <NUM> are used to generate the logistic regression model <NUM>.

The logistic regressor <NUM> determines a logistic probability distribution based on the vulnerability vectors <NUM> and the classified vulnerability data <NUM>. The logistic regressor <NUM> fits a logistic probability distribution to the classified vulnerability data <NUM> as a function of the probability vectors <NUM>. The fitting process comprises determining N+<NUM> parameters (where N is the number of classifiers) which most closely fit a logistic probability distribution to the vulnerability data <NUM>.

The trained classifiers <NUM> and the output of the logistic regressor <NUM> are sent to form the ensemble of classifiers <NUM> by an AVIS <NUM>. The AVIS controller <NUM>, invokes the different elements of the AVIS <NUM> to automatically identify flaws in input software artifacts. If a software artifact is input, then the NLP <NUM> will process the software artifact and generate a software artifact vector. The AVIS controller <NUM> then passes the software artifact vector to classifier ensemble <NUM>, which generates a probability value indicating a probability that the corresponding software component has a flaw. If an open source repository or other target is submitted to the AVIS <NUM>, then the data scraper <NUM> is invoked to scrape the identified target and pass the scraped software artifacts to the NLP <NUM>.

<FIG> depicts the use of a natural language processor for generating vulnerability vectors. Based on a list of open source libraries <NUM>, a number of open source data repositories <NUM> are scraped over a network <NUM> by a data scraper <NUM> to collect CBR data <NUM>. The CBR data <NUM> is sent by the data scraper <NUM> to an NLP <NUM>. The NLP <NUM> generates vulnerability data <NUM>, which comprises a set of vulnerability vectors <NUM>. The NLP <NUM>. outputs counts of words or phrases in vulnerability detection vectors 520A-C based on an embedding of the CBR data <NUM> in a vector space <NUM>.

The NLP <NUM> performs operations to classify words and phrases from the CBR data <NUM> according to their indication of vulnerabilities to generate the vulnerability data <NUM>. The vulnerability data <NUM> comprises a set of vulnerability vectors <NUM><NUM>, which are frequency counts of words or phrases indicative of vulnerabilities as classified by the NLP <NUM>. The association of words and phrases with vulnerabilities is determined by the vulnerability detection vectors 520A-C. The vulnerability detection vectors 520A-C may be manually determined based on words or phrases known to indicate vulnerabilities in open source libraries. The size of each of the vulnerability detection vectors 520A-C may range from several terms to hundreds of terms; for example, a typical vulnerability detection vector may comprise <NUM> words or phrases (tokens). A corresponding vector space <NUM> is constructed with dimension equal to the sum of dimensions of the vulnerability detection vectors 520A-C.

The NLP <NUM> embeds the CBR data <NUM> into the vector space <NUM> based on the distribution of words or phrases in the CBR data <NUM>. By embedding the CBR data <NUM> into the vector space <NUM>, each of the words or phrases in the CBR data <NUM> can be compared to words or phrases in the vulnerability detection vectors 520A-C; based on their cosine similarity. The cosine similarity of two vectors is given by:
<MAT>.

Wherein A and B are vectors and θ is the angle between them. A similarity of <NUM> indicates identical parallel vectors while a similarity of <NUM> indicates orthogonal vectors. Each of the components of the vulnerability detection vectors are mutually orthogonal vectors in the vector space <NUM>. New words or phrases can be embedded in the vector space <NUM> based on their association with words already in the vector space <NUM>.

As the CBR data <NUM> is classified by the NLP <NUM>, words and phrases are embedded in the vector space <NUM> based on their proximity to words in the vector space <NUM>. For example, a new word in the CBR data <NUM> can be embedded in the vector space by taking the sum of each vector corresponding to a word or phrase in the vulnerability detection vectors 520A-C and dividing by the distance of the two words. For example, if some CBR data <NUM> comprises the phrase "The exploit renders my computer unresponsive due to injection of faulty data". Embedding the word "unresponsive" is possible by using the words "exploit" (which is <NUM> words away), "injection" (which is <NUM> words away) and "faulty" (which is <NUM> words away) to generate a vector for "unresponsive in the vector space. By adding each weighted word association to a vector with dimension equal to the sum of the dimensions of the vulnerability detection vectors 520A-C, the vector for the word "unresponsive" may be represented by:
<MAT>.

Wherein the entry <NUM>/<NUM> corresponds to "exploit", the entry <NUM>/<NUM> corresponds to "injection", and the entry <NUM>/<NUM> corresponds to "faulty". In this way, every word or phrase in the CBR data <NUM> can be embedded into the vector space <NUM>. In some cases, words in the CBR data <NUM> may be associated with one another as part of the embedding process. In some cases, the embedding may use a different weighting scheme to associate words and phrases. In some cases, there may exist a threshold to remove associations between words which are distant. For example, the association of two words with more than <NUM> words between them may be ignored. Such thresholds are illustrated by the ellipses about words or phrases in the vulnerability detection vectors 520A-C in the vector space <NUM>. Any words outside of the three ellipses would be ignored when generating the vulnerability data <NUM>.

Based on the embedding of the CBR data <NUM> in the vector space <NUM>, the NLP <NUM> generates the vulnerability data <NUM> by summing the vectors of each word or phrase in the CBR data <NUM> to generate the vulnerability vectors <NUM>, which comprise the vectors strong _vuln_count, medium_vuln_count, and weak_vuln_count. These vectors are generated by summing the associations of each word in the CBR data <NUM> with each word in the vulnerability detection vectors 520A, 520B, and 520C, respectively.

A feature of the NLP <NUM> is that, because it processes a combination of natural language and generic computer science terms, it is "language agnostic" in the sense that the programming languages of open source libraries do not affect the ability of the NLP <NUM> to generate the vulnerability data <NUM> for vulnerability identification.

<FIG> is a graph depicting metrics for an automatic vulnerability identification system. The graph <NUM> has an x-axis <NUM> measuring the probability threshold for predictions made by an AVIS and a y-axis <NUM> depicting multiple metrics for the AVIS. A line <NUM> depicts the recall rate of predictions made by the AVIS and a line <NUM> depicts the precision of predictions made by the AVIS. The two lines <NUM>, <NUM> are dependent upon the probability threshold of the AVIS (i.e., the probability threshold for accepting or rejecting predictions made by the AVIS as introduced by <FIG>). For various applications, it may be beneficial to increase or decrease the probability threshold of the AVIS to ensure a greater recall rate with lower precision or a greater precision with lower recall rate.

The values shown in the graph <NUM> are taken from a trained AVIS. The high values of precision and recall rate are indicative of an improvement in the technology of automatic vulnerability identification for open source libraries. Predictions made by an AVIS can be sorted into one of four categories when classified vulnerability data is available. The four categories are true positives, false positives, true negatives, and false negatives. True positives correspond to the AVIS correctly detecting a vulnerability. False positives correspond to the AVIS incorrectly detecting a vulnerability. True negatives correspond to the AVIS correctly detecting no vulnerabilities. False negatives correspond to the AVIS incorrectly detecting no vulnerabilities. The precision of predictions made by the AVIS is a measure of how many vulnerabilities were correctly detected based on the number of true positives and false positives predicted by the AVIS and is given by:
<MAT>.

At equilibrium (where precision and recall rate converge), a precision of over <NUM>% is given by the disclosed method by using a probability threshold of <NUM>%. The recall rate of predictions made by the AVIS is a measure of how many vulnerabilities were detected based on the number of true positives and false negatives predicted by the AVIS and is given by:
<MAT>.

At equilibrium, a recall rate of over <NUM>% is given by the disclosed method by using a probability threshold of <NUM>%. With a lower probability threshold, a recall rate of over <NUM>% can be achieved.

In some cases, clients may request the AVIS to use a custom probability threshold in accordance with client needs as an operational parameter in client requests. For example, a project which requires a high level of stability may specify a low probability threshold for the classifier ensemble (e.g., a probability value <= <NUM> is classified as not related to a flaw) to maximize the recall rate. This would guarantee the client the detection of a greater number of vulnerabilities (at the cost of a larger number of false positives as well). For clients which do not require as much stability in their projects, a higher probability threshold may be used to expedite the development process by identifying potential vulnerabilities with high precision.

By adjusting the probability threshold of the AVIS, the balance of precision to recall rate may be adjusted. As the probability threshold of the AVIS decreases the recall rate decreases while the precision rate increases. As a default, the AVIS may use a probability threshold that produces equal values for precision and recall rate.

The flowcharts are provided to aid in understanding the illustrations and are not to be used to limit scope of the claims. The flowcharts depict example operations that can vary within the scope of the claims. Additional operations may be performed; fewer operations may be performed; the operations may be performed in parallel; and the operations may be performed in a different order. For example, the operations depicted in blocks <NUM>-<NUM> can be performed in parallel or concurrently. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by program code. The program code may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable machine or apparatus.

As will be appreciated, aspects of the disclosure may be embodied as a system, method or program code/instructions stored in one or more machine-readable media. Accordingly, aspects may take the form of hardware, software (including firmware, resident software, micro-code, etc.), or a combination of software and hardware aspects that may all generally be referred to herein as a "circuit," "module" or "system. " The functionality presented as individual modules/units in the example illustrations can be organized differently in accordance with any one of platform (operating system and/or hardware), application ecosystem, interfaces, programmer preferences, programming language, administrator preferences, etc..

Any combination of one or more machine readable medium(s) may be utilized. The machine readable medium may be a machine readable signal medium or a machine readable storage medium. A machine readable storage medium may be, for example, but not limited to, a system, apparatus, or device, that employs any one of or combination of electronic, magnetic, optical, electromagnetic, infrared, or semiconductor technology to store program code. More specific examples (a non-exhaustive list) of the machine readable storage medium would include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a machine readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine readable storage medium is not a machine readable signal medium.

A machine readable signal medium may include a propagated data signal with machine readable program code embodied therein, for example, in baseband or as part of a carrier wave. A machine readable signal medium may be any machine readable medium that is not a machine readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a machine readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for aspects of the disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as the Java® programming language, C++ or the like; a dynamic programming language such as Python; a scripting language such as Perl programming language or PowerShell script language; and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on a stand-alone machine, may execute in a distributed manner across multiple machines, and may execute on one machine while providing results and or accepting input on another machine.

The program code/instructions may also be stored in a machine readable medium that can direct a machine to function in a particular manner, such that the instructions stored in the machine readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

<FIG> depicts an example computer system with an automatic vulnerability identification system. The computer system includes a processor <NUM> (possibly including multiple processors, multiple cores, multiple nodes, and/or implementing multi-threading, etc.). The computer system includes memory <NUM>. The memory <NUM> may be system memory (e.g., one or more of cache, SRAM, DRAM, zero capacitor RAM, Twin Transistor RAM, eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM, etc.) or any one or more of the above already described possible realizations of machine-readable media. The computer system also includes a bus <NUM> (e.g., PCI, ISA, PCI-Express, HyperTransport® bus, InfiniBand® bus, NuBus, etc.) and a network interface <NUM> (e.g., a Fiber Channel interface, an Ethernet interface, an internet small computer system interface, SONET interface, wireless interface, etc.). The system also includes automatic vulnerability identification system <NUM>. The automatic vulnerability identification system <NUM> identifies vulnerabilities in open source components (e.g., libraries or projects) based on commit messages and bug reports. Any one of the previously described functionalities may be partially (or entirely) implemented in hardware and/or on the processor <NUM>. For example, the functionality may be implemented with an application specific integrated circuit, in logic implemented in the processor <NUM>, in a co-processor on a peripheral device or card, etc. Further, realizations may include fewer or additional components not illustrated in <FIG> (e.g., video cards, audio cards, additional network interfaces, peripheral devices, etc.). The processor <NUM> and the network interface <NUM> are coupled to the bus <NUM>. Although illustrated as being coupled to the bus <NUM>, the memory <NUM> may be coupled to the processor <NUM>.

While the aspects of the disclosure are described with reference to various implementations and exploitations, it will be understood that these aspects are illustrative and that the scope of the claims is not limited to them. In general, techniques for automatic vulnerability identification as described herein may be implemented with facilities consistent with any hardware system or hardware systems. Many variations, modifications, additions, and improvements are possible.

Plural instances may be provided for components, operations or structures described herein as a single instance. Finally, boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the disclosure. In general, structures and functionality presented as separate components in the example configurations may be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements may fall within the scope of the disclosure.

Claim 1:
A method comprising:
training and testing a set of classifiers (416a-f) with k-fold cross validation on a first dataset (<NUM>, <NUM>) comprising vectors (<NUM>, 520A-C) representing software components (<NUM>, <NUM>) and labels (<NUM>),
wherein each label indicates whether a respective one of the software components has a vulnerability or flaw,
wherein a set of probability values (<NUM>) generated for each of the vectors by the set of classifiers from the testing is stored;
training a logistic regression model (<NUM>) with the stored sets of probability values and the labels; and
combining the trained set of classifiers and the trained logistic regression model to generate an ensemble (<NUM>, <NUM>, <NUM>) that indicates whether a software component of said software components has a vulnerability or flaw.