# Collecting Vulnerable Source Code from Open-Source Repositories for Dataset Generation
## 2 Methodology
### Algorithm Pipeline  
The overall workflow of SVCP4C is depicted in Figure 3. The workflow is split into five phases. In the first phase, the crawler requests information about the source code identified as vulnerable. For this phase, several sources for gathering BufferOverflow information have been analyzed. We initially checked the Common Weakness Enumeration (CWE) [53] and Common Vulnerability Enumeration. CWE is a community-developed list of common software security weaknesses. It represents the starting point for weakness identification, mitigation, and prevention from a source code point of view. CVE presents a list of entries for publicly known cybersecurity vulnerabilities. CWE, which focuses on software approaches, identifies software vulnerabilities such as CWE-120: Buffer Copy without Checking Size of Input; CWE-121: Stack-based Buffer Overflow; and CWE-122: Heap-based Buffer Overflow. On top of them, there are formal datasources such as the National Vulnerability Database (NVD) [54] or informal such as CVEdetails.com [55]. Although these sources are enough for some vulnerability analysis [56], they do not present a direct link to the source code that represents the buffer overflow, and it is necessary to crawl in-depth, looking for public repositories (if possible) to reach the vulnerability and parse it. Given this scenario, we decided to focus our efforts on lists that provide information about possible buffer overflow issues that could be raised in a project. Thus, we initially had two possibilities: (1) massively download github repositories and analyze the source code locally and (2) analyze results of current _Software As A Service_ tools, that provide this information in the cloud. We analyzed current cloud solutions offering this information. Although there are several options like Datree.io, Codescene.io, or kiuwan.com, given the REST API limitations imposed by licenses and types of accounts, and because we had experience with SonarQube, we decided to choose SonarCloud for our research. In the second phase, the crawlers collect the source code files identifies in the previous phase. We take advantage of the possibility that SonarCloud offers of retrieving the issues of a given source code file (previous phase) and download that particular file, simplifying the process of data collection. In the third phase, SVCP4C appends the vulnerable lines as comments to the original source code files. This way the files are easier to handle in further analysis. In the forth phase, the commented source code is stored in the local file system. In the last phase, the result of the whole process is released to the public. The data is publicly available at the repository hosted by Github. The source code of SVCP4C is released as well. Figure 3: SonarCloud detecting use of an insecure function. Figure 2: SonarCloud detecting use of an insecure function. The pipeline of our algorithm is illustrated in Listing 1. Some parameters that appear in the algorithm require further contextualization:  
* \(p\). This parameter represents page number. SonarCloud responds with at most 500 results per page. If one query generates more than 500 results, p is pre-incremented and the web service is requested again. * _ps_. This parameter represents page size. SonarCloud allows users to specify how many results they want to see per page, in our case per HTTP response. ps is a constant equal to 500 as it is the maximum page size allowed by SonarCloud and we want to retrieve as much information as possible. * _remainingResults_. This parameter represents how many results are left. That is, if the query generated more than 500 results, remainingResults is checked to request again. SVCP4C performs several HTTP requests to SonarCloud's REST API.