# Collecting Vulnerable Source Code from Open-Source Repositories for Dataset Generation
## 3 Discussion
### Constraints Related to SonarCloud's Web API  
One of the most important restrictions to face when querying SonarCloud's web API is the one commonly known as _the 10,000 issue limit_. This constraint implies that every single request made to /api/issues/search will be responded to only with the first 10,000 results. As SonarCloud's prior, and now deprecated, documentation page states: \"If the number of issues is greater than 10,000, only the first 10,000 ones are returned by the web service\" [59]. Even though the quoted sentence comes from older documentation, the limit still applies nowadays, despite being undocumented. There are many questions in different forums, from users just like us, asking about this very same limit. The response is always the same: there is no way around it.  
Another drawback SonarCloud presents is the lack of vulnerability-type filtering. That is, the ability to retrieve only source code files that are tagged as vulnerable to a given vulnerability (stack-based buffer overflow, format string, integer overflow...). Although filtering by issue type (VULNERABILITY, HOTSPOT...) is possible, filtering based on vulnerability type is a much needed feature to gather specific source code. From TOOBAD4 ML's perspective, such a feature would ease the parsing job. Currently, the datasets gathered by SVCP4C include all kinds of vulnerabilities, not only buffer overflow.  
Up to this point during development, we faced some problems whose solution(s) can be directly seen in Listing 1. For example, we found out that we cannot just append every result of the queries asking for vulnerabilities into one single file because the result is a mal-formatted JSON. This is because SonarCloud sends JSON objects as responses and, as such, these include the opening and closing square brackets. A JSON file, to be well-formatted, must include a single JSON object, that is, a single pair of square opening and closing brackets. The solution we adopted is requesting the first 500 results (page 1) and write them to a file. Immediately after, we parse the file and request the corresponding source code. When we get it, we request the next 500 (page 2) vulnerabilities, write (not append) them to a file and, once again, request the source code. This loop goes on until we reach the 10,000-results limit imposed by SonarCloud. This is reflected in steps 23 and 34 of Listing 1.  
Another characteristic behavior of SonarCloud's web API is that each vulnerable code line within the same source code file is treated as a different issue. That is, a source file with 13 different vulnerabilities is translated into 13 different issues when querying SonarCloud. At the moment of retrieving the corresponding source file, the same file would be downloaded 13 times, whereas only the tagged line would be different. The solution implemented in SVCP4C is to compile all vulnerable lines that refer to the same file, download the file, and append the lines as a comment at the end of the file (step 47 in Listing 1).  
Figure 4: Reports of most extended overflows and analysis of issues per file.  
Finally, additional checks are required when requesting source files because SonarCloud references missing files. That is, it maintains the list of issues even though the file those issues arise from does not exist anymore. When attempting to download a missing source file the result is a file whose sole content is a JSON list called \"errors\" containing \"msg\" keys. The solution implemented in SVCP4C is rather straightforward, the content that is about to be written out is first inspected and, if it contains any \"errors\" JSON list, we skip it. Step 39 of Listing 1 shows the check.