# Unveiling the Landscape of Operating System Vulnerabilities
## 4 Research Methodology
### Data Collection and Processing
#### 4.1.2 Data Scraping Method  
Algorithm 1 shows the method of data scraping from the NVD website using the CVE IDs collected from CVE Mitre. An example of main_url for CVE-2019-15222 is as follows: [[https://nvd.nist.gov/vuln/detail/CVE-2019-15222](https://nvd.nist.gov/vuln/detail/CVE-2019-15222)]([https://nvd.nist.gov/vuln/detail/CVE-2019-15222](https://nvd.nist.gov/vuln/detail/CVE-2019-15222)) accessed on 28 June 2023.  
Information about any CVE ID can be extracted by placing the CVE ID at the end of the main_url. We used requests, BeautifulSoup, and pandas libraries to perform the data scraping and cleaning. In each step, different attributes of \"_id_\", \"_class_\", etc., were used to scrape the corresponding data. For instance, the CVSS scores lie in the \"_div_\" block with the attribute of \"id_\" = \"_vulnCrossPanel_\". The function _get_warning_text_ checks for the warning status name with attribute \"_data-testid_\" = \"_vuln-warning-status-name_\". The warning texts can be _rejected, awaiting_analysis, undergoing_analysis, deferred_, etc., based on the provided status. In a similar manner, we created the _get_cvs_scores_ method to extract the CVSSv3-NIST, CVSSv3-CNA, CVSSv2-NIST, and CVSSv2-CNA. The function _get_cwe_ collects the CWE ID if present in the _main_url_. Finally, the _product_type, product_name, vendor_, and _corresponding versions_ are extracted using _get_all_products_data_ method. We created this method by extracting the CPE2.3 records present in the main_url and splitting the required products' data. Each extraction was followed by the timestamp since the data may change over time. There were 278,146 CVE entries to scrape; therefore, we leveraged Python's _multiprocessing_ library to run the scraping algorithm in parallel.  
```
Function_scrape-data-from-nvd(nvd_base_url,cve_ids) Input:nvd_base_url and cve_ids Output:Dataframe containing product's type, name, vendor, versions, CVSS scores, CWE ID and scraping timestamp requests \(\leftarrow\) object of python requests library nvd_base_url \(\leftarrow\)\"[[https://nvd.nist.gov/vuln/detail/](https://nvd.nist.gov/vuln/detail/)]([https://nvd.nist.gov/vuln/detail/](https://nvd.nist.gov/vuln/detail/))\" (accessed on 1 June 2023) foreachcve_id \(\in\)cve_idsdo main_url \(\leftarrow\)nvd_base_url + cve_id response \(\leftarrow\) requests.get(main_url) timestamp \(\leftarrow\) record the timestamp using datetime library page_data \(\leftarrow\) BeautifulSoup(response.content, \"html.parser\") warning_text \(\leftarrow\)get_warning_text(\"data-testid\": \"vuln-warning-status-name\") ifwarning_text then Save the cve_id in corresponding warning_text CSV file continue end  table \(\leftarrow\)page_data.find(\"_table_\",attrs=\"_id\": \"_vulnDetailTableView\"_) div \(\leftarrow\)table.find(\"_div_\",attrs=\"_class\": \"_row\"_) div \(\leftarrow\)div.find(\"_div_\",attrs=\"_class\": \"_col-lg-9 col-md-7 col-sm-12\") cvss_div \(\leftarrow\)div.find(\"_div_\",attrs=\"_id\": \"_vulnCussPanel\"_) cvss_scores = get_cvss_scores(cvss_div) div \(\leftarrow\)div.find(\"_div_\",attrs=\"_class\": \"_row col-sm-12\") cve_div \(\leftarrow\)div.find(\"_div_\",attrs=\"_id\": \"_vulnTechnicalDetailsDiv\"_) cve_id \(\leftarrow\)get_cve(cve_div) all_prod_div \(\leftarrow\)find the next div block with empty \"_id\"_ and \"_class\"_ attributes has_prod, all_products_details \(\leftarrow\)get_all_products_data(cve_id, all_prod_div) ifhas_prod then Append all_products_details (type, name, vendor, versions), cvss_scores, cve_id, timestamp into the dataframe end else Append the cve_id to no_products_found dataframe end end return dataframe with values product_type, product_name, vendor_name, product_versions, cvss_scores, cve_id, timestamp end
```  
**Algorithm 1** Data Collection Pseudocode