Project: StackRox (Provisioning / Security & Compliance)
File: 0005-scannerv4-read-cvss-data.md

Author(s): Yi Li <yli3@redhat.com> Created: [2023-10-10] Status: Updated Updated By: . The National Vulnerability Database (NVD) has announced the retirement of its JSON feeds by December 15, 2023. This creates the need to change our existing CVSS data updater pipeline for Scanner V4, which currently relies on these JSON feeds. The NVD JSON feed offers bundled data on a yearly basis, spanning from 2002 to the present year. A benefit is that only two requests are needed in the workflow for each year (one for the meta file and one for the data bundle). However, a limitation is the absence of filtering capabilities, such as by CVSS score or modification date. In contrast, the NVD API allows data retrieval through specific HTTP URL parameters, such as publish date, modified date, or CVSS score, enabling more precise data acquisition. The updated NVD CVSS GitHub Workflow (CI) has transitioned to sourcing data from the NVD CVE API, rather than the NVD JSON feeds, ensuring the timely update of the NVD CVSS data bundle in Google Storage. The workflow is designed to download data by specifying a range of start and end dates, spanning from the year 2002 to the current year. For instance, a command listed below could be used: ``` curl "https://services.nvd.nist.gov/rest/json/cves/2.0?noRejected&pubStartDate=2021-08-04T00:00:00.000&pubEndDate=2021-10-22T00:00:00.000&startIndex=0" > 2021-0-0.json ``` We will break every year down to 4 quarters, such as: ``` periods=( "${year}-01-01T00:00:00.000 ${year}-03-31T23:59:59.999" "${year}-04-01T00:00:00.000 ${year}-06-30T23:59:59.999" "${year}-07-01T00:00:00.000 ${year}-09-30T23:59:59.999" "${year}-10-01T00:00:00.000 ${year}-12-31T23:59:59.999" ) ``` For each quarter, it iteratively exhausts the startIndex to capture all associated data in that date range. After successfully downloading all JSON files for a given year, we consolidate the 'vulnerabilities' field from each file into a single comprehensive JSON file. Upon generating these annual consolidated files, we then compress them into a singular archive for efficient storage and handling. Our decision to divide the data retrieval into four quarters annually is both clear and concise, aligning with the NVD's maximum supported range of 120 days per request. We've chosen to base our API requests on publish date starting from year 2002 to ensure vulnerability reports are enriched with either CVSS V3 score or CVSS V2 score if V3 score is missing. The CVSS Updater in Central will be modified to download compressed file (generated by CI mentioned above) in the Google bucket. This eliminates the need for JSON parsing in the updater. Scanner V4 will pull the data bundle from Central and populate the Matcher DB with the enrichment

A custom enricher is essential for Scanner V4, as it will provide a mechanism to retrieve JSON data from Central, parse the enrichment details, and store them in the Scanner V4 database. Additionally, our data will be sourced from the NVD CVE API rather than JSON feeds, so the custom enricher can accommodate the distinct data formats. The customized enricher also filters out data missing both CVSS V2 score and CVSS V3 score. This also offers greater flexibility, allowing us to extract more information from the fetched JSONs than what Claircore offers. The CVSS updater checks the modification date of the data bundle in Google bucket for every 30 mins and it downloads the data once the modification date has been changed. The CVSS Updater in Central will attempt to download the data bundle from the Google bucket using exponential retries in case of any failures, up to a maximum interval within the next loop. Should the GitHub Actions workflow (CI) experience difficulties in accessing or downloading segments of the NVD data due to an HTTP request failurepotentially caused by the NVD API being unavailable or due to rate limitingthe affected HTTP request should be automatically retried following an exponential backoff strategy. In the event that all retry attempts fail, the workflow will be terminated, and the system will await the next scheduled update in 4 hours. The team will be promptly notified of any aborted or failed workflows via Slack. We will also support manually triggering in this GH workflow so that gives up flexibility to re-download the data when necessary. One drawback of the NVD CVE API is its rate limit: only 5 requests are allowed per 30 seconds without an API Key. This restriction necessitates that our workflow fetch data at a slower pace. Additionally, transitioning from the NVD JSON feed to the NVD CVE API brings in pagination limitations. For example, when fetching data with pubStartDate=2021-08-04 and pubEndDate=2021-10-22, the API provides a maximum of 2000 CVEs per response. This means multiple requests are required to retrieve the complete dataset for that date range. The result is an increased number of smaller JSON files. Consequently, we need to compress all JSON files into a single archived file such as tar.gz or tar.xz. Following this modification, Central will no longer have a dependency on Claircore. The Claircore enricher will be used in Scanner V4, which adopts Claircore as its primary scanning engine by design. Given that we will terminate the workflow in the event of incomplete data, there may be a delay of several hours before we can acquire the most updated data bundle. However, should such a workflow termination occur, the team will be promptly notified. Also, it helps the data to be updated with the support of manually triggering the workflow as mentioned above.