Project: VMClarity (Provisioning / Security & Compliance)
File: change-asset-findings-relationships.md

Note: this RFC template follows HashiCrop RFC format described | | | ||| | Created | 2023-01-15 | | Status | WIP\| InReview \| Approved \| Obsolete | | Owner | ramizpolic | | Approvers | | This RFC proposes adding `AssetFinding` API model and its supporting logic to achieve API consistency, improve efficiency, and enable aggregation. Each finding defines some specific security details (e.g. vulnerability) discovered on an asset. The same finding can be discovered on multiple assets by different asset scans. This means that there is many-to-many relationship between findings, assets, and asset scans. However, the existing describe findings with one-to-one relationship to assets and asset scans. In addition, the treats every new finding as unique without performing any checks. Together, this can introduce issues for multiple reasons: Each `Finding` is coupled with an `Asset` it was discovered on and the `AssetScan` that discovered it. This leads to unnecessary data duplication as each `Finding` with different associations to these models will be treated as unique. In addition, the lack of an actual uniqueness check completely ignores the data already present in the database. Together, this creates performance and memory utilization overheads. The model differs from the existing association patterns between models compared to e.g. `Asset`, `AssetScan`, and `AssetScanEstimation`. This introduces complexities due to a lack of proper aggregation by overloading the `Finding` data returned by the API or shown on the UI. Findings should serve as a collection of all security details discovered so far, irrelevant of the assets they were discovered on or the scans that discovered them. This means that the existing `Asset` relationship can be dropped from the `Finding` model. Instead, to express the relationship between assets and findings, new `AssetFinding` model should be added. This ensures many-to-many relationship between the two, i.e. a single finding can be discovered on multiple assets, and an asset can contain multiple findings. A similar approach can be used for asset scans and findings, although this is less relevant (check section). Instead, the `AssetScan` relationship in `Finding` should be preserved, but it should be noted that it represents the first scan that discovered a given finding. The database logic should implement uniqueness check similar to the existing logic as shown . The data required for the check can be extracted directly from the actual object. Uniqueness checks are required for both `Finding` and `AssetFinding` models. To provide statistical data and aggregation capabilities between other models, `Finding` should be expanded with the `assetsCount` property. Together, these approaches address the API consistency and data duplication issues. Take the number of packages that can be discovered on a single production container image as a reference. In a real-world scenario, many security findings can be discovered on a specific asset. Additionally, assets and findings are versioned models where every new version is treated as a completely new

Similarly, asset scans are also versioned in terms of schedule, i.e. new asset scans can be created weekly. Due to the current nature of these models (e.g. creating a new `Finding` or `Asset` for each version, or `AssetScan` on schedule), the size of the tables can grow rapidly. The reason to have `AssetFinding` and `AssetScanFinding` as separate tables relates to time and space complexity. This is also why a unified table between all three models is not viable. Example A To keep track of `#assets = 100`, `#assetScans = 100`, and `#findings = 100` with no versioning changes and no additional scans requires: ``` R(asset, assetScan, finding) = #asset #assetScan #finding = 100^3 items - Unified table is larger than both other tables combined R(asset, finding) = #asset * #finding = 100^2 items - AssetFinding table R(assetScan, finding) = #assetScan * #finding = 100^2 items - AssetScanFinding table ``` Example B Unlike the previous example, this takes into account the versioning and scheduling which is a better metric for real-world usage. To keep track of `#assets = 100`, `#assetScans = 100`, and `#findings = 100` with 10 new versions and 10 new assets for each scan requires: ``` R(asset, assetScan, finding, version, schedule) = 10^5 * 100^2 - Combined relationship table grows too quickly compared to other tables R(asset, finding, version) = 10^2 * 100^2 - AssetFinding table depends on assets and findings, i.e. versioning; also transiently depends on scans, but this is omitted for simplicity. R(assetScan, finding, version, schedule) = 10^2 * 100^2 - Scans can be scheduled to occur more often than the versioning changes on other models ``` Therefore, it makes sense to keep track of associations in separate tables between these models to reduce size and improve efficiency. Otherwise, sharding or more complex approaches must be used which are not viable options at this stage. This RFC does not intend to propose changes regarding the relationship between findings and asset scans. It is assumed that the asset scan in a given finding represents the last scan that discovered it, while the timestamps are used for object lifecycle tracking. If required to keep track of all asset scans that discovered a specific finding and vice-versa, a new `AssetScanFinding` model can be added. Apart from statistical analysis, this offers no additional benefit as we usually do not care much about the actual scans but rather their results in terms of discovery of new assets and findings. This behavior, if required, can be addressed following the same approach as described in the section. Adding the aggregation methods to the `uibackend` API was considered but abandoned as it does not address the data duplication issue. The new models should be described as follows: ```yaml Finding: type: object allOf: $ref: '#/components/schemas/Metadata' type: object properties: id: type: string

# added in case some of the non-unique identifiers are updated type: integer firstSeen: description: When this finding was first discovered by a scan type: string format: date-time lastSeen: description: When this finding was last discovered by a scan type: string format: date-time lastSeenBy: $ref: '#/components/schemas/AssetScanRelationship' assetsCount: description: Total number of assets that contain this finding type: integer findingInfo: anyOf: $ref: '#/components/schemas/PackageFindingInfo' $ref: '#/components/schemas/VulnerabilityFindingInfo' $ref: '#/components/schemas/MalwareFindingInfo' $ref: '#/components/schemas/SecretFindingInfo' $ref: '#/components/schemas/MisconfigurationFindingInfo' $ref: '#/components/schemas/RootkitFindingInfo' $ref: '#/components/schemas/ExploitFindingInfo' $ref: '#/components/schemas/InfoFinderFindingInfo' discriminator: propertyName: objectType mapping: Package: '#/components/schemas/PackageFindingInfo' Vulnerability: '#/components/schemas/VulnerabilityFindingInfo' Malware: '#/components/schemas/MalwareFindingInfo' Secret: '#/components/schemas/SecretFindingInfo' Misconfiguration: '#/components/schemas/MisconfigurationFindingInfo' Rootkit: '#/components/schemas/RootkitFindingInfo' Exploit: '#/components/schemas/ExploitFindingInfo' InfoFinder: '#/components/schemas/InfoFinderFindingInfo' ``` The API changes impact the database schema and should be handled accordingly. In addition, the database-related logic such as bootstrapping the demo data needs to be updated to reflect these changes. For the current case, assume that the `assetsCount` is always zero. The uniqueness check can be added similarly to the existing implementations. An example is given below. ```go func (s FindingsTableHandler) checkUniqueness(finding models.Finding) (models.Finding, error) { discriminator, err := finding.FindingInfo.ValueByDiscriminator() if err != nil { return nil, fmt.Errorf("failed to get value by discriminator: %w", err) } var filter string switch info := discriminator.(type) { case models.PackageFindingInfo: filter = fmt.Sprintf("uniqueness query for package finding", info) case models.VulnerabilityFindingInfo: filter = fmt.Sprintf("uniqueness query for vulnerability finding", info) // implementation of other cases default: return nil, fmt.Errorf("finding type is not supported (%T): %w", discriminator, err) } // implementation of the actual check return nil, nil } ``` ```yaml AssetFinding: type: object allOf: $ref: '#/components/schemas/Metadata' type: object properties: id: type: string asset: $ref: '#/components/schemas/AssetRelationship' finding: $ref: '#/components/schemas/FindingRelationship' ``` The implementation should address the following cases: Update database schema and implement related database table. Additional table index should be created for `(asset.ID, finding.ID)` fields. Add uniqueness check for the new database model Add CRUD controller logic for the new model to handle `/asset-findings` and `/asset-finding/{id}` routes. The filters for `GET /asset-finding` can contain either `findingID` or `assetID` to get the list of related objects for a given filter. This allows fetching the objects for a query such as which assets/findings are associated for given finding/asset. The UI can make use of this data by extracting only the required fields (e.g. `.asset`) from the `[]AssetFinding` response. When performing CRUD operations, note that the `assetsCount` property of the related `Finding` can change either by decrementing or incrementing it. Described in section. This RFC has no visible impacts on the UX. This RFC changes the following UI components: `/findings/{findingType}` table drops the Asset Name and Asset location columns. Instead, they are replaced with a single Asset Count column that shows the number of assets related to a given finding. The `firstSeen` field needs to be used to describe when the asset was discovered. `/findings/{findingType}/{findingID}` changes Asset details menu item by showing the table instead of a single asset. The relevant data is obtained by fetching `[]AssetFinding` from `GET /asset-findings` for a given `findingID`. Subsequently, the asset data for the table is extracted from the returned array by extracting the `.asset` key.