Repository: cloud-bulldozer/orion
Pull request #126: [PERFSCALE-3433] Add virt-density support
Author: login: josecastillolema
Labels: name: enhancement
description: New feature or request
Created: 2025-04-22T08:56:29Z | Merged: 2025-05-28T13:12:52Z

## Type of change

- [ ] Refactor
- [x] New feature
- [ ] Bug fix
- [ ] Optimization
- [ ] Documentation Update

## Description

Add support for the virt-density workload

## Testing

```
orion cmd --config examples/metal-perfscale-cpt-virt-density.yaml --lookback 15d --hunter-analyze
2025-04-22 11:06:03,315 - Orion      - INFO - file: orion.py - line: 125 - 🏹 Starting Orion in command-line mode
2025-04-22 11:06:03,321 - Orion      - INFO - file: utils.py - line: 241 - The test metal-perfscale-cpt-virtdensity has started
2025-04-22 11:06:03,321 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=perf_scale_ci-*
2025-04-22 11:06:03,737 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-04-22 11:06:03,864 - Orion      - INFO - file: utils.py - line: 52 - Collecting vmiReadyLatency
2025-04-22 11:06:03,864 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-04-22 11:06:03,988 - Orion      - ERROR - file: utils.py - line: 72 - Couldn't get metrics vmiReadyLatency, exception 'timestamp'
2025-04-22 11:06:03,989 - Orion      - INFO - file: utils.py - line: 52 - Collecting apiserverCPU
2025-04-22 11:06:03,989 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-04-22 11:06:04,771 - Orion      - INFO - file: utils.py - line: 52 - Collecting multusCPU
2025-04-22 11:06:04,772 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-04-22 11:06:05,469 - Orion      - INFO - file: utils.py - line: 52 - Collecting monitoringCPU
2025-04-22 11:06:05,469 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-04-22 11:06:06,505 - Orion      - INFO - file: utils.py - line: 52 - Collecting ovnCPU
2025-04-22 11:06:06,505 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-04-22 11:06:07,645 - Orion      - INFO - file: utils.py - line: 52 - Collecting etcdCPU
2025-04-22 11:06:07,645 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-04-22 11:06:07,915 - Orion      - INFO - file: utils.py - line: 52 - Collecting etcdDisk
2025-04-22 11:06:07,915 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-04-22 11:06:08,065 - Orion      - INFO - file: utils.py - line: 52 - Collecting kubelet
2025-04-22 11:06:08,066 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
metal-perfscale-cpt-virtdensity
===============================
time                       uuid                                  buildUrl                                                                                                                                                                                                                          apiserverCPU_avg    multusCPU_avg    monitoringCPU_avg    ovnCPU_avg    etcdCPU_avg    etcdDisk_avg    kubelet_avg
-------------------------  ------------------------------------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  ------------------  ---------------  -------------------  ------------  -------------  --------------  -------------
2025-04-09 17:19:56 +0000  796c69d4-e6da-4686-a6d6-19b559cb60ab  https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/63576/rehearse-63576-periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1909955347544543232             39.9121          1.2819               1.65655       7.38165        27.179       0.00211551        82.5442
2025-04-11 12:18:30 +0000  cec16171-6032-4c8e-a45d-f0e2a6f33a50  https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1910604335176224768                                                                   38.1838          1.29651              1.7238        8.39057        26.3422      0.00230283        85.4776
2025-04-12 12:17:41 +0000  3b43430c-6247-48e1-92b3-96935f2bb2b3  https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1910966521669095424                                                                   34.9846          1.10082              1.83616       6.72837        26.9721      0.00219237        91.5808
2025-04-13 12:19:50 +0000  80f35bd0-fb8a-4af0-830c-5e4240d69c1b  https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1911328930829176832                                                                   37.374           1.16559              1.77984       6.50982        25.1575      0.00193925        90.7012
2025-04-18 12:46:06 +0000  5161c287-ee85-4196-a39c-62b59d1d036d  https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1913140951249850368                                                                   42.0289          1.19714              1.83381       5.99746        25.1435      0.00205797        93.1571
2025-04-19 12:21:43 +0000  9cb213cc-b1f8-4819-8ee9-5c92086531ed  https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1913503093274710016                                                                   49.3949          1.06481              1.82749       6.61046        26.247       0.00223316        91.3276
2025-04-20 14:21:05 +0000  a6dc9ea4-3c6b-404a-90e4-2a5d69ff999d  https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1913865506993999872                                                                   36.1069          1.20301              1.95915       5.7773         25.3389      0.00212432        85.5051
```



Diff:
diff --git a/examples/metal-perfscale-cpt-virt-density.yaml b/examples/metal-perfscale-cpt-virt-density.yaml
new file mode 100644
index 00000000..37c888c6
--- /dev/null
+++ b/examples/metal-perfscale-cpt-virt-density.yaml
@@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: VMReady
+      metric_of_interest: P99
+      not:
+        jobConfig.name: "garbage-collection"
+      labels:
+        - "[Jira: PerfScale]"
+
+    - name:  apiserverCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-kube-apiserver
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: kube-apiserver]"
+
+    - name: multusCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-multus
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: multus]"
+
+    - name: monitoringCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-monitoring
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: monitoring]"
+
+    - name:  ovnCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-ovn-kubernetes
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: Networking / ovn-kubernetes]"
+
+    - name:  etcdCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-etcd
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: etcd]"
+
+    - name:  etcdDisk
+      metricName : 99thEtcdDiskBackendCommitDurationSeconds
+      metric_of_interest: value
+      agg:
+        value: duration
+        agg_type: avg
+      labels:
+        - "[Jira: etcd]"
+
+    - name: kubelet
+      metricName : kubeletCPU
+      metric_of_interest: value
+      labels:
+        - "[Jira: Node]"
+      agg:
+        value: cpu
+        agg_type: avg
diff --git a/test.bats b/test.bats
index afb22476..16de2c61 100644
--- a/test.bats
+++ b/test.bats
@@ -86,14 +86,17 @@ setup() {
   run_cmd orion cmd --config "examples/readout-control-plane-node-density.yaml" --hunter-analyze --output-format json --save-output-path=output.json --node-count True
 }
 
-
 @test "orion cmd readout netperf tcp with junit output " {
   export es_benchmark_index="k8s-netperf"
   run_cmd orion cmd --config "examples/readout-netperf-tcp.yaml" --output-format junit --hunter-analyze --save-output-path=output.xml
 }
 
+@test "orion cmd virt-density " {
+  run_cmd orion cmd --config examples/metal-perfscale-cpt-virt-density.yaml --lookback 15d --hunter-analyze
+}
+
 @test "orion cmd small scale cluster density with anomaly detection " {
-  run_cmd orion cmd --config "examples/small-scale-cluster-density.yaml" --lookback 5d --anomaly-detection 
+  run_cmd orion cmd --config "examples/small-scale-cluster-density.yaml" --lookback 5d --anomaly-detection
 }
 
 @test "orion cmd small scale node density cni anomaly detection with a window " {


Reviews:
user: login: josecastillolema
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T09:12:21Z
user: login: josecastillolema
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T09:14:52Z
user: login: jeniferh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-22T18:05:39Z
user: login: jtaleric
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-30T21:11:03Z
user: login: jeniferh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T21:14:38Z
user: login: jtaleric
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-01T15:09:07Z
user: login: josecastillolema
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-02T08:22:27Z
user: login: jtaleric
body: /lgtm
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-08T18:59:47Z
user: login: vishnuchalla
body: lgtm
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-08T19:00:58Z
user: login: chentex
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-20T09:23:26Z
user: login: rsevilla87
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T13:04:47Z
user: login: rsevilla87
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T13:10:08Z
user: login: rsevilla87
body: Can you add a test case for this @josecastillolema ?
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-20T14:29:34Z
user: login: rsevilla87
body: lgtm
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-27T13:42:54Z
user: login: chentex
body: /lgmt
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-28T13:11:25Z

Review comments:
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
+      metric_of_interest: P99
+      not:
+        jobConfig.name: "garbage-collection"
+      labels:
+        - "[Jira: PodLatency]"
+
+    - name:  apiserverCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-kube-apiserver
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: kube-apiserver]"
+
+    - name: multusCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-multus
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: multus]"
+
+    - name: monitoringCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-monitoring
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: monitoring]"
+
+    - name:  ovnCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-ovn-kubernetes
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: Networking / ovn-kubernetes]"
+
+    - name:  etcdCPU
+      metricName : containerCPU
+      labels.namespace.keyword: openshift-etcd
+      metric_of_interest: value
+      agg:
+        value: cpu
+        agg_type: avg
+      labels:
+        - "[Jira: etcd]"
+
+    - name:  etcdDisk
+      metricName : 99thEtcdDiskBackendCommitDurationSeconds
+      metric_of_interest: value
+      agg:
+        value: duration
+        agg_type: avg
+      labels:
+        - "[Jira: etcd]"
+
+    - name: kubelet
+      metricName : kubeletCPU
+      metric_of_interest: value
+      labels:
+        - "[Jira: Node]"
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: josecastillolema
body: Should we change the label here @jtaleric ?
created_at: 2025-04-22T09:12:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
+      metric_of_interest: P99
+      not:
+        jobConfig.name: "garbage-collection"
+      labels:
+        - "[Jira: VmiLatency]"
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: josecastillolema
body: Does this jira label sound good @jeniferh ?
created_at: 2025-04-22T09:14:52Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
+      metric_of_interest: P99
+      not:
+        jobConfig.name: "garbage-collection"
+      labels:
+        - "[Jira: VmiLatency]"
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: jeniferh
body: sounds good to me
created_at: 2025-04-22T18:05:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
+      metric_of_interest: P99
+      not:
+        jobConfig.name: "garbage-collection"
+      labels:
+        - "[Jira: VmiLatency]"
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: jtaleric
body: hm is that a proper Jira project? since it is specific to our team, just use `"[Jira: PerfScale]"`
created_at: 2025-04-30T21:11:03Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
+      metric_of_interest: P99
+      not:
+        jobConfig.name: "garbage-collection"
+      labels:
+        - "[Jira: VmiLatency]"
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: jeniferh
body: are these labels that somehow go to perfscale project? 
created_at: 2025-04-30T21:14:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
+      metric_of_interest: P99
+      not:
+        jobConfig.name: "garbage-collection"
+      labels:
+        - "[Jira: VmiLatency]"
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: jtaleric
body: that is the eventual plan if there is a regression. 
created_at: 2025-05-01T15:09:07Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
+      metric_of_interest: P99
+      not:
+        jobConfig.name: "garbage-collection"
+      labels:
+        - "[Jira: VmiLatency]"
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: josecastillolema
body: Updated to **[Jira: PerfScale]**
created_at: 2025-05-02T08:22:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: chentex
body: After viewing the code all seems good, but the data there is nothing matching `quantileName: Ready`
We should be using `quantileName: VMReady`
This fixes the Error and show data

```
orion cmd --config examples/metal-perfscale-cpt-virt-density.yaml --lookback 40d --hunter-analyze
2025-05-20 11:19:10,597 - Orion      - INFO - file: orion.py - line: 125 - 🏹 Starting Orion in command-line mode
2025-05-20 11:19:10,603 - Orion      - INFO - file: utils.py - line: 241 - The test metal-perfscale-cpt-virtdensity has started
2025-05-20 11:19:10,603 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=perf_scale_ci-*
2025-05-20 11:19:11,307 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-20 11:19:12,226 - Orion      - INFO - file: utils.py - line: 52 - Collecting vmiReadyLatency
2025-05-20 11:19:12,226 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-20 11:19:12,438 - Orion      - INFO - file: utils.py - line: 52 - Collecting apiserverCPU
2025-05-20 11:19:12,439 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-20 11:19:13,497 - Orion      - INFO - file: utils.py - line: 52 - Collecting multusCPU
2025-05-20 11:19:13,497 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-20 11:19:14,493 - Orion      - INFO - file: utils.py - line: 52 - Collecting monitoringCPU
2025-05-20 11:19:14,493 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-20 11:19:15,921 - Orion      - INFO - file: utils.py - line: 52 - Collecting ovnCPU
2025-05-20 11:19:15,921 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-20 11:19:17,310 - Orion      - INFO - file: utils.py - line: 52 - Collecting etcdCPU
2025-05-20 11:19:17,310 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-20 11:19:17,824 - Orion      - INFO - file: utils.py - line: 52 - Collecting etcdDisk
2025-05-20 11:19:17,824 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-20 11:19:18,029 - Orion      - INFO - file: utils.py - line: 52 - Collecting kubelet
2025-05-20 11:19:18,030 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
metal-perfscale-cpt-virtdensity
===============================
time                       uuid         buildUrl       vmiReadyLatency_P99    apiserverCPU_avg    multusCPU_avg    monitoringCPU_avg    ovnCPU_avg    etcdCPU_avg    etcdDisk_avg    kubelet_avg
-------------------------  --------------  --------------  ---------------------  ------------------  ---------------  -------------------  ------------  -------------  --------------  -------------
2025-04-18 12:47:04 +0000  5161c287 https:/     73959             42.0289          1.19714              1.83381       5.99746        25.1435      0.00205797        93.1571
2025-04-19 12:22:42 +0000  9cb213cc  https:/    74392             49.3949          1.06481              1.82749       6.61046        26.247       0.00223316        91.3276
```
created_at: 2025-05-20T09:23:18Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: rsevilla87
body: Just for the record, these are the supported fields:

https://github.com/kube-burner/kube-burner/blob/75d3a6f37e374bcdc4e169030af3c104784caec1/pkg/measurements/vmi_latency.go#L436-L445
created_at: 2025-05-20T13:04:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,92 @@
+tests :
+  - name : metal-perfscale-cpt-virtdensity
+    index: {{ es_metadata_index }}
+    benchmarkIndex: {{ es_benchmark_index }}
+    metadata:
+      platform: BareMetal
+      clusterType: self-managed
+      masterNodesType.keyword: ""
+      masterNodesCount: 3
+      workerNodesType.keyword: ""
+      workerNodesCount: 4
+      benchmark.keyword: virt-density
+      ocpVersion: {{ version }}
+      networkType: OVNKubernetes
+
+    metrics :
+    - name:  vmiReadyLatency
+      metricName: vmiLatencyQuantilesMeasurement
+      quantileName: Ready
path: examples/metal-perfscale-cpt-virt-density.yaml
user: login: rsevilla87
body: And yeah, **VMReady** is the right metric as @chentex said
created_at: 2025-05-20T13:10:08Z
author_association: MEMBER

PR comments:
user: login: josecastillolema
created_at: 2025-04-22T09:02:02Z
body: @jeniferh @capolrik besides **_vmiReadyLatency_** are you interested in tracking any particular components besides the ones that are already present in the file? Maybe some components of the CNV namespace? I need help understanding which components are the relevant ones
author_association: MEMBER
user: login: josecastillolema
created_at: 2025-04-22T09:02:39Z
body: @vishnuchalla @chentex @rsevilla87 @jtaleric  any ideas why **_vmiReadyLatency_** may be missing?
```
2025-04-22 10:53:42,232 - Orion      - ERROR - file: utils.py - line: 72 - Couldn't get metrics vmiReadyLatency, exception 'timestamp'
```
author_association: MEMBER
user: login: jeniferh
created_at: 2025-04-22T18:14:09Z
body: > @jeniferh @capolrik besides **_vmiReadyLatency_** are you interested in tracking any particular components besides the ones that are already present in the file? Maybe some components of the CNV namespace? I need help understanding which components are the relevant ones

I think its valuable to include the ones you have here for any potential pod control plane comparisons. Historically we've been tracking some virt-launcher pod memory usage but it hasn't proven to be very valuable lately. 

I think later we should explore adding some of the Kubevirt work queue metrics from the [upstream dashboard](https://github.com/kubevirt/monitoring/blob/main/dashboards/grafana/kubevirt-control-plane.json), but we'll need to experiment more with that. Also I think potentially tracking virt-api, virt-operator, virt-contoller, and virt-handler cpu & mem metrics (in openshift-cnv naemspace) could be interesting, but we can save that for when we have the metric collection in better shape.
author_association: CONTRIBUTOR
user: login: josecastillolema
created_at: 2025-05-09T13:50:01Z
body: Let's not merge this until we understand why **vmiReadyLatency** may be missing:
```
2025-04-22 10:53:42,232 - Orion      - ERROR - file: utils.py - line: 72 - Couldn't get metrics vmiReadyLatency, exception 'timestamp'
```

Asked @chentex to take a look into this :pleading_face: 
author_association: MEMBER
user: login: josecastillolema
created_at: 2025-05-27T13:06:55Z
body: Thanks a lot @chentex , looking good now:
```
orion cmd --config examples/metal-perfscale-cpt-virt-density.yaml --lookback 15d --hunter-analyze
2025-05-27 15:05:50,956 - Orion      - INFO - file: orion.py - line: 125 - 🏹 Starting Orion in command-line mode
2025-05-27 15:05:50,961 - Orion      - INFO - file: utils.py - line: 241 - The test metal-perfscale-cpt-virtdensity has started
2025-05-27 15:05:50,962 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=perf_scale_ci-*
2025-05-27 15:05:51,358 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-27 15:05:51,852 - Orion      - INFO - file: utils.py - line: 52 - Collecting vmiReadyLatency
2025-05-27 15:05:51,852 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-27 15:05:51,963 - Orion      - INFO - file: utils.py - line: 52 - Collecting apiserverCPU
2025-05-27 15:05:51,964 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-27 15:05:52,578 - Orion      - INFO - file: utils.py - line: 52 - Collecting multusCPU
2025-05-27 15:05:52,578 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-27 15:05:52,855 - Orion      - INFO - file: utils.py - line: 52 - Collecting monitoringCPU
2025-05-27 15:05:52,855 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-27 15:05:53,305 - Orion      - INFO - file: utils.py - line: 52 - Collecting ovnCPU
2025-05-27 15:05:53,305 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-27 15:05:53,799 - Orion      - INFO - file: utils.py - line: 52 - Collecting etcdCPU
2025-05-27 15:05:53,799 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-27 15:05:53,946 - Orion      - INFO - file: utils.py - line: 52 - Collecting etcdDisk
2025-05-27 15:05:53,947 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
2025-05-27 15:05:54,078 - Orion      - INFO - file: utils.py - line: 52 - Collecting kubelet
2025-05-27 15:05:54,079 - Matcher    - INFO - file: matcher.py - line: 65 - Executing query against index=ripsaw-kube-burner-*
metal-perfscale-cpt-virtdensity
===============================
time                       uuid                                  buildUrl                                                                                                                                                                    vmiReadyLatency_P99    apiserverCPU_avg    multusCPU_avg    monitoringCPU_avg    ovnCPU_avg    etcdCPU_avg    etcdDisk_avg    kubelet_avg
-------------------------  ------------------------------------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  ---------------------  ------------------  ---------------  -------------------  ------------  -------------  --------------  -------------
2025-05-15 12:27:50 +0000  ea0bff22-ffa7-43dc-85dd-dc4a022ed3ea  https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1922925098721873920                  76915             40.7914         1.21801               1.79307       7.43709        24.9181      0.00203687        92.8551
2025-05-16 12:18:17 +0000  e645e2c7-1b8e-4e42-ade1-5751fbd96639  https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1923287804293419008                  75964             35.2032         1.08224               1.78848       7.17251        26.5389      0.00225182        93.4423
2025-05-20 12:17:14 +0000  d9751153-c7da-43a7-869c-1fc0826826f4  https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-eng-ocp-qe-perfscale-ci-main-metal-4.19-latest-x86-daily-virt-6nodes/1924737115002769408                 128378             35.0956         0.946354              1.70518       5.90578        21.1809      0.00182848        70.7433
```
author_association: MEMBER
user: login: josecastillolema
created_at: 2025-05-27T13:15:19Z
body: Tests added @rsevilla87 , I think we are good to merge, ptal
author_association: MEMBER
user: login: josecastillolema
created_at: 2025-05-27T13:26:07Z
body: Btw look at the **vmiReadyLatency_P99** perf regression from the 16th to the 20th:
```
time                                  vmiReadyLatency_P99 
-------------------------            -------------------------
2025-05-15 12:27:50 +0000                   76915
2025-05-16 12:18:17 +0000                   75964
2025-05-20 12:17:14 +0000                  128378
```

Could this be because of the preload image issue @rsevilla87 ?
author_association: MEMBER
user: login: rsevilla87
created_at: 2025-05-27T13:42:29Z
body: > Btw look at the **vmiReadyLatency_P99** perf regression from the 16th to the 20th:
> 
> ```
> time                                  vmiReadyLatency_P99 
> -------------------------            -------------------------
> 2025-05-15 12:27:50 +0000                   76915
> 2025-05-16 12:18:17 +0000                   75964
> 2025-05-20 12:17:14 +0000                  128378
> ```
> 
> Could this be because of the preload image issue @rsevilla87 ?

difficult to say, but it could be, correct. Let's try to update the kb version asap in CPT
author_association: MEMBER
user: login: rsevilla87
created_at: 2025-05-27T13:45:46Z
body: Hey @chentex , we can't merge this one unless you give your blessings :)
author_association: MEMBER