Repository: infrawatch/feature-verification-tests
Pull request #238: Add basic tests for VM power monitoring metrics
Author: login: rsafrono
Created: 2025-03-11T22:00:25Z | Merged: 2025-03-27T18:09:18Z

Added 2 tests:
Test idle VM power monitoring metrics are not zero 
Test dynamic VM power monitoring metrics are increasing


Diff:
diff --git a/ci/vars/osp18_env.yml b/ci/vars/osp18_env.yml
index 8ff4ad94..1d7f977f 100644
--- a/ci/vars/osp18_env.yml
+++ b/ci/vars/osp18_env.yml
@@ -6,3 +6,5 @@ stack_flavor: "m1.small"
 stack_network: "private"
 stack_external_network: "public"
 stack_name: "vnf"
+# kepler_test_vm should be created before executing the test suite
+kepler_test_vm: "kepler_test_vm"
diff --git a/roles/telemetry_verify_metrics/tasks/main.yml b/roles/telemetry_verify_metrics/tasks/main.yml
index dda94574..13774d0c 100644
--- a/roles/telemetry_verify_metrics/tasks/main.yml
+++ b/roles/telemetry_verify_metrics/tasks/main.yml
@@ -78,3 +78,21 @@
     file: verify_openstack_network_exporter_metrics.yml
   tags: test
   when: '"openstack_network_exporter" in telemetry_verify_metrics_metric_sources_to_test'
+
+- name: Check if kepler test VM is running
+  ansible.builtin.shell: |
+    {{ openstack_cmd }} server show {{ kepler_test_vm }} -c status -f value
+  register: kepler_vm_status
+  ignore_errors: true
+  changed_when: false
+  when: '"kepler_vm" in telemetry_verify_metrics_metric_sources_to_test'
+
+- name: Verify Kepler VM metrics are being exposed and stored
+  ansible.builtin.include_tasks:
+    file: verify_kepler_vm_metrics.yml
+  tags:
+    - test
+    - power_monitoring
+  when: '"kepler_vm" in telemetry_verify_metrics_metric_sources_to_test and "ACTIVE" in kepler_vm_status.stdout'
+  vars:
+    metrics_cmd_prefix: "{{ openstack_cmd }} metric show --disable-rbac -c value -f value"
diff --git a/roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml b/roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
new file mode 100644
index 00000000..590bbd0b
--- /dev/null
+++ b/roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
@@ -0,0 +1,33 @@
+- name: |
+    TEST Check that kepler shows idle power consumption metric for VM and it is not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    set -euxo pipefail
+    {{ metrics_cmd_prefix }} "kepler_vm_package_joules_total{'vm_id'='{{ kepler_test_vm }}', 'mode'='idle'}"
+  register: result
+  changed_when: false
+  failed_when: result.rc != 0 or result.stdout|float == 0
+
+- name: |
+    TEST Check that kepler shows dynamic metrics for VM and they are increasing
+    RHOSO-13952
+    Step 1 Get an initial value of VM dynamic metrics and check that it's not zero
+  ansible.builtin.shell: |
+    set -euxo pipefail
+    {{ metrics_cmd_prefix }} "kepler_vm_package_joules_total{'vm_id'='{{ kepler_test_vm }}', 'mode'='dynamic'}"
+  register: value1
+  changed_when: false
+  failed_when: value1.rc != 0 or value1.stdout|float == 0
+
+- name: |
+    TEST Check that kepler shows dynamic metrics for VM and they are increasing
+    RHOSO-13952
+    Step 2 Get a new value of VM dynamic metrics and compare with the initial value
+  ansible.builtin.shell: |
+    set -euxo pipefail
+    {{ metrics_cmd_prefix }} "kepler_vm_package_joules_total{'vm_id'='{{ kepler_test_vm }}', 'mode'='dynamic'}"
+  register: value2
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: value2.rc == 0 and (value2.stdout|float > value1.stdout|float)


Reviews:
user: login: elfiesmelfie
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-12T11:04:07Z
user: login: rsafrono
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T11:30:58Z
user: login: rsafrono
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T15:10:41Z
user: login: rsafrono
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T15:20:34Z
user: login: rsafrono
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T15:23:55Z
user: login: vyzigold
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T19:55:09Z
user: login: rsafrono
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-12T21:19:08Z
user: login: elfiesmelfie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-13T18:50:37Z
user: login: elfiesmelfie
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-13T18:55:13Z
user: login: rsafrono
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-16T09:36:16Z
user: login: elfiesmelfie
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-19T17:30:12Z
user: login: vyzigold
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T08:14:40Z
user: login: vyzigold
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-20T13:36:42Z
user: login: vyzigold
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T08:40:43Z
user: login: elfiesmelfie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-24T18:20:10Z
user: login: elfiesmelfie
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-24T18:20:42Z
user: login: rsafrono
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T06:11:11Z
user: login: vyzigold
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T07:11:11Z
user: login: elfiesmelfie
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-27T18:07:16Z

Review comments:
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: elfiesmelfie
body: Please add this as an ansible var, since it is used later as well.

I generally advise people to avoid using grep, etc, so that the debugging is easier when things go wrong. 
i.e. using grep means that when the expected value isn't there, the stdout is just blank, and it means we need to add some debugging tasks when trying to figure out what went wrong.

created_at: 2025-03-12T10:56:13Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    {{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep idle | cut -f3 -d ' '
+  register: result
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: result.rc == 0 and result.stdout|float != 0
+
+- name: |
+    TEST Check kepler shows dynamic (accumulated) power consumption metrics for VM and they are increasing
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: elfiesmelfie
body: Are these keplar VM metrics related to a workload created by OpenStack?
Or are they platform metrics? i.e. does keplar produce different metrics whether it's running on a VM vs baremetal?

I suspect that the first one is the case (i.e. workload metrics for a VM created by OpenStack.
In that case, where is the VM spawned, since this test won't work unless there is a VM?

created_at: 2025-03-12T10:58:31Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    {{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep idle | cut -f3 -d ' '
+  register: result
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: result.rc == 0 and result.stdout|float != 0
+
+- name: |
+    TEST Check kepler shows dynamic (accumulated) power consumption metrics for VM and they are increasing
+    RHOSO-13952
+  ansible.builtin.shell: |
+    set -x
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    value1=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    sleep 20
+    value2=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    (( $(echo "$value2 > $value1" | bc -l) ))
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: elfiesmelfie
body: Is it possible to replace the use of bc with some ansible?
Can the two values be converted to floats and then compared?

e.g. something like "(value2 | float) < (value1 | float)"
This could be used in the `failed_when` in the task.
created_at: 2025-03-12T11:01:21Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    {{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep idle | cut -f3 -d ' '
+  register: result
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: result.rc == 0 and result.stdout|float != 0
+
+- name: |
+    TEST Check kepler shows dynamic (accumulated) power consumption metrics for VM and they are increasing
+    RHOSO-13952
+  ansible.builtin.shell: |
+    set -x
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    value1=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    sleep 20
+    value2=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    (( $(echo "$value2 > $value1" | bc -l) ))
+  register: result
+  delay: 10
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: elfiesmelfie
body: I suggest splitting the task, so that there are two separate tasks for getting value1 and value2.

I don't think that the retries here are going to be effective, since you are always getting two values that are 20 second apart. They retries are insurance in case any part of the shell command fails in some other way.
If you split the task, you can get value1 plus wait the 20 seconds, and then retry getting value2 until it changes.

created_at: 2025-03-12T11:04:03Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    {{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep idle | cut -f3 -d ' '
+  register: result
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: result.rc == 0 and result.stdout|float != 0
+
+- name: |
+    TEST Check kepler shows dynamic (accumulated) power consumption metrics for VM and they are increasing
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: rsafrono
body: The metrics I plan to test here are openstack VM metrics that kepler collects. Kepler produces also other metrics but my team is focusing on VM metrics. The tests require a VM pre-spawned. I did it this way in order to avoid pushing workload creation/deletion commands into the tests repo since my team plans to run the tests on NFV baremetal environments and workload creation commands use many NFV specifics. So our CI jobs will handle creation and deletion of NFV-specific ports and servers (I consider adding one more server with different characteristics, e.g. other port types, more CPUs and RAM amount, to compare with the first one, in further tests).
The tests are disabled by default, in case anyone wants to run the tests need to take care of creating the VM with vm_id "kepler_test_vm" specified in ci/vars/osp18_env.yml and only then enable kepler_vm tests in telemetry_verify_metrics_metric_sources_to_test. I also added a comment in that file that VM should be pre-created.
created_at: 2025-03-12T11:30:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    {{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep idle | cut -f3 -d ' '
+  register: result
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: result.rc == 0 and result.stdout|float != 0
+
+- name: |
+    TEST Check kepler shows dynamic (accumulated) power consumption metrics for VM and they are increasing
+    RHOSO-13952
+  ansible.builtin.shell: |
+    set -x
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    value1=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    sleep 20
+    value2=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    (( $(echo "$value2 > $value1" | bc -l) ))
+  register: result
+  delay: 10
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: rsafrono
body: Thanks for the idea. I was thinking that test should be inside a single task. 
I've put now each test step to a separate task. Second one has retries until value increases. 
created_at: 2025-03-12T15:10:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    {{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep idle | cut -f3 -d ' '
+  register: result
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: result.rc == 0 and result.stdout|float != 0
+
+- name: |
+    TEST Check kepler shows dynamic (accumulated) power consumption metrics for VM and they are increasing
+    RHOSO-13952
+  ansible.builtin.shell: |
+    set -x
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    value1=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    sleep 20
+    value2=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    (( $(echo "$value2 > $value1" | bc -l) ))
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: rsafrono
body: done, replaced bc with ansible condition
created_at: 2025-03-12T15:20:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: rsafrono
body: done, added a var. Unfortunately, not sure how to replace grep. The CLI command does not provide proper filtering options. Added set -euxo pipefail to ensure task fail in case grep not produces an output
created_at: 2025-03-12T15:23:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: vyzigold
body: You can use the PromQL and specify the correct label values you want for the filtering. PromQL is quite powerful and the observabilityclient supports it fully. See an example of 2 commands here https://paste.opendev.org/show/bxm9HYpIvXLJtW39CQrA/ 

I don't have kepler VM metrics at hand right now to test it, but I guess a one of the commands you want here would look something like: `openstack metric query --disable-rbac -c value -f value "kepler_vm_platform_joules_total{'vm_id'='{{ kepler_test_vm }}', 'mode'='idle'}"`
created_at: 2025-03-12T19:55:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: rsafrono
body: thanks, updated the PR to use this approach
created_at: 2025-03-12T21:19:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    {{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep idle | cut -f3 -d ' '
+  register: result
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: result.rc == 0 and result.stdout|float != 0
+
+- name: |
+    TEST Check kepler shows dynamic (accumulated) power consumption metrics for VM and they are increasing
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: elfiesmelfie
body: If the VM is expected to be created externally, there should, IMHO, be a check in this role to make sure that the VM exists, and skip/fail the tests (with a warning/message) if it doesn't exist.

Something like: Can't run this test, pre-req not met. There is no VM called XXX.
This makes it so much easier to debug if someone naively enables the check for keplar vm metrics.

Alternatively, these tests can be run with the rest of the keplar metrics, if there's a VM present.
So, there's no additional config option to add for baremetal vs VM metrics i.e. check all the keplar metrics you can 

Whatever you choose to do, this dependency should be noted in the README for the role.


created_at: 2025-03-13T18:50:37Z
author_association: COLLABORATOR
diff_hunk: @@ -78,3 +78,13 @@
     file: verify_openstack_network_exporter_metrics.yml
   tags: test
   when: '"openstack_network_exporter" in telemetry_verify_metrics_metric_sources_to_test'
+
+- name: Verify Kepler VM metrics are being exposed and stored
+  ansible.builtin.include_tasks:
+    file: verify_kepler_vm_metrics.yml
+  tags:
+    - test
+    - power_monitoring
+  when: '"kepler_vm" in telemetry_verify_metrics_metric_sources_to_test'
path: roles/telemetry_verify_metrics/tasks/main.yml
user: login: elfiesmelfie
body: I suggest adding another check for the VM metrics. If `keplar in telemetry_verify_metrics_metric_sources_to_test `, AND the expected VM exists, then check for the VM metrics.

Does it need to be a particular VM, or could you fall back to any VM?

created_at: 2025-03-13T18:55:10Z
author_association: COLLABORATOR
diff_hunk: @@ -78,3 +78,13 @@
     file: verify_openstack_network_exporter_metrics.yml
   tags: test
   when: '"openstack_network_exporter" in telemetry_verify_metrics_metric_sources_to_test'
+
+- name: Verify Kepler VM metrics are being exposed and stored
+  ansible.builtin.include_tasks:
+    file: verify_kepler_vm_metrics.yml
+  tags:
+    - test
+    - power_monitoring
+  when: '"kepler_vm" in telemetry_verify_metrics_metric_sources_to_test'
path: roles/telemetry_verify_metrics/tasks/main.yml
user: login: rsafrono
body: Added the condition for the VM existence and status.

Regarding the fallback to any arbitrary VM, TBH, I would prefer checking particular VMs, since I have plans to check also (in some other test) that an extra VM with more CPUs and memory consumes more power. Also there is one more test (in plans) that executes stress-ng tool inside the VM. This supposes that VM is suitable for this.
created_at: 2025-03-16T09:36:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    {{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep idle | cut -f3 -d ' '
+  register: result
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: result.rc == 0 and result.stdout|float != 0
+
+- name: |
+    TEST Check kepler shows dynamic (accumulated) power consumption metrics for VM and they are increasing
+    RHOSO-13952
+  ansible.builtin.shell: |
+    set -x
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    value1=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    sleep 20
+    value2=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    (( $(echo "$value2 > $value1" | bc -l) ))
+  register: result
+  delay: 10
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: vyzigold
body: I have a similar worry as Roman. @elfiesmelfie do you think the XML reporting will be correct with the tasks split like this? The 2 tasks have the same test ID. Unfortunately we can't run it without baremetal machines to see what happens.
created_at: 2025-03-20T08:14:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -78,3 +78,20 @@
     file: verify_openstack_network_exporter_metrics.yml
   tags: test
   when: '"openstack_network_exporter" in telemetry_verify_metrics_metric_sources_to_test'
+
+- name: Check if kepler test VM is running
path: roles/telemetry_verify_metrics/tasks/main.yml
user: login: vyzigold
body: This task needs a "changed_when" according to the linter.
created_at: 2025-03-21T08:40:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+- name: |
+    TEST Check kepler shows idle power consumption metrics for VM and they are not zero
+    RHOSO-13583
+  ansible.builtin.shell: |
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    {{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep idle | cut -f3 -d ' '
+  register: result
+  delay: 10
+  retries: 3
+  changed_when: false
+  until: result.rc == 0 and result.stdout|float != 0
+
+- name: |
+    TEST Check kepler shows dynamic (accumulated) power consumption metrics for VM and they are increasing
+    RHOSO-13952
+  ansible.builtin.shell: |
+    set -x
+    params="--disable-rbac kepler_vm_platform_joules_total -c value -c mode -c vm_id -f value"
+    value1=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    sleep 20
+    value2=`{{ openstack_cmd }} metric show $params | grep {{ kepler_test_vm }} | grep dynamic | cut -f3 -d ' '`
+    (( $(echo "$value2 > $value1" | bc -l) ))
+  register: result
+  delay: 10
path: roles/telemetry_verify_metrics/tasks/verify_kepler_vm_metrics.yml
user: login: elfiesmelfie
body: If you're concerned, you can remove the TEST from both, and add an assert task below with the test in in.

e.g. 

Remove "TEST" from the two  tasks
Add a third task that asserts that `value2.rc == 0 and (value2.stdout|float > value1.stdout|float)` (or something similar)
created_at: 2025-03-24T18:20:10Z
author_association: COLLABORATOR
diff_hunk: @@ -78,3 +78,20 @@
     file: verify_openstack_network_exporter_metrics.yml
   tags: test
   when: '"openstack_network_exporter" in telemetry_verify_metrics_metric_sources_to_test'
+
+- name: Check if kepler test VM is running
path: roles/telemetry_verify_metrics/tasks/main.yml
user: login: elfiesmelfie
body: It can be `changed_when: false`
created_at: 2025-03-24T18:20:33Z
author_association: COLLABORATOR
diff_hunk: @@ -78,3 +78,20 @@
     file: verify_openstack_network_exporter_metrics.yml
   tags: test
   when: '"openstack_network_exporter" in telemetry_verify_metrics_metric_sources_to_test'
+
+- name: Check if kepler test VM is running
path: roles/telemetry_verify_metrics/tasks/main.yml
user: login: rsafrono
body: done
created_at: 2025-03-25T06:11:11Z
author_association: CONTRIBUTOR

PR comments:
user: login: softwarefactory-project-zuul[bot]
created_at: 2025-03-12T21:33:18Z
body: Build failed (check pipeline).  Post `recheck` (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/7db3980750a34dc9af575e486f8e7197

:x: [openstack-k8s-operators-content-provider ](https://softwarefactory-project.io/zuul/t/rdoproject.org/build/cf86aa1e3a2d43c7a24292ffbddae1bc) **FAILURE** in 15m 11s
:warning: [functional-metric-verification-tests-osp18 ](https://softwarefactory-project.io/zuul/t/rdoproject.org/build/ea548792fc8147c0980bb634104c130e) SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

author_association: NONE
user: login: rsafrono
created_at: 2025-03-12T21:39:03Z
body: recheck
author_association: CONTRIBUTOR
user: login: rsafrono
created_at: 2025-03-13T13:18:18Z
body: recheck
author_association: CONTRIBUTOR