Repository: red-hat-storage/cephci
Pull request #4684: CEPH-83611098 and CEPH-83611099
Author: login: jprakash-redhat
Created: 2025-04-14T10:39:19Z | Merged: 2025-04-21T12:26:39Z

# Description

1. CEPH-83611098 (Validate Warning at Ceph cluster on zero listener in a subsystem)
2. CEPH-83611099(Validate Warning message or Status on NVMe Service with single Gateway Node.)

```
All test logs located here: /Users/pjayaprakash/logs/CEPH_83611098_and_CEPH_83611099

TEST NAME                        TEST DESCRIPTION                                               DURATION                         STATUS                    COMMENTS
Validate Warning at Ceph clust   Warning at Ceph cluster on zero listener in a subsystem        0:42:40.413972                   Pass
Validate Warning message or St   Warning message or Status on NVMe Service with single Gatewa   0:18:14.199582                   Pass
```



Diff:
diff --git a/suites/squid/nvmeof/tier-2_nvmeof-alerts_health-checks-events.yaml b/suites/squid/nvmeof/tier-2_nvmeof-alerts_health-checks-events.yaml
index 69e0989ec26..3830b4a3810 100644
--- a/suites/squid/nvmeof/tier-2_nvmeof-alerts_health-checks-events.yaml
+++ b/suites/squid/nvmeof/tier-2_nvmeof-alerts_health-checks-events.yaml
@@ -139,3 +139,44 @@ tests:
       module: test_ceph_nvmeof_alerts_events.py
       name: Validate NVMeoFMissingListener alert
       polarion-id: CEPH-83611097
+
+  - test:
+      abort-on-fail: true
+      config:
+        gw_nodes:
+          - node6
+          - node5
+        rbd_pool: rbd
+        time_to_fire: 600
+        gw_group: gw_group1
+        listener_port: 4420
+        test_case: CEPH-83611098
+        cleanup:
+          - pool
+          - gateway
+      desc: Warning at Ceph cluster on zero listener in a subsystem
+      destroy-cluster: false
+      module: test_ceph_nvmeof_alerts_events.py
+      name: Validate Warning at Ceph cluster on zero listener in a subsystem
+      polarion-id: CEPH-83611098
+
+  - test:
+      abort-on-fail: true
+      config:
+        gw_nodes:
+          - node4
+          - node5
+        rbd_pool: rbd
+        time_to_fire: 300
+        gw_group: gw_group1
+        subsystems:                             # Configure subsystems
+          - nqn: nqn.2016-06.io.spdk:cnode1
+        test_case: CEPH-83611099
+        cleanup:
+          - pool
+          - gateway
+      desc: Warning message or Status on NVMe Service with single Gateway Node.
+      destroy-cluster: false
+      module: test_ceph_nvmeof_alerts_events.py
+      name: Validate Warning message or Status on NVMe Service with single Gateway Node.
+      polarion-id: CEPH-83611099
diff --git a/tests/nvmeof/test_ceph_nvmeof_alerts_events.py b/tests/nvmeof/test_ceph_nvmeof_alerts_events.py
index d5524c91bed..8fb2edd048a 100644
--- a/tests/nvmeof/test_ceph_nvmeof_alerts_events.py
+++ b/tests/nvmeof/test_ceph_nvmeof_alerts_events.py
@@ -2,6 +2,7 @@
 Test suite that verifies NVMe alerts and NVMe Health checks.
 """
 
+from copy import deepcopy
 from json import dumps, loads
 from random import choice
 from urllib.parse import urljoin, urlparse
@@ -427,10 +428,164 @@ def check_warning_status(gw, alert=True):
     )
 
 
+def test_ceph_83611098(ceph_cluster, config):
+    """[CEPH-83611098] Warning at Ceph cluster on zero listener in a subsystem.
+
+    NVMeoFZeroListenerSubsystem alert helps user to add listeners to the subsystem,
+    otherwise it would be not visible and able to connect from initators.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+
+    time_to_fire = config["time_to_fire"]
+    intervel = 100
+    alert = "NVMeoFZeroListenerSubsystem"
+    msg = "No listeners added to {subsystem_name} subsystem"
+    gateway_nodes = deepcopy(config.get("gw_nodes"))
+
+    LOG.info("Deploy nvme service")
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+
+    # Configure subsystems
+    nvmegwcl1 = ha.gateways[0]
+    subsystem1 = f"nqn.2016-06.io.spdk:cnode{generate_unique_id(4)}"
+    sub1_args = {"subsystem": subsystem1}
+    config.update({"nqn": subsystem1})
+    nvmegwcl1.subsystem.add(**{"args": {**sub1_args, **{"no-group-append": True}}})
+
+    # Check for alert
+    # NVMeoFZeroListenerSubsystem prometheus alert should be firing
+    LOG.info(
+        "NVMeoFZeroListenerSubsystem should be firing because no listener is configured"
+    )
+    events = PrometheusAlerts(ha.orch)
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(subsystem_name=subsystem1),
+        interval=intervel,
+    )
+
+    # Add the listener and check alert is inactive state or not
+    LOG.info("Add the listener")
+    configure_listeners(ha, [gateway_nodes[0]], config)
+    events.monitor_alert(
+        alert, timeout=time_to_fire, state="inactive", interval=intervel
+    )
+
+    # Delete the listener and alert should be in firing state
+    LOG.info("Delete the listener")
+    listener_args = {"nqn": subsystem1, "listener_port": config["listener_port"]}
+    configure_listeners(ha, [gateway_nodes[0]], listener_args, action="delete")
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(subsystem_name=subsystem1),
+        interval=intervel,
+    )
+
+    # Add the listener and check alert is inactive state or not
+    LOG.info("Add the listener back")
+    configure_listeners(ha, [gateway_nodes[0]], config)
+    events.monitor_alert(
+        alert, timeout=time_to_fire, state="inactive", interval=intervel
+    )
+
+    # Add one more subsystem and ceph prometheus alert to be in firing state
+    LOG.info("Add one more subsystem")
+    subsystem2 = f"nqn.2016-06.io.spdk:cnode{generate_unique_id(4)}"
+    sub2_args = {"subsystem": subsystem2}
+    nvmegwcl1.subsystem.add(**{"args": {**sub2_args, **{"no-group-append": True}}})
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(subsystem_name=subsystem2),
+        interval=intervel,
+    )
+
+    LOG.info("CEPH-83611098 - {alert} alert validated successfully.")
+
+
+def test_ceph_83611099(ceph_cluster, config):
+    """[CEPH-83611099] Warning message or Status on NVMe Service with single Gateway Node.
+
+    NVMeoFSingleGateway alert indicates to user to add more gateways
+    to the service in order to have High Availability.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+
+    time_to_fire = config["time_to_fire"]
+    intervel = 60
+    alert = "NVMeoFSingleGateway"
+    msg = "The gateway group {gw_group} consists of a single gateway - HA is not possible on cluster"
+
+    # Deploy nvmeof service
+    LOG.info("deploy nvme service")
+    gateway_nodes = deepcopy(config.get("gw_nodes"))
+    random_gateway = choice(gateway_nodes)
+    config.update({"gw_nodes": [random_gateway]})
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+    # Check for alert
+    # NVMeoFSingleGateway prometheus alert should be firing
+    events = PrometheusAlerts(ha.orch)
+
+    LOG.info("Check NVMeoFSingleGateway should be firing")
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(gw_group=config["gw_group"]),
+        interval=intervel,
+    )
+
+    # Add one more gateway and NVMeoFSingleGateway alert should be in inactive state
+    LOG.info("Add one more gateway and NVMeoFSingleGateway should be in inactive state")
+    config.update({"gw_nodes": gateway_nodes})
+    deploy_nvme_service(ceph_cluster, config)
+    events.monitor_alert(
+        alert, timeout=time_to_fire, state="inactive", interval=intervel
+    )
+
+    # scale down to one gateway and NVMeoFSingleGateway alert should be in firing state
+    LOG.info(
+        "scale down to one gateway and NVMeoFSingleGateway should be in firing state"
+    )
+    random_gateway = choice(gateway_nodes)
+    config.update({"gw_nodes": [random_gateway]})
+    deploy_nvme_service(ceph_cluster, config)
+    LOG.info(
+        "scaled down to one gateway, NVMeoFSingleGateway should be in firing state"
+    )
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(gw_group=config["gw_group"]),
+        interval=intervel,
+    )
+
+    # Scale up to 2 gateways and alert should be in inactive state
+    LOG.info("Add one more gateway and NVMeoFSingleGateway should be in inactive state")
+    config.update({"gw_nodes": gateway_nodes})
+    deploy_nvme_service(ceph_cluster, config)
+    events.monitor_alert(
+        alert, timeout=time_to_fire, state="inactive", interval=intervel
+    )
+
+    LOG.info("CEPH-83611099 - NVMeoFSingleGateway validated successfully.")
+
+
 testcases = {
     "CEPH-83610948": test_ceph_83610948,
     "CEPH-83610950": test_ceph_83610950,
     "CEPH-83611097": test_ceph_83611097,
+    "CEPH-83611098": test_ceph_83611098,
+    "CEPH-83611099": test_ceph_83611099,
 }
 
 


Reviews:
user: login: rahullepakshi
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T09:33:53Z
user: login: HaruChebrolu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T13:05:42Z
user: login: HaruChebrolu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T13:26:24Z
user: login: jprakash-redhat
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T14:28:26Z
user: login: jprakash-redhat
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T14:30:21Z
user: login: chebroluharika
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T10:46:28Z
user: login: rahullepakshi
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T10:51:33Z
user: login: HaruChebrolu
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T12:26:03Z

Review comments:
diff_hunk: @@ -427,10 +428,164 @@ def check_warning_status(gw, alert=True):
     )
 
 
+def test_ceph_83611098(ceph_cluster, config):
+    """[CEPH-83611098] Warning at Ceph cluster on zero listener in a subsystem.
+
+    NVMeoFZeroListenerSubsystem alert helps user to add listeners to the subsystem,
+    otherwise it would be not visible and able to connect from initators.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+
+    time_to_fire = 600
path: tests/nvmeof/test_ceph_nvmeof_alerts_events.py
user: login: HaruChebrolu
body: can we consider passing these via config for flexibility across environments. WDYT?
created_at: 2025-04-16T13:05:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -427,10 +428,164 @@ def check_warning_status(gw, alert=True):
     )
 
 
+def test_ceph_83611098(ceph_cluster, config):
+    """[CEPH-83611098] Warning at Ceph cluster on zero listener in a subsystem.
+
+    NVMeoFZeroListenerSubsystem alert helps user to add listeners to the subsystem,
+    otherwise it would be not visible and able to connect from initators.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+
+    time_to_fire = 600
+    intervel = 100
+    alert = "NVMeoFZeroListenerSubsystem"
+    msg = "No listeners added to {subsystem_name} subsystem"
+    gateway_nodes = deepcopy(config.get("gw_nodes"))
+
+    LOG.info("Deploy nvme service")
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+
+    # Configure subsystems
+    nvmegwcl1 = ha.gateways[0]
+    subsystem1 = f"nqn.2016-06.io.spdk:cnode{generate_unique_id(4)}"
+    sub1_args = {"subsystem": subsystem1}
+    config.update({"nqn": subsystem1})
+    nvmegwcl1.subsystem.add(**{"args": {**sub1_args, **{"no-group-append": True}}})
+
+    # Check for alert
+    # NVMeoFZeroListenerSubsystem prometheus alert should be firing
+    LOG.info(
+        "NVMeoFZeroListenerSubsystem should be firing because no listener is configured"
+    )
+    events = PrometheusAlerts(ha.orch)
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(subsystem_name=subsystem1),
+        interval=intervel,
+    )
+
+    # Add the listener and check alert is inactive state or not
+    LOG.info("Add the listener")
+    configure_listeners(ha, [gateway_nodes[0]], config)
+    events.monitor_alert(
+        alert, timeout=time_to_fire, state="inactive", interval=intervel
+    )
+
+    # Delete the listener and alert should be in firing state
+    LOG.info("Delete the listener")
+    listener_args = {"nqn": subsystem1, "listener_port": config["listener_port"]}
+    configure_listeners(ha, [gateway_nodes[0]], listener_args, action="delete")
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(subsystem_name=subsystem1),
+        interval=intervel,
+    )
+
+    # Add the listener and check alert is inactive state or not
+    LOG.info("Add the listener back")
+    configure_listeners(ha, [gateway_nodes[0]], config)
+    events.monitor_alert(
+        alert, timeout=time_to_fire, state="inactive", interval=intervel
+    )
+
+    # Add one more subsystem and ceph prometheus alert to be in firing state
+    LOG.info("Add one more subsystem")
+    subsystem2 = f"nqn.2016-06.io.spdk:cnode{generate_unique_id(4)}"
+    sub2_args = {"subsystem": subsystem2}
+    nvmegwcl1.subsystem.add(**{"args": {**sub2_args, **{"no-group-append": True}}})
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(subsystem_name=subsystem2),
+        interval=intervel,
+    )
+
+    LOG.info("CEPH-83611098 - {alert} alert validated successfully.")
+
+
+def test_ceph_83611099(ceph_cluster, config):
+    """[CEPH-83611099] Warning message or Status on NVMe Service with single Gateway Node.
+
+    NVMeoFSingleGateway alert indicates to user to add more gateways
+    to the service in order to have High Availability.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+
+    time_to_fire = 300
+    intervel = 60
+    alert = "NVMeoFSingleGateway"
+    msg = "The gateway group {gw_group} consists of a single gateway - HA is not possible on cluster"
+
+    # Deploy nvmeof service
+    LOG.info("deploy nvme service")
+    gateway_nodes = deepcopy(config.get("gw_nodes"))
+    random_gateway = choice(gateway_nodes)
+    config.update({"gw_nodes": [random_gateway]})
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+    # Check for alert
+    # NVMeoFSingleGateway prometheus alert should be firing
+    events = PrometheusAlerts(ha.orch)
+
+    LOG.info("Check NVMeoFSingleGateway should be firing")
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(gw_group=config["gw_group"]),
+        interval=intervel,
+    )
+
+    # Add one more gateway and NVMeoFSingleGateway alert should be in inactive state
+    LOG.info("Add one more gateway and NVMeoFSingleGateway should be in inactive state")
+    config.update({"gw_nodes": gateway_nodes})
+    deploy_nvme_service(ceph_cluster, config)
path: tests/nvmeof/test_ceph_nvmeof_alerts_events.py
user: login: HaruChebrolu
body: Is it necessary to include a sleep after deployment? Do the alerts start firing immediately once the deployment is complete?
created_at: 2025-04-16T13:26:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -427,10 +428,164 @@ def check_warning_status(gw, alert=True):
     )
 
 
+def test_ceph_83611098(ceph_cluster, config):
+    """[CEPH-83611098] Warning at Ceph cluster on zero listener in a subsystem.
+
+    NVMeoFZeroListenerSubsystem alert helps user to add listeners to the subsystem,
+    otherwise it would be not visible and able to connect from initators.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+
+    time_to_fire = 600
+    intervel = 100
+    alert = "NVMeoFZeroListenerSubsystem"
+    msg = "No listeners added to {subsystem_name} subsystem"
+    gateway_nodes = deepcopy(config.get("gw_nodes"))
+
+    LOG.info("Deploy nvme service")
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+
+    # Configure subsystems
+    nvmegwcl1 = ha.gateways[0]
+    subsystem1 = f"nqn.2016-06.io.spdk:cnode{generate_unique_id(4)}"
+    sub1_args = {"subsystem": subsystem1}
+    config.update({"nqn": subsystem1})
+    nvmegwcl1.subsystem.add(**{"args": {**sub1_args, **{"no-group-append": True}}})
+
+    # Check for alert
+    # NVMeoFZeroListenerSubsystem prometheus alert should be firing
+    LOG.info(
+        "NVMeoFZeroListenerSubsystem should be firing because no listener is configured"
+    )
+    events = PrometheusAlerts(ha.orch)
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(subsystem_name=subsystem1),
+        interval=intervel,
+    )
+
+    # Add the listener and check alert is inactive state or not
+    LOG.info("Add the listener")
+    configure_listeners(ha, [gateway_nodes[0]], config)
+    events.monitor_alert(
+        alert, timeout=time_to_fire, state="inactive", interval=intervel
+    )
+
+    # Delete the listener and alert should be in firing state
+    LOG.info("Delete the listener")
+    listener_args = {"nqn": subsystem1, "listener_port": config["listener_port"]}
+    configure_listeners(ha, [gateway_nodes[0]], listener_args, action="delete")
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(subsystem_name=subsystem1),
+        interval=intervel,
+    )
+
+    # Add the listener and check alert is inactive state or not
+    LOG.info("Add the listener back")
+    configure_listeners(ha, [gateway_nodes[0]], config)
+    events.monitor_alert(
+        alert, timeout=time_to_fire, state="inactive", interval=intervel
+    )
+
+    # Add one more subsystem and ceph prometheus alert to be in firing state
+    LOG.info("Add one more subsystem")
+    subsystem2 = f"nqn.2016-06.io.spdk:cnode{generate_unique_id(4)}"
+    sub2_args = {"subsystem": subsystem2}
+    nvmegwcl1.subsystem.add(**{"args": {**sub2_args, **{"no-group-append": True}}})
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(subsystem_name=subsystem2),
+        interval=intervel,
+    )
+
+    LOG.info("CEPH-83611098 - {alert} alert validated successfully.")
+
+
+def test_ceph_83611099(ceph_cluster, config):
+    """[CEPH-83611099] Warning message or Status on NVMe Service with single Gateway Node.
+
+    NVMeoFSingleGateway alert indicates to user to add more gateways
+    to the service in order to have High Availability.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+
+    time_to_fire = 300
+    intervel = 60
+    alert = "NVMeoFSingleGateway"
+    msg = "The gateway group {gw_group} consists of a single gateway - HA is not possible on cluster"
+
+    # Deploy nvmeof service
+    LOG.info("deploy nvme service")
+    gateway_nodes = deepcopy(config.get("gw_nodes"))
+    random_gateway = choice(gateway_nodes)
+    config.update({"gw_nodes": [random_gateway]})
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+    # Check for alert
+    # NVMeoFSingleGateway prometheus alert should be firing
+    events = PrometheusAlerts(ha.orch)
+
+    LOG.info("Check NVMeoFSingleGateway should be firing")
+    events.monitor_alert(
+        alert,
+        timeout=time_to_fire,
+        msg=msg.format(gw_group=config["gw_group"]),
+        interval=intervel,
+    )
+
+    # Add one more gateway and NVMeoFSingleGateway alert should be in inactive state
+    LOG.info("Add one more gateway and NVMeoFSingleGateway should be in inactive state")
+    config.update({"gw_nodes": gateway_nodes})
+    deploy_nvme_service(ceph_cluster, config)
path: tests/nvmeof/test_ceph_nvmeof_alerts_events.py
user: login: jprakash-redhat
body: Alerts won't fire immediately after the deployment is complete, it will take 300 seconds here to start fire.
created_at: 2025-04-16T14:28:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -427,10 +428,164 @@ def check_warning_status(gw, alert=True):
     )
 
 
+def test_ceph_83611098(ceph_cluster, config):
+    """[CEPH-83611098] Warning at Ceph cluster on zero listener in a subsystem.
+
+    NVMeoFZeroListenerSubsystem alert helps user to add listeners to the subsystem,
+    otherwise it would be not visible and able to connect from initators.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+
+    time_to_fire = 600
path: tests/nvmeof/test_ceph_nvmeof_alerts_events.py
user: login: jprakash-redhat
body: this is particular to this test. for each alert it will vary, so thought of keeping in script itself
created_at: 2025-04-16T14:30:21Z
author_association: CONTRIBUTOR

PR comments:
user: login: HaruChebrolu
created_at: 2025-04-16T14:43:27Z
body: @jprakash-redhat 
CEPH_83611098 - 
1. Don't we need to validate the zero listeners condition for multiple subsystems.
2. And do we need to validate the existence of alert after we bring down (stop/shutdown/power off) the gateway node where that listener was configured.
author_association: CONTRIBUTOR
user: login: jprakash-redhat
created_at: 2025-04-21T05:27:25Z
body: > @jprakash-redhat CEPH_83611098 -
> 
> 1. Don't we need to validate the zero listeners condition for multiple subsystems.
[JP] ---- Already we are validating alert for multiple subsystems
> 2. And do we need to validate the existence of alert after we bring down (stop/shutdown/power off) the gateway node where that listener was configured.
[JP] --- HA is not applicable here because once we configure listener alert will be in inactive state


author_association: CONTRIBUTOR