Repository: red-hat-storage/cephci
Pull request #4628: CEPH-83610948 - NVMe healthcheck on gateway unavailability
Author: login: sunilkumarn417
Labels: name: nvmeof
name: 8.1
description: Ceph 8.1 feature automation
Created: 2025-04-01T16:18:27Z | Merged: 2025-04-07T08:18:31Z

# Description

This test case validates the warning which would be get hit on any gateway unavailability and Raise a healthcheck warning under ceph status and ceph health.


```
2025-04-02 16:37:19,566 - cephci - run:964 - INFO -
All test logs located here: /Users/sunilkumarn/logs/8_1_upstream-alerts-04

All test logs located here: /Users/sunilkumarn/logs/8_1_upstream-alerts-04

TEST NAME                        TEST DESCRIPTION                                               DURATION                         STATUS                    COMMENTS
Validate GW unavailability hea   GW failure or unavailability alert via healthcheck warning     0:03:25.185400                   Pass
```





Diff:
diff --git a/suites/squid/nvmeof/tier-2_nvmeof-alerts_health-checks-events.yaml b/suites/squid/nvmeof/tier-2_nvmeof-alerts_health-checks-events.yaml
new file mode 100644
index 00000000000..9ef864a9bc3
--- /dev/null
+++ b/suites/squid/nvmeof/tier-2_nvmeof-alerts_health-checks-events.yaml
@@ -0,0 +1,104 @@
+# NVMeoF alerts and testing
+# cluster configuration file: conf/squid/nvmeof/ceph_nvmeof_functional.yaml
+# Inventory: conf/inventory/rhel-9.3-server-x86_64-xlarge.yaml or later versions
+
+tests:
+# Set up the cluster
+  - test:
+      abort-on-fail: true
+      module: install_prereq.py
+      name: install ceph pre-requisites
+  - test:
+      abort-on-fail: true
+      config:
+        verify_cluster_health: true
+        steps:
+          - config:
+              command: bootstrap
+              service: cephadm
+              args:
+                mon-ip: node1
+                registry-url: registry.redhat.io
+                allow-fqdn-hostname: true
+                log-to-file: true
+          - config:
+              command: add_hosts
+              service: host
+              args:
+                attach_ip_address: true
+                labels: apply-all-labels
+          - config:
+              command: apply
+              service: mgr
+              args:
+                placement:
+                  label: mgr
+          - config:
+              command: apply
+              service: mon
+              args:
+                placement:
+                  label: mon
+          - config:
+              command: apply
+              service: osd
+              args:
+                all-available-devices: true
+      desc: RHCS cluster deployment using cephadm
+      destroy-cluster: false
+      module: test_cephadm.py
+      name: deploy cluster
+
+#  Test cases to be executed
+  - test:
+      abort-on-fail: true
+      config:
+        command: add
+        id: client.1
+        nodes:
+          - node10
+        install_packages:
+          - ceph-common
+        copy_admin_keyring: true
+      desc: Setup client on NVMEoF gateway
+      destroy-cluster: false
+      module: test_client.py
+      name: configure Ceph client for NVMe tests
+      polarion-id: CEPH-83573758
+
+  - test:
+      abort-on-fail: true
+      config:
+        gw_nodes:
+          - node6
+          - node5
+        rbd_pool: rbd
+        gw_group: gw_group1
+        do_not_create_image: true
+        rep-pool-only: true
+        rep_pool_config:
+          pool: rbd
+        subsystems:                             # Configure subsystems with all sub-entities
+          - nqn: nqn.2016-06.io.spdk:cnode1
+            serial: 1
+            bdevs:
+            - count: 2
+              size: 5G
+              lb_group: node6
+            - count: 2
+              size: 5G
+              lb_group: node5
+            listener_port: 4420
+            listeners:
+              - node6
+              - node5
+            allow_host: "*"
+        test_case: CEPH-83610948
+        cleanup:
+          - pool
+          - gateway
+      desc: GW failure or unavailability alert via healthcheck warning
+      destroy-cluster: false
+      module: test_ceph_nvmeof_alerts_events.py
+      name: Validate GW unavailability healthcheck warning
+      polarion-id: CEPH-83610948
diff --git a/tests/nvmeof/test_ceph_nvmeof_alerts_events.py b/tests/nvmeof/test_ceph_nvmeof_alerts_events.py
new file mode 100644
index 00000000000..68696c17333
--- /dev/null
+++ b/tests/nvmeof/test_ceph_nvmeof_alerts_events.py
@@ -0,0 +1,156 @@
+"""
+Test suite that verifies NVMe alerts and NVMe Health checks.
+"""
+
+from json import loads
+from random import choice
+
+from ceph.ceph import Ceph
+from ceph.nvmegw_cli import NVMeGWCLI
+from ceph.parallel import parallel
+from tests.nvmeof.test_ceph_nvmeof_high_availability import (
+    HighAvailability,
+    configure_subsystems,
+    deploy_nvme_service,
+    teardown,
+)
+from tests.rbd.rbd_utils import initial_rbd_config
+from utility.log import Log
+from utility.retry import retry
+
+LOG = Log(__name__)
+
+
+def test_ceph_83610948(ceph_cluster, config):
+    """[CEPH-83610948] Raise a healthcheck when a gateway is in UNAVAILABLE state.
+
+    This test case validates the warning which would be get hit on any gateway unavailability
+    and Raise a healthcheck warning under ceph status and ceph health.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+    rbd_pool = config["rbd_pool"]
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+
+    with parallel() as p:
+        for subsys_args in config["subsystems"]:
+            subsys_args["ceph_cluster"] = ceph_cluster
+            p.spawn(configure_subsystems, rbd_pool, ha, subsys_args)
+
+    # Validate the Health warning states
+    @retry(ValueError, tries=6, delay=5)
+    def check_warning_status(gw, alert=True):
+        health, _ = ha.orch.shell(args=["ceph", "health", "detail", "--format", "json"])
+        health = loads(health)
+
+        if not alert:
+            if "NVMEOF_GATEWAY_DOWN" not in health["checks"]:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted disappeared - {health}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="AVAILABLE"):
+                    return True
+            raise ValueError(f"NVMEOF_GATEWAY_DOWN warning still appears!!! - {health}")
+
+        if alert and "NVMEOF_GATEWAY_DOWN" in health["checks"]:
+            nvme_gw_down = health["checks"]["NVMEOF_GATEWAY_DOWN"]
+            if HEALTH_CHECK_WARN == nvme_gw_down:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted SUCCESSFULLY - {nvme_gw_down}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="UNAVAILABLE"):
+                    return True
+            raise ValueError(
+                f"NVMEOF_GATEWAY_DOWN is alerted, but different messages - {nvme_gw_down}"
+            )
+        raise ValueError(f"NVMEOF_GATEWAY_DOWN health warning not alerted - {health}.")
+
+    # Fail a Gateway
+    fail_gw = choice(ha.gateways)
+    HEALTH_CHECK_WARN = {
+        "severity": "HEALTH_WARN",
+        "summary": {
+            "message": "1 gateway(s) are in unavailable state; gateway might be down, try to redeploy.",
+            "count": 1,
+        },
+        "detail": [
+            {
+                "message": f"NVMeoF Gateway 'client.{fail_gw.daemon_name}' is unavailable."
+            },
+        ],
+        "muted": False,
+    }
+    if not ha.system_control(fail_gw, action="stop", wait_for_active_state=False):
+        raise Exception(f"[ {fail_gw.daemon_name} ] GW Stop failed..")
+    check_warning_status(fail_gw)
+
+    # Start GW back and alert should not be visible
+    if not ha.system_control(fail_gw, action="start"):
+        raise Exception(f"[ {fail_gw.daemon_name} ] GW Start failed..")
+    check_warning_status(fail_gw, alert=False)
+
+    LOG.info(
+        "CEPH-83610948 - GW Unavailability healthcheck warning validated successfully."
+    )
+
+
+testcases = {"CEPH-83610948": test_ceph_83610948}
+
+
+def run(ceph_cluster: Ceph, **kwargs) -> int:
+    """Return the status of the Ceph NVMEof HA test execution.
+
+    - Configure Gateways
+    - Configures Initiators and Run FIO on NVMe targets.
+    - Perform failover and failback.
+    - Validate the IO continuation prior and after to failover and failback
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        kwargs: Key/value pairs of configuration information to be used in the test.
+
+    Returns:
+        int - 0 when the execution is successful else 1 (for failure).
+
+    Example:
+
+        # Execute the nvmeof GW test
+            - test:
+                name: Ceph NVMeoF deployment
+                desc: Configure NVMEoF gateways and initiators
+                config:
+                    gw_nodes:
+                     - node6
+                     - node7
+                    rbd_pool: rbd
+                    do_not_create_image: true
+                    rep-pool-only: true
+                    cleanup-only: true                          # only for cleanup
+
+    """
+    LOG.info("Starting Ceph Ceph NVMEoF deployment.")
+    config = kwargs["config"]
+    rbd_obj = initial_rbd_config(**kwargs)["rbd_reppool"]
+
+    overrides = kwargs.get("test_data", {}).get("custom-config")
+    for key, value in dict(item.split("=") for item in overrides).items():
+        if key == "nvmeof_cli_image":
+            NVMeGWCLI.NVMEOF_CLI_IMAGE = value
+            break
+
+    try:
+        # NVMe alert test case to run
+        if config.get("test_case"):
+            test_case_run = testcases[config["test_case"]]
+            test_case_run(ceph_cluster, config)
+        return 0
+    except Exception as err:
+        LOG.error(err)
+    finally:
+        if config.get("cleanup"):
+            teardown(ceph_cluster, rbd_obj, config)
+
+    return 1


Reviews:
user: login: Manohar-Murthy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T12:06:30Z
user: login: Manohar-Murthy
body: Lets go through validation.
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T12:07:08Z
user: login: sunilkumarn417
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T05:17:58Z
user: login: Manohar-Murthy
body: lgtm.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T07:06:51Z
user: login: rahullepakshi
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T07:50:39Z
user: login: sunilkumarn417
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T11:36:22Z
user: login: rahullepakshi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T03:44:04Z
user: login: HaruChebrolu
body: lgtm
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T04:00:04Z
user: login: vamahaja
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T08:17:06Z

Review comments:
diff_hunk: @@ -0,0 +1,156 @@
+"""
+Test suite that verifies NVMe alerts and NVMe Health checks.
+"""
+
+from json import loads
+from random import choice
+
+from ceph.ceph import Ceph
+from ceph.nvmegw_cli import NVMeGWCLI
+from ceph.parallel import parallel
+from tests.nvmeof.test_ceph_nvmeof_high_availability import (
+    HighAvailability,
+    configure_subsystems,
+    deploy_nvme_service,
+    teardown,
+)
+from tests.rbd.rbd_utils import initial_rbd_config
+from utility.log import Log
+from utility.retry import retry
+
+LOG = Log(__name__)
+
+
+def test_ceph_83610948(ceph_cluster, config):
+    """[CEPH-83610948] Raise a healthcheck when a gateway is in UNAVAILABLE state.
+
+    This test case validates the warning which would be get hit on any gateway unavailability
+    and Raise a healthcheck warning under ceph status and ceph health.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+    rbd_pool = config["rbd_pool"]
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+
+    with parallel() as p:
+        for subsys_args in config["subsystems"]:
+            subsys_args["ceph_cluster"] = ceph_cluster
+            p.spawn(configure_subsystems, rbd_pool, ha, subsys_args)
+
+    # Validate the Health warning states
+    @retry(ValueError, tries=6, delay=5)
+    def check_warning_status(gw, alert=True):
+        health, _ = ha.orch.shell(args=["ceph", "health", "detail", "--format", "json"])
+        health = loads(health)
+
+        if not alert:
+            if "NVMEOF_GATEWAY_DOWN" not in health["checks"]:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted disappeared - {health}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="AVAILABLE"):
+                    return True
+            raise ValueError(f"NVMEOF_GATEWAY_DOWN warning still appears!!! - {health}")
+
+        if alert and "NVMEOF_GATEWAY_DOWN" in health["checks"]:
+            nvme_gw_down = health["checks"]["NVMEOF_GATEWAY_DOWN"]
+            if HEALTH_CHECK_WARN == nvme_gw_down:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted SUCCESSFULLY - {nvme_gw_down}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="UNAVAILABLE"):
+                    return True
+            raise ValueError(
+                f"NVMEOF_GATEWAY_DOWN is alerted, but different messages - {nvme_gw_down}"
+            )
+        raise ValueError(f"NVMEOF_GATEWAY_DOWN health warning not alerted - {health}.")
+
+    # Fail a Gateway
+    fail_gw = choice(ha.gateways)
+    HEALTH_CHECK_WARN = {
+        "severity": "HEALTH_WARN",
+        "summary": {
+            "message": "1 gateway(s) are in unavailable state; gateway might be down, try to redeploy.",
+            "count": 1,
+        },
+        "detail": [
+            {
+                "message": f"NVMeoF Gateway 'client.{fail_gw.daemon_name}' is unavailable."
+            },
+        ],
+        "muted": False,
+    }
+    if not ha.system_control(fail_gw, action="stop", wait_for_active_state=False):
+        raise Exception(f"[ {fail_gw.daemon_name} ] GW Stop failed..")
+    check_warning_status(fail_gw)
+
+    # Start GW back and alert should not be visible
path: tests/nvmeof/test_ceph_nvmeof_alerts_events.py
user: login: Manohar-Murthy
body: Why would alert not be visible once GW comes up? lets discuss on validation .
created_at: 2025-04-02T12:06:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+"""
+Test suite that verifies NVMe alerts and NVMe Health checks.
+"""
+
+from json import loads
+from random import choice
+
+from ceph.ceph import Ceph
+from ceph.nvmegw_cli import NVMeGWCLI
+from ceph.parallel import parallel
+from tests.nvmeof.test_ceph_nvmeof_high_availability import (
+    HighAvailability,
+    configure_subsystems,
+    deploy_nvme_service,
+    teardown,
+)
+from tests.rbd.rbd_utils import initial_rbd_config
+from utility.log import Log
+from utility.retry import retry
+
+LOG = Log(__name__)
+
+
+def test_ceph_83610948(ceph_cluster, config):
+    """[CEPH-83610948] Raise a healthcheck when a gateway is in UNAVAILABLE state.
+
+    This test case validates the warning which would be get hit on any gateway unavailability
+    and Raise a healthcheck warning under ceph status and ceph health.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+    rbd_pool = config["rbd_pool"]
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+
+    with parallel() as p:
+        for subsys_args in config["subsystems"]:
+            subsys_args["ceph_cluster"] = ceph_cluster
+            p.spawn(configure_subsystems, rbd_pool, ha, subsys_args)
+
+    # Validate the Health warning states
+    @retry(ValueError, tries=6, delay=5)
+    def check_warning_status(gw, alert=True):
+        health, _ = ha.orch.shell(args=["ceph", "health", "detail", "--format", "json"])
+        health = loads(health)
+
+        if not alert:
+            if "NVMEOF_GATEWAY_DOWN" not in health["checks"]:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted disappeared - {health}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="AVAILABLE"):
+                    return True
+            raise ValueError(f"NVMEOF_GATEWAY_DOWN warning still appears!!! - {health}")
+
+        if alert and "NVMEOF_GATEWAY_DOWN" in health["checks"]:
+            nvme_gw_down = health["checks"]["NVMEOF_GATEWAY_DOWN"]
+            if HEALTH_CHECK_WARN == nvme_gw_down:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted SUCCESSFULLY - {nvme_gw_down}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="UNAVAILABLE"):
+                    return True
+            raise ValueError(
+                f"NVMEOF_GATEWAY_DOWN is alerted, but different messages - {nvme_gw_down}"
+            )
+        raise ValueError(f"NVMEOF_GATEWAY_DOWN health warning not alerted - {health}.")
+
+    # Fail a Gateway
+    fail_gw = choice(ha.gateways)
+    HEALTH_CHECK_WARN = {
+        "severity": "HEALTH_WARN",
+        "summary": {
+            "message": "1 gateway(s) are in unavailable state; gateway might be down, try to redeploy.",
+            "count": 1,
+        },
+        "detail": [
+            {
+                "message": f"NVMeoF Gateway 'client.{fail_gw.daemon_name}' is unavailable."
+            },
+        ],
+        "muted": False,
+    }
+    if not ha.system_control(fail_gw, action="stop", wait_for_active_state=False):
+        raise Exception(f"[ {fail_gw.daemon_name} ] GW Stop failed..")
+    check_warning_status(fail_gw)
+
+    # Start GW back and alert should not be visible
path: tests/nvmeof/test_ceph_nvmeof_alerts_events.py
user: login: sunilkumarn417
body: if GW is active, alert is not required. I dont see the reason why we need alert for ACTIVE gateway.
created_at: 2025-04-03T05:17:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+"""
+Test suite that verifies NVMe alerts and NVMe Health checks.
+"""
+
+from json import loads
+from random import choice
+
+from ceph.ceph import Ceph
+from ceph.nvmegw_cli import NVMeGWCLI
+from ceph.parallel import parallel
+from tests.nvmeof.test_ceph_nvmeof_high_availability import (
+    HighAvailability,
+    configure_subsystems,
+    deploy_nvme_service,
+    teardown,
+)
+from tests.rbd.rbd_utils import initial_rbd_config
+from utility.log import Log
+from utility.retry import retry
+
+LOG = Log(__name__)
+
+
+def test_ceph_83610948(ceph_cluster, config):
+    """[CEPH-83610948] Raise a healthcheck when a gateway is in UNAVAILABLE state.
+
+    This test case validates the warning which would be get hit on any gateway unavailability
+    and Raise a healthcheck warning under ceph status and ceph health.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+    rbd_pool = config["rbd_pool"]
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+
+    with parallel() as p:
+        for subsys_args in config["subsystems"]:
+            subsys_args["ceph_cluster"] = ceph_cluster
+            p.spawn(configure_subsystems, rbd_pool, ha, subsys_args)
+
+    # Validate the Health warning states
+    @retry(ValueError, tries=6, delay=5)
+    def check_warning_status(gw, alert=True):
+        health, _ = ha.orch.shell(args=["ceph", "health", "detail", "--format", "json"])
+        health = loads(health)
+
+        if not alert:
+            if "NVMEOF_GATEWAY_DOWN" not in health["checks"]:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted disappeared - {health}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="AVAILABLE"):
+                    return True
+            raise ValueError(f"NVMEOF_GATEWAY_DOWN warning still appears!!! - {health}")
+
+        if alert and "NVMEOF_GATEWAY_DOWN" in health["checks"]:
+            nvme_gw_down = health["checks"]["NVMEOF_GATEWAY_DOWN"]
+            if HEALTH_CHECK_WARN == nvme_gw_down:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted SUCCESSFULLY - {nvme_gw_down}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="UNAVAILABLE"):
+                    return True
+            raise ValueError(
+                f"NVMEOF_GATEWAY_DOWN is alerted, but different messages - {nvme_gw_down}"
+            )
+        raise ValueError(f"NVMEOF_GATEWAY_DOWN health warning not alerted - {health}.")
+
+    # Fail a Gateway
+    fail_gw = choice(ha.gateways)
+    HEALTH_CHECK_WARN = {
+        "severity": "HEALTH_WARN",
+        "summary": {
+            "message": "1 gateway(s) are in unavailable state; gateway might be down, try to redeploy.",
+            "count": 1,
+        },
+        "detail": [
+            {
+                "message": f"NVMeoF Gateway 'client.{fail_gw.daemon_name}' is unavailable."
+            },
+        ],
+        "muted": False,
+    }
+    if not ha.system_control(fail_gw, action="stop", wait_for_active_state=False):
+        raise Exception(f"[ {fail_gw.daemon_name} ] GW Stop failed..")
+    check_warning_status(fail_gw)
+
+    # Start GW back and alert should not be visible
path: tests/nvmeof/test_ceph_nvmeof_alerts_events.py
user: login: rahullepakshi
body: We are actually listing GWs and GW groups in ceph -s from 8.1 active/inactive
created_at: 2025-04-03T07:49:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+"""
+Test suite that verifies NVMe alerts and NVMe Health checks.
+"""
+
+from json import loads
+from random import choice
+
+from ceph.ceph import Ceph
+from ceph.nvmegw_cli import NVMeGWCLI
+from ceph.parallel import parallel
+from tests.nvmeof.test_ceph_nvmeof_high_availability import (
+    HighAvailability,
+    configure_subsystems,
+    deploy_nvme_service,
+    teardown,
+)
+from tests.rbd.rbd_utils import initial_rbd_config
+from utility.log import Log
+from utility.retry import retry
+
+LOG = Log(__name__)
+
+
+def test_ceph_83610948(ceph_cluster, config):
+    """[CEPH-83610948] Raise a healthcheck when a gateway is in UNAVAILABLE state.
+
+    This test case validates the warning which would be get hit on any gateway unavailability
+    and Raise a healthcheck warning under ceph status and ceph health.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+    rbd_pool = config["rbd_pool"]
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+
+    with parallel() as p:
+        for subsys_args in config["subsystems"]:
+            subsys_args["ceph_cluster"] = ceph_cluster
+            p.spawn(configure_subsystems, rbd_pool, ha, subsys_args)
+
+    # Validate the Health warning states
+    @retry(ValueError, tries=6, delay=5)
+    def check_warning_status(gw, alert=True):
+        health, _ = ha.orch.shell(args=["ceph", "health", "detail", "--format", "json"])
+        health = loads(health)
+
+        if not alert:
+            if "NVMEOF_GATEWAY_DOWN" not in health["checks"]:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted disappeared - {health}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="AVAILABLE"):
+                    return True
+            raise ValueError(f"NVMEOF_GATEWAY_DOWN warning still appears!!! - {health}")
+
+        if alert and "NVMEOF_GATEWAY_DOWN" in health["checks"]:
+            nvme_gw_down = health["checks"]["NVMEOF_GATEWAY_DOWN"]
+            if HEALTH_CHECK_WARN == nvme_gw_down:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted SUCCESSFULLY - {nvme_gw_down}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="UNAVAILABLE"):
+                    return True
+            raise ValueError(
+                f"NVMEOF_GATEWAY_DOWN is alerted, but different messages - {nvme_gw_down}"
+            )
+        raise ValueError(f"NVMEOF_GATEWAY_DOWN health warning not alerted - {health}.")
+
+    # Fail a Gateway
+    fail_gw = choice(ha.gateways)
+    HEALTH_CHECK_WARN = {
+        "severity": "HEALTH_WARN",
+        "summary": {
+            "message": "1 gateway(s) are in unavailable state; gateway might be down, try to redeploy.",
+            "count": 1,
+        },
+        "detail": [
+            {
+                "message": f"NVMeoF Gateway 'client.{fail_gw.daemon_name}' is unavailable."
+            },
+        ],
+        "muted": False,
+    }
+    if not ha.system_control(fail_gw, action="stop", wait_for_active_state=False):
+        raise Exception(f"[ {fail_gw.daemon_name} ] GW Stop failed..")
+    check_warning_status(fail_gw)
+
+    # Start GW back and alert should not be visible
path: tests/nvmeof/test_ceph_nvmeof_alerts_events.py
user: login: sunilkumarn417
body: @rahullepakshi I did not get your comment. 
But on the GW DOWN alert we have two validations, 
1. Validate the warning message from `ceph health detail`.
2. Another one to check GW availability inside `check_warning_status()`
created_at: 2025-04-03T11:36:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,156 @@
+"""
+Test suite that verifies NVMe alerts and NVMe Health checks.
+"""
+
+from json import loads
+from random import choice
+
+from ceph.ceph import Ceph
+from ceph.nvmegw_cli import NVMeGWCLI
+from ceph.parallel import parallel
+from tests.nvmeof.test_ceph_nvmeof_high_availability import (
+    HighAvailability,
+    configure_subsystems,
+    deploy_nvme_service,
+    teardown,
+)
+from tests.rbd.rbd_utils import initial_rbd_config
+from utility.log import Log
+from utility.retry import retry
+
+LOG = Log(__name__)
+
+
+def test_ceph_83610948(ceph_cluster, config):
+    """[CEPH-83610948] Raise a healthcheck when a gateway is in UNAVAILABLE state.
+
+    This test case validates the warning which would be get hit on any gateway unavailability
+    and Raise a healthcheck warning under ceph status and ceph health.
+
+    Args:
+        ceph_cluster: Ceph cluster object
+        config: test case config
+    """
+    rbd_pool = config["rbd_pool"]
+    deploy_nvme_service(ceph_cluster, config)
+    ha = HighAvailability(ceph_cluster, config["gw_nodes"], **config)
+
+    with parallel() as p:
+        for subsys_args in config["subsystems"]:
+            subsys_args["ceph_cluster"] = ceph_cluster
+            p.spawn(configure_subsystems, rbd_pool, ha, subsys_args)
+
+    # Validate the Health warning states
+    @retry(ValueError, tries=6, delay=5)
+    def check_warning_status(gw, alert=True):
+        health, _ = ha.orch.shell(args=["ceph", "health", "detail", "--format", "json"])
+        health = loads(health)
+
+        if not alert:
+            if "NVMEOF_GATEWAY_DOWN" not in health["checks"]:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted disappeared - {health}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="AVAILABLE"):
+                    return True
+            raise ValueError(f"NVMEOF_GATEWAY_DOWN warning still appears!!! - {health}")
+
+        if alert and "NVMEOF_GATEWAY_DOWN" in health["checks"]:
+            nvme_gw_down = health["checks"]["NVMEOF_GATEWAY_DOWN"]
+            if HEALTH_CHECK_WARN == nvme_gw_down:
+                LOG.info(
+                    f"NVMEOF_GATEWAY_DOWN health check warning alerted SUCCESSFULLY - {nvme_gw_down}."
+                )
+                if ha.check_gateway_availability(gw.ana_group_id, state="UNAVAILABLE"):
+                    return True
+            raise ValueError(
+                f"NVMEOF_GATEWAY_DOWN is alerted, but different messages - {nvme_gw_down}"
+            )
+        raise ValueError(f"NVMEOF_GATEWAY_DOWN health warning not alerted - {health}.")
+
+    # Fail a Gateway
+    fail_gw = choice(ha.gateways)
+    HEALTH_CHECK_WARN = {
+        "severity": "HEALTH_WARN",
+        "summary": {
+            "message": "1 gateway(s) are in unavailable state; gateway might be down, try to redeploy.",
+            "count": 1,
+        },
+        "detail": [
+            {
+                "message": f"NVMeoF Gateway 'client.{fail_gw.daemon_name}' is unavailable."
+            },
+        ],
+        "muted": False,
+    }
+    if not ha.system_control(fail_gw, action="stop", wait_for_active_state=False):
+        raise Exception(f"[ {fail_gw.daemon_name} ] GW Stop failed..")
+    check_warning_status(fail_gw)
+
+    # Start GW back and alert should not be visible
path: tests/nvmeof/test_ceph_nvmeof_alerts_events.py
user: login: rahullepakshi
body: @sunilkumarn417 Please ignore that. this looks good. I can explain better once I have 8.1 cluster
created_at: 2025-04-04T03:44:04Z
author_association: CONTRIBUTOR

PR comments:
user: login: Manohar-Murthy
created_at: 2025-04-02T12:04:49Z
body: Are you checking alert from system status or from alerts pool?
author_association: CONTRIBUTOR
user: login: sunilkumarn417
created_at: 2025-04-03T05:37:40Z
body: > Are you checking alert from system status or from alerts pool?

This is Ceph Health warning check whichis available ceph status and ceph health detail. We dont have prometheus alert for this one.
author_association: CONTRIBUTOR
user: login: sunilkumarn417
created_at: 2025-04-03T05:37:50Z
body: > Lets go through validation.

Sure
author_association: CONTRIBUTOR
user: login: Manohar-Murthy
created_at: 2025-04-03T07:06:33Z
body: Thanks Sunil for walking me through validation. I was under the impression that this is a prometheous alert. 
Will approve the PR.
author_association: CONTRIBUTOR